Keeping and Dropping Labels

Less frequently, you may want to keep or drop individual labels from an object. For example, some targets supply a lot of unnecessary extra (non-identifying) labels on time series that are not interesting later on and just pollute both the TSDB and your query output. The labelkeep and labeldrop actions allow you to selectively keep or drop some labels.

Rule structure

A labelkeep relabeling rule has the following structure:

action: labelkeep
regex: <regular expression>  # Defaults to '(.*)' (matching any value)

The labelkeep action performs the following steps, in sequence:

  1. It matches the regular expression in regex against all label names.
  2. It keeps only those labels that match.

The labeldrop action works like labelkeep, but drops a label rather than keeping it.

Use case examples

Let's look at some example use cases for the labelkeep action.

Removing HA replica labels from alerts

When running two identical Prometheus servers as a highly available (HA) pair, often both servers are configured to have an external label (via the global configuration option external_labels) that indicates which replica they represent, e.g. replica: A and replica: B. Before sending alerts to the same Alertmanager instance from both replicas, Prometheus needs to remove this replica label so that Alertmanager does not interpret the incoming alerts as different (otherwise, you will get two notifications for the same alert, one with the label replica="A" and one with replica="B"!).

You can remove this replica label with the following labeldrop alert relabeling rule:

action: labeldrop
regex: replica

Removing unneeded labels from metrics

Some targets (like cAdvisor in the past) attach extra labels to each time series that are not necessary to uniquely identify each series, but just provide extra information about the target or series that you may not want to store.

To remove any labels starting with info_ from scraped metrics, you could write a metric relabeling rule like this:

action: labeldrop
regex: info_.*