Keeping and Dropping Labels
Less frequently, you may want to keep or drop individual labels from an object. For example, some targets supply a lot of unnecessary extra (non-identifying) labels on time series that are not interesting later on and just pollute both the TSDB and your query output. The labelkeep
and labeldrop
actions allow you to selectively keep or drop some labels.
Rule structure
A labelkeep
relabeling rule has the following structure:
action: labelkeep
regex: <regular expression> # Defaults to '(.*)' (matching any value)
The labelkeep
action performs the following steps, in sequence:
- It matches the regular expression in
regex
against all label names. - It keeps only those labels that match.
The labeldrop
action works like labelkeep
, but drops a label rather than keeping it.
Use case examples
Let's look at some example use cases for the labelkeep
action.
Removing HA replica labels from alerts
When running two identical Prometheus servers as a highly available (HA) pair, often both servers are configured to have an external label (via the global configuration option external_labels
) that indicates which replica they represent, e.g. replica: A
and replica: B
. Before sending alerts to the same Alertmanager instance from both replicas, Prometheus needs to remove this replica label so that Alertmanager does not interpret the incoming alerts as different (otherwise, you will get two notifications for the same alert, one with the label replica="A"
and one with replica="B"
!).
You can remove this replica
label with the following labeldrop
alert relabeling rule:
action: labeldrop
regex: replica
Removing unneeded labels from metrics
Some targets (like cAdvisor in the past) attach extra labels to each time series that are not necessary to uniquely identify each series, but just provide extra information about the target or series that you may not want to store.
To remove any labels starting with info_
from scraped metrics, you could write a metric relabeling rule like this:
action: labeldrop
regex: info_.*