Keeping and Dropping Labels
Less frequently, you may want to keep or drop individual labels from an object. For example, some targets supply a lot of unnecessary extra (non-identifying) labels on time series that are not interesting later on and just pollute both the TSDB and querying use cases. The
labeldrop actions allow us to selectively keep or drop some labels.
labelkeep relabeling rule has the following structure:
action: labelkeep regex: <regular expression> # Defaults to '(.*)' (matching any value)
labelkeep action performs the following steps, in sequence:
- It matches the regular expression in
regexagainst all label names.
- It keeps only those labels that match.
labeldrop action works like
labelkeep, but drops a label rather than keeping it.
Use Case Examples
Let's look at an example use case for the
Removing HA Replica Labels from Alerts
When running two identical Prometheus servers as a highly available (HA) pair, often both servers are configured to have an external label (via the global configuration option
external_labels) that indicates which replica they represent, e.g.
replica: A and
replica: B. Before sending alerts to the same Alertmanager instance from both replicas, Prometheus needs to remove this replica label so that Alertmanager does not interpret the incoming alerts as different (otherwise, you will get two notifications for the same alert!).
You can achieve that with the following
action: labeldrop regex: replica
Removing Unneeded Labels from Metrics
Some targets (like cAdvisor in the past) attach extra labels to each time series that are not necessary to uniquely identify each series, but just provide extra information about the target or series that you may not want to store.
To remove any labels starting with
info_, you could write a rule like this:
action: labeldrop regex: info_.*