Keeping and Dropping Objects

Another common use case is wanting to dropping or keeping entire labeled objects from the final result set. The keep and drop relabeling actions allow controlling this.

Depending on which type of relabeling configuration they are used in, they allow you to decide:

  • ...which targets coming from service discovery should get scraped,
  • ...which specific series samples to scrape from a target or send to remote storage,
  • ...which alerts to send to Alertmanager.

NOTE: Prometheus 2.41 also added the keepequal and dropequal actions that allow you to keep or drop a target based on the equality of source and target labels. The use cases for these actions are fairly uncommon though, so we won't cover them in detail here.

Rule structure

A keep relabeling rule has the following structure:

action: keep
source_labels: [<source label name list>]
separator: <source labels separator>      # Defaults to ';'
regex: <regular expression>               # Defaults to '(.*)' (matching any value)

The keep action performs the following steps, in sequence:

  1. It concatenates the values of the labels listed in source_labels using the provided separator value.
  2. It tests whether the regular expression in regex matches the concatenated string from the previous step.
    • If there is no match, the object is removed from the final output list.
    • If there is a match, the object is kept.

The drop action works like keep, but drops an object rather than keeping it.

Use case examples

Let's look at some example use cases for the keep and drop actions.

Scraping only targets with a specific scrape annotation

When using target relabeling, you may want to only scrape targets that have a specific metadata label (provided by the service discovery mechanism) that indicates that they should be scraped. For example, the following lets you only scrape targets that have a value for the = true annotation in Kubernetes:

action: keep
source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true

Note that the annotation name is just an example and not standardized.

Storing only specific metrics

When using metric_relabel_configs to control how a target is scraped, you can use the following rule to only store metrics whose metric name starts with the prefix api_ or http_:

action: keep
source_labels: [__name__]
regex: '(api_|http_).*'