Keeping and Dropping Objects

Another common use case is wanting to filter labeled objects. The keep and drop relabeling actions allow controlling this.

Depending on which type of relabeling configuration they are used in, they allow us to decide:

  • ...which targets coming from service discovery should get scraped,
  • ...which specific series samples to scrape from a target or send to remote storage,
  • ...which alerts to send to Alertmanager.

Rule Structure

A keep relabeling rule has the following structure:

action: keep
source_labels: [<source label name list>]
separator: <source labels separator>      # Defaults to ';'
regex: <regular expression>               # Defaults to '(.*)' (matching any value)

The keep action performs the following steps, in sequence:

  1. It concatenates the values of the labels listed in source_labels using the provided separator value.
  2. It tests whether the regular expression in regex matches the concatenated string from the previous step.

    • If there is no match, the object is removed from the final output list.
    • If there is a match, the object is kept.

The drop action works like keep, but drops an object rather than keeping it.

Use Case Examples

Let's look at some example use cases for the keep and drop actions.

Scraping Only Targets with a Scrape Annotation

When using target relabeling, you may want to only scrape targets that have a specific metadata label (provided by the service discovery mechanism) that indicates that they should be scraped. For example, the following lets you only scrape targets that have a value for the example.io/should_be_scraped = true annotation in Kubernetes:

action: keep
source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true

Not that the annotation name is just an example and not standardized.

Storing Only Specific Metrics

When using metric_relabel_configs to control how a target is scraped, you can use the following rule to only store metrics whose metric name starts with the prefix api_ or http_:

action: keep
source_labels: [__name__]
regex: '(api_|http_).*'