Keeping and Dropping Objects
Another common use case is wanting to dropping or keeping entire labeled objects from the final result set. The keep
and drop
relabeling actions allow controlling this.
Depending on which type of relabeling configuration they are used in, they allow you to decide:
- ...which targets coming from service discovery should get scraped,
- ...which specific series samples to scrape from a target or send to remote storage,
- ...which alerts to send to Alertmanager.
NOTE: Prometheus 2.41 also added the
keepequal
anddropequal
actions that allow you to keep or drop a target based on the equality of source and target labels. The use cases for these actions are fairly uncommon though, so we won't cover them in detail here.
Rule structure
A keep
relabeling rule has the following structure:
action: keep
source_labels: [<source label name list>]
separator: <source labels separator> # Defaults to ';'
regex: <regular expression> # Defaults to '(.*)' (matching any value)
The keep
action performs the following steps, in sequence:
- It concatenates the values of the labels listed in
source_labels
using the providedseparator
value. - It tests whether the regular expression in
regex
matches the concatenated string from the previous step.- If there is no match, the object is removed from the final output list.
- If there is a match, the object is kept.
The drop
action works like keep
, but drops an object rather than keeping it.
Use case examples
Let's look at some example use cases for the keep
and drop
actions.
Scraping only targets with a specific scrape annotation
When using target relabeling, you may want to only scrape targets that have a specific metadata label (provided by the service discovery mechanism) that indicates that they should be scraped. For example, the following lets you only scrape targets that have a value for the example.io/should_be_scraped = true
annotation in Kubernetes:
action: keep
source_labels: [__meta_kubernetes_service_annotation_example_io_should_be_scraped]
regex: true
Note that the annotation name is just an example and not standardized.
Storing only specific metrics
When using metric_relabel_configs
to control how a target is scraped, you can use the following rule to only store metrics whose metric name starts with the prefix api_
or http_
:
action: keep
source_labels: [__name__]
regex: '(api_|http_).*'