In rare circumstances, you may want to scale a Prometheus setup horizontally by running multiple almost identical Prometheus server replicas, but each scraping only a subset of all targets of a service. The hashmod relabeling action (combined with the keep action) allows us to shard targets for this scenario.

Rule structure

A hashmod relabeling rule has the following structure:

action: hashmod
source_labels: [<source label name list>]
modulus: <modulus value>
target_label: <target label>

This action performs the following steps, in sequence:

1. It concatenates the values of the labels listed in source_labels using the provided separator value.
2. It calculates the hash (currently implemented as an MD5 sum) of the concatenated string.
3. It applies the modulus provided in modulus to the hash (operation: hash % modulus), to constrain the hash to a value between 0 and modulus - 1.
4. It stores the modulus value from the previous step in the label provided as target_label.

Use case examples

Let's look at an example use case for the hashmod action.

Sharding targets in a service

The prime use case for using the hashmod action is to split up the overall group of targets of a service into shards for each replica of a horizontally-scaled Prometheus server setup. This is achieved by first calculating the hash-based modulus for each target based on one or multiple of its labels, and then only keeping targets that have a specific output modulus value. For example, to shard targets on their instance label and only keep the instances for shard 2 (out of 10 shards), you could combine a hashmod with a keep action like this:

- action: hashmod
  source_labels: [instance]
  modulus: 10
  target_label: __tmp_hashmod
- action: keep
  source_labels: [__tmp_hashmod]
  regex: 2