Hashing and Sharding on Label Values
In rare circumstances, you may want to scale a Prometheus setup horizontally by running multiple almost identical Prometheus server replicas, but each scraping only a subset of all targets of a service. The
hashmod relabeling action helps us in sharding targets for this scenario.
hashmod relabeling rule has the following structure:
action: replace source_labels: [<source label name list>] modulus: <modulus value> target_label: <target label>
This action performs the following steps, in sequence:
- It concatenates the values of the labels listed in
source_labelsusing the provided
- It calculates the hash (currently implemented as an MD5 sum) of the concatenated string.
- It applies the modulus provided in
modulusto the hash, to constrain the hash to a value between
modulus - 1.
- It stores the modulus value from the previous step in the label provided as
Use Case Examples
Let's look at an example use case for the
The prime use case for using the
hashmod action is to split up the overall group of targets of a service into shards for each replica of a horizontally-scaled Prometheus server setup. This is achieved by first calculating the hash-based modulus for each target based on one or multiple of its labels, and then only keeping targets that have a specific output modulus value. For example, to shard targets on their
instance label and only keeping the instances for shard 2 (out of 10 shards), you could combine a
hashmod with a
keep action like this:
- action: hashmod source_labels: [instance] modulus: 10 target_label: __tmp_hashmod - action: keep source_labels: [__tmp_hashmod] regex: 2