Hidden Labels and Metadata
Labels starting with a double underscore (__
) are automatically removed after the last relabeling rule and thus do not make it into the final labels of an object. These specially prefixed labels can be used for storing and passing around extra metadata during the relabeling phase to make relabeling decisions or changes to the object's labels. For example, a service discovery mechanism may initially supply additional metadata for each target in these "hidden" labels.
Scrape control labels
For targets, some of these hidden labels have a special meaning and control how Prometheus should scrape a target:
__address__
: This contains the TCP address that should be scraped for the target. It is initially set to the target's<host>:<port>
pair provided by the service discovery mechanism. After relabeling, Prometheus sets theinstance
label to the value of__address__
if you do not set theinstance
label explicitly to another value before that.__scheme__
: This contains the HTTP scheme (http
orhttps
) with which the target should be scraped. Defaults tohttp
.__metrics_path__
: This contains the HTTP path to scrape metrics from. Defaults to/metrics
.__param_<name>
: This contains HTTP query parameter names and their values that should be sent along with a scrape request.
You can set or override each of these labels using relabeling rules to produce custom scrape behaviors for individual targets.
Service discovery metadata labels
Additionally, service discovery mechanisms can provide a set of labels starting with __meta_
that contain discovery-specific metadata about a target. For example, when discovering pods in a Kubernetes cluster, the Kubernetes service discovery engine will provide a __meta_kubernetes_pod_name
label for each pod target, containing the name of the discovered pod, and a __meta_kubernetes_pod_ready
label indicating whether the pod is in the ready state or not. You can find all the labels that the Kubernetes service discovery provides in the configuration documentation. Other service discovery mechanisms provide similar metadata.
Temporary labels
If you need to store a temporary label value in a relabeling step (for use in a subsequent relabeling step), you can use the __tmp
label name prefix. Labels starting with __tmp
are guaranteed to never be used by Prometheus itself.