Hidden Labels and Metadata
Labels starting with a double underscore (
__) are automatically removed after the last relabeling rule and thus do not make it into the final labels of an object. These specially prefixed labels can be used for storing and passing around extra metadata during the relabeling phase to make relabeling decisions or changes to the object's labels. For example, a service discovery mechanism may initially supply additional metadata for each target in these "hidden" labels.
Scrape control labels
For targets, some of these hidden labels have a special meaning and control how Prometheus should scrape a target:
__address__: This contains the TCP address that should be scraped for the target. It is initially set to the target's
<host>:<port>pair provided by the service discovery mechanism. After relabeling, Prometheus sets the
instancelabel to the value of
__address__if you do not set the
instancelabel explicitly to another value before that.
__scheme__: This contains the HTTP scheme (
https) with which the target should be scraped. Defaults to
__metrics_path__: This contains the HTTP path to scrape metrics from. Defaults to
__param_<name>: This contains HTTP query parameter names and their values that should be sent along with a scrape request.
You can set or override each of these labels using relabeling rules to produce custom scrape behaviors for individual targets.
Service discovery metadata labels
Additionally, service discovery mechanisms can provide a set of labels starting with
__meta_ that contain discovery-specific metadata about a target. For example, when discovering pods in a Kubernetes cluster, the Kubernetes service discovery engine will provide a
__meta_kubernetes_pod_name label for each pod target, containing the name of the discovered pod, and a
__meta_kubernetes_pod_ready label indicating whether the pod is in the ready state or not. You can find all the labels that the Kubernetes service discovery provides in the configuration documentation. Other service discovery mechanisms provide similar metadata.
If you need to store a temporary label value in a relabeling step (for use in a subsequent relabeling step), you can use the
__tmp label name prefix. Labels starting with
__tmp are guaranteed to never be used by Prometheus itself.