Hidden Labels and Metadata

Hidden labels

Labels starting with a double underscore (__) are treated specially in that they are removed after relabeling. A source of labeled objects (such as a service discovery mechanism producing labeled targets) may initially attach these "hidden" labels to provide extra metadata about a labeled object. These special labels can be used during the relabeling phase to make decisions or changes to the object's labels.

Scrape control labels

For targets, some of these hidden labels have a special meaning and control how a target should be scraped:

  • __address__: Contains the TCP address that should be scraped for the target. It initially defaults to the <host>:<port> pair provided by the service discovery mechanism. Prometheus sets the instance label to the value of __address__ after relabeling if you do not set the instance label explicitly to another value before that.
  • __scheme__: Contains the HTTP scheme (http or https) with which the target should be scraped. Defaults to http.
  • __metrics_path__: Contains the HTTP path to scrape metrics from. Defaults to /metrics.
  • __param_<name>: Contains HTTP query parameter names and their values.

Each of these labels can be set or overwritten using relabeling rules to produce custom scrape behaviors for individual targets.

Service discovery metadata labels

Additionally, service discovery mechanisms can provide a set of labels starting with __meta_ that contain discovery-specific metadata about a target. For example, when discovering pods in a Kubernetes cluster, the Kubernetes service discovery engine will provide a __meta_kubernetes_pod_name label for each pod target, containing the name of the discovered pod, and a __meta_kubernetes_pod_ready label indicating whether the pod is in the ready state or not. You can find all the labels that the Kubernetes service discovery provides in the configuration documentation.

Temporary labels

If a relabeling step needs to save a value to a temporary label (to be processed in a subsequent relabeling step), you can use the __tmp label name prefix. Labels starting with __tmp are guaranteed to never be used by Prometheus itself.