You can define and load recording rules into Prometheus in a two-step process:

1. Define your recording rules in a separate rule file.
2. Load the rule file into Prometheus by referencing it from the main Prometheus configuration file.

Creating rule files

You can define recording rules in so-called rule files. Rule files are YAML-based configuration files that are separate from the main Prometheus configuration file. Each rule file contains a list of rule groups (top-level groups field), and each group may in turn contain both recording rules and alerting rules. We cover alerting rules in our Alerting with Prometheus training and will only focus on recording rules in this training. Each rule group has the following fields:

• name: The group name is used to identify a rule group in the Prometheus UI (and in metrics about the rule evaluation itself).
• interval: (optional) This is the interval at which Prometheus should evaluate rules in this group. If not specified, the global default evaluation interval (global.evaluation_interval setting) is used. Use custom per-group intervals with caution, as having different recording intervals in one server can be confusing.
• limit: (optional) A numeric upper limit for the number of output series that each recording or alerting rule is allowed to produce. You can set this as a safeguard in case you are worried about certain rules creating too many output series for Prometheus to handle.
• rules: The actual list of recording and alerting rules in this group.

Recording rules are defined in the rules list of a rule group, and are distinguished from alerting rules by having a record field (vs. the alert field).

Each recording rule has the following fields:

• record: The name of the output metric. This is the metric name that will be used for the resulting stored time series.
• expr: The PromQL expression to evaluate. The result of the expression is stored as a new set of time series with the name specified in the record field.
• labels: (optional) A set of labels to attach to the resulting time series before storing them. These labels will override any conflicting labels on the output time series before the result is stored. Setting additional labels like this is rarely required for recording rules.

The following example shows a rule file (e.g. my-recording-rules.yml) that defines two recording rules:

groups:
- name: my-recording-rules
  rules:
  - record: path:http_requests:rate5m # The new output metric name.
    expr: sum without(instance) (rate(http_requests_total{job="my-job"}[5m]))
  - record: instance_mode:node_cpu:rate1m # The new output metric name.
    expr: sum without(cpu) (rate(node_cpu_seconds_total{job="node"}[1m]))
    labels:
      my_label: my_value

Note that the output metric names consist of colon-separated components (with colons being one of the valid characters allowed in metric names) – this is just a voluntary naming convention that we will discuss in the next section.

The example above contains a single rule group with the name my-recording-rules. All rules in the same group are evaluated in sequence, while rules from different groups are evaluated in parallel. This means that the path:http_requests:rate5m rule will be evaluated before the instance_mode:node_cpu:rate1m rule, since they are part of the same group. This is important to keep in mind when writing rules, especially when the output metric name of one rule is used as an input for another and you don't want to risk using a stale result from a previous evaluation cycle as your input for the current one.

For many use cases, keeping all of your rules in a single group is sufficient, but you can also choose to split your rules into multiple groups if you either want to increase the evaluation efficiency (via parallelization), or if you want to organize your rules into thematically named groups so that they become easier to understand and maintain.

Loading rule files into Prometheus

To load a rule file into Prometheus for evaluation, you need to reference it from the main Prometheus configuration file. You can do this by adding a rule_files section to the top level of the configuration file, which specifies a list of file paths that should be loaded.

The following example rule_files stanza shows how to load the rule file from the previous section into Prometheus:

rule_files:
- "my-recording-rules.yml"