Let's create an example recording rule based on the metrics of a demo service that we are going to monitor. First, let's monitor three instances of the demo service and load our rule file (that we will create in the next step) into Prometheus by adding a rule_files configuration directive to the top level of your Prometheus configuration file.

Set your Prometheus configuration file to the following contents:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - demo-rules.yml

scrape_configs:
  - job_name: 'demo'
    static_configs:
      - targets:
        - 'demo.promlabs.com:10000'
        - 'demo.promlabs.com:10001'
        - 'demo.promlabs.com:10002'

The demo service exposes a demo_api_request_duration_seconds_count counter metric (derived from a histogram, thus the duration_seconds component in the base name of the metric) split up by the path, method, and status labels.

Based on that metric, we could calculate the total request rate for each path/method combination, aggregated over the entire job and all status codes, like this:

sum without(status, instance) (
  rate(demo_api_request_duration_seconds_count{job="demo"}[5m])
)

Let's create a recording rule that records the result of this expression into a new metric name called path_method:demo_api_request_duration_seconds_count:rate5m.

Save the following recording rule as a file named demo-rules.yml in the same directory as the main Prometheus configuration file:

groups:
- name: demo-service-rules
  rules:
  # Calculate and record the total request rate for each path/method combination.
  - record: path_method:demo_api_request_duration_seconds_count:rate5m
    expr: |
      sum without(status, instance) (
        rate(demo_api_request_duration_seconds_count{job="demo"}[5m])
      )

Reload the Prometheus configuration by sending a HUP signal to the Prometheus process:

killall -HUP prometheus

Prometheus should now evaluate the configured recording rule every 15 seconds.