Prometheus Recording Rules

Prometheus Recording Rules well-documented but I’d not previously used this feature.

The simple(st?) prometheus.yml:

global:

rule_files:
- rules.yaml

scrape_configs:
- job_name: prometheus-server
  static_configs:
  - targets:
    - localhost:9090

NOTE Prometheus doesn’t barf if the rules.yaml does not exist. Corrolary, if it can’t find rules.yaml, the /rules endpoint will be empty.

I wanted something that was guaranteed to be available: prometheus_http_requests_total

When I started using Prometheus, I was flummoxed when I encountered colons (:) in metric names but learned that these are used (uniquely) for defining recording rules

This is not a good example of a recording rule since it doesn’t save any metric calculation but:

rules.yml:

groups:
- name: foo
  interval: 60m
  labels:
    foo: foo
  rules:
  - record: foo:https
    expr: prometheus_http_requests_total{}

NOTE Intentionally exaggeratedly long interval of 60m

Then:

VERS=v3.2.1

podman run \
--interactive --tty --rm \
--name=prometheus \
--publish=9090:9090/tcp \
--volume=${PWD}/prometheus.yml:/etc/prometheus/prometheus.yml \
--volume=${PWD}/rules.yaml:/etc/prometheus/rules.yaml \
docker.io/prom/prometheus:${VERS} \
--config.file=/etc/prometheus/prometheus.yml \
--web.enable-lifecycle

NOTE Don’t forget to mount both prometheus.yml and rules.yml into the container.

And:

The underlying metric (prometheus_http_requests_total):

NOTE scraped (by default) every minute.

The recording rule (foo:https):

NOTE recorded (calculated) by configuration, every hour (hence the gaps).

• Prometheus