I continue to learn “tricks” with Prometheus and (kube-prometheus ⊃ Prometheus Operator).

For too long, I’ve lived with Alertmanager alerts including “View in Alertmanager” and “Source” hyperlinks that don’t work because they defaulted incorrectly.

The solution when running either Prometheus or Alertmanager (same for both) is to use the flag:

--web.external-url="${EXTERNAL_URL}"

Conveniently, when running kube-prometheus, there are Prometheus and Alertmanager CRDs for configuring Prometheus and Alertmanager and these include the field:

externalUrl: "${EXTERNAL_URL}"

The question is: How to configure this value?

Prometheus Recording Rules well-documented but I’d not previously used this feature.

The simple(st?) prometheus.yml:

global:

rule_files:
- rules.yaml

scrape_configs:
- job_name: prometheus-server
  static_configs:
  - targets:
    - localhost:9090

NOTE Prometheus doesn’t barf if the rules.yaml does not exist. Corrolary, if it can’t find rules.yaml, the /rules endpoint will be empty.

I wanted something that was guaranteed to be available: prometheus_http_requests_total

When I started using Prometheus, I was flummoxed when I encountered colons (:) in metric names but learned that these are used (uniquely) for defining recording rules

I’m using Pushover’s (generous) 30-day trial. IIUC thereafter (for personal use) the app’s $5 for a perpetual license. That seems very reasonable to me.

I find Prometheus’ documentation “light”. Everything’s there but the docs feel oriented to the power|frequent user. I use Prometheus infrequently and struggle to understand the docs.

The AlertManager configuration for Pushover is ok but I struggled to understand the reference to (Golang) templates:

# Notification title.
[ title: <tmpl_string> | default = '{{ template "pushover.default.title" . }}' ]

# Notification message.
[ message: <tmpl_string> | default = '{{ template "pushover.default.message" . }}' ]

# A supplementary URL shown alongside the message.
[ url: <tmpl_string> | default = '{{ template "pushover.default.url" . }}' ]

I’m familiar with the Go’s templating but I was unclear how to interpret these configuration references. As I thought it, I assumed these must reference default templates (shipped with AlertManager) and found these here: