Ingress contains no valid backends

Using MicroK8s with the new observability addon which uses Helm to install kube-prometheus.

This results in various Resources including several Service’s:

kubectl get services \
--namespace=observability

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)
alertmanager-operated                     ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP
kube-prom-stack-kube-prome-alertmanager   ClusterIP   10.152.183.201   <none>        9093/TCP
kube-prom-stack-kube-prome-operator       ClusterIP   10.152.183.44    <none>        443/TCP
kube-prom-stack-kube-prome-prometheus     ClusterIP   10.152.183.206   <none>        9090/TCP
kube-prom-stack-kube-state-metrics        ClusterIP   10.152.183.126   <none>        8080/TCP
prometheus-operated                       ClusterIP   None             <none>        9090/TCP

I’m using Tailscale Kubernetes Operator to expose MicroK8s services using Ingress to my tailnet.

Initially, I bound a Prometheus Ingress to prometheus-operated but the Tailscale Machine was not created, the Ingress wasn’t exposed to the tailnet and the Tailscale Operator logged:

{
"level":"warn",
"ts":"2024-02-20T18:01:19Z",
"logger":"ingress-reconciler",
"msg":"Ingress contains no valid backends",
"ingress-ns":"observability",
"ingress-name":"prometheus"
}

I was puzzled until I realized that prometheus-operated is a headless service with ClusterIP: None

kubectl get service/prometheus-operated \
--namespace=observability \
--output=yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    operated-prometheus: "true"
  name: prometheus-operated
  namespace: observability
spec:
  # Headless
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http-web
    port: 9090
    protocol: TCP
    targetPort: http-web
  selector:
    app.kubernetes.io/name: prometheus
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

There’s a parallel service kube-prom-stack-kube-prome-prometheus on 9090 and this is a good canddiate for the Ingress:

kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
  name: prometheus
spec:
  defaultBackend:
    service:
      name: kube-prom-stack-kube-prome-prometheus
      port:
        number: 9090
  ingressClassName: tailscale
  tls:
  - hosts:
    - prometheus

alertmanager-operated is similarly headless but its analog kube-prom-stack-kube-prome-alertmanager on 9093 may be used.

When either Ingress is so-created, they can be accessed using {name}.{tailnet}.ts.net:443