Ingress contains no valid backends
Using MicroK8s with the new observability addon which uses Helm to install kube-prometheus.
This results in various Resources including several Service’s:
kubectl get services \
--namespace=observability
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP
kube-prom-stack-kube-prome-alertmanager ClusterIP 10.152.183.201 <none> 9093/TCP
kube-prom-stack-kube-prome-operator ClusterIP 10.152.183.44 <none> 443/TCP
kube-prom-stack-kube-prome-prometheus ClusterIP 10.152.183.206 <none> 9090/TCP
kube-prom-stack-kube-state-metrics ClusterIP 10.152.183.126 <none> 8080/TCP
prometheus-operated ClusterIP None <none> 9090/TCP
I’m using Tailscale Kubernetes Operator to expose MicroK8s services using Ingress to my tailnet.
Initially, I bound a Prometheus Ingress to prometheus-operated but the Tailscale Machine was not created, the Ingress wasn’t exposed to the tailnet and the Tailscale Operator logged:
{
"level":"warn",
"ts":"2024-02-20T18:01:19Z",
"logger":"ingress-reconciler",
"msg":"Ingress contains no valid backends",
"ingress-ns":"observability",
"ingress-name":"prometheus"
}
I was puzzled until I realized that prometheus-operated is a headless service with ClusterIP: None
kubectl get service/prometheus-operated \
--namespace=observability \
--output=yaml
apiVersion: v1
kind: Service
metadata:
labels:
operated-prometheus: "true"
name: prometheus-operated
namespace: observability
spec:
# Headless
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http-web
port: 9090
protocol: TCP
targetPort: http-web
selector:
app.kubernetes.io/name: prometheus
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
There’s a parallel service kube-prom-stack-kube-prome-prometheus on 9090 and this is a good canddiate for the Ingress:
kind: Ingress
apiVersion: networking.k8s.io/v1
metadata:
name: prometheus
spec:
defaultBackend:
service:
name: kube-prom-stack-kube-prome-prometheus
port:
number: 9090
ingressClassName: tailscale
tls:
- hosts:
- prometheus
alertmanager-operated is similarly headless but its analog kube-prom-stack-kube-prome-alertmanager on 9093 may be used.
When either Ingress is so-created, they can be accessed using {name}.{tailnet}.ts.net:443