Gateway LoadBalancer alert runbook

Source rule: cluster/gateway/monitoring.yaml (PrometheusRule gateway-ip-alerts, namespace networking, group gateway.loadbalancer).

⚠️ Read this first — both alerts in this group are currently non-functional

Both rules are built on kube_service_status_load_balancer_ingress, which kube-state-metrics emits as an info-style gauge: the value is always 1, and the LoadBalancer IP is carried in the ip label, not in the value. Verified live against KSM on 2026-07-20:

kube_service_status_load_balancer_ingress{namespace="networking",service="cilium-gateway-main-home",uid="5c69200e-…",ip="10.1.30.10",hostname=""} 1

Consequences:

• GatewayLoadBalancerDown can never fire. A service that loses its LB IP does not produce a 0-valued series — the series disappears entirely. == 0 therefore matches nothing, ever.
• GatewayLoadBalancerIPChanged can never fire. The value is a constant 1, so changes() over it is always 0. An IP rotation creates a new series with a different ip label while the old one goes stale; changes() does not observe this.

Both exprs were executed against live Prometheus and returned empty results, as predicted by the metric shape.

Treat this group as providing no gateway coverage. If you arrived here from a firing alert, something has changed in the rule — re-read the source file before trusting anything below. Fixing this requires a Git change to cluster/gateway/monitoring.yaml; see Fixing the rules.

Context

Bare-metal, dual-site. There is no cloud provider — LoadBalancer IPs are assigned by Cilium LB IPAM from CiliumLoadBalancerIPPool (cluster/cilium-lb/ip-pool*.yaml) and announced on the LAN by ARP via CiliumL2AnnouncementPolicy (home, office). Pools are dedicated VLAN 30 ranges: home 10.1.30.10–250, office 10.2.30.10–250. Pool membership is selected by the lbipam.cilium.io/pool service label.

L2 announcement ownership is held per service by a lease in kube-system named cilium-l2announce-<namespace>-<service>, and policies exclude control-plane nodes deliberately so CP restarts do not migrate worker-held leases.

Gateway services in scope (live, networking namespace):

Service	IP	Site
cilium-gateway-main-home	10.1.30.10	home
cilium-gateway-main	10.1.30.12	home
cilium-gateway-private-home	10.1.30.13	home
cilium-gateway-main-office	10.2.30.10	office
cilium-gateway-private-office	10.2.30.12	office

Note coredns-lan-home / coredns-lan-office also hold pool IPs (10.1.30.11 / 10.2.30.11) and are matched by GatewayLoadBalancerDown's unfiltered expr, but not by the IPChanged regex.

---

GatewayLoadBalancerIPChanged

Severity: warning · Component: gateway LoadBalancer

Fires when: changes(kube_service_status_load_balancer_ingress{service=~"cilium-gateway-.*"}[5m]) > 0 holds for 1m.

Restricted by regex to services named cilium-gateway-*. As written this cannot fire — see the warning above. The intent is to detect a gateway LoadBalancer IP rotating, which would leave LAN DNS overrides pointing at a dead address.

Likely causes (were the rule functional)

1. LB IPAM reassigned the IP after the service was deleted and recreated — e.g. a Gateway resource renamed or recreated in Git, so Cilium allocated the next free address from the pool rather than the previous one.
2. The lbipam.cilium.io/pool label on the Gateway changed, moving the service to the other site's pool (home ↔ office).
3. Pool block edited in cluster/cilium-lb/ip-pool*.yaml such that the previously held address is no longer in range.
4. IP explicitly pinned/changed in the Gateway spec in Git.

Diagnose (all read-only)

# Current assigned IPs for every gateway service
kubectl get svc -n networking -l 'io.cilium.gateway/owning-gateway' \
  -o custom-columns='NAME:.metadata.name,TYPE:.spec.type,IP:.status.loadBalancer.ingress[*].ip'

# What LAN DNS currently hands out — compare against the above
dig +short <hostname>.cauda.dev

# Pool definitions and which pool each service is bound to
kubectl get ciliumloadbalancerippool -o wide
kubectl get svc -n networking --show-labels | grep lbipam

# Operator reconciliation history
kubectl -n kube-system logs deploy/cilium-operator --tail=200 | grep -iE 'lbipam|ipam|loadbalancer'

PromQL — list the distinct IPs seen per gateway service over the last 24h (more than one row per service means the IP moved):

count by (namespace, service, ip) (
  count_over_time(kube_service_status_load_balancer_ingress{service=~"cilium-gateway-.*"}[24h])
)

Resolution

The IP is assigned by Cilium from the pool; do not patch the Service to force an address — it will be reverted by ArgoCD and is forbidden by the repo's GitOps rules. Correct actions:

• If the new IP is acceptable: update the LAN DNS overrides (cluster/coredns-lan/) to the new address. This is a Git change → PR → ArgoCD sync.
• If the IP must be restored to its previous value: pin it in Git via the Gateway's LB IPAM annotation / pool block in cluster/cilium-lb/ip-pool*.yaml. Git change, not a live edit.
• If the service moved to the wrong site's pool, correct the lbipam.cilium.io/pool label on the Gateway in cluster/gateway/. Git change.

---

GatewayLoadBalancerDown

Severity: critical · Component: gateway LoadBalancer

Fires when: kube_service_status_load_balancer_ingress == 0 holds for 2m.

There is no service selector on this expr — it is cluster-wide across every LoadBalancer service, not just gateways (contrary to what the alert name implies). And, as established above, it cannot fire, because the metric is never 0.

Likely causes (were the rule functional)

1. LB IPAM could not allocate — the site pool is exhausted, or the service's lbipam.cilium.io/pool label matches no pool, so no IP is ever assigned.
2. cilium-operator is down or crash-looping and not reconciling LB IPAM (see the separate cilium-operator rules in cluster/monitoring/cilium-alerts.yaml).
3. The Gateway/Service was removed or renamed in Git and ArgoCD pruned it.
4. IP assigned but not reachable: the L2 announcement lease has no holder, or no eligible node matches the CiliumL2AnnouncementPolicy node selector (requires zone label, non-control-plane, and cilium.io/l2-eligible=true). This is the most operationally likely real failure and the one the metric would not catch even if the rule worked — the IP is still assigned, just unannounced.

Diagnose (all read-only)

# Does the service have an ingress IP at all?
kubectl describe svc -n networking <service-name>

# Pool capacity / conditions — look for exhaustion
kubectl get ciliumloadbalancerippool -o yaml

# Operator health and errors
kubectl -n kube-system get pods -l name=cilium-operator -o wide
kubectl -n kube-system logs deploy/cilium-operator --tail=200 | grep -iE 'error|fail|lbipam'

# L2 announcement leases — HOLDER must be a live, eligible worker node
kubectl get leases -n kube-system | grep cilium-l2announce

# Which nodes are eligible to announce
kubectl get ciliuml2announcementpolicy -o yaml
kubectl get nodes -L topology.kubernetes.io/zone,cilium.io/l2-eligible

kubectl get events -n networking --sort-by=.lastTimestamp | tail -30

PromQL — LoadBalancer services that have no ingress IP (this is the correct detection, validated live):

kube_service_spec_type{type="LoadBalancer"}
  unless on(namespace, service) kube_service_status_load_balancer_ingress

Resolution

• Pool exhausted or mislabelled → widen the block or fix the lbipam.cilium.io/pool label. Both are Git changes in cluster/cilium-lb/ or cluster/gateway/ → PR → ArgoCD.
• Service pruned unexpectedly → restore the Gateway definition in cluster/gateway/. Git change. Do not kubectl apply it back by hand.
• cilium-operator unhealthy → this is a live workload fault, not a config fault; follow the cilium-operator runbook. Restarting the operator pod is a remediation action, so treat it as an operational decision, not something this runbook auto-authorises.
• Lease has no holder / no eligible node → confirm at least one worker in that zone carries cilium.io/l2-eligible=true. The label is managed by the node-l2-health-labeler (cluster/cilium-lb/node-l2-health-labeler.yaml); if it is wrongly withheld, fix the labeler logic in Git. Do not hand-label nodes — it will drift back.

---

Fixing the rules

Both alerts need their expr replaced in cluster/gateway/monitoring.yaml. This is a Git change → PR → ArgoCD sync. Suggested exprs, both executed against live Prometheus and confirmed to parse and to be correctly quiet in the current healthy state:

# Replaces GatewayLoadBalancerDown — catches LB services with no ingress IP.
# Left side currently matches 7 services, right side removes those that have an IP.
- alert: GatewayLoadBalancerDown
  expr: |
    kube_service_spec_type{type="LoadBalancer", namespace="networking"}
      unless on(namespace, service) kube_service_status_load_balancer_ingress
  for: 2m

# Replaces GatewayLoadBalancerIPChanged — >1 distinct ip series per service
# within the window means the address moved. Currently returns exactly 1 per
# gateway service, so the ">1" threshold is correctly silent.
- alert: GatewayLoadBalancerIPChanged
  expr: |
    count by (namespace, service) (
      count_over_time(kube_service_status_load_balancer_ingress{service=~"cilium-gateway-.*"}[15m])
    ) > 1
  for: 1m

Consider also adding coverage for an unannounced IP (lease with no holder), which neither the current nor the replacement rules detect and which is the more probable real-world gateway outage on this cluster.

Both rules should carry a namespace="networking" matcher so GatewayLoadBalancerDown stops silently covering every LoadBalancer service in the cluster, or be deliberately renamed if cluster-wide coverage is intended.