kps-self-alerts¶

Source rule: cluster/monitoring/kps-self-alerts.yaml (PrometheusRule kps-self-alerts, namespace monitoring, group kps_self, interval: 1m).

Self-monitoring alerts for the kube-prometheus-stack itself. All 4 alerts in the group are covered below. Every expr / for value quoted here was checked against the rule file; label existence was checked against live series.

Everything in this repo is GitOps. No fix below is applied with kubectl. Each resolution states explicitly whether it needs a Git change or is purely an investigation step. All commands in this runbook are read-only.

As-deployed context (verified): Prometheus replicas: 1, Alertmanager replicas: 1, release name kube-prometheus-stack in namespace monitoring.

PrometheusTargetDown¶

Severity: warning Fires when: up == 0 for 10m.

Note the expr is unfiltered — this is not scoped to one job. It fires one alert per down target across every scrape job in the cluster. A Grafana outage therefore fires both this and GrafanaDown; treat the pair as one incident, not two.

Likely causes¶

The monitored pod is crashed or not ready (OOMKilled, CrashLoopBackOff, Pending).
Service/Endpoint mismatch, or a CiliumNetworkPolicy blocking Prometheus from reaching the target port.
The ServiceMonitor selector no longer matches any Service (label drift after a chart bump).

Diagnose (read-only)¶

# Which targets are down, and in which job
kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  -- promtool query instant http://localhost:9090 'up == 0'

kubectl get pods -n <namespace> -l <target-selector>
kubectl describe svc <service-name> -n <namespace>   # confirm Endpoints is non-empty
kubectl get servicemonitor -n monitoring
kubectl get ciliumnetworkpolicies -A                 # if connectivity is suspected

Resolution¶

Broken workload → fix the app config in Git; ArgoCD syncs it. Do not kubectl edit.
Wrong port / selector in monitoring config → edit the ServiceMonitor manifest in Git.
Both paths are Git changes. There is no correct live action here.

PrometheusScrapePoolExceedTarget¶

Severity: warning Fires when: prometheus_target_scrape_pool_exceeded_target_limit_total > 0 for 5m.

Two corrections that matter during an incident:

This metric is a counter, not a gauge of current target count. > 0 means the target limit was hit at least once since this Prometheus process started — it does not mean the pool is over the limit right now. The counter does not decay, so once tripped the alert stays firing until Prometheus restarts, even after the underlying cause is fixed in Git.
The metric carries no scrape_pool label on this deployment (verified: its only labels are container, endpoint, instance, job, namespace, pod, service). It is a single per-Prometheus-instance counter. Any diagnostic filtering on scrape_pool="..." returns an empty result.

Known source-rule defect: the alert's description interpolates {{ $labels.scrape_pool }}, which renders empty for the same reason. Fixing that is a Git change to cluster/monitoring/kps-self-alerts.yaml; it is deliberately not made as part of running this runbook.

Likely causes¶

A ServiceMonitor selector too broad, matching far more endpoints than intended.
targetLimit / sampleLimit set too low for current cluster size.
Overlapping ServiceMonitors producing duplicate targets.

Diagnose (read-only)¶

kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  -- promtool query instant http://localhost:9090 \
     'prometheus_target_scrape_pool_exceeded_target_limit_total'

# Which pool actually grew — use target counts, since the counter cannot tell you
kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  -- promtool query instant http://localhost:9090 \
     'sort_desc(count by (job) (up))'

kubectl get servicemonitor -A
kubectl logs -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus | grep -i "target_limit\|exceeded"

Because the counter is not per-pool, identify the offending pool from the count by (job) (up) breakdown and the operator logs, not from the alert labels.

Resolution: Narrow the ServiceMonitor selector, or raise targetLimit in the kube-prometheus-stack values if the growth is legitimate. Both are Git changes in argocd-apps/monitoring/kube-prometheus-stack.yaml or the ServiceMonitor manifest. Note the alert will keep firing until Prometheus restarts even after the Git fix lands — confirm the fix via the target counts, not by waiting for the alert to clear.

AlertmanagerConfigInconsistent¶

Severity: critical

Fires when¶

count by (service) (
  count_values by (service) ("config_hash", alertmanager_config_hash)
) > 1

for 5m.

Read this first — the alert is currently inert. Alertmanager runs replicas: 1 on this cluster (verified: exactly one alertmanager_config_hash series, service="kube-prometheus-stack-alertmanager"). With a single replica there is exactly one config hash, so the count can never exceed 1. This alert cannot fire as deployed.

Consequences: - If it does fire, the meaningful conclusion is that Alertmanager was scaled to more than one replica — check that first, before assuming a reload failure. - Multi-replica failure modes (one replica stuck while others reloaded) are not possible at replicas: 1 and should not be chased. - The flip side is the real current risk: a single Alertmanager is a single point of failure for all notification delivery, and this alert provides no coverage for that. Silent notification loss is not detected here.

Diagnose (read-only)¶

kubectl get alertmanager -n monitoring \
  -o custom-columns='NAME:.metadata.name,REPLICAS:.spec.replicas'
kubectl get pods -n monitoring -l app.kubernetes.io/name=alertmanager

kubectl exec -n monitoring prometheus-kube-prometheus-stack-prometheus-0 -c prometheus \
  -- promtool query instant http://localhost:9090 'alertmanager_config_hash'

kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 \
  -c alertmanager | grep -i "reload\|error"

Resolution: If replicas > 1 and hashes genuinely diverge, the cause is a partial reload — usually a config the reloader rejected. Fix the Alertmanager config in Git and let ArgoCD sync; the reloader picks it up. If a forced clean reload is needed, do it by changing a pod annotation in Git, not with a live kubectl rollout restart. Raising the replica count to make this alert meaningful (and to remove the notification SPOF) is likewise a Git change to the kube-prometheus-stack values.

GrafanaDown¶

Severity: warning Fires when: up{job=~".*grafana.*"} == 0 for 5m.

The regex resolves against the live job kube-prometheus-stack-grafana (ServiceMonitor kube-prometheus-stack-grafana exists and the target is up), so this alert is wired correctly and will fire for a real Grafana outage. It overlaps PrometheusTargetDown by design — see that entry.

Likely causes¶

Grafana pod not running: CrashLoopBackOff, OOMKilled, or Pending.
Longhorn PVC mount failure blocking startup — Grafana uses longhorn-office, 5Gi, and is pinned via nodeSelector to topology.kubernetes.io/zone: office. A Longhorn issue in the office zone takes Grafana down with it.
Grafana up but failing its /api/health probe (OIDC or datasource init).

Diagnose (read-only)¶

kubectl get pods -n monitoring -l app.kubernetes.io/name=grafana
kubectl describe pod -n monitoring -l app.kubernetes.io/name=grafana
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana --tail=200

kubectl get pvc -n monitoring
kubectl describe pvc -n monitoring -l app.kubernetes.io/name=grafana
kubectl get volumes.longhorn.io -n longhorn-system      # office-zone volume health

Resolution¶

Resource limits, probe timings, or Grafana config → Git change in argocd-apps/monitoring/kube-prometheus-stack.yaml, then ArgoCD sync.
Longhorn volume fault → diagnose via the Longhorn UI / read-only CRs first. deploymentStrategy: Recreate plus a single RWO volume means a stuck volume attachment blocks the new pod; resolve the attachment at the Longhorn layer rather than deleting Kubernetes objects by hand.
Grafana is a dashboard surface, not a data path. Alerting and metric collection continue while it is down — do not escalate this as a monitoring outage.