Skip to content

Alert runbook — cluster-alerts

Source rule file: cluster/monitoring/alerting-rules.yaml (PrometheusRule monitoring/cluster-alerts)

All 52 alerts in that PrometheusRule are covered below. Every "Fires when" is transcribed from the live expr + for:.

GitOps rule: this cluster is reconciled by ArgoCD. Every diagnostic command here is read-only. Resolutions that change cluster state are marked [Git] — they are made by editing a manifest and letting ArgoCD reconcile, never by a live kubectl mutation. The few genuinely live actions are marked [Live — operator approval] and must not be run reflexively from this document.

Velero alerts are deliberately not here — they live in cluster/monitoring/velero-alerts.yaml.

Pod health (pod-health)

PodCrashLooping

Fires when: rate(kube_pod_container_status_restarts_total[15m]) * 60 * 15 > 0, sustained for: 15m. Severity: warning. Note this is effectively "any restart at all in the 15m window", not a rate threshold — the * 60 * 15 scales the per-second rate back to a restart count. Likely causes: crash on startup (bad config, missing secret), OOMKill (see ContainerOOMKilled), or a failing dependency (DB / network).

Diagnose

  • kubectl get pods -n <namespace>
  • kubectl describe pod <pod> -n <namespace> — Events and Last State reason
  • kubectl logs <pod> -n <namespace> --previous Resolution: [Git] fix the application config or resource limits in the manifest. Do not delete or restart pods by hand; ArgoCD reconciles the corrected manifest.

PodNotReady

Fires when: kube_pod_status_ready{condition="true"} == 0, excluding pods matched by kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1, for: 15m. Severity: warning. The unless clause exists so completed Job pods (always ready=0) do not alert. Likely causes: readiness probe failing, network policy blocking probe traffic, or a volume that never mounted.

Diagnose

  • kubectl describe pod <pod> -n <namespace> — Conditions and Events
  • kubectl logs <pod> -n <namespace>
  • PromQL: kube_pod_status_ready{condition="true", namespace="<ns>", pod="<pod>"} Resolution: [Git] adjust probe thresholds, fix startup logic, or correct the NetworkPolicy.

ContainerOOMKilled

Fires when: kube_pod_container_status_terminated_reason{reason="OOMKilled"} == 1, for: 0m (immediate). Severity: warning. Likely causes: memory limit too low, memory leak, or a traffic spike.

Diagnose

  • kubectl describe pod <pod> -n <namespace> — Last State reason, exit code 137
  • kubectl top pod <pod> -n <namespace>
  • PromQL: container_memory_working_set_bytes{pod="<pod>"} (a gauge — graph it directly; do not wrap in rate()) Resolution: [Git] raise resources.limits.memory in the workload manifest, or fix the leak. Kyverno's check-memory-limits policy applies — audit every container and sidecar when editing.

Node health (node-health)

NodeHighCPU

Fires when: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85, for: 10m. Severity: warning. Likely causes: heavy workloads (CNPG backups, Trivy scans), noisy neighbours, control-plane pressure.

Diagnose

  • kubectl top nodes
  • kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node>
  • PromQL: rate(node_cpu_seconds_total{mode!="idle", instance="<instance>"}[5m]) — breaks usage down by mode (user / system / iowait) Resolution: [Git] rebalance or right-size workload requests/limits. If the driver is control-plane, check the etcd alerts below for disk I/O pressure.

NodeHighMemory

Fires when: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90, for: 10m. Severity: critical. Likely causes: leak in a system daemon, aggressive application caching, or workload density beyond node capacity.

Diagnose

  • kubectl top nodes
  • kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<node>
  • PromQL: node_memory_MemAvailable_bytes{instance="<instance>"} / node_memory_MemTotal_bytes{instance="<instance>"} Resolution: [Git] reduce workload density or add capacity. Node VM sizing changes go through Terraform/Ansible, not a live edit.

NodeDiskSpaceLow

Fires when: (1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) * 100 > 85, for: 10m. Severity: warning. Likely causes: accumulated container logs, un-GC'd images, Longhorn replica growth.

Diagnose

  • kubectl describe node <node> — Conditions
  • PromQL: node_filesystem_avail_bytes{instance="<instance>", fstype!~"tmpfs|overlay"} — identifies the specific mountpoint
  • On-node (read-only, via PVE qm guest exec for unreachable nodes): df -h, du -sh /var/lib/kubelet/pods/* Resolution: [Git] tune kubelet image/log GC settings in the Ansible node config. Do not hand-delete files on a node as the first move — identify what is growing first.

NodeNotReady

Fires when: kube_node_status_condition{condition="Ready",status="true"} == 0, for: 5m. Severity: critical. Likely causes: network partition, kubelet crash, disk or memory pressure.

Diagnose

  • kubectl get nodes
  • kubectl describe node <node> — look for DiskPressure / MemoryPressure / NetworkUnavailable
  • kubectl logs -n kube-system -l k8s-app=cilium --tail=50
  • For an unreachable node, use PVE (qm guest exec) rather than treating it as needing console access. Resolution: Usually not a Git fix. A single node down in a quorum cluster is the design working — do not escalate reflexively. Investigate kubelet/host state; if a DaemonSet config error caused it, that fix is [Git]. Do not drain workers for routine recovery (insufficient spare capacity — reboot in place).

Storage (storage-alerts)

PVCAlmostFull

Fires when: kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes * 100 > 85, for: 10m. Severity: warning. Likely causes: data growth (logs, DB WAL), missing retention policy, undersized initial request.

Diagnose

  • kubectl get pvc -n <namespace>
  • kubectl describe pvc <pvc> -n <namespace>
  • PromQL: kubelet_volume_stats_used_bytes{persistentvolumeclaim="<pvc>", namespace="<ns>"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim="<pvc>", namespace="<ns>"} Resolution: [Git] increase the PVC size in the owning manifest (Longhorn supports online expansion) and/or configure log rotation.

PVCFull

Fires when: same ratio > 95, for: 5m. Severity: critical. Likely causes: as above, at the critical threshold. Application failure is imminent.

Diagnose

  • kubectl get pvc -n <namespace>
  • kubectl logs <pod> -n <namespace> — look for "no space left on device"
  • Check whether Longhorn snapshots are consuming the space. Resolution: [Git] raise the PVC size in the manifest. ArgoCD picks the change up on its normal sync; there is no correct hand-applied shortcut here.

Certificates (cert-alerts)

CertificateExpiringSoon

Fires when: certmanager_certificate_expiration_timestamp_seconds - time() < 21 * 24 * 3600, for: 1h. Severity: warning. The threshold is 21d (not 30d) deliberately: cert-manager renews LE wildcards at ~30d remaining, so 21d means renewal is overdue, not merely starting. Likely causes: cert-manager renewal failure (DNS-01 challenge, ACME rate limit) or issuer misconfiguration.

Diagnose

  • kubectl get certificates -A
  • kubectl describe certificate <cert> -n <namespace>
  • kubectl logs -n cert-manager -l app.kubernetes.io/component=controller --tail=100 Resolution: [Git] fix the issuer config; verify the Cloudflare API token secret is valid (sealed secret → [Git]).

CertificateExpiryCritical

Fires when: the same expression with < 7 * 24 * 3600, for: 1h. Severity: critical. Renewal has been failing for roughly 23 days by this point.

Diagnose

  • kubectl get certificaterequests -n <namespace>
  • kubectl describe order <order> -n <namespace> — ACME challenge errors
  • kubectl describe challenge -n <namespace> Resolution: [Git] fix the issuer or token secret. If a forced renewal is needed ahead of the next reconcile, that is [Live — operator approval]; correct the Git state first so the fix is not reverted.

Database — CNPG (database-alerts)

CNPGInstanceDown

Fires when: cnpg_collector_up == 0, for: 5m. Severity: critical. Likely causes: Postgres pod crashed, node failure, or scrape path broken.

Diagnose

  • kubectl get pods -n <namespace>
  • kubectl get cluster -n <namespace> (CNPG Cluster CR status)
  • kubectl logs <pod> -n <namespace>
  • PromQL: cnpg_collector_up Resolution: [Git] fix the underlying config/storage cause; the CNPG operator restarts the instance itself. Never cordon + force-delete a CNPG pod — it cascades (Postgres cannot reschedule → -rw service empty → every DB-backed app down).

CNPGReplicationLag

Fires when: cnpg_pg_replication_lag > 30 (seconds), for: 10m. Severity: warning. Likely causes: high write load, slow cross-site link, replica I/O bottleneck. prefer-standby backups skew this.

Diagnose

  • PromQL: cnpg_pg_replication_lag — trend per pod
  • kubectl exec <replica-pod> -n <namespace> -- psql -c "SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();" (read-only SELECT) Resolution: [Git] raise replica resources or address query load. Check cross-site latency if replication spans sites.

CNPGReplicationLagCritical

Fires when: cnpg_pg_replication_lag > 300, for: 5m. Severity: critical. The standby is unusable for failover.

Diagnose

  • kubectl logs <replica-pod> -n <namespace>
  • kubectl exec <replica-pod> -n <namespace> -- psql -c "SELECT pg_is_in_recovery();" (read-only SELECT) Resolution: [Git] address primary load or replica sizing. Rebuilding a wedged replica is [Live — operator approval]: it is a destructive resync and must be an explicit decision, not a runbook reflex.

CNPGOperatorCrashLoop

Fires when: increase(kube_pod_container_status_restarts_total{namespace="cnpg-system", container="manager"}[1h]) > 3, for: 15m. Severity: warning. HA (replicaCount: 2) should mask this; treat a firing alert as a bug candidate.

Diagnose

  • kubectl logs -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg --previous
  • kubectl describe pod -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg Resolution: [Git] correct the operator config or pin a different operator version.

CNPGOperatorLeaderFlapping

Fires when: changes(leader_election_master_status{namespace="cnpg-system"}[15m]) > 2, for: 5m. Severity: warning. Likely causes: apiserver reachability, network jitter, or resource pressure on the operator pods.

Diagnose

  • kubectl logs -n cnpg-system -l app.kubernetes.io/name=cloudnative-pg
  • kubectl get --raw /healthz Resolution: [Git] raise operator resources. If the apiserver itself is unstable, that is the load-bearing cause — see the apiserver alerts.

CNPGLastBackupTooOld

Fires when: time() - the newest of barman_cloud_cloudnative_pg_io_last_available_backup_timestamp (max by namespace) or cnpg_collector_last_available_backup_timestamp (max by cluster, namespace) > 36 * 3600, for: 30m. Severity: critical. Both metric families are OR'd because clusters on the barman-cloud plugin leave the old cnpg_collector_* family at 0 forever. Schedule is 24h, so >36h means a ScheduledBackup missed or failed.

Diagnose

  • kubectl get scheduledbackups -A
  • kubectl get backups -n <namespace>
  • kubectl logs -n cnpg-system -l app.kubernetes.io/name=barman-cloud
  • PromQL: barman_cloud_cloudnative_pg_io_last_available_backup_timestamp Resolution: [Git] fix the barman config or ObjectStore credentials (sealed secret). Verify connectivity to the S3-compatible target.

CNPGContinuousArchivingStalled

Fires when: cnpg_collector_last_failed_archive_time > cnpg_collector_last_available_archive_time, or barman_cloud_cloudnative_pg_io_last_failed_backup_timestamp > barman_cloud_cloudnative_pg_io_last_available_backup_timestamp, for: 15m. Severity: critical. The PITR recovery window is shrinking while this fires.

Diagnose

  • kubectl logs <primary-pod> -n <namespace> — archiver errors
  • kubectl logs -n cnpg-system -l app.kubernetes.io/name=barman-cloud Resolution: [Git] fix the archive path, storage permissions, or credentials.

CNPGPluginDown

Fires when: kube_deployment_status_replicas_available{deployment="barman-cloud", namespace="cnpg-system"} == 0, for: 10m. Severity: critical. The CNPG-I gRPC backend is unreachable — no backups and no WAL archiving run at all.

Diagnose

  • kubectl get deployment barman-cloud -n cnpg-system
  • kubectl describe deployment barman-cloud -n cnpg-system
  • kubectl logs -n cnpg-system -l app.kubernetes.io/name=barman-cloud Resolution: [Git] fix the deployment spec or image reference.

CNPGPVCAlmostFull

Fires when: kubelet_volume_stats_used_bytes{persistentvolumeclaim=~".*-pgdata|.*-pgwal"} / kubelet_volume_stats_capacity_bytes > 0.85, for: 15m. Severity: warning. Note the threshold is the ratio 0.85, not a percentage.

Diagnose

  • kubectl get pvc -n <namespace>
  • kubectl exec <primary-pod> -n <namespace> -- du -sh /var/lib/postgresql/data/pg_wal (read-only) Resolution: [Git] expand the volume. Investigate why WAL is accumulating — an inactive replication slot (below) is a common cause.

CNPGInactiveReplicationSlot

Fires when: cnpg_pg_replication_slots_active == 0, for: 30m. Severity: warning. An inactive slot retains WAL indefinitely — this is the upstream cause of a filling pgwal volume.

Diagnose

  • kubectl exec <primary-pod> -n <namespace> -- psql -c "SELECT slot_name, active, wal_status FROM pg_replication_slots;" (read-only SELECT)
  • kubectl get pods -n <namespace> — compare live replicas against slot names Resolution: If the consumer is genuinely gone the slot must be dropped, which is [Live — operator approval]: pg_drop_replication_slot is a destructive DB operation with no GitOps representation. Confirm the consumer is not merely temporarily disconnected first. If the slot belongs to a replica that should exist, the fix is [Git] — restore the missing replica instead.

Workload health (deployment-alerts)

DeploymentReplicasMismatch

Fires when: kube_deployment_spec_replicas != kube_deployment_status_replicas_ready, for: 15m. Severity: warning. Likely causes: crash loop, image pull failure, unschedulable due to resources, node down.

Diagnose

  • kubectl get deployment -n <namespace>
  • kubectl describe deployment <deployment> -n <namespace>
  • kubectl get events -n <namespace> --sort-by=.lastTimestamp Resolution: [Git] fix the underlying pod problem (image, config, resources).

StatefulSetReplicasMismatch

Fires when: kube_statefulset_status_replicas_ready != kube_statefulset_status_replicas, for: 15m. Severity: warning. Likely causes: as above, but PVC binding failures and ordinal blocking are the common StatefulSet-specific cases.

Diagnose

  • kubectl get statefulset -n <namespace>
  • kubectl describe statefulset <statefulset> -n <namespace>
  • kubectl get pvc -n <namespace> — check for Pending claims Resolution: [Git] fix the pod or PVC template.

ArgoCD (argocd-alerts)

ArgoCDAppOutOfSync

Fires when: argocd_app_info{sync_status!="Synced"} == 1, for: 30m. Severity: warning. Likely causes: live drift from an out-of-band change, a failed sync operation, or a chart/manifest change that will not apply.

Diagnose

  • argocd app diff <app> (ArgoCD CLI, read-only) or the ArgoCD UI
  • kubectl get application <app> -n argocd -o yaml — status and conditions
  • Beware the RespectIgnoreDifferences=true + global ignoreDifferences footgun: an ignored field holding a stale value can silently block sync. Resolution: [Git] the manifest is the source of truth — if the live state is the correct one, encode it in Git; if the manifest is correct, fix whatever blocks it from applying. Do not resolve drift by hand-editing the live resource.

ArgoCDAppDegraded

Fires when: argocd_app_info{health_status="Degraded"} == 1, for: 15m. Severity: critical. Likely causes: unhealthy pods, missing resources, or failed rollout within the app.

Diagnose

  • kubectl get application <app> -n argocd -o yaml — identifies the degraded resource
  • Drill into that resource with kubectl describe / kubectl logs Resolution: [Git] fix the failing resource's manifest.

Security — Trivy (trivy-alerts)

TrivyCriticalVulnerability

Fires when: sum by (namespace, resource_name) (trivy_image_vulnerabilities{severity="Critical"}) > 0, for: 1h. Severity: warning.

Diagnose

  • kubectl get vulnerabilityreports -A
  • kubectl describe vulnerabilityreport <report> -n <namespace>
  • PromQL: trivy_image_vulnerabilities{severity="Critical"} Resolution: [Git] bump the image tag to a patched version (pinned — never latest).

TrivyHighVulnerabilitySpike

Fires when: sum by (namespace) (trivy_image_vulnerabilities{severity="High"}) > 20, for: 1h. Severity: warning. Note this is an aggregate per namespace, not per image.

Diagnose

  • PromQL: sum by (namespace, resource_name) (trivy_image_vulnerabilities{severity="High"}) — attributes the count to specific images
  • kubectl get vulnerabilityreports -n <namespace> Resolution: [Git] update the offending images.

TrivyVulnerabilityScannerProducingNoData

Fires when: absent(trivy_image_vulnerabilities), for: 6h. Severity: warning. Means the scanner has produced no vulnerability data for 6h — a blind spot, not a clean result.

Diagnose

  • kubectl get pods,jobs -n security -l app.kubernetes.io/name=trivy-operator — is the operator/scan job running?
  • kubectl get vulnerabilityreports -A | head — any reports at all?
  • kubectl logs -n security deploy/trivy-operator | tail — scan errors, RBAC, or registry-auth failures Resolution: restart the trivy-operator if wedged; if it is a registry-auth failure, [Git] fix the scanner's pull credentials.

Argo Workflows (argo-workflows-alerts)

ArgoWorkflowRunnerFailed / ArgoWorkflowFailureBurst / ArgoWorkflowFailureRateHigh

Fire when: workflows enter Failed/Error — a single failure (for: 15m), a burst above the 0.26/h baseline, or an elevated failure rate over 6h (baseline 2.7%, for: 30m). Severity: warning, per namespace.

Diagnose

  • argo list -n <namespace> --status Failed,Error — the failing workflows
  • argo logs -n <namespace> <workflow> — the failing step's logs
  • argo get -n <namespace> <workflow> — which DAG node failed Resolution: fix the failing step (usually [Git] the WorkflowTemplate/image/inputs); re-run with argo resubmit.

ArgoWorkflowsControllerMetricsMissing

Fires when: absent(argo_workflows_is_leader), for: 30m — no argo_workflows_* series, so the failure alerts above cannot fire (a blind spot).

Diagnose

  • kubectl get servicemonitor -A | grep -i argo — is the controller ServiceMonitor present?
  • kubectl get pods -n <ns> -l app.kubernetes.io/name=argo-workflows-workflow-controller — controller up?
  • Check controller.metricsConfig.enabled in the argo-workflows values. Resolution: [Git] restore the controller ServiceMonitor / metricsConfig; verify the target in Prometheus.

cert-manager (certmanager-alerts)

Metric labels verified live: the controller scrapes under job="cert-manager", the webhook under job="webhook".

CertManagerCertNotReady

Fires when: certmanager_certificate_ready_status{condition="True"} == 0, for: 30m. Severity: warning.

Diagnose

  • kubectl describe certificate <cert> -n <namespace>
  • kubectl logs -n cert-manager -l app.kubernetes.io/component=controller Resolution: [Git] fix the issuer or DNS configuration.

CertManagerControllerDown

Fires when: max(up{job="cert-manager"}) == 0, for: 10m. Severity: critical. max(...) == 0 means every controller target is down — a single-replica restart does not trip this. No issuance or renewal happens while this fires.

Diagnose

  • kubectl get pods -n cert-manager
  • kubectl logs -n cert-manager -l app.kubernetes.io/component=controller
  • PromQL: up{job="cert-manager"} Resolution: [Git] fix the controller deployment config or resources.

CertManagerWebhookDown

Fires when: max(up{job="webhook"}) == 0, for: 10m. Severity: critical. This blocks all cert-manager API writes (Certificate/Issuer admission) and can block ArgoCD syncs touching those CRDs.

Diagnose

  • kubectl get pods -n cert-manager
  • kubectl logs -n cert-manager -l app.kubernetes.io/component=webhook
  • PromQL: up{job="webhook"} Resolution: [Git] fix the webhook deployment. Check the webhook's own serving certificate has not expired.

CertManagerACMEErrors

Fires when: sum(increase(certmanager_http_acme_client_request_count{status!~"2.."}[15m])) > 0, for: 15m. Severity: warning. The status label is the HTTP response code. This catches renewal breakage weeks before CertificateExpiryCritical would.

Diagnose

  • kubectl logs -n cert-manager -l app.kubernetes.io/component=controller — specific ACME errors
  • PromQL: sum by (status) (increase(certmanager_http_acme_client_request_count{status!~"2.."}[15m])) — distinguishes rate-limit (429) from auth failure (403) Resolution: [Git] fix the Cloudflare DNS-01 token (sealed secret) or issuer config. Rate limiting resolves on its own; do not retry into it.

etcd (etcd-cauda)

These are cauda-custom rules; the kube-prometheus-stack defaults are also loaded.

EtcdPeerRTTSlow

Fires when: histogram_quantile(0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket{job=~".*etcd.*"}[5m])) > 1 (second), for: 10m. Severity: warning. Replaces the kps default etcdMemberCommunicationSlow, whose 0.15s threshold is incompatible with this dual-site cluster: cross-site peer RTT over the UniFi gateway tunnel is 400–780ms by design; same-site is ~10–25ms.

Diagnose

  • PromQL: the expression above, grouped by the To label — tells you whether the slow peer is cross-site (expected baseline) or same-site (real degradation)
  • kubectl logs -n kube-system -l component=etcd Resolution: If both peers are same-site, check node CPU/disk pressure. If cross-site, this is the tunnel's normal characteristic — do not invent a firewall or MTU explanation.

EtcdNoLeader

Fires when: max by (job) (etcd_server_has_leader) == 0, for: 1m. Severity: critical. The write path is stalled.

Diagnose

  • kubectl get pods -n kube-system -l component=etcd
  • kubectl logs -n kube-system -l component=etcd — leader election errors
  • PromQL: etcd_server_has_leader Resolution: [Live — operator approval] and not a Git change. Quorum recovery is a manual procedure — follow docs/runbooks/etcd-restore.md rather than improvising. Read the procedure before acting.

EtcdDBSizeApproachingQuota

Fires when: etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes > 0.80, for: 10m. Severity: warning. Defrag + compact run every 4h.

Diagnose

  • PromQL: etcd_mvcc_db_total_size_in_bytes / etcd_server_quota_backend_bytes
  • kubectl logs -n kube-system -l component=etcd — compaction errors
  • Check for resource churn: a large PolicyReport count is a known contributor (see KyvernoReportsBacklog). Resolution: [Git] raise quota-backend-bytes in the etcd static pod config (Ansible), or reduce the churn source.

EtcdBackupMissing

Fires when: time() - max(kube_cronjob_status_last_successful_time{namespace="kube-system",cronjob="etcd-snapshot"}) > 8 * 3600, for: 30m. Severity: critical. Expected cadence is every 6h.

Diagnose

  • kubectl get cronjob etcd-snapshot -n kube-system
  • kubectl get jobs -n kube-system — find the most recent etcd-snapshot Job
  • kubectl logs -n kube-system job/<job-name> Resolution: [Git] fix the CronJob spec or the snapshot destination. If the cause is node disk space, see NodeDiskSpaceLow.

EtcdHighCommitDurations

Fires when: histogram_quantile(0.99, rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) > 0.25 (250ms), for: 10m. Severity: warning.

Diagnose

  • PromQL: rate(etcd_disk_backend_commit_duration_seconds_sum[5m]) / rate(etcd_disk_backend_commit_duration_seconds_count[5m]) — mean commit time alongside the p99
  • kubectl top nodes Resolution: [Git] reduce etcd load or move etcd to faster storage (Ansible/Terraform node config).

kube-apiserver (apiserver-cauda)

ApiserverAuditLogBackpressure

Fires when: increase(apiserver_audit_event_total[5m]) == 0 and apiserver_audit_level_total{level!="None"} > 0, for: 10m. Severity: warning. Audit is enabled but nothing is being recorded — the pipeline is stuck, not idle.

Diagnose

  • kubectl logs -n kube-system -l component=kube-apiserver — audit errors
  • PromQL: apiserver_audit_event_total and apiserver_audit_level_total
  • Check control-plane node disk (read-only df -h via PVE guest exec) Resolution: [Git] fix the audit policy or log rotation config in the apiserver manifest (Ansible-managed static pod). Disk exhaustion is the common root cause.

ApiserverRequestThrottled

Fires when: sum(rate(apiserver_flowcontrol_rejected_requests_total[5m])) by (flow_schema, priority_level) > 0, for: 10m. Severity: warning.

Diagnose

  • PromQL: the expression above — names the saturated FlowSchema and priority level
  • kubectl get flowschema, kubectl get prioritylevelconfiguration Resolution: [Git] tune APF concurrency shares. Identify the client driving the traffic first — throttling is usually a symptom of a hot-looping controller.

Kyverno (kyverno-alerts)

The kyverno-recording group also defines five recording rules (kyverno:admission_request_duration_seconds:p99_5m, :p50_5m, kyverno:policy_results:rate_5m, kyverno:policy_results_fail:rate_5m, kyverno:cluster_policy:count). Recording rules do not alert; two of them back the latency alerts below.

KyvernoAdmissionControllerDown

Fires when: kube_deployment_status_replicas_available{deployment="kyverno-admission-controller", namespace="kyverno"} < kube_deployment_spec_replicas{...}, for: 10m. Severity: warning. Any missing replica trips this (replicas: 3), not total failure.

Diagnose

  • kubectl get deployment kyverno-admission-controller -n kyverno
  • kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller Resolution: [Git] fix the deployment config or resources.

KyvernoAdmissionControllerAllDown

Fires when: the same ..._replicas_available metric == 0, for: 2m. Severity: critical. forceFailurePolicyIgnore=true means policies fail open — violations pass admission silently while this fires.

Diagnose

  • kubectl get pods -n kyverno
  • kubectl describe deployment kyverno-admission-controller -n kyverno Resolution: [Git] fix the deployment. Treat the fail-open window as a security gap — re-audit anything admitted during it.

KyvernoReportsControllerDown

Fires when: kube_deployment_status_replicas_available{deployment="kyverno-reports-controller", namespace="kyverno"} == 0, for: 10m. Severity: warning. Impact: PolicyReports go stale — dashboards and audit views are no longer trustworthy.

Diagnose

  • kubectl get deployment kyverno-reports-controller -n kyverno
  • kubectl logs -n kyverno -l app.kubernetes.io/component=reports-controller Resolution: [Git] fix the deployment config.

KyvernoBackgroundControllerDown

Fires when: kube_deployment_status_replicas_available{deployment="kyverno-background-controller", namespace="kyverno"} == 0, for: 15m. Severity: warning. Impact: background mutate/generate rules stop firing on existing resources. New admissions are unaffected.

Diagnose

  • kubectl get deployment kyverno-background-controller -n kyverno
  • kubectl logs -n kyverno -l app.kubernetes.io/component=background-controller Resolution: [Git] fix the deployment config.

KyvernoCleanupControllerDown

Fires when: kube_deployment_status_replicas_available{deployment="kyverno-cleanup-controller", namespace="kyverno"} == 0, for: 30m. Severity: warning. Impact: CleanupPolicy / DeletingPolicy resources are not reconciled.

Diagnose

  • kubectl get deployment kyverno-cleanup-controller -n kyverno
  • kubectl logs -n kyverno -l app.kubernetes.io/component=cleanup-controller Resolution: [Git] fix the deployment config.

KyvernoAdmissionLatencyP99High

Fires when: the recording rule kyverno:admission_request_duration_seconds:p99_5m > 1 (second), for: 15m. Severity: warning.

Diagnose

  • PromQL: kyverno:admission_request_duration_seconds:p99_5m and :p50_5m — a high p99 with normal p50 points at a specific slow policy
  • kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller Resolution: [Git] simplify or narrow the offending policy's match rules, or raise admission controller resources.

KyvernoAdmissionLatencyNearWebhookTimeout

Fires when: kyverno:admission_request_duration_seconds:p99_5m > 8, for: 5m. Severity: critical. The webhook timeout is 10s — at p99 > 8s requests start timing out and policies silently soft-fail. Diagnose: as above, urgently. Resolution: [Git] reduce policy cost or scale the admission controller. This is a precursor to an undetected fail-open state.

KyvernoPolicyValidationFailureSpike

Fires when: kyverno:policy_results_fail:rate_5m > on() (kyverno:policy_results_fail:rate_5m offset 1h * 3 + 0.05), for: 15m. Severity: warning. That is: the current failure rate exceeds 3x the rate one hour ago, plus a 0.05/s floor to suppress noise at low volumes. It is a relative spike detector, not an absolute threshold. Likely causes: a newly deployed violating workload, or a policy just promoted to Enforce.

Diagnose

  • PromQL: sum by (policy_name, rule_result) (rate(kyverno_policy_results_total[5m]))
  • kubectl get policyreports -A Resolution: [Git] fix the violating workload manifest, or reconsider the newly-enforced policy.

KyvernoPolicyLoadFailures

Fires when: increase(kyverno_policy_changes_total{policy_change_type="fail"}[10m]) > 0, for: 5m. Severity: warning. Likely causes: malformed policy CRD, schema drift after an upgrade, or a permission issue.

Diagnose

  • kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller
  • kubectl get clusterpolicies / kubectl get policies -A — check Ready status Resolution: [Git] fix the policy manifest. Validate with the /verify-kyverno-policies skill before merging.

KyvernoReportsBacklog

Fires when: kube_customresource_count{group="wgpolicyk8s.io", kind="PolicyReport"} > 8000, for: 30m. Severity: warning. Approaching the etcd pressure threshold (~10k reports / ~470MB) — this is an upstream cause of EtcdDBSizeApproachingQuota.

Diagnose

  • PromQL: kube_customresource_count{group="wgpolicyk8s.io", kind="PolicyReport"}
  • kubectl get policyreports -A --no-headers | wc -l Resolution: [Git] configure --allowedResults filtering or adopt reports-server.

KyvernoControllerRestarts

Fires when: increase(kube_pod_container_status_restarts_total{namespace="kyverno"}[1h]) > 3, for: 5m. Severity: warning. Covers any container in the kyverno namespace, not just the admission controller.

Diagnose

  • kubectl get pods -n kyverno
  • kubectl logs <pod> -n kyverno --previous
  • kubectl describe pod <pod> -n kyverno — check for OOMKill Resolution: [Git] fix resources or config. Note the Kyverno check-memory-limits policy applies to its own chart bumps.

KyvernoWebhookErrors

Fires when: sum(rate(kyverno_admission_requests_total{webhook=~".*kyverno.*", code=~"5.."}[5m])) > 0.1 (per second), for: 10m. Severity: warning.

Diagnose

  • PromQL: sum by (code, webhook) (rate(kyverno_admission_requests_total{webhook=~".*kyverno.*", code=~"5.."}[5m]))
  • kubectl logs -n kyverno -l app.kubernetes.io/component=admission-controller Resolution: [Git] fix the failing policy or controller config.

KyvernoForceFailurePolicyIgnoreActive

Fires when: sum(kyverno_admission_requests_total{rule_validation_failure_action="ignore"}) > 0, for: 1h. Severity: info. This is an intentional reminder, not a fault: a cluster-wide failurePolicy override is in effect, so Kyverno fails open. Diagnose: no incident response required. Confirm the override is still intended. Resolution: [Git] disable forceFailurePolicyIgnore per-policy once admission HA is trusted. Not urgent; do not page on this.

KyvernoExceptionExpired

Fires when: count(kube_customresource_info{group="kyverno.io", kind="PolicyException"}) > 0, for: 1h. Severity: warning. Read the name carefully: this fires whenever any PolicyException exists — it cannot actually detect expiry, because kube-state-metrics does not expose the expiry annotation as a timestamp metric. It is a prompt to run the audit, not evidence that something expired.

Diagnose

  • kubectl get policyexceptions -A -o yaml — inspect expiry annotations by hand
  • Run the exception-audit Job defined at cluster/kyverno/test/expiration-for-policyexceptions Resolution: [Git] remove expired PolicyException manifests. An expired exception left in place is a live policy hole.