Skip to content

Loki log-based alerts

Source rule: cluster/monitoring/loki-log-alerts.yaml (ConfigMap loki-log-alerts, ns monitoring, label loki_rule="", group cauda-flow-pipeline, interval: 1m).

These are not PrometheusRule alerts. They are LogQL rules evaluated by the Loki ruler, loaded by the loki-sc-rules sidecar into /rules/fake/ (tenant fake, since auth_enabled=false) and dispatched to Alertmanager with external label source=loki. Every expression below is LogQL over log streams — not PromQL over metrics. Query them in Grafana against the Loki datasource, not Prometheus.

All cluster changes go through Git → ArgoCD. Every command below is read-only; do not apply, patch, scale, delete, edit, annotate, cordon, or drain anything while working these alerts.

Pipeline context: Alloy (ns monitoring, app.kubernetes.io/name=alloy) ships Hubble and Tetragon flow logs into Loki. Cilium 1.19.6 and Tetragon 1.7.0 both run in ns kube-system. The Hubble export applies a fieldMask and a denyList (see argocd-apps/infra/cilium.yaml) — flows with a kube-system source or destination are dropped before they reach Loki, so kube-system traffic is invisible to every Hubble alert here.

HubbleRelaySilent

Severity: critical · team: platform

Fires when: absent_over_time({flow_log="hubble"}[10m]) holds for: 5m — the flow_log="hubble" stream has produced zero entries over a trailing 10-minute window, and that absence persists for a further 5 minutes. Worst case this is ~15 minutes of blind time before the page.

Likely causes

  1. hubble-relay pods down or crash-looping, or the Cilium agent's Hubble export is disabled/misconfigured.
  2. Alloy is not scraping or not pushing — stuck, OOMKilled, or failing to reach the Loki write path.
  3. The Loki write path itself is broken (distributor/ingester unavailable). Expect LokiIngestionStalled alongside if so.
  4. Genuine total quiet: the denyList drops kube-system-sourced flows, so a cluster that is only running kube-system traffic can legitimately go silent. Confirm before escalating.

Diagnose (read-only)

kubectl -n kube-system get pods -l k8s-app=hubble-relay -o wide
kubectl -n kube-system logs -l k8s-app=hubble-relay --tail=100
kubectl -n monitoring get pods -l app.kubernetes.io/name=alloy -o wide
kubectl -n monitoring logs -l app.kubernetes.io/name=alloy --tail=100
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose   # Cilium >=1.16 ships cilium-dbg, not cilium

LogQL — confirm the silence and find when it started:

count_over_time({flow_log="hubble"}[15m])

Resolution

  • Crash-looping relay or Alloy pods: read the logs and fix the cause. If the fix is a config/values/resource-limit change, it is a Git change to argocd-apps/infra/cilium.yaml (Hubble/relay) or the Alloy values, merged and synced by ArgoCD — do not edit live objects.
  • Transient pod restarts self-heal; the alert resolves once the stream resumes.
  • If Loki itself is the problem, work LokiIngestionStalled first — this alert is downstream of it.

TetragonRingbufLoss

Severity: warning · team: security

Fires when: sum(count_over_time({flow_log="tetragon"} |~ "ringbuf.*lost"[5m])) > 0 holds for: 2m. Note the threshold is greater than zero, not a rate — a single ringbuf-loss line in the trailing 5m window is enough, and because the 5m window keeps the count non-zero, one event alone can satisfy the 2m for:. Treat this as "any loss at all", not "sustained heavy loss".

Why it matters: dropped ringbuf events mean LSM/kprobe security observations never reached userspace. Those events are gone — this is a gap in the security audit trail, not just a performance blip.

Likely causes

  1. Event volume from an active workload exceeds the ring buffer drain rate (build jobs, scanners, mass file access).
  2. Ring buffer sized too small for the enabled TracingPolicy set (see cluster/tetragon/ — fileless-exec, kernel-module-load, privilege-escalation, sensitive-file-access, writable-dir-exec).
  3. The Tetragon export/userspace consumer is backlogged — CPU starvation or IO wait on the node.

Diagnose (read-only)

kubectl -n kube-system get pods -l app.kubernetes.io/name=tetragon -o wide
kubectl -n kube-system logs -l app.kubernetes.io/name=tetragon -c tetragon --tail=200 | grep -i "lost\|ringbuf"
kubectl top nodes
kubectl top pods -n kube-system -l app.kubernetes.io/name=tetragon

LogQL — quantify loss and attribute it to a node (mirror the alert's own aggregation):

sum(count_over_time({flow_log="tetragon"} |~ "ringbuf.*lost"[5m]))
sum by (node) (count_over_time({flow_log="tetragon"} |~ "ringbuf.*lost"[30m]))

Resolution

  • Identify the workload driving the event volume first; a single noisy pod is often the whole cause and may itself be the finding.
  • Increasing the ring buffer or Tetragon's resources is a Git change to apps/security/tetragon/values.yaml (the Helm values file referenced by argocd-apps/security/tetragon.yaml). That file currently sets no ring-buffer key, so you are adding one — check the Tetragon 1.7.0 chart's values schema for the correct key name before writing it rather than assuming. Merge and let ArgoCD sync; do not kubectl edit the DaemonSet.
  • Narrowing an over-broad TracingPolicy is likewise a Git change under cluster/tetragon/.

HubbleDropSurge

Severity: warning · team: platform

Fires when: sum(rate({flow_log="hubble"} | json | verdict="DROPPED"[5m])) > 50 holds for: 5m — the rate of log entries whose parsed verdict field equals DROPPED exceeds 50 per second, sustained for 5 minutes.

Likely causes

  1. A CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy just shipped that blocks legitimate traffic — correlate against recent ArgoCD syncs.
  2. A workload retry-storming against an endpoint it is not permitted to reach.
  3. Scanning or probing activity, internal or external.
  4. A backend genuinely down, so its clients' traffic is dropped rather than refused.

Diagnose (read-only)

The Hubble export fieldMask determines which fields exist after | json. Nested fields flatten with an underscore, so the available labels are source_namespace, source_pod_name, destination_namespace, destination_pod_name, IP_source, IP_destination, verdict, drop_reason_desc, traffic_direction, node_name, is_reply. There are no source_ip / destination_ip fields — grouping by those returns nothing.

sum by (drop_reason_desc) (rate({flow_log="hubble"} | json | verdict="DROPPED"[5m]))
sum by (source_namespace, source_pod_name, destination_namespace, destination_pod_name) (rate({flow_log="hubble"} | json | verdict="DROPPED"[5m]))
topk(10, sum by (IP_source, IP_destination) (rate({flow_log="hubble"} | json | verdict="DROPPED"[5m])))

drop_reason_desc is the fastest discriminator: POLICY_DENIED points at a network policy, while connection-tracking or unroutable reasons point at a broken backend or plumbing.

kubectl get cnp -A                     # CiliumNetworkPolicy (namespaced)
kubectl get ccnp                       # CiliumClusterwideNetworkPolicy
kubectl -n kube-system exec ds/cilium -- cilium-dbg status --verbose
kubectl -n kube-system logs -l k8s-app=cilium --tail=100

Resolution

  • If POLICY_DENIED dominates and the denied traffic is legitimate, the fix is a Git change to the policy under cluster/network-policies/ or the owning app's manifests. Never kubectl edit or kubectl delete a CiliumNetworkPolicy to clear a page — ArgoCD will revert it and you will have lost the audit trail.
  • If a workload is the source, the fix belongs in that workload's manifests (also Git).
  • If the traffic is genuinely hostile, the drops are the system working correctly; capture the source and treat it as a security finding rather than an alert to silence.

KyvernoAdmissionDenyBurst

Severity: warning · team: platform

Fires when: sum(rate({namespace="kyverno"} |~ "blocking admission request"[5m])) > 5 holds for: 5m — Kyverno log lines matching blocking admission request exceed 5 per second, sustained for 5 minutes.

The regex is version-sensitive: Kyverno >= 1.10 logs blocking admission request. The earlier policy.*blocked pattern never matched and was fixed in the 2026-05-29 audit. If Kyverno's log format changes on upgrade, this alert silently stops firing — re-verify the string after any Kyverno version bump.

Likely causes

  1. A newly merged policy is stricter than the workloads currently deployed — the classic case here is the check-memory-limits policy rejecting a chart bump whose sidecar has no resource limits.
  2. A controller reconcile-looping against a denied resource, retrying at high frequency (this is what turns a single bad manifest into a >5/s rate).
  3. An ArgoCD sync repeatedly attempting to apply non-compliant manifests.

Diagnose (read-only)

kubectl -n kyverno get pods -o wide
kubectl -n kyverno logs -l app.kubernetes.io/component=admission-controller --tail=200 | grep -i "blocking admission request"
kubectl get clusterpolicy
kubectl get policyreport -A --sort-by=.metadata.creationTimestamp | tail -30
kubectl get cpol -o wide

LogQL — mirror the alert, then attribute the denials:

sum(rate({namespace="kyverno"} |~ "blocking admission request"[5m]))
{namespace="kyverno"} |~ "blocking admission request"

Reading the raw matching lines identifies the offending resource and the policy that rejected it, which is the whole diagnosis.

Resolution

  • The near-universal fix is a Git change: either bring the workload manifest into compliance (usually adding resources.limits) or adjust the policy under cluster/cluster-policies/. Both go through PR → ArgoCD.
  • Do not delete or edit the ClusterPolicy live to unblock a deploy. Kyverno denials are the guardrail working; disabling the guardrail to ship a non-compliant manifest defeats the control and ArgoCD will revert it anyway.
  • If the rate is driven by a retry loop, fixing the underlying manifest stops the burst — there is no separate throttle to tune.

LokiIngestionStalled

Severity: critical · team: platform

Fires when: absent_over_time({namespace="monitoring", pod=~"loki-.*"}[5m]) holds for: 5m — Loki's own pod logs have produced zero entries over a trailing 5-minute window, sustained a further 5 minutes.

This is the silent-observability backstop: Loki failing to log about itself. The metric-based LokiDown alert should fire first; if only this one fires, suspect the collection path rather than Loki itself. The pod=~"loki-.*" selector deliberately matches all four Loki roles — the earlier service_name="loki" selector did not exist in this ingest pipeline and was fixed in the 2026-05-29 audit.

Caveat: if Loki's write path is fully broken, an alert evaluated by the Loki ruler about Loki may not fire at all. Absence of this page is not evidence of health — cross-check Prometheus.

Likely causes

  1. Loki pods down, crash-looping, or evicted.
  2. Alloy is not scraping ns monitoring — check Alloy first if Loki pods look healthy.
  3. Storage backend unavailable or full, so ingesters reject writes.
  4. Disk/PVC pressure on the nodes running Loki.

Diagnose (read-only)

kubectl -n monitoring get pods -l app.kubernetes.io/name=loki -o wide
kubectl -n monitoring get pods | grep '^loki-'
kubectl -n monitoring describe pod -l app.kubernetes.io/name=loki
kubectl -n monitoring logs -l app.kubernetes.io/name=loki --tail=200
kubectl -n monitoring logs -l app.kubernetes.io/name=loki --previous --tail=100   # last crash
kubectl -n monitoring get pvc
kubectl -n monitoring get events --sort-by=.lastTimestamp | tail -30
kubectl top pods -n monitoring

LogQL — confirm the gap (run against Loki; if Loki is down this will simply fail, which is itself the answer):

count_over_time({namespace="monitoring", pod=~"loki-.*"}[15m])

Cross-check from Prometheus, which is an independent path and stays available when Loki is not:

up{namespace="monitoring"}
kube_pod_status_phase{namespace="monitoring", pod=~"loki-.*"} == 1
kube_pod_container_status_restarts_total{namespace="monitoring", pod=~"loki-.*"}

Resolution

  • Crash-looping pods: read --previous logs for OOM or storage errors. Raising limits, fixing storage config, or resizing a PVC are all Git changes to the Loki values, merged and synced by ArgoCD.
  • Transient restarts self-heal and the alert clears on its own.
  • Full storage backend: free space or extend capacity at the backend; the retention config itself is a Git change (see /loki-retention-audit).
  • While Loki is down you are blind to all log-based alerts in this group, including the Hubble and Tetragon ones above. Treat restoring Loki as the priority and re-check the other four once ingestion resumes.