Skip to content

Authentik alerts

Source rule: cluster/monitoring/authentik-alerts.yaml (PrometheusRule authentik-alerts, namespace monitoring, groups authentik-availability, authentik-worker, authentik-tasks-blueprints, authentik-outposts, authentik-enterprise).

Authentik is the cluster IdP. A full outage 502s every OIDC app and every proxy-outpost-backed app cluster-wide, so the availability alerts in this file are paging-grade.

Verified topology (from cluster/network-policies/cnp-authentik.yaml, argocd-apps/auth/authentik.yaml, cluster/cloudnative-pg/cluster.yaml):

Component Selector Notes
Server pods app.kubernetes.io/name=authentik,app.kubernetes.io/component=server HPA minReplicas: 2, maxReplicas: 6
Worker pods app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker Fixed replicas: 2 — no HPA
Proxy outposts app.kubernetes.io/name=authentik-outpost-proxy, or goauthentik.io/outpost-name=authentik-proxy-outpost-{home,office} Deployments ak-outpost-authentik-proxy-outpost-{home,office}
PostgreSQL ns databases, cnpg.io/cluster=pg-cluster Service pg-cluster-rw. cnpg-system is the operator namespace, not the data plane

Ports: 9000 = app HTTP (this is where /-/health/live/ and /-/health/ready/ live), 9300 = Prometheus metrics. Do not curl the health endpoints on 9300.

Worker-metric dependency. authentik_tasks_workers, authentik_tasks_errors_total, authentik_tasks_queued and django_db_new_connection_errors_total come from the worker ServiceMonitor (worker.metrics.serviceMonitor.enabled=true). If that ServiceMonitor is removed or the worker target stops being scraped, these series vanish and the alerts below go inactive rather than firing — absence of alerts is not evidence of health. Confirm the target exists before trusting silence.

GitOps. This cluster is mutation-free outside Git. Every command in this runbook is read-only. All fixes land as a PR that ArgoCD reconciles — never kubectl apply/patch/scale/delete/edit/annotate/cordon/drain a live object to resolve one of these alerts.

AuthentikServerAllTargetsDown

Severity: critical

Fires when: absent(up{job="authentik-server-metrics"} == 1) sustained for: 3m — no authentik-server-metrics target has reported up == 1 for three minutes.

Note this uses absent(), so it also fires if the job disappears entirely (ServiceMonitor deleted, relabelling change) — not only if the pods are down.

Likely causes

  1. The authentik-server Deployment is crashed or CrashLoopBackOff.
  2. Pods are running but not scrapeable — Service/Endpoint or CiliumNetworkPolicy change blocking Prometheus on port 9300.
  3. The ServiceMonitor itself was removed or its release: kube-prometheus-stack label dropped, so the job no longer exists.
  4. Simultaneous loss of both sites.

Diagnose (read-only)

  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server -o wide
  • kubectl describe deployment authentik-server -n authentik
  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server --tail=100
  • kubectl get servicemonitor -n authentik -o wide — confirm the server ServiceMonitor still exists
  • PromQL: up{job="authentik-server-metrics"}

Resolution: If pods are down, read the logs for the reason — the recurring one is Postgres unreachability (see AuthentikDBConnectionErrors). If the ServiceMonitor or a policy is at fault, that is a Git change: edit argocd-apps/auth/authentik.yaml or cluster/network-policies/cnp-authentik.yaml and let ArgoCD sync. Do not hand-apply a Service or ServiceMonitor to restore scraping.

AuthentikServerTargetsDegraded

Severity: warning

Fires when: count(up{job="authentik-server-metrics"} == 1) < 2 sustained for: 5m — fewer than two server targets are up.

The HPA floor is minReplicas: 2, so normal scale-down cannot take the count below 2. Reaching this state means pods are failing, unschedulable, or unscrapeable — not that autoscaling shrank the deployment.

Likely causes

  1. One or more pods failing the readiness probe (most often Postgres connectivity) and dropping out of Endpoints.
  2. Pods pending — insufficient CPU/memory to schedule, or a node down in one site.
  3. Eviction or node pressure removing a replica.

Diagnose (read-only)

  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server -o wide
  • kubectl get hpa -n authentik
  • kubectl describe pod -n authentik <pod-name> — check Events for scheduling or probe failures
  • kubectl top pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server
  • PromQL: count(up{job="authentik-server-metrics"} == 1)

Resolution: If pods are unready, chase the readiness cause (usually DB). If pods are Pending for capacity, resource requests or maxReplicas are a Git change in argocd-apps/auth/authentik.yaml — the memory request was already raised to 768Mi and max to 6 for this reason. Never scale the deployment live to clear the alert.

AuthentikServerReplicasNotReady

Severity: warning

Fires when: kube_deployment_status_replicas_ready{namespace="authentik", deployment="authentik-server"} < kube_deployment_spec_replicas{namespace="authentik", deployment="authentik-server"} sustained for: 10m.

Depends on kube-state-metrics being healthy. This catches pods whose /metrics stays up while /-/health/ready/ fails — the shape of the 2026-06-03 incident (3/5 office-1 pods NotReady).

Likely causes

  1. /-/health/ready/ failing because a PostgreSQL connection cannot be established — per upstream, that check is the DB check.
  2. Startup exceeding the probe budget (startupProbe is 30s delay, 10s period, 60 failures).
  3. CPU throttling or OOMKill before the pod reaches ready.

Diagnose (read-only)

  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server
  • kubectl describe pod -n authentik <pod-name> — probe failure counts and OOMKilled status
  • kubectl logs -n authentik <pod-name> --tail=100
  • kubectl exec -n authentik <pod-name> -- curl -s http://localhost:9000/-/health/ready/ — health is on 9000, not 9300
  • kubectl get cluster -n databases pg-cluster and kubectl get pods -n databases -l cnpg.io/cluster=pg-cluster
  • PromQL: kube_deployment_status_replicas_ready{namespace="authentik", deployment="authentik-server"}

Resolution: Confirm pg-cluster-rw is reachable from the affected pod's node before touching Authentik. If the DB is healthy and the app is erroring, read the logs. Probe timings and resource limits are a Git change in argocd-apps/auth/authentik.yaml.

AuthentikNoWorkersConnected

Severity: critical

Fires when: max(authentik_tasks_workers) < 1 sustained for: 10m.

No worker means no blueprint apply, no outpost reconcile, no certificate rotation and no cleanup. The IdP keeps serving logins, so this is silent unless alerted on.

Likely causes

  1. Worker pods are down or crashlooping.
  2. Workers are running but cannot register with the server (DNS, network policy, or DB).
  3. The worker ServiceMonitor is not scraping, so the series is absent — note max() of an empty vector produces no result, so a missing target makes this alert go quiet rather than fire. Verify the target directly.

Diagnose (read-only)

  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker -o wide
  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker --tail=100
  • PromQL: authentik_tasks_workers — if this returns nothing at all, the problem is scraping, not workers
  • PromQL: up{job=~"authentik-worker.*"} — confirm the worker target exists

Resolution: Worker replica count is fixed at 2 in argocd-apps/auth/authentik.yaml; there is no HPA to blame. If workers are missing, check the ArgoCD sync status of the authentik Application. Restoring replicas or fixing the ServiceMonitor is a Git change, not a live scale.

AuthentikWorkerVersionMismatch

Severity: warning

Fires when: max(authentik_tasks_workers{version_matched="False"}) > 0 sustained for: 15m — at least one connected worker reports a version that does not match the server.

Likely causes

  1. A partial rollout left workers and server on different versions.
  2. A stuck ReplicaSet keeping old worker pods alive.
  3. Image pull failure leaving a pod on a cached older image.

Diagnose (read-only)

  • kubectl get rs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker
  • kubectl describe deployment authentik-worker -n authentik
  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'
  • PromQL: authentik_tasks_workers{version_matched="False"}

Resolution: Do not run kubectl rollout restart. Completing the roll is a Git change — correct the image tag in argocd-apps/auth/authentik.yaml and let ArgoCD reconcile. If a ReplicaSet is genuinely wedged and Git already holds the right version, escalate rather than deleting pods by hand.

AuthentikDBConnectionErrors

Severity: critical

Fires when: sum(rate(django_db_new_connection_errors_total{namespace="authentik"}[5m])) > 0 sustained for: 5m — any non-zero rate of new-connection failures over a 5m window, held for 5m.

This is the leading indicator for AuthentikServerReplicasNotReady: connection failures break /-/health/ready/, pods leave the Service, and auth degrades.

Likely causes

  1. pg-cluster unhealthy, failing over, or unreachable.
  2. Connection-limit or pool exhaustion on the CNPG side.
  3. CiliumNetworkPolicy blocking authentik → databases on 5432.

Diagnose (read-only)

  • kubectl get cluster -n databases pg-cluster
  • kubectl get pods -n databases -l cnpg.io/cluster=pg-cluster -o wide
  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server --tail=100
  • PromQL: sum(rate(django_db_new_connection_errors_total{namespace="authentik"}[5m]))
  • PromQL: sum(rate(django_db_new_connection_errors_total{namespace="authentik"}[5m])) by (pod) — is it one pod or all of them?

Resolution: Fix the database first; Authentik recovers on its own once connections succeed. Connection limits, pooler settings and network policy are all Git changes (cluster/cloudnative-pg/, cluster/network-policies/cnp-authentik.yaml). Note Authentik connects to pg-cluster-rw directly — check whether the pooler is in path before tuning pool sizes.

AuthentikBlueprintApplyFailing

Severity: warning

Fires when: sum(rate(authentik_tasks_errors_total{namespace="authentik", actor_name=~"authentik.blueprints.*"}[15m])) > 0 sustained for: 15m.

Blueprint apply is how Git-declared providers, outposts and policy bindings reach the live IdP. Failures mean the running IdP is drifting from Git while ArgoCD still reports Synced.

Likely causes

  1. An invalid or newly-broken blueprint in argocd-apps/auth/authentik-blueprints.yaml.
  2. A blueprint referencing an object that does not exist yet (ordering dependency).
  3. DB lock or timeout during apply.

Diagnose (read-only)

  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker --tail=200 | grep -i blueprint
  • PromQL: sum by (actor_name) (rate(authentik_tasks_errors_total{namespace="authentik", actor_name=~"authentik.blueprints.*"}[15m]))
  • kubectl get configmap -n authentik — blueprint ConfigMaps mounted for the worker

Resolution: Identify the failing blueprint from the worker log, then fix it in the Git repository — this is always a Git change; blueprints are not editable live in a way that survives. Do not "fix" drift by editing objects in the Authentik UI: the next successful apply will revert it, and the drift-detector will flag it.

AuthentikOutpostReconcileFailing

Severity: warning

Fires when: sum(rate(authentik_tasks_errors_total{namespace="authentik", actor_name=~"authentik.outposts.*"}[15m])) > 0 sustained for: 15m.

The worker cannot push configuration to the proxy outposts, so outposts keep serving stale forward-auth config for media and infra apps.

Likely causes

  1. Outpost pods down or unreachable from the worker.
  2. The outpost's service account or token is invalid.
  3. Kubernetes API errors while the integration reconciles the outpost Deployment.

Diagnose (read-only)

  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker --tail=200 | grep -i outpost
  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik-outpost-proxy -o wide
  • kubectl get deployment -n authentik ak-outpost-authentik-proxy-outpost-home ak-outpost-authentik-proxy-outpost-office
  • PromQL: sum by (actor_name) (rate(authentik_tasks_errors_total{namespace="authentik", actor_name=~"authentik.outposts.*"}[15m]))

Resolution: Outpost Deployments are created by the Authentik Kubernetes integration from IdP state, not from manifests in this repo — so there is no Git file to edit for the Deployment itself. Fix the underlying cause (worker health, token, API permissions). Outpost definitions do come from blueprints, so a bad outpost definition is a Git change in the blueprints app.

AuthentikTaskErrorsElevated

Severity: warning

Fires when: sum by (actor_name) (rate(authentik_tasks_errors_total{namespace="authentik"}[15m])) > 0.05 sustained for: 20m — any single actor exceeding 0.05 errors/sec.

This is the broad backstop for actors without a dedicated alert (flows, policies, crypto, SCIM). The threshold is 0.05/s, not "greater than zero" — low-rate errors are tolerated here and caught by the specific alerts above.

Likely causes

  1. Flow or stage execution failures from a recent policy change.
  2. An external dependency failing (SMTP, LDAP, SCIM target).
  3. Certificate rotation or crypto task errors.

Diagnose (read-only)

  • PromQL: topk(5, sum by (actor_name) (rate(authentik_tasks_errors_total{namespace="authentik"}[15m]))) — identify the actor first
  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker --tail=200 | grep -i error
  • PromQL: sum by (actor_name) (rate(authentik_tasks_errors_total{namespace="authentik"}[15m]))

Resolution: Always start from the actor_name in the alert label — the remediation differs entirely per actor. Configuration owned by blueprints is a Git change; an external dependency outage is fixed at the dependency.

AuthentikTaskQueueBacklog

Severity: warning

Fires when: sum(authentik_tasks_queued{namespace="authentik"}) > 50 sustained for: 15m.

A growing queue delays outpost reconcile, certificate rotation and cleanup.

Likely causes

  1. Workers wedged or stuck on a slow task — check authentik_tasks_workers first.
  2. A burst of work (mass blueprint sync, large SCIM push) that will drain on its own.
  3. Workers CPU-starved.

Diagnose (read-only)

  • PromQL: sum(authentik_tasks_queued{namespace="authentik"}) — check the trend; flat-and-high means wedged, sawtooth means draining
  • PromQL: authentik_tasks_workers
  • kubectl top pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker
  • kubectl logs -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=worker --tail=100

Resolution: Worker count is a fixed replicas: 2 with no HPA, so this will not self-heal by scaling. If workers are genuinely undersized, raising worker.replicas or worker resources is a Git change in argocd-apps/auth/authentik.yaml.

AuthentikProxyOutpostDown

Severity: critical

Fires when: up{job=~"ak-outpost-authentik-proxy-outpost-(home|office)-metrics"} == 0 sustained for: 5m.

The regex deliberately matches only the live home/office outposts — legacy duplicate outpost targets from 2026.2.1 are excluded so they cannot page. Each proxy outpost is the forward-auth gate for roughly 10-15 apps; losing one 502s that zone's apps.

Likely causes

  1. The outpost pod crashed or is not scheduled.
  2. Prometheus cannot reach port 9300 on the outpost (network policy).
  3. Node failure in the affected zone.

Diagnose (read-only)

  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik-outpost-proxy -o wide
  • kubectl describe pod -n authentik <outpost-pod-name>
  • kubectl logs -n authentik <outpost-pod-name> --tail=100
  • PromQL: up{job=~"ak-outpost-authentik-proxy-outpost-(home|office)-metrics"} — the job label in the alert tells you which zone

Resolution: Outpost Deployments are reconciled by the Authentik worker from IdP state, so recovery normally follows from a healthy worker (see AuthentikNoWorkersConnected). Do not delete or scale the outpost Deployment by hand. Network-policy fixes are a Git change in cluster/network-policies/cnp-authentik.yaml.

AuthentikOutpostDisconnected

Severity: warning

Fires when: authentik_outpost_connection{outpost_type="proxy"} == 0 and on (pod) up{job=~"ak-outpost-authentik-proxy-outpost-(home|office)-metrics"} == 1 sustained for: 10m.

The and on (pod) join is the point: the outpost pod is up and scrapeable but reports no websocket to the Authentik core. It keeps serving traffic using stale provider config — strictly worse than being down, because nothing fails over.

Likely causes

  1. Network path from outpost to authentik-server broken (CiliumNetworkPolicy, DNS).
  2. The server is overloaded and rejecting or dropping websocket connections.
  3. The outpost process is wedged and not re-establishing the connection.

Diagnose (read-only)

  • kubectl logs -n authentik <outpost-pod-name> --tail=100 | grep -iE 'connect|websocket'
  • PromQL: authentik_outpost_connection{outpost_type="proxy"}
  • kubectl get pods -n authentik -l app.kubernetes.io/name=authentik,app.kubernetes.io/component=server — is the server healthy?
  • kubectl exec -n authentik <outpost-pod-name> -- curl -s -o /dev/null -w '%{http_code}' http://authentik-server.authentik.svc:9000/-/health/ready/

Resolution: Check server health and the outpost→server policy on port 9000 first. Policy changes are a Git change in cluster/network-policies/cnp-authentik.yaml. If the outpost pod is genuinely wedged with everything else healthy, note that there is no Git-side annotation to bump — the outpost Deployment is operator-managed — so escalate rather than restarting it by hand.

AuthentikLicenseExpiringSoon

Severity: warning

Fires when: min(authentik_enterprise_license_expiry_seconds) > 0 and min(authentik_enterprise_license_expiry_seconds) < 1209600 sustained for: 1h — expiry under 14 days.

The > 0 guard is load-bearing: on community edition the metric reads 0, and without the guard the alert would fire permanently. If this alert fires, an enterprise license genuinely is configured and genuinely is expiring.

Likely causes

  1. The enterprise license is approaching expiry and has not been renewed.

Diagnose (read-only)

  • PromQL: min(authentik_enterprise_license_expiry_seconds)
  • PromQL: min(authentik_enterprise_license_expiry_seconds) / 86400 — days remaining
  • kubectl get secret -n authentik — locate the license secret (do not print its contents)

Resolution: Renew with Authentik, then update the sealed license secret in the repository. This is a Git change end to end — seal the new value and let ArgoCD sync it. Per the secret-rotation rule, seal and merge first, then roll the pods, since env secretKeyRef values are snapshotted at pod creation.