Skip to content

Runbook — LiteLLM DB credential rotation

Rotate the PostgreSQL password for the CNPG litellm role and the LiteLLM gateway's DATABASE_URL, keeping the two GitOps-owned SealedSecrets in sync.

Status: OPEN operator action for issue #2886. The Prisma-argv exposure that leaked this credential is already closed in-repo (derivative image 1.93.0-cordana1, PR #2888): all three source-pinned --from-url/--to-url call sites now resolve the URL from the child DATABASE_URL environment. The already-leaked password must still be rotated on the live cluster. This runbook is operator-executed — it needs new secret material, kubeseal against the live controller, and live reconciliation observation. No plaintext ever lands in the repo.

Two-secret ownership boundary

The password lives in two SealedSecrets with distinct owners. Both must carry the identical password or LiteLLM cannot authenticate to Postgres.

SealedSecret → Secret Namespace Owner Role
pg-cluster-litellm (username, password; kubernetes.io/basic-auth, label cnpg.io/reload: "true") databases CNPG Source of truth. The managed role litellm in cluster/cloudnative-pg/cluster.yaml reconciles the Postgres role password from this Secret.
litellm-database (DATABASE_URL) ai LiteLLM app Consumer. The full postgresql://litellm:<password>@… URL. Its password component must equal the role password.

Files: sealed-secrets/sealed/pg-cluster-litellm.sealed.yaml, sealed-secrets/sealed/litellm-database.sealed.yaml. Both sit under the edit-protected sealed-secrets/sealed/** path — reseal them from plaintext with kubeseal, never hand-edit the ciphertext.

Why the ordering is consumer-first and self-gating

  • App env secretKeyRef is snapshotted at pod create; resealing a Secret never mutates a running pod in place.
  • The Deployment carries secret.reloader.stakater.com/auto: "true", so resealing litellm-database makes Stakater Reloader roll it.
  • The rollout strategy is maxUnavailable: 0, maxSurge: 1: the surge pod comes up on the NEW URL while the OLD pod keeps serving. The surge pod cannot pass its DB-backed readiness probe until the Postgres role password also matches, so it stays NotReady and the old pod stays available — exactly the guarantee in step 1. Completing the role reseal (step 2) lets the surge pod authenticate and the rollout finishes.
  • The PreSync litellm-migrations Job belongs to the LiteLLM Application, not to the SealedSecret Application, so resealing these Secrets does not re-run the schema writer.

Prerequisites

  • kubeseal pointed at the live sealed-secrets controller (see /seal-secret).
  • kubectl context on the site whose CNPG cluster owns the litellm role.
  • A new URL-safe password (percent-encode any reserved characters in the URL).

Steps

  1. Stage the consumer Secret. Reseal litellm-database with the new password embedded in DATABASE_URL — every other URL component unchanged (the CNPG read-write service pg-cluster-rw.databases.svc.cluster.local). Commit and let ArgoCD sync. Reloader surges a new pod; confirm the existing pod stays Ready and serving while the surge pod is NotReady:
kubectl -n ai get pods -l app.kubernetes.io/name=litellm -w

  1. Rotate the role Secret. Reseal pg-cluster-litellm with the SAME password (raw, in the password key; username stays litellm). Commit and let ArgoCD sync.

  2. Confirm CNPG reconciled the role. The cnpg.io/reload label drives the operator to apply the new password to the managed role:

kubectl -n databases get secret pg-cluster-litellm -o jsonpath='{.metadata.resourceVersion}{"\n"}'
kubectl -n databases get cluster pg-cluster -o jsonpath='{.status.managedRolesStatus}{"\n"}'
  1. Confirm the rollout. The surge pod authenticates and goes Ready, the old pod drains, and the Application is Synced/Healthy with zero restarts:
kubectl -n ai get pods -l app.kubernetes.io/name=litellm
kubectl -n ai rollout status deploy/litellm

Then prove a DB-backed operation still works (an authenticated /key/info read or a spend-logged completion; see alerts/litellm-gateway.md).

  1. Destroy plaintext. Shred the new-password working files; the value now exists only inside the two SealedSecrets and the live cluster.

Rollback

  • Before step 2 (role not yet rotated): revert only litellm-database to its prior sealed value. Reloader rolls back to the old, working URL; the role never changed.
  • After step 2 (role rotated): restore the prior pg-cluster-litellm Secret FIRST so CNPG reconciles the role back to the old password, THEN restore litellm-database. Reverting the consumer first would point the pod at the old URL while the role holds the new password — an auth failure.

What can go wrong

  • Resealing both Secrets in one commit is fine, but the surge pod may crashloop until CNPG finishes reconciling the role — that is the self-gating behaviour, not a failure. Do not force-delete the old pod.
  • A non-URL-safe password breaks DATABASE_URL parsing without breaking the raw role password, leaving the two Secrets disagreeing. Percent-encode or choose a URL-safe alphabet.
  • Never hand-edit sealed-secrets/sealed/*.sealed.yaml ciphertext; always reseal from plaintext with kubeseal.

References

  • issue #2886; PR #2888 (argv fix)
  • apps/ai/litellm/README.md → "Rollout, schema ownership, and drain"
  • cluster/cloudnative-pg/cluster.yaml (managed role litellm), cluster/cloudnative-pg/databases.yaml
  • apps/ai/litellm/AGENTS.md