Skip to content

Runbook — secret rotation 2026-04

Triggered by gitleaks historical scan on 2026-04-15 (16 unique findings across 934 commits).

Status: HEAD clean (0 findings as of fix commit). Historical commits still contain plaintext values — they remain valid auth tokens until rotated.

What to rotate (in order)

Tier 0 — high blast radius

# Item Where to rotate Where it was leaked
1 ArgoCD admin API token (JWT, admin:apiKey, iat 2026-03-12) ArgoCD UI → User Info → Generate API token → revoke old .mcp.json:8 (commits before fix)
2 Grafana admin password (jdlzIhE…) kubectl -n monitoring exec -it prometheus-grafana-0 -- grafana-cli admin reset-admin-password <new> then update sealed secret argocd-apps/monitoring/kube-prometheus-stack.yaml:20 at commit 5d38b9e0

Tier 1 — *arr / homepage widget API keys

For each app: web UI → Settings → General → Security → API Key → Generate (revokes old).

# App URL
3 Sonarr https://sonarr.media.cauda.dev
4 Sonarr 4K https://sonarr-4k.media.cauda.dev
5 Sonarr Anime https://sonarr-anime.media.cauda.dev
6 Radarr https://radarr.media.cauda.dev
7 Radarr 4K https://radarr-4k.media.cauda.dev
8 Lidarr https://lidarr.media.cauda.dev
9 Prowlarr https://prowlarr.media.cauda.dev
10 SABnzbd https://sabnzbd.media.cauda.dev → Config → General → API Key → Change
11 Overseerr https://seerr.media.cauda.dev → Settings → API Key → Refresh
12 Paperless-ngx https://docs.office.cauda.dev → User → My Profile → API Token → Regenerate
13 Gitea https://git.dev.cauda.dev → Settings → Applications → Generate New Token

Tier 2 — verify (✅ verified clean 2026-04-16)

History audit confirms none of the four candidates were ever plaintext-committed — all references in tracked files were either variable substitutions, sealed-secret templates with CHANGE_ME placeholders, or already-encrypted SealedSecret blobs. No Tier-2 rotation action is required.

# Item Audit result
14 Cloudflare API token NOT FOUND in history. terraform/init-tfvars.sh references vault.yml; terraform.tfvars.example has placeholder only.
15 Synology DSM k8s-csi password NOT FOUND in history. sealed-secrets/synology-csi-client-info.secret.yaml.tpl has CHANGE_ME_STRONG_PASSWORD placeholder; deployed sealed-secrets contain only encrypted Ag… blobs.
16 QNAP NAS password NOT FOUND in history. ansible/inventory/home.yml references {{ vault_qnap_home_password }}; actual ansible/vault.yml is gitignored.
17 Authentik bootstrap admin password NOT FOUND in history. sealed-secrets/authentik.secret.yaml.tpl has CHANGE_ME placeholder; sealed-secrets are encrypted blobs.

After rotating

For each app where the new key is needed by Homepage:

  1. Fill in sealed-secrets/homepage-widget-keys.secret.yaml (copy from the .tpl).
  2. Seal it via /seal-secret skill or:
kubeseal --controller-namespace=sealed-secrets --controller-name=sealed-secrets \
         --format=yaml < sealed-secrets/homepage-widget-keys.secret.yaml \
       > sealed-secrets/sealed/homepage-widget-keys.sealed.yaml
shred -u sealed-secrets/homepage-widget-keys.secret.yaml
  1. Wire the SealedSecret into the deploy via an Application that references sealed/.
  2. ArgoCD sync → Homepage pod restart → widgets repopulate.

Optional: history rewrite

Even after rotation, the old values stay in git history. They're useless tokens once rotated, but:

  • A scrape of the repo (e.g., GitHub fork) preserves them.
  • Any future leaks of metadata (issue date, naming patterns) could correlate.

To remove from history:

# Install git-filter-repo: https://github.com/newren/git-filter-repo
git filter-repo --replace-text replacements.txt
git push --force-with-lease origin main

Where replacements.txt lists <old-value>==>***REDACTED*** per line.

Consequences:

  • All commit SHAs change. All open branches/PRs need rebase.
  • All collaborators must re-clone (git pull won't work).
  • GitHub caches refs for ~90 days post-deletion (use GHSA cache flush if needed).
  • Tags and submodule pins pointing to old SHAs break.

Recommendation: rotate first (most of the protection). Defer history rewrite unless a specific compliance requirement demands it.

Verification

After rotation:

# 1. HEAD scan (should be clean)
gitleaks detect --source . --config .gitleaks.toml --no-git --redact

# 2. Historical scan (still shows old values until rewrite, but they're now invalid)
gitleaks detect --source . --config .gitleaks.toml --redact