Skip to content

0015. Hybrid GitOps for Apps with PVC-Only Runtime State

  • Status: proposed
  • Date: 2026-05-27
  • Deciders: lusoris
  • Related: PR #436, PR #441, PR #442

Context

Several self-hosted apps in this cluster (Jellyfin, Gitea, qBittorrent, and others) keep significant configuration in a SQLite/SQL database or on-disk file tree under /config that:

  1. Is not rendered from Helm values — it's the product of first-run wizards, in-app settings dialogs, sideloaded plugin DLLs, OAuth dance results, or user-driven library scans.
  2. Cannot reasonably be re-derived from git on cluster rebuild — the data is the user-visible state of the application (watch history, library scan results, OAuth tokens, sideloaded plugins), not config.
  3. Has a strict subset of state that must stay in git for bootstrap correctness — sealed credentials for LDAP/OIDC bind, a break-glass admin password, ConfigMap-templated plugin XML so we don't ship secrets through the wizard, etc.

Jellyfin (PRs #436 + #441) is the forcing function for codifying the pattern. Its state breakdown:

Lives in git Lives on PVC
Helm chart (apps/media/jellyfin/) jellyfin.db (users, library scan, watch history)
LDAP-Auth.xml ConfigMap (template only — no secrets) Library VirtualFolders definitions in library.db
jellyfin-bootstrap-admin SealedSecret (break-glass admin) Per-item metadata + posters under /config/data/metadata/
jellyfin-ldap-creds SealedSecret (LDAP bind password) Plugin DLL binaries (downloaded by init container, sideloaded for Last.fm)
HTTPRoute, ServiceMonitor, etc. Trakt + Last.fm OAuth refresh tokens
install-plugins init container's plugin manifest Local user preferences (theme, layout)

Without an explicit decision, two failure modes recur:

  • "Just nuke and re-apply" — drops watch history, scrobble streaks, and library metadata enrichment that took hours to accumulate. The PVC is genuinely irreplaceable.
  • "Just back up everything" — drifts into the assumption that the PVC alone is sufficient, which loses the bootstrap secrets and the bind-password contract with Authentik on a cluster rebuild.

The setup that just shipped is deliberately hybrid: chart + sealed secrets + ConfigMap template are sufficient for a cold-start up to the wizard; the PVC carries everything that accumulates after the wizard.

Decision

Adopt Hybrid GitOps as the explicit pattern for any app where runtime state cannot reasonably round-trip through git:

  1. Git carries the bootstrap contract. Helm chart, sealed secrets for any externally-shared credential (LDAP bind, OAuth client secrets, break-glass admin), ConfigMap templates for files that normally hold both static config and secrets (e.g. LDAP-Auth.xml). An init container substitutes secret values at pod start so the git artifact never carries plaintext.
  2. PVC carries accumulated state. Database, scan results, metadata cache, sideloaded plugin binaries, OAuth tokens. The PVC is backed up via Longhorn RecurringJob (daily + weekly groups per zone, see cluster/longhorn/recurring-jobs-home.yaml).
  3. Every Hybrid app gets a restore runbook. A two-flow document under docs/runbooks/<app>-restore.md:
  4. Flow A: PVC restore from Longhorn backup (fast path, no wizard re-drive).
  5. Flow B: Full bootstrap from scratch (re-drives the wizard via the app's API, re-adds libraries, re-pulls plugin DLLs, points users at OAuth re-link UIs).
  6. Every Hybrid app gets a /<app>-restore skill mirroring the runbook as a checklist for future Claude sessions.
  7. Break-glass admin is mandatory. Apps that delegate auth to Authentik (LDAP or OIDC) must also ship a sealed local-DB admin credential. Day-to-day logins use SSO; the break-glass account exists so an Authentik outage doesn't lock out the operator.

Alternatives considered

  • Full GitOps (round-trip everything through git). Rejected for apps like Jellyfin where the dominant on-disk state is binary metadata and SQLite — not human-editable, not diff-friendly, and large enough to bloat the repo. The CI / pre-commit gates would also block on routine library scans.
  • PVC-only (treat the app as opaque, back up the PVC and call it done). Rejected because cluster rebuild requires the LDAP bind password to be present in LDAP-Auth.xml at pod start; if the only source of truth is the PVC, a rebuild lands a Jellyfin that can't authenticate any user. The break-glass admin solves the same problem from the other side.
  • External secrets operator + opaque PVC. Rejected for the same reason as PVC-only — ESO solves credential delivery but not the bootstrap wizard or library-add steps. Hybrid GitOps subsumes this case: ESO would slot in cleanly as an alternative to SealedSecrets for the sealed-credential half, but the PVC + runbook + skill obligations are unchanged.
  • Velero PVC backups instead of Longhorn RecurringJob. Defer. Velero is on the deep-dive queue; when it ships, Hybrid apps can opt-in to Velero schedules and the runbook's Flow A gains a second source. The pattern itself does not depend on Longhorn.

Consequences

Positive:

  • Cluster rebuild for a Hybrid app is now a documented Flow B path with bounded time-to-recovery (60–120 min for Jellyfin).
  • Lost-PVC recovery is a documented Flow A path with sub-30 min recovery, assuming a recent Longhorn backup exists.
  • The break-glass admin pattern stops "Authentik down → I can't log in to fix Jellyfin" cascades.
  • Future apps with the same shape (Sonarr, Radarr, Jellyseerr, Romm, ...) inherit the template; future deep-dives should audit for missing restore runbooks against this ADR.

Negative / costs:

  • Two artifacts per Hybrid app (runbook + skill) — modest write cost per app, payoff is operational.
  • The init container's plugin-manifest stays out-of-band from the Renovate dependency-pinning loop. Currently mitigated by pinning plugin versions in values.yaml; not in the digest-pinning sweep. Track separately if the Last.fm fork or any other sideloaded plugin gets a security advisory.
  • A successful Flow B leaves Trakt + Last.fm OAuth tokens unset. Users must re-link via the plugin's Settings page. This is an accepted trade-off; OAuth refresh tokens are intentionally not backed up.

References