Skip to content

Jellyfin Restore — Hybrid GitOps (PRs #436, #441)

Recover the home-zone Jellyfin server when the config PVC is lost, corrupted, or the cluster is rebuilt. Runtime state (DB, users, library scan, plugin DLLs, OAuth tokens) lives on the config PVC and is not in git; only the Helm chart, ConfigMap-templated LDAP-Auth.xml, and two sealed secrets are tracked. Expected recovery window:

  • Flow A (PVC restore from Longhorn backup): 10–20 min.
  • Flow B (full bootstrap from scratch): 60–120 min (depends on library scan).

Related: PR #436 (shared authentik LDAP outpost + jellyfin-ldap provider), PR #441 (LDAP-Auth ConfigMap + bootstrap-admin sealed secret). Helm values: apps/media/jellyfin/values.yaml. HTTPRoute: cluster/gateway/routes/media.yaml (jellyfin.media.cauda.dev, main-home gateway).

When to use this runbook

  • The Longhorn volume backing media/jellyfin-config-jellyfin-0 PVC is lost, corrupted, or replicas are unrecoverable.
  • Full cluster rebuild (post-etcd-restore — see etcd-restore.md — or post-disaster).
  • jellyfin.db corruption surfaces as repeated 500s on /health or Microsoft.Data.Sqlite.SqliteException in the pod log.
  • A bad plugin DLL upgrade wedges the server (Jellyfin won't boot if a plugin throws on load).
  • LDAP-Auth.xml landed without the substituted bind password (init container failed silently) — covered as a subset of Flow B step "LDAP rewiring" below.

Do not use this runbook for:

  • Single missed library scan — use the Jellyfin UI's Scan All Libraries button.
  • Authentik unreachable but Jellyfin pod is healthy — that's an Authentik runbook (/verify-oncall is irrelevant; check the ak-outpost-authentik-ldap-outpost pod in the authentik namespace).

Preflight

Answer these before touching anything.

  1. Is a restore actually required?
  2. Pod CrashLoopBackOff? Check kubectl -n media logs jellyfin-0 -c app --previous first. Often a missing plugin DLL or bad XML — fix in place, no restore needed.
  3. Library count looks short? That's a metadata sync, not state loss. Run Scan Library from the UI.
  4. Scope. What's missing — the PVC, the namespace, or the whole cluster?
  5. Backup availability. Longhorn home-daily-backup retains 3 copies, home-weekly-backup retains 4 (see cluster/longhorn/recurring-jobs-home.yaml). List candidates before wiping anything:
kubectl -n longhorn-system get backup \
  -l backup-volume=$(kubectl -n media get pvc jellyfin-config-jellyfin-0 \
     -o jsonpath='{.spec.volumeName}') \
  --sort-by=.metadata.creationTimestamp \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state,CREATED:.metadata.creationTimestamp,SIZE:.status.size
# Expect: at least one Completed entry in the last 48 h.
  1. Coordinate.
  2. Silence the KubePodCrashLooping + LonghornVolumeStatusWarning alerts for ~2 h (/alertmanager-silence-add jellyfin).

  3. Freeze ArgoCD self-heal on the jellyfin Application so a half-restored PVC doesn't trigger a re-create mid-restore:

    argocd app set jellyfin --sync-policy none

  4. Notify users of jellyfin.media.cauda.dev that the server is in maintenance.

Flow A: PVC restored from a Longhorn backup

Fastest path. Requires at least one Completed Longhorn Backup object for the jellyfin-config-jellyfin-0 PVC's volume.

A1. Pick the backup

BACKUP=$(kubectl -n longhorn-system get backup \
  -l backup-volume=$(kubectl -n media get pvc jellyfin-config-jellyfin-0 \
     -o jsonpath='{.spec.volumeName}') \
  --sort-by=.metadata.creationTimestamp \
  -o jsonpath='{.items[-1].metadata.name}')
echo "Latest backup: $BACKUP"

Verify its status.state == Completed and the creation timestamp predates whatever broke the volume.

A2. Scale Jellyfin down

kubectl -n media scale statefulset jellyfin --replicas=0
kubectl -n media wait --for=delete pod -l app.kubernetes.io/name=jellyfin --timeout=120s

If the StatefulSet's pod is wedged, force-delete after the wait times out:

kubectl -n media delete pod jellyfin-0 --grace-period=0 --force

A3. Delete the broken PVC + (if it's still around) PV

kubectl -n media delete pvc jellyfin-config-jellyfin-0
# If the PV doesn't auto-release within 30 s:
kubectl get pv | grep jellyfin-config-jellyfin-0
kubectl delete pv <pv-name>

A4. Create a Longhorn Volume from the backup, pre-bind a PVC

The cleanest path is to use the Longhorn UI (https://longhorn.home.cauda.dev/#/backup), select the backup, choose Restore Latest Backup, name the new volume jellyfin-config-restore-<YYYYMMDD>, set replicas=2, then create a PV and PVC pre-bound via volumeName.

CLI-only path (see also: memory entry "PVC migration: pre-bind via volumeName"):

# 1. Trigger the restore (creates a Volume object).
kubectl -n longhorn-system apply -f - <<EOF
apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: jellyfin-config-restore-$(date +%Y%m%d)
  namespace: longhorn-system
spec:
  fromBackup: "$(kubectl -n longhorn-system get backup "$BACKUP" -o jsonpath='{.status.url}')"
  numberOfReplicas: 2
  size: "42949672960"  # 40Gi in bytes — match the original PVC size
  dataLocality: best-effort
  accessMode: rwo
EOF

# 2. Wait for the volume to be attached + restore complete.
kubectl -n longhorn-system get volume jellyfin-config-restore-$(date +%Y%m%d) \
  -o jsonpath='{.status.state} {.status.restoreRequired}{"\n"}'
# Expect: detached false  (state=detached, restoreRequired=false).

# 3. Create the PV + PVC pre-bound by volumeName (NOT claimRef alone).
VOL=jellyfin-config-restore-$(date +%Y%m%d)
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: $VOL
spec:
  capacity:
    storage: 40Gi
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: longhorn-home
  csi:
    driver: driver.longhorn.io
    fsType: ext4
    volumeHandle: $VOL
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jellyfin-config-jellyfin-0
  namespace: media
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: longhorn-home
  volumeName: $VOL
  resources:
    requests:
      storage: 40Gi
EOF

A5. Scale Jellyfin back up

kubectl -n media scale statefulset jellyfin --replicas=1
kubectl -n media wait --for=condition=ready pod/jellyfin-0 --timeout=300s

Jump straight to the Smoke tests section at the bottom. Flow A is done; no plugin re-download, no library re-add, no admin re-create needed — the PVC carries all of that.

Flow B: Full bootstrap from scratch

Use this when no Longhorn backup is recoverable (worst case) or the entire cluster was rebuilt. Expect 60–120 min depending on library size; the dominant cost is the initial library scan over ~40 TB of NFS.

B1. Confirm the sealed secrets and ConfigMap are present in git

After cluster rebuild, ArgoCD will reconcile these from apps/media/jellyfin/ and sealed-secrets/sealed/. Verify:

kubectl -n media get secret jellyfin-bootstrap-admin jellyfin-ldap-creds
# Expect: both present, type Opaque.

kubectl -n media get configmap ldap-auth-template -o jsonpath='{.data.LDAP-Auth\.xml}' | head -5
# Expect: <PluginConfiguration ...> header, @@LDAP_BIND_DN@@ placeholder visible.

If either secret is missing, the SealedSecret controller hasn't unsealed yet — kubectl -n sealed-secrets logs deploy/sealed-secrets will show why (usually: the cluster's sealing key changed and the sealed YAML was generated against a different one — recover the cluster key from the offline backup before continuing).

B2. Let Jellyfin come up on an empty PVC

ArgoCD will create the PVC fresh from the chart's volumeClaimTemplate. The init containers run on first boot:

  • render-ldap-auth writes /config/config/plugins/configurations/LDAP-Auth.xml with the bind DN + password substituted from jellyfin-ldap-creds.
  • install-plugins downloads the 15 officially-listed plugins from repo.jellyfin.org and the Last.fm plugin from lusoris/jellyfin-plugin-lastfm.

Watch the init containers complete:

kubectl -n media logs jellyfin-0 -c render-ldap-auth
kubectl -n media logs jellyfin-0 -c install-plugins
# Expect: no "WARNING: Failed to download" lines.

If install-plugins fails for any plugin, see the "Plugin DLL re-download URLs" section below — you can sideload manually.

B3. Drive the setup wizard via API

Jellyfin's first-run wizard runs interactively in the browser by default but accepts a fully-scripted bootstrap. Get the caudaadmin password from the sealed secret:

ADMIN_USER=$(kubectl -n media get secret jellyfin-bootstrap-admin -o jsonpath='{.data.USERNAME}' | base64 -d)
ADMIN_PASS=$(kubectl -n media get secret jellyfin-bootstrap-admin -o jsonpath='{.data.PASSWORD}' | base64 -d)
echo "Admin: $ADMIN_USER"

Port-forward and complete the wizard:

kubectl -n media port-forward svc/jellyfin 8096:8096 &
PF=$!

# Step 1 — language
curl -sS -X POST http://127.0.0.1:8096/Startup/Configuration \
  -H 'Content-Type: application/json' \
  -d '{"UICulture":"en-US","MetadataCountryCode":"DE","PreferredMetadataLanguage":"en"}'

# Step 2 — initial admin user
curl -sS -X POST http://127.0.0.1:8096/Startup/User \
  -H 'Content-Type: application/json' \
  -d "{\"Name\":\"$ADMIN_USER\",\"Password\":\"$ADMIN_PASS\"}"

# Step 3 — remote-access (we proxy via Gateway API, no UPnP needed)
curl -sS -X POST http://127.0.0.1:8096/Startup/RemoteAccess \
  -H 'Content-Type: application/json' \
  -d '{"EnableRemoteAccess":true,"EnableAutomaticPortMapping":false}'

# Step 4 — finish
curl -sS -X POST http://127.0.0.1:8096/Startup/Complete

kill $PF

B4. Authenticate, capture an API key

kubectl -n media port-forward svc/jellyfin 8096:8096 &
PF=$!

AUTH=$(curl -sS -X POST http://127.0.0.1:8096/Users/AuthenticateByName \
  -H 'Content-Type: application/json' \
  -H 'Authorization: MediaBrowser Client="restore", Device="cli", DeviceId="restore-cli", Version="1.0.0"' \
  -d "{\"Username\":\"$ADMIN_USER\",\"Pw\":\"$ADMIN_PASS\"}")

TOKEN=$(echo "$AUTH" | jq -r .AccessToken)
USERID=$(echo "$AUTH" | jq -r .User.Id)
# Never print the token — it grants full API access and would persist in
# shell history, scrollback and any terminal recording of the restore.
[ -n "$TOKEN" ] && echo "auth ok (token acquired)" || { echo "auth FAILED"; exit 1; }
echo "UserId: $USERID"

B5. Re-add the four libraries

The libraries map 1:1 to the chart's NFS mount paths. Re-add each via the Media Library API (/Library/VirtualFolders).

Library CollectionType Path
Movies movies /data/media/movies/standard and /data/media/movies/4k
Shows tvshows /data/media/tvshows/standard and /data/media/tvshows/4k
Anime tvshows /data/media/anime
Music music /data/media/music

Add each library (Movies shown; repeat for the others):

curl -sS -X POST "http://127.0.0.1:8096/Library/VirtualFolders?name=Movies&collectionType=movies&refreshLibrary=false" \
  -H "Authorization: MediaBrowser Token=\"$TOKEN\"" \
  -H 'Content-Type: application/json' \
  -d '{
    "LibraryOptions": {
      "EnablePhotos": false,
      "EnableRealtimeMonitor": true,
      "EnableChapterImageExtraction": false,
      "ExtractChapterImagesDuringLibraryScan": false,
      "PathInfos": [
        {"Path":"/data/media/movies/standard"},
        {"Path":"/data/media/movies/4k"}
      ],
      "MetadataCountryCode": "DE",
      "PreferredMetadataLanguage": "en"
    }
  }'

Repeat for Shows, Anime, Music with the paths from the table. For Anime, set the same collectionType=tvshows as Shows; Jellyfin distinguishes them only by path.

Trigger a library scan once all four are in:

curl -sS -X POST "http://127.0.0.1:8096/Library/Refresh" \
  -H "Authorization: MediaBrowser Token=\"$TOKEN\""
kill $PF

The scan takes 30–90 min depending on metadata-provider rate limits.

B6. Plugin DLL re-download URLs

If install-plugins failed for a specific plugin, sideload it directly. URLs below are pinned to versions known-good as of 2026-05-27.

# Exec into the pod (init container has finished, only the app remains).
kubectl -n media exec -it jellyfin-0 -c app -- sh -c '
  mkdir -p /config/plugins
  cd /tmp

  # First-party plugins — Jellyfin manifest repo. Filename pattern:
  # https://repo.jellyfin.org/files/plugin/<slug>/<slug>_<version>.zip
  for p in \
    "chapter-segments-provider|chapter-segments-provider_4.0.0.0|Chapter Segments Provider" \
    "discogs|discogs_2.0.0.0|Discogs" \
    "fanart|fanart_14.0.0.0|Fanart" \
    "kodi-sync-queue|kodi-sync-queue_15.0.0.0|Kodi Sync Queue" \
    "ldap-authentication|ldap-authentication_23.0.0.0|LDAP Authentication" \
    "local-intros|local-intros_4.0.0.0|Local Intros" \
    "lrclib-lyrics|lrclib-lyrics_3.0.0.0|LrcLib Lyrics" \
    "subtitle-extract|subtitle-extract_7.0.0.0|Subtitle Extract" \
    "session-cleaner|session-cleaner_5.0.0.0|Session Cleaner" \
    "thetvdb|thetvdb_20.0.0.0|TheTVDB" \
    "tmdb-box-sets|tmdb-box-sets_12.0.0.0|TMDb Box Sets" \
    "trakt|trakt_30.0.0.0|Trakt" \
    "transcode-killer|transcode-killer_4.0.0.0|Transcode Killer" \
    "tvmaze|tvmaze_13.0.0.0|TVmaze" \
    "webhook|webhook_21.0.0.0|Webhook" \
  ; do
    SLUG=$(echo "$p" | cut -d"|" -f1)
    FILE=$(echo "$p" | cut -d"|" -f2)
    NAME=$(echo "$p" | cut -d"|" -f3)
    wget -q -O /tmp/p.zip "https://repo.jellyfin.org/files/plugin/$SLUG/${FILE}.zip"
    mkdir -p "/config/plugins/$NAME"
    unzip -o /tmp/p.zip -d "/config/plugins/$NAME"
  done

  # Last.fm (fork): https://github.com/lusoris/jellyfin-plugin-lastfm
  wget -q -O /tmp/lastfm.zip \
    https://github.com/lusoris/jellyfin-plugin-lastfm/releases/download/10.11.10.0/lastfm_10.11.10.0.zip
  mkdir -p "/config/plugins/Last.fm"
  unzip -o /tmp/lastfm.zip -d /config/plugins/Last.fm
'

# Restart Jellyfin so it picks up the new DLLs.
kubectl -n media delete pod jellyfin-0

The Last.fm fork's release SHA-256 is da05da97433eed2d18f447c0a4107bd2173b36073d202693c0c47f7fac3edda8 (verify after wget if integrity is in question).

B7. LDAP rewiring

Verify the shared LDAP outpost is up before Jellyfin tries to bind:

kubectl -n authentik get deploy ak-outpost-authentik-ldap-outpost \
  -o jsonpath='{.status.replicas}/{.status.readyReplicas} ready{"\n"}'
# Expect: 1/1 ready (or the configured replica count).

kubectl -n authentik exec deploy/ak-outpost-authentik-ldap-outpost -- \
  /bin/sh -c 'wget -qO- localhost:9300/ping'
# Expect: pong.

Test the bind from the Jellyfin pod's perspective:

kubectl -n media exec jellyfin-0 -c app -- /bin/sh -c '
  apk add --no-cache openldap-clients > /dev/null 2>&1 || true
  ldapsearch -x -H ldap://ak-outpost-authentik-ldap-outpost.authentik.svc.cluster.local:3389 \
    -D "cn=ldapservice,ou=users,dc=ldap,dc=cauda,dc=dev" \
    -w "$JELLYFIN_LDAP_BIND_PASSWORD" \
    -b "dc=ldap,dc=cauda,dc=dev" \
    "(objectClass=user)" cn uid mail
'
# Expect: at least one entry for the bootstrap test user, exit code 0.

If the bind fails:

  • Invalid credentials (49) — jellyfin-ldap-creds password drifted from authentik's authentik-secrets[LDAP_BIND_PASSWORD]. Both must match. See the seal-first-rollout-second memory note before resealing.
  • Can't contact LDAP server (-1) — outpost pod unreachable; check authentik logs.

In the Jellyfin UI: Dashboard → Plugins → LDAP-Auth → Settings. Click Save once to force the plugin to re-load LDAP-Auth.xml. Then click Test — should show one or more matching users.

Smoke tests

Run all of these after Flow A or Flow B. Each step has a pass/fail signal — don't paper over a fail.

  1. Pod healthy:
kubectl -n media get pod jellyfin-0 \
  -o jsonpath='{.status.containerStatuses[0].ready} {.status.containerStatuses[0].restartCount}{"\n"}'
# Expect: "true 0" (or low restart count if Flow B).
  1. HTTPS path serves:
curl -sS -o /dev/null -w '%{http_code}\n' https://jellyfin.media.cauda.dev/health
# Expect: 200.

  1. Admin login (break-glass): in a private browser window, open https://jellyfin.media.cauda.dev/web/index.html#!/login.html, sign in as caudaadmin with the password from the sealed secret. Should land on the dashboard.

  2. LDAP user login: sign in as any Authentik user that's a member of the LDAP Bind Users group. Should land on the home screen, library tiles populated.

  3. Library counts (compare to a known good snapshot from STATE.md, if one was captured):

kubectl -n media exec jellyfin-0 -c app -- sqlite3 /config/data/jellyfin.db \
  "SELECT Name, (SELECT COUNT(*) FROM TypedBaseItems WHERE TopParentId = mi.guid) AS items FROM MediaProviders mi;"
# Or via UI: Dashboard → Libraries → row counts.
  1. Transcode works: play any 4K title in a browser that doesn't support HEVC direct (e.g. Firefox). Jellyfin should fall back to GPU transcode. Verify the Intel iGPU is in use:
kubectl -n media exec jellyfin-0 -c app -- intel_gpu_top -J 2>/dev/null | head -20
# Expect: non-zero engines.video activity during playback.

  1. Plugin list: Dashboard → Plugins shows all 16 plugins (15 first-party + Last.fm) at Active status. No Disabled by admin or Failed to load states.

  2. OAuth-token-using plugins (Trakt, Last.fm): these intentionally ship with placeholder credentials. After restore they will need user-driven OAuth re-link via the plugin's Settings page. Note in STATE.md that scrobble history before the restore date may not re-sync.

Post-restore housekeeping

  1. Re-enable ArgoCD self-heal:
argocd app set jellyfin --sync-policy automated --auto-prune --self-heal

  1. Lift the Alertmanager silence.

  2. Append a restore record to STATE.md: date · flow used · backup timestamp restored from · time-to-healthy · anything unexpected.

  3. If Flow B: schedule a Longhorn backup immediately so the next incident has a recovery point:

kubectl -n longhorn-system annotate volume \
  $(kubectl -n media get pvc jellyfin-config-jellyfin-0 -o jsonpath='{.spec.volumeName}') \
  recurring-job.longhorn.io/source=enabled \
  recurring-job-group.longhorn.io/home=enabled --overwrite
# Or trigger an on-demand backup from the Longhorn UI.

Tracking

  • Owner: @lusoris.
  • First drill: run Flow A end-to-end against a disposable media-drill namespace with a clone of the production PVC. Target: next Longhorn or Jellyfin chart bump.
  • Flow B drill: harder to run safely — needs a separate cluster or a scratch namespace + temporary HTTPRoute. Defer until a Velero restore drill is in place.