Skip to content

paperless-ngx 3 rebuild — reset & cutover (#2106)

Runtime steps for the operator, around the merge of the rebuild PR (branch feat/paperless3-rebuild). Design + decisions: .workingdir2/research/31-paperless3-rebuild/{02-design.md,99-decisions-for-operator.md}. The app is verified ~empty; the reset is pre-approved (D4/D5).

What the PR changes: all four PVCs become explicit manifests — data → paperless-data-office (longhorn-office block), media/consume/export → static NFS PVs at fixed NAS paths pre-bound via volumeName; v3 env surface (trusted proxies, archive generation, stability delay, duplicate rejection, filename format); nightly document_exporter CronJob to the /volume4 backup share; office PVC quota 12 → 16 (swap overlap).

Prerequisites

  • Workstation in the office zone (or cluster API via cloudflared).
  • SSH access to the office Synology (192.168.2.19).
  • kubectl read + exec against the cluster; ArgoCD UI/CLI access.

1. NAS prep (BEFORE merge — pods cannot mount missing NFS subdirs)

ssh <admin>@192.168.2.19
mkdir -p /volume7/k8s-apps/paperless/media /volume7/k8s-apps/paperless/consume
mkdir -p /volume4/k8s-backups/paperless/export
chown -R 1000:1000 /volume7/k8s-apps/paperless /volume4/k8s-backups/paperless

Operator TODO (NAS-side, D5, #2106): create the SMB shared folder for the scanner on /volume7/k8s-apps/paperless/consume in DSM (Control Panel → Shared Folder → point at that path; give the scanner account write access). Needed before the scanner workflow works; not needed for the cutover itself. Subdirs the scanner creates become paperless tags.

2. Safety dump (BEFORE the DB reset — even though the app is ~empty)

PRIMARY=$(kubectl get cluster -n databases pg-cluster -o jsonpath='{.status.currentPrimary}')
kubectl exec -n databases "$PRIMARY" -c postgres -- \
  pg_dump -U postgres --clean --if-exists -d paperless \
  > paperless-pre-rebuild-$(date +%F).sql

3. DB reset to a clean schema (v3 migrates into it on first boot)

Keeps the paperless database itself (the CNPG Database CR keeps reconciling its existence); only the contents reset. The running old pod will error from here until the roll — expected, this is the window.

kubectl exec -n databases "$PRIMARY" -c postgres -- psql -U postgres -d paperless -c \
  'DROP SCHEMA public CASCADE; CREATE SCHEMA public AUTHORIZATION paperless; GRANT ALL ON SCHEMA public TO paperless;'

4. Merge the PR

Auto-sync does the rest in one sync: static PVs + new PVCs created and pre-bound, deployment rolls onto the new volumes, exporter CronJob created, old chart-managed PVCs pruned (deletion completes once the old pod terminates and releases them). No manual scale-down needed — the volume swap forces the roll, and app-of-apps selfHeal would revert a manual sync-policy pause anyway.

5. Readback checks (in order)

# PVCs: 4 new Bound, old 4 gone
kubectl get pvc -n office | grep paperless
# static PVs bound to the right claims
kubectl get pv paperless-media-office paperless-consume-office paperless-export-office
# pod up, migrations clean, consumer POLLING (not inotify), no deprecation warnings
kubectl logs -n office deploy/paperless-ngx-paperless | grep -iE 'polling|migrat|warn|error' | head -50

Then functionally:

  1. Login via https://docs.office.cauda.dev (Authentik) works.
  2. Upload a test PDF in the UI → consumed, OCR runs, AI suggestions arrive in German (B60 cold start can take minutes on first call).
  3. Scanner end-to-end: drop a PDF on the consume SMB share → picked up within ~75 s (60 s poll + 15 s stability), subdir becomes a tag.
  4. Exporter first run — trigger manually instead of waiting for 01:30:
kubectl create job -n office --from=cronjob/paperless-ngx-exporter paperless-export-manual
kubectl logs -n office job/paperless-export-manual -f
ssh <admin>@192.168.2.19 ls -la /volume4/k8s-backups/paperless/export   # manifest.json + version, non-empty

6. Cleanup

# old dynamic PVs are Released (SC reclaim=Retain) — remove the objects;
# file data stays in the old pvc-UUID subdirs on the NAS
kubectl get pv | grep Released | grep office
kubectl delete pv <the four released pvc-… volumes>

After a verified week: remove the old /volume7/k8s-apps/office-paperless-ngx-*-pvc-* subdirs on the NAS. Append the cutover result to STATE.md.

What can go wrong

  • New pod Pending / PVC mount failure — NAS subdir missing (step 1) or a static PV/PVC pre-bind mismatch (volumeName ↔ claimRef names).
  • PVCs denied by quota — mitigated by the 12→16 bump in the PR; check kubectl describe resourcequota -n office.
  • Old PVCs stuck Terminating — old pod still holds them; check the rollout. Last resort is the finalizer-patch pattern from the /volume7 migration — only after confirming nothing mounts them.
  • Migrations fail on first boot — schema not clean; re-run step 3, then delete the pod.
  • Exporter PermissionError on export dir — ownership must be 1000:1000 (step 1).
  • Later: v3.0.0 stable re-pin — bump the digest in apps/office/paperless-ngx/values.yaml (single anchor covers deployment AND CronJob); the next nightly export re-runs against the new version. Exports are version-locked — never restore an old export into a newer paperless.