Skip to content

Nextcloud production rebuild — reset install (issue #2108)

Fresh reset install of the office Nextcloud (D3) onto the static NFS data PV (D1), with the D4 app suite, B60 assistant wiring (D5) and aligned crash-consistent backups (D6). Design record: .workingdir2/research/32-nextcloud-rebuild/ (01-research, 02-design, 99-decisions). All Git-side config ships in PR branch feat/nextcloud-rebuild; this runbook is the ordered runtime part.

Prerequisites

  • PR from feat/nextcloud-rebuild reviewed; do NOT merge before phase 3 is complete.
  • Workstation with argocd + kubectl (read/write for the documented unfreeze/reset commands below — these are sanctioned runtime ops, all declarative state stays in Git).
  • SSH to the office NAS (192.168.2.19, DSM — see synology-nas-setup.md).
  • Verify the instance is still effectively empty before any destructive step: du -sh of the live data dir ≈ 192M / only akadmin (verify-live-before-destructive; if real user data has landed, STOP and re-plan per design Option B).

Steps

Phase 1 — freeze + safety net (before merge)

  1. Disable auto-sync on the app:
argocd app set nextcloud --sync-policy none
  1. Maintenance mode on:
kubectl exec -n office deploy/nextcloud -c nextcloud -- \
  runuser -u www-data -- env HOME=/var/www php occ maintenance:mode --on
  1. DB safety dump (rollback anchor; CNPG PITR also covers it):
kubectl exec -n databases pg-cluster-1 -c postgres -- \
  pg_dump -U postgres -Fc nextcloud > nextcloud-pre-rebuild-$(date +%F).dump
  1. NAS snapshot of the current data share (DSM → Snapshot Replication → k8s-apps → Take Snapshot), label it pre-nextcloud-rebuild.

Phase 2 — storage prep (NAS side)

  1. Create the new static data dir; the chart mounts subPath data, and kubelet would auto-create it root-owned, so pre-create with UID 33:
ssh <admin>@192.168.2.19
sudo mkdir -p /volume7/k8s-apps/nextcloud-data/data
sudo chown -R 33:33 /volume7/k8s-apps/nextcloud-data
sudo chmod 770 /volume7/k8s-apps/nextcloud-data/data

Phase 3 — reset (destructive; dump + snapshot from phase 1 are the rollback)

  1. Drop + recreate the DB. The Database CR pg-cluster-nextcloud (cluster/cloudnative-pg/databases.yaml) has reclaim retain, so deleting the CR would NOT drop the DB — do it via psql-as-postgres, matching the CR spec (name nextcloud, owner nextcloud) so the CR stays truthful:
kubectl exec -n databases pg-cluster-1 -c postgres -- \
  psql -U postgres -c "DROP DATABASE nextcloud;"
kubectl exec -n databases pg-cluster-1 -c postgres -- \
  psql -U postgres -c "CREATE DATABASE nextcloud OWNER nextcloud;"
  1. Delete the workload + old config volume for a clean /var/www/html (auto-sync is off; the sync in phase 4 recreates them):
kubectl delete deployment -n office nextcloud
kubectl delete pvc -n office nextcloud-nextcloud
kubectl delete pv nextcloud-restored
  1. Optional but recommended: reseal a fresh nextcloud-admin-creds (/seal-secret, ns office) — the fresh autoinstall consumes it as the initial admin. Seal BEFORE the phase-4 sync (seal first, rollout second).

Phase 4 — merge + fresh install

  1. Merge the PR, then sync with prune (removes the old nextcloud-nextcloud-data PVC; its nfs-apps PV is Retain, the old pvc-UUID dir stays on the NAS until phase-6 cleanup):
argocd app sync nextcloud --prune

  1. Watch first boot (autoinstall + hook; startup probe allows ~5 min):

    kubectl get pvc -n office        # nextcloud-data-nfs must be Bound
kubectl logs -n office deploy/nextcloud -c nextcloud -f

    The before-starting hook is self-asserting: any READBACK MISMATCH line fails the boot instead of silently no-opping.

Phase 5 — verification (silent-no-op rule: GET everything back)

  1. Instance + storage:

    kubectl exec -n office deploy/nextcloud -c nextcloud -- \
  runuser -u www-data -- env HOME=/var/www php occ status
ssh <admin>@192.168.2.19 ls /volume7/k8s-apps/nextcloud-data/data
# expect exactly ONE appdata_<instanceid> dir, .ncdata marker present

  2. Config readback (hook asserts these too — confirm once by hand):

    occ config:system:get dbhost                  # pg-cluster-rw.databases.svc.cluster.local
occ config:system:get maintenance_window_start  # 1
occ config:app:get core backgroundjobs_mode   # cron
occ app:list | grep -E 'calendar|contacts|notes|deck|previewgenerator|assistant|integration_openai|user_oidc'
occ config:app:get integration_openai url     # http://qwen3-8b-b580-router.ai.svc.cluster.local/v3 (B580; standalone B60 8b retired)
occ config:app:get integration_openai request_timeout  # 240
occ config:app:get previewgenerator squareSizes          # 64 256
occ config:app:get previewgenerator fillWidthHeightSizes # 256 4096
occ config:app:get preview jpeg_quality       # 60

  3. OIDC: log in at https://cloud.office.cauda.dev as akadmin via Authentik → user is provisioned, has admin rights (occ user:info akadmin shows group admin).

  4. Functional round-trip: upload + download a file (lands under /volume7/k8s-apps/nextcloud-data/data/akadmin/files/ on the NAS), edit a document in Collabora, open the Photos/Files grid (previews render).

  5. Assistant end-to-end (B60 scale-from-zero may take 1-2 min on the first request): Assistant → chat prompt → answer arrives; then

    occ taskprocessing:task:list   # task shows successful

  6. occ files:scan --all (fast no-op on a fresh instance; proves FS perms), exporter metrics green in Grafana (nextcloud-exporter target up).

Phase 6 — unfreeze + cleanup

  1. Re-enable auto-sync:

    argocd app set nextcloud --sync-policy automated --auto-prune --self-heal

  2. After a soak (suggest: next NAS snapshot cycle), delete the released old dynamic-data PV and remove the old /volume7/k8s-apps/office-nextcloud-nextcloud-data-pvc-* dir on the NAS (content is preserved in the pre-nextcloud-rebuild snapshot).

  3. Append the outcome to STATE.md; close #2108.

Operator DSM TODOs (D6/D2 — not k8s work)

  • Btrfs snapshot schedule on the k8s-apps share (covers nextcloud-data): hourly ×24 + daily ×14 + weekly ×8, anchored ~02:00 so snapshots, CNPG daily backup (02:00) and maintenance_window_start=1 (01:00-05:00 UTC job window) align — crash-consistent restore = CNPG PITR to snapshot time + occ files:scan --all + occ maintenance:repair.
  • Off-NAS copy (Hyper Backup) + any NAS-layer at-rest encryption: deferred to #2105 (D2 = no Nextcloud SSE; NAS-layer if desired).

Deferred (recorded, intentionally NOT in this change)

  • Redis: bundled, standalone, no persistence — left exactly as-is (operator declined changes here). Design recommendation kept on record: delete the orphaned redis-data-nextcloud-redis-master-0 PVC (8Gi longhorn-office-no-backup, #1976 orphan list).
  • notify_push (helm chart PR #581 still open), full-text search (no NC 34 fts release), AppAPI/ExApps (Docker-socket only), imaginary — all design-out per 02-design.md.
  • Collabora CODE 26.04 bump — separate /bump-helm-chart.
  • NC 35 upgrade — after 2026-09-16, gated on chart + user_oidc 8.11.
  • Optional weekly consistent-backup CronJob (D6 option B) — revisit once real user data lands.

What can go wrong

  • nextcloud-data-nfs PVC stuck Pending — PV claimRef already populated with a stale UID from a previous bind attempt: kubectl patch pv nextcloud-data-nfs --type json -p '[{"op":"remove","path":"/spec/claimRef/uid"}]' (documented unwedge; PV/PVC pair itself stays Git-managed).
  • "Cannot write into data directory" — the data subPath dir was auto-created root-owned (phase 2 skipped): re-do phase 2, restart pod.
  • occ app:install fails — app store unreachable from the pod; retry (transient) or check egress. The hook fails the boot loudly.
  • config:app:set warns about LAZY/SENSITIVE state — a value was previously written with different flags; the hook's flags mirror integration_openai's own setters, accept the migration prompt-free behaviour (--no-interaction keeps current state) and re-run.
  • First boot slow on NFS — known (nextcloud/helm#10); startup probe budget is 5 min, don't panic before it trips.
  • Rollback: revert the PR merge, argocd app sync nextcloud, restore DB pg_restore -U postgres -d nextcloud <dump> (after DROP/CREATE DATABASE), restore the data dir from the pre-nextcloud-rebuild NAS snapshot into the pvc-UUID dir the reverted dynamic PVC binds to, then occ maintenance:mode --off + occ files:scan --all.