Skip to content

0040. Migrating home PVCs to the Longhorn v2 storage class

  • Status: accepted
  • Date: 2026-07-14
  • Deciders: lusoris
  • Related: PR #2487, BUG-40, BUG-41

Context

Home-zone persistent volumes ran on the Longhorn v1 data engine, whose replicas lived on a filesystem-type disk at /var/lib/longhorn on each home worker's 800 GiB OS root. The home PVE host's backing storage is NVMe, so the v1 engine's iSCSI/tgtd datapath sat between the workload and near-native NVMe throughput. The operator raised this as a backlog item on 2026-07-13 (.workingdir2/archive/HANDOFF-2026-07-13-cluster-modernization.md): use the Longhorn v2 (SPDK) data engine on the home node for an AI-I/O speedup on model loads and caches, with SPDK, hugepages, a raw block device, per-node engine opt-in and kernel support named as prerequisites. Longhorn 1.12.0 (2026-06-02) had made the V2 Data Engine GA, which is what made this actionable.

The enabling work landed first. #2462 flipped v2DataEngine in the chart's defaultSettings, which did not take effect because Longhorn defaultSettings only apply to unset settings and the v2-data-engine Setting already existed as false; #2463 added the explicit ArgoCD-managed Setting resource (cluster/longhorn/v2-data-engine-setting.yaml) that forces the value. Both live home workers were then prepared with 2 GiB hugepages, the nvme-tcp/uio_pci_generic/vfio_pci modules, and a dedicated 1 TiB block-type Longhorn disk (VM 9003 and VM 9004, scsi1).

The remaining gap was that no workload actually requested the v2 class. This change is that request: the desired-state repoint of the home application fleet.

A measured v1-versus-v2 throughput comparison was listed as an intended output (longhorn-v2-benchmark) in the 2026-07-14 handoff but no such artifact exists in the repository. The expected performance gain is therefore recorded here as the stated motive, not as a measured result.

Decision

Repoint the desired-state storageClass / storageClassName of 25 home-zone applications from longhorn-home to longhorn-home-v2 in a single batch. Databases were excluded from this batch; pg-media was cut over separately with its own rolling procedure and resizeInUseVolumes: false. The office zone stays on v1.

longhorn-home-v2 (cluster/longhorn/v2-storageclass.yaml) is an explicit, opt-in class — not the cluster default. It sets dataEngine: "v2", numberOfReplicas: "1", reclaimPolicy: Retain, volumeBindingMode: Immediate, dataLocality: best-effort, nodeSelector: zone-home and backupTargetName: home-nas. The v1-only tunables disableRevisionCounter and unmapMarkSnapChainRemoved are deliberately omitted because the v2 engine manages revision and unmap internally.

This PR changed desired state only. Because storageClassName is immutable on a bound PVC, each volume's actual move to v2 was a separate, workload-specific runtime procedure — fresh provision, restore, rolling replacement, or a cold-copy with a verified quiesced source. All of those completed between 2026-07-14 and 2026-07-15; live Longhorn inventory afterwards showed 44 v2 volumes, 18 office/out-of-scope v1 volumes, and zero home v1 volumes.

Alternatives considered

  • In-place conversion of existing v1 volumes to v2. Not available: Longhorn never converts a volume's data engine in place, which is why every cutover had to be a recreation, copy or restore. This is recorded in the class file itself rather than as a rejected option — it is a constraint, not a choice.
  • Making longhorn-home-v2 the default class, or migrating the office zone too. Rejected. The class is deliberately explicit and office remains on v1, because only the two home workers have the hugepages, kernel modules and dedicated block disks that v2 requires.

Beyond those, no further alternatives are recorded. An evaluation of an alternative storage backend (simplyblock-evaluation) appears in the 2026-07-14 handoff's list of intended research outputs, but the artifact is not present in the repository, so nothing can be cited from it. The rationale for preferring Longhorn v2 over any other NVMe-oriented backend was not recorded.

Consequences

  • The home 2-replica tier is effectively retired. longhorn-home-v2 has no 2-replica sibling, and the workloads that justified longhorn-home-2replica (gitea, jellyfin, pg-media) are now all on the single-replica v2 class. Grep confirms longhorn-home-2replica has zero remaining consumers; the class is still defined but unused. Single-node-loss survivability for those volumes now depends on backup restore rather than a second live replica.
  • Home scheduled snapshots and backups are currently paused. BUG-41: on Longhorn 1.12.0 a v2 backup's temporary snapshot NVMe path teardown can fail live-frontend validation, remove the live frontend and force mounted filesystems read-only (upstream P0 longhorn#13331). All home RecurringJobs in cluster/longhorn/recurring-jobs-home.yaml are therefore pointed at the empty home-v2-maintenance-paused group. This is still the state in Git: the home zone has had no automatic snapshot or backup since 2026-07-15, pending 1.12.1+. The same P0 remounted ai/release-watch and media/configarr state PVCs read-only (issue #2546); both CronJobs were suspended in PR #2548.
  • Backup owner election is not v2-aware. BUG-40: Longhorn can elect a Backup owner on a node that has only a v1 instance manager, which stalls the backup with failed to find running instance manager ... data engine v2. The workaround is to reassign the status ownerID to a v2-capable home node; no data repair was needed.
  • Temporary migration quota headroom is still in place. Retained v1 and replacement v2 claims had to coexist, so the ai and harbor ResourceQuotas were raised. The handoff says to reduce them after the migration; ai is still at 800 Gi / 20 PVCs and harbor at 250 Gi / 10 PVCs, versus 180 Gi and 160 Gi / 8 before the migration. This is outstanding cleanup.
  • Evacuating the OS roots unblocked the worker rebuild. Once no v1 replicas remained on /var/lib/longhorn, both home workers' 800 GiB OS roots could be replaced with 50 GiB roots plus dedicated /var/lib/containerd disks.
  • The v2 class is Retain, matching the longhorn-home convention, so a mistaken PVC deletion orphans the PV rather than wiping it.

References

  • PR #2487 — feat(storage): batch home PVCs -> Longhorn v2 (25 apps)
  • PRs #2462, #2463 — v2 data-engine enablement and Setting-level enforcement
  • .workingdir2/archive/HANDOFF-2026-07-15-home-longhorn-v2-and-vm-roots.md — active operational tracker for the migration and the worker root rebuild
  • .workingdir2/archive/HANDOFF-2026-07-13-cluster-modernization.md — the operator's originating backlog item
  • .workingdir2/bugs/08-known-bugs.md — BUG-40, BUG-41
  • cluster/longhorn/v2-storageclass.yaml, cluster/longhorn/v2-data-engine-setting.yaml, cluster/longhorn/recurring-jobs-home.yaml