Longhorn alert runbook¶

Source rule file: cluster/monitoring/longhorn-alerts.yaml (PrometheusRule longhorn-alerts, group longhorn-storage-health, namespace monitoring).

All commands below are read-only. This cluster is strictly GitOps: no kubectl apply/patch/delete/scale/edit/annotate, no cordon/drain, no live Longhorn setting edits through the UI. Every configuration fix lands as a PR under cluster/longhorn/ and is applied by ArgoCD.

Useful resource names (the Longhorn CRDs are *.longhorn.io, not longhornvolume/longhornnode): volumes.longhorn.io, nodes.longhorn.io, replicas.longhorn.io, engines.longhorn.io, backups.longhorn.io, snapshots.longhorn.io, instancemanagers.longhorn.io, settings.longhorn.io, backuptargets.longhorn.io, recurringjobs.longhorn.io. Replicas and engines carry the labels longhornvolume=<volume> and longhornnode=<node>; backups carry backup-volume=<volume> and backup-target=<target>.

LonghornVolumeFaulted¶

Fires when: longhorn_volume_robustness == 3 (FAULTED) for 5m. Severity critical. All replicas of the volume are lost or unusable.

Likely causes¶

Every node hosting a replica failed or lost its disk at once.
Replica data corrupted on all copies, so the engine refuses to attach.
A single-replica volume lost its only replica (these read FAULTED immediately, not degraded).

Diagnose¶

kubectl -n longhorn-system get volumes.longhorn.io <volume> -o yaml
kubectl -n longhorn-system get replicas.longhorn.io -l longhornvolume=<volume> -o wide
kubectl -n longhorn-system get engines.longhorn.io -l longhornvolume=<volume> -o wide
kubectl -n longhorn-system logs ds/longhorn-manager --tail=200 | grep -i <volume>
kubectl get nodes -o wide
kubectl -n longhorn-system get backups.longhorn.io -l backup-volume=<volume>

PromQL: longhorn_volume_robustness{volume="<volume>"} (3 = faulted, 2 = degraded, 1 = healthy).

Resolution¶

This is a data-availability incident. First establish whether a usable backup exists (last command above; Completed state). Restore is a data-plane recovery action, performed by creating a restored volume from the backup in the Longhorn UI — it is not a Git change and it must not overwrite the faulted volume in place. There is no longhornctl restore subcommand; do not look for one.

If the root cause is inadequate replication or a missing backup schedule, that fix is a Git change: replica count and zone placement live in cluster/longhorn/zone-storageclasses.yaml, backup schedules in cluster/longhorn/recurring-jobs-{home,office}.yaml.

LonghornFilesystemReadOnly¶

Fires when: max by (instance, device, fstype) (node_filesystem_readonly{device=~"/dev/longhorn/.+", fstype=~"ext4|xfs"}) == 1 for 2m. Severity critical. The OS remounted the filesystem read-only. The Longhorn Volume CR can still report healthy — a v2 frontend can disappear underneath a healthy-looking CR, so trust this alert over the CR.

Likely causes¶

Block-layer I/O errors propagated up to ext4/xfs, which remounted read-only to protect data.
Filesystem metadata corruption detected by the kernel.
The v2 data-engine frontend for the device went away while the volume stayed attached.

Diagnose¶

kubectl describe node <node>
kubectl -n longhorn-system logs ds/longhorn-manager --field-selector spec.nodeName=<node> --tail=200
kubectl -n longhorn-system get instancemanagers.longhorn.io -o wide | grep <node>
kubectl -n longhorn-system get volumes.longhorn.io -o wide | grep <node>

On the node itself (read-only, via SSH or qm guest exec on the PVE host):

dmesg -T | tail -100
mount | grep /dev/longhorn

PromQL: node_filesystem_readonly{device=~"/dev/longhorn/.+"} and rate(node_disk_io_time_seconds_total[5m]) for the backing device.

Resolution¶

Stop writes to the affected volume before anything else — a read-only remount usually means the kernel already saw I/O errors. Identify the workload with the mount and scale it down through Git (its Deployment/StatefulSet manifest), never with a live kubectl scale.

Recovery is a data-plane action: detach and reattach the volume so the engine rebuilds the frontend, done from the Longhorn UI. If the underlying disk is failing, retiring that disk or node is a Git change under cluster/longhorn/ (disk config) plus the Ansible/Terraform node definition.

LonghornVolumeDegraded¶

Fires when: longhorn_volume_robustness == 2 (DEGRADED) for 30m. Severity warning. Fewer healthy replicas than the volume spec asks for. The 30-minute window is deliberate: a rebuild after a node blip is normal and should self-heal well inside it.

Likely causes¶

A node hosting a replica is down or its instance-manager is wedged, so the rebuild cannot start.
No schedulable disk has room — Longhorn will not place a replica below the 25% min-available floor.
Rebuild is throttled behind concurrent-replica-rebuild-per-node-limit and genuinely still running.

Diagnose¶

kubectl -n longhorn-system get volumes.longhorn.io <volume> -o yaml
kubectl -n longhorn-system get replicas.longhorn.io -l longhornvolume=<volume> -o wide
kubectl -n longhorn-system get nodes.longhorn.io -o wide
kubectl -n longhorn-system logs ds/longhorn-manager --tail=200 | grep -i -e rebuild -e <volume>
kubectl get nodes

PromQL: longhorn_volume_robustness == 2, plus (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) * 100 to see whether any node can actually take a replica.

Resolution¶

If a node is down, fix the node — see LonghornNodeDown below. If it is a wedged office instance-manager, that is the known BUG-29 pattern (Longhorn iSCSI wedge, not hardware); bouncing the instance-manager pod is a live action, so confirm the wedge before touching it, and never cordon or force-delete CNPG pods as collateral.

Do not manually delete stuck replicas. If the desired replica count or zone placement is wrong, that is a Git change in cluster/longhorn/zone-storageclasses.yaml; rebuild concurrency lives in cluster/longhorn/settings-robustness.yaml.

LonghornNodeStorageFull¶

Fires when: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) * 100 > 80 for 10m. Severity warning. Threshold is 80%, not 70% — storageMinimalAvailablePercentage: 25 already reserves headroom, so 70% was noisy.

Likely causes¶

Volume growth on this node outpacing snapshot cleanup.
Snapshot accumulation — retention on the recurring jobs is too generous for the churn.
Replica placement skewed onto this node.

Diagnose¶

kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml
kubectl -n longhorn-system get replicas.longhorn.io -l longhornnode=<node> -o wide
kubectl -n longhorn-system get snapshots.longhorn.io
kubectl -n longhorn-system get recurringjobs.longhorn.io
kubectl top nodes

PromQL: (longhorn_node_storage_usage_bytes / longhorn_node_storage_capacity_bytes) * 100 and longhorn_node_storage_capacity_bytes per node.

Resolution¶

Both real fixes are Git changes. Snapshot/backup retention and the snapshot-cleanup / filesystem-trim jobs are declared in cluster/longhorn/recurring-jobs-{home,office}.yaml (note: Kyverno enforces zone pinning on these — see cluster/kyverno/longhorn-recurring-job-zone-pinning.yaml). Replica rebalancing behaviour is the replica-auto-balance Setting in cluster/longhorn/settings-robustness.yaml.

Adding physical capacity means a node disk change plus its Longhorn disk config in Git. Never delete files from a Longhorn disk by hand — that orphans replicas.

LonghornDiskFull¶

Fires when: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) * 100 > 80 for 10m. Severity warning. Same threshold as above but scoped to one disk ({{ $labels.disk }}) rather than the node total — a node can look fine while one disk is full.

Likely causes¶

Replicas concentrated on one disk while others on the same node sit idle.
Snapshot data on this disk not being cleaned up.
Disk smaller than its peers, so the same replica count fills it first.

Diagnose¶

kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml
kubectl -n longhorn-system get replicas.longhorn.io -l longhorndiskuuid=<disk> -o wide
kubectl -n longhorn-system get snapshots.longhorn.io

PromQL: (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) * 100 > 80 — the disk label is the disk UUID, which maps to a path in the nodes.longhorn.io CR above.

Resolution¶

Rebalancing is driven by the replica-auto-balance Setting in cluster/longhorn/settings-robustness.yaml and by disk tags in the node's Longhorn disk config — both Git changes. Retention tuning is in cluster/longhorn/recurring-jobs-{home,office}.yaml. Do not delete data from the disk directly.

LonghornNodeDown¶

Fires when: count(longhorn_node_status{condition="ready"} == 0) > 0 for 10m. Severity warning. Counts Longhorn nodes whose ready condition is 0. Note the alert reports only a count — it does not name the node, so start by finding which one.

Scheduling on k8s-worker-office-3 is intentionally disabled; that does not clear readiness, so a not-ready result there is still a real signal.

Likely causes¶

Node genuinely down (reboot, kernel panic, power loss).
longhorn-manager pod on that node is not running or cannot reach the manager peers.
Node is up but its instance-manager/iSCSI stack is wedged — the office-node pattern that looks like a disk failure but is not (BUG-29).

Diagnose¶

kubectl -n longhorn-system get nodes.longhorn.io -o wide
kubectl get nodes -o wide
kubectl -n longhorn-system get pods -l app=longhorn-manager -o wide
kubectl -n longhorn-system logs ds/longhorn-manager --tail=200
kubectl -n longhorn-system get instancemanagers.longhorn.io -o wide
kubectl describe node <node>

PromQL: longhorn_node_status{condition="ready"} == 0 — this form gives the node label the alert omits.

Resolution¶

Diagnose the node first; for an unreachable node use the PVE host and qm guest exec rather than declaring it needs console access. A single node down in a quorum cluster is the design working — do not escalate on that alone.

Do not cordon or drain to "fix" this. If a node is being retired or its scheduling posture must change, that is a Git change in the Ansible/Terraform node definition and cluster/longhorn/ disk config.

LonghornBackupFailed¶

Fires when: longhorn_backup_state == 4 (Error) for 15m. Severity warning. Longhorn 1.11+ exposes no backup-target-health metric, so a failed backup is the actual signal that a zone's S3 target is unreachable.

Likely causes¶

The zone MinIO/S3 target is down or unreachable over the LAN.
Credentials rotated in Git but the secret was not resealed, or the backup target references a stale secret.
Backup target out of capacity.

Diagnose¶

kubectl -n longhorn-system get backuptargets.longhorn.io
kubectl -n longhorn-system get backups.longhorn.io | grep -i error
kubectl -n longhorn-system get backups.longhorn.io <backup> -o yaml
kubectl -n longhorn-system logs ds/longhorn-manager --tail=200 | grep -i backup
kubectl -n longhorn-system get secret minio-home-backup-creds minio-office-backup-creds

The backuptargets.longhorn.io listing shows AVAILABLE and LASTSYNCEDAT per target — that is the fastest reachability check. Note the default target is unused and normally reports unavailable; only home-nas and office-nas matter.

Do not dump the credential secret with -o yaml; the listing above confirms existence without printing the data.

PromQL: longhorn_backup_state == 4.

Resolution¶

If the target is unreachable, fix MinIO/the NAS — a data-plane/infra action. Backups retry on their next scheduled run.

If credentials or the target URL are wrong, that is a Git change: targets are declared in cluster/longhorn/backup-target-{home,office}.yaml and the credentials are sealed secrets. Reseal first, then let the rollout follow — an already-running pod holds the old secret value.

LonghornVolumeBackupStale¶

Fires when: longhorn_volume_last_backup_at > 0 and (time() - longhorn_volume_last_backup_at) > 129600 for 30m. Severity warning. 129600 s = 36 h. Daily backups run at 04:00, so >36 h means two cycles were missed. The > 0 guard excludes volumes that were never backed up (e.g. *-no-backup storage-class volumes), so this only fires on volumes that previously had a backup and then stopped.

Likely causes¶

Backups are erroring — check LonghornBackupFailed first, it is usually the same incident.
The volume lost its recurring-job group label, so no schedule selects it any more.
The recurring job itself is paused or its group was renamed (note the home-v2-maintenance-paused group currently on the home jobs).

Diagnose¶

kubectl -n longhorn-system get volumes.longhorn.io <volume> -o yaml
kubectl -n longhorn-system get backups.longhorn.io -l backup-volume=<volume>
kubectl -n longhorn-system get recurringjobs.longhorn.io
kubectl -n longhorn-system get backuptargets.longhorn.io

PromQL: (time() - longhorn_volume_last_backup_at) / 3600 for hours since last backup across all volumes.

Resolution¶

If the target is broken, resolve LonghornBackupFailed and the schedule catches up on its own.

If the volume fell out of its schedule, that is a Git change: recurring jobs and their groups are in cluster/longhorn/recurring-jobs-{home,office}.yaml, the group label is applied via the storage class in cluster/longhorn/zone-storageclasses.yaml, and cluster/longhorn/longhorn-no-backup.yaml defines the deliberately-unbacked classes. Confirm the volume is supposed to be backed up before treating this as a fault.

If the recurring job group is paused, un-pausing it is a Git change too — do not re-enable it live.

LonghornInstanceManagerCPUHigh¶

Fires when: (longhorn_instance_manager_cpu_usage_millicpu / longhorn_instance_manager_cpu_requests_millicpu) * 100 > 300 for 15m. Severity warning. Usage above 300% of the request (not of a core, and not a limit) — the instance manager is allowed to burst, so this means sustained pressure, not a hard ceiling.

Likely causes¶

Rebuild storm — several replicas rebuilding at once after a node returned.
Genuinely heavy volume I/O on that node.
guaranteed-instance-manager-cpu set too low for the replica count this node now carries.

Diagnose¶

kubectl top pods -n longhorn-system --sort-by=cpu
kubectl -n longhorn-system get instancemanagers.longhorn.io -o wide
kubectl -n longhorn-system get replicas.longhorn.io -l longhornnode=<node> -o wide
kubectl -n longhorn-system get volumes.longhorn.io -o wide | grep -i degraded
kubectl top nodes

PromQL: (longhorn_instance_manager_cpu_usage_millicpu / longhorn_instance_manager_cpu_requests_millicpu) * 100.

Resolution¶

If it is a rebuild storm, it is self-limiting — confirm rebuilds are progressing (degraded volume count falling) and let it finish.

If it is sustained under normal load, the fix is a Git change: the guaranteed-instance-manager-cpu Setting in cluster/longhorn/settings-robustness.yaml, alongside concurrent-replica-rebuild-per-node-limit if rebuild concurrency is the driver. Changing instance-manager CPU restarts the instance managers, so treat it as a maintenance action, not a hot fix. Do not scale or edit the pods directly.