Skip to content

Velero alert runbook

Source rule: cluster/monitoring/velero-alerts.yaml (PrometheusRule velero-alerts, ns monitoring).

All 11 alerts in that rule are documented below. Metric names and labels are those of Velero v1.18 (pkg/metrics/metrics.go), the running version.

GitOps rule: every command in this runbook is read-only. Velero objects (Schedules, BSLs, Backups, credentials secrets) are ArgoCD-managed — fixes land as a Git change, never as a live kubectl apply/patch/edit/delete. Where a live action is genuinely the fix, it is called out explicitly.

Topology referenced by these alerts: - BSL default → office MinIO (minio.office.svc:9000, ns office) - BSL home → home MinIO (minio-home.media.svc:9000, ns media) - Schedules: velero-daily-resources, velero-weekly-resources, velero-daily-arr-stack (cron 30 2 * * *, ns media, TTL 72h — source for the monthly restore-drill)

velero-backup-health

VeleroBackupFailure

Severity: warning Fires when: increase(velero_backup_failure_total[24h]) > 0 sustained for: 1h — at least one backup reached the Failed phase in the last 24h.

Likely causes

  1. BackupStorageLocation connectivity (MinIO down, or the cross-site link degraded).
  2. Velero pod resource exhaustion (OOMKilled) before completion.
  3. Invalid resource references in the Schedule template (e.g. a deleted CRD).

Diagnose

  • kubectl get backups -n velero --sort-by=.metadata.creationTimestamp | tail -10
  • kubectl logs deploy/velero -n velero --tail=200
  • velero backup describe <name> -n velero (read-only)
  • PromQL: increase(velero_backup_failure_total[24h]) > 0 — the schedule label names the failing schedule.

Resolution: If the cause is configuration (excluded resources, storageLocation, credentials), fix the Schedule or sealed secret in Git and let ArgoCD sync. If it was a transient BSL blip, the next scheduled run clears the alert — do not hand-run a backup via CLI.

VeleroBackupPartialFailure

Severity: warning Fires when: increase(velero_backup_partial_failure_total[24h]) > 0 sustained for: 1h — backups completed but individual items errored. Restore from such a backup may be incomplete.

Likely causes

  1. RBAC denial on specific namespaces or resource types.
  2. Transient API-server timeouts during item collection.
  3. Resources that error on read/serialization.

Diagnose

  • kubectl get backups -n velero --sort-by=.metadata.creationTimestamp | tail -5
  • kubectl describe backup <name> -n velero — inspect the Warnings/Errors counts in status.
  • velero backup describe <name> -n velero --details (read-only)
  • PromQL: increase(velero_backup_partial_failure_total[24h])

Resolution: Requires a Git change to the Schedule template (includedNamespaces, excludedResources, itemOperationTimeout) or to the Velero ClusterRole if RBAC is the gap. Do not patch the Backup object — Backups are immutable records of a run.

VeleroBackupValidationFailure

Severity: warning Fires when: increase(velero_backup_validation_failure_total[6h]) > 0 sustained for: 30m — a Backup spec was rejected at validation, before execution.

Likely causes

  1. Schedule references a non-existent storageLocation or an invalid TTL.
  2. Malformed backup template in the ArgoCD-managed manifest.
  3. Missing required fields in the generated Backup.

Diagnose

  • kubectl get schedules -n velero
  • kubectl describe schedule <name> -n velero — the template and last-run status.
  • PromQL: increase(velero_backup_validation_failure_total[6h]) — schedule label identifies the offender.

Resolution: Always a Git change — correct the Schedule manifest in the repo. A validation failure is deterministic and will repeat every run until the spec is fixed; it will not self-clear.

velero-backup-freshness

VeleroDailyBackupMissing

Severity: critical Fires when: time() - max(velero_backup_last_successful_timestamp{schedule="velero-daily-resources"}) > 48 * 3600 sustained for: 1h — no successful daily-resources backup in over 48h, i.e. two consecutive misses.

Likely causes

  1. Persistent BSL outage (MinIO down).
  2. Velero controller pod down or in CrashLoopBackOff.
  3. Cross-site link down, blocking the BSL in use.

Diagnose

  • kubectl get pods -n velero
  • kubectl logs deploy/velero -n velero --since=48h | grep -iE "error|fail"
  • kubectl get backupstoragelocations -n velero (short name bsl)
  • PromQL: max(velero_backup_last_successful_timestamp{schedule="velero-daily-resources"}) — exact last-success epoch.

Resolution: If Velero itself is unhealthy, treat the pod state as the symptom and find the cause (config, quota, node). If the BSL is down, restore the MinIO service. Config fixes go through Git; only genuine infrastructure recovery (MinIO service, node, link) is a live action.

VeleroWeeklyBackupMissing

Severity: warning Fires when: time() - max(velero_backup_last_successful_timestamp{schedule="velero-weekly-resources"}) > 9 * 24 * 3600 sustained for: 1h — the weekly (Sun 03:00) run has not succeeded in over 9 days, i.e. a missed Sunday plus slack.

Likely causes

  1. Same as the daily alert — BSL outage or Velero unavailability across the weekly window.
  2. Schedule cron/timezone misalignment.
  3. Namespace ResourceQuota blocking pod creation during the window.

Diagnose

  • kubectl get backups -n velero | grep weekly-resources
  • kubectl describe schedule velero-weekly-resources -n velero
  • kubectl get resourcequota -n velero
  • PromQL: max(velero_backup_last_successful_timestamp{schedule="velero-weekly-resources"})

Resolution: Cron, quota, and schedule-template fixes are Git changes. If the miss was a one-off during an outage that has since cleared, the next Sunday run resolves the alert.

VeleroArrStackBackupStale

Severity: warning Fires when: time() - max(velero_backup_last_successful_timestamp{schedule="velero-daily-arr-stack"}) > 50 * 3600 sustained for: 1h. This is the restore-drill's source backup; 50h catches a miss while a usable candidate (TTL 72h) still exists.

Why it matters: the monthly restore-drill restores the newest arr-stack backup. Stale here means the drill silently validates an old backup.

Likely causes

  1. Volume/namespace issues in ns media blocking the backup.
  2. BSL unreachable at the 02:30 run time.
  3. Velero pod restart across the 02:30 window.

Diagnose

  • kubectl get backups -n velero | grep arr-stack
  • kubectl logs deploy/velero -n velero --since=50h | grep arr-stack
  • kubectl describe schedule velero-daily-arr-stack -n velero
  • PromQL: max(velero_backup_last_successful_timestamp{schedule="velero-daily-arr-stack"})

Resolution: Template problems are a Git change. If the cause is a transient storage or network fault that has cleared, the 02:30 run restores freshness — do not trigger a backup by hand.

VeleroNoBackupMetrics

Severity: warning Fires when: absent(velero_backup_last_successful_timestamp) sustained for: 30m.

Why it matters: this is the watchdog on the watchdog. With no series, every freshness alert above goes silent rather than firing — absent() is the only rule that still triggers.

Likely causes

  1. Velero pod down or not Ready, so the ServiceMonitor target is empty.
  2. ServiceMonitor / label-selector drift so Prometheus no longer selects the target.
  3. NetworkPolicy blocking Prometheus from scraping the metrics port (container port 8085, http-monitoring).

Diagnose

  • kubectl get pods -n velero
  • kubectl get svc,endpoints -n velero
  • kubectl get servicemonitor -n velero -o yaml
  • PromQL: up{namespace="velero"} — confirm the scrape target's exact job label on the Prometheus Targets page before filtering on job=.

Resolution: ServiceMonitor labels and NetworkPolicy are ArgoCD-managed — fix in Git. Velero's liveness/readiness probes hit /metrics, so a Ready pod that exports nothing points at the scrape path, not at Velero.

velero-infra-health

VeleroBSLUnavailable

Severity: critical Fires when: velero_backup_location_status_gauge == 0 sustained for: 10m for any BackupStorageLocation (label backup_location_name). The 10m window tolerates a transient cross-site tunnel blip but pages within minutes of a real outage.

Impact: one BSL down = off-site redundancy lost (the other still serves backup and restore). Both down = nothing writable or restorable.

Likely causes

  1. MinIO down at the corresponding site.
  2. Cross-site tunnel down.
  3. Bad credentials in the velero-credentials / velero-credentials-home secrets.

Diagnose

  • kubectl get backupstoragelocations -n velero — which location is Unavailable.
  • kubectl logs deploy/velero -n velero | grep -i backupstoragelocation — validation errors.
  • kubectl get pods -n office -l app=minio and kubectl get pods -n media -l app=minio — MinIO at both sites.
  • PromQL: velero_backup_location_status_gauge == 0

Resolution: Credential fixes are a Git change (re-seal the secret; note that env secretKeyRef values snapshot at pod create, so a rollout is needed after the sealed secret syncs). MinIO or link recovery is infrastructure work. Do not patch the BSL object to force it Available.

VeleroRepoMaintenanceFailing

Severity: warning Fires when: increase(velero_repo_maintenance_failure_total[6h]) > 2 sustained for: 1h — note the threshold is more than 2 failures, not any failure. Single transient failures are tolerated because retries succeed.

Why it matters: failed maintenance pods accumulate in Error state against the velero ResourceQuota (30 pods).

Likely causes

  1. Network instability to MinIO on the cross-site path.
  2. Bucket permission or repository state problems.
  3. ResourceQuota exhaustion preventing maintenance pods from starting.

Diagnose

  • kubectl get pods -n velero | grep -i maintenance
  • kubectl get resourcequota -n velero
  • kubectl logs deploy/velero -n velero | grep -i maintenance — the repository_name label identifies the repo.
  • PromQL: increase(velero_repo_maintenance_failure_total[6h])

Resolution: Quota sizing, loadAffinity pinning to office, and NetworkPolicy are Git changes. Do not delete the Error pods to clear the quota — that hides the cause and the controller recreates them; fix the underlying reachability or quota first.

VeleroRestoreFailed

Severity: warning Fires when: increase(velero_restore_failed_total[1h]) > 0 sustained for: 5m.

Why it matters: if this is the monthly restore-drill, the backups may not be restorable at all. That is the most serious class of signal in this runbook — investigate immediately.

Likely causes

  1. Incomplete or corrupt backup data (often downstream of a prior partial failure).
  2. StorageClass mismatch between the backup and the current cluster.
  3. Longhorn volume restore problems (missing snapshot, insufficient space).

Diagnose

  • kubectl get restores -n velero --sort-by=.metadata.creationTimestamp | tail -5
  • kubectl describe restore <name> -n velero
  • velero restore describe <name> -n velero --details (read-only)
  • kubectl get volumes.longhorn.io -n longhorn-system — check target volume state.
  • PromQL: increase(velero_restore_failed_total[1h])

Resolution: Restore-drill definitions and any StorageClass mapping live in Git — fix there. Per repo policy, restore drills target sandbox namespaces only, never live data. If the backup itself is unrestorable, escalate: that invalidates the recovery path, not just this run.

VeleroBackupDeletionFailing

Severity: warning Fires when: increase(velero_backup_deletion_failure_total[24h]) > 0 sustained for: 2h — expired backups are not being garbage-collected, so the office MinIO bucket grows unbounded.

Likely causes

  1. BSL write/delete permissions revoked, or object-lock on the bucket.
  2. Network partition preventing object deletion.
  3. Velero GC controller erroring against the API server.

Diagnose

  • kubectl get backups -n velero — look for accumulating Expired/Deleting entries.
  • kubectl get deletebackuprequests -n velero
  • kubectl logs deploy/velero -n velero | grep -iE "deletion|gc"
  • PromQL: increase(velero_backup_deletion_failure_total[24h])

Resolution: Credential and BSL-permission fixes are a Git change. Do not delete Backup objects by hand to reclaim space — that orphans the objects in the bucket without releasing them. Fix the delete path, then let GC drain the backlog.