Skip to content

Media library alert runbook

Source rule: cluster/monitoring/media-library-alerts.yaml (PrometheusRule media-library-alerts, namespace monitoring, group media-library-health).

Context

Three alerts cover the media namespace library stack: Jellyfin serving errors, *arr backlog stagnation, and the data/media ZFS pool on the PVE host.

Two of the three carry verification caveats recorded during this runbook's review — see the flagged blocks below. Read them before trusting a "no alert firing" signal as evidence of health.

.workingdir2/archive/HANDOFF-media-stack.md still lists a fourth alert, StashScenesUnstashboxed, as living in this file. It does not exist in this rule file or anywhere else in the repo. The handoff doc is stale on that point.
Namespace (workloads) media
Namespace (rule) monitoring
*arr metrics source exportarr v2.3.0 sidecar per app, own Service + ServiceMonitor each
Jellyfin metrics sources native /metrics (prometheus-net / ASP.NET) + jellyfin-exporter (:9594)
ZFS metrics source node_exporter on the PVE01-home host (PR #548), not a pod

Note that sonarr, sonarr-4k, sonarr-anime, radarr, radarr-4k and lidarr are separate Helm releases, each with its own ServiceMonitor and therefore its own job label.

JellyfinHighErrorRate

Severity: warning · Component: media · Subsystem: jellyfin

Fires when: the 5xx share of Jellyfin HTTP responses exceeds 5% and stays above it continuously for 10m.

sum(rate(jellyfin_http_responses_total{status=~"5.."}[5m]))
/
clamp_min(sum(rate(jellyfin_http_responses_total[5m])), 1) > 0.05

Two details that change how you read this:

  • The rate window is 5m; the for: is 10m. The alert needs the 5m-windowed ratio to stay over the threshold for a full 10 minutes, so a short error burst will not page.
  • The denominator is clamp_min(..., 1), floored at 1 request/sec. Below ~1 rps of total traffic the denominator is held at 1, so the computed ratio understates the true error share. On a quiet library, a genuinely broken Jellyfin can sit under 5% and never fire.

Caveat — metric producer not confirmed. jellyfin_http_responses_total appears nowhere in this repo except inside this expression: no dashboard references it, and jellyfin-exporter documents only jellyfin_media_count, jellyfin_now_playing_state and jellyfin_transcoding_sessions (apps/media/jellyfin-exporter/README.md). Native Jellyfin /metrics emits prometheus-net runtime and ASP.NET/Kestrel series, which use different names. Confirm the series exists before treating silence as health — run the bare selector below. If it returns nothing, this alert cannot fire and correcting the rule is a Git change.

Likely causes

  1. Hardware transcode failure — Arc dGPU driver or device-plugin unavailability causing FFmpeg to fail out.
  2. FFmpeg crashes on unsupported codecs or damaged media files.
  3. Plugin crash surfacing as 500s.
  4. Library database corruption or unavailability blocking metadata reads.

Diagnose (all read-only)

kubectl -n media logs -l app.kubernetes.io/name=jellyfin --tail=200
kubectl -n media logs -l app.kubernetes.io/name=jellyfin --previous --tail=200
kubectl -n media describe pod -l app.kubernetes.io/name=jellyfin
kubectl -n media top pod -l app.kubernetes.io/name=jellyfin
kubectl -n media get pod -l app.kubernetes.io/name=jellyfin -o wide

PromQL:

jellyfin_http_responses_total
sum(rate(jellyfin_http_responses_total{status=~"5.."}[5m])) by (status)
sum(rate(jellyfin_http_responses_total[5m]))
jellyfin_transcoding_sessions

Run the bare jellyfin_http_responses_total selector first. An empty result means the alert is inert and the incident must be diagnosed from logs, not from this alert.

Resolution

  • Transcode-path faults are usually node-level (driver, gpu.intel.com/xe advertisement). Diagnose on the node; any device-plugin or node config change is a Git change under cluster/ or the Ansible tree.
  • A bad image or plugin version is fixed by bumping the pinned tag/digest in apps/media/jellyfin/values.yaml and letting ArgoCD sync — a Git change, not a live edit.
  • Corrupt media files are fixed at the source file, outside the cluster.
  • If a pod restart is genuinely the remedy, note that an ArgoCD sync does not restart an unchanged pod. Restarting is a live mutation and is out of scope for this GitOps cluster without operator sign-off; prefer landing the real fix in Git.

ArrMissingBacklogStale

Severity: warning · Component: media · Subsystem: arr

Fires when: the combined missing count is > 0 and has recorded zero changes over a 7-day window sampled hourly, holding for 1h.

(
  (sonarr_episode_missing_total +
   radarr_movie_missing_total +
   ignoring(__name__) lidarr_album_missing_total) > 0
)
and on(job)
(changes(
  (sonarr_episode_missing_total +
   radarr_movie_missing_total +
   ignoring(__name__) lidarr_album_missing_total)[7d:1h]) == 0)

The for: 1h is easy to miss and matters: this is a slow-burn alert, not a live signal. The [7d:1h] is a subquery — a 7-day range evaluated at 1-hour resolution.

Caveat — cross-job vector matching. Each *arr runs its own exportarr sidecar behind its own ServiceMonitor (cluster/monitoring/servicemonitors.yaml), so sonarr_episode_missing_total, radarr_movie_missing_total and lidarr_album_missing_total carry different job/instance/pod labels. Prometheus arithmetic between instant vectors matches one-to-one on the remaining labels, so these three series have no matching partners and the sum plausibly evaluates to an empty vector — meaning the alert never fires. The and on(job) and the {{ $labels.job }} in the annotation also imply per-job evaluation, which contradicts summing across jobs. Verify by running the sum expression directly before trusting silence. Fixing this requires rewriting the rule, which is a Git change.

Likely causes (assuming the alert is live)

  1. Broken or unreachable Prowlarr indexer — searches return nothing, so the backlog never drains.
  2. Custom-format or quality-profile gating so strict that no available release qualifies.
  3. Stuck download-client queue (qBittorrent / SABnzbd integration failure, stalled items).
  4. Network or credential failure between the *arr apps and Prowlarr or the download clients.

Diagnose (all read-only)

kubectl -n media logs -l app.kubernetes.io/name=sonarr --tail=100      # repeat: radarr, lidarr
kubectl -n media logs -l app.kubernetes.io/name=prowlarr --tail=100
kubectl -n media logs -l app.kubernetes.io/name=qbittorrent --tail=100
kubectl -n media logs -l app.kubernetes.io/name=sabnzbd --tail=100
kubectl -n media get pods -o wide

PromQL — query each metric separately, since the combined expression is the suspect part:

sonarr_episode_missing_total
radarr_movie_missing_total
lidarr_album_missing_total
changes(sonarr_episode_missing_total[7d:1h])

The *arr Wanted view and the Prowlarr indexer health page are the authoritative UI checks; both are read-only surfaces as long as you do not trigger re-searches.

Resolution

  • Indexer credentials/availability live in Prowlarr config, managed by configarr / the bootstrap Jobs — correcting them is a Git change to the relevant chart values or ConfigMap, then an ArgoCD sync.
  • Quality profiles and custom formats are likewise Git changes via configarr.
  • A genuinely hung download client needs a pod restart, which is a live mutation — do not perform it as a routine step. Land the underlying fix in Git, or escalate to the operator if a restart is unavoidable.

DataMediaZfsHigh

Severity: critical · Component: media · Subsystem: storage

Fires when: data/media ZFS consumption exceeds 90% and holds for 30m.

(
  sum(node_zfs_zpool_dataset_used_bytes{dataset="data/media"})
  /
  (sum(node_zfs_zpool_dataset_used_bytes{dataset="data/media"}) +
   sum(node_zfs_zpool_dataset_available_bytes{dataset="data/media"}))
) > 0.90

Ratio is used / (used + available). Both series come from node_exporter running on the PVE01-home host, not from a pod in the cluster. data/media is hierarchical — child datasets share the parent's free space, hence the sum().

This is the one alert in this group with no verification caveat: the metrics and the mechanism check out.

Likely causes

  1. Library growth outrunning pool capacity — the ordinary case.
  2. Accumulated ZFS snapshots holding space that deleted files would otherwise release.
  3. Large incomplete downloads or temp files, e.g. a stalled qBittorrent/SABnzbd job.
  4. *arr history/queue retention holding onto imported artifacts.

Longhorn metadata bloat, ARC eviction and scan churn are consequences of the pool filling, not causes — do not chase them as the root cause.

Diagnose (all read-only)

PromQL first — it needs no host access:

node_zfs_zpool_dataset_used_bytes{dataset="data/media"}
node_zfs_zpool_dataset_available_bytes{dataset="data/media"}
node_zfs_zpool_dataset_used_bytes{dataset=~"data/media.*"}
predict_linear(node_zfs_zpool_dataset_available_bytes{dataset="data/media"}[6h], 7*24*3600)

The predict_linear line estimates available bytes a week out; a negative result means the pool fills within the week.

On the PVE01-home host (read-only commands only):

zfs list -o name,used,avail,refer -r data/media
zfs list -t snapshot -o name,used,creation -s used -r data/media
zpool list -o name,size,alloc,free,cap,health

In-cluster context:

kubectl -n media get pvc
kubectl -n media logs -l app.kubernetes.io/name=qbittorrent --tail=100

The zfs commands must be run on the PVE host. There is no node-exporter pod to exec into for this data — node-exporter here is a host-level install, and the exporter container would carry neither a shell nor the zfs binary.

Resolution

  • Pruning snapshots and deleting media are live host actions on PVE, outside the GitOps flow. They are destructive and permanent — get operator confirmation before removing anything, and prefer the largest expirable snapshots identified by the zfs list -t snapshot query above.
  • Changing snapshot retention policy is a Git change to the snapshot job/cron definition, not an ad-hoc host edit.
  • Growing the pool or adjusting quotas is a Git change in the infrastructure-as-code tree (Ansible/Terraform), plus the physical capacity work.
  • *arr history/queue retention settings are a Git change via configarr.
  • If the alert is firing because the threshold is genuinely too tight for normal operation, adjust it in cluster/monitoring/media-library-alerts.yaml — a Git change. Do not edit the PrometheusRule live; ArgoCD will revert it.