Skip to content

0019. Long-term metrics sink: VictoriaMetrics

  • Status: proposed
  • Date: 2026-05-29
  • Deciders: lusoris

Context

The in-cluster Prometheus (kube-prometheus-stack) runs with --storage.tsdb.retention.time=2d and no remote_write configured. Anything older than ~48h is gone for good. Three concrete consequences have surfaced:

  1. Dead-alert audit is impossible. "Which PrometheusRule alerts have not fired in the last 30 days?" is a load-bearing hygiene question — it tells us which rules to retire, retune, or promote from warning to critical. With a 2-day window we cannot answer it at all; the recent prometheus-rule-deduplicator work (apps/ai/prometheus-rule-deduplicator/) compounds the loss because we have no way to validate that dedup did not silently mask a 30-day-rare alert.
  2. Postmortem rot. Incidents older than two days have to be reconstructed from Loki logs alone. Time-series context (CPU, memory, request rate, replica counts, queue depth) is gone before a weekend postmortem can land. The recent Authentik / open-webui network-policy work would have benefitted from a 7-day connection graph; we had ~36h.
  3. No long-trend dashboards. Monthly cost (per-namespace CPU/RAM integrated over time), capacity planning (Longhorn volume growth week-over-week), and "is this app's footprint creeping?" cannot be built. Grafana dashboards that ask for [30d] silently fall back to sparse data.

A long-term sink is needed. The cluster already has two adjacent primitives that constrain the choice:

  • MinIO Office is the existing S3-compatible backup sink (s3://*@us-east-1/, see memory: reference_synology_backup_target.md for the cross-site backup contract). An object-store-backed sink would inherit MinIO's availability profile.
  • Longhorn provides replicated PVCs cluster-locally. A PVC-backed sink avoids any cross-site round-trip on the query path.

Decision

Adopt VictoriaMetrics (single-node, vmsingle) as the long-term metrics sink. Configure the cluster Prometheus to remote_write into it; keep Prometheus's 2-day local TSDB for short-range recording-rule work and immediate Grafana queries.

Rationale:

  1. Single-binary install. vmsingle is one container, one PVC, one Service. No operator, no distributed components, no consensus layer to operate. Matches the cluster's "operate as little infra as possible" posture per PRINCIPLES.md.
  2. Native Prometheus remote_write target. No Prometheus chart-fork is needed — kube-prometheus-stack ships a top-level prometheus.prometheusSpec.remoteWrite field. The change is a values delta, not a code change.
  3. PVC-backed, not S3-backed. Our query patterns (30-day-aggregation dashboards, dead-alert audits) are read-amplified — every load-time evaluation rescans the window. Object-store TSDB (Thanos / Mimir) pays for that in GET round-trips and bucket-cache misses; a Longhorn PVC pays it in replica-local disk read with no egress. MinIO Office stays the backup sink, not the live query path.
  4. Fits our cardinality. Cluster currently emits ~5 000 active series. vmsingle documented RSS at this scale is well under 100 MB; published benchmarks put the per-series cost about an order of magnitude lower than Prometheus's own. Retention of 1 year on ~5 k series fits comfortably in a 10 GB PVC.
  5. VictoriaMetrics speaks PromQL (with MetricsQL extensions). Grafana datasource swap is one-field — no dashboard rewrites.

Alternatives considered

  • Thanos. Rejected. Sidecar + querier + compactor + store-gateway is four moving parts for a five-thousand-series workload. The bucket cache, the deduplication semantics, and the compaction-overlap failure modes all have non-trivial operational surface. Justified at fleet scale, over-engineered here.
  • Mimir. Rejected. Simpler than Thanos in concept but still a distributed system (ingesters, queriers, store-gateways, compactors) with a Memberlist gossip plane to operate. The reference deployment assumes object storage as the hot path, which is the wrong shape for our query mix.
  • Increase Prometheus TSDB retention to 30 d in place. Rejected as insufficient. Prometheus's per-series memory cost grows roughly linearly with retention; 30 d at current cardinality would push the pod past its memory limit and trigger Kyverno check-memory-limits violations on the next chart bump (see memory: feedback_kyverno_admission_on_chart_bumps.md). It also does not solve the "remote_write to a sink that survives a cluster rebuild" requirement.

Consequences

Positive:

  • Dead-alert audit becomes runnable as a vmselect-backed query over ALERTS{alertstate="firing"} integrated over 30 d. The prometheus-rule-deduplicator cron can extend to a vm-rule-audit cron in the same pattern.
  • Postmortem reconstruction up to the configured retention horizon (1 year by default) — CPU/memory/request-rate graphs survive into the postmortem window.
  • Monthly cost and capacity dashboards become buildable. Grafana acquires a second datasource (victoria-metrics) selected by the dashboard's time range.

Operational:

  • New ArgoCD app apps/monitoring/victoria-metrics/ with the victoria-metrics-single chart pinned per /bump-helm-chart conventions.
  • ~10 GB Longhorn PVC, two-replica (matches default Longhorn class).
  • kube-prometheus-stack values.yaml gains a remoteWrite block pointing at the in-cluster vmsingle Service. Prometheus retention stays at 2 d — vmsingle is the long-term sink, not a replacement.
  • Grafana datasource provisioning gains a second entry. Long-trend dashboards (cost, capacity, dead-alert audit, weekly cardinality) target victoria-metrics; short-range dashboards (alerting, drilldown, recent incidents) keep targeting prometheus. The split is documented per dashboard.
  • New /verify-victoria-metrics skill alongside the existing /verify-prometheus-targets to confirm scrape-to-sink round-trip.

Negative / costs:

  • One more component to keep version-pinned and renovate-tracked.
  • Two datasources means dashboard authors must pick the right one; partly mitigated by the documented split and a Grafana datasource-default rule per folder.
  • VictoriaMetrics' MetricsQL is a superset of PromQL, but queries copied from VM-only dashboards back into Prometheus can fail silently. Convention: victoria-metrics dashboards are authoritative for long-range, prometheus dashboards stay PromQL-pure.

References

  • Earlier-in-session finding: prometheus --storage.tsdb.retention.time=2d with no remote_write, surfaced during the dead-alert audit attempt that prompted this ADR. The audit query (count_over_time(ALERTS{alertstate="firing"}[30d])) returned empty because the underlying series did not exist beyond the 2-day window.
  • memory: reference_synology_backup_target.md — MinIO Office is the cross-site backup sink; remains the backup path, not the live query path under this decision.
  • memory: feedback_kyverno_admission_on_chart_bumps.md — informs the "retention-in-place is not viable" alternative.
  • apps/ai/prometheus-rule-deduplicator/ — dedup cron that needs the long-term sink to validate its own output.
  • apps/ai/cluster-coverage-auditor/ — consolidated hygiene workflow whose Prometheus target-coverage lane also benefits from a multi-week window.