Skip to content

0034. Improve Paperclip Resilience and Data Management

  • Status: accepted
  • Date: 2026-07-12
  • Deciders: lusoris
  • Related: PR #2322

Context

The paperclip pod runs a single-threaded event loop. When that loop wedged, the TCP port stayed open and the shallow /health endpoint still returned 200 in under a second, so both a tcpSocket liveness probe and a /health probe kept passing while real work — /api/companies, /api/health — timed out. As a result Kubernetes never restarted the hung pod, producing prolonged downtime. Separately, the heartbeat_runs table and its JSONB payloads grew without bound, degrading the same loop.

Decision

Add an /api/health endpoint that exercises the database and loop path, and switch the liveness probe from tcpSocket/shallow /health to httpGet /api/health with a deliberately tolerant threshold (period 20s, timeout 18s, failureThreshold 9, roughly three minutes sustained) so a genuine multi-minute wedge triggers a restart while transient busy or slow periods do not. Readiness stays on the shallow /health. The mandatory Host header avoids better-auth host-trust 403s being read as probe failures. Cap the JSONB drain and prune old heartbeat_runs rows to keep the loop healthy.

Alternatives considered

  • A short liveness timeout: rejected — it killed busy-but-alive pods mid-drain and re-grew the queue, a health-starvation loop.
  • Keeping tcpSocket/shallow /health liveness: never detects a wedged loop because the socket and shallow endpoint stay up.
  • An external watchdog: more moving parts than a probe that already runs.

Consequences

  • A real wedge now self-heals through a restart; a busy drain is no longer falsely restarted.
  • Liveness depends on the DB/loop path, so the Host header and tolerant thresholds are load-bearing and documented inline in the deployment.
  • Complements ADR 0031's db-janitor for bounding heartbeat_runs growth.

References

  • PR #2322 — fix(paperclip): self-heal wedged event loop + cap the JSONB drain
  • apps/ai/paperclip/templates/deployment.yaml (readiness/liveness probes)