Skip to content

0035. Retention Period Adjustment for Paperclip DB Janitor

  • Status: accepted
  • Date: 2026-07-13
  • Deciders: lusoris
  • Related: PR #2376

Context

The db-janitor introduced in ADR 0031 pruned all run payloads uniformly at two days. The result_json payloads of succeeded runs are the training corpus for the tiny-specialists SFT pipeline (mine_runs.py), and a two-day window was too short to reliably rebuild that corpus. The payloads of failed, cancelled, and timed-out runs carry no such value and sit on the wedge path when they accumulate.

Decision

Split the payload retention policy by run status. Succeeded runs keep their payloads for 14 days (runsPayloadSuccessRetentionDays) so the SFT corpus survives a rebuild cycle; failed, cancelled, and timed-out runs keep the existing 2-day window (runsPayloadRetentionDays). The additional roughly 50-70 MB of succeeded payloads is not on the wedge path.

Alternatives considered

  • Keep a uniform 2-day window: loses the SFT corpus.
  • Retain all payloads for a long window: reintroduces the bloat and wedge risk ADR 0031 removed.
  • Export the corpus to a separate store on each run: more moving parts than extending retention for one status class.

Consequences

  • The tiny-specialists SFT corpus survives long enough to be rebuilt.
  • Disk use rises modestly but stays bounded, and the extra rows are off the wedge path.
  • Refines the retention policy set by ADR 0031; both are parameterised in the janitor's values.yaml.

References

  • PR #2376 — feat(paperclip-db-janitor): succeeded-run payloads keep 14d — rebuild the SFT corpus
  • apps/ai/paperclip-db-janitor/values.yaml, apps/ai/paperclip-db-janitor/files/prune.sql