Skip to content

0047. Adjusting Timeouts and Retry Logic for Improved System Reliability

  • Status: accepted
  • Date: 2026-07-24
  • Deciders: lusoris
  • Related: PR #3006

Context

Two independent reliability failures were observed on issue #2959. First, Hindsight 0.8.4 used its default 120-second HINDSIGHT_API_LLM_TIMEOUT and closed queued lm_studio/qwen/qwen3-8b requests before the backend completed; LiteLLM recorded these downstream cancellations as HTTP 499 with zero tokens and zero cost. The cross-site LM Studio instance accepts a four-request queue, so a healthy request can legitimately wait behind several generations or a cold model load, yet LiteLLM's physical-leg timeout for that route was also only 120 seconds. Second, Paperclip 2026.707.0's one-shot asynchronous startup codex_local managed-home reconciliation failed on a single transient PostgreSQL DNS lookup (getaddrinfo EAI_AGAIN pg-cluster-rw.databases.svc.cluster.local) and never retried, leaving persisted managed homes unreconciled until another restart even though the hostname was correct and subsequently resolved.

Decision

  • Raise the LiteLLM LM Studio Qwen3 8B physical-leg timeout from 120 to 1,800 seconds (apps/ai/litellm/values.yaml), so queued local work and cold loads complete instead of being converted into retry amplification.
  • Set Hindsight's client deadline to 2,100 seconds via a new HINDSIGHT_API_LLM_TIMEOUT environment variable rendered from llm.timeout (apps/ai/hindsight). The client deadline is deliberately higher than the gateway leg deadline so LiteLLM owns timeout, routing, and fallback behavior.
  • Patch Paperclip's server entry point to retry the idempotent startup reconciliation only on EAI_AGAIN, with bounded backoff of 1, 2, 4, and 8 seconds; all other errors still fail immediately, and a persistent DNS fault still surfaces after the bounded retries.
  • Raise Paperclip's CPU request from 250m to one full core (the six-core burst limit is unchanged) so migrations, auth-home reconciliation, queued-run recovery, and UI/API startup are not throttled inside a 250m share.

Alternatives considered

  • Keep the 120-second defaults and rely on retries: rejected — the timeout fired on healthy queued work, producing misleading provider-shaped 499 cancellations and retry amplification rather than failures.
  • Let the client (Hindsight) own the deadline with a value at or below LiteLLM's leg timeout: rejected — the gateway must own timeout, routing, and fallback; a lower client deadline reproduces the observed cancellation pattern.
  • Retry all Paperclip startup reconciliation errors or retry without a bound: rejected — only the transient resolver failure is safe to retry on the idempotent path; anything wider would mask real faults.
  • Relax Paperclip's global concurrency cap as part of this repair: explicitly deferred for separate review against the admission controller and LiteLLM queue contract.

Consequences

  • Queued LM Studio work may hold a request open for up to 30 minutes before LiteLLM bounds it, so genuine backend hangs on this leg take longer to surface; in exchange, completed local work is preserved and LiteLLM spend records reflect real outcomes instead of client disconnects.
  • A transient startup DNS failure no longer leaves Paperclip's managed homes unreconciled until another restart; recurring EAI_AGAIN outside the bounded startup path is treated as a separate DNS incident, not grounds for widening retries.
  • Paperclip permanently reserves one CPU core on its node.
  • Chart tests pin the new values (Hindsight env 2100, LiteLLM leg 1800, Paperclip CPU request 1 / limit 6), and the model-deployment projections were regenerated for the changed LiteLLM values.
  • No database hostname, GPU assignment, model alias, provider order, or live cluster object was changed.

References

  • PR #3006 — fix(ai): preserve queued local inference work