Skip to content

0046. Retire the bounded-lane-champion challenger; make baseline-adopted a signable outcome

  • Status: accepted
  • Date: 2026-07-20
  • Deciders: lusoris
  • Related: issue #2559 (P1-RAG-01), PR #2769, PR #2772, PR #2774, PR #2790, ADR-0042

Context

P1-RAG-01's acceptance runs a paired-fusion evaluation: a baseline arm (reciprocal_rank_fusion_k60_int64) against a challenger arm (bounded_lane_champion_rank_sum_int64). The evaluator seals a signed receipt only when state == "passed", which requires challenger_gain — the challenger must not regress and must meet the minimum gain (multidomain_evaluator.go, MultidomainEvaluationGates.ChallengerGain).

The operator-ratified freeze v2 was evaluated and failed that gate: all six absolute per-class threshold cells passed on both arms, but the challenger regressed on the kubernetes class (MRR 1.0→0.833) after policy re-admission rotated the document IDs. No receipt was sealed. Because a receipt cannot be signed unless the challenger wins, and the accepted signed output of P1-RAG-01 is the input P1-MCP-01 is blocked on, the whole downstream chain stalled.

The question was whether the challenger could win on a better case set, or is structurally worse. TestMineSeparatingCaseShapes (PR #2790) answers it against the committed corpus, not by argument:

  • 166 of 792 (query, hypothetical-gold) pairs separate the two arms (21%).
  • Of the 83 distinct separating shapes, only 2 straddle the metric cutoff k=10 — only those can move recall@10 / MRR@10 / nDCG@10. The other 81 flip ordering below rank 10, where the metrics are blind, so authoring them would reproduce the tie.
  • Those 2 fold to one distinct contest, kubernetes-02-typemeta, where the champion ranks the gold at #10 (inside the cutoff) and the baseline at #11 (outside) — the champion wins it by a single rank.
  • 0 of the suite's 11 real golds appear in the separating set, consistent with the committed receipt's exact tie.

So the champion's entire metric-moving advantage over baseline across the whole corpus is one knife-edge contest won by one rank. Everywhere else it ties or regresses. RRF's reciprocal 1/(k+r) dampens a poor lane; the champion's rank-sum accumulates it linearly, so a single weak lane sinks an otherwise strong hit. Baseline is the robust profile.

Decision

The framing is deliberately "ship the robust default now", not "the champion is permanently worse". Deciding a fusion profile on 11 hand-authored cases is premature optimisation: the metric-moving evidence is a single one-rank contest, which is noise at this scale. The real profile signal comes from live retrieval at data volume, not a frozen micro-suite. So:

  1. Adopt baseline RRF-k60 as the production fusion profile now, and retire bounded_lane_champion_rank_sum as an adoption candidate for this suite. The point is to get P1-RAG-01 signed and the retrieval path live and ingesting real corpus — a profile bake-off can be re-run on real data later, when there is enough of it to separate the arms meaningfully. This ADR does not claim the champion is forever inferior; it claims baseline is the sane default to ship, and 11 cases cannot justify adopting anything else.
  2. Make "baseline adopted, challenger rejected" a signable acceptance outcome. The paired-fusion evaluation still runs both arms and records the full comparison, but a fairly-evaluated challenger that does not beat baseline now yields a terminal baseline_adopted state that seals a receipt — instead of an unsignable failed. The receipt records, truthfully, that the challenger was evaluated and did not win, and that baseline is adopted.

This is not a weakening of the gate. The existing passed path is unchanged: a challenger that genuinely wins (challenger_gain true) still seals a passed receipt and remains the promotion path. baseline_adopted requires all absolute bars — held_out_quality, citation_lineage, deterministic_replay — to pass exactly as before; it only reinterprets a non-winning-but-fully-valid run as "baseline retained" rather than a failure, which is the honest reading of a bake-off the incumbent won. promotable and execution_authorized stay false; adoption of baseline is not promotion of the challenger.

Alternatives considered

  • Author freeze v4 on the one separating contest. Technically possible — the champion legitimately wins kubernetes-02-typemeta. Rejected: an acceptance basis resting on a single one-rank win proves little beyond that case and is fragile. The mining evidence shows the champion is not broadly better.
  • Set the challenger arm equal to baseline for an honest tie. Rejected: forbidden by the validators — both evaluation_freeze.go and multidomain_evaluation.go hard-pin the challenger profile to the champion, so a same-profile freeze returns freeze comparison target unsupported.
  • Leave it failed and keep retrying. Rejected: the champion cannot win the current suite, and v3's results are observed, so no amount of re-running seals a receipt. The chain stays blocked.

Consequences

  • P1-MCP-01 unblocks on a baseline_adopted signed receipt from P1-RAG-01.
  • The evaluator gains a baseline_adopted terminal state and its seal condition; the change ships with mutation tests proving neither the new nor the existing seal path can be satisfied vacuously.
  • The receipt is signed with the sealed evaluator-v2 key already in the trust root (evaluator-allowed-signers, PR #2771); v1 is retained so prior receipts keep verifying.
  • The paired-fusion framework is preserved: a future, genuinely-better challenger can still be proposed and, if it wins, sealed as passed.

References

  • tools/knowledge-retrieval/multidomain_evaluator.go — the gate and state.
  • tools/knowledge-retrieval/fusion_discrimination_test.go TestMineSeparatingCaseShapes — the corpus evidence.
  • .workingdir2/cluster-2/audits/p1-rag-01-freeze-v2-acceptance-2026-07-20.md — the failed run.