Repository navigation
Injection quality: run a fresh calibrated eval — committed results predate all 14 quality features #95
Description
Split out of #54 (Injection quality Tier 3), which bundled three items of very different cost and
ownership behind one checkbox. This is the one that is runnable now, and it is the decision input
for the other two.
The problem
Current retrieval quality is unknown. The committed results in eval/results/ are the March run:
baseline-study-20260302-163649.log,ablation-study-20260302-170130.logablation-report-final.md— "85 completed results across 13 tasks"
The only commit touching eval/results/ since is 0ea99cd (2026-07-23), which was release hygiene —
untracking committed-but-gitignored data files — not a new run. So every committed number
predates the 14 shipped quality features (FileCategory/classify, doc→source BridgeMode, gate
threshold, session dedup, hot topics, sectioned output, query preprocessing, recency boosting, doc
demotion, configurable weights, bobbin calibrate + auto-calibration guard).
The work
Run the baseline study plus ablations with calibrated configs against today's main, and commit the
results with the commit SHA they were produced from.
Why first
The other two Tier 3 items — embedding-model upgrade (#54 item 1) and chunk-boundary refinement
(#54 item 2) — are both expensive and both justified only by numbers nobody currently has.
Committing to a model migration or a chunker redesign against unknown quality is choosing blind.
This issue is the cheapest of the three and gates the value of both.
Note on the original deferral
#54 deferred this item because "heavy eval loads are constrained in the current build environment".
That constraint no longer applies — builds, tests, CI runs and benchmarks are available again as of
2026-09-15 — so the reason this sat unrun has expired.
Activity
scbrown commentedon Sep 16, 2026
Both harness blockers under this issue are now FIXED and merged
Attempting this eval surfaced two defects that would have made a fresh study worthless. Both are on
main:5fd5d41~/.claude/settings.jsonaside host-wide for every agent run16300d3passed = exit_code == 0, and cargo output parsed as pytest)The second one matters directly here: any study run before
16300d3produced a pass-rate thatcould not fail on a task whose test filter missed. That includes the archived results this issue
exists to replace, so they should be re-checked against their stored outputs before being used as a
comparison baseline.
Measured inputs for the scale decision
A partial run was made before it was stopped — 2 runs of the 8 planned, task
ruff-001, one attemptper arm:
Both produced a real, semantically-correct fix. Notes on reading these:
16300d3scorer, i.e.they are the vacuous signal itself. Do not cite "2/2 passed" from it.
costs more because it enlarges the prompt is backwards: every turn resends accumulated history, so
turn count dominates prompt size, and injection buys fewer exploration turns than it adds context.
For reference, the full study as scripted is 96 agent runs — 12 no-bobbin + 12 with-bobbin + 72
ablations (4 tasks × 6 variants × 3 attempts), each with a
$5.00per-run cap.One thing to check before running it
run-baseline-study.shpasses--force-budget, which disables the harness's own pre-flight check.That check is a token-expiry guard (it aborts only if the OAuth token has under an hour left) —
it is not a spend cap, and nothing in the harness caps spend. Worth knowing before choosing a
scale, since the flag's name suggests otherwise.
Comparability caveat
flask-001..005are now ineval/tasks/_quarantined/, and the archivedablation-report-final.mdlists flask tasks among its data. Old-vs-new comparisons must be restricted to the non-quarantined
set.
scbrown commentedon Sep 16, 2026
Adding a concrete instance to the archived-baseline caveat already noted here, because it is more
specific than "needs re-checking" and it names the artifact a reader would actually cite.
eval/results/ablation-report-final.md(committed 2026-03-02) carries aPass%column, and itstargets include cargo-built ones —
ruffis the most-referenced target in the file and is itself aRust project, with
cargoappearing throughout. That report predates the zero-tests fix by roughlysix months.
The issue is not that those numbers are known wrong. It is that nothing in the report
distinguishes a real pass from a vacuous one, because the signal that produced the column was
later found unable to fail for runs whose output parsed as a suite that never executed. A file named
-finalwith aPass%column is precisely the artifact that gets cited later without anyonere-deriving it.
Same applies to the other committed results carrying cargo/Rust targets:
baseline-study-20260302-163649.log,ablation-study-20260302-170130.log,interim-ablation-report.md,fresh-report.md.Suggested handling, in preference order:
is a different study, not a validation of the old one.
was produced by a signal since shown unable to fail. That is cheap, survives, and reaches the
reader who finds the file rather than this thread.
by the same defect, so blanket-invalidating the whole report would discard sound numbers.
Flagging the asymmetry that makes this worth doing rather than noting: an over-cautious annotation
costs a line of text, while an uncaveated
Pass%in a file namedfinalcosts whoever builds on it.scbrown commentedon Oct 12, 2026
Current-source triage at
main94bfc9353d9d5f7a256e673c489aa691c3155b6a: the next evaluation path is now the v2 harness rather than rerunning the archived v1 study unchanged.runner.cli l0runs offline retrieval with no agent or LLM calls. It invokes the production injection hook, supports all ablation/baseline arms, and writes per-arm JSONL results.BOBBIN_*settings and uses an empty XDG configuration so host configuration cannot redirect retrieval to a shared server.The concrete next step is a frozen current-source L0 run and ablations with task/config/binary provenance, followed by the preregistered analysis. Preserve the earlier warning about archived v1
Pass%: a rerun is a new study, not validation of the old scorer's verdicts. This source check provides no new evaluation result and does not clear the evidence gate for #96 or #97.scbrown commentedon Oct 13, 2026
A bounded v2 L0 run is complete for
go-001: one task, 11 arms, 300-line budget, not the full 40-task study. This used the production injection path with no agent or LLM calls and isolated retrieval configuration.Provenance: harness commit
94bfc9353d9d5f7a256e673c489aa691c3155b6a, Bobbin 0.27.3, and the parent of Go fixing commit6aef900af4eeb4a35d3ff1237a41cf3f63c56c24.All 11 arms produced context; none returned an error or skip. Gold coverage for this task is one file and one pre-image line,
src/runtime/metrics/sample.go:45.A scoring positive control covering the exact gold span returns file recall, hunk recall and density of 1. That is an instrument control, not a retrieval result.
Both index commands completed successfully, but the last progress count was 10164/10165 files. Complete file coverage is therefore not claimed. The guarded run reported no changes to the watched user configuration or process resources.
This is one task's observed miss, not a general quality comparison or a completed study. The remaining preregistered task matrix is still needed before drawing aggregate conclusions. The earlier caveat about archived v1 Pass% remains unchanged.
docs(eval): retain capped L0 result for #95 (#193)
scbrown commentedon Oct 17, 2026
The bounded fresh L0 run and its exact partial results are now published in PR #193, merged as
e48be5bc7eafa9e66fa284ac891ffa5e67c2e234.The run hit its six-hour hard deadline (exit 124). It produced 330 of 1,320 planned cells across 10 of 40 tasks: 213 injected outcomes, 83 skips, and 34 errors. The report, sanitized rows, and frozen provenance are committed. These partial measurements do not establish full-cohort retrieval quality, and this issue remains open.
No second campaign or silent continuation has been started. The next decision is whether a separately bounded indexing/reuse redesign can complete the preregistered matrix without increasing the resource/time envelope. Model and chunk-boundary decisions in #96/#97 remain gated on usable complete evidence; this partial run does not justify either change.
scbrown commentedon Oct 21, 2026
2026-10-21 owner disposition: retain open; repair the measurement contract before another campaign
Re-read the committed partial report at current main. It remains 330/1,320 cells across 10/40 tasks, with 213 injected, 83 skipped and 34 errors, stopped at the six-hour cap. This is not a complete calibrated baseline and does not release #96/#97.
The concrete next source work is to discriminate the 34
no parseable chunk headerserrors and reconcile the 23 dedup-worded skips against the preregistered--no-dedupcontract, with positive/negative fixtures. Then prepare bounded indexing/reuse that preserves exact task, arm, source, model and configuration provenance within the existing resource envelope. Do not silently enlarge the cap or resume the old output stream.Owner: Gennaro. Resume the evaluation step only when those source/fixture contracts and the bounded reuse plan are actually delivered; freeze their identities before a separately bounded run. No new campaign, agent/model call, GPU work or shared index change was started by this triage. The historical v1
Pass%caveat remains; a new run cannot retroactively validate those old scorer verdicts. Existing partial results remain intact and this issue stays open.scbrown commentedon Oct 23, 2026
2026-10-23 triage — owner: Gennaro. Keep open; the next deliverable is measurement-contract repair, not another evaluation campaign.
New source delivery since the October 21 disposition: #204 merged as
3d9073b41ca6dffc15583852a74911f20396d29a. Its identity-sealed opt-in scorer qualifies native workspace paths and refuses ambiguous, linked or unreadable inputs without numeric recall. That is a source/instrument repair; it neither rescored the historical rows nor produced new retrieval-quality evidence.At frozen current main
fddcdb7b1bdf885528dbf0b432958125afe9c29d, the committed report remains 330/1,320 cells across 10/40 tasks: 213 injected, 83 skipped, 34 errors. Next I will reconcile the 34 header errors and 23 dedup-worded skips against the production-output and preregistered no-dedup contracts, accounting for #204's opt-in scoring convention. Deliverable: paired positive/refusal fixtures and an explicit disposition for each outcome family, then a bounded indexing/reuse plan with frozen task/arm/config/model/scoring identities. Preserve original rows; a changed scoring convention requires separate output.A further campaign starts only after those contracts and its bounded execution envelope are accepted. This triage starts no build, indexing, agent/model call or GPU run, raises no cap, and does not release #96/#97. Historical v1 Pass% remains separately caveated.
scbrown commentedon Oct 24, 2026
2026-10-24 status — owner: Gennaro. The measurement-contract repair has started and its source changes have landed; a new calibrated campaign has not started.
Since #204 (merge
3d9073b4, qualified scoring paths and unreadable-input refusal), #210 merged ase010faafon October 23. Its opt-inindependent-hook-outcomes/v1contract treats contamination, contradictory/unsupported output and unexplained silence as errors with null efficacy metrics and quarantine, rather than skips or numeric zero recall. Valid headers and explicit operational/gate skips retain their meanings. Validation recorded 623 evaluation tests and 24 focused contract controls, including inverse controls.The reconciliation accounts for all 330 historical cells: 213 injected, 34 header errors, 23 dedup-worded skips, 42 operational skips and 18 gate skips. The scalar archive lacks the stdout needed to determine the exact cause of the 34 header errors; that cause remains unknown. Original rows and legacy scoring remain unchanged. No new retrieval-quality conclusion follows, and #96/#97 remain gated.
Next: prepare the bounded-run readiness packet: enroll an exact current native runtime in sealed reuse (it currently admits exact 0.28.0 bytes), measure aggregate storage, and obtain acceptance of the existing six-hour/4-CPU/6-GiB/no-swap/no-GPU envelope. The proposed 32-GiB storage ceiling is neither accepted nor proven sufficient. ETA: readiness disposition by October 25, 18:00 UTC; campaign completion has no defensible ETA until those gates are resolved. No cap increase or silent continuation of the old campaign.
scbrown commentedon Nov 2, 2026
Owner: Gennaro. Next deliverable: the bounded-run readiness packet; a new campaign has not started.
#210 is merged (
e010faaf), completing the independent hook-outcome measurement-contract repair. Current installed Bobbin is 0.31.0 (c2042121); its exact binary is not enrolled in the sealed-reuse admission table at sourcece58f01e. That table still admits only the previously qualified runtime. Enrolling a new binary requires paired native cold/derived and per-cell state-isolation controls; a version label alone is insufficient.Storage and execution acceptance also remain open. The retained 6.47 GB inventory covers 21 built/partial indexes, not the aggregate workspace, derived copies, journals, quarantine and outputs for the full matrix. The proposed 32 GiB ceiling is neither accepted nor proven sufficient. The original six-hour / four-CPU / 6 GiB / no-swap / no-GPU envelope remains unchanged.
This waits on exact-runtime qualification, bounded aggregate-storage proof and acceptance of a newly frozen execution packet. Those are the next steps before any separately authorized run with separate outputs; no silent continuation or cap increase. Campaign completion has no defensible ETA yet. The original partial results remain intact, and #96/#97 stay gated on usable #95 evidence. No new campaign, model/provider call, GPU work or shared-index change was started.
scbrown commentedon Nov 3, 2026
Owner: Gennaro. Readiness-packet update: the exact-runtime reuse qualification is now delivered; the full calibrated campaign remains held.
#226 merged as
2502bb686cb49f3b932f171974bbbd7f8e536e78. It enrolls the qualified native 0.31 profile after paired cold/derived equivalence and per-cell isolation controls, with pinned ONNX Runtime 1.24.1 and a single loaded runtime library. This resolves the earlier source-enrollment gate for that exact profile; a version label or later installed binary is not automatically qualified.This is source/runtime qualification, not a fresh retrieval result. The retained study remains 330/1,320 cells across 10/40 tasks. The earlier #204/#210 measurement-contract repairs and the archived-result caveats still apply; #96/#97 remain gated.
Remaining readiness work: measure aggregate storage for the full task matrix, including source workspaces, cold/derived indexes, journals, quarantine, models and outputs; then obtain acceptance of a newly frozen execution packet. The historical 6.47 GB inventory covers only 21 built/partial indexes. The proposed 32 GiB ceiling is neither accepted nor proven sufficient. The existing six-hour / four-CPU / 6 GiB / no-swap / no-GPU envelope is unchanged.
Next deliverable is that storage/allocation and frozen-identity packet, not a campaign start. No model/provider call, new index build, GPU work, shared-store change or continuation of the old output stream is authorized by this update. Campaign completion has no defensible ETA until the remaining gates are accepted.
scbrown commentedon Nov 4, 2026
Owner status: the calibrated evaluation campaign remains held. The exact native 0.31 reuse qualification was delivered in #226; it is not a fresh quality result or qualification of later runtime versions. Retained coverage remains 330/1,320 cells across 10/40 tasks.
The remaining hold is unproven aggregate storage and an unaccepted frozen execution packet. The historical 6.47 GB inventory covers only 21 built/partial indexes, not the full workspaces, cold/derived indexes, journals, quarantine, models and outputs. The proposed 32 GiB ceiling is not proven sufficient or accepted.
Resume only after that aggregate-storage/allocation proof and the newly frozen exact-runtime execution packet are accepted within the existing six-hour / four-CPU / 6 GiB / no-swap / no-GPU envelope. No campaign, model/provider call, index build or shared-index change starts from this status update. #96/#97 remain gated on usable #95 results.