This repository was archived by the owner on Oct 31, 2026. It is now read-only.
Repository navigation
Run the ruling-2, v3 and level-2 drivers through the held-out referee - #13
Merged
Merged
Conversation
A driver names a JSON endpoint file and the SHA-256 of its canonical JSON; the loader refuses a file whose digest differs, names no key, and builds the seeded worker clients: one fixed seed in round robin for the deterministic policy, seed_base plus the repetition for the stochastic one. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
…eferee The referee gains decide_many (every first run in one pool, every confirming run in another), ruling 2 on a deterministic endpoint (the repeatability probe, the parent discordance stop and a receipt naming the endpoint configuration digest), stochastic policy v3 (the interleaved schedule, identity controls and admit_v3), and a lookup that rebuilds a stored candidate without returning its checks. The improver receives counts, pooled rates and signed decisions. Every per-case held-out table goes to a results file signed with the decision key, and each decision's verification bundle (receipt, reports, traces, harnesses, or a v3 cohort's execution records) is written as the decision is signed. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
ratchet.verify.results checks the results envelope's signature, then for each listed decision verifies the bundled receipt (re-grading every held-out run from the disclosed case documents) and recomputes the per-case table: for ruling 2 the outcome cells, stable recoveries, broken protected cases and C1 flips from the bundled reports; for v3 the per-panel counts from execution records that must digest to the signed arm digests, with each execution graded again from its trace. The repeatability probe is recomputed from its traces and the parent discordance from the bundled parent runs. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
Both drivers name the referee factory policy_parts and pass the endpoint configuration by file and digest. run_det runs the training cases and hands the referee the parent and candidate training runs; the referee rebuilds the candidate from its stored decision, probes repeatability, runs P1, P2, C1 and C2 and signs. run_v3 runs nothing itself: every v3 case is held out, so the whole schedule runs in the referee. Each driver writes counts, signed verdicts and the digest of the referee's signed results file, and no held-out case id. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
…eree The level-2 comparison no longer keeps its own referee. The bench holds training cases only and a referee; the parent's held-out runs, every candidate's held-out runs, replay over held-out traces, admission and signing happen in the referee, which receives each candidate's training run with its target cluster and returns counts. The arm report keeps the admission yield, gate failures and C1 flip counts; which held-out cases flipped is in the referee's signed results file. The worker is configured by an endpoint file the referee resolves by digest. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
The ruling-2, v3 and improver-instruction result pages each get a dated note: the runs on them predate the referee path and keep their numbers; on the referee path their per-case tables come from the results file the referee signs, checked with python -m ratchet.verify.results. RUNNING, VERIFY, ARCHITECTURE and the changelog describe the endpoint file, the policy methods and the results file. Signed-off-by: Sankalp Gilda <23521054+astrogilda@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routes the remaining in-process readers of held-out data through the held-out referee, so no driver outside the referee reads held-out cases, traces or grades.
decide_many,deterministic,stochasticandcandidate. The ruling-2, deterministic and stochastic v3 drivers now run inside it.referee_live:policy_parts), so a driver names the configuration by reference instead of holding it.run_detandrun_v3are routed through the referee.instruction_searchloses its private referee and uses the shared one, and its bench holds training cases only.python -m ratchet.verify.results <file> --case-suiteverifies that file against its bundles and recounts and regrades the parent and candidate executions from their traces.docs/results/r2det.md,docs/results/v3-live.mdanddocs/results/improver-instructions.mdcarry dated notes that cite the results file for their held-out tables. RUNNING, VERIFY, ARCHITECTURE and the changelog describe the new path.No historical cell is re-run here; the existing results keep their recorded numbers.
Edited 2026-10-17: the body previously held another pull request's description by mistake; replaced with this one.