From 56c8c5d9dae6e3dc30f42e76bf77971daea34150 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 22:15:12 -0700 Subject: [PATCH 1/8] Audit the gates themselves, not just the product A gate that has never once passed is more likely broken than the product is to be uniformly incapable of exactly that one thing. gate-health.ts reads past MSBench runs and reports, per gate, whether it has ever actually done its job. Ports the idea from #1669, whose input file does not exist here. MSBench records only passed/failed plus a nullable error, so the distinctions are reconstructed from eval.json, the exec table in session.sqlite, error.json and final-agent-config.json. Two differences from #1669 matter: - Void instances discard their passes as well as their failures. A rate-limited run that produced nothing still records a PASS for a negative assertion, because COUNT(*) = 0 is trivially true against an empty table. 7 of 26 instances in the corpus are void. - Not-applicable results never enter the pass numerator. N/A graders exit 0, so MSBench scores them as passes; an always-N/A gate would otherwise report 16-for-16 while testing nothing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 171 +++++++ evals/msbench/gate-health.ts | 942 +++++++++++++++++++++++++++++++++++ evals/package.json | 1 + 3 files changed, 1114 insertions(+) create mode 100644 evals/msbench/gate-health.ts diff --git a/evals/msbench/README.md b/evals/msbench/README.md index 2abb204e0..1f828ff62 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1061,6 +1061,177 @@ Two traps, both of which cost real time: msbench-cli run … --benchmark . . ``` +## Auditing the gates themselves + +[`gate-health.ts`](gate-health.ts) audits the **instrument** rather than the product. It +reads past runs and asks, per gate, whether that gate has ever actually done its job. + +```bash +export PATH="$HOME/.msbench-venv/bin:$PATH" +cd evals && npm run gate-health # every run in the local cache +npm run gate-health -- 2026082579322454 … # specific runs, extracted on demand +``` + +The motivating case comes from the sibling suite in #1669: its `worker` gate recorded +**16 failures and zero passes across every run ever executed** before anyone noticed the +storage probe was signing its Azurite requests with a corrupted account key. Azurite +answered 403 to everything, so no generated app could have passed regardless of quality. +Ten percent of the corpus was being charged for a harness defect. *A gate that has never +once passed is far more likely to be broken than the product is to be uniformly incapable +of exactly that one thing.* + +| Verdict | What it suggests | What to do | +| --- | --- | --- | +| `never-passed` | The gate may be impossible to satisfy — broken probe, wrong credential, bad fixture | Re-grade the named run and read the grader's own stderr | +| `never-failed` | The gate may be vacuous; it has never discriminated | Check it can go red at all; certification is the cheap way | +| `always-not-applicable` | The gate has never rendered a verdict — **and MSBench scored every one as a pass** | Group by `reason=`; usually one missing prerequisite | +| `never-attempted` | The gate never got the chance to run | Fix what is upstream; the gate is not the problem | +| `healthy` | Has both passed and failed | Nothing | + +**None of these prove a defect.** Each is a reason to look before quoting a score. + +### How many runs before a verdict means anything + +Verdicts resting on fewer than `--min-runs` runs (default **3**) are printed but marked +`(low confidence)` and never fail the process. This matters more than it sounds: on the +current corpus **25 of 30 gates are `never-failed`**, which is what a young suite looks +like, not a broken one — most gates have one to fourteen observations. The number to watch +is whether that ratio survives corpus growth, not its value today. + +Only a `never-passed` gate with at least `--min-runs` runs sets exit **1**. + +### The four inputs, and what had to be inferred + +#1669 read a `cor-validation.json` carrying a per-gate `status` and an explicit +`notAttempted` flag. **No such file exists here.** MSBench records only `passed: true | +false` plus a nullable `error`, so every distinction is reconstructed from four artifacts +of an extraction: + +| Input | Gives | +| --- | --- | +| `vsc-output/eval.json` → `details[]` | the verdicts; the only gate name MSBench carries is the assertion comment | +| `vsc-output/session.sqlite` → `exec` table | exit **1** vs exit **3** offline, and the N/A marker on stderr | +| `output/error.json` → `type` | the instance was **void** — see below | +| `vsc-output/configs/final-agent-config.json` | declared assertions, so a run with **no `eval.json` at all** still names the gates that never ran | + +The `assertions` table in `session.sqlite` is always empty; `eval.json` is the authority. + +### Void instances corrupt the tally in both directions + +#1669's cascade is per-gate, matched on the prose of a failure reason. Ours is structured +and coarser: `error.json` marks a whole instance void. Every verdict in a void instance is +discarded — **including the passes**, which is the part #1669 does not model. + +That is not theoretical. Two runs in the current corpus: + +| Run | Fault | Recorded | What actually happened | +| --- | --- | --- | --- | +| [`2026082583236973`](https://msbenchapp.azurewebsites.net/run-analysis/2026082583236973) | `RATE_LIMIT` | 1/4 — including a **pass** for `Agent should not fall back to the chat question tool` | The agent produced **literally nothing**. A `COUNT(*) = 0` assertion is trivially true against an empty table. | +| [`2026082467189297`](https://msbenchapp.azurewebsites.net/run-analysis/2026082467189297) | `X_EXTENSION_ACTIVATION_ERROR` | 4/7 | **All four "passes" are the negative assertions**; all three "failures" are the extension never activating. | + +So a naive pass rate over these manufactures failures the product never earned *and* +credits passes it never earned. **Seven of twenty-six instances in the corpus are void.** + +> **Any run predating #1706 may contain vacuous passes.** The liveness sentinel added +> there fails such runs outright, but only going forward. Anyone re-grading or +> trend-plotting historical runs should assume the older half of the corpus is +> contaminated in both directions. + +### This report is the safety mechanism for the not-applicable convention + +The fidelity and runtime gates emit a machine-readable marker on stderr: + +``` +NOT_APPLICABLE gate= reason= detail="…" +``` + +**and exit 0.** Because `assertZeroExitCode` compiles to `SELECT COUNT(*) > 0 FROM exec +WHERE exitCode = 0 …`, MSBench scores every N/A as a **pass**. A gate that is N/A across +the whole corpus therefore reports **16-for-16** — #1669's defect with the sign flipped, +and the inverted form is worse, because 0-for-16 looks alarming while 16-for-16 looks like +success and nobody investigates a passing gate. This is live rather than hypothetical: the +five `runtime-*` gates emit `functionsHostUnavailable` on *every* current stimulus, since +all four are Azure Functions and the container has no `func` binary. + +MSBench assertions are binary — there is no "neither" — so this cannot be fixed at the +assertion layer. **This report is the only place it can be caught**, which is why exit 0 +was only defensible on the assumption that this tool exists and behaves as follows. If you +are tempted to simplify any of it, this is what you would be breaking: + +1. **N/A is its own bucket**, alongside passed / failed / notAttempted. Never folded into + passed, never silently dropped. +2. **Every rate excludes N/A from both numerator and denominator.** A gate that ran 16 + times, was N/A 16 times and passed 0 real times has *no applicable observations* — the + `rate` column prints `n/a`, and `n/a` never means 100%. +3. **Always-N/A gates are grouped by `reason=`**, so one absent prerequisite reads as a + single actionable line rather than five mystery gates. + +Detection keys off the **marker, not the exit code**, so the tool survives the convention +changing again. + +### Gate identity, and a known limitation + +MSBench carries no gate id — `eval.json` identifies an assertion only by its comment. That +is unstable: `requirements.json should be valid JSON carrying a questions array` and +`requirements.json satisfies the requirements contract` are **the same gate** before and +after it moved from SQL to `exec:`, so under comment identity it appears as two gates with +7 and 3 runs rather than one with 10. **A gate can silently reset its own history by being +reworded.** + +The default `--identity gate` mitigates this by keying `exec:` gates on the grader's +filename — the same id `gate=` is derived from, and the same id the certification manifest +uses — which also recovers a stable identity for runs recorded *before* the convention +existed. Two consequences worth knowing: + +- It is deliberately **coarser**: every `validate-requirements.ts` invocation is one gate + regardless of its flags. Use `--identity comment` for the raw per-assertion view. +- It only helps `exec:` gates. The SQL assertions over `files` / `toolCalls` / + `llm_responses` have no stderr and no grader file, so they stay comment-keyed. + +### Where the data lives — and why this is not a laptop-only tool + +Worth stating plainly, because the opposite is easy to assume: + +- **Run *data* is remote.** `msbench-cli extract` is served by the backend — extracting an + unknown id reports `Requesting run metadata from remote service`. **Any run id you have + access to can be audited from any machine**, free and without tokens. The local + `~/Library/Application Support/msbench/runs` directory is a cache, not the source. +- **Run *discovery* is local-only today.** With no arguments the tool can only enumerate + this machine's cache. The CLI already supports `list runs --kusto --created_by + --lookback`, which would make discovery team-wide, but the `MSBench User` role does not + appear to grant Kusto DB read: + + ``` + Corp: Principal 'aaduser=…' is not authorized to read database 'ces_telemetry_prod' + AME: Principal 'aaduser=…' is not authorized to read database 'msbench' + ``` + + (`ces-westus3-adx.westus3` and `msbdikustoprodeus2.eastus2` respectively.) That is a + concrete, filable access gap and the entire fix for discovery. +- **Kusto could not answer this question even with access.** The ingested views — + `CESBenchmarkInstanceStatusV2View`, `CESBenchmarkRunStatusV2View`, + `CESBenchmarkMetricsDedupView`, `CESBenchmarkMetadataDedupView` — carry run and instance + status, timings, tags, agent, model and resolved rate. **Per-assertion `details[]` is + ingested nowhere.** Gate-level health is only computable from extracted artifacts. + +The tool is therefore **run-id-driven and indifferent to provenance**. The day Kusto read +lands, `msbench-cli list runs --kusto` piped into `npm run gate-health` works with no +change to the tool. In CI the ids are known by construction anyway. + +### Flags + +| Flag | | +| --- | --- | +| `--extracted ` | Audit an existing extraction; skips `msbench-cli` entirely. Repeatable. | +| `--min-runs ` | Runs required before a verdict counts as confident (default 3). | +| `--identity gate\|comment` | Gate identity scheme; see above. | +| `--refresh` | Re-extract even when the cache has the run. | +| `--json` | Machine-readable report, including the full declared-but-never-seen list. | + +Extractions are cached in `.regrade/`, shared with +[`regrade.ts`](#re-grading-a-past-run-for-free), so a run pulled by either tool is already +on disk for the other. + ## Running in CI [`.github/workflows/msbench-evals.yml`](../../.github/workflows/msbench-evals.yml) runs diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts new file mode 100644 index 000000000..751003cc3 --- /dev/null +++ b/evals/msbench/gate-health.ts @@ -0,0 +1,942 @@ +#!/usr/bin/env node +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the MIT License. See LICENSE.md in the project root for license information. + *--------------------------------------------------------------------------------------------*/ + +/** + * Audits the evaluation instrument rather than the product. + * + * A gate that has never once passed is far more likely to be broken than the product is to be + * uniformly incapable of exactly that one thing. In the suite this idea comes from, the `worker` + * gate recorded 16 failures and zero passes across every run ever executed before anyone noticed + * that the storage probe signed its Azurite requests with a corrupted account key — Azurite + * answered 403 to every request, so no generated app could have passed regardless of quality. + * Ten percent of the corpus was being charged for a harness defect. + * + * Four signals matter, and each maps to a distinct instrument failure: + * never-passed — the gate may be impossible to satisfy (broken probe, wrong credential, bad fixture) + * never-failed — the gate may be vacuous; it has never discriminated between good and bad output + * always-n/a — the gate is dead weight; no scenario has ever exercised it + * never-attempted — the gate never got the chance to run; this indicts everything upstream of it + * + * None of these are proof of a defect. All of them are reasons to go look before quoting a score. + * + * ## What is different here, and why + * + * The original reads `cor-validation.json` files emitted by an SDK-driven runner that recorded a + * per-gate `status` and a `notAttempted` flag. **No such file exists in this world.** MSBench + * records only `passed: true | false` plus a nullable `error`, so every distinction above has to be + * reconstructed from four inputs (see `readInstance`). Two consequences are worth stating up front. + * + * **1. Cascade is per-instance here, not per-gate — and it corrupts the tally in both directions.** + * + * The original matched cascade on the prose of a gate's failure reason. Ours is structured and + * coarser: `error.json` marks a whole instance void (`RATE_LIMIT`, `X_EXTENSION_ACTIVATION_ERROR`, + * `X_MODEL_NOT_FOUND_ERROR`, `X_ASSERTION_DOES_NOT_COMPILE`). Every verdict in a void instance is + * discarded — **including the passes**, which is the part the original does not model. + * + * That is not a theoretical refinement. Run `2026082583236973` was rate limited, the agent produced + * literally nothing, and the gate `Agent should not fall back to the chat question tool` is recorded + * as **passing** — because a negative assertion (`COUNT(*) = 0`) is trivially true against an empty + * table. Run `2026082467189297` died in extension activation and scored 4/7, where all four "passes" + * are the negative assertions and all three "failures" are just the extension never starting. A + * naive pass rate over those runs manufactures failures the product never earned *and* credits + * passes it never earned. Six of twenty-six instances in the current corpus are void. + * + * The liveness sentinel (PR #1706) prevents this going forward by failing such runs outright. It + * does nothing for the runs already recorded, which is why this tool still has to discard them. + * + * **2. Gate identity is the comment string, and it does not survive editing.** + * + * MSBench carries no gate id — `eval.json` identifies an assertion only by its human comment. So + * `requirements.json should be valid JSON carrying a questions array` and `requirements.json + * satisfies the requirements contract` are the same gate before and after it moved from a SQL + * assertion to an `exec:` grader, and are counted here as two gates with 7 and 3 runs rather than + * one with 10. **A gate can silently reset its own history by being reworded.** The fix is a stable + * `gate=` token on every verdict line (see `NOT_APPLICABLE_MARKER`); until stimuli are + * retrofitted, identity is best-effort and `--min-runs` is doing real work. + * + * Runs straight off source via Node's built-in type stripping — no build step. + */ + +import { spawnSync } from 'node:child_process'; +import { existsSync, readFileSync, readdirSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { DatabaseSync } from 'node:sqlite'; +import { fileURLToPath } from 'node:url'; + +const HERE = dirname(fileURLToPath(import.meta.url)); + +/** + * Shared with `regrade.ts` on purpose. Both tools want the same extraction of the same run, so a + * run pulled by either is already on disk for the other. Gitignored. + */ +const CACHE_ROOT = join(HERE, '.regrade'); + +/** Today's declared gates, used only to spot ones no run has ever exercised. Never written to. */ +const STIMULI_DIR = join(HERE, 'config', 'stimuli'); + +/** + * Mirrors `graderHarness.ts`. Exit 3 means the grader itself broke — it is not a verdict about the + * product, so it can neither pass nor fail a gate. + */ +const EXIT_GRADER_ERROR = 3; + +/** + * The structured not-applicable marker, agreed with the fidelity-gates and runtime-gates sessions: + * + * NOT_APPLICABLE gate= reason= detail="…" + * + * ## This detection is the safety mechanism for the whole not-applicable convention + * + * An N/A grader **exits 0**. MSBench's `assertZeroExitCode` compiles to + * `SELECT COUNT(*) > 0 FROM exec WHERE exitCode = 0 …`, so **MSBench scores every N/A as a pass.** + * A gate that is not applicable across the entire corpus therefore reports 16-for-16. + * + * That is the 0-for-16 defect in this file's header with the sign flipped, and the inverted form is + * worse: 0-for-16 looks alarming, 16-for-16 looks like success, and nobody investigates a passing + * gate. MSBench assertions are binary — there is no "neither" — so this cannot be fixed at the + * assertion layer. **This report is the only place it can be caught**, which is why exit 0 was only + * defensible on the assumption that this tool exists and behaves as documented below. + * + * The contract, which is deliberately stated as a contract and not a preference: + * + * 1. N/A is its own bucket, alongside passed / failed / notAttempted. Never folded into passed, + * never silently dropped. + * 2. Every rate excludes N/A from **both** numerator and denominator. A gate that ran 16 times, + * was N/A 16 times and passed 0 real times has no pass rate — it has *no applicable + * observations*. See `passRate`, which returns undefined rather than 100%. + * 3. Always-N/A gates are grouped by reason code, so one missing prerequisite reads as one + * actionable line rather than N alarming ones. + * + * Detection keys off **the marker, not the exit code**. That was the right call while the exit code + * was still being argued, and it stays right: it means this tool keeps working if the convention + * ever changes again. + */ +const NOT_APPLICABLE_MARKER = /^NOT_APPLICABLE\s+(?:gate=(\S+)\s+)?(?:reason=(\S+))?/mu; + +/** + * `gate=` on a `PASS:` / `FAIL:` / `NOT_APPLICABLE` line. Fidelity derives the id from the + * grader's filename, which is also the certification manifest's validator id — so it doubles as a + * join key to the grader-certification reports under `evals/results/grader-certification/`. + */ +const GATE_ID = /^(?:PASS|FAIL|NOT_APPLICABLE)\b[^\n]*?\bgate=(\S+)/mu; + +/** + * Below this many runs a verdict is a coincidence with a label on it. Verdicts are still printed, + * but marked low-confidence and never used to fail the process. + */ +const DEFAULT_MIN_RUNS = 3; + +type Verdict = 'never-passed' | 'never-failed' | 'always-not-applicable' | 'never-attempted' | 'healthy'; + +interface GateTally { + passed: number; + /** Failures where the gate actually ran and rendered a verdict about the product. */ + failed: number; + /** "Failures" that are really upstream cascade, a void instance, or a broken grader. */ + notAttempted: number; + notApplicable: number; + /** Distinct runs, and distinct instances — a 5-instance run is one run but five observations. */ + runs: Set; + instances: number; + exampleFailure?: string; + exampleFailureRun?: string; + /** Why the gate did not run, counted by cause, so the report groups rather than just totals. */ + notAttemptedReasons: Map; + /** Why the gate was inapplicable, counted by reason code. */ + notApplicableReasons: Map; +} + +interface GateRow { + gate: string; + tally: GateTally; + verdict: Verdict; + confident: boolean; +} + +interface Options { + runIds: string[]; + extractedDirs: string[]; + minRuns: number; + json: boolean; + refresh: boolean; + /** `false` keys gates by grader id where available; `true` keys by the raw assertion comment. */ + identityByComment: boolean; +} + +class GateHealthError extends Error { } + +let jsonMode = false; +function log(message: string): void { + if (jsonMode) { + console.error(message); + } else { + console.log(message); + } +} + +// --------------------------------------------------------------------------- +// Arguments +// --------------------------------------------------------------------------- + +const USAGE = `Audit the gates themselves across past MSBench runs. Costs zero tokens. + +Usage: + node gate-health.ts [run-id...] [options] + +With no run ids, every run in the local MSBench cache is audited. Run ids that are not +cached are fetched with \`msbench-cli extract\`, which is server-backed — so any run id +you have access to works from any machine, not just the one that submitted it. + +Options: + --extracted Audit an existing extraction directory. Repeatable. + --min-runs Runs required before a verdict is treated as confident (default ${DEFAULT_MIN_RUNS}). + --identity 'gate' (default) keys gates by grader id, so rewording an assertion does not + start a fresh history; 'comment' keys by the raw assertion comment. + --refresh Re-extract even when the cache already has the run. + --json Machine-readable report on stdout. + -h, --help This message. + +Exit codes: + 0 no confident never-passed gate + 1 at least one gate ran often enough to judge and never once passed +`; + +function parseArgs(argv: string[]): Options { + const options: Options = { + runIds: [], + extractedDirs: [], + minRuns: DEFAULT_MIN_RUNS, + json: false, + refresh: false, + identityByComment: false, + }; + + for (let index = 0; index < argv.length; index++) { + const arg = argv[index]; + switch (arg) { + case '-h': + case '--help': + console.log(USAGE); + process.exit(0); + break; + case '--json': + options.json = true; + break; + case '--refresh': + options.refresh = true; + break; + case '--identity': { + const value = argv[++index]; + if (value !== 'gate' && value !== 'comment') { + throw new GateHealthError("--identity takes 'gate' or 'comment'"); + } + options.identityByComment = value === 'comment'; + break; + } + case '--extracted': { + const value = argv[++index]; + if (!value) { + throw new GateHealthError('--extracted needs a directory'); + } + options.extractedDirs.push(resolve(value)); + break; + } + case '--min-runs': { + const value = Number(argv[++index]); + if (!Number.isInteger(value) || value < 1) { + throw new GateHealthError('--min-runs needs a positive integer'); + } + options.minRuns = value; + break; + } + default: + if (arg.startsWith('-')) { + throw new GateHealthError(`Unknown option ${arg}`); + } + options.runIds.push(arg); + } + } + return options; +} + +// --------------------------------------------------------------------------- +// Locating runs +// --------------------------------------------------------------------------- + +/** + * MSBench's own run cache. This is where run *discovery* is local-only: the CLI can list runs from + * Kusto (`list runs --kusto`), but that needs a Kusto read grant the `MSBench User` role does not + * include, so without explicit run ids all we can enumerate is this machine's history. The run + * *data* is not local — see `extractRun`. + */ +function localRunIds(): string[] { + const candidates = [ + process.env.MSBENCH_DATA_DIR ? join(process.env.MSBENCH_DATA_DIR, 'runs') : undefined, + join(homedir(), 'Library', 'Application Support', 'msbench', 'runs'), + join(homedir(), '.local', 'share', 'msbench', 'runs'), + process.env.APPDATA ? join(process.env.APPDATA, 'msbench', 'runs') : undefined, + ].filter((path): path is string => path !== undefined); + + for (const root of candidates) { + if (!existsSync(root)) { + continue; + } + const ids = readdirSync(root, { withFileTypes: true }) + .filter(entry => entry.isDirectory() && /^\d+$/u.test(entry.name)) + .map(entry => entry.name) + .sort(); + if (ids.length > 0) { + return ids; + } + } + return []; +} + +/** + * Extraction is served by the MSBench backend, not by the local cache — extracting an unknown id + * reports `Requesting run metadata from remote service`. A cached `results.zip` is unzipped without + * a network call, so re-auditing the same corpus stays fast and offline. + */ +function extractRun(runId: string, refresh: boolean): string | undefined { + const dir = join(CACHE_ROOT, runId); + if (!refresh && findInstances(dir).length > 0) { + return dir; + } + + const result = spawnSync('msbench-cli', ['extract', '--run_id', runId, '--output', dir], { + encoding: 'utf8', + }); + if (result.error && (result.error as NodeJS.ErrnoException).code === 'ENOENT') { + throw new GateHealthError( + 'msbench-cli is not on PATH. Run:\n' + + ' export PATH="$HOME/.msbench-venv/bin:$PATH"\n' + + 'Invoking it by absolute path breaks its plugin discovery, so it has to be on PATH.' + ); + } + if (result.status !== 0) { + log(` ! ${runId}: extract failed, skipping (${(result.stderr ?? '').trim().split('\n').pop() ?? 'no detail'})`); + return undefined; + } + return dir; +} + +interface Instance { + name: string; + vscOutput: string; + outputDir: string; +} + +/** One `-output/output/vsc-output` tree per instance, as `regrade.ts` finds them. */ +function findInstances(root: string): Instance[] { + if (!existsSync(root)) { + return []; + } + return readdirSync(root, { withFileTypes: true }) + .filter(entry => entry.isDirectory() && entry.name.endsWith('-output')) + .map(entry => ({ + name: entry.name.replace(/-output$/u, ''), + outputDir: join(root, entry.name, 'output'), + vscOutput: join(root, entry.name, 'output', 'vsc-output'), + })) + .sort((a, b) => a.name.localeCompare(b.name)); +} + +// --------------------------------------------------------------------------- +// Reading one instance +// --------------------------------------------------------------------------- + +function readJson(path: string): T | undefined { + if (!existsSync(path)) { + return undefined; + } + try { + return JSON.parse(readFileSync(path, 'utf8')) as T; + } catch { + return undefined; + } +} + +/** + * `error.json` is the structured cascade signal. Its presence means the instance is void: whatever + * `eval.json` says about it describes an agent that never got to run. + */ +function instanceFault(instance: Instance): string | undefined { + for (const path of [join(instance.outputDir, 'error.json'), join(instance.vscOutput, 'error.json')]) { + const parsed = readJson<{ type?: string }>(path); + if (parsed) { + return parsed.type ?? 'UNKNOWN_ERROR'; + } + if (existsSync(path)) { + return 'UNREADABLE_ERROR_JSON'; + } + } + return undefined; +} + +interface ExecRow { + exitCode: number; + stdErr: string; +} + +/** + * The `exec` table is where exit 1 and exit 3 stay distinct. MSBench collapses both into + * "non-zero", so without this a broken grader is indistinguishable from a product regression — + * and would be tallied as a real failure against the gate. + */ +function readExecRows(sqlitePath: string): Map { + const rows = new Map(); + if (!existsSync(sqlitePath)) { + return rows; + } + let db: DatabaseSync | undefined; + try { + db = new DatabaseSync(sqlitePath, { readOnly: true }); + for (const row of db.prepare('SELECT command, exitCode, stdErr FROM exec').all()) { + const { command, exitCode, stdErr } = row as { command: string; exitCode: number; stdErr: string }; + rows.set(command, { exitCode, stdErr: stdErr ?? '' }); + } + } catch { + // A missing or unreadable exec table just means no exec attribution for this instance. + } finally { + db?.close(); + } + return rows; +} + +/** `[AUTOGENERATED] Check 0 exit code for command: ''. Original comment: ` */ +const AUTOGENERATED_EXEC = /^\[AUTOGENERATED\] Check 0 exit code for command: '([\s\S]*)'\. Original comment: ([\s\S]*)$/u; + +/** The gate name a human wrote, recovered from the wrapper MSBench generates for `exec:`. */ +function shortComment(comment: string): string { + if (!comment.startsWith('[AUTOGENERATED]')) { + return comment; + } + const exec = comment.match(AUTOGENERATED_EXEC); + if (exec) { + return exec[2]; + } + const marker = '. Original comment: '; + const index = comment.lastIndexOf(marker); + return index === -1 ? comment : comment.slice(index + marker.length); +} + +function execCommandOf(comment: string): string | undefined { + return comment.match(AUTOGENERATED_EXEC)?.[1]; +} + +interface NotApplicable { + reason: string; +} + +function parseNotApplicable(stdErr: string): NotApplicable | undefined { + const match = stdErr.match(NOT_APPLICABLE_MARKER); + if (!match) { + return undefined; + } + return { reason: match[2] ?? 'unspecified' }; +} + +/** + * The gate's identity, best available. + * + * MSBench carries no gate id, so the fallback is the human comment — which is unstable, because a + * gate that gets reworded starts a fresh history under a new name (see this file's header). Two + * things improve on it: + * + * 1. An explicit `gate=` token on a verdict line, once graders emit it. + * 2. For an `exec:` gate, the grader's **filename**, which is what `gate=` is derived from. That + * recovers a stable identity for runs recorded *before* the convention existed, so the + * transition does not split every gate's history in two. + * + * Filename identity is deliberately coarser than the comment: every `validate-requirements.ts` + * invocation is one gate regardless of its flags, matching the certification manifest's validator + * id. Use `--identity comment` for the raw, per-assertion view. + */ +function graderFilename(command: string): string | undefined { + const match = command.match(/([\w.-]+)\.ts\b/u); + return match ? match[1].replace(/^validate-/u, '') : undefined; +} + +function gateIdentity(comment: string, stdErr: string | undefined, byComment: boolean): string { + const human = shortComment(comment); + if (byComment) { + return human; + } + const declared = stdErr?.match(GATE_ID)?.[1]; + if (declared) { + return declared; + } + const command = execCommandOf(comment); + const derived = command ? graderFilename(command) : undefined; + return derived ?? human; +} + +// --------------------------------------------------------------------------- +// Tallying +// --------------------------------------------------------------------------- + +function tallyOf(tallies: Map, gate: string): GateTally { + let tally = tallies.get(gate); + if (!tally) { + tally = { + passed: 0, + failed: 0, + notAttempted: 0, + notApplicable: 0, + runs: new Set(), + instances: 0, + notAttemptedReasons: new Map(), + notApplicableReasons: new Map(), + }; + tallies.set(gate, tally); + } + return tally; +} + +function bump(counter: Map, key: string): void { + counter.set(key, (counter.get(key) ?? 0) + 1); +} + +interface StoredEval { + resolved?: boolean; + details?: { comment: string; query?: string; passed: boolean; error: string | null }[]; +} + +interface AgentConfig { + promptSteps?: { assertions?: { comment?: string; query?: string; exec?: string; assertZeroExitCode?: boolean }[] }[]; +} + +/** Assertions the run declared, used when it produced no `eval.json` to name the gates that never ran. */ +function declaredGates(configPath: string, identityByComment: boolean): string[] { + const config = readJson(configPath); + if (!config) { + return []; + } + const gates: string[] = []; + for (const step of config.promptSteps ?? []) { + for (const assertion of step.assertions ?? []) { + // Non-asserting `exec:` entries generate no check, exactly as upstream drops them. + if (!assertion.comment || (assertion.exec !== undefined && assertion.assertZeroExitCode === false)) { + continue; + } + const derived = !identityByComment && assertion.exec ? graderFilename(assertion.exec) : undefined; + gates.push(derived ?? shortComment(assertion.comment)); + } + } + return gates; +} + +function analyzeInstance( + tallies: Map, + runId: string, + instance: Instance, + identityByComment: boolean, +): void { + const fault = instanceFault(instance); + const stored = readJson>(join(instance.vscOutput, 'eval.json')); + const instanceKey = stored ? Object.keys(stored)[0] : undefined; + const details = instanceKey ? stored?.[instanceKey]?.details ?? [] : undefined; + + // No eval.json at all: the run rendered no verdicts, so name the gates from the config it was + // given. Skipping the instance instead would hide the very gates that never got to run. + if (details === undefined) { + const configPath = join(instance.vscOutput, 'configs', 'final-agent-config.json'); + for (const gate of declaredGates(configPath, identityByComment)) { + const tally = tallyOf(tallies, gate); + tally.runs.add(runId); + tally.instances++; + tally.notAttempted++; + bump(tally.notAttemptedReasons, fault ?? 'NO_EVAL_JSON'); + } + return; + } + + const execRows = readExecRows(join(instance.vscOutput, 'session.sqlite')); + + for (const detail of details) { + const command = execCommandOf(detail.comment); + const exec = command ? execRows.get(command) : undefined; + const gate = gateIdentity(detail.comment, exec?.stdErr, identityByComment); + const tally = tallyOf(tallies, gate); + tally.runs.add(runId); + tally.instances++; + + const notApplicable = exec ? parseNotApplicable(exec.stdErr) : undefined; + + // Order matters. A void instance discards everything, including passes: the agent produced + // nothing, so a negative assertion passed trivially rather than meaningfully. + if (fault) { + tally.notAttempted++; + bump(tally.notAttemptedReasons, fault); + } else if (notApplicable) { + // Never `passed`, even though the grader exited 0 and MSBench scored it as a pass. + // This branch is the entire safety mechanism for the exit-0 convention. + tally.notApplicable++; + bump(tally.notApplicableReasons, notApplicable.reason); + } else if (detail.error) { + // The assertion never compiled or returned a non-boolean. `passed: false` would launder + // a harness fault into a product verdict. + tally.notAttempted++; + bump(tally.notAttemptedReasons, 'ASSERTION_ERROR'); + } else if (exec?.exitCode === EXIT_GRADER_ERROR) { + tally.notAttempted++; + bump(tally.notAttemptedReasons, 'GRADER_EXIT_3'); + } else if (detail.passed) { + tally.passed++; + } else { + tally.failed++; + if (!tally.exampleFailure) { + const evidence = (exec?.stdErr ?? '').split('\n').map(line => line.trim()).find(Boolean); + tally.exampleFailure = (evidence ?? detail.query ?? '').slice(0, 140); + tally.exampleFailureRun = runId; + } + } + } +} + +function classify(tally: GateTally): Verdict { + const rendered = tally.passed + tally.failed; + if (rendered === 0 && tally.notApplicable > 0 && tally.notAttempted === 0) { + return 'always-not-applicable'; + } + if (rendered === 0) { + // Ran nowhere versus ran and never succeeded: the first indicts everything upstream, the + // second indicts the gate. Collapsing them is what manufactures phantom failures. + return 'never-attempted'; + } + if (tally.passed === 0) { + return 'never-passed'; + } + if (tally.failed === 0) { + return 'never-failed'; + } + return 'healthy'; +} + +/** + * Pass rate over **applicable** observations only. Not-applicable and never-attempted results are + * excluded from the numerator *and* the denominator, so a gate with nothing to judge returns + * `undefined` — rendered as "n/a", never as 100%. This is contract point 2 in + * `NOT_APPLICABLE_MARKER`; folding N/A into the numerator is exactly the bug this tool exists to + * catch, and it would be reintroduced here if anywhere. + */ +function passRate(tally: GateTally): number | undefined { + const applicable = tally.passed + tally.failed; + return applicable === 0 ? undefined : tally.passed / applicable; +} + +// --------------------------------------------------------------------------- +// Reporting +// --------------------------------------------------------------------------- + +function topReasons(counter: Map, limit = 3): string { + return [...counter.entries()] + .sort((a, b) => b[1] - a[1]) + .slice(0, limit) + .map(([reason, count]) => `${reason}\u00d7${count}`) + .join(', '); +} + +/** Gates declared in today's stimuli. A gate here that no run mentions has never been exercised. */ +async function declaredToday(identityByComment: boolean): Promise> { + const declared = new Map(); + if (!existsSync(STIMULI_DIR)) { + return declared; + } + let parse: (source: string) => unknown; + try { + ({ parse } = await import('yaml')); + } catch { + return declared; + } + for (const file of readdirSync(STIMULI_DIR).filter(name => /\.ya?ml$/u.test(name))) { + let config: AgentConfig; + try { + config = parse(readFileSync(join(STIMULI_DIR, file), 'utf8')) as AgentConfig; + } catch { + continue; + } + for (const step of config?.promptSteps ?? []) { + for (const assertion of step.assertions ?? []) { + if (!assertion.comment || (assertion.exec !== undefined && assertion.assertZeroExitCode === false)) { + continue; + } + const derived = !identityByComment && assertion.exec ? graderFilename(assertion.exec) : undefined; + const gate = derived ?? shortComment(assertion.comment); + declared.set(gate, [...(declared.get(gate) ?? []), file.replace(/\.ya?ml$/u, '')]); + } + } + } + return declared; +} + +function printTable(rows: GateRow[]): void { + const width = Math.min(Math.max(...rows.map(row => row.gate.length)) + 2, 70); + console.log(''); + console.log( + `${'GATE'.padEnd(width)}${'pass'.padStart(6)}${'fail'.padStart(6)}` + + `${'n/att'.padStart(7)}${'n/a'.padStart(6)}${'runs'.padStart(6)}${'rate'.padStart(7)} VERDICT` + ); + console.log('-'.repeat(width + 38)); + for (const { gate, tally, verdict, confident } of rows) { + const name = gate.length > width - 2 ? `${gate.slice(0, width - 5)}...` : gate; + const rate = passRate(tally); + console.log( + `${name.padEnd(width)}${String(tally.passed).padStart(6)}${String(tally.failed).padStart(6)}` + + `${String(tally.notAttempted).padStart(7)}${String(tally.notApplicable).padStart(6)}` + + `${String(tally.runs.size).padStart(6)}` + + // "n/a" rather than 100%: a gate with no applicable observations has no pass rate. + `${(rate === undefined ? 'n/a' : `${Math.round(rate * 100)}%`).padStart(7)}` + + ` ${verdict}${confident ? '' : ' (low confidence)'}` + ); + } + console.log(''); + console.log('rate = passes over *applicable* observations. Not-applicable and never-attempted'); + console.log('results are excluded from both sides of it, so "n/a" means nothing was judged —'); + console.log('it never means 100%.'); +} + +function printFindings(rows: GateRow[], minRuns: number, unexercised: Map): number { + const of = (verdict: Verdict): GateRow[] => rows.filter(row => row.verdict === verdict); + const suspect = of('never-passed'); + const starved = of('never-attempted'); + const dead = of('always-not-applicable'); + const vacuous = of('never-failed'); + + console.log(''); + console.log('='.repeat(78)); + console.log('WHAT TO GO AND LOOK AT'); + console.log('='.repeat(78)); + + let actionable = 0; + + if (suspect.length > 0) { + actionable++; + console.log(''); + console.log('NEVER PASSED — a gate that ran and never once succeeded is more likely broken than'); + console.log('the product is to be uniformly incapable of exactly that one thing.'); + for (const { gate, tally, confident } of suspect) { + console.log(` * ${gate}`); + console.log(` ${tally.failed} failure(s), 0 passes across ${tally.runs.size} run(s)${confident ? '' : ' — too few runs to judge yet'}`); + if (tally.exampleFailure) { + console.log(` e.g. ${tally.exampleFailure}`); + console.log(` reproduce: npm run regrade -- ${tally.exampleFailureRun}`); + } + } + } + + if (starved.length > 0) { + actionable++; + console.log(''); + console.log('NEVER ATTEMPTED — these never got the chance to run. This indicts whatever is'); + console.log('upstream of them, not the gates, and it says nothing at all about the product.'); + for (const { gate, tally } of starved) { + console.log(` * ${gate}`); + console.log(` ${tally.notAttempted} blocked observation(s) over ${tally.runs.size} run(s): ${topReasons(tally.notAttemptedReasons)}`); + } + console.log(' Fix the upstream cause; these gates cannot report anything until you do.'); + } + + if (dead.length > 0) { + actionable++; + console.log(''); + console.log('ALWAYS NOT-APPLICABLE — these have never rendered a verdict about the product.'); + console.log('MSBench scored every one of them as a PASS, because an N/A grader exits 0. Left to'); + console.log('the raw numbers each of these gates looks perfect while testing nothing, so this'); + console.log('section is the only thing standing between a coverage hole and a green dashboard.'); + // Grouped by reason so a single missing prerequisite reads as one line, not N mystery gates. + const byReason = new Map(); + for (const { gate, tally } of dead) { + const [reason] = [...tally.notApplicableReasons.entries()].sort((a, b) => b[1] - a[1])[0] ?? ['unspecified']; + byReason.set(reason, [...(byReason.get(reason) ?? []), gate]); + } + for (const [reason, gates] of [...byReason.entries()].sort((a, b) => b[1].length - a[1].length)) { + console.log(''); + console.log(` * reason=${reason} — ${gates.length} gate(s), 0 applicable observations`); + console.log(` ${gates.join(', ')}`); + console.log(' Either a stimulus that exercises this is missing, or a prerequisite is'); + console.log(` absent from the run environment. One cause, ${gates.length} gate(s) to recover.`); + } + } + + if (unexercised.size > 0) { + actionable++; + console.log(''); + console.log('DECLARED BUT NEVER SEEN — in today\'s stimuli, absent from every run audited.'); + console.log('Usually means the stimulus has not been run since the gate was added. If you'); + console.log('audited a subset of runs, expect this list to be long and mostly uninteresting.'); + const listed = [...unexercised.entries()].slice(0, 8); + for (const [gate, stimuli] of listed) { + console.log(` * ${gate} [${stimuli.join(', ')}]`); + } + if (unexercised.size > listed.length) { + console.log(` ... and ${unexercised.size - listed.length} more (--json for the full list)`); + } + } + + if (vacuous.length > 0) { + console.log(''); + const confident = vacuous.filter(row => row.confident); + console.log(`NEVER FAILED — ${vacuous.length} gate(s) have never discriminated between good and bad`); + console.log('output. At this corpus size that is expected rather than alarming: a young suite'); + console.log('mostly passes. Watch whether it stays true as the corpus grows.'); + if (confident.length > 0) { + console.log(` Worth a look first (>= ${minRuns} runs and still never red):`); + for (const { gate, tally } of confident) { + console.log(` * ${gate}: ${tally.passed} passes, 0 failures over ${tally.runs.size} runs`); + } + } else { + console.log(` None has reached ${minRuns} runs yet, so none is worth investigating on this evidence.`); + } + } + + if (actionable === 0 && vacuous.length === 0) { + console.log(''); + console.log('Nothing to investigate: every gate has both passed and failed at least once.'); + } + return suspect.filter(row => row.confident).length; +} + +function printPreamble(runs: string[], instances: number, voidInstances: number, minRuns: number): void { + console.log(''); + console.log('Gate health — auditing the instrument, not the product'); + console.log(`${runs.length} run(s), ${instances} instance(s). Verdicts below ${minRuns} runs are marked low confidence.`); + if (voidInstances > 0) { + console.log( + `${voidInstances} of ${instances} instance(s) were void (the agent never really ran); every verdict in them,\n` + + 'including the passes, is discarded rather than counted. A negative assertion passes\n' + + 'trivially against an empty session, so a void pass is as meaningless as a void failure.' + ); + } +} + +// --------------------------------------------------------------------------- +// Main +// --------------------------------------------------------------------------- + +async function main(): Promise { + const options = parseArgs(process.argv.slice(2)); + jsonMode = options.json; + + const roots: { runId: string; dir: string }[] = []; + for (const dir of options.extractedDirs) { + if (!existsSync(dir)) { + throw new GateHealthError(`--extracted ${dir} does not exist`); + } + roots.push({ runId: dir, dir }); + } + + let runIds = options.runIds; + if (runIds.length === 0 && roots.length === 0) { + runIds = localRunIds(); + if (runIds.length === 0) { + throw new GateHealthError( + 'No run ids given and no local MSBench run cache found.\n' + + 'Pass run ids explicitly — extraction is server-backed, so any run you have access\n' + + 'to works from any machine: node gate-health.ts 2026082579322454 …' + ); + } + log(`Auditing ${runIds.length} run(s) from the local MSBench cache.`); + } + + for (const runId of runIds) { + const dir = extractRun(runId, options.refresh); + if (dir) { + roots.push({ runId, dir }); + } + } + + const tallies = new Map(); + const auditedRuns: string[] = []; + let instanceCount = 0; + let voidInstances = 0; + + for (const { runId, dir } of roots) { + const instances = findInstances(dir); + if (instances.length === 0) { + log(` ! ${runId}: no instances found in ${dir}`); + continue; + } + auditedRuns.push(runId); + for (const instance of instances) { + instanceCount++; + if (instanceFault(instance)) { + voidInstances++; + } + analyzeInstance(tallies, runId, instance, options.identityByComment); + } + } + + if (tallies.size === 0) { + throw new GateHealthError('No assertion results found in any audited run; nothing to audit.'); + } + + const rows: GateRow[] = [...tallies.entries()] + .map(([gate, tally]) => ({ + gate, + tally, + verdict: classify(tally), + confident: tally.runs.size >= options.minRuns, + })) + .sort((a, b) => a.gate.localeCompare(b.gate)); + + const unexercised = new Map(); + for (const [gate, stimuli] of await declaredToday(options.identityByComment)) { + if (!tallies.has(gate)) { + unexercised.set(gate, stimuli); + } + } + + if (options.json) { + console.log(JSON.stringify({ + runs: auditedRuns, + instances: instanceCount, + voidInstances, + minRuns: options.minRuns, + gates: rows.map(({ gate, tally, verdict, confident }) => ({ + gate, + verdict, + confident, + passed: tally.passed, + failed: tally.failed, + notAttempted: tally.notAttempted, + notApplicable: tally.notApplicable, + runs: [...tally.runs], + instances: tally.instances, + notAttemptedReasons: Object.fromEntries(tally.notAttemptedReasons), + notApplicableReasons: Object.fromEntries(tally.notApplicableReasons), + exampleFailure: tally.exampleFailure, + exampleFailureRun: tally.exampleFailureRun, + })), + declaredButNeverSeen: Object.fromEntries(unexercised), + }, undefined, 2)); + const confidentSuspects = rows.filter(row => row.verdict === 'never-passed' && row.confident).length; + process.exitCode = confidentSuspects > 0 ? 1 : 0; + return; + } + + printPreamble(auditedRuns, instanceCount, voidInstances, options.minRuns); + printTable(rows); + const confidentSuspects = printFindings(rows, options.minRuns, unexercised); + + console.log(''); + console.log('None of these verdicts proves a defect. Each is a reason to look before quoting a score.'); + if (confidentSuspects > 0) { + console.log(''); + console.log(`FAIL: ${confidentSuspects} gate(s) ran in ${options.minRuns}+ runs and never once passed.`); + process.exitCode = 1; + } +} + +main().catch((error: unknown) => { + if (error instanceof GateHealthError) { + console.error(`\n${error.message}\n`); + process.exitCode = 2; + return; + } + throw error; +}); diff --git a/evals/package.json b/evals/package.json index 055483b22..a64c483a4 100644 --- a/evals/package.json +++ b/evals/package.json @@ -12,6 +12,7 @@ "typecheck": "tsc --noEmit -p tsconfig.json", "certify": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON src/graderCertification.ts", "regrade": "node --disable-warning=ExperimentalWarning msbench/regrade.ts", + "gate-health": "node --disable-warning=ExperimentalWarning msbench/gate-health.ts", "ci:local": "node ci-local.ts", "msbench:self-test": "node msbench/verify-run.ts --self-test" }, From d7fd6fc6d01f436175f70d620a6bfbff90afa675 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 22:25:05 -0700 Subject: [PATCH 2/8] Document how to read a verdict, using the sentinel as a worked example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The liveness sentinel is declared in all five stimuli and appears in zero of the 21 audited runs, because every run predates #1706. That is benign now and is expected to resolve when the scaffold runs land — but if it is still never-attempted after those, the same verdict means a real bug. A verdict here is a question with a date on it, not a finding. Also records the measured before/after for gate identity: four validate-requirements.ts variants as separate rows (three never-failed on one or two runs each) versus one requirements gate reading 5 pass / 1 fail / 7 runs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/evals/msbench/README.md b/evals/msbench/README.md index 1f828ff62..ee6cfaf17 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1090,6 +1090,23 @@ of exactly that one thing.* **None of these prove a defect.** Each is a reason to look before quoting a score. +### How to read a verdict: the sentinel, right now + +The liveness sentinel is a worked example, and it is in the report today. It is declared in +**all five stimuli** and appears in **zero of the 21 runs audited**, so it reports as +*declared but never seen*. That is correct and entirely benign — every run in the corpus +predates #1706, which added it. + +It is also the useful half of the lesson. **The verdict is expected to resolve on its own:** +the scaffold runs being submitted now are the first that will carry the sentinel. If it is +*still* never-attempted once those have landed, that is a real bug rather than a historical +artifact, and the same verdict means something completely different. + +That is how every verdict here should be read — as a question with a date on it, not a +finding. "Has never passed" is only interesting relative to *when the gate was last changed +and which runs have happened since*, which is why `--min-runs` and explicit run scoping both +exist. + ### How many runs before a verdict means anything Verdicts resting on fewer than `--min-runs` runs (default **3**) are printed but marked @@ -1181,7 +1198,12 @@ reworded.** The default `--identity gate` mitigates this by keying `exec:` gates on the grader's filename — the same id `gate=` is derived from, and the same id the certification manifest uses — which also recovers a stable identity for runs recorded *before* the convention -existed. Two consequences worth knowing: +existed. The difference is measurable on the current corpus: under comment identity the +four `validate-requirements.ts` variants appear as four separate rows, three of them +`never-failed` on one or two runs each; under grader identity they are one `requirements` +gate reading **5 pass / 1 fail / 7 runs / healthy**. Same data, and only the second is true. + +Two consequences worth knowing: - It is deliberately **coarser**: every `validate-requirements.ts` invocation is one gate regardless of its flags. Use `--identity comment` for the raw per-assertion view. From a7addc62b26c7d18f93d00239a8e01686c31d6b9 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 22:41:39 -0700 Subject: [PATCH 3/8] Handle the class= token and the reversal of the N/A exit code to 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Not-applicable graders now exit 3 rather than 0, and carry an explicit class=outOfScope|environmentGap alongside gate= and reason=. Because detection keys off the stderr marker rather than the exit code, the reversal needed no change to the bucketing — but three things did change: - Marker tokens are parsed order-independently. The live format is gate= class= reason= detail=, and a fixed-order regex would have silently stopped recognising N/A if a producer reordered them. - class=environmentGap is tallied as notAttempted, not notApplicable. A missing func binary is not dead weight: nobody decided the gate was unnecessary. An absent or unrecognised class is read as environmentGap, which is the safe direction — it says something is in the way rather than this gate is pointless. - never-attempted is now grouped by cause, as always-not-applicable already was, since that is where the five runtime-* gates land. Under exit 3 MSBench records N/A as passed: false, so the risk this protects against is inverted: N/A must now stay out of the failed bucket rather than the passed one, or the product gets charged for a missing prerequisite. Verified against fixtures covering both classes, a genuinely crashed grader (exit 3, no marker) and a real product failure (exit 1) — all four discriminate, and neither N/A lands in passed or failed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 73 +++++++++++++------ evals/msbench/gate-health.ts | 136 ++++++++++++++++++++++++----------- 2 files changed, 147 insertions(+), 62 deletions(-) diff --git a/evals/msbench/README.md b/evals/msbench/README.md index ee6cfaf17..12ba70289 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1084,7 +1084,7 @@ of exactly that one thing.* | --- | --- | --- | | `never-passed` | The gate may be impossible to satisfy — broken probe, wrong credential, bad fixture | Re-grade the named run and read the grader's own stderr | | `never-failed` | The gate may be vacuous; it has never discriminated | Check it can go red at all; certification is the cheap way | -| `always-not-applicable` | The gate has never rendered a verdict — **and MSBench scored every one as a pass** | Group by `reason=`; usually one missing prerequisite | +| `always-not-applicable` | The gate has never rendered a verdict — **and MSBench scored every one as a failure** | Group by `reason=`; usually one missing prerequisite | | `never-attempted` | The gate never got the chance to run | Fix what is upstream; the gate is not the problem | | `healthy` | Has both passed and failed | Nothing | @@ -1154,37 +1154,68 @@ credits passes it never earned. **Seven of twenty-six instances in the corpus ar > trend-plotting historical runs should assume the older half of the corpus is > contaminated in both directions. -### This report is the safety mechanism for the not-applicable convention +### Not-applicable, and the convention that got reversed The fidelity and runtime gates emit a machine-readable marker on stderr: ``` -NOT_APPLICABLE gate= reason= detail="…" +NOT_APPLICABLE gate= class= reason= detail="…" ``` -**and exit 0.** Because `assertZeroExitCode` compiles to `SELECT COUNT(*) > 0 FROM exec -WHERE exitCode = 0 …`, MSBench scores every N/A as a **pass**. A gate that is N/A across -the whole corpus therefore reports **16-for-16** — #1669's defect with the sign flipped, -and the inverted form is worse, because 0-for-16 looks alarming while 16-for-16 looks like -success and nobody investigates a passing gate. This is live rather than hypothetical: the -five `runtime-*` gates emit `functionsHostUnavailable` on *every* current stimulus, since -all four are Azure Functions and the container has no `func` binary. - -MSBench assertions are binary — there is no "neither" — so this cannot be fixed at the -assertion layer. **This report is the only place it can be caught**, which is why exit 0 -was only defensible on the assumption that this tool exists and behaves as follows. If you -are tempted to simplify any of it, this is what you would be breaking: +**and exit 3**, which MSBench records as `passed: false`. So an N/A is scored as a +**failure**, and a gate that is N/A across the whole corpus reads **0-for-16** — the story +at the top of this section exactly, except this time the gate is fine and the environment +is the problem. That is live rather than hypothetical: the five `runtime-*` gates emit +`functionsHostUnavailable` on *every* current stimulus, because all four are Azure +Functions and the container has no `func` binary. + +**It was very nearly the opposite, and the reversal is worth recording.** Exit 0 was ruled +first, explicitly on the grounds that this tool's always-not-applicable verdict made it +safe. That premise was false. MSBench writes `exitCode = 0` as `passed: true`, `resolved` +derives from it, and the run-analysis site, `msbench-cli report` and Kusto all publish that +number — so this report could say "not applicable" while the headline said green, and +**nobody investigates green**. Observing inflation is not the same as being able to undo +it. The ruling was reversed on that basis: exit 3 is *pessimistic and recoverable*, exit 0 +was *optimistic and unrecoverable*. + +MSBench assertions are binary — there is no "neither" — so neither convention can be fixed +at the assertion layer, and this report stays the only place N/A is visible as N/A. Three +behaviours are therefore a **contract**, not a preference. If you are tempted to simplify +any of them, this is what you would be breaking: 1. **N/A is its own bucket**, alongside passed / failed / notAttempted. Never folded into - passed, never silently dropped. + `passed` — and, under exit 3, **never folded into `failed`**, which is now the live risk + and would charge the product for a missing binary. 2. **Every rate excludes N/A from both numerator and denominator.** A gate that ran 16 times, was N/A 16 times and passed 0 real times has *no applicable observations* — the - `rate` column prints `n/a`, and `n/a` never means 100%. -3. **Always-N/A gates are grouped by `reason=`**, so one absent prerequisite reads as a - single actionable line rather than five mystery gates. + `rate` column prints `n/a`, which means nothing was judged, not 0% and not 100%. +3. **Always-N/A gates are grouped by `reason=`.** Under exit 3 this is what separates "five + gates are broken" from "one binary is missing, here is the install command". + +Detection keys off the **marker, not the exit code** — which is why reversing exit 0 to +exit 3 needed no code change at all. One ordering detail matters: the marker is checked +*before* the exit-3 grader-error branch, so a genuinely crashed grader (exit 3, no marker) +stays distinct from a not-applicable one. + +#### The two classes, and why the split is mechanical + +`class=` is on the line rather than in a lookup table here, so a new reason code cannot +silently default into the wrong bucket. Each gate family owns its own reason-to-class +mapping, so adding a reason is never a shared edit. + +| `class=` | Means | Tallied as | Because | +| --- | --- | --- | --- | +| `outOfScope` | The scenario genuinely does not apply — `ecosystemNotSupported`, `noFrontendDeclared` | `notApplicable` | This is what "dead weight" is meant to find | +| `environmentGap` | A prerequisite is missing — `functionsHostUnavailable`, `datastoreRequiresContainer` | `notAttempted` | Nobody decided the gate was unnecessary; the environment could not run it | + +An unrecognised or absent `class=` is read as `environmentGap`. That is the safe direction: +it reports "something is in the way" rather than "this gate is pointless". `noProjectManifestFound` +is the case that motivates it — it most likely means the tree was never staged, and reporting +that as dead weight would invite deleting a gate to fix a staging bug. -Detection keys off the **marker, not the exit code**, so the tool survives the convention -changing again. +A reason meaning *"we tried and it did not work"* does not belong on this path at all. That +is a product failure and must go red; routing one through N/A turns a real bug into a +self-suppressing green. ### Gate identity, and a known limitation diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index 751003cc3..d5652f9d9 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -87,35 +87,66 @@ const EXIT_GRADER_ERROR = 3; /** * The structured not-applicable marker, agreed with the fidelity-gates and runtime-gates sessions: * - * NOT_APPLICABLE gate= reason= detail="…" + * NOT_APPLICABLE gate= class= reason= detail="…" * - * ## This detection is the safety mechanism for the whole not-applicable convention + * `class=` is the mechanical split this tool asked for, and it is on the line rather than in a + * lookup table here so that a new reason code cannot silently default into the wrong bucket. Each + * gate family owns its own reason-to-class mapping, so adding a reason is never a shared edit: * - * An N/A grader **exits 0**. MSBench's `assertZeroExitCode` compiles to - * `SELECT COUNT(*) > 0 FROM exec WHERE exitCode = 0 …`, so **MSBench scores every N/A as a pass.** - * A gate that is not applicable across the entire corpus therefore reports 16-for-16. + * outOfScope — the scenario genuinely does not apply (`ecosystemNotSupported`). Dead weight, + * and the thing the always-not-applicable verdict is looking for. + * environmentGap — a prerequisite is missing (`functionsHostUnavailable`). **Not** dead weight: + * nobody has decided this gate is unnecessary, the environment simply cannot run + * it. Tallied with cascade, because that is what it is. * - * That is the 0-for-16 defect in this file's header with the sign flipped, and the inverted form is - * worse: 0-for-16 looks alarming, 16-for-16 looks like success, and nobody investigates a passing - * gate. MSBench assertions are binary — there is no "neither" — so this cannot be fixed at the - * assertion layer. **This report is the only place it can be caught**, which is why exit 0 was only - * defensible on the assumption that this tool exists and behaves as documented below. + * That distinction is load-bearing. `noProjectManifestFound` most likely means the tree was never + * staged; reported as dead weight it would read "this gate is unnecessary, delete it" — the exact + * inversion this whole tool exists to prevent. A missing or unrecognised `class=` is therefore read + * as `environmentGap`, which is the safe direction: it says "something is in the way" rather than + * "this gate is pointless". + * + * Note what does **not** belong here at all: a reason meaning "we tried and it did not work" is a + * product failure and must go red. Routing one through the N/A path turns a real bug into a + * self-suppressing green. + * + * ## Why this detection matters, and why it survived the convention changing + * + * An N/A grader **exits 3**, and MSBench records that as `passed: false`. So an N/A is scored as a + * **failure**, and a gate that is not applicable across the whole corpus reads 0-for-16 — which is + * this file's opening story exactly, except the gate is fine and the environment is the problem. + * + * It was very nearly the opposite. Exit 0 was ruled first, on the explicit grounds that this tool's + * always-not-applicable verdict made it safe. That premise was wrong: MSBench writes `exitCode = 0` + * as `passed: true`, `resolved` derives from it, and the run-analysis site, `msbench-cli report` and + * Kusto all publish that number. This report could say "not applicable" while the headline said + * green, and **nobody investigates green**. Observing inflation is not the same as undoing it. The + * ruling was reversed on that basis: exit 3 is pessimistic and recoverable, exit 0 was optimistic + * and unrecoverable. * * The contract, which is deliberately stated as a contract and not a preference: * - * 1. N/A is its own bucket, alongside passed / failed / notAttempted. Never folded into passed, - * never silently dropped. + * 1. N/A is its own bucket, alongside passed / failed / notAttempted. It is never folded into + * `passed` **and never into `failed`** — under exit 3 the second is the live risk, and it + * would charge the product for a missing binary. * 2. Every rate excludes N/A from **both** numerator and denominator. A gate that ran 16 times, * was N/A 16 times and passed 0 real times has no pass rate — it has *no applicable - * observations*. See `passRate`, which returns undefined rather than 100%. - * 3. Always-N/A gates are grouped by reason code, so one missing prerequisite reads as one - * actionable line rather than N alarming ones. + * observations*. See `passRate`, which returns undefined rather than 0%. + * 3. Always-N/A gates are grouped by reason code. Under exit 3 this is what separates "five gates + * are broken" from "one binary is missing, here is the install command". * - * Detection keys off **the marker, not the exit code**. That was the right call while the exit code - * was still being argued, and it stays right: it means this tool keeps working if the convention - * ever changes again. + * Detection keys off **the marker, not the exit code** — which is why the reversal from exit 0 to + * exit 3 required no change to any of it. That ordering matters in one specific way: the marker is + * checked *before* the exit-3 grader-error branch, so a legitimately-crashed grader (exit 3, no + * marker) stays distinct from a not-applicable one. */ -const NOT_APPLICABLE_MARKER = /^NOT_APPLICABLE\s+(?:gate=(\S+)\s+)?(?:reason=(\S+))?/mu; +const NOT_APPLICABLE_MARKER = /^NOT_APPLICABLE\b([^\n]*)/mu; + +/** + * `key=value` on the marker line. Parsed order-independently rather than as one fixed-order regex: + * the token order is not part of the agreed contract, and a producer reordering them must not + * silently turn every N/A back into a pass. + */ +const MARKER_TOKEN = /\b(gate|class|reason)=("[^"]*"|\S+)/gu; /** * `gate=` on a `PASS:` / `FAIL:` / `NOT_APPLICABLE` line. Fidelity derives the id from the @@ -430,14 +461,25 @@ function execCommandOf(comment: string): string | undefined { interface NotApplicable { reason: string; + /** `true` only when the grader positively declared the scenario out of scope. */ + outOfScope: boolean; } function parseNotApplicable(stdErr: string): NotApplicable | undefined { - const match = stdErr.match(NOT_APPLICABLE_MARKER); - if (!match) { + const line = stdErr.match(NOT_APPLICABLE_MARKER); + if (!line) { return undefined; } - return { reason: match[2] ?? 'unspecified' }; + const tokens = new Map(); + for (const [, key, value] of line[1].matchAll(MARKER_TOKEN)) { + tokens.set(key, value.replace(/^"|"$/gu, '')); + } + return { + reason: tokens.get('reason') ?? 'unspecified', + // Anything other than an explicit outOfScope is treated as an environment gap — see + // NOT_APPLICABLE_MARKER. Never guess a gate into the dead-weight bucket. + outOfScope: tokens.get('class') === 'outOfScope', + }; } /** @@ -575,8 +617,16 @@ function analyzeInstance( } else if (notApplicable) { // Never `passed`, even though the grader exited 0 and MSBench scored it as a pass. // This branch is the entire safety mechanism for the exit-0 convention. - tally.notApplicable++; - bump(tally.notApplicableReasons, notApplicable.reason); + if (notApplicable.outOfScope) { + tally.notApplicable++; + bump(tally.notApplicableReasons, notApplicable.reason); + } else { + // An environment gap is not dead weight: nobody decided this gate was unnecessary, + // the environment just could not run it. Reporting it as dead weight would invite + // deleting a gate to fix a missing binary. + tally.notAttempted++; + bump(tally.notAttemptedReasons, notApplicable.reason); + } } else if (detail.error) { // The assertion never compiled or returned a non-boolean. `passed: false` would launder // a harness fault into a product verdict. @@ -633,14 +683,6 @@ function passRate(tally: GateTally): number | undefined { // Reporting // --------------------------------------------------------------------------- -function topReasons(counter: Map, limit = 3): string { - return [...counter.entries()] - .sort((a, b) => b[1] - a[1]) - .slice(0, limit) - .map(([reason, count]) => `${reason}\u00d7${count}`) - .join(', '); -} - /** Gates declared in today's stimuli. A gate here that no run mentions has never been exercised. */ async function declaredToday(identityByComment: boolean): Promise> { const declared = new Map(); @@ -734,20 +776,32 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map(); + for (const row of starved) { + const [cause] = [...row.tally.notAttemptedReasons.entries()].sort((a, b) => b[1] - a[1])[0] ?? ['unknown']; + byCause.set(cause, [...(byCause.get(cause) ?? []), row]); + } + for (const [cause, gates] of [...byCause.entries()].sort((a, b) => b[1].length - a[1].length)) { + console.log(''); + console.log(` * ${cause} — ${gates.length} gate(s) blocked, 0 real verdicts between them`); + for (const { gate, tally } of gates) { + console.log(` ${gate} (${tally.notAttempted} blocked over ${tally.runs.size} run(s))`); + } } - console.log(' Fix the upstream cause; these gates cannot report anything until you do.'); } if (dead.length > 0) { actionable++; console.log(''); console.log('ALWAYS NOT-APPLICABLE — these have never rendered a verdict about the product.'); - console.log('MSBench scored every one of them as a PASS, because an N/A grader exits 0. Left to'); - console.log('the raw numbers each of these gates looks perfect while testing nothing, so this'); - console.log('section is the only thing standing between a coverage hole and a green dashboard.'); + console.log('MSBench scored every one of them as a FAILURE, because an N/A grader exits 3.'); + console.log('Left to the raw numbers each looks like a broken gate; grouped by reason they are'); + console.log('usually one missing prerequisite or one gate wired to a stack it never covered.'); // Grouped by reason so a single missing prerequisite reads as one line, not N mystery gates. const byReason = new Map(); for (const { gate, tally } of dead) { @@ -758,8 +812,8 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map 1 ? 'these gates are' : 'this gate is'} genuinely dead weight.`); } } From 0706263a98663255593bda9d26fb8a95d0e4fc97 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 22:44:28 -0700 Subject: [PATCH 4/8] Read outOfScope as a wiring bug rather than dead weight MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Applicability is decided at wiring time, in the stack declaration, not discovered by a gate at runtime. So a gate reporting class=outOfScope was attached to a stack it cannot answer for — a config bug with an owner. "Dead weight, consider deleting" is only correct if the gate is out of scope for every stack in the corpus, which this report cannot determine on its own, so it now says so rather than implying the stronger conclusion. Also drops stray markdown emphasis from terminal output. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 24 ++++++++++++++++-------- evals/msbench/gate-health.ts | 19 ++++++++++++------- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/evals/msbench/README.md b/evals/msbench/README.md index 12ba70289..d361cd2a7 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1084,8 +1084,8 @@ of exactly that one thing.* | --- | --- | --- | | `never-passed` | The gate may be impossible to satisfy — broken probe, wrong credential, bad fixture | Re-grade the named run and read the grader's own stderr | | `never-failed` | The gate may be vacuous; it has never discriminated | Check it can go red at all; certification is the cheap way | -| `always-not-applicable` | The gate has never rendered a verdict — **and MSBench scored every one as a failure** | Group by `reason=`; usually one missing prerequisite | -| `never-attempted` | The gate never got the chance to run | Fix what is upstream; the gate is not the problem | +| `always-not-applicable` | Declared `class=outOfScope` every time — the gate was wired to a stack it cannot answer for | Fix the stack wiring; only consider deleting if it's out of scope everywhere | +| `never-attempted` | The gate never got the chance to run — cascade, or `class=environmentGap` | Fix what is upstream or install the prerequisite; the gate is not the problem | | `healthy` | Has both passed and failed | Nothing | **None of these prove a defect.** Each is a reason to look before quoting a score. @@ -1205,17 +1205,25 @@ mapping, so adding a reason is never a shared edit. | `class=` | Means | Tallied as | Because | | --- | --- | --- | --- | -| `outOfScope` | The scenario genuinely does not apply — `ecosystemNotSupported`, `noFrontendDeclared` | `notApplicable` | This is what "dead weight" is meant to find | -| `environmentGap` | A prerequisite is missing — `functionsHostUnavailable`, `datastoreRequiresContainer` | `notAttempted` | Nobody decided the gate was unnecessary; the environment could not run it | +| `outOfScope` | The gate should not have been wired to this stack — `ecosystemNotSupported`, `noFrontendDeclared` | `notApplicable` | Applicability is a wiring-time decision; seeing it at runtime is a config bug with an owner | +| `environmentGap` | The gate applies, the machine cannot run it — `functionsHostUnavailable`, `datastoreRequiresContainer` | `notAttempted` | Nobody decided the gate was unnecessary; we genuinely are not testing something we claim to | An unrecognised or absent `class=` is read as `environmentGap`. That is the safe direction: -it reports "something is in the way" rather than "this gate is pointless". `noProjectManifestFound` -is the case that motivates it — it most likely means the tree was never staged, and reporting -that as dead weight would invite deleting a gate to fix a staging bug. +it reports "something is in the way" rather than "this gate should not be here". +`noProjectManifestFound` is the case that motivates it — it most likely means the tree was +never staged, and reporting that as a wiring or scope problem would invite deleting a gate +to fix a staging bug. + +Note what `always-not-applicable` does **not** license. Because applicability is decided at +wiring time, a gate reporting `outOfScope` is a **wiring bug with an owner** — it was +attached to a stack it cannot answer for. "Dead weight, consider deleting" is only correct +if the gate is out of scope for *every* stack in the corpus, which this report cannot tell +you by itself. Read the stack declarations before removing anything. A reason meaning *"we tried and it did not work"* does not belong on this path at all. That is a product failure and must go red; routing one through N/A turns a real bug into a -self-suppressing green. +self-suppressing green. Naming matters here too: a reason code that describes a harness +capability gap as though it were a product outcome tells the reader not to investigate. ### Gate identity, and a known limitation diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index d5652f9d9..0ea130693 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -798,11 +798,16 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map 0) { actionable++; console.log(''); - console.log('ALWAYS NOT-APPLICABLE — these have never rendered a verdict about the product.'); - console.log('MSBench scored every one of them as a FAILURE, because an N/A grader exits 3.'); - console.log('Left to the raw numbers each looks like a broken gate; grouped by reason they are'); - console.log('usually one missing prerequisite or one gate wired to a stack it never covered.'); - // Grouped by reason so a single missing prerequisite reads as one line, not N mystery gates. + console.log('ALWAYS OUT-OF-SCOPE — these declared class=outOfScope every time they ran, and'); + console.log('MSBench scored each as a FAILURE, because an N/A grader exits 3. Nothing here is'); + console.log('evidence about the generated app.'); + console.log(''); + console.log('Applicability is a wiring-time decision. A gate that is out of scope for the stack'); + console.log('it was wired to is a WIRING BUG WITH AN OWNER — fix the stack declaration, not the'); + console.log('gate. "Dead weight, consider deleting" is only the right reading if the gate is out'); + console.log('of scope for every stack in the corpus, which this report cannot tell you on its'); + console.log('own; check the stack declarations before removing anything.'); + // Grouped by reason so a single cause reads as one line, not N mystery gates. const byReason = new Map(); for (const { gate, tally } of dead) { const [reason] = [...tally.notApplicableReasons.entries()].sort((a, b) => b[1] - a[1])[0] ?? ['unspecified']; @@ -812,8 +817,8 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map 1 ? 'these gates are' : 'this gate is'} genuinely dead weight.`); + console.log(` Wired to ${gates.length > 1 ? 'stacks these gates' : 'a stack this gate'} cannot answer for, or never wired to one`); + console.log(' that exercises it.'); } } From 012e5adf9eec0b9d72f2124ce48c97fc3c50704d Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 22:48:35 -0700 Subject: [PATCH 5/8] Track the producers' class= semantics and stop overstating out-of-scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ecosystemNotSupported is classified by its producer as a gap to close, not as out of scope: a Go project has a plan, a tree and a real fidelity question, and there is simply no analyser for it. Calling that dead weight would suggest deleting the gate when the correct action is to write the analyser. The docs here said otherwise; the code never did, because it buckets on class= alone and never interprets reason codes. Also stops the out-of-scope section implying more than it knows. The report sees the runs it was given, so it can say "out of scope for the stacks observed" and no more — "dead weight everywhere" is a coverage claim it has no evidence for. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 20 +++++++++++++++----- evals/msbench/gate-health.ts | 29 ++++++++++++++++------------- 2 files changed, 31 insertions(+), 18 deletions(-) diff --git a/evals/msbench/README.md b/evals/msbench/README.md index d361cd2a7..f0ec44d93 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1205,8 +1205,15 @@ mapping, so adding a reason is never a shared edit. | `class=` | Means | Tallied as | Because | | --- | --- | --- | --- | -| `outOfScope` | The gate should not have been wired to this stack — `ecosystemNotSupported`, `noFrontendDeclared` | `notApplicable` | Applicability is a wiring-time decision; seeing it at runtime is a config bug with an owner | -| `environmentGap` | The gate applies, the machine cannot run it — `functionsHostUnavailable`, `datastoreRequiresContainer` | `notAttempted` | Nobody decided the gate was unnecessary; we genuinely are not testing something we claim to | +| `outOfScope` | The gate should not have been wired to this stack — `noFrontendDeclared`, `noHealthPathDeclared` | `notApplicable` | Applicability is a wiring-time decision; seeing it at runtime is a config bug with an owner | +| `environmentGap` | The gate applies, the machine cannot run it — `functionsHostUnavailable`, `ecosystemNotSupported` | `notAttempted` | Nobody decided the gate was unnecessary; we genuinely are not testing something we claim to | + +Note which side `ecosystemNotSupported` sits on, because it is the instructive one. A Go +project is **not** a scenario with nothing to test — it has a plan, a tree and a real +fidelity question; we simply have no analyser for it. Classified `outOfScope` it would tell +someone to delete the datastore gate because it keeps not applying to Go, when the correct +action is to write the Go analyser. The producer owns that judgement, which is exactly why +this tool buckets on `class=` and never on the reason code. An unrecognised or absent `class=` is read as `environmentGap`. That is the safe direction: it reports "something is in the way" rather than "this gate should not be here". @@ -1216,9 +1223,12 @@ to fix a staging bug. Note what `always-not-applicable` does **not** license. Because applicability is decided at wiring time, a gate reporting `outOfScope` is a **wiring bug with an owner** — it was -attached to a stack it cannot answer for. "Dead weight, consider deleting" is only correct -if the gate is out of scope for *every* stack in the corpus, which this report cannot tell -you by itself. Read the stack declarations before removing anything. +attached to a stack it cannot answer for. Beyond that, this report can only ever say *out +of scope for the stacks actually observed*. "Dead weight everywhere, delete it" is a claim +about coverage that the report has no evidence for: it sees the runs it was given, not the +set of stacks that exist. Overstating it once would teach people to discount the verdict +entirely, so it deliberately stops short. Read the stack declarations before removing +anything. A reason meaning *"we tried and it did not work"* does not belong on this path at all. That is a product failure and must go red; routing one through N/A turns a real bug into a diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index 0ea130693..83a4b4589 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -93,17 +93,20 @@ const EXIT_GRADER_ERROR = 3; * lookup table here so that a new reason code cannot silently default into the wrong bucket. Each * gate family owns its own reason-to-class mapping, so adding a reason is never a shared edit: * - * outOfScope — the scenario genuinely does not apply (`ecosystemNotSupported`). Dead weight, - * and the thing the always-not-applicable verdict is looking for. - * environmentGap — a prerequisite is missing (`functionsHostUnavailable`). **Not** dead weight: - * nobody has decided this gate is unnecessary, the environment simply cannot run - * it. Tallied with cascade, because that is what it is. + * outOfScope — the gate should not have been wired to this stack (`noFrontendDeclared`). + * Applicability is a wiring-time decision, so this is a config bug with an + * owner rather than proof the gate is unnecessary. + * environmentGap — the gate applies, the machine cannot run it (`functionsHostUnavailable`, + * `ecosystemNotSupported`). Not dead weight: nobody has decided this gate is + * unnecessary. Tallied with cascade, because that is what it is. * - * That distinction is load-bearing. `noProjectManifestFound` most likely means the tree was never - * staged; reported as dead weight it would read "this gate is unnecessary, delete it" — the exact - * inversion this whole tool exists to prevent. A missing or unrecognised `class=` is therefore read - * as `environmentGap`, which is the safe direction: it says "something is in the way" rather than - * "this gate is pointless". + * `ecosystemNotSupported` sits on the environmentGap side, which is the instructive case: a Go + * project has a plan, a tree and a real fidelity question — there is simply no analyser for it. + * Bucketed as dead weight it would suggest deleting the gate when the correct action is to write + * the analyser. That judgement belongs to the producer, which is why this tool buckets on `class=` + * alone and never interprets reason codes. A missing or unrecognised `class=` is read as + * `environmentGap`, the safe direction: it says "something is in the way" rather than "this gate + * should not be here". * * Note what does **not** belong here at all: a reason meaning "we tried and it did not work" is a * product failure and must go red. Routing one through the N/A path turns a real bug into a @@ -804,9 +807,9 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map(); for (const { gate, tally } of dead) { From d7f5f2f141fc553f4ab1bbeab67b11ac196f02e8 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 23:18:46 -0700 Subject: [PATCH 6/8] Distinguish a reader fault from a run with no output, loudly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An extraction with no instances had two causes collapsed into one quiet log line: the archive holds instance output we failed to load, or the run genuinely has none yet. Those need opposite responses, and collapsing them is this tool's own version of the failure class it exists to find — a silently dropped run under-reports coverage invisibly, and does so towards "never executed", the loudest verdict here. The reader now cross-checks the cached results.zip. An archive containing *-output.zip members while the extraction yielded no instances is reported as a READER FAULT in the preamble, naming the runs and stating that no tally below should be quoted. An archive with no output member is reported as pending or missing, which is a corpus fact rather than a bug. Also hardens marker parsing: field scanning stops before detail=, so a detail string containing the literal text class= or reason= cannot be read as a field. detail= is now JSON.stringify on both emitters and is treated as opaque. Verified: findInstances is order-independent (total scan, no first-match) against directories with -output entries at adversarial readdir positions; and a marker whose detail embeds a decoy class=outOfScope is bucketed by the real class. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/gate-health.ts | 146 +++++++++++++++++++++++++++++++---- 1 file changed, 133 insertions(+), 13 deletions(-) diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index 83a4b4589..5108bc48b 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -301,6 +301,16 @@ function parseArgs(argv: string[]): Options { // Locating runs // --------------------------------------------------------------------------- +/** Candidate locations of MSBench's per-machine run cache. */ +function msbenchRunRoots(): string[] { + return [ + process.env.MSBENCH_DATA_DIR ? join(process.env.MSBENCH_DATA_DIR, 'runs') : undefined, + join(homedir(), 'Library', 'Application Support', 'msbench', 'runs'), + join(homedir(), '.local', 'share', 'msbench', 'runs'), + process.env.APPDATA ? join(process.env.APPDATA, 'msbench', 'runs') : undefined, + ].filter((path): path is string => path !== undefined); +} + /** * MSBench's own run cache. This is where run *discovery* is local-only: the CLI can list runs from * Kusto (`list runs --kusto`), but that needs a Kusto read grant the `MSBench User` role does not @@ -308,14 +318,7 @@ function parseArgs(argv: string[]): Options { * *data* is not local — see `extractRun`. */ function localRunIds(): string[] { - const candidates = [ - process.env.MSBENCH_DATA_DIR ? join(process.env.MSBENCH_DATA_DIR, 'runs') : undefined, - join(homedir(), 'Library', 'Application Support', 'msbench', 'runs'), - join(homedir(), '.local', 'share', 'msbench', 'runs'), - process.env.APPDATA ? join(process.env.APPDATA, 'msbench', 'runs') : undefined, - ].filter((path): path is string => path !== undefined); - - for (const root of candidates) { + for (const root of msbenchRunRoots()) { if (!existsSync(root)) { continue; } @@ -369,6 +372,10 @@ function findInstances(root: string): Instance[] { if (!existsSync(root)) { return []; } + // A total scan, deliberately: no first-match, no break, no index assumption. Directory order + // from readdirSync is not guaranteed, so anything order-sensitive here would drop instances + // non-deterministically — and it would fail towards "never executed", this tool's loudest + // verdict. `diagnoseEmptyExtraction` is the paired check for the same hazard. return readdirSync(root, { withFileTypes: true }) .filter(entry => entry.isDirectory() && entry.name.endsWith('-output')) .map(entry => ({ @@ -379,6 +386,95 @@ function findInstances(root: string): Instance[] { .sort((a, b) => a.name.localeCompare(b.name)); } +/** + * An extraction with no instances has two very different causes, and collapsing them is the same + * "passes for the wrong reason" shape this tool exists to find — pointed at the tool itself. + * + * - The run's archive contains instance output we failed to load. That is a **reader fault**, and + * it must be loud: a corpus consumer that silently drops runs under-reports gate coverage + * invisibly, and it fails towards "never executed". + * - The archive genuinely has no instance output yet. That is a corpus fact — a pending or + * missing blob — and it is not this tool's bug. + * + * Distinguishing them requires looking at the archive rather than trusting the extraction, so this + * reads the cached `results.zip` member list directly. `results.zip` member order is an artifact of + * packing order and is not guaranteed, which is precisely why the presence of an output member is + * checked rather than its position. + */ +function diagnoseEmptyExtraction(runId: string): string { + const archive = runArchivePath(runId); + if (!archive) { + return 'no instances found (no cached archive to cross-check — run may be pending or missing)'; + } + const members = zipMemberNames(archive); + if (members === undefined) { + return `no instances found (could not read ${archive} to cross-check)`; + } + const outputMembers = members.filter(name => name.endsWith('-output.zip')); + if (outputMembers.length === 0) { + return `no instances found; the archive carries no *-output.zip member either ` + + `(${members.length} member(s)). The run produced no instance output — pending or missing, ` + + 'not a reader fault.'; + } + return `READER FAULT: ${archive} contains ${outputMembers.length} *-output.zip member(s) ` + + `(${outputMembers.join(', ')}) but the extraction yielded no instances. The data exists and ` + + 'was not loaded — treat this as a bug in this tool or in extraction, NOT as a corpus fact.'; +} + +/** The cached archive MSBench keeps per run, if this machine has it. */ +function runArchivePath(runId: string): string | undefined { + if (!/^\d+$/u.test(runId)) { + return undefined; + } + for (const root of msbenchRunRoots()) { + const candidate = join(root, runId, 'results.zip'); + if (existsSync(candidate)) { + return candidate; + } + } + return undefined; +} + +/** + * Member names from a zip's end-of-central-directory record. Implemented here rather than pulled in + * as a dependency because it is only ever used to answer "does this archive contain instance + * output", and being wrong in the conservative direction (returning undefined) is harmless. + */ +function zipMemberNames(archive: string): string[] | undefined { + let buffer: Buffer; + try { + buffer = readFileSync(archive); + } catch { + return undefined; + } + // Locate the end-of-central-directory signature, scanning back over the max comment length. + const EOCD = 0x06054b50; + let eocd = -1; + for (let offset = buffer.length - 22; offset >= Math.max(0, buffer.length - 22 - 0xffff); offset--) { + if (buffer.readUInt32LE(offset) === EOCD) { + eocd = offset; + break; + } + } + if (eocd === -1) { + return undefined; + } + const count = buffer.readUInt16LE(eocd + 10); + let pointer = buffer.readUInt32LE(eocd + 16); + const names: string[] = []; + for (let index = 0; index < count; index++) { + if (pointer + 46 > buffer.length || buffer.readUInt32LE(pointer) !== 0x02014b50) { + return names.length > 0 ? names : undefined; + } + const nameLength = buffer.readUInt16LE(pointer + 28); + const extraLength = buffer.readUInt16LE(pointer + 30); + const commentLength = buffer.readUInt16LE(pointer + 32); + names.push(buffer.subarray(pointer + 46, pointer + 46 + nameLength).toString('utf8')); + pointer += 46 + nameLength + extraLength + commentLength; + } + return names; +} + // --------------------------------------------------------------------------- // Reading one instance // --------------------------------------------------------------------------- @@ -473,13 +569,18 @@ function parseNotApplicable(stdErr: string): NotApplicable | undefined { if (!line) { return undefined; } + // `detail=` is opaque and free-form, and the two emitters escape it differently (JSON.stringify + // vs. quote-substitution). Stop scanning before it: otherwise a detail containing the literal + // text `reason=` would be read as a field. The three fields that matter are closed vocabularies + // and all precede `detail=`, so cutting here cannot lose them. + const beforeDetail = line[1].split(/\s+detail=/u)[0]; const tokens = new Map(); - for (const [, key, value] of line[1].matchAll(MARKER_TOKEN)) { + for (const [, key, value] of beforeDetail.matchAll(MARKER_TOKEN)) { tokens.set(key, value.replace(/^"|"$/gu, '')); } return { reason: tokens.get('reason') ?? 'unspecified', - // Anything other than an explicit outOfScope is treated as an environment gap — see + // Anything other than an explicit outOfScope is treated as a gap — see // NOT_APPLICABLE_MARKER. Never guess a gate into the dead-weight bucket. outOfScope: tokens.get('class') === 'outOfScope', }; @@ -863,10 +964,23 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map row.confident).length; } -function printPreamble(runs: string[], instances: number, voidInstances: number, minRuns: number): void { +function printPreamble(runs: string[], instances: number, voidInstances: number, minRuns: number, + readerFaults: string[], skipped: string[]): void { console.log(''); console.log('Gate health — auditing the instrument, not the product'); console.log(`${runs.length} run(s), ${instances} instance(s). Verdicts below ${minRuns} runs are marked low confidence.`); + if (readerFaults.length > 0) { + // Loud, and deliberately separate from a pending run: this says the numbers below are + // incomplete for a reason that is our fault, so nothing here should be quoted. + console.log(''); + console.log(`READER FAULT on ${readerFaults.length} run(s): ${readerFaults.join(', ')}`); + console.log('Their archives contain instance output that failed to load, so every tally below'); + console.log('is missing data. Fix the reader before believing any verdict — a dropped run'); + console.log('makes gates look never-executed, which is exactly what this report shouts about.'); + } + if (skipped.length > 0) { + console.log(`Skipped ${skipped.length} run(s) with no instance output (pending or missing): ${skipped.join(', ')}`); + } if (voidInstances > 0) { console.log( `${voidInstances} of ${instances} instance(s) were void (the agent never really ran); every verdict in them,\n` + @@ -914,13 +1028,19 @@ async function main(): Promise { const tallies = new Map(); const auditedRuns: string[] = []; + const readerFaults: string[] = []; + const skipped: string[] = []; let instanceCount = 0; let voidInstances = 0; for (const { runId, dir } of roots) { const instances = findInstances(dir); if (instances.length === 0) { - log(` ! ${runId}: no instances found in ${dir}`); + // Never just shrug and continue: a dropped run under-reports coverage invisibly, and it + // does so in the direction of "never executed" — the loudest verdict here. + const diagnosis = diagnoseEmptyExtraction(runId); + log(` ! ${runId}: ${diagnosis}`); + (diagnosis.startsWith('READER FAULT') ? readerFaults : skipped).push(runId); continue; } auditedRuns.push(runId); @@ -981,7 +1101,7 @@ async function main(): Promise { return; } - printPreamble(auditedRuns, instanceCount, voidInstances, options.minRuns); + printPreamble(auditedRuns, instanceCount, voidInstances, options.minRuns, readerFaults, skipped); printTable(rows); const confidentSuspects = printFindings(rows, options.minRuns, unexercised); From 9e2bd2d4c0c5636c765d5579dd53be3b9566c3ff Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 23:19:34 -0700 Subject: [PATCH 7/8] Record the third-class consequence at the parse site Bucketing anything that is not explicitly outOfScope as a gap is the safe default and makes the coverageGap rename a no-op. It also means a future third class would land in the gap bucket silently, so that consequence is now written where the next reader will see it: a third class needs an explicit ruling and this line updated to match. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/gate-health.ts | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index 5108bc48b..528a30357 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -580,8 +580,14 @@ function parseNotApplicable(stdErr: string): NotApplicable | undefined { } return { reason: tokens.get('reason') ?? 'unspecified', - // Anything other than an explicit outOfScope is treated as a gap — see - // NOT_APPLICABLE_MARKER. Never guess a gate into the dead-weight bucket. + // Deliberate default-by-exclusion: anything that is not explicitly `outOfScope` is treated + // as a gap. That is the safe direction — never guess a gate into the dead-weight bucket — + // and it makes a rename of the gap class (environmentGap -> coverageGap) a no-op here. + // + // The consequence, recorded so it is a decision rather than an accident: if a THIRD class is + // ever introduced it lands silently in the gap bucket. A third class therefore needs an + // explicit ruling on which bucket it joins, and this line updated to match. Raise it before + // emitting one. outOfScope: tokens.get('class') === 'outOfScope', }; } From eb43ecc60a28fa1f35331c10f13cb6c472056580 Mon Sep 17 00:00:00 2001 From: alexweininger Date: Tue, 25 Aug 2026 23:21:06 -0700 Subject: [PATCH 8/8] Say plainly that filename identity fixes only the smaller half MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Keying exec: gates on the grader filename stops that half of the corpus from forking histories when an assertion is reworded. It does nothing for SQL assertions over files/toolCalls/llm_responses, which have no stderr and no grader file — and those are the majority. That limit is measured rather than feared: three pairs in the corpus are one gate wearing two names, because sibling stimuli word the same assertion differently. All three are SQL assertions, so no identity scheme available here can merge them. The real fix is upstream — one canonical string per shared gate plus a drift check. The report now warns about it where it actually surfaces: a gate reworded in one stimulus appears under "declared but never seen" with its old text while running fine under the new one, and "this wording has never run" is indistinguishable from "this gate has never run" without checking. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- evals/msbench/README.md | 22 ++++++++++++++++++++-- evals/msbench/gate-health.ts | 5 +++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/evals/msbench/README.md b/evals/msbench/README.md index f0ec44d93..84c3338f4 100644 --- a/evals/msbench/README.md +++ b/evals/msbench/README.md @@ -1256,8 +1256,26 @@ Two consequences worth knowing: - It is deliberately **coarser**: every `validate-requirements.ts` invocation is one gate regardless of its flags. Use `--identity comment` for the raw per-assertion view. -- It only helps `exec:` gates. The SQL assertions over `files` / `toolCalls` / - `llm_responses` have no stderr and no grader file, so they stay comment-keyed. +- **It is a partial fix, and the larger half is untouched.** Filename identity stops + `program`/`exec:` gates from getting worse. SQL assertions over `files` / `toolCalls` / + `llm_responses` have no stderr and no grader file, so they keep comment identity — and + they are the majority of gates. + +That second point is measured, not feared. Three pairs in the current corpus are one gate +wearing two names, because sibling stimuli word the same assertion differently: + +| | | +| --- | --- | +| `Sentinel; …or the negative checks below are vacuous` | `Sentinel; …or every check below is vacuous` | +| `Agent should not open the plan view to approve the plan itself` | `Agent should not take over planning by opening the plan view` | +| `Agent should not fall back to the chat question tool` | `Agent should refuse with a message, not by asking a chat question` | + +All three are SQL assertions, so no identity scheme available here can merge them, and the +count grows as stimuli are added. **The actual fix is upstream**: one canonical string per +shared gate plus a drift check that fails when a stimulus deviates. Until that lands, treat +run counts for SQL-assertion gates as a lower bound, and read a `never-attempted` verdict on +one of them as possibly meaning "this wording has never run" rather than "this gate has +never run". ### Where the data lives — and why this is not a laptop-only tool diff --git a/evals/msbench/gate-health.ts b/evals/msbench/gate-health.ts index 528a30357..32a4e18dd 100644 --- a/evals/msbench/gate-health.ts +++ b/evals/msbench/gate-health.ts @@ -938,6 +938,11 @@ function printFindings(rows: GateRow[], minRuns: number, unexercised: Map