Audit the gates themselves, not just the product

evals/msbench/README.md

@@ -1061,6 +1061,266 @@ Two traps, both of which cost real time:
   msbench-cli run … --benchmark <benchmark>.<instance_id> <benchmark>.<instance_id>
   ```
 
+## Auditing the gates themselves
+
+[`gate-health.ts`](gate-health.ts) audits the **instrument** rather than the product. It
+reads past runs and asks, per gate, whether that gate has ever actually done its job.
+
+```bash
+export PATH="$HOME/.msbench-venv/bin:$PATH"
+cd evals && npm run gate-health                    # every run in the local cache
+npm run gate-health -- 2026082579322454 …          # specific runs, extracted on demand
+```
+
+The motivating case comes from the sibling suite in #1669: its `worker` gate recorded
+**16 failures and zero passes across every run ever executed** before anyone noticed the
+storage probe was signing its Azurite requests with a corrupted account key. Azurite
+answered 403 to everything, so no generated app could have passed regardless of quality.
+Ten percent of the corpus was being charged for a harness defect. *A gate that has never
+once passed is far more likely to be broken than the product is to be uniformly incapable
+of exactly that one thing.*
+
+| Verdict | What it suggests | What to do |
+| --- | --- | --- |
+| `never-passed` | The gate may be impossible to satisfy — broken probe, wrong credential, bad fixture | Re-grade the named run and read the grader's own stderr |
+| `never-failed` | The gate may be vacuous; it has never discriminated | Check it can go red at all; certification is the cheap way |
+| `always-not-applicable` | Declared `class=outOfScope` every time — the gate was wired to a stack it cannot answer for | Fix the stack wiring; only consider deleting if it's out of scope everywhere |
+| `never-attempted` | The gate never got the chance to run — cascade, or `class=environmentGap` | Fix what is upstream or install the prerequisite; the gate is not the problem |
+| `healthy` | Has both passed and failed | Nothing |
+
+**None of these prove a defect.** Each is a reason to look before quoting a score.
+
+### How to read a verdict: the sentinel, right now
+
+The liveness sentinel is a worked example, and it is in the report today. It is declared in
+**all five stimuli** and appears in **zero of the 21 runs audited**, so it reports as
+*declared but never seen*. That is correct and entirely benign — every run in the corpus
+predates #1706, which added it.
+
+It is also the useful half of the lesson. **The verdict is expected to resolve on its own:**
+the scaffold runs being submitted now are the first that will carry the sentinel. If it is
+*still* never-attempted once those have landed, that is a real bug rather than a historical
+artifact, and the same verdict means something completely different.
+
+That is how every verdict here should be read — as a question with a date on it, not a
+finding. "Has never passed" is only interesting relative to *when the gate was last changed
+and which runs have happened since*, which is why `--min-runs` and explicit run scoping both
+exist.
+
+### How many runs before a verdict means anything
+
+Verdicts resting on fewer than `--min-runs` runs (default **3**) are printed but marked
+`(low confidence)` and never fail the process. This matters more than it sounds: on the
+current corpus **25 of 30 gates are `never-failed`**, which is what a young suite looks
+like, not a broken one — most gates have one to fourteen observations. The number to watch
+is whether that ratio survives corpus growth, not its value today.
+
+Only a `never-passed` gate with at least `--min-runs` runs sets exit **1**.
+
+### The four inputs, and what had to be inferred
+
+#1669 read a `cor-validation.json` carrying a per-gate `status` and an explicit
+`notAttempted` flag. **No such file exists here.** MSBench records only `passed: true |
+false` plus a nullable `error`, so every distinction is reconstructed from four artifacts
+of an extraction:
+
+| Input | Gives |
+| --- | --- |
+| `vsc-output/eval.json` → `details[]` | the verdicts; the only gate name MSBench carries is the assertion comment |
+| `vsc-output/session.sqlite` → `exec` table | exit **1** vs exit **3** offline, and the N/A marker on stderr |
+| `output/error.json` → `type` | the instance was **void** — see below |
+| `vsc-output/configs/final-agent-config.json` | declared assertions, so a run with **no `eval.json` at all** still names the gates that never ran |
+
+The `assertions` table in `session.sqlite` is always empty; `eval.json` is the authority.
+
+### Void instances corrupt the tally in both directions
+
+#1669's cascade is per-gate, matched on the prose of a failure reason. Ours is structured
+and coarser: `error.json` marks a whole instance void. Every verdict in a void instance is
+discarded — **including the passes**, which is the part #1669 does not model.
+
+That is not theoretical. Two runs in the current corpus:
+
+| Run | Fault | Recorded | What actually happened |
+| --- | --- | --- | --- |
+| [`2026082583236973`](https://msbenchapp.azurewebsites.net/run-analysis/2026082583236973) | `RATE_LIMIT` | 1/4 — including a **pass** for `Agent should not fall back to the chat question tool` | The agent produced **literally nothing**. A `COUNT(*) = 0` assertion is trivially true against an empty table. |
+| [`2026082467189297`](https://msbenchapp.azurewebsites.net/run-analysis/2026082467189297) | `X_EXTENSION_ACTIVATION_ERROR` | 4/7 | **All four "passes" are the negative assertions**; all three "failures" are the extension never activating. |
+
+So a naive pass rate over these manufactures failures the product never earned *and*
+credits passes it never earned. **Seven of twenty-six instances in the corpus are void.**
+
+> **Any run predating #1706 may contain vacuous passes.** The liveness sentinel added
+> there fails such runs outright, but only going forward. Anyone re-grading or
+> trend-plotting historical runs should assume the older half of the corpus is
+> contaminated in both directions.
+
+### Not-applicable, and the convention that got reversed
+
+The fidelity and runtime gates emit a machine-readable marker on stderr:
+
+```
+NOT_APPLICABLE gate=<gate-id> class=<outOfScope|environmentGap> reason=<reasonCode> detail="…"
+```
+
+**and exit 3**, which MSBench records as `passed: false`. So an N/A is scored as a
+**failure**, and a gate that is N/A across the whole corpus reads **0-for-16** — the story
+at the top of this section exactly, except this time the gate is fine and the environment
+is the problem. That is live rather than hypothetical: the five `runtime-*` gates emit
+`functionsHostUnavailable` on *every* current stimulus, because all four are Azure
+Functions and the container has no `func` binary.
+
+**It was very nearly the opposite, and the reversal is worth recording.** Exit 0 was ruled
+first, explicitly on the grounds that this tool's always-not-applicable verdict made it
+safe. That premise was false. MSBench writes `exitCode = 0` as `passed: true`, `resolved`
+derives from it, and the run-analysis site, `msbench-cli report` and Kusto all publish that
+number — so this report could say "not applicable" while the headline said green, and
+**nobody investigates green**. Observing inflation is not the same as being able to undo
+it. The ruling was reversed on that basis: exit 3 is *pessimistic and recoverable*, exit 0
+was *optimistic and unrecoverable*.
+
+MSBench assertions are binary — there is no "neither" — so neither convention can be fixed
+at the assertion layer, and this report stays the only place N/A is visible as N/A. Three
+behaviours are therefore a **contract**, not a preference. If you are tempted to simplify
+any of them, this is what you would be breaking:
+
+1. **N/A is its own bucket**, alongside passed / failed / notAttempted. Never folded into
+   `passed` — and, under exit 3, **never folded into `failed`**, which is now the live risk
+   and would charge the product for a missing binary.
+2. **Every rate excludes N/A from both numerator and denominator.** A gate that ran 16
+   times, was N/A 16 times and passed 0 real times has *no applicable observations* — the
+   `rate` column prints `n/a`, which means nothing was judged, not 0% and not 100%.
+3. **Always-N/A gates are grouped by `reason=`.** Under exit 3 this is what separates "five
+   gates are broken" from "one binary is missing, here is the install command".
+
+Detection keys off the **marker, not the exit code** — which is why reversing exit 0 to
+exit 3 needed no code change at all. One ordering detail matters: the marker is checked
+*before* the exit-3 grader-error branch, so a genuinely crashed grader (exit 3, no marker)
+stays distinct from a not-applicable one.
+
+#### The two classes, and why the split is mechanical
+
+`class=` is on the line rather than in a lookup table here, so a new reason code cannot
+silently default into the wrong bucket. Each gate family owns its own reason-to-class
+mapping, so adding a reason is never a shared edit.
+
+| `class=` | Means | Tallied as | Because |
+| --- | --- | --- | --- |
+| `outOfScope` | The gate should not have been wired to this stack — `noFrontendDeclared`, `noHealthPathDeclared` | `notApplicable` | Applicability is a wiring-time decision; seeing it at runtime is a config bug with an owner |
+| `environmentGap` | The gate applies, the machine cannot run it — `functionsHostUnavailable`, `ecosystemNotSupported` | `notAttempted` | Nobody decided the gate was unnecessary; we genuinely are not testing something we claim to |
+
+Note which side `ecosystemNotSupported` sits on, because it is the instructive one. A Go
+project is **not** a scenario with nothing to test — it has a plan, a tree and a real
+fidelity question; we simply have no analyser for it. Classified `outOfScope` it would tell
+someone to delete the datastore gate because it keeps not applying to Go, when the correct
+action is to write the Go analyser. The producer owns that judgement, which is exactly why
+this tool buckets on `class=` and never on the reason code.
+
+An unrecognised or absent `class=` is read as `environmentGap`. That is the safe direction:
+it reports "something is in the way" rather than "this gate should not be here".
+`noProjectManifestFound` is the case that motivates it — it most likely means the tree was
+never staged, and reporting that as a wiring or scope problem would invite deleting a gate
+to fix a staging bug.
+
+Note what `always-not-applicable` does **not** license. Because applicability is decided at
+wiring time, a gate reporting `outOfScope` is a **wiring bug with an owner** — it was
+attached to a stack it cannot answer for. Beyond that, this report can only ever say *out
+of scope for the stacks actually observed*. "Dead weight everywhere, delete it" is a claim
+about coverage that the report has no evidence for: it sees the runs it was given, not the
+set of stacks that exist. Overstating it once would teach people to discount the verdict
+entirely, so it deliberately stops short. Read the stack declarations before removing
+anything.
+
+A reason meaning *"we tried and it did not work"* does not belong on this path at all. That
+is a product failure and must go red; routing one through N/A turns a real bug into a
+self-suppressing green. Naming matters here too: a reason code that describes a harness
+capability gap as though it were a product outcome tells the reader not to investigate.
+
+### Gate identity, and a known limitation
+
+MSBench carries no gate id — `eval.json` identifies an assertion only by its comment. That
+is unstable: `requirements.json should be valid JSON carrying a questions array` and
+`requirements.json satisfies the requirements contract` are **the same gate** before and
+after it moved from SQL to `exec:`, so under comment identity it appears as two gates with
+7 and 3 runs rather than one with 10. **A gate can silently reset its own history by being
+reworded.**
+
+The default `--identity gate` mitigates this by keying `exec:` gates on the grader's
+filename — the same id `gate=` is derived from, and the same id the certification manifest
+uses — which also recovers a stable identity for runs recorded *before* the convention
+existed. The difference is measurable on the current corpus: under comment identity the
+four `validate-requirements.ts` variants appear as four separate rows, three of them
+`never-failed` on one or two runs each; under grader identity they are one `requirements`
+gate reading **5 pass / 1 fail / 7 runs / healthy**. Same data, and only the second is true.
+
+Two consequences worth knowing:
+
+- It is deliberately **coarser**: every `validate-requirements.ts` invocation is one gate
+  regardless of its flags. Use `--identity comment` for the raw per-assertion view.
+- **It is a partial fix, and the larger half is untouched.** Filename identity stops
+  `program`/`exec:` gates from getting worse. SQL assertions over `files` / `toolCalls` /
+  `llm_responses` have no stderr and no grader file, so they keep comment identity — and
+  they are the majority of gates.
+
+That second point is measured, not feared. Three pairs in the current corpus are one gate
+wearing two names, because sibling stimuli word the same assertion differently:
+
+| | |
+| --- | --- |
+| `Sentinel; …or the negative checks below are vacuous` | `Sentinel; …or every check below is vacuous` |
+| `Agent should not open the plan view to approve the plan itself` | `Agent should not take over planning by opening the plan view` |
+| `Agent should not fall back to the chat question tool` | `Agent should refuse with a message, not by asking a chat question` |
+
+All three are SQL assertions, so no identity scheme available here can merge them, and the
+count grows as stimuli are added. **The actual fix is upstream**: one canonical string per
+shared gate plus a drift check that fails when a stimulus deviates. Until that lands, treat
+run counts for SQL-assertion gates as a lower bound, and read a `never-attempted` verdict on
+one of them as possibly meaning "this wording has never run" rather than "this gate has
+never run".
+
+### Where the data lives — and why this is not a laptop-only tool
+
+Worth stating plainly, because the opposite is easy to assume:
+
+- **Run *data* is remote.** `msbench-cli extract` is served by the backend — extracting an
+  unknown id reports `Requesting run metadata from remote service`. **Any run id you have
+  access to can be audited from any machine**, free and without tokens. The local
+  `~/Library/Application Support/msbench/runs` directory is a cache, not the source.
+- **Run *discovery* is local-only today.** With no arguments the tool can only enumerate
+  this machine's cache. The CLI already supports `list runs --kusto --created_by
+  --lookback`, which would make discovery team-wide, but the `MSBench User` role does not
+  appear to grant Kusto DB read:
+
+  ```
+  Corp: Principal 'aaduser=…' is not authorized to read database 'ces_telemetry_prod'
+   AME: Principal 'aaduser=…' is not authorized to read database 'msbench'
+  ```
+
+  (`ces-westus3-adx.westus3` and `msbdikustoprodeus2.eastus2` respectively.) That is a
+  concrete, filable access gap and the entire fix for discovery.
+- **Kusto could not answer this question even with access.** The ingested views —
+  `CESBenchmarkInstanceStatusV2View`, `CESBenchmarkRunStatusV2View`,
+  `CESBenchmarkMetricsDedupView`, `CESBenchmarkMetadataDedupView` — carry run and instance
+  status, timings, tags, agent, model and resolved rate. **Per-assertion `details[]` is
+  ingested nowhere.** Gate-level health is only computable from extracted artifacts.
+
+The tool is therefore **run-id-driven and indifferent to provenance**. The day Kusto read
+lands, `msbench-cli list runs --kusto` piped into `npm run gate-health` works with no
+change to the tool. In CI the ids are known by construction anyway.
+
+### Flags
+
+| Flag | |
+| --- | --- |
+| `--extracted <dir>` | Audit an existing extraction; skips `msbench-cli` entirely. Repeatable. |
+| `--min-runs <n>` | Runs required before a verdict counts as confident (default 3). |
+| `--identity gate\|comment` | Gate identity scheme; see above. |
+| `--refresh` | Re-extract even when the cache has the run. |
+| `--json` | Machine-readable report, including the full declared-but-never-seen list. |
+
+Extractions are cached in `.regrade/<run-id>`, shared with
+[`regrade.ts`](#re-grading-a-past-run-for-free), so a run pulled by either tool is already
+on disk for the other.
+
 ## Running in CI
 
 [`.github/workflows/msbench-evals.yml`](../../.github/workflows/msbench-evals.yml) runs