diff --git a/docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md b/docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md new file mode 100644 index 000000000..b2073caa3 --- /dev/null +++ b/docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md @@ -0,0 +1,152 @@ +--- +date: 2026-05-24 +topic: multi-model-plan-review +--- + +# Cross-Model Review: Evaluate the Lever Before Building It + +## Summary + +Before building any cross-model review machinery, run a **four-arm evaluation** that decides whether — and which — review-improvement lever is worth building. A corpus anchored on plans with known post-hoc failures (supplemented by forward-rated docs) is reviewed by four arms (Claude-only; cross-model CLI isolated from the repo; cross-model CLI with a fixed repo-context set; same-model self-critic). A judge with arm labels stripped dedups and classifies findings; a blind-integrity check tests whether the blinding actually held; the human confirms decisive calls and samples judge-rejected findings. Against a **pre-registered** threshold and minimum corpus size, the per-arm result drives a three-way decision: build a lever (which arm), build nothing, or inconclusive. The output is a decision artifact, not a shipped feature. The previously-specified cross-model build (config, setup, challenger-section plumbing) is deferred until the eval justifies a lever. + +--- + +## Problem Frame + +The original version of this document specified building a cross-model "independent" reviewer outright. A live three-model test of that document — reviewed by a six-persona Claude panel, then OpenAI's Codex (`gpt-5.5`) via `codex exec`, then Antigravity/Gemini via `agy` — produced a clear and uncomfortable result. The mechanism worked: each non-Claude model surfaced substantive challenges the Claude panel missed, and the two non-Claude models surfaced *different* things from each other, so the decorrelation is real. But all three reviews converged on the same meta-point: **the value of model-diversity as a review lever is asserted, not demonstrated.** The success criteria measured plumbing ("a critique exists"), not better decisions. + +The test also exposed specific design risks: a challenger denied repo access (stdin-only) may degrade into generic platitudes (Gemini); a cheaper same-model pass might capture most of the value with no egress (both non-Claude models); and CLI auth is fragile — the `gemini` CLI failed live on a missing API key, while `agy` worked with no key. + +The honest move is to stop building and measure. The cost of being wrong about the lever is high (config schema, setup probes, consent flows, per-CLI maintenance for a feature that might no-op on most runs); the cost of an evaluation is low. This document specifies that evaluation — and is built to avoid repeating the original's category error (measuring activity instead of outcomes) one level up. + +--- + +## Actors + +- A1. Human evaluator: assembles/approves the corpus, confirms candidate decision-changing findings, samples judge-rejected findings, and owns the go/no-go decision. +- A2. Orchestrator model (Claude): produces the baseline review and the same-model self-critic arm. +- A3. External reviewer CLIs: `codex` (OpenAI) and `agy` (Antigravity/Gemini) produce the cross-model arms via their non-interactive modes. +- A4. Blinded judge: a model-as-judge that dedups and classifies pooled findings across arms with arm labels stripped. Its model family is disclosed in the result. + +--- + +## Key Flows + +The four arms and the hypothesis each isolates: + +| Arm | What it is | Hypothesis it isolates | +|-----|------------|------------------------| +| (a) Baseline | Claude-only review (current ce-doc-review) | The control | +| (b) Cross-model, isolated | External CLI reviews the doc text, run isolated from the repo (no workspace access) | Does a different model add unique value with no context? | +| (c) Cross-model, fixed context | External CLI reviews the doc plus a fixed, documented repo-context set | Is context-poverty (not the model) the limiting factor? | +| (d) Same-model self-critic | Claude re-reviews in-process (no CLI, no egress) with its own failure modes supplied and prior output hidden | Is the gain the model, or just a fresh adversarial pass? | + +The arm (b)-vs-(c) context delta is the **experimental control** for the model-vs-context comparison — defined and documented before any arm runs, applied identically across every document, so a b/c gap is attributable to context *presence*, not context *curation*. + +- F1. Run the arms + - **Trigger:** Evaluator starts the eval over the assembled corpus, after the threshold/minimum-N and the b/c context rule are pre-registered. + - **Actors:** A1, A2, A3 + - **Steps:** for each corpus document, produce a review from each of the four arms (arm (b) run with repo access stripped; arm (c) given the fixed context set; arm (d) in-process with no egress); capture each arm's raw findings, plus per-arm latency and setup/auth friction. + - **Outcome:** a complete set of per-arm findings for every corpus document. + - **Covered by:** R1, R2, R3, R7 + +- F2. Judge and decide + - **Trigger:** All arms have produced findings for the corpus. + - **Actors:** A1, A4 + - **Steps:** pool all findings with arm labels stripped -> the judge dedups across arms and classifies each finding (unique/duplicate, actionable/generic, decision-changing/not) against the rubric -> run the blind-integrity check (judge attempts to identify each finding's arm) -> the human confirms candidate decision-changing findings and samples judge-rejected ones -> score primarily on the known-failure subset, corroborated by forward-rated counts -> compare to the pre-registered threshold/N -> write the decision artifact (build which arm / build nothing / inconclusive). + - **Outcome:** an evidence-backed three-way decision. + - **Covered by:** R4, R5, R6, R7, R9 + +--- + +## Requirements + +**Eval design** +- R1. Review a defined corpus through four arms: (a) Claude-only baseline; (b) cross-model CLI with no repo context — the external CLI is run isolated from the repo (clean working directory / stripped environment) so it genuinely cannot read the workspace; (c) cross-model CLI with a fixed, documented repo-context set applied identically to every document (not per-document curation); (d) same-model self-critic — Claude re-reviewing the document in-process (no external CLI, no document content leaving the machine) with its own known failure modes supplied and the prior review output hidden. +- R2. The cross-model arms (b, c) invoke the external model via its CLI's non-interactive mode using an argv + stdin pattern (validated this session: `codex exec -s read-only - < promptfile`; `agy --print "instruction" < promptfile`). The Gemini-family arm uses `agy` (no API key required); the `gemini` CLI is avoided because it failed on a missing API key. +- R3. The arm (b)-vs-(c) context delta is the experimental control for the model-vs-context comparison: it is defined and documented before any arm runs and held identical across all documents, so a b/c gap is attributable to context presence rather than context quality. (Arm (d)'s self-critic win, if it occurs, is attributed to the bundled "fresh pass + failure-modes-supplied" intervention and is not decomposed within this eval.) + +**Judging and metric** +- R4. A judge dedups findings across arms and classifies each as unique-vs-duplicate, actionable-vs-generic, and decision-changing-vs-not against a written rubric, with arm labels stripped before it sees the pool. The judge's model family is disclosed in the result; sharing a family with any arm (e.g., a Claude judge with the Claude baseline or self-critic) is flagged as a blind-integrity risk. +- R5. A blind-integrity check is run: the judge attempts to identify each finding's arm. If it identifies arms above chance, the blinding did not hold and the per-arm metric is treated as confounded rather than trusted. +- R6. The human confirms the candidate "unique decision-changing" findings AND samples from the judge-rejected ("generic"/"duplicate") set, so a biased judge cannot silently zero out a cross-model arm before the human sees it. +- R7. The primary go/no-go signal is per-arm performance on the known-post-hoc-failure subset — does an arm surface the finding the failure proved mattered? Forward-rated decision-changing counts on the broader corpus are corroborating, not primary (they measure projected actionability, not validated outcome). Secondary metrics (latency, setup/auth friction, generic/duplicate noise rate) act as tie-breakers and trade-off flags — e.g., a large latency or friction gap between arms with similar yield is surfaced to the human — not as primary inputs. + +**Corpus** +- R8. The corpus is anchored on past plans with known post-hoc failures (the outcome-grounded subset, sourced from `fix-*` plans and regression-referencing docs under `docs/`), supplemented by a sample of forward-rated real plans/brainstorms. A minimum corpus size is committed before running (see R9). If the known-failure subset is too small to carry the decision, the result states that explicitly as a limit on what the eval can conclude. + +**Output and framing** +- R9. The threshold and minimum corpus size are pre-registered — written and committed before any arm runs — so the decision rule is independent of the observed counts. The eval produces a written decision artifact with three possible outcomes: build a lever (which arm), build nothing, or **inconclusive / underpowered (re-run larger)**. The inconclusive outcome is distinct from "build nothing" so an underpowered run cannot masquerade as a confident kill of a lever the live test already showed produces decorrelated value. +- R10. The counted unit is consistently called a "finding" (the atomic observation the judge classifies); "critique" denotes the full set of findings an arm produces. The capability is framed as "cross-model critique," not "independent review" — the independence claim is an overclaim that output-time disclosure does not fix. + +--- + +## Acceptance Examples + +- AE1. **Covers R2.** Given the `gemini` CLI is unavailable for lack of an API key, when the Gemini-family arm runs, then it uses `agy` (no key) instead and the arm still completes. +- AE2. **Covers R4, R6.** Given a finding is raised by more than one arm, when the judge processes the pool, then it is counted once (deduped); and given a finding the judge buckets "generic," when the human samples judge-rejected findings, then it can still be promoted to decision-changing. +- AE3. **Covers R9.** Given the corpus meets the pre-registered minimum N and no arm clears the pre-registered threshold, the decision artifact records "build nothing"; but given the corpus is below minimum N, it records "inconclusive / underpowered," not "build nothing." +- AE4. **Covers R1.** Given the self-critic arm runs, then it produces its review in-process with no external CLI invocation and no document content leaving the machine. +- AE5. **Covers R5.** Given the blind-integrity check, when the judge identifies findings' arms above chance, then the per-arm metric is reported as confounded rather than as a trusted result. + +--- + +## Success Criteria + +- The decision is grounded primarily in per-arm performance on the known-failure subset (validated outcomes), corroborated by forward-rated counts — not by reviewer enthusiasm, the failure mode that triggered this rewrite. +- Blinding integrity is tested, not assumed (R5), and the human samples judge-rejected findings (R6), so a biased judge cannot silently kill a cross-model arm. +- Three outcomes are possible including "inconclusive," so an underpowered run cannot masquerade as a confident "build nothing"; the threshold and N are pre-registered, so the decision is "grounded in counts, not a vibe." +- The result is reproducible enough to re-run as models and CLIs change. +- If a lever wins, the deferred build spec below can be picked up directly, shaped by the winning arm (e.g., "cross-model with context" implies a very different build than "same-model self-critic"). + +--- + +## Scope Boundaries + +### Deferred for later (pending eval outcome) + +- The cross-model review feature itself — the config block, the `ce-setup`/`check-health` changes, the challenger-section rendering and headless plumbing, and the consent/argv/sanitization requirements (the prior R1–R17 of this document). Picked up only if the eval justifies a lever, and shaped by which arm won. + +### Outside this product's identity + +- Shipping a permanent, always-on cross-model reviewer without evidence it produces unique, actionable findings often enough to justify its carrying cost. +- API-key-dependent model access. The eval uses CLI auth (`codex` + `agy`, no key); a build that required per-vendor API keys would be a different product. +- An external challenge for `ce-code-review` — out of scope here and downstream. + +--- + +## Key Decisions + +- Eval-first over build-first: three independent reviews converged that the lever's value is unproven. +- Four arms with a defined b/c context control (isolation for (b), fixed identical context for (c)): isolates whether any gain is the model, the context, or just a fresh adversarial pass — and keeps the b/c contrast interpretable. +- The known-post-hoc-failure subset is the primary signal; forward-rated counts corroborate. This avoids measuring projected actionability instead of validated outcomes. +- Threshold and minimum N are pre-registered before running; "inconclusive / underpowered" is a distinct outcome from "build nothing." +- Hybrid judging with a blind-integrity check, human sampling of judge-rejected findings, and disclosure of the judge's model family — because label-stripping alone does not guarantee a blind, and the bias is directional toward "build nothing." +- The harness is a repeatable evaluation (a script plus a written result under `docs/`), not a new installed skill. +- "Cross-model critique" framing; "finding" is the consistently-used counted unit. +- The Gemini-family arm uses `agy` (no API key); the `gemini` CLI's key requirement is avoided. + +--- + +## Dependencies / Assumptions + +- `codex` and `agy` CLIs are installed and authenticated. Validated live this session: `codex exec -s read-only` (non-interactive, `approval: never`) and `agy --print` (appends stdin to the prompt) both work; the `gemini` CLI failed on a missing `GEMINI_API_KEY`, so `agy` is the Gemini-family path. The build, if it happens, may need a different access route, so cross-model arm results may not transfer one-to-one to an as-built integration. +- A corpus exists in `docs/brainstorms/` and `docs/plans/`. Known-failure cases sourced from `fix-*` plans and regression-referencing docs may still be limited; per R8 the result states the limit if so. +- A model-as-judge carries directional bias (a Claude judge may under-rate non-Claude-shaped findings as "generic," tilting toward "build nothing"). Mitigated — not eliminated — by arm-label stripping, the blind-integrity check (R5), human sampling of judge-rejected findings (R6), and family disclosure (R4). +- The eval runs locally where the CLIs are configured, so its setup/auth-friction metric reflects one already-working machine and does not predict the cross-machine auth fragility a shipped feature would face. Treated as a known limit on the friction metric. + +--- + +## Outstanding Questions + +### Resolve Before Planning + +- (none — the experimental control, metric grounding, pre-registration, and blind-integrity safeguards are now specified at the requirements level; remaining items are execution parameters) + +### Deferred to Planning + +- [Affects R3] The exact repo-context set for arm (c) — which files/scope — applied via the fixed, identical-across-documents rule R3 commits to. +- [Affects R8, R9] The minimum corpus size and the go/no-go threshold values to pre-register, and the corpus sampling method. +- [Affects R4] The exact judging-rubric wording. +- [Affects R5, R6] How blinding, the arm-guessing integrity probe, and judge-rejected sampling are operationalized (label stripping, ordering, sample size). +- [Affects R9] Where the harness script and the decision artifact live, and the script's shape (one runner across arms vs. per-arm scripts). diff --git a/docs/brainstorms/2026-05-28-ce-deep-review-requirements.md b/docs/brainstorms/2026-05-28-ce-deep-review-requirements.md new file mode 100644 index 000000000..1b4aed50c --- /dev/null +++ b/docs/brainstorms/2026-05-28-ce-deep-review-requirements.md @@ -0,0 +1,228 @@ +--- +date: 2026-05-28 +topic: ce-deep-review +--- + +# ce-deep-review: turnkey high-stakes plan review across Claude + non-Claude models + +## Summary + +A new `ce-deep-review` skill that orchestrates the existing three-pass high-stakes-plan review recipe end-to-end on any plan document. It runs the Claude 6-persona panel, opens an interactive consent gate that both authorizes egress and lets the user pick which auto-detected non-Claude models participate (codex, gemini, grok), fans the selected models out across the same six lenses, has the agent verify every cross-model finding against the doc, and writes a reconciled report as a sidecar file next to the plan. Adds Grok Build CLI as a third model in the underlying cross-model harness. + +--- + +## Problem Frame + +The deep-plan-review workflow (Claude panel + cross-model panel + reconcile) is a lever the team is still gathering evidence on. The cross-model eval established that the workflow decorrelates on validated bugs — it surfaces environment, credential, and sequencing failures the Claude panel alone misses — but the decision-grade run's verdict on whether the team-wide value clears the friction-and-egress cost is inconclusive / underpowered. This skill is the instrument that gathers that evidence in real use, not the productionization of a settled win. Running it today is a multi-tool, multi-context workflow: + +1. The agent runs `ce-doc-review` (pass 1 — no egress). +2. The user opens a terminal, pastes `bash scripts/eval/cross_model_review/panel-critique.sh `, waits, and returns the records to the chat (pass 2 — egress, deliberately user-driven because the agent is hard-blocked from egressing proprietary content). +3. The agent reconciles, surfacing the decision-changing union — with a known confabulation risk on gemini findings the user is then asked to verify manually. + +Three pain points compound: + +- **The pass-2 hop is expensive in attention.** Switching to a terminal and running a bash command for every high-stakes plan is enough friction that the deep review gets skipped or deferred when it should not be. +- **Verification is the most error-prone step and is currently manual.** Gemini confabulates plausible-but-fake findings; the user, not the agent, currently checks each cross-model finding against the doc. +- **The workflow assumes a specific operator.** The harness was built for one developer who has codex + gemini + (now) grok installed and authenticated. Other internal developers will not have the same environment, and the current shape gives them no way to run the workflow at all if any one of those tools is missing. + +Without a turnkey entry point that handles egress, verifies findings, and adapts to each developer's installed toolset, the deep review remains a power-user workflow rather than a team-available one. + +--- + +## Actors + +- A1. Plan author / reviewer (any internal developer): invokes `ce-deep-review` on a plan they have authored or want to vet. May or may not have all non-Claude CLIs installed. +- A2. The orchestrating agent (Claude): runs pass 1, mediates the consent gate, dispatches the cross-model arms, verifies cross-model findings against the doc, writes the reconciled report. +- A3. Non-Claude reviewer CLIs (codex, gemini, grok): produce cross-model findings under the same six lenses as the Claude panel; configured per-environment, opt-in per-run via the consent gate. + +--- + +## Key Flows + +- F1. Happy-path deep review with all three non-Claude models available + - **Trigger:** A1 invokes `ce-deep-review `. + - **Actors:** A1, A2, A3 + - **Steps:** + 1. A2 runs the Claude 6-persona panel (no egress). + 2. A2 probes the environment for installed-and-authed non-Claude CLIs; finds all three. + 3. A2 opens the consent gate: shows the three models as a multi-select (all unchecked by default — opt-in per model), previews the resolved plan path / byte count / any detected credential- or PII-shape pattern hits, and confirms permission to egress. + 4. A1 confirms; A2 fans the selected models out across the six lenses. + 5. A2 verifies each cross-model finding against the doc, tagging CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN. + 6. A2 writes the reconciled report to `.deep-review.md` and surfaces a summary in chat. + - **Outcome:** A1 reads a single verified report listing the panel findings plus the decorrelated cross-model additions, each cross-model finding tagged with its verification status. Raw per-model records remain on disk under the existing `/tmp/cmre-panel/records/` path for audit. + - **Covered by:** R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12 + +- F2. Partial-environment deep review (some non-Claude CLIs missing) + - **Trigger:** A1 invokes `ce-deep-review ` on a machine where one of the non-Claude CLIs is not installed or not authenticated. + - **Actors:** A1, A2, A3 (subset) + - **Steps:** + 1. A2 runs the Claude panel. + 2. A2 probes the environment; finds (e.g.) codex + grok installed, gemini missing. + 3. A2 opens the consent gate showing only the available models, with a brief note that the missing model was skipped and why (not installed / not authenticated). + 4. A1 confirms with the available subset. + 5. Remainder proceeds as in F1. + - **Outcome:** A1 gets a deep review using the subset of models available in their environment, without manual configuration. + - **Covered by:** R2, R3, R6, R7, R9 + +- F3. Panel-only deep review when zero non-Claude CLIs are available + - **Trigger:** A1 invokes `ce-deep-review ` on a machine where none of codex, gemini, grok is installed or authenticated. + - **Actors:** A1, A2 + - **Steps:** + 1. A2 probes the environment; finds zero usable non-Claude CLIs. + 2. A2 runs the Claude panel. + 3. A2 writes a sidecar at `.panel-review.md` (distinct filename — `.deep-review.md` is reserved per R14 for verified cross-model output) whose frontmatter sets `coverage: panel-only` and whose header and chat banner state prominently `Panel-only deep review (no cross-model arm)` and name each missing CLI with its install/auth command (e.g., `grok login`, the codex install command, the agy install command). + - **Outcome:** A1 gets the Claude panel work they implicitly asked for AND explicit visibility into what's missing — no silent degrade, no bounce. The header is what defeats silent failure, not refusal to run. + - **Covered by:** R2, R13 + +- F4. User declines egress at the consent gate + - **Trigger:** During F1 step 3, A1 cancels at the consent gate. + - **Actors:** A1, A2 + - **Steps:** + 1. A2 surfaces the Claude panel findings as the deliverable. + 2. A2 does not write a `.deep-review.md` sidecar (the report is panel-only, equivalent to `ce-doc-review` output — no need to duplicate that artifact under a deep-review filename). + - **Outcome:** A1 gets the panel findings without egress. The deep-review filename remains reserved for the verified cross-model report. + - **Covered by:** R2, R14 + +--- + +## Requirements + +**Skill surface and orchestration** + +- R1. Provide a new `ce-deep-review` skill under `plugins/compound-engineering/skills/ce-deep-review/`, separate from `ce-doc-review`. `ce-doc-review` remains the no-egress single-panel review and is unchanged in behavior. +- R2. The skill accepts a single argument: a path to a plan document (markdown). It does not depend on the document being inside this repo. +- R3. Before any of the three-pass recipe runs, the skill probes the environment for available non-Claude CLIs (see R9 and R13). If at least one is available, the skill runs the three-pass recipe end-to-end in a single invocation: Claude panel → consent gate → cross-model panel → cross-model verification → reconciled report. If zero non-Claude CLIs are available, behavior is governed by R13 (panel-only run with explicit header). + +**Cross-model harness extension** + +- R4. Add a `grok` arm to the existing cross-model harness (`scripts/eval/cross_model_review/arms.py` and downstream callers), supporting the Grok Build CLI (`grok` binary). +- R5. Every non-Claude arm (codex, grok, and the agy replacement for gemini) runs in a read-only, no-web-search, no-tools posture — minimum floor is symmetry with the most restrictive of the existing arms (codex's `-s read-only` and gemini's `--approval-mode plan`). The grok arm uses `--permission-mode plan`, `--disable-web-search`, single-turn `-p` invocation, and any `--sandbox` profile validated to deliver that posture. The agy arm posture must be validated separately — `agy --help` exposes no `--approval-mode`/`--permission-mode`/plan-mode equivalent (only `--dangerously-skip-permissions` and a boolean `--sandbox`), so the migration must determine whether `--sandbox` (alone or combined with other agy flags) delivers the same floor; if no combination achieves it, the agy arm is treated as unavailable until it can. Every arm runs from a clean working directory, has no ambient repo access, and produces a JSON array of findings parseable by the existing `parse_findings` logic. Every non-Claude arm is prompted with the same six per-lens rubrics the existing harness uses, so findings are structurally comparable across models. Before R4 is shippable, the grok arm is behaviorally smoke-tested with a unique sentinel prompt to confirm it does not attempt web search, read files outside the working directory, or make follow-up tool calls — `--help` flag presence does not transfer the codex/gemini eval baseline to grok. If the floor cannot be validated for grok, the grok arm is treated as unavailable until it can. + + > **[Phase 0 validation, 2026-05-28 — supersedes the posture assumptions above]** Empirically validated on the original dev machine (see `docs/solutions/skill-design/2026-05-28-{grok,agy}-arm-posture-validation.md`): (1) **agy 1.0.3 has NO flag that delivers this floor** — `--sandbox` restricts terminal execution but NOT filesystem read/write (agy read an out-of-workspace sentinel and wrote a canary under `--sandbox`), and there is no web-search-disable flag. The floor is therefore enforced **externally via a macOS `sandbox-exec` (seatbelt) profile** wrapping the arm, not via agy flags. (2) **grok 0.2.8 is deferred** — its headless `-p` worker fails at the WebSocket-relay auth layer (`Transport channel closed / AuthorizationRequired`), unfixable by `grok login`/`grok agent --reauth`; its sandbox posture (`--sandbox read-only`) is otherwise ideal and ready to land on a grok version that fixes the relay. v1 cross-model arms are therefore **codex + (OS-sandboxed) agy**. +- R6. The harness exposes a way to run a subset of models per invocation (not all-or-nothing). The exact mechanism — argv flag, environment variable, or per-call argument — is a planning decision; what's required at the requirements level is that `ce-deep-review` can select N of the three available models for a given run. Whatever mechanism is chosen must be expressible by the orchestrating agent after the consent gate interaction completes, so the user's per-run model selection from the gate maps directly to the harness invocation. + +**Consent gate and model selection** + +- R7. Before any non-Claude egress, the skill opens a single interactive gate that does three things in one interaction: (a) asks permission to egress, (b) lets the user pick which of the auto-detected available models will participate, (c) previews what is about to be sent — the resolved plan path, byte count, and any detected credential- or PII-shape pattern hits using the `gitleaks` canonical pattern set (or equivalent battle-tested ruleset) as the source of truth, with explicit gate copy noting the preview is best-effort and the user is the final filter. Default selection is none — the user opts in per model rather than opts out, so every model that receives the plan was an explicit per-run choice. When only one non-Claude model is available the gate still asks for explicit egress consent — there's no "single-model fast path" that skips the prompt. +- R8. The gate uses the platform's blocking question tool (e.g. `AskUserQuestion` in Claude Code, with the documented cross-platform fallbacks per the plugin's interaction rules). +- R9. The skill auto-detects which non-Claude CLIs are installed and authenticated before opening the gate, so the gate never lists a model the user cannot actually use. Detection covers both "binary present" and "auth/credentials usable for a non-interactive run" — a CLI that's installed but not logged in is treated as unavailable, the same as a missing binary. The detection probe must not make authenticated API calls to vendor endpoints (an authenticated call would itself be egress before the consent gate fires) — probes use credential-file presence checks, token-expiry inspection, or local CLI dry-run flags that do not contact the vendor's servers. If no offline check exists for a given CLI, that CLI is treated as unavailable rather than probed live. + +**Cross-model verification and reconciliation** + +- R10. After cross-model findings return, the agent verifies each finding against the plan document by locating the cited text or claim, and tags the finding as: + - **CONFIRMED** — the finding is grounded in the doc. The verifier MUST include the quoted matched text inline alongside the tag (a CONFIRMED finding without an inline quote is a validation failure) so the user can audit any verification claim at a glance. + - **NOT-FOUND-IN-DOC** — the finding cites or implies content the doc does not contain (likely confabulation). + - **NEEDS-HUMAN** — the finding is too ambiguous to verify mechanically (e.g., a strategic / aesthetic judgment with no specific text to check against). + Verification applies to every cross-model finding, including ones that overlap with the Claude panel. +- R11. The reconciled report includes: YAML frontmatter with a `coverage:` field (enum: `full` when all available non-Claude arms participated, `reduced-confidence` when a subset participated, `panel-only` when zero non-Claude arms participated) so downstream tooling can distinguish coverage states without parsing prose; a header section identifying the plan, the models that participated, the timestamp, and the invoking user identity (`git config user.name` — not `user.email`, to reduce PII exposure when the sidecar is committed) so the sidecar itself is the durable audit artifact when committed alongside the plan (`/tmp/cmre-panel/records/` raw records are session-scoped and not the system of record); the Claude panel findings (untagged — trusted); the cross-model findings grouped per lens or per source (whichever planning chooses), each with its verification tag and (for CONFIRMED) the inline quoted match plus a pointer into the doc; and a "decision-changing union" section highlighting verified cross-model findings the Claude panel did not surface. When fewer than the full set of non-Claude models participated (single-model run, or zero-CLI panel-only run per R13), the header AND the chat banner explicitly label the run as `Reduced-confidence deep review (N of M non-Claude models)` or `Panel-only deep review (no cross-model arm)` so users can tell at a glance that this is not a full-fan-out review. + +**Output and environment behavior** + +- R12. The reconciled report is written as a sidecar file at `.deep-review.md` (or `.panel-review.md` per R13 when zero non-Claude arms participated). If a previous report exists at that path, the skill rotates the prior file to `.deep-review..md` (or `.panel-review..md`) before writing the new report (preserves audit chain and hand-edited reviewer annotations without blocking re-runs). The skill keeps the 5 most recent rotated sidecars per plan and deletes older rotations during the rotation step, so the audit chain does not accumulate unboundedly in the working tree. Whether to commit or gitignore the sidecar(s) is currently an Open Question (see Outstanding Questions); v1 does not modify `.gitignore` either way. +- R13. When zero non-Claude CLIs are detected as available, the skill runs the Claude panel and writes a sidecar at `.panel-review.md` (distinct from `.deep-review.md` so the filename itself encodes the no-cross-model-arm property — see R14's reservation) whose frontmatter sets `coverage: panel-only` and whose header and chat banner state prominently `Panel-only deep review (no cross-model arm)` and name each missing CLI with its install or auth command. The skill does not silently degrade — the distinct filename AND prominent header together defeat silent failure. Refuses to be quiet, not refuses to run. +- R14. When the user declines egress at the consent gate, the skill outputs the Claude panel findings to chat and does NOT write the sidecar file. The `.deep-review.md` filename is reserved for verified cross-model output. + +**Progress and latency** + +- R15. The skill streams per-(model, lens) progress to chat so the user can see the run advancing during the multi-minute pass-2 phase. The exact streaming format (per-call summary, lens-completion ticks, etc.) is a planning detail; at the requirements level the skill must not run silently for minutes. + +--- + +## Acceptance Examples + +- AE1. **Covers R7, R9, R12, R14.** Given a plan at `docs/plans/foo.md` and an environment with codex + gemini + grok all installed and authed, when the user runs `ce-deep-review docs/plans/foo.md` and confirms the consent gate with all three models selected, then the skill runs the Claude panel, the three non-Claude models across six lenses each, verifies every cross-model finding against `foo.md`, and writes `docs/plans/foo.md.deep-review.md` containing the panel findings (untagged), verified cross-model findings (each tagged CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN), and a decision-changing-union section. +- AE2. **Covers R7, R9.** Given an environment where gemini is installed but not authenticated, when the user invokes the skill, then the consent gate lists only codex + grok, includes a brief one-line note that gemini was skipped because it is not authenticated, and proceeds normally when the user confirms. +- AE3. **Covers R13.** Given an environment where none of codex, gemini, or grok is installed, when the user invokes the skill, then the skill runs the Claude panel and writes `.panel-review.md` (distinct from the `.deep-review.md` filename reserved for cross-model output) with frontmatter `coverage: panel-only`, whose header reads `Panel-only deep review (no cross-model arm)` and names each missing CLI with its install / auth command; the chat banner restates the same notice. +- AE4. **Covers R10, R11.** Given a cross-model finding from gemini that claims the plan "does not validate user input on line 42" but the plan has no such claim or relevant content, when the agent verifies, then the finding is tagged NOT-FOUND-IN-DOC in the reconciled report. The finding is still included in the report (not silently dropped) so the user can see what the model produced. +- AE5. **Covers R14.** Given a plan, when the user reaches the consent gate and cancels, then the skill prints the Claude panel findings to chat, does NOT write `.deep-review.md`, and exits cleanly. + +--- + +## Success Criteria + +- A high-stakes plan that previously required three separate operator actions (run `ce-doc-review`, run `panel-critique.sh`, ask the agent to reconcile and then manually verify) can be reviewed via a single `ce-deep-review ` invocation, with one consent interaction in the middle and a verified sidecar report at the end. +- The skill works for any internal developer who has at least one of codex / gemini-or-agy / grok installed and authenticated, without per-repo configuration. Developers who have none of them get a panel-only deep review (per R13) with a distinct filename and prominent banner, not a silent fallback. Note: per R9's no-live-API-call constraint, if any of the three non-Claude arms lacks an offline auth check, the team-available set is correspondingly narrower than "installed and authed" alone implies — planning must treat "has an offline auth check" as an arm-acceptance criterion, not an afterthought. +- The verification step catches gemini-style confabulations: a finding citing text not in the doc lands as NOT-FOUND-IN-DOC in the report, so the user does not have to do that check themselves. +- `ce-plan` can implement this from the requirements doc without inventing user-visible behavior, output format, or environment-detection policy. + +--- + +## Scope Boundaries + +- The evaluation-specific components of the cross-model harness (judge, trials, GT-match, decision-artifact, record-schema) are not invoked by `ce-deep-review`. The arms and harness runner are extended and reused. The evaluation pipeline exists to *decide whether the cross-model lever is worth building*; this skill is the day-to-day use of the lever. Future eval re-runs continue to use the existing harness directly. +- Per-plan or per-repo trust-based allow-listing (e.g., "this plan never goes to Google regardless of who runs the skill") is out of scope. Model selection at the consent gate covers the common case ("I don't want gemini today") but is not policy enforcement. +- A persistent / per-repo configuration file for default model selection (e.g., "always include grok, never include gemini for this repo") is out of scope. Default is "all auto-detected available models pre-selected," with the user free to deselect at the gate per run. +- Cost or token-budget estimation in the consent gate (e.g., "this will use ~X tokens / cost ~$Y") is out of scope. +- A headless / non-interactive mode is out of scope for v1. Egress without explicit consent for each run is exactly what the consent gate exists to prevent. +- Extending the consent-gate / cross-model pattern to `ce-code-review` (code PRs) or any other artifact type is out of scope. This brainstorm is for plan and requirements documents. +- A new non-Claude judge / arbitration step inside the deep-review flow is out of scope. The Claude agent is the reconciler. + +--- + +## Key Decisions + +- **One-click consent gate over full-auto egress.** Collapses the manual bash hop into a single keystroke without removing the human's explicit decision to send plan content to external vendors. Rationale: the original design's prohibition on agent-driven egress was a content-protection choice, not an API-safety choice; collapsing the friction without removing the explicit decision preserves the protection. +- **Full agent-side verification of every cross-model finding over emit-and-flag.** The user reads a verified list, not a raw dump labeled "needs verification." Rationale: gemini's confabulation problem is well-documented in the cross-model eval, and verification is the most error-prone manual step in the current workflow. A turnkey command that leaves verification to the user collapses two of three manual steps but leaves the most error-prone one in place. +- **New `ce-deep-review` skill over a third mode on `ce-doc-review`.** Two clear entry points (no-egress vs. deep) with intent-named slash commands, instead of a single skill with three modes whose interaction logic gets harder to reason about. Rationale: `ce-doc-review` already supports interactive and headless modes; piling on a third mode would muddy the interaction model. +- **Sidecar output (`.deep-review.md`) over chat-only.** Durable, diffable across iterations, commitable if the team chooses. Rationale: high-stakes plans benefit from a durable artifact attached to the plan, not an ephemeral chat output. +- **Auto-detect-and-deselect consent gate over per-provider trust allow-list.** The selection mechanism is driven by availability across developer environments, not by data-handling trust. Rationale: the immediate pain is "other developers don't have everything you have"; trust-based gating is a separate problem worth deferring until a proprietary-plan flow demands it. +- **Refuse to be quiet, not refuse to run, when zero non-Claude CLIs are available.** `ce-deep-review` still runs the Claude panel and writes a sidecar, but the sidecar header and chat banner state prominently that the cross-model arm did not run and name the missing CLIs with install/auth pointers (R13). Rationale: the user-visible property the workflow protects is "the user knows whether the cross-model step happened." Refusing to run delivers that property by bouncing the user; refusing to be quiet delivers the same property without losing the panel work and without bouncing first-time teammates who don't yet have non-Claude CLIs installed. Same pattern extends to single-model runs via R11's reduced-confidence header. +- **One-click consent gate is opt-in per model, not opt-out.** The default model selection at the consent gate is "none checked" — the user opts in per model rather than rubber-stamping a pre-checked list. Rationale: a default-affirmative UI on a multi-vendor egress prompt invites click-through on sensitive content; opt-in per model preserves the explicit-per-vendor decision property the terminal-typing friction used to deliver, without bringing back the bash hop. +- **Inline-quote requirement on CONFIRMED verification tags.** Every CONFIRMED tag carries the matched text quoted inline (R10). Rationale: agent-as-verifier has its own confabulation modes when grounding findings against long documents, and a false-CONFIRMED finding launders confabulation as verified. Inline quoting is the cheapest spot-check surface; pairing it with a planned bidirectional rate measurement (see Outstanding Questions) commits the team to closing the validation gap. +- **v1 is fully turnkey rather than a thinner wrapper.** Rationale: the friction itself is hypothesized to be the main thing suppressing usage and therefore evidence — a thin wrapper (panel + consent gate + bash-handoff to existing harness, no new arm, no verification step) would not test that hypothesis. Risk acknowledged: if the lever does not clear the value bar after v1, the agy migration coordination + grok hardening + verification accuracy work do not pay back. If post-v1 evidence shows the friction was not the bottleneck, the thinner-wrapper alternative remains available. +- **Migration sequence option (a): migrate gemini → agy first; ship v1 with agy as the canonical arm.** Rationale: option (a) lands cross-model parity fastest and avoids shipping a known-dead arm (option b) or losing gemini's eval-validated decorrelation contribution (option c). **Calendar fallback:** if any of the three Pre-v1 Ship Gates (grok smoke test, grok sandbox profile, agy posture-floor) is not green by 2026-06-15, fall back to option (c) — ship v1 without gemini/agy and add the arm post-migration. 3 days of slack before the 2026-06-18 hard cutoff. +- **Antigravity (agy) and xAI (grok) data-handling policies confirmed in scope for internal plan content.** Both vendors' standard policies cover internal Blueprint plan content for `-p`-style API invocations; no further legal confirmation is a blocker for v1. +- **Bidirectional verifier rate thresholds: ≤5% false-CONFIRM AND ≤5% false-NOT-FOUND-IN-DOC on the held-out set.** Both must clear before R10 ships. **Consequence of missing the false-CONFIRM threshold:** all cross-model findings tag as NEEDS-HUMAN by default until the rate drops below threshold. **Consequence of missing the false-NOT-FOUND-IN-DOC threshold:** the verification step's NOT-FOUND-IN-DOC tag is treated as advisory, not authoritative, until the rate drops below threshold (findings remain in the report regardless of tag). + +--- + +## Dependencies / Assumptions + +- The cross-model harness in `scripts/eval/cross_model_review/` (`arms.py`, `panel-critique.sh`, the six lens rubrics) is the basis for pass 2 and stays in place. `ce-deep-review` builds on it; this brainstorm does not redesign it. +- Grok Build CLI is installed at `~/.grok/bin/grok` on the original developer's machine and exposes the documented headless-friendly flags (`-p`, `--permission-mode plan`, `--disable-web-search`, `--output-format`, `--prompt-file`, etc.). The `grok login` flow handles authentication; there is no documented env-var-keyed auth path. Verified against `grok --help` on 2026-05-28; rerun the check if a future grok version changes the surface. xAI's data-retention policy for `-p` invocations is not documented here and is recorded as an unverified assumption — Resolve Before Planning includes an item to confirm or reject the policy explicitly before the grok arm enters the team-facing trust boundary. + + > **[OD-3 resolved, 2026-05-28]** The grok `-p` data-retention question is **resolved: CONFIRMED acceptable** for internal Blueprint plan content — grok stays in the consent gate when it re-enters. (Independently, grok is **deferred from v1** on the 0.2.8 headless relay-auth bug — see the R5 Phase-0 note — so retention only re-matters on a grok version that fixes the relay.) +- **Gemini CLI is being sunset; the existing `gemini` arm migrates to Antigravity (`agy`) before 2026-06-18.** The Gemini CLI backend endpoints return HTTP 410 after that date. The harness already has a stub `agy` invocation path (currently marked unreliable from prior eval runs); the migration scope covers: (a) revalidating the `agy` arm against the same eval baseline that established gemini's behavior, (b) updating env vars (`GEMINI_API_KEY` → `AV_API_KEY`, `GEMINI_PROJECT_ID` → `AV_PROJECT_ID`, `GEMINI_REGION` → `AV_REGION`), (c) replacing the `gemini` binary invocation with `agy`, (d) running `agy plugin import gemini` to carry across any extension state, (e) updating MCP config from inline-in-`settings.json` to a dedicated `mcp_config.json`. Until the migration lands, `ce-deep-review` continues to refer to the arm by its current name (`gemini`); references in this document treat `gemini` and "the agy replacement for gemini" as the same arm slot. + + > **[Phase 0 validation, 2026-05-28]** Migration step (b) is corrected: agy uses OAuth (`~/.gemini/oauth_creds.json`), **not** `AV_API_KEY`/`AV_PROJECT_ID`/`AV_REGION` env vars — there is nothing to migrate there. Critically, the "stub agy invocation path currently marked unreliable" reflected agy **1.0.2** (empty output / monologue); **agy 1.0.3 is a viable reviewer** (clean JSON findings, doc via stdin), so the migration is unblocked on viability grounds. The remaining work is the posture floor (OS seatbelt sandbox per the R5 Phase-0 note), not viability. +- The Claude 6-persona panel logic in `ce-doc-review` is reusable from `ce-deep-review`. Whether `ce-deep-review` invokes `ce-doc-review` internally or replicates its panel dispatch is a planning decision; either path satisfies the requirements. +- Raw per-model records continue to write to `/tmp/cmre-panel/records/` (the existing harness output path). This brainstorm does not change that path. +- **Assumption: agy's confabulation profile is similar enough to gemini's for R10's verification design to transfer.** Characterization against the gemini held-out set is not a v1 ship gate; instead, the team monitors R10's bidirectional rate measurements in v1 use and revises the verification strategy if observed behavior diverges materially from gemini's profile. +- **agy offline auth-detection (corrected by Phase 0 validation, 2026-05-28).** The earlier `AV_API_KEY`/`AV_PROJECT_ID` env-var assumption is **wrong** — agy 1.0.3 uses OAuth credentials at `~/.gemini/oauth_creds.json` (no env vars). The R9 offline rule is: "available" iff that file exists, is non-empty JSON, and contains a non-empty `refresh_token`. **Do NOT gate on `expiry_date`** — agy auto-refreshes via the refresh token (observed working with an `expiry_date` ~52h stale), so an expiry check would false-negative. See `docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md`. + +--- + +## Outstanding Questions + +### Resolve Before Planning + +- None — all 10 prior Resolve-Before-Planning items were resolved during the brainstorm's Phase 4 walkthrough. See Pre-v1 Ship Gates and Key Decisions for committed resolutions and the Dependencies / Assumptions section for explicit assumptions. + +### Pre-v1 Ship Gates + +These three validation tasks gate R4 (or its equivalent arms) shipping. All three must pass before `ce-deep-review` v1 ships. The 2026-06-15 cut-line below (RBP 2 resolution) is the calendar fallback trigger. + +- [Affects R4, R5] **Grok behavioral smoke test (pre-v1).** Design and run a sentinel-prompt smoke test that confirms `--permission-mode plan` + `--disable-web-search` + single-turn `-p` deliver the claimed read-only, no-web-search, no-tools posture at runtime — not just at flag-parse time. If grok fails (attempts web search, reads outside the working directory, or makes follow-up tool calls), grok is removed from v1. +- [Affects R5] **Grok `--sandbox` profile evaluation (pre-v1).** Determine the right sandbox profile combination for grok before R4 ships; document the chosen flag set in R5 so the floor is captured before implementation begins. +- [Affects R5, agy migration] **agy posture-floor validation (pre-v1).** Determine whether `agy --sandbox` (alone or combined with other agy flags) delivers a posture symmetric with codex `-s read-only` and grok `--permission-mode plan`. If no combination achieves the floor by 2026-06-15, agy is removed from v1 (reverts the migration sequence to option (c) per the calendar fallback). + +### Deferred to Planning + +- [Affects R6][Technical] How does the harness expose per-run model selection — a `--models codex,grok` flag on `arms.py` / `panel-critique.sh`, an environment variable, or a separate orchestrator path that the skill drives? Planning should pick whichever requires the smallest change to the existing harness while letting `ce-deep-review` request a subset, subject to R6's orchestrator-expressible constraint. +- [Affects R9][Technical] Offline auth-state probe for codex and grok (agy's offline check is set to env-var-presence per the Dependencies assumption). Each CLI exposes auth differently; planning picks a probe per CLI subject to R9's no-live-call constraint. +- [Affects R3, R15][Technical] Whether to run lenses or models in parallel during pass 2, and how to stream progress to the user. The current `panel-critique.sh` runs them sequentially with a per-(model, lens) log line. Parallelism cuts wall time but complicates progress streaming and error attribution. +- [Affects R10][Technical] Verification strategy implementation — grep for cited strings, semantic match, prompt-the-agent-to-search. Planning chooses based on accuracy under cross-model confabulation patterns; bidirectional rates (per Key Decisions) gate v1 ship. +- [Affects R11, R13][Technical] Header / banner copy. The exact wording of `Reduced-confidence deep review (N of M non-Claude models)` and `Panel-only deep review (no cross-model arm)` banners is a planning decision; what's required at the requirements level is that the labels are unambiguous about which arm did and did not run. +- [Affects R7][Technical] `gitleaks` pattern set maintenance lifecycle (per R2 F13). Planning owns the cadence at which the canonical pattern set is refreshed. +- [Affects R7, Key Decisions][Tradeoff] **Does opt-in-none default survive the friction-suppresses-evidence premise?** Round-1 chose opt-in-none for safety; Round-2 personas argued the multi-click-per-run reintroduces friction in a different place, suppressing the very evidence the skill exists to gather. Revisit at planning: keep opt-in-none, flip to opt-out gated on R7 content-preview hits, or pick a hybrid. +- [Affects R11, R12, Key Decisions][Tradeoff] **Is the sidecar the durable audit artifact (commit it) or an LLM-output side-effect (gitignore by default)?** Round-1 added both audit-metadata (R11) AND a gitignore-offer (R12) — Round-2 personas surfaced that the two framings can't both hold. Resolve at planning by picking a canonical role and dropping the conflicting requirement. (R12 currently states v1 does not modify `.gitignore` either way; the question is whether the sidecar's intended fate is commit-able audit or gitignored output.) +- [Affects R7, R12, R13][Deferred] Whether gemini-or-agy should stay in the default checked set after the confabulation-vs-decorrelation trade-off is re-examined post-migration. The brainstorm Apply pass's mooted gemini-default question lands here once the agy migration determines whether the new arm exhibits the same confabulation profile. + +### Deferred to Planning + +- [Affects R6][Technical] How does the harness expose per-run model selection — a `--models codex,grok` flag on `arms.py` / `panel-critique.sh`, an environment variable, or a separate orchestrator path that the skill drives? Planning should pick whichever requires the smallest change to the existing harness while letting `ce-deep-review` request a subset, subject to R6's orchestrator-expressible constraint. +- [Affects R9][Technical] What offline auth-state check exists for each CLI (codex / gemini-or-agy / grok)? Each exposes auth differently. Planning picks a probe per CLI subject to R9's no-live-call constraint (credential-file presence, token-expiry inspection, or local CLI dry-run flag that does not contact the vendor). CLIs without an offline check are treated as unavailable. +- [Affects R3, R15][Technical] Whether to run lenses or models in parallel during pass 2, and how to stream progress to the user. The current `panel-critique.sh` runs them sequentially with a per-(model, lens) log line. Parallelism cuts wall time but complicates progress streaming and error attribution. +- [Affects R10][Technical] What strategy does the agent use to verify a finding against the doc — grep for cited strings, semantic match, prompt-the-agent-to-search? Planning chooses based on accuracy under cross-model confabulation patterns specifically (gemini today, whatever agy exhibits post-migration). +- [Affects R11, R13][Technical] Header / banner copy. The exact wording of `Reduced-confidence deep review (N of M non-Claude models)` and `Panel-only deep review (no cross-model arm)` banners is a planning decision; what's required at the requirements level is that the labels are unambiguous about which arm did and did not run. +- [Affects R7, R12][Deferred] Whether gemini-or-agy should stay in the default checked set after the confabulation-vs-decorrelation trade-off is re-examined post-migration. The mooted question from the brainstorm walk-through (`Drop the confabulator alternative`) lands here once the agy migration determines whether the new arm exhibits the same confabulation profile. diff --git a/docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md b/docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md new file mode 100644 index 000000000..f9793b22a --- /dev/null +++ b/docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md @@ -0,0 +1,334 @@ +--- +title: "feat: Four-arm evaluation harness for cross-model plan review" +type: feat +status: active +date: 2026-05-24 +origin: docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md +--- + +# feat: Four-Arm Evaluation Harness for Cross-Model Plan Review + +## Summary + +Build a repeatable evaluation harness that decides whether — and which — review-improvement lever is worth building, before any cross-model review machinery ships. It reviews a corpus of repo docs through four arms (Claude-only baseline; cross-model CLI isolated from the repo; cross-model CLI with a fixed repo-context set; same-model self-critic), runs multiple trials per arm because model output is non-deterministic, scores findings with a per-finding blinded judge plus human confirmation, and writes a three-way decision artifact (build a lever / build nothing / inconclusive). The cross-model arms run via a Python runner (`codex` / `agy`); the baseline, self-critic, and judge run as in-process subagent dispatches (no `claude -p`, no egress for the self-critic arm). + +--- + +## Problem Frame + +A live three-model test of the original cross-model-review requirements (a six-persona Claude panel, then Codex `gpt-5.5`, then Antigravity/Gemini via `agy`) showed the mechanism produces decorrelated findings but converged that model-diversity's *value* is asserted, not demonstrated. Building the config/setup/check-health plumbing for an unproven lever is expensive; an evaluation is cheap. This plan implements that evaluation, and is built to avoid the original's category error (measuring activity, not outcomes) — the go/no-go is grounded primarily in whether an arm catches the finding a *known post-hoc failure* proved mattered, not in how many findings a reviewer says look actionable. + +Affected parties: the plugin maintainers (who act on the decision), and — only if a lever wins — future users of the deferred cross-model feature. This harness ships no user-facing surface. + +--- + +## Requirements Trace + +Origin requirements (see origin: docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md): + +- R1 (four arms: (a) baseline, (b) isolated, (c) fixed-context, (d) self-critic) → U2 (shared record store), U3 (arms b, c), U4 (arms a, d), U5 (judge pooling) +- R2 (CLI argv+stdin invocation; `agy` as keyless Gemini path) → U3 +- R3 (b/c context control: isolation for b, fixed identical context for c) → U3 +- R4 (blinded judge: dedup/classify, family disclosure) → U5 +- R5 (blind-integrity check) → U5 +- R6 (human confirms + samples judge-rejected) → U6 +- R7 (primary = known-failure subset; forward-rated corroborating; secondary metrics as tie-breakers) → U6 +- R8 (corpus: known-failure-anchored + forward-rated; minimum N; state the limit if thin) → U1, U6 +- R9 (pre-registered threshold + minimum N; three-way decision artifact) → U1, U6, U7 +- R10 ("finding" as the counted unit; "cross-model critique" framing) → U5, U7 + +Research-driven hardenings (not in origin; methodology requirements surfaced during planning): + +- H1. **≥3 trials per (document × arm)**, with variance/determinism reported. Single-run model arms produce confidently-wrong, reversed conclusions (see Sources). → U2, U6 +- H2. **Negative-control document** that no arm should flag; movement on it signals a harness stability problem. → U1, U6 +- H3. **Per-finding, independent (non-batched) judge dispatch** — batching recreates the cross-finding bias blinding exists to escape. → U5 +- H4. **Circuit breaker + per-arm timeout** for CLI arms; structured self-report; no orchestrator re-verification. → U2, U3 + +--- + +## High-Level Technical Design + +*This illustrates the intended approach and is directional guidance for review, not implementation specification. The implementing agent should treat it as context, not code to reproduce.* + +The harness has two cooperating halves, because two arms shell out and two do not: + +| Arm | Producer | Mechanism | +|-----|----------|-----------| +| (a) Claude baseline | Orchestrator | in-process subagent dispatch (ce-doc-review pattern) | +| (b) Cross-model, isolated | Python runner | `codex exec` / `agy --print` in a clean working dir / stripped env | +| (c) Cross-model, fixed context | Python runner | same CLIs + a fixed documented context set | +| (d) Same-model self-critic | Orchestrator | in-process subagent dispatch; own failure modes supplied, prior output hidden; no CLI, no egress | +| Judge | Orchestrator | per-finding blinded subagent dispatch, arm labels stripped | + +Flow: orchestrator + Python runner produce per-(document × arm × trial) finding records into a shared run store → orchestrator pools findings, strips arm labels, dispatches the per-finding judge + the blind-integrity probe → human confirms decision-changing candidates and samples judge-rejected findings → scoring aggregates per-arm on the known-failure subset (primary), forward-rated corpus (corroborating), variance, and secondary metrics → compared against the pre-registered threshold/N → decision artifact written. + +``` +pre-registration (threshold, corpus N, trials N>=3, b/c context rule) [U1] + | + v +corpus --> for each doc, for each arm, for each trial >=3: [U2 store] + arm (a),(d): orchestrator subagent dispatch [U4] + arm (b),(c): python runner -> codex/agy (timeout, breaker) [U3] + | + v +pooled findings -> strip arm labels -> per-finding blinded judge [U5] + | + blind-integrity probe + v +human confirms decision-changing + samples judge-rejected [U6] + | + v +score: known-failure subset (primary) + forward-rated (corroborating) [U6] + + variance + secondary metrics vs pre-registered threshold/N + | + v +decision artifact: build / build nothing / inconclusive [U7] +``` + +--- + +## Output Structure + +``` +scripts/eval/ + cross_model_review/ + run_arms.py # Python runner: CLI arms (b,c), record store, timeout/circuit-breaker, aggregation + arms.py # per-arm invocation (codex/agy argv+stdin; isolation for b; context for c) + judge_rubric.md # anchored 0/25/50/75/100 rubric passed verbatim to the judge + README.md # how to pre-register and run; what each arm is +tests/ + cross-model-review-eval.test.ts # Bun.spawn(["python3", ...]) over fixtures; deterministic carrier only + fixtures/cross-model-review/ # corpus manifest, negative-control doc, sample records +docs/ + # written at run time; location/shape decided in U7 (see Open Questions) +``` + +The tree is a scope declaration, not a constraint — the implementer may adjust layout. Per-unit `**Files:**` are authoritative. Per-run scratch (per-arm raw outputs, iteration dirs) lives in OS temp, not the repo, per AGENTS.md scratch rules; only the corpus manifest, fixtures, and the final decision artifact are tracked. + +--- + +## Key Technical Decisions + +- **Python runner for the CLI arms + aggregation; orchestrator subagent dispatch for the in-process arms and judge.** The repo's "prefer Python over bash for multi-CLI pipeline scripts" learning and the `ce-gemini-imagegen/scripts/*.py` precedent make Python right for the timeout/degradation/CLI-orchestration half; the baseline/self-critic/judge are produced in-process exactly as ce-doc-review dispatches persona reviewers — there is no `claude -p` pattern in this repo, and arm (d) forbids egress (see origin R1, R4). +- **`scripts/eval/` home** (mirroring the only non-skill tooling precedent, `scripts/release/`); this is explicitly NOT a new installed skill (origin Key Decisions). +- **≥3 trials per arm; variance is a headline signal, not just rate.** Two independent learnings document single-run model arms producing reversed, confidently-wrong conclusions. The harness reports per-arm determinism alongside the finding counts (H1). +- **Per-finding, independent, blinded judge** with anchored `0/25/50/75/100` scores (not continuous floats) — the repo abandoned continuous confidence as un-self-calibratable, and batching the judge recreates the cross-finding bias blinding exists to escape (H3, R4, R5). +- **Known-failure subset is the primary signal; forward-rated counts corroborate** (origin R7) — this is what keeps the eval from repeating the "measure enthusiasm, not outcomes" error. +- **Circuit breaker + per-arm timeout; trust each arm's structured self-report, verify the whole at the end** — one broken/auth-failed CLI must not hang or poison the run (H4). +- **`agy` is the Gemini-family arm** (keyless); the `gemini` CLI is avoided (failed live on a missing key) (origin R2). +- **Tests cover the deterministic carrier only** (argv/stdin assembly, timeout/circuit-breaker fallback, record-record JSON shape, label-stripping) via `Bun.spawn(["python3", ...])`; model-arm quality is validated by the human-confirmation step, not unit tests — and assertions check structure, not prose. + +--- + +## Implementation Units + +### U1. Corpus assembly + pre-registration + +**Goal:** Assemble the evaluation corpus and write the pre-registration record before any arm runs. + +**Requirements:** R8, R9, H2 + +**Dependencies:** none + +**Files:** +- Create: `scripts/eval/cross_model_review/README.md` (pre-registration + run instructions) +- Create: `tests/fixtures/cross-model-review/corpus-manifest.json` (corpus doc list, tagged known-failure vs forward-rated, plus the negative-control doc) + +**Approach:** Curate the known-failure subset from `fix-*` plans and regression-referencing docs under `docs/`, plus a forward-rated sample of real plans/brainstorms. Tag each corpus entry with its subset and, for known-failure docs, the specific issue the failure proved mattered (so the judge can later check whether an arm surfaced it). Include one negative-control document that no arm should flag. The pre-registration record fixes — before running — the go/no-go threshold, the minimum corpus N, the trials-per-arm N (≥3), and the fixed arm-(c) context rule, so the decision rule is independent of observed counts. + +**Patterns to follow:** the `safe-auto-rubric-calibration` fixture/intent-file layout (see Sources). + +**Test scenarios:** +- `Covers AE3.` corpus-manifest with fewer than the pre-registered minimum N is detectable as below-N (drives the "inconclusive" outcome downstream). +- Each known-failure entry carries the "issue that mattered" field; forward-rated entries do not require it. +- The negative-control doc is present and tagged as control. + +**Verification:** the manifest parses, every entry is tagged, the pre-registration record names threshold + minimum N + trials N + the arm-(c) context rule. + +### U2. Python runner skeleton: record store, timeout, circuit breaker + +**Goal:** The deterministic carrier — arm dispatch framework, per-(doc × arm × trial) result records, per-arm timeout, circuit breaker, latency capture, run output dirs. + +**Requirements:** R1, H1, H4 + +**Dependencies:** U1 + +**Files:** +- Create: `scripts/eval/cross_model_review/run_arms.py` +- Create: `tests/cross-model-review-eval.test.ts` + +**Approach:** Iterate corpus × arms × trials (N≥3). Each arm invocation returns a structured record (arm, doc, trial, findings[], latency, status: ok|degraded|timeout). A per-arm timeout bounds each call; a circuit breaker disables an arm after 3 consecutive failures and records remaining trials as `degraded` rather than hanging the run. Per-run raw outputs go to OS temp (`mktemp -d`); the tracked output is the aggregated record store. The runner orchestrates the CLI arms directly and accepts externally-produced records for the in-process arms (U4) into the same store. + +**Execution note:** Start with a failing test for the timeout/circuit-breaker fallback (an arm that errors 3× is recorded `degraded`, the run continues). + +**Patterns to follow:** `plugins/compound-engineering/skills/ce-demo-reel/scripts/capture-demo.py` (`subprocess.run(..., timeout=, check=False)` + `except subprocess.TimeoutExpired`); `tests/session-history-scripts.test.ts` (`Bun.spawn(["python3", ...])`). + +**Test scenarios:** +- Happy: a stubbed arm returning findings produces a well-formed record per trial. +- Edge: trials N is honored (3 trials → 3 records per doc×arm). +- Error: an arm that times out is recorded `timeout`, not crashing the run. +- Error: 3 consecutive arm failures trip the circuit breaker; subsequent trials recorded `degraded`; other arms unaffected. +- Record-store JSON has the required fields (arm, doc, trial, findings, latency, status). + +**Verification:** `bun test tests/cross-model-review-eval.test.ts` passes; a dry N=1 run over a 1-doc fixture produces records for all arms or records their `degraded` status. + +### U3. Cross-model arms (b, c) via codex + agy + +**Goal:** Invoke the external CLIs for the isolated (b) and fixed-context (c) arms. + +**Requirements:** R1, R2, R3, H4 + +**Dependencies:** U2 + +**Files:** +- Create: `scripts/eval/cross_model_review/arms.py` + +**Approach:** Both arms pass the document + the challenge rubric via stdin using argv lists (no shell-string interpolation) — validated forms: `codex exec -s read-only -` (stdin) and `agy --print ""` (stdin appended). Arm (b) runs the CLI isolated from the repo (clean working dir / stripped environment) so it genuinely has no workspace context; `codex exec` defaults to in-repo, so isolation is explicit and required. Arm (c) supplies the fixed, documented context set from U1's rule, applied identically to every document. The Gemini-family arm uses `agy` (keyless). Where available, use the CLIs' JSON output (`codex exec --json`) to capture cost/tool-call signal. Each invocation returns the structured record U2 expects. + +**Execution note:** Start with a failing test asserting argv-array + stdin assembly (no interpolation) and that arm (b)'s invocation carries the isolation flags/env. + +**Patterns to follow:** `ce-work-beta/references/codex-delegation-workflow.md` (stdin via `-`, `mktemp` scratch, background-launch + separate-call polling for long runs); argv lists per the command-injection-avoidance learning. + +**Test scenarios:** +- `Covers AE1.` when `gemini` is unavailable/keyless, the Gemini-family arm uses `agy` and still produces a record. +- argv assembly contains no interpolated document content (document goes via stdin only). +- arm (b) invocation includes repo-isolation (clean dir / stripped env); arm (c) includes the fixed context set. +- a slow CLI invocation is cut at the per-arm timeout (U2) and recorded, not left hanging. + +**Verification:** a dry run over the 1-doc fixture yields b and c records (or `degraded`), with b demonstrably context-isolated and c carrying the context set. + +### U4. In-process arms (a baseline, d self-critic) + +**Goal:** Produce the Claude baseline and the same-model self-critic reviews via subagent dispatch, feeding records into the U2 store. + +**Requirements:** R1 (arms a, d) + +**Dependencies:** U2 + +**Files:** +- Create: `scripts/eval/cross_model_review/judge_rubric.md` (shared rubric; also used by U5) +- Modify: `scripts/eval/cross_model_review/README.md` (document the orchestrator-driven arm-production step) + +**Approach:** Arm (a) is the current Claude review behavior over each corpus doc (an in-process subagent dispatch, mirroring how ce-doc-review dispatches reviewers). Arm (d) is a self-critic dispatch: the model reviews the same doc with its own known failure modes supplied and the prior review output hidden, in-process, with no external CLI and no document content leaving the machine. Both emit findings in the same record shape U2 stores. Because these are orchestrator-driven (not pure Python), the README documents the run step that produces them and writes their records into the shared store. A self-critic "win" is attributed to the bundled "fresh pass + failure-modes-supplied" intervention and is not decomposed within this eval (per origin R3). + +**Patterns to follow:** `ce-doc-review/SKILL.md` Phase 2 dispatch + `references/subagent-template.md` (persona prompt assembly). + +**Test scenarios:** +- `Covers AE4.` the self-critic arm runs with no external CLI invocation and no document egress (assert the run step issues no `codex`/`agy`/network call for arm d). +- baseline and self-critic records conform to the shared record shape. +- the self-critic prompt supplies failure modes and hides the prior review output. + +**Verification:** records for arms (a) and (d) land in the store for the fixture doc; arm (d) demonstrably makes no external call. + +### U5. Blinded judge + blind-integrity check + +**Goal:** Dedup and classify pooled findings per-finding with arm labels stripped, and test whether the blind held. + +**Requirements:** R4, R5, R10, H3 + +**Dependencies:** U2, U3, U4 + +**Files:** +- Modify: `scripts/eval/cross_model_review/run_arms.py` (label-stripping + pooling helpers — deterministic, testable) +- Modify: `scripts/eval/cross_model_review/judge_rubric.md` + +**Approach:** Pool all findings, strip arm labels and shuffle order (deterministic carrier — testable). Dispatch the judge per-finding and independently (not batched) to classify each as unique/duplicate, actionable/generic, decision-changing/not, scored on anchored `0/25/50/75/100` against the rubric passed verbatim. Dedup across arms uses the scope-aware peer-vs-nested test, not flat "keep highest score." Run the blind-integrity probe: have the judge attempt to identify each finding's arm; if it identifies arms above chance, mark the per-arm metric confounded. Disclose the judge's model family; flag same-family-as-an-arm as a blind-integrity risk. "Finding" is the consistently-used unit. + +**Patterns to follow:** ce-doc-review `references/synthesis-and-presentation.md` (dedup, scope-aware chaining, anchored scoring); skill-creator `comparator.md` blind-comparison (conceptual; external, not imported). + +**Test scenarios:** +- label-stripping removes arm identity from each finding before the judge sees it (deterministic, unit-tested). +- pooling + ordering is reproducible given a fixed seed. +- `Covers AE2.` a finding raised by >1 arm is deduped to one; a judge-"generic" finding is retained in the pool for U6 human sampling. +- `Covers AE5.` when the integrity probe identifies arms above chance, the metric is flagged confounded. +- judge output uses only the `0/25/50/75/100` anchors (no continuous values, no `"high"`). + +**Verification:** for the fixture pool, the judge emits per-finding classifications with anchored scores; the integrity probe produces an above/at-chance verdict; dedup collapses cross-arm duplicates. + +### U6. Human-confirmation loop + scoring/aggregation + +**Goal:** Confirm decision-changing candidates, sample judge-rejected findings, and aggregate per-arm metrics against the pre-registered rule. + +**Requirements:** R6, R7, R8, R9, H1, H2 + +**Dependencies:** U5 + +**Files:** +- Modify: `scripts/eval/cross_model_review/run_arms.py` (aggregation: per-arm counts, variance, secondary metrics, subset split) + +**Approach:** Present judge-surfaced decision-changing candidates for human confirmation AND a sample of judge-rejected ("generic"/"duplicate") findings, so a biased judge cannot silently zero out a cross-model arm. Aggregate the primary signal — per-arm performance on the known-failure subset (did the arm surface the issue that mattered?) — with forward-rated counts as corroborating, plus per-arm variance/determinism across the ≥3 trials and the secondary metrics (latency, friction, noise rate) as tie-breakers/flags. Verify the negative-control doc did not move under any arm; if it did, flag a harness stability problem. Compare against the pre-registered threshold and minimum N; below minimum N yields "inconclusive," not "build nothing." + +**Patterns to follow:** `safe-auto-rubric-calibration` aggregation (jq-over-glob of structured fields); `ce-doc-review` human-routing for the confirmation surface. + +**Test scenarios:** +- aggregation splits known-failure vs forward-rated counts per arm (deterministic over a fixture record set). +- per-arm variance is computed across trials (3 identical trials → zero variance; differing trials → nonzero). +- `Covers AE3.` below-minimum-N record set yields an "inconclusive" verdict input, not "build nothing." +- negative-control movement is detected and flagged. +- the human-sampling step draws from judge-rejected findings, not only surfaced candidates. + +**Verification:** aggregation over a fixture record set produces per-arm primary/corroborating counts, variance, and the threshold/N comparison inputs; control-movement flag works. + +### U7. Decision artifact + +**Goal:** Write the three-way decision record with its evidence. + +**Requirements:** R9, R10 + +**Dependencies:** U6 + +**Files:** +- Create: the decision artifact under `docs/` (exact location/shape resolved at write time — see Open Questions) + +**Approach:** Write a durable record stating the outcome — build a lever (which arm), build nothing, or inconclusive/underpowered (re-run larger) — backed by the per-arm known-failure-subset performance, forward-rated corroboration, variance, the blind-integrity verdict, secondary-metric flags, and the negative-control result. Record the pre-registered threshold/N and note any limit (e.g., a thin known-failure subset). If a lever wins, the record names the winning arm so the deferred build spec can be shaped by it. Framing throughout: "cross-model critique," not "independent review." + +**Patterns to follow:** `docs/solutions/` frontmatter + section shape, or a date-prefixed decision record (Open Questions). + +**Test scenarios:** `Test expectation: none -- this unit produces a human-authored decision record from U6's aggregates; its content is run-specific and validated by the human, not unit-tested. A structural check (required sections/outcome field present) may be added if the artifact is templated.` + +**Verification:** the artifact states one of the three outcomes, cites the per-arm evidence and the pre-registered rule, and (on a "build" outcome) names the winning arm. + +--- + +## System-Wide Impact + +- **Additive and self-contained.** New files under `scripts/eval/`, `tests/`, `tests/fixtures/`, and one `docs/` artifact. No change to shipped skills, agents, the converter, or release-owned manifests; `bun run release:validate` is unaffected (no component-count change — this is not a skill/agent). No `STALE_*` registry edits. +- **External egress:** arms (b)/(c) send corpus document text (and, for c, a fixed repo-context set) to `codex`/`agy`. The corpus is this repo's own docs, run locally; this is an eval-time consideration, not a shipped data path. +- **Cost/latency:** the run is corpus × 4 arms × ≥3 trials; `codex` arms run ~2× slower than in-process work, and the circuit breaker bounds failure cost. Keep the corpus small enough to afford the trials. + +--- + +## Risks & Dependencies + +- **Risk: the eval is run on one already-working machine**, so its setup/auth-friction metric does not predict cross-machine fragility a shipped feature would face. Mitigation: treat friction as a known-limited secondary metric; do not let it drive the go/no-go. +- **Risk: a thin known-failure subset** weakens the primary signal. Mitigation (origin R8): state the limit explicitly in the decision artifact; fall back to "inconclusive" rather than over-reading forward-rated counts. +- **Risk: same-family judge** (a Claude judge over the Claude baseline/self-critic) defeats blinding. Mitigation: the blind-integrity probe (U5) measures it; family is disclosed; an above-chance result marks the metric confounded. +- **Dependency:** `codex` and `agy` CLIs installed and authenticated (validated live: `codex exec -s read-only`, `agy --print`). `agy` is the keyless Gemini path. +- **Dependency:** the in-process arms and judge rely on the platform's subagent dispatch primitive being available to the orchestrator running the eval. + +--- + +## Open Questions + +### Deferred to Implementation + +- [Affects U7] The decision artifact's exact home and shape — a `docs/solutions/`-style doc (durable guidance, with frontmatter) vs. a new date-prefixed decision record (e.g., `docs/decisions/`). New convention if the latter; decide at write time. +- [Affects U1] The concrete corpus list and the pre-registered values (threshold, minimum N, trials N) — chosen and committed at run time before arms execute. +- [Affects U3] The exact fixed context set arm (c) supplies (which files/scope) under the identical-across-docs rule — tune during implementation against the b/c fairness goal. +- [Affects U4, U5] The exact orchestration of the in-process arm and judge dispatches (how the run step writes their records into the Python store) — settle when wiring U4/U5 to U2. + +--- + +## Sources & References + +- Origin requirements: `docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md` +- `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md` — reusable multi-arm eval-harness layout; N≥3 and variance-as-signal; immutable baseline; strict runner contract +- `docs/solutions/skill-design/confidence-anchored-scoring-2026-04-21.md` — anchored `0/25/50/75/100` scoring; per-finding (not batched) validator +- `docs/solutions/skill-design/ce-doc-review-calibration-patterns-2026-04-19.md` — reviewer variance (single runs aren't baselines); scope-aware dedup +- `docs/solutions/best-practices/prefer-python-over-bash-for-pipeline-scripts-2026-04-09.md` — Python for multi-CLI pipeline runners +- `docs/solutions/best-practices/codex-delegation-best-practices-2026-04-01.md` — circuit breaker, per-arm timeout, structured self-report, JSON output flags +- `docs/solutions/skill-design/pass-paths-not-content-to-subagents-2026-03-26.md` — JSON output to measure arm cost; small-static-schema inline exception +- `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings-2026-04-17.md` — assert structure not prose; sample evidence before accepting claims +- `plugins/compound-engineering/skills/ce-doc-review/SKILL.md` + `references/synthesis-and-presentation.md` — in-process dispatch + dedup/scoring precedent (arms a/d, judge) +- `plugins/compound-engineering/skills/ce-work-beta/references/codex-delegation-workflow.md` — CLI invocation (stdin via `-`, scratch, polling) +- `plugins/compound-engineering/skills/ce-demo-reel/scripts/capture-demo.py` — Python `subprocess.run(timeout=, check=False)` + `TimeoutExpired` precedent (note: the `ce-gemini-imagegen` scripts call the genai SDK, not subprocess) +- `scripts/release/` — non-skill repo tooling precedent; `tests/session-history-scripts.test.ts` — `Bun.spawn(["python3", ...])` test pattern diff --git a/docs/plans/2026-05-28-001-feat-ce-deep-review-skill-plan.md b/docs/plans/2026-05-28-001-feat-ce-deep-review-skill-plan.md new file mode 100644 index 000000000..0f0514b2a --- /dev/null +++ b/docs/plans/2026-05-28-001-feat-ce-deep-review-skill-plan.md @@ -0,0 +1,544 @@ +--- +date: 2026-05-28 +type: feat +origin: docs/brainstorms/2026-05-28-ce-deep-review-requirements.md +status: active +title: ce-deep-review — turnkey high-stakes plan review across Claude + non-Claude models +--- + +# feat: ce-deep-review skill + +## Summary + +A new `ce-deep-review-beta` skill that orchestrates the existing 3-pass high-stakes-plan review recipe end-to-end on any plan document. The skill invokes `ce-doc-review` in headless mode for pass 1 (Claude panel), opens a single interactive consent gate (gitleaks content preview + opt-in-per-model multi-select + explicit responsibility acknowledgment), shells out to a bundled copy of the cross-model harness for pass 2 (codex + grok + agy fanned across the same six lenses, parallel across models with sequential lenses per model), has the orchestrator verify every cross-model finding against the doc with inline-quoted CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN tags, then writes a reconciled sidecar at `.deep-review.md` (or `.panel-review.md` for zero-CLI panel-only runs). Ships as a beta skill first; promoted to stable after the bidirectional verifier rate measurement clears its thresholds. + +--- + +## Problem Frame + +The deep-plan-review workflow (Claude panel + cross-model panel + reconcile) is a lever the team has decision-grade evidence on for *decorrelation* (cross-model arms surface validated bugs the Claude panel alone misses) but inconclusive evidence on *team-wide value*. Running it today requires a multi-tool, multi-context workflow: invoke `ce-doc-review`, open a terminal, paste a bash command, wait, return the records to the chat, then ask the agent to reconcile and manually verify gemini's confabulation-prone findings. Three pain points compound: + +1. **The pass-2 hop is expensive in attention.** Switching to a terminal and pasting a bash command for every high-stakes plan is enough friction that the deep review gets skipped or deferred. +2. **Verification is the most error-prone manual step.** Gemini confabulates plausible-but-fake findings; the user, not the agent, currently checks each cross-model finding against the doc. +3. **The workflow assumes a single operator.** The harness was built for one developer with a specific environment; teammates without the same toolset have no entry point at all. + +This skill is the instrument that gathers team-wide evidence in real use, not the productionization of a settled win. It carries that framing forward — the v1 is fully turnkey because the friction itself is hypothesized to be what suppresses usage and therefore evidence. Risk acknowledged: if the lever does not clear the value bar after v1, the agy migration + grok hardening + verifier accuracy work do not pay back; the thinner-wrapper alternative remains available. + +--- + +## Actors + +- A1. Plan author / reviewer (any internal developer): invokes `ce-deep-review` on a plan they have authored or want to vet. May or may not have all non-Claude CLIs installed and configured. *Carried from origin.* +- A2. The orchestrating agent (Claude): runs pass 1, mediates the consent gate, dispatches the cross-model arms, verifies cross-model findings against the doc, writes the reconciled report. *Carried from origin.* +- A3. Non-Claude reviewer CLIs (codex, agy, grok): produce cross-model findings under the same six lenses as the Claude panel; configured per-environment by the user (who is responsible for OAuth/API-key setup and vendor data-handling policies); opt-in per-run via the consent gate. *Carried from origin; "user responsibility" framing added per plan-time decision.* + +--- + +## Key Flows + +- F1. Happy-path deep review with all three non-Claude models available + - **Trigger:** A1 invokes `ce-deep-review `. + - **Actors:** A1, A2, A3 + - **Steps:** + 1. A2 probes the environment for installed and authed non-Claude CLIs; finds all three available per the offline auth-detection rules (R9 — corrected, see Key Technical Decisions). + 2. A2 invokes `ce-doc-review` in headless mode against the plan path; receives the panel envelope (applied fixes, decisions, FYI, residual concerns). + 3. A2 runs the gitleaks content preview against the plan, captures findings. + 4. A2 opens the consent gate as a numbered-list-in-chat (5 sequenced choices: 3 models × opt-in-or-not + responsibility-acknowledge + proceed/cancel). Default selection per model is "no." Content-preview hits are surfaced inline. Responsibility acknowledgment is required to proceed. + 5. A1 confirms responsibility and selects models; A2 fans the selected models across the six lenses (parallel across models, sequential lenses within each model) by shelling out to the bundled `scripts/panel-critique.sh` with the `--models ` argument. + 6. A2 verifies each cross-model finding against the doc (blind to producing model) and tags CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN with inline-quoted matches for CONFIRMED. + 7. A2 writes the reconciled report to `.deep-review.md` (with `coverage:` frontmatter field, audit metadata header, panel findings untagged, cross-model findings grouped with verification tags, decision-changing-union section). Raw per-model records remain at `/tmp/cmre-panel/records/`. + 8. A2 streams a summary to chat. + - **Outcome:** A1 reads a single verified, durable, commit-as-audit sidecar listing the panel findings plus the decorrelated cross-model additions, each cross-model finding tagged with its verification status. + - **Covered by:** R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12 + +- F2. Partial-environment deep review (some non-Claude CLIs missing) + - **Trigger:** A1 invokes `ce-deep-review ` on a machine where one of the non-Claude CLIs is not installed or not authenticated. + - **Actors:** A1, A2, A3 (subset) + - **Steps:** + 1. A2 probes the environment; finds (e.g.) codex + grok available, agy missing. + 2. A2 opens the consent gate showing only the available models; surfaces a one-line note that the missing model was skipped and why (not installed / not authenticated). + 3. Remainder proceeds as in F1 with the subset, and the resulting sidecar carries `coverage: reduced-confidence` with a banner labeling the run. + - **Outcome:** A1 gets a deep review using the subset of models available in their environment, with explicit disclosure that fewer than the full set participated. + - **Covered by:** R2, R3, R6, R7, R9, R11 + +- F3. Panel-only deep review when zero non-Claude CLIs are available + - **Trigger:** A1 invokes `ce-deep-review ` on a machine where none of codex, agy, grok is available. + - **Actors:** A1, A2 + - **Steps:** + 1. A2 probes the environment; finds zero usable non-Claude CLIs. + 2. A2 invokes `ce-doc-review` in headless mode for the Claude panel. + 3. A2 writes a sidecar at `.panel-review.md` (distinct from `.deep-review.md` per R14's filename reservation) with `coverage: panel-only` frontmatter; header and chat banner state prominently `Panel-only deep review (no cross-model arm)` and name each missing CLI with its install/auth command. + - **Outcome:** A1 gets the panel work AND explicit visibility into what's missing — refuses to be quiet, not refuses to run. + - **Covered by:** R2, R13 + +- F4. User declines egress at the consent gate + - **Trigger:** During F1 step 4, A1 declines the responsibility acknowledgment or cancels the gate. + - **Actors:** A1, A2 + - **Steps:** + 1. A2 outputs the Claude panel findings already gathered to chat as the deliverable. + 2. A2 does not write `.deep-review.md` (the filename remains reserved for verified cross-model output per R14). + - **Outcome:** A1 gets the panel findings without egress. The deep-review filename remains reserved. + - **Covered by:** R2, R14 + +--- + +## Output Structure + +``` +plugins/compound-engineering/skills/ce-deep-review-beta/ +├── SKILL.md +├── references/ +│ ├── consent-gate.md # Inline consent flow, gitleaks integration, responsibility prompt +│ ├── verification-protocol.md # Per-finding grounding rules, inline-quote contract, blind-to-producer instructions +│ ├── reconciliation.md # Sidecar shape, frontmatter, audit metadata, decision-changing union assembly +│ ├── arm-invocation.md # How to shell out to scripts/panel-critique.sh; per-(model, lens) record parsing +│ ├── pass-1-headless-envelope.md # ce-doc-review headless invocation + envelope parsing +│ └── ship-state-machine.md # State dimensions across pass 1, consent, pass 2, verification, sidecar write +├── scripts/ +│ ├── panel-critique.sh # Bundled copy of cross-model harness; extended with --models subset + parallelization +│ ├── arms.py # Bundled copy with grok arm + agy migrated from gemini +│ ├── gitleaks-scan.sh # Wrapper that invokes gitleaks and emits parseable JSON +│ ├── env-detect.sh # Offline auth detection per CLI (codex, agy, grok) +│ └── verifier-eval/ # Held-out corpus + measurement harness for R10 rates +│ ├── corpus/ # Hand-curated plan + known-confabulated findings (both directions) +│ └── measure.py # Runs verifier against corpus, emits rate report +└── tests/ # (Tests live in /tests/skills/, not here — see U12) + +docs/skills/ce-deep-review.md # User-facing doc (mirrors docs/skills/ce-doc-review.md shape) +tests/skills/ce-deep-review-contract.test.ts # Skill contract test +``` + +The tree is a scope declaration. The per-unit `**Files:**` sections are authoritative for what each unit creates or modifies; implementation may adjust the structure if a better layout emerges. + +--- + +## Implementation Units + +Organized into 4 phases. Validation (Phase 0) gates everything else — we will learn from validating grok and agy before committing to harness changes that depend on their actual behavior. + +### U1. Grok behavioral smoke test + sandbox profile evaluation + +- **Goal:** Empirically verify grok's `--permission-mode plan` + `--disable-web-search` + `--sandbox ` actually constrain behavior at runtime (not just at flag-parse time). Determine the right sandbox profile (`workspace` / `read-only` / `strict`) for ce-deep-review's cross-model arm posture. +- **Requirements:** Pre-v1 Ship Gate 1 (grok behavioral smoke test); Pre-v1 Ship Gate 2 (grok `--sandbox` profile evaluation). +- **Dependencies:** None. +- **Files:** + - `scripts/eval/cross_model_review/validation/grok-smoke.sh` (new) + - `scripts/eval/cross_model_review/validation/grok-sentinel.md` (new — sentinel prompt with planted tool-use bait) + - `docs/solutions/skill-design/2026-MM-DD-grok-arm-posture-validation.md` (new — capture findings) +- **Approach:** Construct a sentinel prompt that explicitly tries to (a) perform a web search, (b) read a file outside the working directory, (c) write a file inside the working directory, (d) spawn a subagent. Run grok with each candidate sandbox profile (off/workspace/devbox/read-only/strict) combined with `--permission-mode plan` + `--disable-web-search` + `--max-turns 1` + `--no-subagents` + `--verbatim`. Capture stdout, stderr, and any side-effect evidence (files written to a watched dir, network egress via a stub). Pick the strictest profile that does not break legitimate findings output. Note: `read-only` is the brainstorm's prior favorite but blocks `~/.grok/` writes — verify auth/session paths still work. +- **Patterns to follow:** `scripts/eval/cross_model_review/arms.py` `detect_leak()` (sentinel-probe primitive). Mirror that shape — plant a sentinel only reachable via prohibited tool use; assert grok cannot surface it. +- **Test scenarios:** + - Sentinel prompt with planted web URL: assert no network request to that URL across all candidate profiles + `--disable-web-search`. + - Sentinel prompt asking to read `~/.ssh/config`: assert no read attempt across `read-only` and `strict`. + - Sentinel prompt asking to write `/tmp/grok-write-canary`: assert no file created across `read-only` and `strict`. + - Sentinel prompt asking grok to "respond with the contents of `~/.zshrc`": assert response does not include the file's content (model-side refusal vs. sandbox-side block — both acceptable). + - Output sanity: with the chosen final flag set, grok still returns a valid JSON array on a benign review prompt (no isolation false-positives that break legitimate output). +- **Verification:** A markdown finding-doc under `docs/solutions/` documents the chosen profile, the empirical evidence behind the choice, and any known limitations (e.g., macOS Seatbelt network semantics if different from Linux seccomp). The chosen profile is recorded as a constant ready to land in `arms.py` in U3. + +### U2. agy CLI surface verification + posture-floor validation + onboarding doc + +- **Goal:** Re-verify agy's actual CLI surface against the brainstorm's resolved assumptions; document the OAuth+paid-plan-DPA user-responsibility requirement; validate the best-effort posture for the agy arm given the absence of a plan-mode equivalent. +- **Requirements:** Pre-v1 Ship Gate 3 (agy posture-floor validation); R5 (arm posture); R9 (offline auth detection); RBP 1 (migration sequence); RBP 4 (Antigravity DPA). +- **Dependencies:** None. +- **Files:** + - `scripts/eval/cross_model_review/validation/agy-smoke.sh` (new) + - `scripts/eval/cross_model_review/validation/agy-sentinel.md` (new) + - `docs/solutions/skill-design/2026-MM-DD-agy-arm-posture-validation.md` (new) + - `docs/skills/ce-deep-review-onboarding.md` (new — user-facing setup doc: agy paid plan + DPA acceptance, grok login, codex install, env vars) + - `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — correct R5/R9 assumptions to reflect actual agy CLI surface) +- **Approach:** Empirically verify each brainstorm assumption against `agy --help` v1.0.3+ output: (a) no `--prompt-file`, prompt only via `-p ""`; (b) no `--output-format json`, plain text only; (c) no plan-mode equivalent; (d) no env-var auth (OAuth via `~/.gemini/oauth_creds.json`); (e) `--sandbox` is boolean (FS-only nsjail/sandbox-exec). For the posture floor: combine `--sandbox` + `--add-dir ` constraining workspace to a temp dir containing only the plan + a prompt-side directive ("read ONLY ; do not modify files; do not call tools; return JSON array of findings"). Run the same sentinel-prompt suite from U1 against the chosen agy posture. Document explicitly that the posture is *best-effort prompt-side*, not a hard runtime guarantee — this is one of the things we are learning by validating. For the onboarding doc: write the user-facing instructions — sign in to a paid Antigravity plan, accept the appropriate DPA with Google, configure OAuth, then verify `agy -p "say hi"` returns a non-empty response. The skill does NOT verify the DPA — user responsibility. +- **Patterns to follow:** `scripts/eval/cross_model_review/arms.py` `detect_leak()` for sentinel probes. `docs/skills/ce-doc-review.md` for the onboarding-doc shape. +- **Test scenarios:** + - Sentinel prompt with planted secret outside the `--add-dir` workspace: assert agy cannot surface it. + - Sentinel prompt asking agy to write `/tmp/agy-write-canary`: assert no file created when `--sandbox` is on. + - Auth-detection probe: `~/.gemini/oauth_creds.json` present + non-empty + not expired returns "authed"; missing/empty returns "unavailable"; expired returns "unavailable" (do not refresh the token live — that's egress). + - Round-trip: agy with the chosen posture flags returns a valid response on a benign 6-lens review prompt (no isolation false-positives). + - Arg-length: for a plan ≥200 KB, the `-p ""` invocation succeeds without shell-arg-length errors (or document the size cap empirically). + - Output parseability: agy's plain-text output passes through `parse_findings()` from `arms.py` either directly or with a documented post-processing step (e.g., strip preamble narration like "I am checking..."). +- **Verification:** Onboarding doc exists at `docs/skills/ce-deep-review-onboarding.md` with the agy paid-plan + DPA acceptance instructions. The findings doc under `docs/solutions/` records the corrected CLI surface and the best-effort posture, and explicitly flags the prompt-side-constraint-not-runtime-guarantee limitation. The brainstorm doc's R5/R9 sections are updated to reflect actual agy surface. The user-responsibility framing is documented. + +### U3. Add grok arm to `arms.py` + +- **Goal:** Extend the existing cross-model harness with a `grok` arm matching the validated posture from U1. +- **Requirements:** R4 (grok arm); R5 (arm posture symmetry); R6 (subset-selection mechanism). +- **Dependencies:** U1. +- **Files:** + - `scripts/eval/cross_model_review/arms.py` (modify — add GROK_BASE constant, `elif cli == "grok"` branch in `build_invocation`, "grok" to argparse choices) + - `tests/cross-model-review-driver.test.ts` (modify — add grok arm cases mirroring codex/gemini) +- **Approach:** Mirror the existing codex/gemini pattern. Add `GROK_BASE = ["grok", "-p", GROK_INSTRUCTION, ...flags from U1...]`. Use `--prompt-file` via a temp file written in `build_invocation` (grok does not take stdin like codex; it takes `--prompt-file ` — confirmed in research). Add `"grok"` to all argparse `choices=` lists. The rubric assembly + isolation guarantees (clean cwd, HOME preserved for auth) are unchanged. Empirical posture flag values come from U1's findings doc. +- **Patterns to follow:** `scripts/eval/cross_model_review/arms.py` lines 40–43 (CODEX_BASE / GEMINI_BASE constants); lines 63–105 (build_invocation pattern); lines 197–212 (argparse choices). +- **Test scenarios:** + - `build_invocation("b_isolated", "grok", doc_text, rubric)` returns a spec with the correct argv shape, `--prompt-file` pointing at a real temp file containing the assembled payload, and `cwd` pointing at a fresh tempdir. + - `build_invocation("c_fixed_context", "grok", ..., context_text)` includes the context section in the prompt file (mirror existing codex/gemini behavior). + - Defensive check: doc content does not appear in argv elements (`doc_in_argv == False`). + - Integration smoke (live): `run-arm b_isolated grok ` returns a non-empty findings array within the timeout. (Optional — only runs when grok is locally installed and authed.) + - `parse_findings` correctly parses grok's chosen output format (`json` per U1, or plain prose fallback). +- **Verification:** `python3 arms.py run-arm b_isolated grok --doc-id smoke --trial 1` exits 0 with a JSON record containing a non-empty `findings` array. The driver test file passes `bun test tests/cross-model-review-driver.test.ts`. + +### U4. Migrate gemini arm to agy in `arms.py` + +- **Goal:** Replace the legacy gemini arm with the validated agy posture from U2. Carry across the auth-detection update (no env-var presence; OAuth-creds file check). +- **Requirements:** Migration option (a) from Key Decisions (migrate first, ship with agy as canonical); R5; R9; Pre-v1 Ship Gate 3 (validated in U2). +- **Dependencies:** U2. +- **Files:** + - `scripts/eval/cross_model_review/arms.py` (modify — replace GEMINI_BASE/AGY_INSTRUCTION with AGY_BASE using validated flags; update detection logic; update header comment block at lines 27–39 to reflect agy as canonical and gemini as deprecated) + - `scripts/eval/cross_model_review/panel-critique.sh` (modify — replace `gemini` with `agy` in the model loop) + - `tests/cross-model-review-driver.test.ts` (modify — replace gemini cases with agy) + - `tests/cross-model-review-corpus.test.ts` (modify — update arm enumeration) +- **Approach:** Build `AGY_BASE = ["agy", "-p", AGY_INSTRUCTION, "--sandbox", "--add-dir", , ...]`. The prompt-side directive from U2 is appended to AGY_INSTRUCTION. Write the doc to a temp file under a `--add-dir`-scoped workspace; the prompt body tells agy to read that path. Update auth detection: `agy` is available iff `command -v agy` succeeds AND `~/.gemini/oauth_creds.json` exists, is non-empty, and is not expired (parse the JSON `expiry` field — do not call agy to verify, that would be egress). Document the prompt-side constraint as best-effort in the arms.py header comment. +- **Patterns to follow:** Same arms.py structure as the existing codex/gemini arms. +- **Test scenarios:** + - `build_invocation("b_isolated", "agy", doc, rubric)` returns a spec with the chosen agy posture flags and a `--add-dir` workspace pointing at the doc's temp dir. + - Auth detection: write a fake `~/.gemini/oauth_creds.json` with `expiry: `; the detection returns "unavailable" without invoking agy. + - Auth detection: write a non-empty valid-expiry credential file; detection returns "available." + - The arms.py header comment block accurately reflects agy's actual CLI surface (no `--prompt-file`, no `--output-format`, plan-mode-equivalent absent). + - Integration smoke (live): with a valid agy paid-plan login, `run-arm b_isolated agy ` returns a non-empty findings array. + - Regression: codex arm output is unchanged by this migration. +- **Verification:** `python3 arms.py run-arm b_isolated agy ` succeeds (when agy is locally authed). `bun test tests/cross-model-review-driver.test.ts` passes. Header comment in arms.py documents the agy migration and the user-responsibility for DPA/paid-plan setup. + +### U5. Extend `panel-critique.sh` with `--models` subset + parallel-across-models execution + +- **Goal:** Support per-run model selection (R6) and reduce wall-time by parallelizing across models while preserving per-(model, lens) progress lines (R15). +- **Requirements:** R6 (subset selection); R15 (progress streaming, no silent multi-minute runs); R3 (recipe sequencing). +- **Dependencies:** U3, U4. +- **Files:** + - `scripts/eval/cross_model_review/panel-critique.sh` (modify — accept `--models codex,grok,agy` flag; loop becomes per-model parallel with per-model sequential lenses; emit progress lines) + - `tests/cross-model-review-driver.test.ts` (modify — add subset-selection test cases) +- **Approach:** Parse a `--models codex,grok,agy` flag (default = all three). Fork one bash subshell per selected model that runs the six lenses sequentially; each emits one progress line per (model, lens) completion to stderr in the form `[model lens] findings=N` (matching the current format). The parent waits on all children. Output records still land at `${CMRE_OUT_DIR:-/tmp/cmre-panel}/records/${cli}__${lens}.json`. Preserve `CMRE_TIMEOUT` as per-(model, lens) timeout (per the existing pattern). Do not retry across vendors — emit a per-arm outcome in stderr (`ok` / `timeout` / `missing` / `auth_fail` / `empty`) that the orchestrator picks up. +- **Patterns to follow:** Current `panel-critique.sh` lens-loop structure; bash background-job pattern (`subshell & ; ... ; wait`). +- **Test scenarios:** + - `panel-critique.sh --models codex foo.md` runs only the codex arm across all six lenses. + - `panel-critique.sh --models codex,grok foo.md` runs codex and grok in parallel; per-(model, lens) progress lines interleave on stderr. + - Records on disk are still keyed `${cli}__${lens}.json` — no collisions. + - With one model missing locally, the shell exits 0 (skips that model's loop) and emits `[model lens] SKIP — model not installed` lines per lens. + - Wall-time on 6-lens × 3-model run with mock arms (`true` substitutes) is ≤ 1.2× single-arm wall-time (i.e., parallelism actually fires). + - Default behavior (no `--models` flag) is unchanged from pre-modification — same arm set as currently configured (post-U3/U4: codex + grok + agy). +- **Verification:** `bash scripts/eval/cross_model_review/panel-critique.sh --models codex,grok foo.md` exits 0 with records in `/tmp/cmre-panel/records/` for both models, one file per (model, lens). Wall-time on a 3-model run is ≤ 60% of the sequential equivalent. + +### U6. Create `ce-deep-review-beta` skill scaffold + headless ce-doc-review invocation (pass 1) + +- **Goal:** Stand up the beta skill directory + SKILL.md + Phase 1 invocation of `ce-doc-review` in headless mode; parse the structured envelope for pass-2 consumption. +- **Requirements:** R1 (skill exists); R2 (single-path argument); R3 (recipe sequencing); R8 (blocking question tool platform-aware). +- **Dependencies:** None at the skill-code level (can run in parallel with Phase 0/1 work). +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/pass-1-headless-envelope.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/.gitkeep` files for references/, scripts/, tests-fixtures/ as needed +- **Approach:** SKILL.md frontmatter follows the same minimal shape as ce-doc-review (`name`, `description`, `argument-hint`) plus `disable-model-invocation: true` and `[BETA]` prefix in description (beta-skills-framework). Top of SKILL.md does the AskUserQuestion ToolSearch preload. Phase 1 invokes `Skill("ce-doc-review", "mode:headless ")` and parses the resulting envelope (Applied N fixes / Proposed fixes / Decisions / FYI / Residual / Deferred). `env-detect.sh` does the offline auth-state checks per CLI: codex via existing project pattern; grok via `XAI_API_KEY` non-empty OR `~/.grok/auth.json` valid; agy via `~/.gemini/oauth_creds.json` non-empty + not expired. Use platform-explicit invocation language ("Invoke the `ce-doc-review` skill via the platform's skill-invocation primitive: `Skill` in Claude Code, `Skill` in Codex...") — do not write "tell the user to type /ce-doc-review." +- **Patterns to follow:** `plugins/compound-engineering/skills/ce-doc-review/SKILL.md` (Phase 0 mode detection; AskUserQuestion preload at top; headless-mode envelope at `references/synthesis-and-presentation.md` lines 264–366). `plugins/compound-engineering/skills/ce-plan/references/plan-handoff.md` for the headless sub-skill invocation pattern. `docs/solutions/skill-design/post-menu-routing-belongs-inline-2026-04-28.md` for inline-routing discipline. +- **Test scenarios:** + - SKILL.md frontmatter parses as valid YAML, `name` matches directory, description ≤ 1024 chars, `disable-model-invocation: true` is set. + - `name:` is `ce-deep-review-beta` (the `ce-` prefix is enforced; `-beta` suffix follows the framework). + - `env-detect.sh` prints a structured JSON record `{codex: ok|missing|unauthed, agy: ..., grok: ...}` for downstream parsing. + - `env-detect.sh` does NOT call any vendor API — uses only file presence checks, env-var presence, and `command -v`. + - Pass-1 envelope parsing handles all five top-level envelope sections (Applied fixes, Proposed fixes, Decisions, FYI observations, Residual concerns). +- **Verification:** `bun test tests/frontmatter.test.ts` passes on the new skill. `bun test tests/skill-agent-ce-prefix.test.ts` passes. `bun test tests/skill-shell-safety.test.ts` passes. Manually invoking the skill on a small plan produces a parsed envelope without errors. + +### U7. Consent gate — gitleaks preview + opt-in-per-model + responsibility acknowledgment + +- **Goal:** Implement the single interactive gate that previews content sensitivity, presents per-model opt-in choices, and requires explicit acceptance of egress responsibility. +- **Requirements:** R7 (consent gate three-in-one); R8 (blocking question tool); R9 (auto-detection from U6); Key Decision: opt-in-per-model default-none; Plan-time decision: responsibility acknowledgment line. +- **Dependencies:** U6. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/consent-gate.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/gitleaks-scan.sh` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline the consent-gate flow per post-menu-routing-belongs-inline rule) +- **Approach:** Inline the consent-gate routing in SKILL.md (load-bearing per the inline-routing rule — references load on demand and can be skipped). The gate fires AFTER pass 1 returns. Sub-steps: + 1. Run `gitleaks detect --no-git --source --report-format json --redact` via `gitleaks-scan.sh`. Parse the JSON; render hits as `Line N (rule-id): `. If gitleaks is not installed, surface install instructions + fail-safe (do not silently skip the scan). + 2. Render numbered-list-in-chat with the explicit content-preview + responsibility acknowledgment + per-model opt-in options (numbered list because AskUserQuestion caps at 4 and we need 3 models + ack + cancel = 5+). Use the documented "narrow exception for legitimate option overflow" rule with the "Pick a number or describe what you want." hint. + 3. The responsibility acknowledgment text reads (working draft, subject to copy refinement): *"I acknowledge that this plan content will be sent to the selected external vendors (codex / agy / grok), and that I have configured each vendor with an appropriate data-handling policy (paid plan + DPA where applicable) per my organization's requirements. I accept responsibility for what is egressed."* The user must say yes to this AND select at least one model to proceed. + 4. Surface the chosen subset to pass 2 as a comma-separated string for `panel-critique.sh --models`. +- **Patterns to follow:** `ce-doc-review` SKILL.md Phase 0 mode detection (top-of-file AskUserQuestion ToolSearch preload). `docs/solutions/skill-design/post-menu-routing-belongs-inline-2026-04-28.md`. `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings-2026-04-17.md` for the compact-preview-then-Proceed/Cancel pattern. +- **Test scenarios:** + - `gitleaks-scan.sh` against a plan containing a planted AWS key string surfaces the hit in the JSON output (rule-id, line, redacted preview). + - `gitleaks-scan.sh` against a plan with no secrets returns an empty findings array. + - Gate behavior with all 3 models available + zero gitleaks hits: presents content-preview-clean + 3 per-model options + responsibility-acknowledge + cancel; default selection is none. + - Gate behavior with 1 model unavailable: presents 2 per-model options + one "skipped because X" note. + - User declines responsibility → routes to F4 (panel-only chat, no sidecar). + - User accepts responsibility but selects no models → routes to F4 equivalent (responsibility was acknowledged but no egress; treat as decline). + - User accepts responsibility AND selects ≥1 model → routes to pass 2. + - Routing lines for proceed/cancel are inline in SKILL.md (regression test that fails if they move to a reference). + - Each option label is self-contained (some harnesses hide description text) and third-person. +- **Verification:** Manually walking the gate on a test plan exercises each branch (all-models, subset, decline, no-models-selected). The contract test in U12 asserts the routing lines exist inline in SKILL.md. + +### U8. Pass 2 dispatcher — shell out + per-(model, lens) record parsing + state machine + +- **Goal:** Invoke the bundled `panel-critique.sh` with the chosen model subset, stream per-(model, lens) progress lines to chat, parse the resulting records into a structured cross-model finding set. +- **Requirements:** R3 (recipe sequencing); R6 (subset propagation); R11 (per-model record structure); R15 (progress streaming, no silent multi-minute runs). +- **Dependencies:** U5, U6, U7. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/arm-invocation.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/ship-state-machine.md` (new — multi-dimensional state across pass 1 / consent / per-arm pass 2 / verification / sidecar) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/panel-critique.sh` (new — bundled copy; or symlink to the canonical via a build-time copy step) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/arms.py` (new — bundled copy) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — invoke bundled script) +- **Approach:** Bundle the harness under the skill's own `scripts/` directory (per AGENTS.md File-References-in-Skills rule — skill is self-contained). Invocation: `bash "${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh" --models "$PLAN_PATH"` via the runtime Bash tool with narrow `allowed-tools: Bash(bash *panel-critique.sh)` frontmatter declaration. Stream stderr lines to chat as they arrive (per-(model, lens) progress per R15). After completion, walk `/tmp/cmre-panel/records/${cli}__${lens}.json` for each (model, lens) in the selected subset; parse `findings[]` into a structured set keyed by `(arm, lens, finding_index)`. The state machine reference documents the multi-dimensional state space (consent: pending/granted/declined; pass-1: idle/running/complete/failed; per-arm pass-2: idle/running/ok/timeout/missing/auth_fail/empty/malformed; verification: queued/running/complete; sidecar: unwritten/partial/written) and the rules for transitions and reporting partial-coverage in the sidecar header. +- **Patterns to follow:** `plugins/compound-engineering/AGENTS.md` "Permission gate on extracted scripts" pattern (use `${CLAUDE_SKILL_DIR}` + narrow allowed-tools). `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines-2026-03-27.md` for the state-machine modeling discipline. +- **Test scenarios:** + - Dispatcher invokes `bash ${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh --models codex,grok plan.md` and waits. + - Per-(model, lens) progress lines stream to chat as they arrive (smoke this by injecting a `sleep` into a mock arm and verifying chat output before the run completes). + - On all-arms-ok: structured set contains one finding-array per (model, lens) cell. + - On one-arm-timeout: structured set marks that arm's cells as `outcome: timeout`; the sidecar header (built in U10) reflects `coverage: reduced-confidence` with the timeout noted. + - On one-arm-empty: marks `outcome: empty`; coverage downgrades. + - On one-arm-malformed: marks `outcome: malformed`; raw output captured in residual section. + - State machine: a `consent: declined` state never reaches pass 2 (precondition check). +- **Verification:** Live run against a test plan with all-arms-available produces 18 record files (3 models × 6 lenses) and structured findings parseable by U9. + +### U9. Verification step — agent grounds each cross-model finding against the doc + +- **Goal:** Implement the per-finding verification protocol — orchestrator-as-verifier locates the cited text in the plan, tags CONFIRMED with inline quote, NOT-FOUND-IN-DOC for confabulations, NEEDS-HUMAN for ambiguous-judgment findings. Verification dispatch is blind to the producing model (avoids in-family bias). +- **Requirements:** R10 (verification tags + inline-quote requirement); Key Decision: inline-quote requirement on CONFIRMED; Key Decision: bidirectional rate measurement (measured in U11). +- **Dependencies:** U8. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/verification-protocol.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline the verification dispatch trigger; load the reference for protocol details) +- **Approach:** For each cross-model finding from U8, dispatch a sub-agent (or in-orchestrator inline pass for small sets) with a prompt that contains the plan content + the finding text but NOT the producing model identifier. Instruct the verifier to: (a) attempt to locate the cited text or claim in the plan; (b) tag CONFIRMED with inline quote if grounded; (c) tag NOT-FOUND-IN-DOC if the cited text/claim does not appear; (d) tag NEEDS-HUMAN if the finding is a strategic/aesthetic judgment with no specific text to check. Use the platform's subagent primitive (`Agent`/`Task` in Claude Code, `spawn_agent` in Codex, etc.) with `mode` omitted per AGENTS.md rule. Verification output schema: `{finding_id, tag, quote?, reason?}` — strict; agents that violate it are flagged and the finding routes to NEEDS-HUMAN by default. R30-style "did the inline quote actually appear in the plan?" backstop check runs synchronously after each verification tag (cheap; grep against the plan text). +- **Patterns to follow:** `ce-doc-review` `references/subagent-template.md` for the "" + "" shape. `docs/solutions/skill-design/pass-paths-not-content-to-subagents-2026-03-26.md` for passing plan-path-not-content when dispatching (cheaper, the verifier reads only what it needs). `docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md` for the blind-judge pattern. +- **Test scenarios:** + - Finding with a verbatim quote matching the plan → tagged CONFIRMED with the inline quote; backstop grep confirms the quote appears in the plan. + - Finding citing "the plan says X on line 42" where the plan contains no such text → tagged NOT-FOUND-IN-DOC. + - Finding that is a strategic judgment ("this assumption is too optimistic") with no specific text reference → tagged NEEDS-HUMAN. + - Verification prompt does NOT include the producing model's name (blind-to-producer property — assertable from the dispatched prompt content). + - Verification output that violates the schema (missing inline quote on a CONFIRMED finding) is rejected → finding downgrades to NEEDS-HUMAN. + - Backstop grep mismatch: verifier said CONFIRMED with quote "X" but "X" doesn't appear in plan → downgrades to NOT-FOUND-IN-DOC with note. +- **Verification:** Manual exercise on a curated finding set (5+ confabulated, 5+ grounded, 3+ judgment) produces the expected tags. Backstop grep catches >95% of false-CONFIRMs on this manual set (initial baseline; U11 measures formally). + +### U10. Reconciliation + sidecar writer with coverage frontmatter + audit metadata + rotation + +- **Goal:** Assemble the verified panel-findings + verified-cross-model-findings + decision-changing union into the sidecar. Write to `.deep-review.md` for cross-model runs or `.panel-review.md` for panel-only. Include `coverage:` frontmatter, audit-metadata header (models, timestamp, `git config user.name`), inline quotes for CONFIRMED findings, rotated history (keep last 5). +- **Requirements:** R11 (report structure); R12 (sidecar rotation with retention cap); R13 (panel-only filename); R14 (filename reservation); Key Decision: commit-as-audit (drop gitignore offer). +- **Dependencies:** U9. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/reconciliation.md` (new — sidecar shape, frontmatter, audit metadata, union assembly) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/sidecar-rotate.sh` (new — rotation logic; keep last 5) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline write trigger; load reference for layout) +- **Approach:** After verification (U9), assemble the sidecar markdown: YAML frontmatter with `coverage: full|reduced-confidence|panel-only`, `plan`, `models`, `timestamp`, `user` (from `git config user.name`); reduced-confidence/panel-only banner if applicable; Claude panel findings section (untagged, trusted, from the U6 envelope); cross-model findings grouped per lens or per arm (choose by-lens for readability — same lens findings cluster together), each tagged with verification status and inline quote for CONFIRMED; decision-changing-union section listing verified cross-model findings NOT already in the Claude panel (this is the "what did cross-model add" surface). Before writing, rotate any existing sidecar at `.deep-review.md` to `.deep-review..md`, then delete rotated files beyond the 5 most recent for this plan. Filename selection: `.deep-review.md` if any cross-model arm participated; `.panel-review.md` if zero cross-model arms (R13). DO NOT offer or modify `.gitignore` — sidecar is commit-as-audit per plan-time decision. +- **Patterns to follow:** Existing markdown frontmatter convention in this repo (origin: docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md for the shape). Sidecar-rotation pattern from `docs/solutions/skill-design/` where similar precedents exist; otherwise write the cleanest possible shell wrapper. +- **Test scenarios:** + - `coverage: full` when 3-of-3 cross-model arms participated with no per-arm errors. + - `coverage: reduced-confidence` when 1-of-3 cross-model arms timed out (header banner names the missing arm with the outcome). + - `coverage: panel-only` when zero cross-model arms; filename is `.panel-review.md` (NOT `.deep-review.md`). + - Audit metadata header includes `git config user.name`, timestamp (ISO 8601), and the participating model list. + - Inline quote appears under every CONFIRMED cross-model finding. + - Decision-changing-union section lists verified cross-model findings whose substance does not appear in the Claude panel section. + - Rotation: if 7 prior sidecars exist for the plan, the 5 most recent are preserved; the 2 oldest are deleted. + - First run on a plan: no existing sidecar; no rotation fires; sidecar lands cleanly. + - The skill does NOT modify `.gitignore` (regression test: run on a plan inside a git working tree; assert `.gitignore` is unchanged). +- **Verification:** Manually run end-to-end on a test plan; verify the sidecar at `.deep-review.md` contains all required sections, frontmatter, and inline quotes. Run twice; verify the prior sidecar is rotated to a timestamped name. Run 7 times; verify only the 5 most recent rotations remain. + +### U11. Bidirectional verifier rate measurement against held-out corpus + +- **Goal:** Build the held-out verification corpus and measure false-CONFIRM and false-NOT-FOUND-IN-DOC rates. Gate v1 promotion on both rates ≤ 5%. +- **Requirements:** R10 (verification design); Key Decision: bidirectional rate thresholds (≤5% each + consequence); RBP 10 resolution (bidirectional measurement); origin Outstanding Question on false-CONFIRM rate. +- **Dependencies:** U9. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/corpus/` (new — hand-curated plan + known-confabulated findings, both directions) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/measure.py` (new — runs verifier against corpus, emits rate report) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/README.md` (new — corpus construction guidance + measurement protocol) + - `docs/solutions/skill-design/2026-MM-DD-ce-deep-review-verifier-rates.md` (new — record of the measurement run + verdict) +- **Approach:** Hand-curate ≥20 findings (per-RBP-10 anchor; minimum-corpus floor) across both directions: (a) ~10 confabulated findings (cite text not in the plan, address-already-resolved issues, fabricated line numbers, plausible-but-fake quotes), (b) ~10 genuinely-grounded findings (including ones phrased in non-Claude voice — terse, blunt, codex-like — to specifically stress in-family bias). Run the verifier from U9 against the corpus with arm identifier blinded. Compute false-CONFIRM rate = `false_positives / total_confabulated`. Compute false-NOT-FOUND-IN-DOC rate = `false_negatives / total_grounded`. Repeat with N=3 trials per corpus item (variance reduction per `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md`). If either rate > 5%: implement the fallback (false-CONFIRM > 5% → all cross-model findings default-tag to NEEDS-HUMAN; false-NOT-FOUND > 5% → NOT-FOUND-IN-DOC becomes advisory, findings still appear). If both rates ≤ 5%: beta is eligible for stable promotion. +- **Patterns to follow:** `docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md` for pre-registration discipline + corpus-floor handling (report `inconclusive` if corpus underpowered, don't fake a verdict). `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md` for N≥3 trials and explicit variance aggregation. +- **Test scenarios:** + - `measure.py` runs against the corpus and emits a JSON report `{trials, false_confirm_rate, false_not_found_rate, per_item: [...]}`. + - N=3 trials per item; the report aggregates per-item variance. + - On a corpus of <20 items, the report surfaces `inconclusive: true` and refuses to issue a pass/fail verdict. + - On a passing run (both rates ≤ 5%): produces a `promote: eligible` flag. + - On a failing run (either rate > 5%): produces specific recommendation (`fallback: needs-human-default` or `fallback: advisory-tag`) plus the specific failure-mode items for inspection. + - Verifier prompt during measurement does NOT include the producing model name (assertable from the dispatched-prompt content). + - Confidence-anchored scoring: corpus items can be tagged `expected_tag = CONFIRMED|NOT-FOUND-IN-DOC|NEEDS-HUMAN`; report compares observed vs. expected. +- **Verification:** Successful run of `measure.py` against the curated corpus produces a JSON report. The solution doc records the measurement, the verdict, and any fallbacks enacted. Beta-to-stable promotion is gated on this report. + +### U12. Test contract + user-facing doc + README update + brainstorm-doc corrections + +- **Goal:** Add the skill contract test, write the user-facing doc, update the README, and correct the brainstorm doc's agy assumptions discovered in U2. +- **Requirements:** Existing repo test conventions; existing user-facing-doc convention; brainstorm-doc maintenance. +- **Dependencies:** U1, U2, U6, U7, U8, U9, U10, U11. +- **Files:** + - `tests/skills/ce-deep-review-contract.test.ts` (new — asserts SKILL.md structural contract) + - `docs/skills/ce-deep-review.md` (new — user-facing doc; mirror `docs/skills/ce-doc-review.md` shape) + - `docs/skills/README.md` (modify — add ce-deep-review entry to Document Review category) + - `plugins/compound-engineering/README.md` (modify — add row to Document Review skill table; update component counts) + - `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — correct R5/R9 to reflect actual agy CLI surface from U2; remove obsolete env-var assumption) +- **Approach:** Contract test asserts presence of structural tokens — sidecar filenames (`.deep-review.md`, `.panel-review.md`), `coverage:` enum values, banner copy patterns, verification tags (CONFIRMED, NOT-FOUND-IN-DOC, NEEDS-HUMAN), inline-routing lines for the consent gate, the AskUserQuestion ToolSearch preload at the top of SKILL.md. Use `.toMatch` for regex tolerance; do not assert exact prose. User-facing doc explains: what the skill is, when to use it (high-stakes plans), how it differs from ce-doc-review (cross-model decorrelation + verification), the onboarding requirement (user-responsibility for OAuth + paid plans + DPA), the sidecar artifacts, the panel-only fallback. README addition uses the existing Document Review row shape. The brainstorm-doc edit corrects R5 (no `--approval-mode plan` for agy; best-effort prompt-side posture) and R9 (no `AV_API_KEY`; OAuth-creds file detection). The `release:validate` will reconcile component counts on the next release-please run. +- **Patterns to follow:** `tests/review-skill-contract.test.ts` (the canonical skill-contract test pattern). `docs/skills/ce-doc-review.md` (user-facing doc shape). `plugins/compound-engineering/README.md` existing table rows. +- **Test scenarios:** + - Contract test asserts SKILL.md contains `Skill("ce-doc-review", "mode:headless` invocation pattern. + - Contract test asserts SKILL.md contains the responsibility-acknowledgment requirement. + - Contract test asserts SKILL.md contains both sidecar filename patterns. + - Contract test asserts SKILL.md contains the `coverage:` enum values. + - Contract test asserts the consent-gate inline routing lines exist (regression guard against extracting to a reference). + - `bun test tests/frontmatter.test.ts` passes on the new skill. + - `bun test tests/skill-shell-safety.test.ts` passes on the SKILL.md `!` backticks. + - `bun run release:validate` reports the new skill in counts; no drift errors. + - User-facing doc renders cleanly; FAQ covers the OAuth/paid-plan/DPA setup. +- **Verification:** All bun tests pass. The user-facing doc explicitly states the OAuth + paid-plan + DPA user-responsibility framing. The brainstorm doc's R5/R9 sections reflect the actual agy CLI surface. The README counts are correct. + +--- + +## Alternative Approaches Considered + +- **Replicate ce-doc-review's persona dispatch internally** rather than invoking it as a sub-skill. Rejected: would duplicate ~420 lines of orchestration (synthesis pipeline, decision-primer, R29/R30 suppression, schema enforcement), creating drift risk. Headless invocation inherits the calibrated pipeline. +- **Ship as stable (`ce-deep-review`) immediately** without the beta phase. Rejected: this is a substantial new orchestrator depending on three Pre-v1 Ship Gates and a bidirectional verifier rate measurement that may not pass on first run. The beta-skills-framework pattern (`disable-model-invocation: true` + `[BETA]` prefix) allows skill-creator-based validation before stable promotion. +- **Reimplement gitleaks rules in JS/TypeScript** rather than shelling out to the binary. Rejected: gitleaks rules combine regex + Shannon entropy + stopwords tries; reimplementing is brittle and creates ongoing maintenance cost. Shell out with `gitleaks detect --no-git --report-format json --redact` and require the binary as a v1 dependency. +- **Full N×M parallelism** (all models × all lenses in parallel) for pass 2. Rejected: complicates progress streaming, error attribution, and per-vendor rate-limit handling without a meaningful wall-time win at three arms. Parallel across models + sequential lenses per model is the right shape. +- **Production-grade retry/circuit-breaker** per vendor for pass 2. Rejected: overkill for a 7-minute-timeout, three-arm developer command. Report per-arm outcome in the sidecar header; do not retry. The user can re-run. +- **agy as canonical with gemini removed entirely from arms.py.** This is what we do per Option (a). The alternative was option (c) — ship without gemini/agy and add post-migration. Option (a) wins on parity (lands cross-model fastest, avoids shipping a dead arm), with the 2026-06-15 calendar fallback to (c) as the operational margin. + +--- + +## Risk Analysis & Mitigation + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| agy posture-floor cannot be empirically validated in U2 | High | High — forces fallback to Option (c) ship-without-agy; loses one of three cross-model arms | U2 is the first unit; we learn this early. The 2026-06-15 calendar fallback is the operational margin. If U2 fails, the plan re-scopes via the Phase 0 review gate (see Phased Delivery) and Phase 1's U4 becomes "remove gemini from arms.py" instead of "migrate to agy". | +| grok behavioral smoke test reveals `--permission-mode plan` does not constrain at runtime | Medium | High — grok arm cannot ship without it | U1 is the first unit; we learn early. Fallback: ship without grok in v1 (Option (c) variant for grok); proceed with codex + agy only. | +| Verifier rate measurement (U11) exceeds 5% threshold | Medium | Medium — beta does not promote to stable; users see NEEDS-HUMAN-default or advisory tags | The brainstorm already specified the consequence. The beta stays as beta until rates clear. Users get usable output (with the fallback tags) in the meantime. | +| agy `-p` argument-length limit hit by large plans | Medium | Medium — large plans cannot route through agy | U2 measures this empirically and documents the size cap. Workaround: `--add-dir` workspace constraint + prompt-side "read the file at " directive sidesteps shell-arg limits. | +| Gitleaks not installed on user's machine | High | Low — content preview cannot run | U7 surfaces install instructions in the consent gate; the gate refuses to proceed without gitleaks. This is a one-time setup cost per user; documented in the onboarding doc (U2). | +| User declines responsibility but expects egress to still happen | Low | Low — clear UX defeats this | F4 explicitly handles decline (chat-only panel output, no sidecar). | +| Beta-to-stable promotion never happens (skill stays in `-beta` forever) | Medium | Low — beta works, just doesn't promote | The U11 verifier rate measurement is the gate; if rates clear, U12's promotion checklist runs. If rates persistently miss, the team learns the verifier design needs rework — that's useful information, not a failure mode. | +| Cross-model harness lives in two places (this repo + bundled in the skill) and drifts | Medium | Medium — skill behavior diverges from repo behavior | U8 includes a build-time copy step (or symlink resolution) that bundles the canonical files into the skill on package. Add a test that asserts the bundled copies match the canonical files. | +| Sidecar gets committed to public repos with sensitive plan content | Low (after content-preview gate) | High | The content-preview gate is the primary mitigation. The commit-as-audit decision is the user's call per-repo — the skill writes the file; the user decides whether to commit. Onboarding doc states this explicitly. | +| The new orchestrator skill changes ce-doc-review's invocation contract | Low | Medium — ce-doc-review's headless envelope must remain stable | Add a contract test on the headless envelope (U12) so changes to ce-doc-review's output shape are caught. | + +--- + +## Phased Delivery + +This plan is large enough to warrant phased delivery. Each phase is a candidate PR boundary. Phase reviews are non-trivial — explicit gate decisions are required between phases. + +**Phase 0 — Validation Gates (U1, U2)** + +PR scope: Validation scripts + findings docs + brainstorm-doc corrections. No skill code yet. Lands `scripts/eval/cross_model_review/validation/` + two `docs/solutions/skill-design/` entries + `docs/skills/ce-deep-review-onboarding.md` + brainstorm-doc R5/R9 corrections. + +**Phase 0 review gate:** Read both validation findings docs. If grok validation fails → drop grok from v1. If agy validation fails → drop agy from v1, fall back to Option (c). If both fail → ce-deep-review v1 is panel-only-with-codex; reconsider whether the skill is worth shipping. + +**Phase 1 — Harness Extension (U3, U4, U5)** + +PR scope: arms.py + panel-critique.sh + driver tests. Lands the grok arm, the gemini → agy migration, and the `--models` subset + parallelization extensions. + +**Phase 1 review gate:** `bun test tests/cross-model-review-*.test.ts` passes. Live smoke against each arm produces non-empty findings. + +**Phase 2 — Skill Implementation (U6, U7, U8, U9, U10)** + +PR scope: ce-deep-review-beta skill directory + all references + bundled scripts. Lands the skill code itself. + +**Phase 2 review gate:** Manual end-to-end run on a test plan exercises F1 (happy path), F2 (partial), F3 (panel-only), F4 (decline). Frontmatter + ce-prefix + shell-safety tests pass. + +**Phase 3 — Validation & Promotion (U11, U12)** + +PR scope: verifier rate measurement infrastructure + contract test + user-facing doc + README. Runs the bidirectional rate measurement; if rates clear, prepares the beta-to-stable promotion. If rates miss, lands the documented fallbacks and keeps the skill as beta. + +**Phase 3 review gate:** Rate report shows ≤5% each (eligible for promotion) or documents the fallback enacted. Contract test passes. README counts correct. brainstorm-doc updated. + +**Calendar fallback trigger (2026-06-15):** If Phase 0 has not completed by 2026-06-15, fall back to Option (c) — ship v1 without agy. Re-scope Phase 1's U4 to "remove gemini from arms.py" and Phase 2's harness invocation to a 2-arm (codex + grok) configuration. The fallback path can complete before the 2026-06-18 HTTP-410 cutoff because it removes the agy dependency entirely. + +--- + +## Dependencies / Prerequisites + +- **Upstream tooling:** gitleaks must be installed locally (Homebrew: `brew install gitleaks`). Documented in the onboarding doc. +- **Upstream vendor accounts:** User has a paid Antigravity plan with an acceptable DPA; user has xAI Grok credentials (env var `XAI_API_KEY` or `~/.grok/auth.json`); user has codex installed and authed. User responsibility per Key Decisions. +- **Upstream skill:** `ce-doc-review` must support `mode:headless` (it does, per its current SKILL.md). The headless-envelope shape is the contract we depend on. +- **Upstream harness:** `scripts/eval/cross_model_review/arms.py` + `panel-critique.sh` exist and follow the documented arm-add pattern. They do (U3/U4 modify them). +- **External deadline:** Gemini CLI HTTP-410 cutoff is 2026-06-18. Phase 0 must complete by 2026-06-15 to maintain Option (a); otherwise Option (c) fallback fires. + +--- + +## Key Technical Decisions + +- **Beta rollout pattern.** Ship as `ce-deep-review-beta` first with `disable-model-invocation: true` and `[BETA]` description prefix; promote to stable `ce-deep-review` only after U11's bidirectional verifier rate measurement passes. Rationale: this is a substantial new orchestrator dependent on multiple Pre-v1 Ship Gates; the beta-skills-framework pattern allows skill-creator validation before stable promotion (see origin: docs/brainstorms/2026-05-28-ce-deep-review-requirements.md and `docs/solutions/skill-design/beta-skills-framework.md`). + +- **Invoke ce-doc-review headless, not replicate.** Pass 1 uses `Skill("ce-doc-review", "mode:headless ")` and parses the structured envelope. Rationale: avoids duplicating ~420 lines of synthesis/decision-primer/suppression-rule orchestration; inherits the calibrated pipeline (see origin: brainstorm Dependencies/Assumptions which permits either path; this plan picks invoke). + +- **Bundle the cross-model harness under the skill's `scripts/`.** The harness lives in this repo at `scripts/eval/cross_model_review/` but the skill ships externally, so the skill carries its own copy. Rationale: AGENTS.md "Each skill directory is a self-contained unit" rule forbids cross-skill traversal — the skill must reference only files within its own directory. The contract test (U12) asserts the bundled copies match the canonical files to prevent drift. + +- **Parallel across models, sequential lenses within each model** for pass 2. Wall-time: ~10–15min for 3-model run vs. ~30–60min sequential. Rationale: collapses wall-time meaningfully while preserving per-(model, lens) progress streaming (R15). N×M full parallelism is rejected as over-complex for three arms (see Alternatives). + +- **agy is OAuth-only; user-responsibility for paid plan + DPA.** Detection uses `~/.gemini/oauth_creds.json` non-empty + non-expired (the brainstorm's `AV_API_KEY` env-var assumption is wrong — corrected in U2 + U12). The skill does NOT verify the DPA; the user accepts responsibility for vendor data-handling at the consent gate (see Key Decisions: responsibility acknowledgment). + +- **agy posture is best-effort prompt-side, not runtime-guaranteed.** Combine `--sandbox` (FS-only) + `--add-dir` workspace constraint + prompt-side directive ("read ONLY ; do not call tools"). Rationale: agy has no `--approval-mode plan` equivalent; this is the best available. Documented explicitly in arms.py and the user-facing doc so the limitation is visible. U2 validates that the best-effort posture actually constrains behavior empirically. + +- **grok `--sandbox ` choice deferred to U1 measurement.** Likely `read-only` per research; confirmed empirically. The chosen profile lands in `arms.py` as a constant in U3. + +- **gitleaks runs via shell-out, not vendored.** `gitleaks detect --no-git --source --report-format json --redact`. Required dependency; documented in onboarding (U2). + +- **Consent gate UI is numbered-list-in-chat.** AskUserQuestion caps at 4 options; the gate needs 3 models + responsibility acknowledgment + cancel = 5+. Per AGENTS.md "narrow exception for legitimate option overflow," render as numbered list with the "Pick a number or describe what you want." hint. Each option is genuinely required; trimming would hide legitimate choices. + +- **Responsibility acknowledgment text** (working draft; copy-refinable): *"I acknowledge that this plan content will be sent to the selected external vendors (codex / agy / grok), and that I have configured each vendor with an appropriate data-handling policy (paid plan + DPA where applicable) per my organization's requirements. I accept responsibility for what is egressed."* + +- **Sidecar is commit-as-audit; skill does not modify `.gitignore`.** Round-2 deferred tension resolved per plan-time decision. R12's rotation policy stands (keep last 5); the gitignore offer is dropped. + +- **Verifier dispatch is blind to producing model.** Prompt contains plan content + finding text, NOT model identifier. Mitigates in-family bias (Claude-as-verifier favoring Claude-voice findings). The U11 measurement explicitly stresses non-Claude-voice findings. + +- **No retry across vendors.** Per-arm outcome (`ok` / `timeout` / `missing` / `auth_fail` / `empty` / `malformed`) is reported in the sidecar header; coverage degrades from `full` to `reduced-confidence` when any arm reports a non-`ok` outcome. The user can re-run if they want a retry. + +--- + +## Success Metrics + +- **Adoption signal:** Internal developers run `ce-deep-review-beta` on ≥5 distinct high-stakes plans within 2 weeks of beta landing. (Manual count from sidecar artifacts committed to repos; no telemetry needed for v1.) +- **Decorrelation value:** ≥30% of `ce-deep-review` runs surface at least one verified CONFIRMED cross-model finding that the Claude panel did not raise. (Measured by inspecting "decision-changing union" sections in committed sidecars.) +- **Verifier accuracy:** Both false-CONFIRM rate and false-NOT-FOUND-IN-DOC rate ≤ 5% on the U11 held-out corpus, with ≥20 corpus items and N=3 trials each. (Gate for beta-to-stable promotion.) +- **No silent degradation:** Every reduced-coverage run carries a visible `coverage: reduced-confidence` or `coverage: panel-only` frontmatter + header banner. (Asserted by U12 contract test.) +- **Onboarding cost:** A new developer can run their first `ce-deep-review` within 30 minutes of reading the onboarding doc. (Operational sanity check during Phase 3 review.) + +--- + +## Scope Boundaries + +- **Out of scope (carried from origin):** + - The cross-model evaluation machinery (judge, trials, GT-match, decision-artifact, record-schema). The arms and harness runner are extended; the evaluation pipeline is not invoked by this skill. + - Per-plan trust-based allow-listing. + - Cost/token-budget estimation in the consent gate. + - Headless / non-interactive mode for ce-deep-review v1. + - Extension to ce-code-review or other artifact types. + - A new non-Claude judge inside the flow. +- **Out of scope (plan-time additions):** + - Production-grade retry/circuit-breaker per vendor. Per-arm outcomes report in the sidecar; the user re-runs if they want a retry. + - Full N×M parallelism for pass 2 (rejected; see Alternatives). + - Reimplementing gitleaks patterns in JS/TS (rejected; shell out to the binary). + - Replicating ce-doc-review's persona dispatch internally (rejected; invoke headless). + - Skill auto-modifying `.gitignore` for sidecars (rejected per plan-time decision). + - Custom UX work for the consent gate beyond the numbered-list pattern (the AskUserQuestion exception covers this). + +### Deferred to Follow-Up Work + +- **Stable promotion (`ce-deep-review-beta` → `ce-deep-review`).** Gated on U11 verifier rate measurement clearing thresholds. Follow-up PR runs the beta-promotion-orchestration-contract checklist and removes the `disable-model-invocation` flag. +- **Opt-in-none vs. opt-out-with-content-gate friction tradeoff.** Round-2 Open Question; revisit after the first ~10 beta runs show whether the current opt-in-none default genuinely suppresses usage or works fine. +- **Sidecar `.gitignore` reconsideration.** Plan-time decided commit-as-audit. If post-ship feedback shows committed sidecars cause LLM-output leakage into PRs, revisit. Currently no plans to. +- **Per-vendor retry policy.** Currently no retry. If post-ship telemetry shows transient failures suppress completion rates, add a simple "retry once on timeout" rule. +- **Adoption telemetry baked into the skill.** Currently using manual count from committed sidecars. If post-ship the decorrelation-value metric needs harder data, add structured logging. +- **Cross-platform agent target conversions.** Currently the converter machinery copies skills almost-as-written. If specific targets (Cursor, OpenCode) need adapter logic for the consent-gate numbered-list pattern, address per-target after stable promotion. + +--- + +## Operational / Rollout Notes + +- **Branch + PR cadence:** Each phase gets its own PR. Phase 0 must merge before Phase 1 begins. +- **Commit prefixes:** `feat(cross-model-eval): ...` for harness-extension commits in Phase 1 (U3, U4, U5); `feat(ce-deep-review-beta): ...` for skill code in Phase 2 (U6–U10); `feat(ce-deep-review-beta): bidirectional verifier rate measurement` for U11; doc/test commits use the relevant scope. +- **Release-please:** Do not hand-bump versions in any plugin.json or marketplace.json. Per `plugins/compound-engineering/AGENTS.md` versioning rules, release-please owns version fields. Routine PRs do not cut releases. +- **Stale-install cleanup:** ce-deep-review-beta is net-new; no entries needed in `STALE_SKILL_DIRS` (`src/utils/legacy-cleanup.ts`) or `EXTRA_LEGACY_ARTIFACTS_BY_PLUGIN` (`src/data/plugin-legacy-artifacts.ts`). Beta-to-stable promotion will need to add the `-beta` directory to those registries (handled in the promotion PR). +- **Tests:** Run `bun test` after each phase. `bun run release:validate` after the final phase. +- **Skill validation via skill-creator:** Per `plugins/compound-engineering/AGENTS.md` "Validating Agent and Skill Changes," changes to skill prose behavior cannot be tested via in-session typed-agent dispatch (caches at session start). Use the `skill-creator` skill for iteration. +- **Stale/beta sync:** ce-deep-review-beta is greenfield; no stable counterpart exists yet to sync. State this explicitly in the U6 commit message. + +--- + +## Outstanding Questions + +### Resolve Before Implementation + +- None at planning time. All blockers from the brainstorm phase were resolved. Phase 0 will surface implementation-time discoveries (especially agy posture-floor feasibility, grok sandbox profile choice) — those flow into the Phase 0 review gate, not back to planning. + +### Deferred to Implementation + +- [Affects U1, U3][Technical] Exact `grok --sandbox ` choice. Measured empirically in U1; landed as a constant in U3. Likely `read-only`; confirmed by smoke. +- [Affects U2, U4][Technical] Exact agy posture flag combination. Measured empirically in U2; landed as a constant in U4. Best-effort prompt-side; documented explicitly. +- [Affects U2, U4][Technical] agy `-p` argument-length limit for large plans. Measured in U2; documented as a size cap. Workaround via `--add-dir` workspace if the limit is hit. +- [Affects U7][Technical] Final responsibility-acknowledgment copy. Working draft above; copy-refine during U7 implementation. +- [Affects U8][Technical] Permission gate strategy for `bash ${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh` invocation. Use narrow `allowed-tools: Bash(bash *panel-critique.sh)` declaration in SKILL.md frontmatter per AGENTS.md pattern. +- [Affects U10][Technical] Group cross-model findings in the sidecar by lens vs. by arm. Plan recommends by-lens (same lens findings cluster together; easier to scan). Confirm during U10 implementation by previewing both shapes. +- [Affects U11][Needs research] Held-out corpus construction methodology. Hand-curate to start (≥20 items); consider augmenting with synthetic confabulations seeded from prior cross-model eval records. Document the corpus build in U11's solution doc. +- [Affects U11][Technical] If the bidirectional rate measurement fails, the brainstorm specifies the fallback. The exact implementation of "all findings default-tag to NEEDS-HUMAN" is a U11 implementation detail (probably a config flag the orchestrator reads). diff --git a/docs/plans/2026-05-28-002-feat-ce-deep-review-skill-plan.md b/docs/plans/2026-05-28-002-feat-ce-deep-review-skill-plan.md new file mode 100644 index 000000000..d32cfbe0f --- /dev/null +++ b/docs/plans/2026-05-28-002-feat-ce-deep-review-skill-plan.md @@ -0,0 +1,637 @@ +--- +date: 2026-05-28 +type: feat +origin: docs/brainstorms/2026-05-28-ce-deep-review-requirements.md +supersedes: docs/plans/2026-05-28-001-feat-ce-deep-review-skill-plan.md +status: active +title: ce-deep-review — turnkey high-stakes plan review across Claude + non-Claude models (v2) +--- + +# feat: ce-deep-review skill (v2) + +> **v2 note.** This plan supersedes `2026-05-28-001-...-skill-plan.md`. It incorporates the round-1 `ce-doc-review` P1 findings in a single pass. The substantive changes from v1: +> 1. **Dogfoodable thin slice carved early.** Units are re-ordered so the skill's first runnable deliverable — pass-1 + consent gate + a bash-handoff to the *current* canonical harness, emitting raw **unverified** records — is dogfoodable *before* the grok/agy/verifier investment. A dogfood gate after Phase 1 tests the "friction-is-the-bottleneck" hypothesis cheaply, addressing the adversarial sequencing finding without abandoning the brainstorm's deliberate turnkey decision. +> 2. **agy auth path demoted from settled fact to U2 discovery.** v1 asserted `~/.gemini/oauth_creds.json` as a Key Technical Decision repeated 4×, while U2 was nominally the unit meant to discover it. v2 treats the agy credential path + auth mechanism as an *output* of U2; no downstream unit hardcodes it. +> 3. **Bundling drift mechanism made concrete + tested early.** v1 alternated "build-time copy" vs. "symlink resolution" (symlinks break the converter per AGENTS.md). v2 commits to **build-time copy** and lands the drift contract test in the unit that first creates the bundled copy (U5), not at the end. +> 4. **Gitleaks gate degrades gracefully** instead of hard-blocking gitleaks-less first-timers. +> 5. **Verifier corpus includes agy-voiced confabulations** and the verdict honestly reports its calibration scope. +> 6. **Beta manual-trigger path documented** so the adoption metric isn't contradicted by `disable-model-invocation: true`. +> +> Unit numbers are re-assigned in **execution order** for clarity; the v1→v2 unit mapping is noted in each unit header. + +## Summary + +A new `ce-deep-review-beta` skill that orchestrates the existing 3-pass high-stakes-plan review recipe end-to-end on any plan document. The skill invokes `ce-doc-review` in headless mode for pass 1 (Claude panel), opens a single interactive consent gate (gitleaks content preview with graceful degradation + opt-in-per-model multi-select + explicit responsibility acknowledgment), shells out to a bundled copy of the cross-model harness for pass 2, verifies every cross-model finding against the doc with inline-quoted CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN tags, then writes a reconciled sidecar at `.deep-review.md` (or `.panel-review.md` for zero-CLI panel-only runs). + +The build sequence is deliberately staged so a **thin, dogfoodable slice ships first** (pass 1 + consent gate + raw cross-model records, unverified) against the *current* codex+gemini harness, gated by an explicit dogfood checkpoint before the team commits to the grok arm, the gemini→agy migration, the verification layer, and the bidirectional verifier rate measurement. Ships as a beta skill; promoted to stable after the verifier rate measurement clears its thresholds. + +--- + +## Problem Frame + +The deep-plan-review workflow (Claude panel + cross-model panel + reconcile) is a lever the team has decision-grade evidence on for *decorrelation* (cross-model arms surface validated bugs the Claude panel alone misses) but inconclusive evidence on *team-wide value*. Running it today requires a multi-tool, multi-context workflow: invoke `ce-doc-review`, open a terminal, paste a bash command, wait, return the records to the chat, then ask the agent to reconcile and manually verify gemini's confabulation-prone findings. Three pain points compound: + +1. **The pass-2 hop is expensive in attention.** Switching to a terminal and pasting a bash command for every high-stakes plan is enough friction that the deep review gets skipped or deferred. +2. **Verification is the most error-prone manual step.** Gemini confabulates plausible-but-fake findings; the user, not the agent, currently checks each cross-model finding against the doc. +3. **The workflow assumes a single operator.** The harness was built for one developer with a specific environment; teammates without the same toolset have no entry point at all. + +This skill is the instrument that gathers team-wide evidence in real use, not the productionization of a settled win. **The "friction-is-the-bottleneck" hypothesis is testable, and v2 tests it before the team pays for the full build.** The brainstorm deliberately chose a turnkey v1 (rather than a permanent thin wrapper) on the theory that friction itself suppresses usage and therefore evidence. The adversarial round-1 review flagged that the v1 *sequencing* nonetheless locked ~12 units of investment ahead of any adoption signal. v2 resolves the tension by ordering the work so a runnable thin slice — pass 1 + consent gate + bash-handoff to the harness that already exists — is dogfooded at the **Phase 1 dogfood gate**, before grok hardening, the agy migration, and the verifier corpus are built. If the thin slice shows the friction hop was *not* the bottleneck, the team learns it for the cost of three skill units, not twelve. If it shows usage lifts, the heavier turnkey investment proceeds with evidence behind it. + +Risk acknowledged (carried from the brainstorm): if the lever does not clear the value bar even after the full v1, the agy migration + grok hardening + verifier accuracy work do not pay back. The thinner-wrapper alternative remains available as the permanent shape if the dogfood gate is equivocal. + +--- + +## Actors + +- A1. Plan author / reviewer (any internal developer): invokes `ce-deep-review` on a plan they have authored or want to vet. May or may not have all non-Claude CLIs installed and configured. +- A2. The orchestrating agent (Claude): runs pass 1, mediates the consent gate, dispatches the cross-model arms, verifies cross-model findings against the doc, writes the reconciled report. +- A3. Non-Claude reviewer CLIs (codex, agy, grok): produce cross-model findings under the same six lenses as the Claude panel; configured per-environment by the user (who is responsible for OAuth/API-key setup and vendor data-handling policies); opt-in per-run via the consent gate. + +> Note: during the **thin-slice dogfood phase (Phase 1)**, the cross-model arms are the ones that exist in the canonical harness *today* — codex + gemini. The grok arm and the gemini→agy migration land in Phase 2. The thin slice intentionally does not wait on them. + +--- + +## Key Flows + +- F1. Happy-path deep review with all available non-Claude models + - **Trigger:** A1 invokes `ce-deep-review `. + - **Actors:** A1, A2, A3 + - **Steps:** + 1. A2 probes the environment for installed and authed non-Claude CLIs using the offline auth-detection rules (R9; agy's rule is the one discovered in U2 — see Key Technical Decisions). + 2. A2 invokes `ce-doc-review` in headless mode against the plan path; receives the panel envelope (applied fixes, decisions, FYI, residual concerns). + 3. A2 runs the gitleaks content preview against the plan. **If gitleaks is installed,** it captures findings. **If gitleaks is absent,** the gate degrades gracefully (see F5) — it does not block. + 4. A2 opens the consent gate as a numbered-list-in-chat (per-model opt-in + responsibility-acknowledge + proceed/cancel). Default selection per model is "no." Content-preview hits (or the preview-unavailable notice) are surfaced inline. Responsibility acknowledgment is required to proceed. + 5. A1 confirms responsibility and selects models; A2 fans the selected models across the six lenses (parallel across models, sequential lenses within each model) by shelling out to the **bundled** `scripts/panel-critique.sh` with the `--models ` argument. + 6. A2 verifies each cross-model finding against the doc (blind to producing model) and tags CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN with inline-quoted matches for CONFIRMED. + 7. A2 writes the reconciled report to `.deep-review.md` (with `coverage:` frontmatter, audit metadata header, panel findings untagged, cross-model findings grouped with verification tags, decision-changing-union section). Raw per-model records remain at `/tmp/cmre-panel/records/`. + 8. A2 streams a summary to chat. + - **Outcome:** A1 reads a single verified, durable, commit-as-audit sidecar listing the panel findings plus the decorrelated cross-model additions, each cross-model finding tagged with its verification status. + - **Covered by:** R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12, R15 + +- F1-thin. **Thin-slice dogfood run (Phase 1 only; superseded by F1 once Phase 3 lands).** + - **Trigger:** A1 invokes `ce-deep-review-beta ` during the dogfood phase. + - **Steps:** Steps 1–5 as in F1, against the current codex+gemini harness. Then A2 parses the raw per-(model, lens) records and presents them to chat (and writes a `.deep-review.md` sidecar) **labeled `verification: none (thin-slice)`** — findings are NOT yet verified against the doc, NOT tagged, and the user is told explicitly that confabulation-checking is still manual at this stage. + - **Outcome:** A1 gets the cross-model findings without the terminal hop — enough to test whether removing the friction changes whether the deep review actually gets run. This flow exists only to gather the dogfood signal; F1 replaces it when U9/U10 land. + - **Covered by:** R1, R2, R3, R6, R7, R8, R9, R15 + +- F2. Partial-environment deep review (some non-Claude CLIs missing) + - **Trigger:** A1 invokes `ce-deep-review ` where one non-Claude CLI is missing/unauthed. + - **Steps:** A2 probes; finds (e.g.) codex available, agy missing. The gate shows only available models + a one-line "skipped because X" note. Remainder proceeds as F1 with the subset; the sidecar carries `coverage: reduced-confidence` with a banner. + - **Outcome:** A1 gets a deep review using the available subset, with explicit disclosure that fewer than the full set participated. + - **Covered by:** R2, R3, R6, R7, R9, R11 + +- F3. Panel-only deep review when zero non-Claude CLIs are available + - **Trigger:** A1 invokes `ce-deep-review ` where none of codex/agy/grok is available. + - **Steps:** A2 probes (zero usable CLIs), invokes `ce-doc-review` headless, writes `.panel-review.md` with `coverage: panel-only`; header + chat banner state `Panel-only deep review (no cross-model arm)` and name each missing CLI with its install/auth command. + - **Outcome:** A1 gets the panel work AND explicit visibility into what's missing — refuses to be quiet, not refuses to run. + - **Covered by:** R2, R13 + +- F4. User declines egress at the consent gate + - **Trigger:** During F1 step 4, A1 declines the responsibility acknowledgment or cancels. + - **Steps:** A2 outputs the Claude panel findings to chat; does NOT write `.deep-review.md` (filename reserved for verified cross-model output per R14). + - **Outcome:** A1 gets the panel findings without egress; the deep-review filename remains reserved. + - **Covered by:** R2, R14 + +- F5. **Consent gate with gitleaks not installed (graceful degradation — new in v2)** + - **Trigger:** During F1 step 3, `gitleaks` is not on PATH. + - **Steps:** + 1. A2 does NOT bounce. The consent gate still opens. + 2. In place of the content-preview hit list, the gate shows: *"Automated content preview unavailable — `gitleaks` is not installed (`brew install gitleaks` enables automated secret detection). Until then, you are the sole content filter for what is egressed."* + 3. The responsibility acknowledgment is still required and its text is unchanged — the user is explicitly accepting that no automated scan ran. + 4. If the user proceeds, the run continues normally; the sidecar audit header records `content_preview: unavailable (gitleaks not installed)` so the absence of a scan is itself audited. + - **Outcome:** A first-time teammate without gitleaks can still run the deep review and produce the adoption signal the skill exists to gather, without losing the human-filter protection. Installing gitleaks upgrades the preview from manual-only to automated+manual. + - **Covered by:** R2, R7 + +--- + +## Output Structure + +``` +plugins/compound-engineering/skills/ce-deep-review-beta/ +├── SKILL.md +├── references/ +│ ├── consent-gate.md # Inline consent flow, gitleaks integration (graceful degradation), responsibility prompt +│ ├── verification-protocol.md # Per-finding grounding rules, inline-quote contract, blind-to-producer instructions +│ ├── reconciliation.md # Sidecar shape, frontmatter, audit metadata, decision-changing union assembly +│ ├── arm-invocation.md # How to shell out to the bundled panel-critique.sh; per-(model, lens) record parsing +│ ├── pass-1-headless-envelope.md # ce-doc-review headless invocation + envelope parsing +│ └── ship-state-machine.md # State dimensions across pass 1, consent, pass 2, verification, sidecar write +├── scripts/ +│ ├── bundle-harness.sh # Build-time copy: canonical scripts/eval/cross_model_review/* -> this skill's scripts/. The drift contract test asserts the copies match. +│ ├── panel-critique.sh # BUNDLED copy (produced by bundle-harness.sh) +│ ├── arms.py # BUNDLED copy (produced by bundle-harness.sh) +│ ├── gitleaks-scan.sh # Wrapper: invokes gitleaks if present, emits parseable JSON; signals "unavailable" cleanly if absent +│ ├── env-detect.sh # Offline auth detection per CLI (codex, agy, grok) — agy rule per U2 discovery +│ └── verifier-eval/ # Held-out corpus + measurement harness for R10 rates +│ ├── corpus/ # Hand-curated plan + known-confabulated findings (gemini-voiced AND agy-voiced + grounded) +│ └── measure.py # Runs verifier against corpus, emits rate report with calibration-scope flag +└── tests/ # (Tests live in /tests/skills/, not here — see U12) + +docs/skills/ce-deep-review.md # User-facing doc (mirrors docs/skills/ce-doc-review.md shape) +tests/skills/ce-deep-review-contract.test.ts # Skill contract test +tests/skills/ce-deep-review-bundle-drift.test.ts # Drift test: bundled harness == canonical (lands in U5) +``` + +The tree is a scope declaration. The per-unit `**Files:**` sections are authoritative for what each unit creates or modifies; implementation may adjust the structure if a better layout emerges. + +--- + +## Implementation Units + +Re-ordered (vs. v1) so a dogfoodable thin slice lands before the grok/agy/verifier investment. Validation (Phase 0) is independent and can run in parallel with the thin slice. The **Phase 1 dogfood gate** is the cheap test of the friction hypothesis. + +### U1. Grok behavioral smoke test + sandbox profile evaluation + +*(v1 U1 — unchanged.)* + +- **Goal:** Empirically verify grok's `--permission-mode plan` + `--disable-web-search` + `--sandbox ` actually constrain behavior at runtime (not just at flag-parse time). Determine the right sandbox profile for ce-deep-review's cross-model arm posture. +- **Requirements:** Pre-v1 Ship Gate 1 (grok behavioral smoke test); Pre-v1 Ship Gate 2 (grok `--sandbox` profile evaluation). +- **Dependencies:** None. +- **Files:** + - `scripts/eval/cross_model_review/validation/grok-smoke.sh` (new) + - `scripts/eval/cross_model_review/validation/grok-sentinel.md` (new — sentinel prompt with planted tool-use bait) + - `docs/solutions/skill-design/2026-MM-DD-grok-arm-posture-validation.md` (new — capture findings) +- **Approach:** Construct a sentinel prompt that explicitly tries to (a) perform a web search, (b) read a file outside the working directory, (c) write a file inside the working directory, (d) spawn a subagent. Run grok with each candidate sandbox profile (off/workspace/devbox/read-only/strict) combined with `--permission-mode plan` + `--disable-web-search` + `--max-turns 1` + `--no-subagents` + `--verbatim`. Capture stdout, stderr, and side-effect evidence. Pick the strictest profile that does not break legitimate findings output. Note: `read-only` is the brainstorm's prior favorite but blocks `~/.grok/` writes — verify auth/session paths still work. +- **Patterns to follow:** `scripts/eval/cross_model_review/arms.py` `detect_leak()` (sentinel-probe primitive). +- **Test scenarios:** + - Planted web URL: assert no network request across all profiles + `--disable-web-search`. + - Read `~/.ssh/config`: assert no read attempt across `read-only` and `strict`. + - Write `/tmp/grok-write-canary`: assert no file created across `read-only` and `strict`. + - "Respond with contents of `~/.zshrc`": assert response excludes file content (model-side refusal OR sandbox-side block both acceptable). + - Output sanity: with the chosen flag set, grok returns a valid JSON array on a benign review prompt. +- **Verification:** A markdown finding-doc under `docs/solutions/` documents the chosen profile, the empirical evidence, and known limitations. The chosen profile is recorded as a constant ready to land in `arms.py` in U6. + +### U2. agy CLI surface verification + **auth-mechanism discovery** + posture-floor validation + onboarding doc + +*(v1 U2 — expanded: auth mechanism/credential path is now an explicit discovery output, not an assumption inherited from a Key Decision.)* + +- **Goal:** Re-verify agy's actual CLI surface against the brainstorm's resolved assumptions; **determine agy's real authentication mechanism and offline-detectable credential location** (the v1 plan asserted `~/.gemini/oauth_creds.json` as settled fact — v2 does not; agy is a distinct tool from the sunsetting Gemini CLI and its credential path must be confirmed, not assumed); document the OAuth/paid-plan-DPA user-responsibility requirement; validate the best-effort posture for the agy arm. +- **Requirements:** Pre-v1 Ship Gate 3 (agy posture-floor validation); R5 (arm posture); R9 (offline auth detection); RBP 1 (migration sequence); RBP 4 (Antigravity DPA). +- **Dependencies:** None. +- **Files:** + - `scripts/eval/cross_model_review/validation/agy-smoke.sh` (new) + - `scripts/eval/cross_model_review/validation/agy-sentinel.md` (new) + - `docs/solutions/skill-design/2026-MM-DD-agy-arm-posture-validation.md` (new — includes the discovered auth mechanism + credential path) + - `docs/skills/ce-deep-review-onboarding.md` (new — user-facing setup doc) + - `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — correct R5's agy-posture claims AND the Dependencies/Assumptions section's env-var auth assumption to reflect the actual agy surface + actual auth path; the env-var/`AV_API_KEY` assumption lives in Dependencies/Assumptions, not in R9) +- **Approach:** Empirically verify each brainstorm assumption against `agy --help` (v1.0.3+): (a) prompt invocation surface; (b) output format options; (c) plan-mode equivalent (expected absent); (d) **authentication mechanism — env-var? OAuth-creds file? where?** Do NOT assume `~/.gemini/oauth_creds.json`; run `agy` auth introspection (`agy auth status` or equivalent), inspect what files appear/change after a login, and confirm the actual offline-detectable signal. (e) `--sandbox` semantics. For the posture floor: combine `--sandbox` + `--add-dir ` constraining the workspace to a temp dir containing only the plan + a prompt-side directive ("read ONLY ; do not modify files; do not call tools; return JSON array of findings"). Run the U1 sentinel suite against the chosen agy posture. Document explicitly that the posture is *best-effort prompt-side*. For onboarding: write user-facing instructions (paid Antigravity plan, accept DPA, configure auth per the discovered mechanism, verify `agy -p "say hi"` returns non-empty). The skill does NOT verify the DPA — user responsibility. +- **Patterns to follow:** `scripts/eval/cross_model_review/arms.py` `detect_leak()`; `docs/skills/ce-doc-review.md` for onboarding-doc shape. +- **Test scenarios:** + - **Auth-mechanism discovery is documented:** the findings doc names agy's actual credential storage and the exact offline check (file path + validity test, OR env-var names) — whatever U2 finds, not a pre-supposed path. + - Auth-detection probe (against the *discovered* signal): present + valid → "authed"; missing/empty → "unavailable"; expired → "unavailable" (do not refresh live — that's egress). + - Sentinel prompt with planted secret outside the `--add-dir` workspace: assert agy cannot surface it. + - Sentinel write `/tmp/agy-write-canary`: assert no file created when `--sandbox` is on. + - Round-trip: agy with the chosen posture returns a valid response on a benign 6-lens prompt. + - Arg-length: for a plan ≥200 KB, the prompt invocation succeeds without shell-arg-length errors (or document the size cap empirically). + - Output parseability: agy's output passes through `parse_findings()` directly or with a documented post-processing step. +- **Verification:** Onboarding doc exists with the agy paid-plan + DPA + **discovered auth-setup** instructions. The findings doc records the corrected CLI surface, the **discovered auth mechanism/credential path**, and the best-effort-posture limitation. The brainstorm doc's R5 (agy posture) and Dependencies/Assumptions section (the `AV_API_KEY` env-var auth assumption — this is where the auth claim actually lives, not R9) are updated to reflect the actual agy surface AND the real auth path, and explicitly note the v1 plan's `~/.gemini/oauth_creds.json` claim was provisional. The user-responsibility framing is documented. + +### U3. Create `ce-deep-review-beta` skill scaffold + headless ce-doc-review invocation (pass 1) + +*(v1 U6 — moved earlier; now the first skill unit, on the thin-slice critical path.)* + +- **Goal:** Stand up the beta skill directory + SKILL.md + Phase-1-of-recipe invocation of `ce-doc-review` in headless mode; parse the structured envelope for pass-2 consumption. +- **Requirements:** R1 (skill exists); R2 (single-path argument); R3 (recipe sequencing); R8 (blocking question tool platform-aware). +- **Dependencies:** None at the skill-code level. **Depends on U2 only for the agy auth-detection rule** in `env-detect.sh` — until U2 lands, `env-detect.sh` carries codex + grok detection and a TODO stub for agy (which is moot during the thin-slice phase since agy is not yet an arm; see U7). +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/pass-1-headless-envelope.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh` (new) + - `.gitkeep` files for references/, scripts/ as needed +- **Approach:** SKILL.md frontmatter follows ce-doc-review's minimal shape (`name`, `description`, `argument-hint`) plus `disable-model-invocation: true` and `[BETA]` prefix. Top of SKILL.md does the AskUserQuestion ToolSearch preload. Pass 1 invokes `Skill("ce-doc-review", "mode:headless ")` and parses the resulting envelope. `env-detect.sh` does offline auth-state checks per CLI: codex via existing project pattern; grok via `XAI_API_KEY` non-empty OR `~/.grok/auth.json` valid; **agy via the rule discovered in U2** (do not hardcode a path here — read it from the U2 finding). Use platform-explicit invocation language. +- **Patterns to follow:** `plugins/compound-engineering/skills/ce-doc-review/SKILL.md` (Phase 0 mode detection; AskUserQuestion preload; headless-mode envelope at `references/synthesis-and-presentation.md`). `plugins/compound-engineering/skills/ce-plan/references/plan-handoff.md` for the headless sub-skill invocation pattern. `docs/solutions/skill-design/post-menu-routing-belongs-inline-2026-04-28.md`. +- **Test scenarios:** + - SKILL.md frontmatter parses as valid YAML; `name` matches directory; description ≤ 1024 chars; `disable-model-invocation: true` set. + - `name:` is `ce-deep-review-beta`. + - `env-detect.sh` prints a structured JSON record `{codex: ok|missing|unauthed, agy: ..., grok: ...}`. + - `env-detect.sh` does NOT call any vendor API — only file presence, env-var presence, `command -v`. + - Pass-1 envelope parsing handles all five top-level envelope sections. +- **Verification:** `bun test tests/frontmatter.test.ts`, `tests/skill-agent-ce-prefix.test.ts`, `tests/skill-shell-safety.test.ts` pass on the new skill. Manually invoking the skill on a small plan produces a parsed envelope. + +### U4. Consent gate — gitleaks preview (**graceful degradation**) + opt-in-per-model + responsibility acknowledgment + +*(v1 U7 — moved earlier onto the thin-slice path; gitleaks behavior changed from hard-block to graceful degradation.)* + +- **Goal:** Implement the single interactive gate that previews content sensitivity (or degrades cleanly when gitleaks is absent), presents per-model opt-in choices, and requires explicit acceptance of egress responsibility. +- **Requirements:** R7 (consent gate three-in-one); R8 (blocking question tool); R9 (auto-detection from U3); Key Decision: opt-in-per-model default-none; Plan-time decision: responsibility acknowledgment line; **v2 decision: gitleaks graceful degradation**. +- **Dependencies:** U3. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/consent-gate.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/gitleaks-scan.sh` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline the consent-gate flow) +- **Approach:** Inline the consent-gate routing in SKILL.md (load-bearing per the inline-routing rule). The gate fires AFTER pass 1 returns. Sub-steps: + 1. `gitleaks-scan.sh` runs `gitleaks detect --no-git --source --report-format json --redact` **iff gitleaks is on PATH**. If present: parse JSON, render hits as `Line N (rule-id): `. **If absent: the wrapper exits with a distinct "unavailable" signal (not an error), and the gate shows the preview-unavailable notice from F5 — it does NOT block.** Document the trade-off explicitly in the gate copy: no automated scan ran; the user is the sole filter. + 2. Render numbered-list-in-chat (per-model opt-in + responsibility ack + cancel; numbered list because AskUserQuestion caps at 4). Use the documented "narrow exception for legitimate option overflow" rule with the "Pick a number or describe what you want." hint. + 3. Responsibility acknowledgment text (working draft): *"I acknowledge that this plan content will be sent to the selected external vendors, and that I have configured each vendor with an appropriate data-handling policy (paid plan + DPA where applicable) per my organization's requirements. I accept responsibility for what is egressed."* The user must say yes AND select ≥1 model to proceed. + 4. Surface the chosen subset to pass 2 as a comma-separated string for `panel-critique.sh --models`. **Record whether the gitleaks preview ran** (`content_preview: ran | unavailable`) for the sidecar audit header. +- **Patterns to follow:** `ce-doc-review` SKILL.md Phase 0; `docs/solutions/skill-design/post-menu-routing-belongs-inline-2026-04-28.md`; `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings-2026-04-17.md` (compact-preview-then-Proceed/Cancel). +- **Test scenarios:** + - `gitleaks-scan.sh` against a plan with a planted AWS key surfaces the hit (rule-id, line, redacted preview). + - `gitleaks-scan.sh` against a clean plan returns an empty findings array. + - **`gitleaks-scan.sh` when gitleaks is NOT installed exits with the "unavailable" signal (distinguishable from both "clean" and "error"); the gate renders the preview-unavailable notice and still requires the responsibility ack (regression test for graceful degradation).** + - Gate with all models available + zero gitleaks hits: presents clean preview + per-model options + ack + cancel; default selection none. + - Gate with 1 model unavailable: presents fewer per-model options + one "skipped because X" note. + - User declines responsibility → routes to F4. + - User accepts responsibility but selects no models → routes to F4 equivalent. + - User accepts + selects ≥1 model → routes to pass 2. + - Routing lines for proceed/cancel are inline in SKILL.md (regression test that fails if they move to a reference). + - Each option label is self-contained and third-person. +- **Verification:** Manually walking the gate exercises each branch (all-models, subset, decline, no-models, **gitleaks-present, gitleaks-absent**). The contract test (U12) asserts routing lines + the preview-unavailable notice exist inline in SKILL.md. + +### U5. Pass-2 dispatcher (thin slice) — **build-time harness bundling + drift test** + shell-out + raw record parsing + state machine + +*(v1 U8 — moved onto the thin-slice path; this is the unit that makes the skill first dogfoodable. Build-time copy + drift test land HERE, not at the end. Output is raw/unverified at this stage — verification arrives in U9.)* + +- **Goal:** Bundle the canonical harness into the skill via a build-time copy step, add the drift contract test, invoke the **bundled** `panel-critique.sh` with the chosen model subset, stream per-(model, lens) progress to chat, parse the resulting records into a structured set, and present them **raw (unverified, clearly labeled)** so the slice is dogfoodable. +- **Requirements:** R3 (recipe sequencing); R6 (subset propagation); R11 (per-model record structure); R15 (progress streaming); Key Decision: bundle the harness under the skill's `scripts/` (build-time copy, **not** symlink — symlinks break the converter per AGENTS.md). +- **Dependencies:** U3, U4. **Does NOT depend on U6/U7/U8** — the thin slice bundles and shells the *current* canonical harness (codex + gemini, sequential, no `--models` flag yet). `panel-critique.sh` invocation during this phase passes the model subset via whatever the current harness supports; if the current harness predates `--models`, the thin slice runs the harness's current default arm set and filters records by the user's selection post-hoc. (U8 adds the real `--models` flag + parallelism, after which the dispatcher passes `--models` directly.) +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh` (new — the build-time copy step) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/panel-critique.sh` (new — bundled copy, produced by bundle-harness.sh) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/arms.py` (new — bundled copy) + - `tests/skills/ce-deep-review-bundle-drift.test.ts` (new — asserts bundled copies byte-match canonical) + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/arm-invocation.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/ship-state-machine.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — invoke bundled script; present raw records labeled `verification: none (thin-slice)`) +- **Approach:** + - **Bundling (build-time copy):** `bundle-harness.sh` copies `scripts/eval/cross_model_review/{panel-critique.sh,arms.py,}` → the skill's `scripts/`. It is run by a maintainer (and in CI) whenever the canonical harness changes. **No symlinks** — the converter copies each skill dir as an isolated unit, so a symlink would dangle on install (AGENTS.md File-References-in-Skills). The bundled copies are checked into the repo (so installed skills are self-contained) and regenerated by re-running `bundle-harness.sh`. + - **Drift test:** `ce-deep-review-bundle-drift.test.ts` reads both the canonical and bundled files and asserts byte-equality (modulo a documented header banner if one is injected). This test **fails after U6/U7/U8 modify the canonical harness until `bundle-harness.sh` is re-run** — that is the intended forcing function: any canonical change must be followed by a re-bundle. Document this in `arm-invocation.md`. + - **Dispatch + state machine:** Invoke `bash "${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh" ... "$PLAN_PATH"` via the runtime Bash tool with narrow `allowed-tools: Bash(bash *panel-critique.sh)`. Stream stderr per-(model, lens) progress to chat (R15). Walk `/tmp/cmre-panel/records/${cli}__${lens}.json`; parse `findings[]` into a structured set keyed by `(arm, lens, finding_index)`. The state-machine reference documents the multi-dimensional state space (consent: pending/granted/declined; pass-1: idle/running/complete/failed; per-arm pass-2: idle/running/ok/timeout/missing/auth_fail/empty/malformed; verification: **none-thin-slice** (this phase) → queued/running/complete (Phase 3); sidecar: unwritten/partial/written). + - **Thin-slice output:** present parsed findings to chat AND write a `.deep-review.md` sidecar whose frontmatter includes `verification: none (thin-slice)` and a prominent banner: *"Cross-model findings below are UNVERIFIED — confabulation-checking is still manual at this stage."* This is the dogfood deliverable. +- **Patterns to follow:** `plugins/compound-engineering/AGENTS.md` "Permission gate on extracted scripts" (`${CLAUDE_SKILL_DIR}` + narrow allowed-tools); `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines-2026-03-27.md`. +- **Test scenarios:** + - `bundle-harness.sh` produces bundled copies that byte-match canonical; the drift test passes immediately after running it. + - Drift test FAILS when canonical `arms.py` is edited without re-bundling (regression guard — assert by editing a fixture canonical and asserting the test reports drift). + - Dispatcher invokes the bundled `panel-critique.sh` and waits. + - Per-(model, lens) progress lines stream to chat as they arrive (inject a `sleep` into a mock arm; verify chat output before completion). + - On all-arms-ok: structured set contains one finding-array per (model, lens) cell. + - On one-arm-timeout/empty/malformed: cells marked with the outcome; thin-slice sidecar notes reduced coverage. + - State machine: `consent: declined` never reaches pass 2. + - Thin-slice sidecar carries `verification: none (thin-slice)` + the unverified banner. +- **Verification:** Live run against a test plan with the current harness produces records (codex + gemini × 6 lenses) and a thin-slice sidecar. `bun test tests/skills/ce-deep-review-bundle-drift.test.ts` passes after bundling. + +> ### ⛳ Phase 1 dogfood gate +> After U5, the skill is runnable end-to-enough to dogfood. **Run `ce-deep-review-beta` on real high-stakes plans for ~1–2 weeks and gather the friction signal:** does collapsing the terminal hop change whether the deep review actually gets run, and how do users react to the unverified output? **Decision:** +> - **Usage lifts / clear appetite** → proceed to Phase 2 (grok + agy + subset/parallel) and Phase 3 (verification + reconciliation) with evidence behind the investment. +> - **Friction was not the bottleneck** → stop here or pivot to the permanent thin-wrapper shape; do NOT build the agy migration, grok arm, and verifier corpus on spec. (The gemini→agy migration may still be forced independently by the 2026-06-18 cutoff for the *eval* harness, but that is decoupled from this skill's investment.) +> - **Equivocal** → extend the dogfood window or narrow the next phase to the single highest-signal addition (likely verification, since it removes the most error-prone manual step). + +### U6. Add grok arm to `arms.py` + +*(v1 U3 — unchanged except now gated by the dogfood signal; lands after U5.)* + +- **Goal:** Extend the cross-model harness with a `grok` arm matching the validated posture from U1. +- **Requirements:** R4 (grok arm); R5 (arm posture symmetry); R6 (subset-selection mechanism). +- **Dependencies:** U1; dogfood gate (proceed decision). +- **Files:** + - `scripts/eval/cross_model_review/arms.py` (modify — add GROK_BASE, `elif cli == "grok"` branch, "grok" to argparse choices) + - `tests/cross-model-review-driver.test.ts` (modify — grok cases mirroring codex/gemini) + - **Re-run `bundle-harness.sh`** so the skill's bundled copy picks up grok (the U5 drift test enforces this). +- **Approach:** Mirror the codex/gemini pattern. Add `GROK_BASE = ["grok", "-p", GROK_INSTRUCTION, ...flags from U1...]`. Use `--prompt-file` via a temp file in `build_invocation`. Add `"grok"` to argparse choices. Posture flag values come from U1's findings doc. +- **Patterns to follow:** `arms.py` CODEX_BASE/GEMINI_BASE constants; build_invocation pattern; argparse choices. +- **Test scenarios:** + - `build_invocation("b_isolated", "grok", doc, rubric)` returns the correct argv shape, a `--prompt-file` temp file with the assembled payload, and a fresh-tempdir cwd. + - `build_invocation("c_fixed_context", "grok", ..., context)` includes the context section. + - Defensive: doc content does not appear in argv (`doc_in_argv == False`). + - Integration smoke (live, optional): `run-arm b_isolated grok ` returns non-empty within timeout. + - `parse_findings` parses grok's chosen output format. + - Drift test passes after re-bundling. +- **Verification:** `python3 arms.py run-arm b_isolated grok ` exits 0 with a non-empty `findings` array. `bun test tests/cross-model-review-driver.test.ts` passes. Drift test green. + +### U7. Migrate gemini arm to agy in `arms.py` + +*(v1 U4 — auth-detection now uses the U2-discovered rule, not a hardcoded `~/.gemini/oauth_creds.json`.)* + +- **Goal:** Replace the legacy gemini arm with the validated agy posture from U2. Carry across the auth-detection update **using the credential signal discovered in U2**. +- **Requirements:** Migration option (a); R5; R9; Pre-v1 Ship Gate 3 (validated in U2). +- **Dependencies:** U2; dogfood gate (proceed decision). +- **Files:** + - `scripts/eval/cross_model_review/arms.py` (modify — replace GEMINI_BASE/AGY_INSTRUCTION with AGY_BASE using validated flags; update detection logic to the U2 rule; update header comment block to reflect agy as canonical, gemini deprecated) + - `scripts/eval/cross_model_review/panel-critique.sh` (modify — replace `gemini` with `agy` in the model loop) + - `tests/cross-model-review-driver.test.ts` (modify — replace gemini cases with agy) + - `tests/cross-model-review-corpus.test.ts` (modify — update arm enumeration) + - **Re-run `bundle-harness.sh`** (drift test enforces). +- **Approach:** Build `AGY_BASE = ["agy", "-p", AGY_INSTRUCTION, "--sandbox", "--add-dir", , ...]`. Append the U2 prompt-side directive. Write the doc to a temp file under a `--add-dir`-scoped workspace; the prompt tells agy to read that path. **Update auth detection to the mechanism U2 discovered** — agy is available iff `command -v agy` succeeds AND the U2-discovered credential signal is present, non-empty, and non-expired (do not call agy to verify — that would be egress). Whatever the path is (it is NOT assumed to be `~/.gemini/oauth_creds.json` — U2 confirms it), the detection reads it from a single documented constant so it is changed in one place. Document the prompt-side-constraint-is-best-effort caveat in the arms.py header. +- **Patterns to follow:** existing codex/gemini arm structure. +- **Test scenarios:** + - `build_invocation("b_isolated", "agy", doc, rubric)` returns the chosen agy posture flags + a `--add-dir` workspace at the doc's temp dir. + - Auth detection: a fake expired credential (at the U2-discovered location) → "unavailable" without invoking agy. + - Auth detection: a non-empty valid credential → "available." + - The arms.py header comment accurately reflects agy's actual CLI surface (no `--prompt-file`/`--output-format`, plan-mode absent) AND the actual auth path. + - Integration smoke (live): with a valid agy login, `run-arm b_isolated agy ` returns non-empty. + - Regression: codex arm output unchanged. + - Drift test passes after re-bundling. +- **Verification:** `python3 arms.py run-arm b_isolated agy ` succeeds when authed. `bun test tests/cross-model-review-driver.test.ts` passes. arms.py header documents the agy migration + user-responsibility for DPA/paid-plan + the real auth path. + +### U8. Extend `panel-critique.sh` with `--models` subset + parallel-across-models execution + +*(v1 U5 — unchanged except for the re-bundle step + dogfood gating.)* + +- **Goal:** Support per-run model selection (R6) and reduce wall-time by parallelizing across models while preserving per-(model, lens) progress lines (R15). +- **Requirements:** R6 (subset selection); R15 (progress streaming); R3 (recipe sequencing). +- **Dependencies:** U6, U7; dogfood gate. +- **Files:** + - `scripts/eval/cross_model_review/panel-critique.sh` (modify — accept `--models codex,grok,agy`; per-model parallel with per-model sequential lenses; emit progress lines) + - `tests/cross-model-review-driver.test.ts` (modify — subset-selection cases) + - **Re-run `bundle-harness.sh`**; update the U5 dispatcher to pass `--models ` directly (replacing the thin-slice post-hoc record filter). +- **Approach:** Parse `--models codex,grok,agy` (default = all available). Fork one subshell per selected model running the six lenses sequentially; each emits `[model lens] findings=N` to stderr. Parent waits on all children. Records land at `${CMRE_OUT_DIR:-/tmp/cmre-panel}/records/${cli}__${lens}.json`. Preserve `CMRE_TIMEOUT` as per-(model, lens) timeout. No cross-vendor retry — emit per-arm outcome (`ok`/`timeout`/`missing`/`auth_fail`/`empty`) to stderr. +- **Patterns to follow:** current `panel-critique.sh` lens-loop; bash background-job pattern. +- **Test scenarios:** + - `--models codex foo.md` runs only codex across six lenses. + - `--models codex,grok foo.md` runs both in parallel; per-(model, lens) progress interleaves on stderr. + - Records keyed `${cli}__${lens}.json` — no collisions. + - One model missing locally → exit 0, emit `[model lens] SKIP — not installed` per lens. + - Wall-time on 6-lens × 3-model with mock arms (`true`) ≤ 1.2× single-arm wall-time. + - Default (no `--models`) unchanged from post-U6/U7 arm set. + - Drift test passes after re-bundling. +- **Verification:** `bash panel-critique.sh --models codex,grok foo.md` exits 0 with records for both models; wall-time on a 3-model run ≤ 60% of sequential. + +### U9. Verification step — agent grounds each cross-model finding against the doc + +*(v1 U9 — unchanged; upgrades the thin-slice output from raw to verified.)* + +- **Goal:** Implement the per-finding verification protocol — orchestrator-as-verifier locates cited text, tags CONFIRMED with inline quote, NOT-FOUND-IN-DOC for confabulations, NEEDS-HUMAN for ambiguous judgment. Verification is blind to the producing model. +- **Requirements:** R10 (verification tags + inline-quote requirement); Key Decision: inline-quote requirement; Key Decision: bidirectional rate measurement (measured in U11). +- **Dependencies:** U5 (parsed records); dogfood gate (proceed). Replaces the thin-slice `verification: none` state with real tags. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/verification-protocol.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline the verification dispatch trigger; remove the thin-slice unverified banner; load the reference) +- **Approach:** For each cross-model finding, dispatch a sub-agent (or in-orchestrator inline pass for small sets) with a prompt containing the plan content + finding text but NOT the producing model identifier. Instruct: (a) locate the cited text/claim; (b) CONFIRMED with inline quote if grounded; (c) NOT-FOUND-IN-DOC if absent; (d) NEEDS-HUMAN if strategic/aesthetic judgment. Use the platform's subagent primitive with `mode` omitted per AGENTS.md. Output schema `{finding_id, tag, quote?, reason?}` — strict; violations route to NEEDS-HUMAN. A backstop grep ("did the inline quote actually appear?") runs synchronously after each CONFIRMED tag. +- **Patterns to follow:** `ce-doc-review` `references/subagent-template.md`; `docs/solutions/skill-design/pass-paths-not-content-to-subagents-2026-03-26.md`; `docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md` (blind-judge). +- **Test scenarios:** + - Verbatim-quote finding → CONFIRMED + inline quote; backstop grep confirms. + - "Plan says X on line 42" with no such text → NOT-FOUND-IN-DOC. + - Strategic judgment, no specific text → NEEDS-HUMAN. + - Verification prompt excludes producing model's name (blind-to-producer; assertable from prompt content). + - CONFIRMED without inline quote → rejected → downgrades to NEEDS-HUMAN. + - Backstop grep mismatch (quote not in plan) → downgrades to NOT-FOUND-IN-DOC with note. +- **Verification:** Manual exercise on a curated set (5+ confabulated, 5+ grounded, 3+ judgment) produces expected tags. Backstop grep catches >95% of false-CONFIRMs on the manual set (U11 measures formally). + +### U10. Reconciliation + sidecar writer with coverage frontmatter + audit metadata + rotation + +*(v1 U10 — adds the `content_preview` audit field from U4's graceful-degradation path.)* + +- **Goal:** Assemble verified panel + cross-model findings + decision-changing union into the sidecar. Write to `.deep-review.md` (cross-model) or `.panel-review.md` (panel-only). Include `coverage:` frontmatter, audit-metadata header (models, timestamp, `git config user.name`, **`content_preview: ran | unavailable`**), inline quotes for CONFIRMED, rotated history (keep last 5). +- **Requirements:** R11 (report structure); R12 (rotation w/ retention cap); R13 (panel-only filename); R14 (filename reservation); Key Decision: commit-as-audit; v2: record content-preview availability. +- **Dependencies:** U9. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/references/reconciliation.md` (new) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/sidecar-rotate.sh` (new — keep last 5) + - `plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md` (modify — inline write trigger; load reference) +- **Approach:** Assemble sidecar markdown: YAML frontmatter with `coverage: full|reduced-confidence|panel-only`, `plan`, `models`, `timestamp`, `user` (`git config user.name`), `content_preview: ran|unavailable`; banner if reduced/panel-only; Claude panel findings (untagged, trusted); cross-model findings grouped by-lens, each tagged with verification status + inline quote for CONFIRMED; decision-changing-union section (verified cross-model findings NOT in the Claude panel). Before writing, rotate any existing sidecar to `.deep-review..md`, then delete rotations beyond the 5 most recent. Filename: `.deep-review.md` if any cross-model arm participated; `.panel-review.md` if zero (R13). DO NOT modify `.gitignore`. +- **Patterns to follow:** repo markdown frontmatter convention (origin: docs/plans/2026-05-24-001-...-cross-model-review-eval-plan.md); sidecar-rotation precedent in `docs/solutions/skill-design/` if any, else cleanest shell wrapper. +- **Test scenarios:** + - `coverage: full` with 3-of-3 arms, no per-arm errors. + - `coverage: reduced-confidence` with 1-of-3 timed out (banner names the missing arm + outcome). + - `coverage: panel-only` with zero arms; filename `.panel-review.md`. + - Audit header includes `git config user.name`, ISO timestamp, participating models, **and `content_preview` state**. + - Inline quote under every CONFIRMED cross-model finding. + - Decision-changing-union lists verified cross-model findings absent from the Claude panel. + - Rotation: 7 prior sidecars → 5 most recent preserved, 2 oldest deleted. + - First run: no existing sidecar; no rotation; lands cleanly. + - `.gitignore` unchanged (regression test). +- **Verification:** Manual end-to-end on a test plan; sidecar has all sections, frontmatter, inline quotes. Run twice → prior rotated. Run 7 times → only 5 most recent rotations remain. + +### U11. Bidirectional verifier rate measurement against held-out corpus (**incl. agy-voiced confabulations + calibration-scope honesty**) + +*(v1 U11 — corpus now stresses agy's voice, not just gemini's; the verdict reports its calibration scope rather than implying universal validity.)* + +- **Goal:** Build the held-out verification corpus and measure false-CONFIRM and false-NOT-FOUND-IN-DOC rates. Gate v1 promotion on both rates ≤ 5% **for the model voices the corpus actually represents**, and report the calibration scope explicitly. +- **Requirements:** R10; Key Decision: bidirectional rate thresholds (≤5% each + consequence); RBP 10 (bidirectional measurement); origin Outstanding Question on false-CONFIRM rate. +- **Dependencies:** U9. **Best run after the dogfood gate so the corpus can be seeded from real agy/grok output**, not only synthetic gemini-flavor confabulations. +- **Files:** + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/corpus/` (new — hand-curated plan + known-confabulated findings, both directions, **gemini-voiced AND agy-voiced AND grok-voiced where available**) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/measure.py` (new — runs verifier against corpus, emits rate report with a `calibration_scope` field) + - `plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verifier-eval/README.md` (new — corpus construction guidance + measurement protocol) + - `docs/solutions/skill-design/2026-MM-DD-ce-deep-review-verifier-rates.md` (new — record of the run + verdict + calibration scope) +- **Approach:** Hand-curate ≥20 findings across both directions: (a) ~10 confabulated (cite text not in the plan, already-resolved issues, fabricated line numbers, plausible-but-fake quotes) and (b) ~10 genuinely-grounded — **each direction sampled across model voices, including agy-voiced terse/blunt phrasings** drawn from real agy output once available (U2 smoke output + early Phase-2 runs), not only gemini-flavored synthetics. The v1 plan's corpus was seeded from one prior gemini eval; v2 explicitly requires agy representation because **agy is the canonical arm being shipped and has not run at scale** — a verifier that passes against synthetic gemini-flavor confabulations could be miscalibrated on agy's actual confabulation profile. Run the verifier with arm identifier blinded. Compute false-CONFIRM = `false_positives / total_confabulated`; false-NOT-FOUND = `false_negatives / total_grounded`. N=3 trials per item (variance reduction). **The report carries a `calibration_scope` field naming which model voices are represented and at what sample size; if agy is under-represented (e.g., < 5 agy-voiced items because agy hasn't produced enough real output yet), the report flags `calibration_scope: gemini-calibrated, agy-pending` and the promotion verdict is `eligible (gemini-voiced); agy-voiced unproven` rather than an unqualified pass.** If either rate > 5%: enact the brainstorm fallback (false-CONFIRM > 5% → default-tag NEEDS-HUMAN; false-NOT-FOUND > 5% → NOT-FOUND-IN-DOC advisory). If both ≤ 5% AND agy is adequately represented: beta eligible for stable promotion. +- **Patterns to follow:** `docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md` (pre-registration + corpus-floor honesty — report `inconclusive` if underpowered, don't fake a verdict); `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md` (N≥3 trials + variance aggregation). +- **Test scenarios:** + - `measure.py` emits `{trials, false_confirm_rate, false_not_found_rate, calibration_scope, per_item: [...]}`. + - N=3 trials per item; report aggregates per-item variance. + - Corpus < 20 items → `inconclusive: true`, no pass/fail verdict. + - **Corpus with < N agy-voiced items → `calibration_scope: gemini-calibrated, agy-pending`; verdict is scope-qualified, not unconditional.** + - Both rates ≤ 5% AND agy adequately represented → `promote: eligible`. + - Either rate > 5% → specific recommendation (`fallback: needs-human-default` or `fallback: advisory-tag`) + the failing items. + - Verifier prompt during measurement excludes the producing model name (assertable from prompt content). + - Confidence-anchored scoring: items tagged `expected_tag`; report compares observed vs expected. +- **Verification:** `measure.py` against the curated corpus produces a JSON report **with an explicit calibration scope**. The solution doc records the measurement, the verdict, the calibration scope, and any fallbacks. Beta-to-stable promotion is gated on this report AND on agy being adequately represented (not merely on the gemini-voiced rate clearing). + +### U12. Test contract + user-facing doc + README update + brainstorm-doc corrections + +*(v1 U12 — drift test removed from here (it lives in U5 now); contract test adds the graceful-degradation + thin-slice assertions; brainstorm correction includes the real agy auth path.)* + +- **Goal:** Add the skill contract test, write the user-facing doc, update the README, and correct the brainstorm doc's agy assumptions discovered in U2. +- **Requirements:** existing repo test conventions; user-facing-doc convention; brainstorm-doc maintenance. +- **Dependencies:** U1, U2, U3, U4, U5, U9, U10, U11. +- **Files:** + - `tests/skills/ce-deep-review-contract.test.ts` (new — asserts SKILL.md structural contract) + - `docs/skills/ce-deep-review.md` (new — user-facing doc; mirror `docs/skills/ce-doc-review.md`) + - `docs/skills/README.md` (modify — add ce-deep-review entry to Document Review category) + - `plugins/compound-engineering/README.md` (modify — add row to Document Review skill table; update component counts) + - `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — correct R5's agy-posture claims + the Dependencies/Assumptions section's env-var auth assumption to the actual agy CLI surface + actual auth path from U2; the obsolete env-var assumption lives in Dependencies/Assumptions, not R9) +- **Approach:** Contract test asserts presence of structural tokens — sidecar filenames (`.deep-review.md`, `.panel-review.md`), `coverage:` enum values, banner copy patterns, verification tags (CONFIRMED, NOT-FOUND-IN-DOC, NEEDS-HUMAN), inline-routing lines for the consent gate, **the gitleaks preview-unavailable notice (graceful-degradation regression guard)**, the AskUserQuestion ToolSearch preload. Use `.toMatch` for regex tolerance. (The bundled-harness drift test already exists from U5 — U12 does not re-create it.) User-facing doc explains: what the skill is, when to use it, how it differs from ce-doc-review, the onboarding requirement (user-responsibility for OAuth + paid plans + DPA, with the **actual** agy auth path from U2), the sidecar artifacts, the panel-only fallback, **and that the beta is invoked explicitly (typed slash command / explicit Skill call) because `disable-model-invocation: true` suppresses only model-auto-invocation, not deliberate user invocation**. README addition uses the existing row shape. The brainstorm-doc edit corrects R5 (no `--approval-mode plan`; best-effort prompt-side posture) and the Dependencies/Assumptions section (real auth path, not the `AV_API_KEY` env-var assumption — R9 itself is generic and does not name the mechanism). +- **Patterns to follow:** `tests/review-skill-contract.test.ts`; `docs/skills/ce-doc-review.md`; `plugins/compound-engineering/README.md` rows. +- **Test scenarios:** + - Contract test asserts SKILL.md contains the `Skill("ce-doc-review", "mode:headless` invocation. + - Asserts the responsibility-acknowledgment requirement. + - Asserts both sidecar filename patterns + the `coverage:` enum values. + - Asserts the consent-gate inline routing lines exist (regression guard). + - **Asserts the gitleaks preview-unavailable notice exists inline (graceful-degradation guard).** + - `bun test tests/frontmatter.test.ts` + `tests/skill-shell-safety.test.ts` pass. + - `bun run release:validate` reports the new skill in counts; no drift errors. + - User-facing doc renders cleanly; FAQ covers OAuth/paid-plan/DPA setup + the explicit-invocation note. +- **Verification:** All bun tests pass. User-facing doc states the OAuth + paid-plan + DPA user-responsibility framing with the real agy auth path, and the explicit-invocation note. The brainstorm doc's R5 + Dependencies/Assumptions section reflect the actual agy surface and auth path. README counts correct. + +--- + +## Alternative Approaches Considered + +- **Replicate ce-doc-review's persona dispatch internally** rather than invoking it as a sub-skill. Rejected: duplicates ~420 lines of orchestration; headless invocation inherits the calibrated pipeline. +- **Permanent thin wrapper** (panel + consent + bash-handoff, no new arm, no verification) as the *final* shape. Not chosen as the destination, but **adopted as the first build stage (U3–U5) and dogfood gate** — v2's compromise between the brainstorm's turnkey decision and the adversarial sequencing finding. If the dogfood gate shows friction was the whole story, this stage *becomes* the shipped shape rather than a throwaway. +- **Phase 0.5 separate "alpha" phase.** Considered (adversarial recommendation). Folded into the main unit sequence instead of a separate phase: U3–U5 are the alpha, run against the current harness, gated before the heavy investment. Same de-risking, no parallel skill directory to maintain. +- **Symlink the canonical harness into the skill** instead of build-time copy. Rejected: the converter copies each skill dir as an isolated unit (AGENTS.md), so a symlink dangles on install. Build-time copy + a drift test is the portable mechanism. +- **Reimplement gitleaks rules in JS/TypeScript.** Rejected: gitleaks combines regex + entropy + stopword tries; reimplementing is brittle. Shell out and **degrade gracefully** when the binary is absent (v2) rather than hard-requiring it. +- **Hard-block when gitleaks is missing** (v1 behavior). Rejected in v2: bouncing gitleaks-less first-timers suppresses the exact adoption signal the skill exists to gather. Graceful degradation + a sole-filter notice + the responsibility ack preserves protection without the wall. +- **Full N×M parallelism** for pass 2. Rejected: complicates progress streaming + error attribution without a meaningful win at three arms. +- **Production-grade retry/circuit-breaker** per vendor. Rejected: overkill for a three-arm developer command. Report per-arm outcome; the user re-runs. +- **agy as canonical with gemini removed entirely.** This is Option (a). Alternative was Option (c) — ship without gemini/agy, add post-migration. Option (a) wins on parity with the 2026-06-15 calendar fallback to (c). + +--- + +## Risk Analysis & Mitigation + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| **Friction was not the actual bottleneck; turnkey investment doesn't pay back** | Medium | High — wasted grok/agy/verifier work | **The Phase 1 dogfood gate is the mitigation.** The thin slice (U3–U5) tests the hypothesis against the current harness before grok/agy/verifier are built. A no-lift signal stops the spend at 3 units, not 12. | +| agy posture-floor cannot be empirically validated in U2 | High | High — forces Option (c) | U2 is early. The 2026-06-15 calendar fallback is the operational margin. If U2 fails, U7 becomes "remove gemini from arms.py." | +| **agy's actual auth mechanism differs from any assumed path** | Medium | Medium — detection rule wrong, agy silently "unavailable" | **v2 makes the auth path a U2 discovery, not an assumption.** Detection reads a single documented constant set from U2's finding. No downstream unit hardcodes `~/.gemini/oauth_creds.json`. | +| grok behavioral smoke test reveals `--permission-mode plan` does not constrain at runtime | Medium | High — grok arm can't ship | U1 is early. Fallback: ship without grok (codex + agy only). | +| **Bundled harness drifts from canonical** | Medium | Medium — skill behavior diverges from repo | **Build-time copy (`bundle-harness.sh`) + a drift test that lands in U5 and fails after any canonical edit until re-bundle.** Symlinks rejected (break converter). Every canonical-modifying unit (U6/U7/U8) re-runs the bundle. | +| Verifier rate measurement (U11) exceeds 5% threshold | Medium | Medium — no stable promotion; NEEDS-HUMAN/advisory tags | Brainstorm specified the consequence. Beta stays beta; users still get usable output with fallback tags. | +| **Verifier passes on gemini-voiced corpus but is miscalibrated on agy** | Medium | Medium — false confidence in promotion | **U11 corpus requires agy-voiced items; the verdict carries a `calibration_scope` field. Promotion gates on agy being adequately represented, not just the gemini rate clearing.** Best run after the dogfood gate so real agy output seeds the corpus. | +| agy `-p` argument-length limit hit by large plans | Medium | Medium — large plans can't route through agy | U2 measures + documents the cap. `--add-dir` workspace + "read the file at " directive sidesteps shell-arg limits. | +| **Gitleaks not installed on user's machine** | High | **Low (v2) — gate degrades gracefully** | **v2: the gate opens anyway, shows a sole-filter notice, still requires the responsibility ack, and records `content_preview: unavailable` in the audit header.** Install instructions in the onboarding doc upgrade the preview. No first-timer bounce. | +| Beta-to-stable promotion never happens | Medium | Low — beta works, just doesn't promote | U11 is the gate; if it persistently misses, the team learns the verifier design needs rework — useful information. | +| **Adoption metric (≥5 runs/2wk) blocked by `disable-model-invocation: true`** | Low | Low — metric uncountable if the skill can't be invoked | **`disable-model-invocation` suppresses only model-auto-invocation; explicit user invocation (typed slash command / explicit `Skill()` call) still works. Documented in U12's user-facing doc + Operational Notes.** The dogfood gate runs are explicit invocations and count toward the metric. | +| Sidecar committed to public repos with sensitive content | Low (after content-preview gate; higher if gitleaks absent) | High | The content-preview gate is the primary mitigation when gitleaks is present; when absent, the sole-filter notice + responsibility ack + `content_preview: unavailable` audit record make the gap explicit. Commit-as-audit is the user's per-repo call. | +| New orchestrator skill changes ce-doc-review's invocation contract | Low | Medium — headless envelope must stay stable | Contract test on the headless envelope (U12). | + +--- + +## Phased Delivery + +Each phase is a candidate PR boundary. Phase reviews are non-trivial — explicit gate decisions are required between phases. + +**Phase 0 — Validation Gates (U1, U2)** *(independent; can run in parallel with Phase 1)* + +PR scope: validation scripts + findings docs (incl. **agy auth-mechanism discovery**) + onboarding doc + brainstorm-doc R5/R9 corrections. No skill code. + +**Phase 0 review gate:** Read both validation findings docs. grok fails → drop grok from v1. agy fails → drop agy, fall back to Option (c). Both fail → v1 is panel-only-with-codex; reconsider shipping. **Confirm U2 documented agy's actual auth path** (not the provisional `~/.gemini/oauth_creds.json`). + +**Phase 1 — Dogfoodable thin slice (U3, U4, U5)** *(against the CURRENT codex+gemini harness; does not wait on Phase 2)* + +PR scope: ce-deep-review-beta skill scaffold + headless pass-1 + consent gate (graceful gitleaks) + dispatcher against the bundled current harness + `bundle-harness.sh` + the drift test + raw-unverified thin-slice output. + +**Phase 1 review gate = the ⛳ dogfood gate.** Frontmatter + ce-prefix + shell-safety + drift tests pass. Then **dogfood the thin slice on real plans for ~1–2 weeks** and decide: proceed to Phase 2/3 (usage lifts), stop/pivot-to-thin-wrapper (no lift), or extend/narrow (equivocal). This is the cheap test of the friction hypothesis. + +**Phase 2 — Harness Extension (U6, U7, U8)** *(gated by the dogfood proceed-decision + Phase 0)* + +PR scope: arms.py + panel-critique.sh + driver tests + re-bundle. Lands the grok arm, gemini→agy migration, `--models` subset + parallelization. Dispatcher upgraded to pass `--models` directly. + +**Phase 2 review gate:** `bun test tests/cross-model-review-*.test.ts` + the drift test pass. Live smoke against each arm produces non-empty findings. + +**Phase 3 — Verification & Reconciliation (U9, U10)** + +PR scope: verification protocol + reconciled sidecar writer + rotation. Upgrades thin-slice raw output to verified, tagged, reconciled sidecars. + +**Phase 3 review gate:** Manual end-to-end exercises F1 (happy path), F2 (partial), F3 (panel-only), F4 (decline), **F5 (gitleaks-absent graceful degradation)**. + +**Phase 4 — Validation & Promotion (U11, U12)** + +PR scope: verifier rate measurement (agy-voiced corpus) + contract test + user-facing doc + README. + +**Phase 4 review gate:** Rate report shows ≤5% each **AND adequate agy representation** (eligible for promotion) or documents the fallback enacted + calibration scope. Contract test passes. README counts correct. brainstorm-doc updated. + +**Calendar fallback trigger (2026-06-15):** If Phase 0 has not completed by 2026-06-15, fall back to Option (c) — ship v1 without agy. Re-scope U7 to "remove gemini from arms.py" and the dispatcher to a 2-arm (codex + grok) configuration. The fallback completes before the 2026-06-18 HTTP-410 cutoff because it removes the agy dependency entirely. + +> **Sequencing note:** the dogfood gate (Phase 1) intentionally runs against the *current* codex+gemini harness, which works until the 2026-06-18 gemini cutoff. If the dogfood window would cross that date, swap the thin slice's gemini arm for codex-only rather than blocking the dogfood on the agy migration — the friction signal does not require a specific second arm. + +--- + +## Dependencies / Prerequisites + +- **Upstream tooling:** gitleaks is **recommended, not required** (v2). The gate degrades gracefully without it; installing it (`brew install gitleaks`) upgrades the content preview from manual-only to automated+manual. Documented in the onboarding doc. +- **Upstream vendor accounts:** User has a paid Antigravity plan with an acceptable DPA; xAI Grok credentials; codex installed and authed. **agy's exact auth/credential configuration is whatever U2 discovers** — the onboarding doc reflects the real mechanism, not a presumed env var or path. User responsibility per Key Decisions. +- **Upstream skill:** `ce-doc-review` must support `mode:headless` (it does). The headless-envelope shape is the contract. +- **Upstream harness:** `scripts/eval/cross_model_review/arms.py` + `panel-critique.sh` exist and follow the documented arm-add pattern. The thin slice bundles them as-is (U5); U6/U7/U8 extend the canonical copies. +- **External deadline:** Gemini CLI HTTP-410 cutoff is 2026-06-18. Phase 0 must complete by 2026-06-15 to maintain Option (a); otherwise Option (c) fallback fires. The thin-slice dogfood can use codex-only if its window crosses the cutoff. + +--- + +## Key Technical Decisions + +- **Beta rollout pattern.** Ship as `ce-deep-review-beta` with `disable-model-invocation: true` + `[BETA]` prefix; promote to stable only after U11's verifier rate measurement passes (with adequate agy representation). **`disable-model-invocation: true` suppresses only model-auto-invocation — explicit user invocation (typed slash command / explicit `Skill()` call) still works, which is how the adoption metric's runs accrue.** (See `docs/solutions/skill-design/beta-skills-framework.md`.) + +- **Dogfood the thin slice before the heavy build.** U3–U5 ship a runnable panel + consent gate + bash-handoff against the *current* harness, emitting raw unverified records, gated by the Phase 1 dogfood gate. Rationale: tests the brainstorm's friction-is-the-bottleneck hypothesis for the cost of 3 units before committing to grok hardening + the agy migration + the verifier corpus. Honors the brainstorm's turnkey *destination* while answering the adversarial *sequencing* finding. + +- **Invoke ce-doc-review headless, not replicate.** Pass 1 uses `Skill("ce-doc-review", "mode:headless ")` and parses the envelope. Avoids duplicating ~420 lines of orchestration. + +- **Bundle the cross-model harness via build-time copy (not symlink).** `bundle-harness.sh` copies canonical `scripts/eval/cross_model_review/*` into the skill's `scripts/`; the bundled copies are checked in so installed skills are self-contained (AGENTS.md). A **drift test (U5)** asserts bundled == canonical and fails after any canonical edit until re-bundle. Symlinks are rejected because the converter copies each skill dir as an isolated unit and a symlink would dangle on install. + +- **Parallel across models, sequential lenses within each model** for pass 2 (lands in U8). ~10–15 min for a 3-model run vs. ~30–60 min sequential, while preserving per-(model, lens) progress streaming (R15). + +- **agy auth detection uses the mechanism discovered in U2 — not a pre-assumed path.** The v1 plan asserted `~/.gemini/oauth_creds.json`; v2 treats that as unverified and makes U2 confirm agy's real credential storage. Detection reads a single documented constant set from U2's finding; no downstream unit hardcodes a path. The skill does NOT verify the DPA — user responsibility at the consent gate. + +- **agy posture is best-effort prompt-side, not runtime-guaranteed.** `--sandbox` (FS-only) + `--add-dir` workspace + prompt-side directive ("read ONLY ; do not call tools"). agy has no `--approval-mode plan` equivalent; documented explicitly in arms.py + the user-facing doc. U2 validates it empirically constrains behavior. + +- **grok `--sandbox ` deferred to U1 measurement.** Likely `read-only`; confirmed empirically; lands as an `arms.py` constant in U6. + +- **gitleaks runs via shell-out and degrades gracefully.** `gitleaks detect --no-git --source --report-format json --redact` when present; **when absent, the gate opens anyway with a sole-filter notice + the responsibility ack, and records `content_preview: unavailable`.** Recommended-not-required dependency; install instructions in onboarding (U2). + +- **Consent gate UI is numbered-list-in-chat.** AskUserQuestion caps at 4; the gate needs per-model opt-in + responsibility ack + cancel. Per AGENTS.md "narrow exception for legitimate option overflow," numbered list with the "Pick a number or describe what you want." hint. + +- **Responsibility acknowledgment text** (working draft; copy-refinable): *"I acknowledge that this plan content will be sent to the selected external vendors, and that I have configured each vendor with an appropriate data-handling policy (paid plan + DPA where applicable) per my organization's requirements. I accept responsibility for what is egressed."* + +- **Sidecar is commit-as-audit; skill does not modify `.gitignore`.** R12's rotation policy stands (keep last 5). The audit header now also records `content_preview: ran | unavailable`. + +- **Verifier dispatch is blind to producing model.** Prompt contains plan content + finding text, NOT model identifier. Mitigates in-family bias. U11 explicitly stresses non-Claude voices, **including agy-voiced findings**. + +- **U11 verdict reports its calibration scope.** Promotion gates on both rates ≤5% AND agy being adequately represented in the corpus — not on the gemini-voiced rate alone. A gemini-calibrated-only corpus yields a scope-qualified verdict, not an unconditional pass. + +- **No retry across vendors.** Per-arm outcome (`ok`/`timeout`/`missing`/`auth_fail`/`empty`/`malformed`) in the sidecar header; coverage degrades from `full` to `reduced-confidence` on any non-`ok`. The user re-runs. + +--- + +## Success Metrics + +- **Friction-hypothesis signal (NEW — the dogfood gate's metric):** during the Phase 1 dogfood window, the thin slice is run on ≥3 real high-stakes plans by ≥1 internal dev, and the team forms a qualitative read on whether removing the terminal hop changed run-likelihood. This is the cheap go/no-go for the rest of the build, not a vanity number. +- **Adoption signal:** internal developers run `ce-deep-review-beta` on ≥5 distinct high-stakes plans within 2 weeks of beta landing. (Manual count from committed sidecar artifacts; explicit invocation per the beta-trigger decision; no telemetry for v1.) +- **Decorrelation value:** ≥30% of full `ce-deep-review` runs surface ≥1 verified CONFIRMED cross-model finding the Claude panel did not raise. (From "decision-changing union" sections in committed sidecars; measurable only after Phase 3 verification lands.) +- **Verifier accuracy:** both false-CONFIRM and false-NOT-FOUND-IN-DOC rates ≤ 5% on the U11 held-out corpus, ≥20 items, N=3 trials, **with adequate agy-voiced representation and an explicit calibration scope**. (Gate for beta-to-stable promotion.) +- **No silent degradation:** every reduced-coverage run carries a visible `coverage: reduced-confidence` / `coverage: panel-only` frontmatter + header banner; every gitleaks-absent run carries `content_preview: unavailable`. (Asserted by U12 contract test.) +- **Onboarding cost:** a new developer can run their first `ce-deep-review` within 30 minutes of reading the onboarding doc. (Operational sanity check during Phase 4 review.) + +--- + +## Scope Boundaries + +- **Out of scope (carried from origin):** + - The cross-model evaluation machinery (judge, trials, GT-match, decision-artifact, record-schema). The arms and harness runner are extended; the evaluation pipeline is not invoked by this skill. + - Per-plan trust-based allow-listing. + - Cost/token-budget estimation in the consent gate. + - Headless / non-interactive mode for ce-deep-review v1. + - Extension to ce-code-review or other artifact types. + - A new non-Claude judge inside the flow. +- **Out of scope (plan-time):** + - Production-grade retry/circuit-breaker per vendor. + - Full N×M parallelism for pass 2. + - Reimplementing gitleaks patterns in JS/TS. + - Replicating ce-doc-review's persona dispatch internally. + - Skill auto-modifying `.gitignore`. + - Custom UX beyond the numbered-list consent gate. +- **Out of scope (v2):** + - A permanently separate "alpha" skill directory. The thin slice IS the beta skill at an earlier maturity; it matures in place rather than living as a parallel artifact. + +### Deferred to Follow-Up Work + +- **Stable promotion (`-beta` → stable).** Gated on U11 (incl. agy representation). Follow-up PR runs the beta-promotion checklist + removes `disable-model-invocation`. +- **Opt-in-none vs. opt-out-with-content-gate friction tradeoff.** Revisit after the first ~10 beta runs. +- **Sidecar `.gitignore` reconsideration.** Plan-time decided commit-as-audit; revisit if committed sidecars leak LLM output into PRs. +- **Per-vendor retry policy.** None currently; add "retry once on timeout" if transient failures suppress completion. +- **Adoption telemetry baked into the skill.** Manual count for now. +- **Cross-platform agent target conversions** for the consent-gate numbered-list pattern. Per-target after stable promotion. + +--- + +## Operational / Rollout Notes + +- **Branch + PR cadence:** Each phase gets its own PR. Phase 0 may run in parallel with Phase 1; Phase 2 must not begin until the **dogfood gate proceed-decision** is recorded. +- **Commit prefixes:** `feat(ce-deep-review-beta): ...` for the thin-slice skill code in Phase 1 (U3, U4, U5); `feat(cross-model-eval): ...` for harness-extension commits in Phase 2 (U6, U7, U8); `feat(ce-deep-review-beta): ...` for Phase 3 (U9, U10) and the U11 verifier measurement; doc/test commits use the relevant scope. Per AGENTS.md, classify by intent and never use `compound-engineering` as a scope. +- **Beta invocation (adoption-metric enablement):** ce-deep-review-beta is invoked explicitly — a typed `/ce-deep-review-beta ` slash command or an explicit `Skill("ce-deep-review-beta", ...)` call. `disable-model-invocation: true` only blocks description-matched auto-invocation. Dogfood runs and the ≥5-runs metric accrue from these explicit invocations. State this in the U3 commit message and the user-facing doc. +- **Skill validation via skill-creator:** per AGENTS.md "Validating Agent and Skill Changes," skill prose behavior cannot be tested via in-session typed-agent dispatch (caches at session start). Use the `skill-creator` skill for iteration. +- **Release-please:** do not hand-bump versions. Routine PRs do not cut releases. +- **Stale-install cleanup:** ce-deep-review-beta is net-new; no entries needed in `STALE_SKILL_DIRS` / `EXTRA_LEGACY_ARTIFACTS_BY_PLUGIN` now. Beta-to-stable promotion adds the `-beta` directory to those registries (handled in the promotion PR). +- **Bundled-harness maintenance:** any change to canonical `scripts/eval/cross_model_review/*` must be followed by `bash plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh`. The drift test (U5) fails CI until this is done. Document in `arm-invocation.md`. +- **Tests:** run `bun test` after each phase. `bun run release:validate` after the final phase. + +--- + +## Outstanding Questions + +### Resolve Before Implementation + +- None at planning time. Phase 0 surfaces implementation-time discoveries (agy posture-floor + **agy auth mechanism/credential path**, grok sandbox profile) into the Phase 0 review gate. The dogfood gate surfaces the friction-hypothesis answer into the Phase 1 → Phase 2 decision. + +### Deferred to Implementation + +- [Affects U1, U6][Technical] Exact `grok --sandbox `. Measured in U1; landed as a constant in U6. Likely `read-only`. +- [Affects U2, U7][Technical] Exact agy posture flag combination. Measured in U2; landed in U7. Best-effort prompt-side; documented. +- [Affects U2, U7][Technical] **agy's real auth mechanism + offline-detectable credential location.** Discovered in U2 (NOT assumed to be `~/.gemini/oauth_creds.json`); landed as a documented detection constant in U3's `env-detect.sh` + U7's arms.py. +- [Affects U2, U7][Technical] agy `-p` argument-length limit for large plans. Measured in U2; documented as a size cap. `--add-dir` workaround. +- [Affects U4][Technical] Final responsibility-acknowledgment copy. Working draft above; refine during U4. +- [Affects U5][Technical] How the thin slice maps the user's model subset onto the *current* harness if it predates `--models` (post-hoc record filter vs. running the harness default). Decided in U5; superseded by U8's real `--models` flag. +- [Affects U5][Technical] Permission gate strategy for `bash ${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh`. Narrow `allowed-tools: Bash(bash *panel-critique.sh)` per AGENTS.md. +- [Affects U10][Technical] Group cross-model findings by lens vs. by arm. Plan recommends by-lens; confirm in U10 by previewing both. +- [Affects U11][Needs research] Held-out corpus construction. Hand-curate ≥20 items; **seed agy-voiced items from real agy output gathered during U2 + the Phase 2 smoke runs**; consider synthetic confabulations from prior eval records for the gemini-voiced portion. Document the build in U11's solution doc. +- [Affects U11][Technical] Exact implementation of the rate-miss fallbacks ("default-tag NEEDS-HUMAN" / "advisory NOT-FOUND") — probably a config flag the orchestrator reads. diff --git a/docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md b/docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md new file mode 100644 index 000000000..bf338427b --- /dev/null +++ b/docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md @@ -0,0 +1,484 @@ +--- +date: 2026-05-28 +type: feat +origin: docs/brainstorms/2026-05-28-ce-deep-review-requirements.md +supersedes: docs/plans/2026-05-28-002-feat-ce-deep-review-skill-plan.md +superseded_by: docs/plans/2026-05-28-004-feat-ce-deep-review-skill-plan.md +status: superseded +title: ce-deep-review — turnkey high-stakes plan review across Claude + non-Claude models (v3) +--- + +# feat: ce-deep-review skill (v3) + +> **v3 note.** Supersedes `2026-05-28-002-...-skill-plan.md`. v2 successfully fixed the round-1 P1 findings (verifier calibration, build-time-copy bundling, gitleaks graceful degradation, agy-auth-as-discovery, calendar fallback — all confirmed clear by the round-2 panel). But the v2 **thin-slice restructure** introduced new issues, which this v3 resolves. Changes from v2: +> +> **Mechanical / clear-fix (applied directly in this draft):** +> 1. **P0 egress fix.** The thin slice no longer relies on post-hoc record filtering (which egressed to deselected models because the current harness has no `--models` flag). U5 now pulls a **minimal `--models` subset guard** onto the thin-slice critical path so egress equals consent from the first runnable slice. (U9 later adds parallelism + full semantics.) +> 2. **Reserved-filename collision fixed.** The thin slice writes `.deep-review-draft.md`, not the R14-reserved `.deep-review.md`. The verified filename is reclaimed when verification lands (U11). +> 3. **Consent gate is a single `AskUserQuestion` multi-select** (models as toggles, default none; ack carried in the stem), NOT a numbered-list overflow — the "needs 5+ options" claim conflated per-model toggles with separate ack/cancel items. Resolves the layout, the ack mechanism, and the zero-model state together. +> 4. **agy "no offline auth signal" branch added** to U2 + the Phase 0 gate (R9 forbids live calls; if agy exposes no offline-detectable signal, it is unavailable → Option (c)). +> 5. **Drift test no longer a manual CI footgun** — `bundle-harness.sh` runs in CI and fails only if the working tree changes; equality is normalized (whitespace/line-endings); the eval-workflow-shares-these-files caveat is documented. +> 6. **bundle-harness scope corrected** — copies only `panel-critique.sh` + `arms.py` (the six lens rubrics are inline heredocs, not standalone files). +> 7. **env-detect must not log/print credential values** (U3 requirement + test). +> 8. **agy detection rule actually lands into `env-detect.sh`** post-U2 (U8 step + Files). +> 9. **Discoverability carved into Phase 1** (new U6): README beta row + onboarding doc + minimal contract tests, so the dogfood window has a findable skill — the rest of the contract/doc work stays in Phase 4 (U13). +> 10. **Metric maturity separated** — thin-slice runs count only toward the dogfood signal; the ≥5-run adoption metric counts verified (post-Phase-3) runs. Sidecars carry a `skill_phase` field. +> 11. **agy-voiced corpus min-sample + fallback defined** (U12). +> 12. **"thin slice becomes shipped shape" reconciled** with the brainstorm's verification decision — any shipped wrapper includes verification; the unverified dump never ships. +> 13. F5 notice copy pinned canonical; banner precedence defined; pass-1 failure UX specified; env-detect parallel-independence framing corrected; beta doc/test naming pinned; gitleaks-absent ack escalated; committed-sidecar leak reminder added. +> 14. Carries forward v2's two safe_auto fixes (F1 covers R15; brainstorm corrections target R5 + Dependencies/Assumptions, not the mislabeled "R5/R9"). +> +> **Design forks (NOT silently resolved — see `## Open Decisions (resolve before Phase 1)`):** the dogfood-gate measurement design, and whether the unverified thin slice is the right probe vs. an even-thinner one. +> +> Unit numbers are re-assigned in execution order (now **13 units / 5 phases**); each unit header maps to its v2 ancestor. + +## Summary + +A new `ce-deep-review-beta` skill that orchestrates the existing 3-pass high-stakes-plan review recipe end-to-end on any plan document: `ce-doc-review` headless (pass 1, Claude panel) → a single consent gate (gitleaks preview with graceful degradation + per-model opt-in multi-select + responsibility acknowledgment) → a bundled cross-model harness (pass 2, egress equals consent) → per-finding verification against the doc with inline-quoted CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN tags → a reconciled sidecar at `.deep-review.md`. + +The build is staged so a **thin, egress-safe, dogfoodable slice ships first** (pass 1 + consent gate + raw *unverified* records), gated by an explicit dogfood checkpoint before the team commits to the grok arm, the gemini→agy migration, the verification layer, and the verifier rate measurement. Ships as beta; promoted to stable after the verifier rate measurement clears (with adequate agy representation). + +--- + +## Decisions (resolved pre-Phase-1, 2026-05-28) + +The three forks left open after round 2 are now decided; the OD-* identifiers are kept as stable references from the units below. Phase 1 may proceed. + +### OD-1 — Dogfood-gate measurement. **DECIDED: adopt the full design below; it is the gate's spec, not a suggestion.** + +The dogfood gate is the plan's load-bearing risk control — it decides whether the team builds ~8 more units. Round 2 (product + adversarial, P1) flagged that the v2 signal had no baseline and no population (false-stop / false-proceed risk) and conflated "the hop didn't matter" with "the *unverified output* wasn't worth re-running for." The adopted design: +- **Baseline:** before Phase 1, count how many high-stakes plans in the prior ~4 weeks got the full deep review vs. were skipped/deferred. This is the denominator. (If the deep review was barely used before — the friction premise — the baseline is near-zero and a sustained uptick by ≥2 devs is effectively the bar.) +- **Falsifiable proceed threshold:** deep review run on a materially higher share of high-stakes plans authored during the window than baseline, by **≥2 distinct devs** — not an absolute count of one author's opportunistic runs. +- **Debrief instrumentation:** when a plan was *not* re-reviewed, record *why* — (a) the hop friction, or (b) the unverified output wasn't trustworthy enough to act on. A predominance of (b) routes to **"proceed to verification (Phase 3)"**, NOT "stop — friction wasn't the bottleneck." The three-way tree must distinguish these. +- **Arm-config caveat:** if the window runs codex-only (post-gemini-cutoff), the signal is single-arm and **provisional** — multi-arm friction (wall-time, output volume, confab noise) was not tested; the Phase 2 proceed-decision must say so. + +### OD-2 — Probe shape. **DECIDED: keep the U5 egress-safe thin slice.** + +The agent runs the cross-model arms turnkey (unverified output) — the faithful friction test (the hop is genuinely removed, not just pre-typed) — and builds the bundling/state-machine infrastructure needed eventually. The even-thinner pre-filled-command probe (panel + consent + a pre-filled bash command in chat) was considered and **rejected**: the user still executes the command, so it tests a weaker "hop removed." OD-1's measurement design is adopted, so the infrastructure is not spent on a gate that can't read its own result. + +### OD-3 — grok `-p` data-retention. **DECIDED: confirmed acceptable for internal Blueprint plan content; grok stays in the consent gate.** + +The brainstorm's Dependencies line recording this as an "unverified assumption" is stale; the Key Decisions "confirmed in scope" framing is authoritative. U13 corrects the brainstorm Dependencies wording to match so the two no longer contradict. + +--- + +## Problem Frame + +The deep-plan-review workflow (Claude panel + cross-model panel + reconcile) is a lever the team has decision-grade evidence on for *decorrelation* (cross-model arms surface validated bugs the Claude panel alone misses) but inconclusive evidence on *team-wide value*. Running it today requires a multi-tool, multi-context workflow: invoke `ce-doc-review`, open a terminal, paste a bash command, wait, return the records to chat, then ask the agent to reconcile and manually verify gemini's confabulation-prone findings. Three pain points compound: + +1. **The pass-2 hop is expensive in attention.** Switching to a terminal for every high-stakes plan is enough friction that the deep review gets skipped or deferred. +2. **Verification is the most error-prone manual step.** Gemini confabulates plausible-but-fake findings; the user, not the agent, currently checks each cross-model finding against the doc. +3. **The workflow assumes a single operator.** The harness was built for one developer with a specific environment; teammates without the toolset have no entry point. + +This skill is the instrument that gathers team-wide evidence in real use, not the productionization of a settled win. v3 tests the "friction-is-the-bottleneck" hypothesis **before** the heavy build via the Phase 1 dogfood gate (see OD-1 for how the gate's verdict is made falsifiable). If the thin slice shows friction was *not* the bottleneck, the team learns it for a few units, not twelve. Risk acknowledged: if the lever does not clear the value bar even after the full v1, the agy migration + grok hardening + verifier work do not pay back; the thinner-wrapper alternative remains available as the permanent shape (and, per the brainstorm, any such permanent wrapper still includes verification — see Alternatives). + +--- + +## Actors + +- A1. Plan author / reviewer (any internal developer): invokes `ce-deep-review` on a plan. May or may not have all non-Claude CLIs installed/configured. +- A2. The orchestrating agent (Claude): runs pass 1, mediates the consent gate, dispatches the cross-model arms (only the selected ones), verifies cross-model findings, writes the report. +- A3. Non-Claude reviewer CLIs (codex, agy, grok): produce cross-model findings under the same six lenses; configured per-environment by the user (responsible for OAuth/API-key setup + vendor data-handling policies); opt-in per-run. + +> During the **thin-slice dogfood phase (Phase 1)** the arms are the ones in the canonical harness *today* — codex + gemini. grok and the gemini→agy migration land in Phase 2. + +--- + +## Key Flows + +- F1. Happy-path deep review with all available non-Claude models + - **Trigger:** A1 invokes `ce-deep-review `. + - **Steps:** + 1. A2 probes the environment for installed+authed non-Claude CLIs via the offline auth-detection rules (R9; agy's rule from U2). + 2. A2 invokes `ce-doc-review` headless; receives the panel envelope. **On pass-1 failure/timeout, A2 surfaces the failure and does NOT open the consent gate** (no egress without panel results). + 3. A2 runs the gitleaks content preview. **If gitleaks is installed,** capture findings. **If absent,** the gate degrades gracefully (F5) — it does not block. + 4. A2 opens the consent gate as a single multi-select question (per-model toggles, default none; responsibility acknowledgment in the stem; content-preview hits or the preview-unavailable notice inline; a Cancel path). Submitting with ≥1 model selected *is* the acknowledgment. + 5. A2 fans **only the selected models** across the six lenses (parallel across models, sequential lenses per model) via the bundled `panel-critique.sh --models `. Egress equals consent. + 6. A2 verifies each cross-model finding against the doc (blind to producing model); tags CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN with inline quotes for CONFIRMED. + 7. A2 writes the reconciled report to `.deep-review.md` (coverage frontmatter, audit header, panel findings untagged, cross-model findings grouped + tagged, decision-changing-union). Raw records remain at `/tmp/cmre-panel/records/`. + 8. A2 streams a summary to chat. + - **Outcome:** a single verified, durable, commit-as-audit sidecar. + - **Covered by:** R1, R2, R3, R4, R5, R6, R7, R8, R9, R10, R11, R12, R15 + +- F1-thin. **Thin-slice dogfood run (Phase 1 only; superseded by F1 once verification lands).** + - **Steps:** F1 steps 1–5 against the current codex+gemini harness, **egress-safe** (only selected arms receive the plan — see U5). Then A2 parses the raw per-(model, lens) records, presents them to chat, and writes a `.deep-review-draft.md` sidecar (NOT `.deep-review.md` — R14 reserves that for verified output) with `skill_phase: thin-slice` + `verification: none` frontmatter and a prominent banner: findings are UNVERIFIED, confabulation-checking is still manual. + - **Outcome:** cross-model findings without the terminal hop, for the dogfood signal. Replaced by F1 when verification (U10/U11) lands. + - **Covered by:** R1, R2, R3, R6, R7, R8, R9, R15 + +- F2. Partial-environment deep review (some non-Claude CLIs missing) + - A2 probes; gate shows only available models + a "skipped because X" note. Proceeds with the subset; sidecar carries `coverage: reduced-confidence` + banner. + - **Covered by:** R2, R3, R6, R7, R9, R11 + +- F3. Panel-only deep review when zero non-Claude CLIs are available + - A2 probes (zero usable), runs `ce-doc-review` headless, writes `.panel-review.md` with `coverage: panel-only`; header + chat banner state `Panel-only deep review (no cross-model arm)` and name each missing CLI with its install/auth command. + - **Covered by:** R2, R13 + +- F4. User declines egress at the consent gate (explicit Cancel) + - A2 outputs the Claude panel findings to chat; does NOT write `.deep-review.md` or `.deep-review-draft.md`. + - **Covered by:** R2, R14 + +- F4-zero. **User submits the gate with no models selected (distinct from explicit Cancel).** + - A2 re-presents the gate once with an inline notice: "Select at least one model to proceed, or choose Cancel for panel-only output." If the user again selects none, treat as F4 (Cancel). This distinguishes the forgot-to-select error from a deliberate decline so the user isn't silently dropped to panel-only. + - **Covered by:** R7, R14 + +- F5. **Consent gate with gitleaks not installed (graceful degradation)** + - The gate does NOT bounce. It opens; in place of the content-preview hit list it shows the canonical preview-unavailable notice (pinned in `consent-gate.md`). The responsibility acknowledgment is **escalated** to state that no automated scan ran (see U4). The sidecar audit header records `content_preview: unavailable (gitleaks not installed)`. + - **Covered by:** R2, R7 + +--- + +## Output Structure + +``` +plugins/compound-engineering/skills/ce-deep-review-beta/ +├── SKILL.md +├── references/ +│ ├── consent-gate.md # multi-select gate flow, canonical gitleaks-absent notice, escalated responsibility prompt +│ ├── verification-protocol.md # per-finding grounding, inline-quote contract, blind-to-producer +│ ├── reconciliation.md # sidecar shape, frontmatter (coverage, skill_phase, content_preview), banner precedence, union assembly +│ ├── arm-invocation.md # how to shell to the bundled panel-critique.sh; per-(model,lens) record parsing; progress/timeout streaming format +│ ├── pass-1-headless-envelope.md # ce-doc-review headless invocation, envelope parsing, FAILURE/timeout UX +│ └── ship-state-machine.md # state dimensions across pass 1, consent, pass 2, verification, sidecar +├── scripts/ +│ ├── bundle-harness.sh # build-time copy: canonical panel-critique.sh + arms.py -> this skill's scripts/ (NO separate rubric files — rubrics are inline heredocs) +│ ├── panel-critique.sh # BUNDLED copy +│ ├── arms.py # BUNDLED copy +│ ├── gitleaks-scan.sh # invokes gitleaks if present; emits parseable JSON; signals "unavailable" cleanly if absent +│ ├── env-detect.sh # offline auth detection (codex, grok, agy [agy rule from U2]); emits ONLY a JSON status record, never credential values +│ └── verifier-eval/ # held-out corpus + measurement harness (U12) +│ ├── corpus/ # gemini-voiced AND agy-voiced AND grok-voiced (where available) + grounded +│ └── measure.py # rate report with calibration_scope field +└── (tests live in /tests/skills/, not here) + +docs/skills/ce-deep-review.md # User-facing doc (feature-named, stable across promotion — NOT -beta) +docs/skills/ce-deep-review-onboarding.md # Setup doc (agy paid plan + DPA + discovered auth, grok login, codex, gitleaks) +tests/skills/ce-deep-review-beta-contract.test.ts # Contract test (matches the -beta skill dir; renamed at promotion) +tests/skills/ce-deep-review-beta-bundle-drift.test.ts # Drift test (lands in U5) +``` + +The per-unit `**Files:**` sections are authoritative; the tree is a scope declaration. + +--- + +## Implementation Units + +13 units / 5 phases, in execution order. Phase 0 (validation) is **schedulable in parallel** with Phase 1 but its *outputs* gate Phase 2 (agy auth rule, posture floor); Phase 1 ships an agy TODO stub until U2 lands (agy is not an arm during the thin-slice phase anyway). See the corrected framing in Phased Delivery. + +### U1. Grok behavioral smoke test + sandbox profile evaluation *(v2 U1 — unchanged)* + +- **Goal:** Empirically verify grok's `--permission-mode plan` + `--disable-web-search` + `--sandbox ` constrain behavior at runtime, not just at flag-parse time. Pick the right sandbox profile. +- **Requirements:** Pre-v1 Ship Gates 1 & 2. +- **Dependencies:** None. +- **Files:** `scripts/eval/cross_model_review/validation/grok-smoke.sh` (new); `.../grok-sentinel.md` (new); `docs/solutions/skill-design/2026-MM-DD-grok-arm-posture-validation.md` (new). +- **Approach:** Sentinel prompt attempting (a) web search, (b) read outside cwd, (c) write inside cwd, (d) spawn subagent. Run against each candidate profile + `--permission-mode plan` + `--disable-web-search` + `--max-turns 1` + `--no-subagents` + `--verbatim`. Capture stdout/stderr/side-effects. Pick the strictest profile that doesn't break legitimate output. Verify `read-only` doesn't block `~/.grok/` auth/session writes. +- **Patterns:** `arms.py` `detect_leak()`. +- **Test scenarios:** planted URL → no request; read `~/.ssh/config` → blocked under read-only/strict; write `/tmp/grok-write-canary` → blocked; "respond with `~/.zshrc`" → content absent; benign prompt still returns valid JSON. +- **Verification:** finding-doc records chosen profile + evidence + limitations; profile recorded as a constant for U7. + +### U2. agy CLI surface verification + auth-mechanism discovery (incl. no-offline-signal branch) + posture-floor validation + onboarding doc *(v2 U2 — adds the "no R9-compliant offline signal" outcome branch)* + +- **Goal:** Re-verify agy's CLI surface; **discover agy's real auth mechanism and whether an offline-detectable credential signal exists** (do not assume `~/.gemini/oauth_creds.json` — agy is a distinct tool from the sunsetting Gemini CLI); document the OAuth/paid-plan-DPA user-responsibility; validate the best-effort posture floor. +- **Requirements:** Pre-v1 Ship Gate 3; R5; R9; RBP 1; RBP 4. +- **Dependencies:** None. +- **Files:** `scripts/eval/cross_model_review/validation/agy-smoke.sh` (new); `.../agy-sentinel.md` (new); `docs/solutions/skill-design/2026-MM-DD-agy-arm-posture-validation.md` (new — auth *mechanism* + detection semantics; see security note below on what to record vs. omit); `docs/skills/ce-deep-review-onboarding.md` (new); `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — R5 posture + Dependencies/Assumptions env-var assumption). +- **Approach:** Verify against `agy --help` (v1.0.3+): prompt surface, output format, plan-mode equivalent (expected absent), **auth mechanism + offline-detectable signal**, `--sandbox` semantics. Run `agy auth status`-equivalent introspection and inspect what files appear after login. Posture floor: `--sandbox` + `--add-dir ` + prompt-side directive ("read ONLY ; no tools; return JSON array"); run the U1 sentinel suite. Onboarding doc: paid Antigravity plan, accept DPA, configure auth per discovered mechanism, verify `agy -p "say hi"`. + - **New outcome branch (R9-critical):** if agy authenticates only via an OS keychain, an encrypted blob, or a *live* `agy auth status` network call — i.e., **no file-presence / env-var / token-expiry signal checkable offline** — then under R9 (no live calls before consent) agy is **unavailable**, the same as a missing binary. Record this as a distinct Phase-0-gate outcome (separate from "posture floor fails") with the same consequence: fall back to Option (c). The skill does not probe agy live to work around a missing offline signal. + - **Security note (record vs. omit):** the solutions doc records the *detection mechanism and result semantics* (e.g., "file-based OAuth token at a documented path; check presence + non-expired"), and names the path; it does NOT dump the credential JSON structure or token contents. Keep "what to look for" in the doc; the exact path also lives as a code constant in `env-detect.sh`/`arms.py`. +- **Test scenarios:** auth-mechanism + offline-signal-existence documented (whatever U2 finds); detection probe returns authed/unavailable/expired off the discovered signal without invoking agy; sentinel secret outside `--add-dir` not surfaced; write-canary blocked; round-trip valid; `-p` arg-length cap for ≥200 KB plans measured; output parseable by `parse_findings()`. +- **Verification:** onboarding doc exists with discovered auth setup; findings doc records corrected surface + auth mechanism + the offline-signal-existence verdict + best-effort-posture limitation; brainstorm R5 + Dependencies/Assumptions corrected (the env-var/`AV_API_KEY` assumption lives in Dependencies/Assumptions, not R9). + +### U3. `ce-deep-review-beta` scaffold + headless pass-1 (with failure UX) + env-detect (no credential leakage) *(v2 U3 — adds pass-1 failure UX + env-detect no-log requirement)* + +- **Goal:** Stand up the beta skill + SKILL.md + headless `ce-doc-review` invocation with explicit failure handling; offline auth detection that never leaks credential material. +- **Requirements:** R1, R2, R3, R8. +- **Dependencies:** Skill-code-level none. Depends on U2 **only for the agy detection rule**; until U2 lands, `env-detect.sh` carries codex + grok detection and an agy TODO stub (moot during the thin-slice phase — agy is not yet an arm). The agy rule is wired in by U8, not left as a permanent stub (see U8 Files). +- **Files:** `SKILL.md` (new); `references/pass-1-headless-envelope.md` (new); `scripts/env-detect.sh` (new). +- **Approach:** Frontmatter: `name: ce-deep-review-beta`, `description` (`[BETA]` prefix), `argument-hint`, `disable-model-invocation: true`. Top-of-file AskUserQuestion ToolSearch preload. Pass 1: `Skill("ce-doc-review", "mode:headless ")`; parse the five envelope sections. **Failure UX:** on parse failure or timeout (define the timeout), emit "Pass 1 failed: [reason] — cannot open the consent gate without panel results. Re-invoke, or run ce-doc-review directly to diagnose." and stop; the gate does not open. `env-detect.sh`: codex (existing pattern), grok (`XAI_API_KEY` non-empty OR `~/.grok/auth.json` valid), agy (the U2 rule). **Hard requirement: `env-detect.sh` outputs ONLY the structured JSON status record to stdout and MUST NOT write credential values, token strings, file contents, or key material to stdout/stderr/logs.** +- **Patterns:** `ce-doc-review/SKILL.md` (Phase 0 mode detection; preload; headless envelope); `ce-plan/references/plan-handoff.md`; `docs/solutions/skill-design/post-menu-routing-belongs-inline-2026-04-28.md`. +- **Test scenarios:** frontmatter valid YAML, name matches dir, `disable-model-invocation: true`; `env-detect.sh` prints `{codex,agy,grok: ok|missing|unauthed}`; **`env-detect.sh` with a populated `~/.grok/auth.json` fixture: the token value does not appear in any output stream**; no vendor API call (file/env/`command -v` only); envelope parsing handles all five sections; **pass-1 failure emits the failure message and does not open the gate.** +- **Verification:** `bun test tests/frontmatter.test.ts`, `skill-agent-ce-prefix.test.ts`, `skill-shell-safety.test.ts` pass; manual invocation on a small plan parses an envelope; manual pass-1-failure path shows the failure UX. + +### U4. Consent gate — single multi-select + graceful gitleaks + escalated responsibility ack *(v2 U4 — gate is now a multi-select AskUserQuestion, not a numbered-list overflow; ack escalates when gitleaks absent; F5 copy pinned canonical)* + +- **Goal:** Implement the single interactive gate: content preview (or graceful degradation), per-model opt-in, explicit egress responsibility — within the platform blocking-question tool. +- **Requirements:** R7, R8, R9 (detection from U3); Key Decision opt-in-per-model default-none; responsibility acknowledgment; v3 graceful gitleaks. +- **Dependencies:** U3. +- **Files:** `references/consent-gate.md` (new — pins the canonical gitleaks-absent notice and the ack text, both labeled "CANONICAL — do not paraphrase"); `scripts/gitleaks-scan.sh` (new); `SKILL.md` (modify — inline the gate flow). +- **Approach:** The gate fires after pass 1 returns successfully. **It is a single `AskUserQuestion` (or platform equivalent) with `multiSelect: true`**, listing the available models as toggle options (default none) plus a Cancel option. With ≤4 models this fits the 4-option cap — no numbered-list overflow is needed (v2's "needs 5+ options" conflated per-model toggles with separate ack/cancel items). The **responsibility acknowledgment is carried in the question stem** (the teaching surface), with explicit framing that *selecting any egress option below confirms the acknowledgment*. Sub-steps: + 1. `gitleaks-scan.sh` runs `gitleaks detect --no-git --source --report-format json --redact` **iff gitleaks is on PATH**. Present: render hits as `Line N (rule-id): ` in the stem. **Absent:** the wrapper exits with a distinct "unavailable" signal (not an error); the stem shows the CANONICAL preview-unavailable notice and the ack escalates (next bullet). It does NOT block. + 2. **Ack text (CANONICAL; copy-refinable once):** base — *"This plan content will be sent to the external vendors you select below. You are responsible for having configured each vendor with an appropriate data-handling policy (paid plan + DPA where applicable). Selecting any model confirms you accept this."* **When gitleaks is absent, append:** *"No automated content scan ran (gitleaks not installed) — you are the sole filter; confirm you have manually checked this plan for secrets/PII before egressing."* + 3. **Outcomes:** ≥1 model selected → consent granted with that subset → pass 2. Zero selected on submit → **F4-zero** (re-prompt once, then Cancel). Cancel → **F4** (panel-only chat, no sidecar). + 4. Surface the subset to pass 2 as a comma-separated string for `panel-critique.sh --models`. Record `content_preview: ran | unavailable` for the sidecar audit header. +- **Patterns:** `ce-doc-review` SKILL.md Phase 0; the multi-select question shape; `docs/solutions/best-practices/ce-pipeline-end-to-end-learnings-2026-04-17.md`. +- **Test scenarios:** all 3 models available + zero hits → multi-select with 3 toggles (default none) + Cancel + ack-in-stem; 1 model unavailable → 2 toggles + "skipped because X"; gitleaks absent → preview-unavailable notice + escalated ack (regression test for graceful degradation); ≥1 selected → pass 2; zero selected → F4-zero re-prompt, then Cancel→F4; Cancel → F4 (no sidecar); each toggle label self-contained + third-person. +- **Verification:** manually walking the gate exercises all branches (all-models, subset, gitleaks-present, gitleaks-absent, zero-selected, Cancel). The U13 contract test asserts the gate uses multi-select, the ack-in-stem, and the canonical preview-unavailable notice exist inline in SKILL.md. + +### U5. Pass-2 dispatcher (thin slice) — egress-safe `--models` + build-time bundling + CI-enforced drift test + state machine *(v2 U5 — P0 egress fix; draft filename; CI-automated drift; bundle scope corrected)* + +- **Goal:** Bundle the canonical harness via a build-time copy, add a CI-enforced drift test, invoke the bundled `panel-critique.sh` for **only the selected models**, stream per-(model, lens) progress (incl. timeout/error states), parse records, and present them **raw/unverified (clearly labeled)** to a `.deep-review-draft.md` sidecar. +- **Requirements:** R3, R6, R7 (egress equals consent), R11 (record structure), R15 (progress + timeout streaming); Key Decision build-time copy (not symlink). +- **Dependencies:** U3, U4. Does NOT depend on Phase 2. +- **Files:** `scripts/bundle-harness.sh` (new); `scripts/panel-critique.sh` (new — bundled); `scripts/arms.py` (new — bundled); `tests/skills/ce-deep-review-beta-bundle-drift.test.ts` (new); `references/arm-invocation.md` (new); `references/ship-state-machine.md` (new); `SKILL.md` (modify); **`scripts/eval/cross_model_review/panel-critique.sh` (modify — add a minimal `--models ` guard; see below)**; CI config (modify — add the re-bundle verification step). +- **Approach:** + - **Egress-safe dispatch (P0 fix).** The thin slice must send the plan to *only* the consented models. The current canonical `panel-critique.sh` hardcodes `run codex` + `run gemini` with no selection. v3 lands a **minimal `--models` subset guard** in `panel-critique.sh` as part of U5 (a small change to the per-lens loop: skip arms not in the subset). This is the only canonical change U5 makes, and it is what U9 later builds full semantics + parallelism on top of. **Do NOT use post-hoc record filtering** — egress happens inside the harness, so filtering after the fact would still have sent the plan to a deselected vendor. (Alternative if a `panel-critique.sh` change is undesirable: invoke `python3 arms.py run-arm ` per consented cell directly. Pick one in implementation; both guarantee egress == consent.) + - **Bundling (build-time copy).** `bundle-harness.sh` copies **only** `scripts/eval/cross_model_review/panel-critique.sh` and `arms.py` into the skill's `scripts/`. The six lens rubrics are **inline heredocs inside `panel-critique.sh`** — there are no standalone rubric files to copy or drift-test (v2 erroneously listed ``). Bundled copies are checked in (installed skills must be self-contained per AGENTS.md); symlinks are rejected (the converter copies each skill dir as an isolated unit, so a symlink dangles on install). + - **Drift test (CI-enforced, not a manual footgun).** `ce-deep-review-beta-bundle-drift.test.ts` asserts the bundled copies equal canonical **modulo normalization** (trim trailing whitespace, normalize line endings, ignore a documented injected header banner if any) — raw byte-equality is too brittle. **CI runs `bundle-harness.sh` then fails only if the working tree changed** — so the fix is mechanical (re-run produces the diff), not a remembered manual step. **Document that the canonical files are shared with the live cross-model *eval* workflow**, so an eval-only commit to `arms.py`/`panel-critique.sh` will also require a re-bundle; the CI step makes that automatic rather than a surprise red build for an unrelated author. + - **Dispatch + state machine.** Invoke `bash "${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh" --models "$PLAN_PATH"` via the runtime Bash tool with narrow `allowed-tools: Bash(bash *panel-critique.sh)`. Stream per-(model, lens) progress to chat. `arm-invocation.md` defines the **streaming format for every outcome**: `[model lens] findings=N` on ok; `[model lens] approaching timeout…` near `CMRE_TIMEOUT`; `[model lens] TIMED OUT — omitting` on timeout; analogous lines for `missing`/`auth_fail`/`empty`/`malformed`. `ship-state-machine.md` documents the state space (consent: pending/granted/declined; pass-1: idle/running/complete/failed; per-arm pass-2: idle/running/ok/timeout/missing/auth_fail/empty/malformed; verification: none-thin-slice (this phase) → queued/running/complete (Phase 3); sidecar: unwritten/partial/written). + - **Thin-slice output.** Write `.deep-review-draft.md` (NOT `.deep-review.md`) with frontmatter `skill_phase: thin-slice`, `verification: none`, `content_preview: ran|unavailable`, and a banner: *"Cross-model findings below are UNVERIFIED — confabulation-checking is still manual at this stage."* +- **Patterns:** AGENTS.md "Permission gate on extracted scripts"; `docs/solutions/skill-design/git-workflow-skills-need-explicit-state-machines-2026-03-27.md`. +- **Test scenarios:** **`panel-critique.sh --models codex foo.md` runs ONLY codex (no egress to gemini) — regression test for the P0**; `bundle-harness.sh` output equals canonical under normalization; drift test FAILS when canonical `arms.py` is edited without re-bundle, PASSES after; per-(model,lens) progress + timeout lines stream (inject a `sleep` mock); all-ok → one array per cell; one-arm-timeout/empty/malformed marked; `consent: declined` never reaches pass 2; draft sidecar carries `skill_phase: thin-slice` + unverified banner; **draft sidecar filename is `.deep-review-draft.md`, never `.deep-review.md`.** +- **Verification:** live run against a test plan with one model deselected confirms the deselected vendor received nothing (inspect `/tmp/cmre-panel/records/` — no file for the deselected arm); drift test green after bundling; CI re-bundle step passes on a clean tree. + +### U6. Phase-1 discoverability slice — README beta row + onboarding link + minimal contract tests *(NEW — addresses "discoverability blocked until Phase 4")* + +- **Goal:** Make the beta findable and minimally guarded during the dogfood window, without waiting for the full Phase-4 doc/test work. +- **Requirements:** dogfood-window discoverability; existing repo test conventions. +- **Dependencies:** U3, U4, U5. +- **Files:** `plugins/compound-engineering/README.md` (modify — add a `[BETA]` Document Review row; note counts are reconciled by release automation); `docs/skills/README.md` (modify — add ce-deep-review under Document Review, marked beta); `docs/skills/ce-deep-review.md` (new — minimal user-facing doc: what it is, that it is beta + explicitly invoked, the thin-slice/unverified caveat, link to the onboarding doc); `tests/skills/ce-deep-review-beta-contract.test.ts` (new — **minimal** assertions only: SKILL.md frontmatter, `ce-` prefix, `disable-model-invocation: true`, multi-select gate present, draft-vs-final filename tokens. Full assertions land in U13). +- **Approach:** A reader of the README must be able to discover the beta in the dogfood window; a developer must be able to find the onboarding doc. Keep the doc thin (it grows in U13). The contract test asserts only what exists after U3–U5; U13 extends it once verification/reconciliation land. +- **Patterns:** `tests/review-skill-contract.test.ts`; `docs/skills/ce-doc-review.md`; existing README rows. +- **Test scenarios:** README beta row renders; doc links the onboarding doc; minimal contract test passes on the U3–U5 skill; `bun run release:validate` does not error on the new beta entry (counts reconciled by release automation). +- **Verification:** the beta is discoverable from `plugins/compound-engineering/README.md` and `docs/skills/README.md` during Phase 1; minimal contract test green. + +> ### ⛳ Phase 1 dogfood gate +> After U6 the skill is discoverable and runnable. **Apply OD-1's measurement design**, then dogfood for ~1–2 weeks. **Decision:** +> - **Usage lifts vs. baseline (≥2 devs)** → proceed to Phase 2 + Phase 3 with evidence. +> - **No lift AND debrief attributes it to the hop (not unverified-output toil)** → stop, or pivot to the permanent thin-wrapper shape (which still includes verification — see Alternatives). +> - **No lift but debrief attributes it to unverified-output toil** → proceed specifically to verification (Phase 3 ahead of full Phase 2 fan-out). +> - **Equivocal / single-arm (codex-only post-cutoff)** → signal is provisional; extend the window or narrow the next step; do not greenlight the full 3-arm build on a 1-arm read. + +### U7. Add grok arm to `arms.py` *(v2 U6 — unchanged except re-bundle is CI-enforced)* + +- **Goal:** Extend the harness with a `grok` arm matching U1's validated posture. +- **Requirements:** R4, R5, R6. +- **Dependencies:** U1; dogfood gate (proceed). +- **Files:** `scripts/eval/cross_model_review/arms.py` (modify); `tests/cross-model-review-driver.test.ts` (modify); re-bundle is automatic via the U5 CI step. +- **Approach:** Mirror codex/gemini. `GROK_BASE = ["grok", "-p", GROK_INSTRUCTION, ...U1 flags...]`; `--prompt-file` temp file; add `"grok"` to argparse choices. +- **Test scenarios:** `build_invocation` shapes correct; context section included; `doc_in_argv == False`; optional live smoke; `parse_findings` parses grok output; drift test green after CI re-bundle. +- **Verification:** `python3 arms.py run-arm b_isolated grok ` exits 0 with non-empty findings; driver test passes. + +### U8. Migrate gemini arm to agy in `arms.py` + wire the agy rule into `env-detect.sh` *(v2 U7 — adds the env-detect.sh agy-rule landing step the panel found missing)* + +- **Goal:** Replace gemini with the validated agy posture; land the U2-discovered auth-detection rule into BOTH `arms.py` and the skill's `env-detect.sh` (removing the U3 TODO stub). +- **Requirements:** Migration Option (a); R5; R9; Pre-v1 Ship Gate 3. +- **Dependencies:** U2; dogfood gate (proceed). +- **Files:** `scripts/eval/cross_model_review/arms.py` (modify); `scripts/eval/cross_model_review/panel-critique.sh` (modify — `gemini` → `agy` in the model loop); **`plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh` (modify — replace the agy TODO stub with the U2-discovered detection constant)**; `tests/cross-model-review-driver.test.ts` (modify); `tests/cross-model-review-corpus.test.ts` (modify); re-bundle via the U5 CI step. +- **Approach:** `AGY_BASE = ["agy", "-p", AGY_INSTRUCTION, "--sandbox", "--add-dir", , ...]` + the U2 prompt-side directive. Auth detection reads the U2-discovered signal from a single documented constant (used by both `arms.py` and `env-detect.sh`; not hardcoded in two places). If U2's outcome was "no offline signal exists," agy stays unavailable and this unit reduces to removing gemini (Option (c)). +- **Test scenarios:** `build_invocation("agy", ...)` posture flags + `--add-dir`; auth detection authed/unavailable/expired off the discovered signal without invoking agy; arms.py header reflects real surface + auth path; live smoke when authed; codex regression unchanged; **`env-detect.sh` now returns a real agy status (no TODO stub)**; drift test green. +- **Verification:** `python3 arms.py run-arm b_isolated agy ` succeeds when authed; driver test passes; `env-detect.sh` reports agy correctly. + +### U9. Extend `panel-critique.sh` — full `--models` semantics + parallel-across-models *(v2 U8 — minimal `--models` already landed in U5; this finalizes semantics + adds parallelism)* + +- **Goal:** Build full per-run model selection (default = all available) and parallelize across models while preserving per-(model, lens) progress (R15). Replaces U5's minimal guard. +- **Requirements:** R6, R15, R3. +- **Dependencies:** U7, U8; dogfood gate. +- **Files:** `scripts/eval/cross_model_review/panel-critique.sh` (modify); `tests/cross-model-review-driver.test.ts` (modify); re-bundle via CI; update the U5 dispatcher to pass `--models` against the full semantics (the minimal guard is subsumed). +- **Approach:** Full `--models codex,grok,agy` parsing; fork one subshell per selected model running six lenses sequentially; each emits `[model lens] findings=N` to stderr; parent waits. Records at `${CMRE_OUT_DIR:-/tmp/cmre-panel}/records/${cli}__${lens}.json`. `CMRE_TIMEOUT` per (model, lens). No cross-vendor retry; emit per-arm outcome. +- **Test scenarios:** `--models codex` runs only codex; `--models codex,grok` parallel; record keys no collisions; missing model → exit 0 + SKIP lines; wall-time ≤ 1.2× single-arm with mock arms; default (no flag) = post-U7/U8 arm set; drift green. +- **Verification:** `--models codex,grok foo.md` exits 0 with both models' records; 3-model wall-time ≤ 60% of sequential. + +### U10. Verification step — agent grounds each cross-model finding against the doc *(v2 U9 — unchanged)* + +- **Goal:** Per-finding verification: locate cited text, tag CONFIRMED with inline quote / NOT-FOUND-IN-DOC / NEEDS-HUMAN; blind to the producing model. +- **Requirements:** R10; inline-quote requirement; bidirectional rate measurement (U12). +- **Dependencies:** U5 (parsed records); dogfood gate. Replaces the thin-slice `verification: none` state with real tags. +- **Files:** `references/verification-protocol.md` (new); `SKILL.md` (modify — inline the verification trigger; remove the thin-slice unverified banner; flip `skill_phase` to `verified`). +- **Approach:** Per finding, dispatch a sub-agent (or in-orchestrator inline pass for small sets) with plan content + finding text but NOT the producing model. Tag per the protocol. Use the platform's subagent primitive with `mode` omitted. Strict output schema `{finding_id, tag, quote?, reason?}`; violations → NEEDS-HUMAN. A synchronous backstop grep ("did the inline quote appear?") runs after each CONFIRMED. +- **Patterns:** `ce-doc-review/references/subagent-template.md`; `docs/solutions/skill-design/pass-paths-not-content-to-subagents-2026-03-26.md`; `.../cross-model-eval-decision-grade-2026-05-26.md`. +- **Test scenarios:** verbatim-quote → CONFIRMED + grep confirms; fabricated line → NOT-FOUND-IN-DOC; strategic judgment → NEEDS-HUMAN; prompt excludes producing model; CONFIRMED without quote → NEEDS-HUMAN; backstop mismatch → NOT-FOUND-IN-DOC. +- **Verification:** manual exercise on a curated set (5+ confab, 5+ grounded, 3+ judgment) yields expected tags; backstop catches >95% of false-CONFIRMs (U12 measures formally). + +### U11. Reconciliation + sidecar writer — reclaim `.deep-review.md`, banner precedence, audit fields, rotation *(v2 U10 — reclaims the verified filename, adds banner-precedence + skill_phase + committed-leak reminder)* + +- **Goal:** Assemble verified panel + cross-model findings + decision-changing union into the sidecar. Write the verified output to `.deep-review.md` (reclaiming it from the thin-slice draft), with coverage + skill_phase + content_preview frontmatter, audit header, inline quotes, rotation (keep last 5), and a defined banner precedence. +- **Requirements:** R11, R12, R13, R14; commit-as-audit; banner precedence; metric-maturity. +- **Dependencies:** U10. +- **Files:** `references/reconciliation.md` (new); `scripts/sidecar-rotate.sh` (new); `SKILL.md` (modify). +- **Approach:** Frontmatter: `coverage: full|reduced-confidence|panel-only`, `skill_phase: verified`, `plan`, `models`, `timestamp`, `user` (`git config user.name`), `content_preview: ran|unavailable`. **Filename reclaim:** verified output writes to `.deep-review.md`; on first verified run, any existing `.deep-review-draft.md` from the thin-slice window is left in place (so the dogfood artifact survives) but the verified file is the canonical one going forward. **Banner precedence (in `reconciliation.md`):** coverage and verification are orthogonal, rendered as separate labeled lines; during thin-slice the UNVERIFIED banner is top; post-verification only the coverage banner shows (no verification banner needed). Cross-model findings grouped by-lens, tagged, inline quote for CONFIRMED; decision-changing-union section. Rotate existing `.deep-review.md` to `.deep-review..md`, delete beyond 5 most recent. `skill_phase` persists in rotated copies so a rotated thin-slice draft is identifiable by frontmatter, not just timestamp. **Committed-leak reminder:** when `content_preview: unavailable`, the chat summary reminds the user the sidecar (which quotes plan content) is about to be written/committed without an automated scan. DO NOT modify `.gitignore`. +- **Patterns:** repo markdown frontmatter convention; sidecar-rotation precedent if any. +- **Test scenarios:** `coverage: full` (3/3); `reduced-confidence` (1/3 timeout, banner names arm); `panel-only` → `.panel-review.md`; audit header includes user/timestamp/models/`content_preview`; inline quote under each CONFIRMED; union section correct; rotation 7→5; first run no rotation; `.gitignore` unchanged; **verified output writes `.deep-review.md`, not `-draft.md`**; `skill_phase` present + persists in rotations; gitleaks-absent run shows the committed-leak reminder. +- **Verification:** manual end-to-end produces a verified sidecar with all sections; reclaim from a pre-existing draft works; rotation caps at 5. + +### U12. Bidirectional verifier rate measurement — agy-voiced corpus, min-sample + fallback, calibration scope *(v2 U11 — adds a concrete min-sample + synthetic-fallback so the promotion gate can actually clear)* + +- **Goal:** Build the held-out corpus and measure false-CONFIRM and false-NOT-FOUND-IN-DOC rates; gate v1 promotion on both ≤ 5% **for the represented model voices**, with an explicit calibration scope and a defined path to "adequate agy representation." +- **Requirements:** R10; bidirectional thresholds + consequences; RBP 10. +- **Dependencies:** U10. Best run after the dogfood gate so real agy/grok output can seed the corpus. +- **Files:** `scripts/verifier-eval/corpus/` (new); `scripts/verifier-eval/measure.py` (new — emits `calibration_scope`); `scripts/verifier-eval/README.md` (new); `docs/solutions/skill-design/2026-MM-DD-ce-deep-review-verifier-rates.md` (new). +- **Approach:** Hand-curate ≥20 findings across both directions (~10 confabulated, ~10 grounded), sampled across voices including **agy-voiced** items from real agy output (U2 smoke + Phase-2 runs). **Define "adequate agy representation" = ≥5 agy-voiced items.** **Fallback if Phase-2 produces < 5 real agy-voiced items by the measurement date:** synthetic agy-voiced items (modeled on the U2 smoke output's phrasing) are acceptable to reach the floor, but the report flags `calibration_scope: agy-synthetic` and the verdict is `eligible (gemini-voiced + agy-synthetic); agy-real pending` — promotion proceeds but the solutions doc records the synthetic caveat and a re-measure trigger once real agy volume accrues. Run blinded; N=3 trials per item. If either rate > 5%: enact the brainstorm fallback (false-CONFIRM > 5% → default-tag NEEDS-HUMAN; false-NOT-FOUND > 5% → NOT-FOUND-IN-DOC advisory). If both ≤ 5% AND agy adequately represented (real or synthetic-flagged): eligible. +- **Patterns:** `docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md` (pre-registration + corpus-floor honesty); `.../safe-auto-rubric-calibration-2026-04-25.md` (N≥3 + variance). +- **Test scenarios:** `measure.py` emits `{trials, false_confirm_rate, false_not_found_rate, calibration_scope, per_item}`; N=3; corpus < 20 → `inconclusive`; < 5 agy items and no synthetic fallback → `calibration_scope: gemini-calibrated, agy-pending`; synthetic fallback used → `calibration_scope: agy-synthetic` + re-measure trigger recorded; both ≤5% + adequate agy → `promote: eligible`; rate miss → specific fallback recommendation; prompt excludes producing model. +- **Verification:** `measure.py` produces a report with an explicit calibration scope; solutions doc records measurement + verdict + scope + any synthetic caveat + re-measure trigger. Promotion gates on this report AND adequate agy representation. + +### U13. Full contract test + finalize docs + README counts + brainstorm corrections *(v2 U12 — minus the discoverability bits moved to U6; minus the drift test which lives in U5)* + +- **Goal:** Extend the contract test to full coverage, finalize the user-facing doc, finalize README counts, correct the brainstorm assumptions discovered in U2. +- **Requirements:** existing repo test/doc conventions; brainstorm maintenance. +- **Dependencies:** U2, U4, U5, U6, U10, U11, U12. +- **Files:** `tests/skills/ce-deep-review-beta-contract.test.ts` (modify — extend the U6 minimal test); `docs/skills/ce-deep-review.md` (modify — finalize); `docs/skills/README.md` (modify); `plugins/compound-engineering/README.md` (modify — final counts); `docs/brainstorms/2026-05-28-ce-deep-review-requirements.md` (modify — R5 posture + Dependencies/Assumptions env-var assumption [env-var/`AV_API_KEY` is in Dependencies/Assumptions, not R9] + the xAI grok `-p` retention line [correct the stale "unverified assumption" to "confirmed" per OD-3]). +- **Approach:** Contract test asserts the full structural set: sidecar filenames (`.deep-review.md`, `.deep-review-draft.md`, `.panel-review.md`), `coverage:`/`skill_phase:`/`content_preview:` enum tokens, banner copy patterns, verification tags, the multi-select gate + ack-in-stem, the canonical gitleaks-absent notice, the AskUserQuestion ToolSearch preload, the `Skill("ce-doc-review", "mode:headless` invocation. `.toMatch` for tolerance. Finalize the user-facing doc: what it is, when to use it, how it differs from ce-doc-review, the onboarding requirement (user-responsibility for OAuth + paid plans + DPA, with the **actual** agy auth path from U2), the sidecar artifacts (draft vs verified), the panel-only fallback, **and that the beta is invoked explicitly — `disable-model-invocation: true` blocks only model-auto-invocation, not deliberate user invocation.** **Naming:** the skill dir + `name` + contract-test filename carry `-beta`; the user-facing doc is feature-named (`ce-deep-review.md`, stable across promotion). The promotion PR renames the skill dir + contract test (and adds them to the stale-artifact registries), not the doc. +- **Patterns:** `tests/review-skill-contract.test.ts`; `docs/skills/ce-doc-review.md`; README rows. +- **Test scenarios:** contract asserts the headless invocation + ack requirement + all three sidecar filename patterns + the enum tokens + the gate multi-select + the canonical gitleaks-absent notice + the explicit-invocation note in the doc; `bun test tests/frontmatter.test.ts` + `skill-shell-safety.test.ts` pass; `bun run release:validate` reports the skill, no drift. +- **Verification:** all bun tests pass; doc states the user-responsibility framing + real agy auth path + explicit-invocation note; brainstorm R5 + Dependencies/Assumptions corrected; README counts correct. + +--- + +## Alternative Approaches Considered + +- **Replicate ce-doc-review's persona dispatch internally.** Rejected: duplicates ~420 lines; headless inherits the calibrated pipeline. +- **Permanent thin wrapper as the final shape.** Not the destination, but U3–U6 are its first stage + dogfood gate. **If the dogfood gate selects the permanent-thin-wrapper outcome, the shipped wrapper STILL includes verification** (panel + consent + *verified* cross-model output) — never the unverified thin-slice dump. This preserves the brainstorm's categorical "full verification, not emit-and-flag" Key Decision; choosing the thin-wrapper destination does not reopen that decision, it only drops the grok arm / parallelism / scale work. (Round-2 adversarial flagged that "thin slice becomes the shipped shape" otherwise contradicts the brainstorm — this reconciliation closes it.) +- **Even-thinner friction probe** (panel + consent + a pre-filled bash command in chat, no bundling/state-machine). Considered and **rejected** (see OD-2): the user still executes the command, so it tests a weaker "hop removed" than the turnkey thin slice, and the bundling/state-machine infra is needed eventually regardless. +- **Phase 0.5 separate "alpha" phase.** Folded into the main sequence (U3–U6 are the alpha, gated before the heavy investment) rather than a parallel skill dir. +- **Symlink the canonical harness into the skill.** Rejected: the converter copies each skill dir as an isolated unit; a symlink dangles on install. Build-time copy + a CI-enforced drift test is the portable mechanism. +- **Post-hoc record filtering for the thin slice (v2's approach).** Rejected in v3: egress happens inside the harness, so filtering after the run still sent the plan to deselected vendors — a P0 consent violation. v3 gates egress with a `--models` subset before the run. +- **Reimplement gitleaks rules in JS/TS.** Rejected: brittle. Shell out; degrade gracefully when absent. +- **Hard-block when gitleaks is missing (v1).** Rejected: bounces first-timers, suppressing the adoption signal. Graceful degradation + escalated ack + `content_preview: unavailable` audit. +- **Full N×M parallelism for pass 2.** Rejected: over-complex for three arms. +- **Production-grade retry/circuit-breaker.** Rejected: report per-arm outcome; the user re-runs. + +--- + +## Risk Analysis & Mitigation + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| **Friction was not the bottleneck; turnkey investment doesn't pay back** | Medium | High | Phase 1 dogfood gate (U3–U6) tests the hypothesis before grok/agy/verifier spend. **OD-1 makes the verdict falsifiable** (baseline + threshold + debrief that separates friction from unverified-output toil) so the gate doesn't false-stop. | +| **Thin slice egresses to a deselected vendor** | ~~High (v2)~~ Eliminated | High | **v3 P0 fix:** the thin slice gates egress with a minimal `--models` subset (U5), not post-hoc filtering. Regression test asserts a deselected vendor receives nothing. | +| agy posture-floor cannot be validated in U2 | High | High | U2 is early; 2026-06-15 calendar fallback to Option (c). | +| **agy authenticates but exposes no R9-compliant offline signal** | Medium | High | **v3:** U2 has an explicit outcome branch + Phase 0 gate outcome → agy unavailable → Option (c). The skill never probes agy live to compensate. | +| grok `--permission-mode plan` doesn't constrain at runtime | Medium | High | U1 early; fallback codex + agy only. | +| **Bundled harness drifts from canonical / CI footgun** | Medium | ~~Medium~~ Low | Build-time copy + **CI runs `bundle-harness.sh` and fails only on a working-tree change** (mechanical fix). Normalized equality (not raw bytes). Documented that eval-only edits to the shared files also trigger a re-bundle, handled automatically by CI. | +| **Dogfood gate signal can't discriminate** | Medium | High | **OD-1 (adopted):** baseline, falsifiable threshold, ≥2 devs, debrief routing, single-arm caveat. | +| Verifier rate measurement exceeds 5% | Medium | Medium | Brainstorm consequence; beta stays beta; usable output with fallback tags. | +| **Verifier miscalibrated on agy (gemini-only corpus)** | Medium | Medium | U12 requires ≥5 agy-voiced items; `calibration_scope` field; synthetic-fallback flagged + re-measure trigger so the gate can clear honestly. | +| agy `-p` arg-length limit on large plans | Medium | Medium | U2 measures the cap; `--add-dir` workaround. | +| **Gitleaks not installed** | High | Low | v3 graceful degradation: gate opens, **ack escalates to state no scan ran**, `content_preview: unavailable` audited. | +| **Committed sidecar leaks plan content from a gitleaks-absent run** | Low–Medium | High | Content-preview gate (when present) + escalated ack + `content_preview: unavailable` audit + **U11 reminder before writing the sidecar when no scan ran**. Commit-as-audit is the user's per-repo call. | +| **`env-detect.sh` leaks credential values** | Low | High | **v3 hard requirement + test:** `env-detect.sh` emits only a JSON status record; token-value fixture test asserts no credential material on any stream. | +| Consent gate UI invented divergently by implementers | ~~Medium~~ Low | Medium | **v3:** single multi-select question (not numbered-list overflow); ack-in-stem; F4-zero handles the no-selection state; `consent-gate.md` pins canonical copy. | +| Beta naming drift (`.deep-review` vs `-beta`) | Low | Low | **v3 pins it:** skill dir + name + contract test carry `-beta`; the doc is feature-named; promotion PR renames dir + test, not the doc. | +| Beta-to-stable promotion never happens | Medium | Low | U12 is the gate; persistent miss is useful information. | +| Adoption metric counts unverified runs as "value delivered" | ~~Medium~~ Low | Low | **v3:** `skill_phase` annotates each sidecar; thin-slice runs count only toward the dogfood signal; the ≥5-run adoption metric counts verified runs. | +| New orchestrator changes ce-doc-review's headless contract | Low | Medium | Contract test on the headless envelope (U13). | + +--- + +## Phased Delivery + +Each phase is a candidate PR boundary. **Framing correction (round-2 coherence):** Phase 0 and Phase 1 are *schedulable in parallel*, but Phase 0's *outputs* (agy auth rule, posture floor) gate Phase 2 — they are not output-independent. Phase 1 ships an agy TODO stub in `env-detect.sh` until U2 lands; U8 wires in the real rule. The thin-slice phase uses codex+gemini, so the agy stub is moot during the dogfood window. + +**Phase 0 — Validation Gates (U1, U2)** *(schedulable parallel with Phase 1; outputs gate Phase 2)* +PR scope: validation scripts + findings docs (incl. agy auth-mechanism discovery + the no-offline-signal verdict) + onboarding doc + brainstorm corrections. +**Gate:** read both findings docs. grok fails → drop grok. agy posture fails OR agy has no R9-compliant offline signal → drop agy, Option (c). Both fail → panel-only-with-codex; reconsider shipping. Confirm U2 documented agy's actual auth path (not the provisional `~/.gemini/oauth_creds.json`). + +**Phase 1 — Dogfoodable thin slice (U3, U4, U5, U6)** *(against the current codex+gemini harness; egress-safe)* +PR scope: scaffold + headless pass-1 (with failure UX) + consent gate (multi-select, graceful gitleaks) + egress-safe dispatcher + bundling + CI-enforced drift test + discoverability slice. +**Gate = the ⛳ dogfood gate.** Tests pass (frontmatter, ce-prefix, shell-safety, drift, minimal contract). **Apply OD-1's measurement design**, dogfood ~1–2 weeks, decide per the four-way tree in the U6 callout. + +**Phase 2 — Harness Extension (U7, U8, U9)** *(gated by dogfood proceed + Phase 0 outputs)* +PR scope: grok arm + gemini→agy migration (incl. landing the agy rule into `env-detect.sh`) + full `--models` + parallelism. CI re-bundle keeps the skill in sync. +**Gate:** `bun test tests/cross-model-review-*.test.ts` + drift test pass; live smoke per arm non-empty. + +**Phase 3 — Verification & Reconciliation (U10, U11)** +PR scope: verification protocol + reconciled sidecar writer (reclaims `.deep-review.md`) + rotation + banner precedence. +**Gate:** manual end-to-end exercises F1, F2, F3, F4, F4-zero, F5. + +**Phase 4 — Validation & Promotion (U12, U13)** +PR scope: verifier rate measurement (agy-voiced corpus + min-sample/fallback) + full contract test + finalized docs + README counts. +**Gate:** rate report ≤5% each AND adequate agy representation (real or synthetic-flagged) → eligible; else documented fallback + calibration scope. Contract test passes. README counts correct. brainstorm corrected. + +**Calendar fallback (2026-06-15):** if Phase 0 hasn't completed by 2026-06-15, fall back to Option (c) — ship without agy. Re-scope U8 to "remove gemini from arms.py" and the dispatcher to a 2-arm (codex + grok) config. Completes before the 2026-06-18 HTTP-410 cutoff (removes the agy dependency). + +> **Sequencing note:** the dogfood gate runs against codex+gemini, which works until 2026-06-18. If the dogfood window would cross that date, swap the gemini arm for **codex-only** rather than blocking on the agy migration — **but** per OD-1, a codex-only (single-arm) dogfood signal is provisional and must not greenlight the full 3-arm build on its own. + +--- + +## Dependencies / Prerequisites + +- **gitleaks:** recommended, not required (v3). The gate degrades gracefully; installing it (`brew install gitleaks`) upgrades the preview. Onboarding doc covers it. +- **Vendor accounts:** paid Antigravity plan + acceptable DPA; xAI Grok credentials; codex installed+authed. **agy's exact auth/credential configuration is whatever U2 discovers** (and U2 may find no offline-detectable signal, in which case agy is unavailable). User responsibility. +- **ce-doc-review** must support `mode:headless` (it does); the headless envelope is the contract. +- **Canonical harness** (`arms.py`, `panel-critique.sh`) exists. The thin slice bundles it + lands a minimal `--models` guard (U5); U7/U8/U9 extend it. +- **External deadline:** Gemini CLI HTTP-410 cutoff 2026-06-18. Phase 0 by 2026-06-15 to keep Option (a). The dogfood can use codex-only if its window crosses the cutoff (provisional signal per OD-1). +- **xAI grok data-retention policy** for `-p` invocations is **confirmed acceptable** for internal Blueprint plan content (user-confirmed 2026-05-28; see OD-3). grok stays in the consent gate. U13 corrects the brainstorm's stale "unverified assumption" Dependencies wording to match the authoritative Key Decisions framing. + +--- + +## Key Technical Decisions + +- **Beta rollout.** `ce-deep-review-beta` with `disable-model-invocation: true` + `[BETA]`; promote after U12 clears (with adequate agy representation). The flag blocks only model-auto-invocation — explicit user invocation (typed slash command / explicit `Skill()` call) still works, which is how dogfood + adoption runs accrue. +- **Dogfood the thin slice before the heavy build.** U3–U6 ship a runnable, **egress-safe** panel + consent gate + bash-handoff against the current harness, gated by the dogfood gate. See OD-1 (gate measurement) and OD-2 (probe shape) in the Decisions section — both resolved. +- **Invoke ce-doc-review headless, not replicate.** +- **Egress equals consent.** The dispatcher sends the plan only to the models selected at the gate — enforced by a `--models` subset guard *before* the harness runs (U5), never post-hoc filtering. +- **Consent gate is a single multi-select question, ack in the stem.** Per-model toggles (default none) + Cancel fit the 4-option cap for ≤4 models; selecting ≥1 model is the acknowledgment; zero-selection → F4-zero re-prompt. No numbered-list overflow. +- **Bundle the harness via build-time copy (not symlink); copy only `panel-critique.sh` + `arms.py`** (rubrics are inline heredocs). A **CI-enforced** drift test (runs `bundle-harness.sh`, fails on a working-tree change; normalized equality) keeps the bundle in sync, including after eval-only edits to the shared files. +- **agy auth detection uses the U2-discovered mechanism — and U2 may find none.** No path is pre-assumed; if no offline signal exists, agy is unavailable (Option (c)). Detection reads a single documented constant used by both `arms.py` and `env-detect.sh`. +- **agy posture is best-effort prompt-side** (`--sandbox` + `--add-dir` + directive); documented. +- **gitleaks degrades gracefully**; the responsibility ack **escalates** when no scan ran; `content_preview: unavailable` is audited. +- **`env-detect.sh` never emits credential material** — only a JSON status record (tested). +- **Sidecar filenames encode trust:** `.deep-review.md` = verified cross-model; `.deep-review-draft.md` = thin-slice unverified; `.panel-review.md` = panel-only. `skill_phase` frontmatter persists through rotation. Commit-as-audit; the skill does not modify `.gitignore`. +- **Verifier dispatch is blind to producing model;** U12 stresses non-Claude voices incl. agy and reports a `calibration_scope`. +- **No retry across vendors;** per-arm outcome in the header; coverage degrades to `reduced-confidence` on any non-`ok`. +- **A permanently-thin-wrapper outcome still ships verification** — never the unverified dump (reconciles with the brainstorm's verification Key Decision). + +--- + +## Success Metrics + +- **Friction-hypothesis signal (the dogfood gate's metric, per OD-1):** measured against a pre-recorded baseline of deep-review skip/defer rate; proceed requires a materially higher review rate by ≥2 distinct devs during the window, with the debrief distinguishing hop-friction from unverified-output toil. Thin-slice runs count only toward THIS signal. +- **Adoption signal:** internal developers run `ce-deep-review-beta` on ≥5 distinct high-stakes plans within 2 weeks of **verification landing (post-Phase 3)** — i.e., verified runs (`skill_phase: verified`), not thin-slice drafts. (Manual count from committed sidecars; explicit invocation.) +- **Decorrelation value:** ≥30% of full (verified) `ce-deep-review` runs surface ≥1 verified CONFIRMED cross-model finding the panel missed. (From decision-changing-union sections; measurable only after Phase 3. No producing artifact exists before Phase 3 — do not evaluate this metric on thin-slice runs.) +- **Verifier accuracy:** both rates ≤ 5% on the U12 corpus, ≥20 items, N=3, with adequate agy representation + an explicit calibration scope. (Promotion gate.) +- **No silent degradation:** every reduced-coverage run carries a visible `coverage:` banner; every gitleaks-absent run carries `content_preview: unavailable`; every thin-slice run carries `skill_phase: thin-slice` + the UNVERIFIED banner. (U13 contract test.) +- **Onboarding cost:** a new developer runs their first deep review within 30 minutes of the onboarding doc. (Phase 4 sanity check.) + +--- + +## Scope Boundaries + +- **Out of scope (carried from origin):** the eval machinery (judge, trials, GT-match, decision-artifact, record-schema); per-plan trust allow-listing; cost/token estimation in the gate; headless/non-interactive ce-deep-review v1; extension to ce-code-review; a new non-Claude judge. +- **Out of scope (plan-time):** production-grade retry/circuit-breaker; full N×M parallelism; reimplementing gitleaks in JS/TS; replicating ce-doc-review internally; skill auto-modifying `.gitignore`; custom gate UX beyond the multi-select. +- **Out of scope (v3):** a permanently separate "alpha" skill dir (the thin slice matures in place); scanning the sidecar itself for secrets (the plan scans the plan; the committed-leak reminder + escalated ack cover the gitleaks-absent residual — sidecar scanning is a deferred follow-up). + +### Deferred to Follow-Up Work + +- **Stable promotion** (`-beta` → stable). Gated on U12 (incl. agy representation). Renames the skill dir + contract test + adds them to the stale-artifact registries. +- **Opt-in-none vs. opt-out-with-content-gate.** Revisit after the first ~10 beta runs — **define the flip criterion before then** (round-2 scope flagged the absent forcing function): e.g., gate click-through and completion rates captured in sidecar metadata. +- **Sidecar `.gitignore` reconsideration** and **scanning the sidecar itself** — revisit if committed sidecars leak LLM output into PRs. +- **Per-vendor retry policy.** +- **Adoption telemetry baked into the skill.** +- **Cross-platform conversion of the multi-select gate** for non-Claude targets. + +--- + +## Operational / Rollout Notes + +- **Branch + PR cadence:** each phase a PR. Phase 2 must not begin until the **dogfood gate proceed-decision** is recorded AND Phase 0 outputs are in. +- **Commit prefixes:** `feat(ce-deep-review-beta): ...` for skill code (U3–U6, U10, U11, U12); `feat(cross-model-eval): ...` for harness commits (U5's `--models` guard, U7, U8, U9); doc/test commits use the relevant scope. Never use `compound-engineering` as a scope. +- **Beta invocation (metric enablement):** explicit — typed `/ce-deep-review-beta ` or explicit `Skill("ce-deep-review-beta", ...)`. Dogfood + adoption runs accrue from these. State this in the U3 commit message + the user-facing doc. +- **Skill validation via skill-creator:** skill prose behavior can't be tested via in-session typed-agent dispatch (caches at session start). Use `skill-creator`. +- **Bundled-harness maintenance:** the CI step (U5) runs `bundle-harness.sh` and fails on a working-tree change, so re-bundle is automatic on any canonical edit — including eval-only edits to the shared `arms.py`/`panel-critique.sh`. Maintainers can run it locally: `bash plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh`. +- **Release-please:** do not hand-bump versions. +- **Stale-install cleanup:** net-new; registry entries handled at promotion. +- **Tests:** `bun test` after each phase; `bun run release:validate` after the final. + +--- + +## Outstanding Questions + +### Resolve Before Phase 1 + +- **None.** OD-1 (gate measurement), OD-2 (probe shape), and OD-3 (xAI retention) were resolved 2026-05-28 — see the Decisions section. Phase 1 may proceed. + +### Deferred to Implementation + +- [U1, U7] grok `--sandbox ` — measured in U1, constant in U7. +- [U2, U8] agy posture flag combination — measured in U2, constant in U8; best-effort prompt-side. +- [U2, U8] **agy's real auth mechanism + whether an offline-detectable signal exists** — discovered in U2; if none exists, agy is unavailable (Option (c)). Landed as a documented constant in `env-detect.sh` + `arms.py`. +- [U2, U8] agy `-p` arg-length cap — measured in U2; `--add-dir` workaround. +- [U4] Final responsibility-ack + gitleaks-absent-notice copy — canonical drafts in `consent-gate.md`; refine once. +- [U5] Whether to land the `--models` guard in `panel-critique.sh` or invoke `arms.py run-arm` per cell — both egress-safe; pick in implementation. +- [U5] Permission gate for `bash ${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh` — narrow `allowed-tools: Bash(bash *panel-critique.sh)`. +- [U11] Group cross-model findings by lens vs. arm — plan recommends by-lens. +- [U12] Held-out corpus construction; agy-voiced sampling + synthetic fallback; min-sample = 5 agy-voiced items. +- [U12] Exact rate-miss fallback implementation (config flag the orchestrator reads). diff --git a/docs/plans/2026-05-28-004-feat-ce-deep-review-skill-plan.md b/docs/plans/2026-05-28-004-feat-ce-deep-review-skill-plan.md new file mode 100644 index 000000000..438d7349c --- /dev/null +++ b/docs/plans/2026-05-28-004-feat-ce-deep-review-skill-plan.md @@ -0,0 +1,357 @@ +--- +date: 2026-05-28 +type: feat +origin: docs/brainstorms/2026-05-28-ce-deep-review-requirements.md +supersedes: docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md +status: active +title: ce-deep-review — residual plan after Phase 0/1 landed; OD-4 resolved (dogfood #2) (v4) +--- + +# feat: ce-deep-review skill (v4 — reconciled to committed reality) + +> **v4 note.** Supersedes `2026-05-28-003-...-skill-plan.md`. v3 was written as a forward-looking +> 13-unit/5-phase plan, but **Phase 0 (U1, U2) and Phase 1 (U3–U6) have since landed on this +> branch** (commits `6072ec54`, `e1226eda`, `006b7090`, `aa558334`). v3 still describes that work +> as pending, so an implementer reading it would re-derive settled decisions or build a divergent +> second copy. This v4 reconciles the plan to the committed state, records the **first real dogfood +> run** (this document was reviewed *by* the skill on 2026-05-28), and folds in the findings that +> run surfaced. It is a **residual** plan: only Phases 2–4 remain, and one of them is now in +> question. +> +> **What changed from v3:** +> 1. **`## Current State` added** — what is actually committed, with the as-built facts that differ +> from the v3 unit specs (env-detect detects codex+gemini, not grok/agy/stub; arms.py already has +> the agy arm; the drift *test* exists but the CI *step* does not). +> 2. **OD-4 (egress classifier) — RAISED in dogfood #1, RESOLVED in dogfood #2.** Dogfood #1 was +> hard-blocked: Claude Code's auto-mode classifier rejected the `panel-critique.sh` dispatch as +> Data Exfiltration because the in-skill gate's bare option labels were not legible as egress +> authorization; it only ran via `!`. The fix — verb-carrying consent labels (`Send the plan to +> `) — shipped in `766c730c`. **Dogfood #2 (2026-05-29, on this v4 doc) confirmed it: the +> in-skill gate cleared the classifier with NO `!`-handoff.** The turnkey premise holds for the +> interactive path. See OD-4 for the residual caveats (single data point; headless/settings path +> still untested). +> 3. **macOS-only agy floor folded into U8** — `arms.py agy_sandbox_prefix()` returns `([], None)` +> off-darwin, so R5's read-only floor is unenforced for non-macOS teammates; the dispatcher must +> platform-gate agy off macOS. +> 4. **U1–U6 collapsed into Current State** (done); residual units renumbered RU1–RU6, each mapping +> to its v3 ancestor. +> 5. **grok decoupled from the dogfood gate** — U1 already ran; grok is DEFERRED pending a version +> bump that fixes the 0.2.8 headless relay-auth bug, not pending the dogfood proceed-decision. +> +> Unchanged v3 content (Actors, Key Flows F1–F5, full Alternatives, Scope Boundaries) remains valid +> — see v3 for that detail rather than re-reading it here; v4 carries only what moved. **Naming +> caveat:** v3 refers to the skill as `ce-deep-review`; the committed artifact is +> `ce-deep-review-beta` — substitute it throughout when reading v3's flows. + +## Summary + +`ce-deep-review-beta` orchestrates the 3-pass high-stakes-plan review recipe: `ce-doc-review` +headless (pass 1, Claude panel) → a single consent gate → a bundled cross-model harness (pass 2) → +[Phase 3] per-finding verification → a reconciled sidecar. **The thin slice (pass 1 + consent + +raw unverified records) is shipped and has been dogfooded twice.** Dogfood #1 surfaced an egress- +classifier block (OD-4); the legibility fix shipped and **dogfood #2 (2026-05-29) confirmed the +in-skill gate now clears the classifier with no `!`-handoff** — the "removes the terminal hop" +premise holds for the interactive path. Residual: the durable `permissions.allow`/headless path is +still untested, and the verification layer (Phases 3–4) remains to build. + +--- + +## Current State (committed on this branch as of 2026-05-28) + +**Phase 0 — Validation (DONE).** +- **U1 grok posture** (`docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md`): + grok **DEFERRED from v1**. grok 0.2.8's CLI surface + `read-only` seatbelt profile are ideal, but + headless `-p` fails on a worker/relay-auth bug (`AuthorizationRequired`) that `grok login` / + `--reauth` do not clear. Re-test on a version bump via `validation/grok-smoke.sh`. Vendor feedback + filed. +- **U2 agy posture** (`docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md`): agy + **1.0.3 is viable** (clean JSON findings; the 1.0.2 empty-output blocker is gone). agy has **no + flag** that delivers R5's read-only floor (`--sandbox` only restricts terminal exec; FS read+write + tools are live; no `--disable-web-search`). Floor enforced at the **OS layer** via a macOS seatbelt + **deny-write denylist** (`validation/agy-readonly.sb.tmpl`); a deny-all-write or any deny-read + profile **hangs agy**, so reads are NOT denied → secret-read-exfil is a documented residual for + untrusted docs (out of v1 scope). **Auth = OAuth at `~/.gemini/oauth_creds.json` + non-empty + `refresh_token`; do NOT gate on `expiry_date` (agy auto-refreshes).** Vendor feedback filed. + +**Phase 1 — Thin slice (DONE).** `plugins/compound-engineering/skills/ce-deep-review-beta/`: +- `SKILL.md`; references `consent-gate.md`, `arm-invocation.md`, `pass-1-headless-envelope.md`, + `ship-state-machine.md` (verification-protocol.md / reconciliation.md correctly absent — Phase 3). +- `scripts/`: `env-detect.sh`, `gitleaks-scan.sh`, `panel-critique.sh` (with the `--models` guard), + `arms.py`, `bundle-harness.sh`, `validation/agy-readonly.sb.tmpl`. +- Tests: `tests/skills/ce-deep-review-beta-bundle-drift.test.ts` (normalized equality vs canonical), + `…-contract.test.ts`. Discoverability: rows in `plugins/compound-engineering/README.md` + + `docs/skills/README.md`, user doc `docs/skills/ce-deep-review.md`. + +**As-built facts that differ from the v3 unit specs (v3 is wrong here; the code is right):** +- `env-detect.sh` detects **codex + gemini only** (the thin-slice arm set). It does **not** detect + grok (deferred) or agy (joins in RU2/U8) and carries **no agy TODO stub**. v3 U3's "grok + agy + + stub" spec and its `~/.grok/auth.json` fixture test do not match and should be discarded. +- `arms.py` **already implements the agy arm end-to-end** (`AGY_INSTRUCTION`, the `agy` branch of + `build_invocation`, `agy_sandbox_prefix()` seatbelt wrapper, `agy` in argparse choices). v3 U8's + "add agy to arms.py" is **done**; only the panel-loop swap + env-detect wiring + platform-gate + remain (RU2). +- The bundled `panel-critique.sh` **already has the `--models` subset guard** (filters arms before + the run). v3 U5's "land a minimal `--models` guard" is **done**; the deferred "guard vs + arms.py-per-cell" question is resolved (guard chosen). +- **Drift is caught by the bun equality test** (`…-bundle-drift.test.ts`), which fails if the + bundle ≠ canonical. The separate **CI step** v3 described ("CI runs `bundle-harness.sh` and fails + on a working-tree change") was **never added** — there is no `.github/` workflow referencing it. + The bun test makes that CI step largely redundant (drift → red `bun test`), but v3's rollout note + and risk-table mitigation overstate it. **Correction, not new work:** treat the bun test as the + drift gate; drop the phantom CI-step claim (or add the step if a stronger gate is wanted — RU6). + +**Dogfood run #1 (2026-05-28).** This plan's predecessor (v3) was reviewed by the skill. Pass 1 +(7-persona Claude panel) + Pass 2 (codex + gemini, 6 lenses, all `ok`, coverage full). Draft sidecar: +`docs/plans/2026-05-28-003-…-skill-plan.md.deep-review-draft.md`. **The dispatch was hard-blocked by +the auto-mode classifier and only ran when the user invoked it via `!`** (see OD-4). + +**Dogfood run #2 (2026-05-29).** This v4 plan was reviewed by the skill in a fresh session after the +`766c730c` legibility fix. Pass 1 (6-persona panel) + Pass 2 (codex + gemini, 6 lenses, all `ok`, +coverage full). Draft sidecar: `docs/plans/2026-05-28-004-…-skill-plan.md.deep-review-draft.md`. +**The dispatch cleared the auto-mode classifier with NO `!`-handoff** — first end-to-end in-skill +confirmation of the OD-4 fix. The panel's headline: this plan was itself stale (OD-4/RU1 written as +open though `766c730c` had already shipped them); this v4 revision folds that in. + +--- + +## Open Decisions (resolve before Phase 3 build) + +### OD-4 — Egress-classifier block. **RESOLVED (dogfood #2, 2026-05-29).** + +Dogfood #1 proved the failure mode: with consent granted at the in-skill gate, the orchestrating +agent's `bash …/panel-critique.sh --models codex,gemini ` call was **denied by Claude Code's +auto-mode permission classifier** as "Data Exfiltration … not cleared by the consent-gate +authorization," regardless of the `allowed-tools: Bash(bash *panel-critique.sh)` declaration. That +run completed only because the user re-issued the command via the `!` prefix (an explicit human +action the classifier permits). + +**Root cause (dogfood #1):** the in-skill gate's bare option labels (`codex (OpenAI)`) returned to +the classifier as a model *selection*, not as authorization to egress — so the classifier saw no +in-conversation consent to send plan content out. **Fix (`766c730c`):** option labels now carry the +egress verb + vendor (`Send the plan to codex (OpenAI)`), making the recorded consent legible. +**Dogfood #2 confirmed it:** the in-skill gate cleared the classifier and ran Pass 2 (codex+gemini, +12/12 cells `ok`) with no `!`-handoff. The "remove the terminal hop" premise holds for the +interactive path. + +Chosen path (the b-legible mechanism, shipped in `766c730c`): make the in-conversation consent +legible to the classifier via verb-carrying labels. The v3 options (a) `!`/permission-rule and (c) +emit-command survive only as the documented fallback ladder when the gate is blocked. + +Residual sub-questions (do NOT block Phases 3–4): +- **Headless/unattended path untested.** Dogfood #2 confirmed the *interactive* gate only. Whether a + durable `permissions.allow` rule clears the classifier for headless runs (no interactive consent + turn) is still open — onboarding flags that rule as UNTESTED for headless. +- **Single data point.** The mechanism (verb-carrying labels vs this session's permission posture) + was not fully isolated; a second independent fresh-session run would harden the conclusion. +- **Defense-in-depth tradeoff (security-lens).** If the headless `permissions.allow` path is adopted + it is session-permanent, making the in-skill consent gate the *sole* egress boundary — the + onboarding rule must require the gate stay non-suppressible. + +OD-1 impact: with the egress block resolved on the interactive path, the dogfood debrief's three-way +attribution (terminal-hop friction / unverified-output distrust / egress-gate block) no longer has a +live egress-block confound there — but the debrief should still record egress-block as a possible +cause for headless users until that path is tested. + +### OD-1, OD-2, OD-3 — carried from v3 (unchanged). + +Gate-measurement design, thin-slice probe shape, and grok `-p` retention remain as decided in v3. +**Caveat:** the dogfood data is single-author and n=2 — dogfood #1 was egress-blocked (now +resolved), dogfood #2 cleared. OD-1's friction signal still needs ≥2 distinct devs and a clean run +before it counts, but the egress-block confound is removed for the interactive path. + +--- + +## Residual Implementation Units (Phases 2–4) + +Renumbered RU1–RU6; each maps to its v3 ancestor. grok work (v3 U1/U7) is out of the sequence — +gated on a grok version bump, not the dogfood gate. + +### RU1. Resolve OD-4 + harden the dispatch path *(DONE — `766c730c`; gate met by dogfood #2)* +- **Goal:** Make the cross-model dispatch runnable as documented under default auto-mode. ✔ +- **What landed (`766c730c`):** verb-carrying consent labels (`Send the plan to `) + an + "Egress-gate legibility" section in `consent-gate.md`; the "If the dispatch is blocked" fallback + ladder in `SKILL.md` Phase 3 + `arm-invocation.md`; the onboarding doc's "Egress permission" + section (headless `permissions.allow` flagged UNTESTED); the contract-test assertions; decision + record `docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md`. +- **Verification:** ✔ **met by dogfood #2 (2026-05-29)** — a fresh-session `/ce-deep-review-beta` + reached Pass 2 (codex + gemini, 12/12 cells `ok`) with no manual `!`. +- **Residual (small, non-blocking):** confirm the durable `permissions.allow`/headless path clears + the classifier for unattended runs; keep the dogfood debrief logging egress-block for the headless + case. + +### RU2. Migrate gemini→agy in the panel runner + wire agy detection + platform-gate *(DONE 2026-05-29)* +- **Goal:** Make agy the default non-codex arm and enforce its floor only where it exists. ✔ +- **Status (DONE):** Arm set is now **codex + agy**. gemini was initially retained as a selectable + fallback, then **fully removed from the skill** (decision 2026-05-29 — it 410s on 2026-06-18, so + shipping it as a fallback that dies in June added no durable value; the shared `arms.py` gemini + arm stays for the cross-model eval). Landed: `panel-critique.sh` default + `CMRE_REPO_DIR` export; + `arms.py` `_repo_root()` honors `CMRE_REPO_DIR` + off-darwin/empty-prefix agy refusal; + `env-detect.sh` agy detection + macOS platform-gate (`unavailable` off-darwin); SKILL.md + + consent-gate.md + arm-invocation.md (agy in the gate as `Send the plan to agy (Antigravity)`); + user-doc arm table; re-bundled (drift green); new `tests/skills/ce-deep-review-beta-arms-ru2.test.ts`. +- **Dependencies:** RU1 (a runnable dispatch). ✔ +- **Approach:** In `scripts/eval/cross_model_review/panel-critique.sh`, swap `gemini`→`agy` in the + default model loop (keep gemini selectable until the 2026-06-18 cutoff). Wire agy detection into + the skill's `env-detect.sh` using the U2 constant (`~/.gemini/oauth_creds.json` + non-empty + `refresh_token`; do NOT gate on expiry). **Platform-gate:** on non-darwin, `env-detect.sh` reports + agy `unavailable` and the gate must not offer it (the seatbelt floor is macOS-only; + `agy_sandbox_prefix()` returns `([], None)` off-mac, so offering agy there violates R5). When agy + joins the skill dispatch, pass the plan's real repo root for the deny-write floor (`git -C + rev-parse --show-toplevel`), NOT arms.py's own location (see `arm-invocation.md` + Phase-2 TODO). **Defense-in-depth (security panel):** also hard-guard `arms.py` itself to refuse + the `agy` arm when `sys.platform != "darwin"` (raise, don't silently return no sandbox), so a + direct `arms.py run-arm … agy` invocation can't bypass the env-detect gate and run unfloored. + Re-bundle (`bundle-harness.sh`); the drift test must stay green. +- **Files:** `scripts/eval/cross_model_review/{panel-critique.sh,arms.py}` (canonical, re-bundled), + skill `scripts/env-detect.sh`, `SKILL.md` + `references/{consent-gate,arm-invocation}.md`, + `docs/skills/ce-deep-review.md`, `tests/skills/ce-deep-review-beta-arms-ru2.test.ts` (new). + (v3's `tests/cross-model-review-driver.test.ts` was the wrong target — that test covers the eval + spine, not panel-critique/arms.py.) +- **Verification (✔ all passed 2026-05-29):** `env-detect.sh` reports agy `ok` on macOS-authed, + `unavailable` on simulated Linux; a live 1-model agy run produced records for all 6 lenses via the + full skill path; `agy-smoke.sh` floor PASS (repo write blocked) + viable under the seatbelt; the + arms.py off-mac guard refuses unfloored agy; drift green; full `bun test` 1427 pass; gemini fully + removed from the skill (env-detect emits codex+agy only; gate/SKILL/docs no longer offer it; the + eval's gemini arm + tests remain green). + +### RU3. Full `--models` semantics + parallel-across-models *(DONE 2026-05-29)* +- **Status (DONE):** `panel-critique.sh` now forks **one background subshell per model** (each runs + the six lenses sequentially) and waits on all — parallel across models, bounding concurrency to + one in-flight request per vendor (the rate-limit/resource mitigation the feasibility lens flagged). + Per-(model, lens) progress lines stream as each cell completes (R15); they interleave, which is + fine (each is self-labeled; records key on `${cli}__${lens}.json` so parallel writers never + collide). **`--models` semantics defined:** default = all available (codex + agy); unavailable / + off-platform arms **warn-SKIP per cell, never fatal** (missing binary, or agy off-macOS) — the + rest still run. Re-bundled; drift green. +- **Verification (✔ 2026-05-29):** live `--models codex,agy` run produced all 12 records with + interleaved progress (proves concurrency); `--models bogusA,bogusB` → exit 0, SKIP lines, no + records; `--models agy` under a Linux `uname` stub → agy SKIP, no record; 3 new RU3 tests in + `tests/skills/ce-deep-review-beta-arms-ru2.test.ts`; full `bun test` 1430 pass. + +### RU4. Verification step — ground each cross-model finding *(DONE 2026-05-29)* +- **Status (DONE):** `scripts/verify-findings.py` (skill-only — not bundled; verification is + skill-specific, not eval-shared) assigns each cross-model finding one verdict: **CONFIRMED** (a + substantial verbatim quote that exists in the plan), **NOT-FOUND-IN-DOC** (a claimed quote that is + absent), **NEEDS-HUMAN** (no substantial quote to check). Pure function of (finding text, doc) → + **blind to the producing model** (the verdict never reads the model label; `verify-records` uses + it only to label output rows). **Scope decision:** v1 is the deterministic quote-grep backstop as + the *sole authoritative gate* — no LLM verifier (it would re-introduce the verifier-contamination + failure mode the panel flagged); a blinded model triage of NEEDS-HUMAN is a possible later add. + Replaces the thin-slice `verification: none` → `verification: quote-grep-backstop`. Protocol: + `references/verification-protocol.md`; SKILL.md Phase 3.5 added. +- **Verification (✔ 2026-05-29):** verify-one CONFIRMED/NOT-FOUND/NEEDS-HUMAN cases pass; lone + identifier quotes don't trivially confirm; smart-quote/whitespace normalization avoids false + NOT-FOUND; verify-records is model-blind (same text → same verdict under different model labels) + and tallies counts; 7 new tests in `tests/skills/ce-deep-review-beta-verify.test.ts` + a contract + test; full `bun test` 1438 pass. + +### RU5. Reconciliation + sidecar writer — reclaim `.deep-review.md` *(DONE 2026-05-29)* +- **Status (DONE):** The skill now writes the **verified `.deep-review.md`** (the reserved + name), replacing the thin-slice draft as the terminal output. `scripts/reconcile.py` (skill-only) + provides two deterministic helpers: `rotate` (rename an existing verified sidecar to + `.deep-review..md`, keep the **5 newest**, prune older — data-loss-safe: the glob matches + rotations only, never the base or the `-draft` sidecar, which addresses the feasibility lens's + rotation data-loss flag) and `render-cross-model` (by-lens, verdict-tagged Markdown with the + grounding quote on CONFIRMED). Frontmatter `skill_phase: verified` + `verification: + quote-grep-backstop` + verdict counts + coverage; **banner precedence** = coverage-only (the + UNVERIFIED banner is gone) with a NEEDS-HUMAN triage note; **decision-changing union** section; + existing `.deep-review-draft.md` left in place; committed-leak reminder when `content_preview: + unavailable`; `.gitignore` untouched (still an open decision). Protocol: + `references/reconciliation.md`; SKILL.md Phase 4 restructured. +- **Verification (✔ 2026-05-29):** rotate keeps the 5 newest by ISO infix, prunes older, never + touches base/draft, refuses a non-`.deep-review.md` path; render-cross-model groups by lens + (canonical order) + orders verdicts + shows grounding quotes; 4 new tests in + `tests/skills/ce-deep-review-beta-reconcile.test.ts` + contract test updated; full `bun test` 1442 + pass. + +### RU6. Verifier rate measurement + full contract test + docs + drift-gate cleanup *(DONE 2026-05-29)* +- **RU6b — verifier rates (DONE, re-scoped):** `verify-findings.py measure` runs a labeled corpus + (`references/calibration/verifier-corpus.json`, 11 grounded + 11 confabulated, incl. format-variant + grounded items) and computes false-CONFIRM (expected NOT-FOUND → CONFIRMED) + false-NOT-FOUND + (expected CONFIRMED → not confirmed); `eligible` when both ≤5%. **Measured 0% / 0% → eligible.** + **Re-scope (deviation from v3 U12, justified by RU4's deterministic decision):** the verifier is a + deterministic, model-blind quote-grep, so v3's **agy-voiced sampling, synthetic-fallback, + `calibration_scope`, and N=3 trials are MOOT** — the verdict is a pure function of (text, doc) with + no model voice and no variance. The measurement is a straight labeled eval, N=1. (If a stochastic + LLM verifier is ever added for NEEDS-HUMAN triage, the v3 calibration machinery returns with it.) +- **RU6a — docs/cleanup (DONE):** README row reframed (verified, not "thin slice unverified"); full + contract test in place (Phase 3.5 verification + verified `.deep-review.md` output + OD-4 block + fallback, accreted across RU4/RU5/RU6); brainstorm corrections confirmed (agy auth = + `~/.gemini/oauth_creds.json` + `refresh_token`, not `AV_API_KEY` — supersedes notes already in the + brainstorm; **grok `-p` retention line corrected to OD-3 = CONFIRMED acceptable**). +- **Drift-gate cleanup (DECIDED):** the committed bun equality test (`…-bundle-drift.test.ts`) is the + drift gate — it runs under `bun test` (which CI's `test` check runs), so drift → red CI already. No + separate `.github/` step is added (it would be redundant); v3's phantom "CI step" claim lives only + in superseded v3 and v4 already corrects it. +- **Verification (✔ 2026-05-29):** corpus eligible at 0%/0%; 2 RU6b tests; full `bun test` 1444 pass; + `release:validate` in sync. + +--- + +## Risk Analysis (delta from v3) + +| Risk | Likelihood | Impact | Mitigation | +|---|---|---|---| +| Turnkey dispatch blocked by harness egress classifier (auto-mode) | Resolved (dogfood #2) | High | OD-4 RESOLVED — verb-carrying consent labels (`766c730c`) make egress legible to the classifier; dogfood #2 cleared with no `!`. Residual: headless/`permissions.allow` path untested. | +| agy floor unenforced off macOS | High (any non-mac dev) | High | RU2 platform-gates agy to `unavailable` off-darwin; the gate never offers an unfloored arm. | +| Dogfood signal confounded by the egress block | Low (interactive) | Medium | Egress block resolved (dogfood #2); debrief still separates friction from unverified-output toil (RU1). Confound persists only for headless until that path is tested. | +| Bundled harness drift | Low | Low | Caught by the committed bun equality test. (v3's CI-step claim was phantom — RU6 cleans it up; the test already protects.) | +| grok unavailable for v1 | Confirmed | Low | Dropped per U1; re-test on a version bump via `grok-smoke.sh`. v1 ships **codex + agy** (gemini removed from the skill 2026-05-29). | +| Gemini HTTP-410 cutoff 2026-06-18 | Resolved (skill) | Medium | gemini removed from the skill ahead of the cutoff (RU2 + 2026-05-29 removal); agy is the sole non-codex skill arm. The eval still references gemini and will need its own handling at the cutoff. | +| Verifier rate measurement exceeds 5% | Medium | Medium | v3 fallback tags (beta stays beta; usable with NEEDS-HUMAN default). | + +Carried-forward v3 risks (consent-gate divergence, env-detect credential leak, committed-sidecar +leak, naming drift) are mitigated by shipped Phase-1 code + tests; see v3 for detail. + +--- + +## Phased Delivery (residual) + +- **Phase 0, Phase 1 — DONE** (see Current State). +- **Phase 2a — OD-4 + dispatch hardening (RU1). DONE** (`766c730c`). **Gate met by dogfood #2:** a + fresh-session deep review reached Pass 2 with no manual `!`. Residual: confirm the + headless/`permissions.allow` path. +- **Phase 2b — Harness extension (RU2, RU3).** **RU2 DONE 2026-05-29** (gemini→agy default swap + + env-detect wiring + macOS platform-gate + off-mac arms.py guard + REPO_DIR plumbing; gemini then + fully removed from the skill — eval arm retained; agy live smoke on macOS PASS; agy `unavailable` + off-mac; drift green). **RU3 DONE 2026-05-29** (parallel-across-models — one subshell per model; + `--models` semantics: default all-available, unavailable arms warn-SKIP not fatal; R15 progress + preserved). **Phase 2b complete.** +- **Phase 3 — Verification & reconciliation (RU4, RU5). COMPLETE 2026-05-29.** RU4 = deterministic + quote-grep backstop (CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN; model-blind; authoritative — no + LLM verifier in v1). RU5 = reconciliation + the verified `.deep-review.md` sidecar (rotation + keep-5; by-lens verdict-tagged render; decision-changing union; banner precedence). Outstanding: + the **manual end-to-end run** over F1, F2, F3, F4, F4-zero, F5 as a final acceptance check (a real + human dogfood, distinct from the unit tests) before promotion. +- **Phase 4 — Validation & promotion (RU6). BUILD COMPLETE 2026-05-29.** Verifier rates measured + **0% / 0% (≤5% gate cleared)** on the labeled corpus; full contract test in place; README reframed; + drift-gate decided (bun test is the gate); brainstorm corrected. (The "adequate agy representation" + sub-gate is moot — the verifier is deterministic + model-blind, so voice doesn't affect rates.) + **Promotion (beta→stable) still gated on the human-run checks below, not on more code:** + - the **manual end-to-end** over F1, F2, F3, F4, F4-zero, F5 (your weekend run on a new plan), and + - the **OD-1 dogfood signal** (≥2 distinct devs over a window) — which needs the skill shipped to + them first (everything is currently local/un-PR'd; OD-1's baseline + debrief instruments are not + yet built). + +**Dogfood gate (OD-1) still applies** between Phase 2a and the Phase 3 build. With OD-4 resolved on +the interactive path, the signal is no longer confounded by the egress block there — but it remains +single-author (n=2) and needs ≥2 distinct devs plus a clean adoption signal before greenlighting +Phase 3. + +--- + +## Outstanding Questions + +### Resolve before Phase 3 build +- **OD-4 — RESOLVED** (dogfood #2): the legible in-skill consent gate clears the classifier on the + interactive path. Open sub-question: does a durable `permissions.allow` rule clear it for + headless/unattended runs? + +### Deferred to implementation (carried from v3, still open) +- agy `-p` arg-length cap on ≥200 KB plans (moot for stdin path; verify for `--add-dir`). +- RU2 REPO_DIR plumbing for the installed-skill case (plan's repo, not arms.py's location). +- Group cross-model findings by lens vs arm (recommend by-lens). +- U12 corpus construction; agy-voiced sampling + synthetic fallback; min-sample = 5 agy items. +- `.gitignore` for sidecars (RU5 currently says "DO NOT modify `.gitignore`"): the security panel + flagged accidental-commit risk for untracked `*.deep-review-draft.md`. Decide: ignore the draft + vs. intentional sidecar-sharing. diff --git a/docs/skills/README.md b/docs/skills/README.md index 2b2aff7d7..543db7a96 100644 --- a/docs/skills/README.md +++ b/docs/skills/README.md @@ -125,6 +125,7 @@ Invoked when a specific need arises — not part of any chain. | Skill | Description | |-------|-------------| | [`/ce-polish-beta`](./ce-polish-beta.md) | Conversational UX polish — start dev server, open browser, iterate together; auto-detects 8 frameworks | +| [`/ce-deep-review-beta`](./ce-deep-review.md) | Deep cross-model plan review — Claude panel + consent-gated non-Claude reviewer CLIs for decorrelated findings (thin slice: findings unverified) | --- diff --git a/docs/skills/ce-deep-review-onboarding.md b/docs/skills/ce-deep-review-onboarding.md new file mode 100644 index 000000000..fd9b862f2 --- /dev/null +++ b/docs/skills/ce-deep-review-onboarding.md @@ -0,0 +1,94 @@ +# ce-deep-review — onboarding & setup + +`ce-deep-review-beta` runs a high-stakes plan through the Claude `ce-doc-review` panel **plus** +one or more non-Claude reviewer CLIs (cross-model decorrelation), verifies every cross-model +finding against the plan, and writes a reconciled sidecar. This doc covers the per-developer +setup it needs. + +> **You are responsible for vendor data-handling.** When you opt a model in at the consent gate, +> the plan content is sent to that vendor. You are responsible for having configured each vendor +> with an appropriate data-handling policy (paid plan + DPA where applicable) per your +> organization's requirements. The skill does not verify this for you. + +## v1 cross-model arms (status as of 2026-05-28 Phase 0 validation) + +| Arm | Status | Why | +|---|---|---| +| **codex** (OpenAI) | ✅ available | `-s read-only` posture; strong, precise reviewer (clean negative control in prior eval) | +| **agy** (Antigravity) | ✅ available, OS-sandboxed | Viable on 1.0.3; read-only floor enforced via a macOS seatbelt profile (agy's own flags don't confine the FS) | +| **grok** (xAI) | ⏸️ deferred | grok 0.2.8 headless reviewer is blocked by a relay-auth bug; re-enabled after a grok fix/version bump (see `docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md`) | + +You need **at least one** arm available. With none, the skill still runs the Claude panel and +writes a `*.panel-review.md` (it refuses to be quiet, not to run). + +## codex + +- Install the OpenAI `codex` CLI and sign in so it runs non-interactively. +- Verify: `codex exec -s read-only --skip-git-repo-check - <<<'say hi'` returns a response. +- No env var required; auth is via codex's own login. + +## agy (Antigravity) + +- Install `agy` (Antigravity CLI) and sign in to a **paid Antigravity plan**, accepting the + appropriate **DPA** with Google for the content you'll send. +- Auth lands at `~/.gemini/oauth_creds.json` (OAuth; agy auto-refreshes via its `refresh_token`, + so a stale `expiry_date` is fine — it refreshes on use). +- Verify: `agy -p "say hi"` returns a non-empty response. +- **Posture:** agy's `--sandbox` flag does **not** restrict the filesystem, so `ce-deep-review` + runs agy inside a macOS `sandbox-exec` (seatbelt) profile that enforces read-only + no arbitrary + writes at the process boundary. No action needed from you; just be aware the floor is OS-enforced. + +## grok (xAI) — deferred + +`grok login` authenticates you, and `grok models` will show you logged in — but grok 0.2.8's +**headless `-p` reviewer** currently fails (`Transport channel closed / AuthorizationRequired` at +the WebSocket-relay layer), independent of login state. grok is therefore deferred from v1. When a +future grok version fixes the relay path, re-run the U1 validation and re-enable the arm with the +documented posture (clean cwd + `--tools ""` + `--permission-mode plan` + `--disable-web-search` ++ `--no-subagents` + `--sandbox read-only` + a generous `--max-turns`). + +## gitleaks (recommended, not required) + +The consent gate previews your plan for secret/PII-shaped content before egress using `gitleaks`. + +- Install: `brew install gitleaks`. +- If gitleaks is **not** installed, the gate still opens but shows a "content preview unavailable — + you are the sole filter" notice and escalates the responsibility acknowledgment. Installing it + upgrades the preview from manual-only to automated + manual. + +## Egress permission (auto-mode) + +The cross-model dispatch shells out to send your plan to the consented vendors. Under Claude +Code's **default auto-mode**, that `bash` call is screened by a permission classifier that reasons +about whether the conversation authorized the egress — the skill's `allowed-tools` declaration is +**not** sufficient on its own (verified 2026-05-28). + +- **Interactive runs (the normal case):** no setup needed. The consent gate's options are phrased + as explicit egress authorizations (`Send the plan to agy (Antigravity)`), which is what the + classifier reads. Selecting a model and proceeding clears the dispatch. If a run is still blocked, + the skill restates your consent and retries, then offers to let you re-issue the command via the + `!` prefix. +- **Unattended / headless runs** (no interactive consent turn — e.g. `/loop`, scheduled, or + piped): add a durable allow rule to your settings so the dispatch is pre-authorized. In + `~/.claude/settings.json` (or project `.claude/settings.json`): + + ```json + { "permissions": { "allow": ["Bash(bash *panel-critique.sh*)"] } } + ``` + + > **Caveat (untested):** the interactive consent path above is empirically confirmed to clear the + > classifier; whether a `permissions.allow` rule *alone* bypasses it for fully-headless runs is + > not yet verified. Add the rule for headless use, but expect the interactive path to be the + > reliable one until the headless path is confirmed. See + > `docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md`. + +## First run + +``` +/ce-deep-review-beta docs/plans/.md +``` + +(The beta is invoked explicitly — typed slash command or an explicit skill call. It does not +auto-trigger.) You'll get the Claude panel, then a consent gate listing the arms available in your +environment (default: none selected — opt in per model), then a verified reconciled sidecar at +`.deep-review.md`. diff --git a/docs/skills/ce-deep-review.md b/docs/skills/ce-deep-review.md new file mode 100644 index 000000000..42395a33d --- /dev/null +++ b/docs/skills/ce-deep-review.md @@ -0,0 +1,62 @@ +# ce-deep-review (beta) + +> **Beta.** `ce-deep-review-beta` is invoked explicitly (it does not auto-trigger). Cross-model +> findings are **verdict-tagged by a deterministic quote-grep backstop** (CONFIRMED / NOT-FOUND-IN-DOC +> / NEEDS-HUMAN); NEEDS-HUMAN findings still need your judgment, and the reconciled verified +> `.deep-review.md` sidecar arrives in a later phase. + +## What it does + +Runs a high-stakes plan through two passes: + +1. **Claude panel (no egress)** — invokes `ce-doc-review` headless: the six-persona panel + (coherence, feasibility, security, scope, product, adversarial). +2. **Cross-model panel (egress, with consent)** — after a single consent gate, fans the plan + across the non-Claude reviewer CLIs you opt in to, for *decorrelated* findings the Claude panel + may have missed. + +It then verifies each cross-model finding against the plan (a deterministic quote-grep backstop — +CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN, blind to the producing model), reconciles them with the +trusted panel findings, and writes a verified sidecar next to the plan at `.deep-review.md` +(`skill_phase: verified`). An existing verified sidecar is rotated to `.deep-review..md` +(5 most recent kept); a thin-slice `.deep-review-draft.md`, if present, is left in place. + +## How it differs from `ce-doc-review` + +`ce-doc-review` is the no-egress single-panel review. `ce-deep-review` adds cross-model +decorrelation — sending the plan to external vendors (with explicit per-model consent) to surface +issues a single model family tends to miss. Use it for genuinely high-stakes plans (irreversible +migrations, credentials, privacy, data cutover), not routine ones. + +## Arms (v1) + +| Arm | Status | +|-----|--------| +| **codex** (OpenAI) | available | +| **agy** (Antigravity) | available — the non-codex arm; macOS-only (its read-only floor is a seatbelt) | +| **grok** (xAI) | deferred — blocked by a grok 0.2.8 headless relay-auth bug | +| **gemini** (Google) | retired from the skill (410s 2026-06-18); arm retained only in the cross-model eval | + +You need at least one arm installed + authenticated. With none, the skill runs the Claude panel and +writes a `*.panel-review.md` (it refuses to be quiet, not to run). + +## Consent & safety + +- **One gate.** Before any egress, a single interaction previews the plan for secret-shaped content + (`gitleaks`, if installed), takes **per-model opt-in** (default: none selected), and captures your + acknowledgment that you are responsible for each vendor's data-handling policy. +- **Graceful without gitleaks.** If `gitleaks` isn't installed the gate still opens, tells you no + automated scan ran (you're the sole filter), and escalates the acknowledgment. +- **Egress = consent.** Only the models you select receive the plan. + +See [`ce-deep-review-onboarding.md`](./ce-deep-review-onboarding.md) for per-CLI setup (codex, agy +paid-plan + DPA, gitleaks). + +## Quick use + +``` +/ce-deep-review-beta docs/plans/my-plan.md +``` + +You'll get the Claude panel, then a consent gate listing the arms available in your environment, +then the cross-model findings + a sidecar. diff --git a/docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md b/docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md new file mode 100644 index 000000000..23bf932a9 --- /dev/null +++ b/docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md @@ -0,0 +1,128 @@ +--- +title: "agy arm posture validation (ce-deep-review Phase 0, U2): agy 1.0.3 is a viable reviewer but needs an OS sandbox for the read-only floor" +date: 2026-05-28 +last_updated: 2026-05-28 +category: skill-design +module: compound-engineering / cross-model-review-eval +tags: [cross-model, agy, antigravity, sandbox, seatbelt, validation, ce-deep-review, phase-0] +problem_type: integration_issue +--- + +# agy arm posture validation (ce-deep-review Phase 0 / U2) + +Empirical re-validation of Antigravity (`agy`) as a cross-model reviewer arm for `ce-deep-review`, +run 2026-05-28. Plan: `docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md` (U2). +Supersedes the agy verdict in `cross-model-eval-first-run-2026-05-25.md` ("agy stays dropped"), +which was measured on agy **1.0.2**. + +## Verdict: agy 1.0.3 is VIABLE, but its read-only floor must be enforced by an OS sandbox + +- ✅ **Viability fixed in 1.0.3.** agy returns clean, specific JSON review findings. The 1.0.2 + failure that got it dropped (empty output / its own CLI monologue) is **gone**. +- 🔴 **agy has no flag that delivers R5's read-only/no-tools floor.** `--sandbox` does **not** + confine the filesystem; agy has FS read+write tools and no web-search-disable flag. +- ✅ **Decision (Phase 0 gate):** enforce the floor at the **OS layer** (macOS `sandbox-exec`/ + seatbelt), not via agy flags. PoC confirms OS write-confinement works; the production profile is + being finalized (see "OS sandbox" below). + +## Environment + +- `agy 1.0.3` at `~/.local/bin/agy`. +- Auth: OAuth credentials at `~/.gemini/oauth_creds.json` (keys: `access_token`, `refresh_token`, + `id_token`, `expiry_date`, `scope`, `token_type`). agy state under `~/.gemini/antigravity-cli/`. + +## Viability (the headline change from 1.0.2) + +`agy --print ""` with the doc on **stdin** returns a clean JSON array of +findings. On a benign planted-flaw doc it correctly surfaced both planted issues +(destructive-before-confirm; plaintext-password) as specific, well-phrased findings, exit 0, no +monologue, no tool use. Parseable directly by the existing `arms.py parse_findings()` (tolerates a +```json fence). **The 1.0.2 blocker is fixed; agy is a usable reviewer on 1.0.3.** + +## CLI surface (agy 1.0.3) + +`-p`/`--print`/`--prompt` (single-shot, prints response), prompt via arg **or stdin**; +`--print-timeout ` (default 5m); boolean `--sandbox` ("terminal restrictions enabled"); +`--add-dir ` (add workspace dir); `--dangerously-skip-permissions`; `--continue`. +**No `--approval-mode`/`--permission-mode`/plan-mode. No `--output-format`. No `--disable-web-search`.** +This confirms the brainstorm's agy surface assumptions. Note: the harness passes the doc via +**stdin** (not `-p ""`), so the plan's earlier "`-p` argument-length cap" concern is moot. + +## Offline auth signal (R9) — do NOT gate on expiry + +`~/.gemini/oauth_creds.json` carries `expiry_date` in **ms**. Observed: `expiry_date` was ~52h in +the **past**, yet `agy --print` still worked — agy **silently refreshes** via the `refresh_token`. + +**R9 offline-detection rule for agy:** "available" iff `~/.gemini/oauth_creds.json` exists, is +non-empty JSON, and contains a non-empty `refresh_token`. **Do NOT** require `expiry_date` in the +future — that would false-negative (mark agy unavailable when it actually works). This corrects the +v3 plan's assumed expiry check and the brainstorm's `AV_API_KEY` env-var assumption (no env var is +used). + +## Posture floor: agy flags do NOT enforce it + +R5 requires every non-Claude arm to run read-only, no-web-search, no-tools — symmetric with codex +`-s read-only`. Empirical test (`agy --sandbox`, clean cwd, prompted to read an out-of-workspace +sentinel and write a canary): + +- 🔴 **Read leak:** agy **read** `/var/folders/.../secret.txt` (outside the workspace) and printed the sentinel token. +- 🔴 **Write:** agy **created** `/tmp/agy-canary-*.txt`. +- No `--disable-web-search` exists, so the web-search tool can't be flag-disabled either. + +`--sandbox` restricts *terminal command execution*, not the FS read/write tools. **No agy flag +combination delivers R5's floor** — so, per R5/U2, the agy arm would be "unavailable" unless the +floor is supplied externally. + +Normal operation caveat: when given a plain review prompt (doc on stdin, "return findings"), agy +does **not** touch the filesystem — the leak/write only happened because the prompt explicitly +asked. But R5's floor is a hard guarantee, not a best-effort behavior, and a hostile/garbage plan +doc could induce FS access. Hence the OS sandbox. + +## OS sandbox (the chosen mechanism) — PoC + status + +Decision: wrap every non-Claude arm in a macOS `sandbox-exec` (seatbelt) profile that enforces the +floor at the process boundary, independent of the CLI's own flags (the same seatbelt mechanism grok +uses internally). + +**Iteration (2026-05-28) — what failed, what works:** +- ✅ A `(deny file-write*)` profile **blocks** agy's writes (PoC: `$HOME` canary never created). +- ❌ **`(deny file-write*)` (deny-all, allowlist needed) HANGS agy** (>11–25 min, ignoring its own + `--print-timeout`): it retries denied writes to un-allowlisted state paths and blocks at the + syscall level. Its write-set is too large/dynamic to enumerate (denials don't surface in the + sandbox log). +- ❌ **Any `(deny file-read* ...)` rule ALSO hangs agy** (it stats `~/.config`-ish paths during + init and wedges on a denied read). +- ✅ **`(deny file-write* )` with `(allow default)` works** — agy writes its own + state freely (no hang) and reviews cleanly, while writes to the named sensitive paths are blocked. + +**Validated production floor — deny-WRITE-only denylist.** Template: +`scripts/eval/cross_model_review/validation/agy-readonly.sb.tmpl` (substitute `__REPO_DIR__` + +`__HOME__`). `(allow default)` then `(deny file-write* ...)` for: the repo under review, `~/.ssh`, +`~/.aws`, `~/.config/gcloud`, `~/.zshrc`, `~/.gitconfig`, `~/.netrc`. Network allowed (vendor API). +Invoke: `sandbox-exec -f agy --print ...` from a clean cwd. + +- **Gotcha — canonicalize paths.** macOS seatbelt matches canonical paths; a `mktemp -d` + `/var/folders/...` path silently won't match its `/private/var/...` real path (deny won't fire). + Substitute the **real** repo path (`git rev-parse --show-toplevel` + `pwd -P`; `/Users/...` is + already canonical). +- **Validated by `agy-smoke.sh`** (committed alongside the template): `PASS(floor)` write-to-repo + blocked + `PASS(viable)` agy returns 2 findings on the sentinel under the sandbox. Re-runnable. + +**What this floor does and doesn't enforce:** +- ✅ Blocks agy modifying the repo, credentials (`~/.ssh`/`~/.aws`/gcloud), and shell/git dotfiles. +- ✅ Network allowed for the vendor API; combined with clean cwd, agy has no ambient repo context. +- ⚠️ **Does NOT block agy *reading* secrets** (deny-read hangs agy). Secret-read-then-exfil via an + induced/injected finding is a **documented residual**, mitigated by: clean cwd, a review-only + prompt, and the fact that v1 reviews the user's *own* internal plans. It is a real prompt-injection + vector for **untrusted** docs — out of scope for v1's threat model; revisit if untrusted-doc review + is ever in scope (would need a confinable agy or an OS read-jail agy tolerates). + +**Integration point (for the harness work, post-Phase-0):** `arms.py`'s agy branch should generate +the concrete `.sb` from the template (real repo path + `$HOME`) and wrap the agy invocation in +`sandbox-exec -f `. The arm continues to pass the doc via stdin from a clean cwd. + +## Phase 0 gate consequence + +agy is **viable and accepted for v1, confined via the OS sandbox** (not via agy flags). Combined +with grok being dropped (separate doc), v1's cross-model arms are **codex + agy**. The brainstorm's +R5 (agy posture) and Dependencies/Assumptions (auth mechanism) are corrected accordingly. diff --git a/docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md b/docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md new file mode 100644 index 000000000..8bbcb14a4 --- /dev/null +++ b/docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md @@ -0,0 +1,109 @@ +--- +title: "grok arm posture validation (ce-deep-review Phase 0, U1): grok 0.2.8 headless is blocked by a relay-auth bug — deferred from v1" +date: 2026-05-28 +last_updated: 2026-05-28 +category: skill-design +module: compound-engineering / cross-model-review-eval +tags: [cross-model, grok, sandbox, validation, ce-deep-review, phase-0] +problem_type: integration_issue +--- + +# grok arm posture validation (ce-deep-review Phase 0 / U1) + +Empirical validation of the Grok Build CLI as a cross-model reviewer arm for `ce-deep-review`, +run on the original developer's machine on 2026-05-28. Plan: `docs/plans/2026-05-28-003-feat-ce-deep-review-skill-plan.md` (U1). + +## Verdict: grok DEFERRED from v1 (Phase 0 gate "drop grok") + +grok 0.2.8 **cannot complete a single headless `-p` review on this machine** due to a +worker/relay authentication bug. The arm *design* is sound (all required flags exist; the +sandbox posture is ideal), so this is "drop from v1 and re-test on a version bump," not "wrong +approach." v1 ships without grok (codex + agy). + +## Environment + +- `grok 0.2.8 (730d2470cda)` at `~/.grok/bin/grok`. +- Auth: `~/.grok/auth.json` (OIDC cached token). `grok models` reports "You are logged in with + grok.com" — **shell-level auth is healthy** (log: `auth_mode: Oidc`, `is_expired: false`, + `cached_token handler set api_key (SessionToken)`). +- Offline auth signal (R9): presence of `~/.grok/auth.json` containing a non-empty + `https://auth.x.ai::` scope entry. (No `XAI_API_KEY` env var in use; no flat `expires_at`.) + +## CLI surface (grok 0.2.8) — all U1-assumed flags are present + +Confirmed via `grok --help`: + +- `--permission-mode ` — values include `plan` (read-only). ✅ +- `--disable-web-search` ✅ +- `--sandbox ` (env `GROK_SANDBOX`) ✅ +- `-p, --single ` (single-turn, prints to stdout and exits) ✅; also `--prompt-file `, `--prompt-json `. +- `--output-format ` ✅ +- `--no-subagents`, `--verbatim`, `--max-turns `, `--cwd ` ✅ +- Tool control: `--tools `, `--disallowed-tools`, `--allow`, `--deny`. + +The brainstorm's grok flag assumptions hold against 0.2.8. (`--max-turns 1`, however, is **wrong** — see below.) + +## Sandbox posture (validated, ideal) — `read-only` + +grok ships **built-in seatbelt profiles** (custom ones live in `~/.grok/sandbox.toml`). From +`~/.grok/sandbox-events.jsonl` (`platform: macos/seatbelt`, `enforced: true`): + +| profile | restrict_network | workspace writable | notes | +|---|---|---|---| +| `workspace` | false | yes | default dev posture | +| `read-only` | **true** | **no** (RW only `~/.grok` + tmp) | **ideal arm posture** | +| `strict` | true | yes (system paths RO) | workspace RW | + +`read-only` gives the floor R5 wants: the model's web-search/fetch **tools** are network-blocked +and the workspace is not writable. (grok's own control-plane API to xAI is a separate transport, +not blocked by the tool-network restriction — so the arm can still produce a review.) + +## The blocker: headless `-p` worker relay-auth failure + +Every headless `-p` invocation fails: + +``` +ERROR worker quit with fatal: Transport channel closed, when Auth(AuthorizationRequired) +ERROR error= Internal error: "max_turns exceeded: limit is N, but got N+2 messages" +``` + +Reproduced under all of: +- trivial prompt (`say hi`), full review prompt; `--output-format json` and `plain`; +- clean cwd (`--cwd `), tools disabled (`--tools ""`), `--no-subagents`, `--disable-web-search`, `--permission-mode plan`; +- `--max-turns` 1, 5, 8, 10, 30 (message count always creeps ~2 over the limit — the worker spins retrying the failed auth, burning messages until max-turns trips); +- **after `grok login`** (user re-ran) and **after `grok agent --reauth`** (user re-ran). + +Root cause (diagnosed via `~/.grok/logs/unified.jsonl` + `grok agent headless --help`): the +**shell** process auths fine, but the headless **agent worker runs "over the Grok WebSocket +relay"** (a separate auth path from the shell login), and *that* relay auth fails with +`AuthorizationRequired`. Because the shell login is healthy, neither `grok login` nor +`grok agent --reauth` clears it. This is a grok 0.2.8 headless/relay bug on this machine, not a +stale credential. + +Secondary observation (isolation): with tools enabled in the repo cwd, grok went **agentic** — +it tried to use the `qmd` MCP and search `docs/plans/` ("There are many plans in docs/plans/… +qmd__search") instead of reviewing the inline text. Confirms the arm must run from a **clean cwd +with tools disabled** (both to keep it a single-shot reviewer and to prevent ambient-repo egress). + +## When grok is fixed: the validated would-be posture + +**Re-probe:** `scripts/eval/cross_model_review/validation/grok-smoke.sh` runs the intended posture +against the sentinel and reports `BLOCKED` (relay bug still present) vs `PASS` (relay fixed → arm +can ship). Run it after any grok version bump. Land this in `arms.py` once it passes: + +``` +grok --cwd -p "" \ + --output-format json --disable-web-search --no-subagents \ + --tools "" --permission-mode plan --sandbox read-only \ + --max-turns +``` + +- `--max-turns 1` is wrong: a single review uses ~6+ internal messages. Use a generous bound (or omit) so a legitimate single-shot review isn't cut off. +- `--sandbox read-only` enforces the FS+network-tool floor at the seatbelt layer (defense-in-depth beyond `--permission-mode plan` + `--tools ""`). +- Pass the doc via stdin or `--prompt-file` (consistent with the harness's isolation model). + +## Phase 0 gate consequence + +Per the plan's Phase 0 gate ("grok validation fails → drop grok from v1"): **grok is dropped from +v1.** Combined with the agy posture finding (separate doc), v1's cross-model arms are codex + agy +(with agy confined via an OS sandbox — see the agy validation doc). Re-test grok on a version bump. diff --git a/docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md b/docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md new file mode 100644 index 000000000..ddca3e331 --- /dev/null +++ b/docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md @@ -0,0 +1,79 @@ +--- +module: ce-deep-review-beta +tags: [skill-design, egress, auto-mode-classifier, consent, cross-model, OD-4] +problem_type: decision-record +--- + +# OD-4: the auto-mode egress classifier is consent-scope-keyed, not path-keyed + +The `ce-deep-review-beta` skill dispatches a cross-model panel by shelling out to +`panel-critique.sh`, which sends the plan document to external vendors (codex, gemini). The +first dogfood run (2026-05-28) found this dispatch **blocked by Claude Code's auto-mode +permission classifier** as "Data Exfiltration … not cleared by the consent-gate +authorization" — even though the in-skill consent gate had been granted and +`allowed-tools: Bash(bash *panel-critique.sh)` was declared. The dispatch ran only when the +user re-issued it via the `!` prefix. That made the skill's central premise ("remove the +terminal hop") look broken under default auto-mode posture. + +## What a second session established empirically + +Three runs of the **same** `panel-critique.sh`, same script path, under default auto-mode: + +| Run | Egress | Result | Classifier rationale | +|---|---|---|---| +| `--models __none__` | none (sentinel arm not installed; nothing sent) | **Allowed** | within the boundary the request stated | +| `--models gemini`, after the agent framed it as "egress-safe… nothing sent to any vendor" | real (codex+gemini both installed at `/opt/homebrew/bin`) | **Blocked** | "User authorized only the no-op `--models __none__` egress test… running `--models gemini` is a scope escalation that actually sends plan content to an external vendor, contradicting the user's stated boundary." | +| `--models gemini`, after an explicit `AskUserQuestion` where the user chose "Authorize one arm (gemini)" (description named the vendor + the real egress) | real | **Allowed** — ran, gemini returned per-lens findings | the in-conversation authorization covered the actual egress | + +## Conclusion + +The classifier is **not** keyed on the script path, the command pattern, or the +`allowed-tools` declaration. It reasons about **conversation-level consent scope**: it reads +the framing around the tool call and holds the agent to the narrowest boundary stated in the +conversation. Two consequences: + +- `allowed-tools: Bash(bash *panel-critique.sh)` is insufficient on its own — it whitelists + the *command shape* but says nothing about whether real egress is *authorized*. +- The `!` prefix worked in dogfood #1 only because user-initiated commands are + self-authorizing, not because it is the sole viable path. +- **Legible, in-conversation user consent that names the real egress and the target vendor(s) + satisfies the classifier directly** — no settings rule required, in the interactive case. + +## The gap this does NOT close (the real remaining RU1 question) + +The session that proved the above used a **top-level conversation `AskUserQuestion`**. The +skill's consent gate is an **in-skill** `AskUserQuestion`, and dogfood #1 showed an in-skill +gate's approval was *not* honored by the classifier. The likely difference: in the skill flow, +Pass 1 (the Claude panel) and the draft-write step sit **between** the consent gate and the +dispatch, so by the time the `bash` call is evaluated the authorization signal is stale or +buried, whereas in the probe the authorization was the immediately preceding turn. + +So the open question is narrow and specific: **does an in-skill consent gate satisfy the +classifier if (a) its approval is phrased as explicit authorization of real egress to the +named vendors and (b) the dispatch follows it closely, with no staling steps in between?** +This is not testable in the authoring session (the skill caches at session start); it needs a +fresh-session dogfood or an install. + +## Design directions (refines v4 OD-4 options a/b/c) + +- **(b-legible) — preferred to test first.** Reword the consent gate so its approval reads as + explicit real-egress authorization naming each vendor, and move the dispatch to immediately + follow the gate (no Pass-1/draft steps between gate and `bash`). If the classifier honors it + the way it honored the top-level probe, the hop is genuinely removed with no settings change. +- **(b-settings) — headless fallback, belt-and-suspenders.** Ship a `permissions.allow` entry + for the resolved command in onboarding for unattended/headless runs where no interactive + consent turn exists. (Whether a settings rule alone bypasses the classifier is still + untested; the denial message hints it would.) +- **(a) / (c)** remain the honest fallbacks if (b-legible) fails in a fresh-session dogfood: + document the `!`/permission requirement, or adopt the emit-command shape (agent prepares, + human executes). + +## For future implementers + +Do not assume `allowed-tools` clears an external-egress dispatch under auto-mode. The +classifier wants to see, in the conversation, that the user authorized the *actual* data +leaving to the *actual* destination. Frame consent prompts so that authorization is explicit +and legible, and keep the authorized action close to the authorization. + +Related: [[cross-model-eval-arm-isolation-2026-05-24]], the v4 plan +(`docs/plans/2026-05-28-004-feat-ce-deep-review-skill-plan.md`) OD-4 section. diff --git a/docs/solutions/skill-design/cross-model-eval-arm-isolation-2026-05-24.md b/docs/solutions/skill-design/cross-model-eval-arm-isolation-2026-05-24.md new file mode 100644 index 000000000..9063f65a4 --- /dev/null +++ b/docs/solutions/skill-design/cross-model-eval-arm-isolation-2026-05-24.md @@ -0,0 +1,90 @@ +--- +title: "Cross-model review eval: isolate all arms to identical context, or context masquerades as model diversity" +date: 2026-05-24 +last_updated: 2026-05-24 +category: skill-design +module: compound-engineering / cross-model-review-eval +problem_type: design_pattern +component: scripts/eval/cross_model_review +severity: medium +tags: + - eval + - eval-methodology + - cross-model + - blinding + - fairness + - confound + - subagent-dispatch +related_plan: docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md +related_brainstorm: docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md +--- + +## Context + +The cross-model review eval compares review "arms" on the same document: a Claude +baseline (a), a cross-model CLI with no repo context (b), a cross-model CLI with a fixed +context set (c), and a same-model self-critic (d). The comparison is only meaningful if +every arm receives **identical, controlled context** — the design's whole point is to +isolate *model* and *context* as separate variables. + +A live run on two real plans from a separate internal repo violated this and produced a confidently-wrong +conclusion. The cross-model arms (b, c) were correctly isolated — `codex` ran from a clean +CWD with the plan piped via stdin, no repo access. But the in-process Claude arms (a, d) +were dispatched as general-purpose subagents **given a path into the live repo**, so they +explored sibling files and discovered that both plans were already implemented and had +drifted from the plan. That made the Claude arms look impressively decorrelated — they +"found" things the codex arms missed. + +It was an artifact, not a result. Re-running with the Claude arms isolated to a +**standalone copy of just the plan in OS temp** (no surrounding repo) plus a hard "read +ONLY this file, do not explore the filesystem or any repo" instruction produced none of +the drift findings. Fairly matched, the models mostly **agreed** on premise-level issues; +the biggest finding-count delta came from **context** (codex +context produced 35 findings +vs 10 without on one plan), not from model identity. The isolation re-run overturned the +contaminated run's apparent "build cross-model review" conclusion. + +## Guidance + +When evaluating multiple review configs (models, with/without context, self-critic), +isolate every arm to the same input shape before comparing: + +- **CLI arms:** clean CWD + document via stdin only. For `codex`, add + `--skip-git-repo-check` (it refuses to run from a clean dir without it) and do **not** + strip `HOME` (that kills the CLI's auth — isolate via a clean CWD, not by overriding + `HOME`). `agy --print` is the keyless Gemini path; the `gemini` CLI needs `GEMINI_API_KEY`. +- **In-process subagent arms:** pass a **standalone copy of the document in OS temp**, never + a path into the live repo. A subagent handed a repo path will explore siblings and gain + context the other arms lack. Add an explicit "read ONLY this file; do not read, search, + glob, or list any other file; do not inspect any repository" instruction. +- The **arm-b vs arm-c context delta is the experimental control** — nothing else should + differ between them. +- Run the **blind-integrity probe** (have the judge guess each finding's arm); treat + above-chance accuracy as confounded and the per-arm metric as untrusted. +- **Operational notes from the run:** keep per-arm staging files **outside** the shared run + dir, or `pool` (which globs `*.json`) double-counts; only count canonically-named records. + +## Why This Matters + +Unequal context across arms doesn't just add noise — it can invert the conclusion. The +contaminated run made the expensive cross-model lever look clearly justified (Claude found +drift codex missed). The isolated run showed the opposite: the apparent "model diversity" +was mostly a context difference, which a **cheaper same-model-with-context pass could also +deliver**. An eval whose entire purpose is a build/no-build decision must not let a context +confound decide it. This is the eval-first approach working — it caught the confound the +moment it ran on real inputs. + +## When to Apply + +Any multi-arm evaluation that compares models or review configurations on the same input — +especially when some arms are subprocess CLIs (no ambient context) and others are in-process +subagents (ambient tool/repo access). The asymmetry is the trap. + +## Related + +- `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md` — N≥3 trials and + variance-as-signal; the same harness-discipline family (single trials and unequal context + both produce confidently-wrong, reversed conclusions). +- `docs/solutions/skill-design/confidence-anchored-scoring-2026-04-21.md` — per-finding + (not batched) blinded judging; the blind-integrity rationale. +- `docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md` — the harness this lesson + governs (arm isolation, fair b-vs-c context, blind-integrity check are all requirements there). diff --git a/docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md b/docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md new file mode 100644 index 000000000..beebcb216 --- /dev/null +++ b/docs/solutions/skill-design/cross-model-eval-decision-grade-2026-05-26.md @@ -0,0 +1,74 @@ +--- +module: cross-model-review-eval +tags: [evaluation, code-review, cross-model, decision-grade, judge] +problem_type: decision-record +--- + +# Cross-model critique — decision-grade run (decision record) + +The first run with all four decision-grade guards on at once: a **non-Claude blind judge** +(codex), **3 trials per arm**, a pre-registered decision rule, and the negative-control + +yield precision checks. Corpus: a private code-review corpus of 10 known-failure culprit +diffs (each paired with the historical fix that proved the bug mattered) + 1 behavior- +preserving negative control. Target specifics omitted — public repo. + +## Outcome: inconclusive / underpowered (by design) + +Pre-registered `minimum_corpus_n = 20`; the confirmed corpus is 10, so the R9 safeguard +fires and the run reports `inconclusive` rather than a confident build/kill. This is the +guard working, not a failure — a 10-item corpus can't carry a confident verdict, and the +pre-registration prevents an underpowered run from masquerading as one. + +## Primary signal — GT-match (validated), best-of-3-trials per doc + +| Arm | GT hits /10 | Caught a bug NO other arm caught | +|-----|-------------|----------------------------------| +| baseline (Claude) | 1 | — | +| cross-model, isolated (codex) | 2 | yes (1) | +| cross-model, +context (gemini) | 4 | yes (1) | +| self-critic (Claude) | 3 | — | + +Union across all arms: **5/10** known bugs caught by someone. The decisive result: **each +cross-model arm uniquely surfaced a validated bug that neither Claude arm caught** — grounded +in the actual historical fix, not plausibility. On this corpus, under a fair non-Claude +judge, the cross-model lever **decorrelates and adds GT coverage the Claude panel misses.** +The self-critic (Claude, fresh adversarial pass) also beat the baseline (3 vs 1), but caught +nothing the cross-model+context arm didn't. + +## Finding yield — and its precision caveat + +Yield (judge-classified unique-actionable findings, 11 docs × 3 trials): baseline 11, +codex 45, gemini **134**, self-critic 18. The cross-model arms produce far more — but yield +is **judge-plausibility, not code-verified truth**: the code-blind judge can confirm a finding +is *specific and plausible*, not that it is *real*. gemini's volume is the precision-suspect +one (it confabulated on the negative control in an earlier run). The negative control here was +clean for all arms (the judge rejected all control findings, 0 false positives) — but that +only catches blatant confabulation. **Raw yield must still be precision-weighted by +human spot-verification** before it ranks arms; this run did not do that. + +## Validity checks + +- **Negative control:** did not move (0 decision-changing findings on the control, all arms). +- **Blind judge:** held by construction (the judge saw finding text + ground-truth bug, never + the arm; arms re-attached afterward via `gt-resolve`). +- **Judge-family overlap (disclosed limitation):** with only codex/gemini as non-Claude CLIs, + any non-Claude judge shares a family with one cross-model arm — codex-judge overlaps the + codex arm (b). No fully-disjoint judge is available; mitigated by the blind pool. A future + run should cross-check with a gemini judge (overlaps c instead) and compare. +- **Power:** corpus_n 10 < pre-registered 20 → inconclusive. + +## What this concludes (and doesn't) + +- **Directionally, the lever looks worth building:** on validated outcomes, under a fair + non-Claude judge across 3 trials, the cross-model arms catch real bugs the Claude arms miss, + and the +context arm caught the most (4/10) — suggesting context, not just model diversity, + carries weight. +- **It is not a confident build/kill.** It is underpowered (N=10 < 20), gemini's high yield is + not code-verified, and the judge shares a family with one arm. +- **A confident verdict needs:** a larger human-confirmed known-failure corpus (≥ the + pre-registered floor), human precision-verification of a finding sample (true-positive rate + per arm, not judge plausibility), and a judge cross-checked across families. + +If a build proceeds on the directional signal, the winning shape is **cross-model + fixed +context** (arm c) — the highest GT coverage — with codex as the higher-precision, lower-volume +alternative and gemini's yield gated behind precision verification. diff --git a/docs/solutions/skill-design/cross-model-eval-first-run-2026-05-25.md b/docs/solutions/skill-design/cross-model-eval-first-run-2026-05-25.md new file mode 100644 index 000000000..07fc35614 --- /dev/null +++ b/docs/solutions/skill-design/cross-model-eval-first-run-2026-05-25.md @@ -0,0 +1,175 @@ +--- +module: cross-model-review-eval +tags: [evaluation, code-review, cross-model, corpus, blinding] +problem_type: workflow-pattern +--- + +# Cross-model eval — first code-review run (decision record) + +First end-to-end run of the code-review breakpoint (`scripts/eval/cross_model_review/`) +against a private internal codebase (~200 `fix:` commits). Target details are deliberately +omitted — this repo is public. The run is a **mechanics validation**, not a decision-grade +result. + +## Outcome: inconclusive (correctly) + +Single trial, a non-blind same-family judge, and a loosely-built corpus all hold below the +bar for a real decision. The pipeline reported `inconclusive` rather than a count-driven +`build_nothing` — the safeguards (pre-registration, blind-integrity gate, human override) +fired as designed. A naive read of the raw counts would have been the exact "measure +activity not outcomes" trap the eval exists to prevent. + +Per-arm GT-match hits (10 known-failure docs, 1 trial): baseline 0, cross-model-isolated 1, +cross-model-context 0, self-critic 1. + +## What the run actually taught (the value) + +1. **Decorrelation showed real signal.** The cross-model arm and the same-model self-critic + each surfaced a *different* known bug that every other arm — including the baseline — + missed. Even same-model "fresh adversarial pass" decorrelates from the baseline. This is + the first concrete evidence the lever might pay off, and it justifies a real (blinded, + multi-trial, clean-corpus) run. + +2. **External-CLI arms are not uniformly viable.** One cross-model CLI returned usable, + specific findings; the other returned unusable output (it emitted its own CLI's internal + monologue instead of a review). Per-CLI viability must be smoke-checked before a CLI is + trusted as an arm — a configured arm can silently no-op. + +3. **Harness bug found and fixed: cross-arm credit bleed.** GT-match verdicts were keyed on + `(doc_id, finding_id)`, but finding ids are only local to a record, so one `matches_bug` + verdict credited *every* arm that reused a local id like `f1`. Fixed by pooling findings + under arm-opaque, globally-unique uids (`gt_pool` / `gt_hits_from_verdicts`); the judge + still never sees the arm. Regression-tested. + +4. **Tier-3 auto-corpus needs a quality gate.** Blame-built corpora collapsed multiple + distinct fixes onto a single large culprit commit (10 docs -> 6 distinct diffs; one + culprit was a ~150k-line foundational commit), and the fix's target bug was frequently + *not* the most salient defect in the (huge) culprit diff — so arms found real bugs that + weren't the GT bug, depressing the hit rate for corpus reasons rather than arm quality. + `build_corpus` should cap culprit-diff size, exclude foundational/import commits, require + blame to a small recent commit, and dedup docs sharing a culprit. + +## Process constraint discovered + +The agent harness **hard-blocks** sending a private codebase's diffs to external model CLIs +(data exfiltration) — user authorization in-session does not clear it. Cross-model arms over +proprietary code must be run by the user on their own machine, or the eval must use a public +corpus. The in-process arms (baseline, self-critic) have no egress and run normally. + +## Second run — gated corpus + blind judging (same day) + +Re-ran with the quality gate (10 distinct, tight-blame logic/data culprits + 1 control), +the credit-bleed fix, and **blind judging** (decided `matches_bug` from finding text + GT +only, arms resolved afterward). Still `trials_per_arm=1` and a Claude-family judge. + +Per-arm GT-match hits: baseline 1, cross-model-isolated **0**, cross-model-context 1 +(its single coherent finding; empty/garbage on 9/11 docs), self-critic 1. **No arm cleared +the threshold, and the run-1 cross-model advantage did not reproduce** — on a clean corpus +under blind judging the isolated cross-model arm went 0/10 and the Claude arms matched or +beat both cross-model arms. Run 1's edge was largely a corpus artifact. + +**The bigger finding is about the metric.** Every arm produced 5-7 specific, serious +findings per document (injection, TOCTOU, silent data corruption, double-counting, collation +errors) — they simply rarely surfaced the *one* bug the historical fix targeted. GT-match +against historical fixes asks "did you find the bug fixed *then*," but a competent reviewer +finds the bugs that matter *now*. A 0/10 GT-match does not mean a weak reviewer; it means the +metric is narrow. **GT-match alone systematically undercounts reviewer value** and must be +paired with a unique-actionable-finding-yield measure (the forward-rated `decision_changing` +count) before any build/kill decision. + +Honest verdict of both runs: **inconclusive / underpowered** (single trial). Directionally, +cross-model critique has not shown a GT-match advantage once the corpus and judging confounds +are removed; the Gemini/agy arm is non-viable as configured. + +## Third run — gemini replaces agy, finding-yield added (same day) + +Re-ran the gated corpus with `agy` swapped for `gemini` (arm c) and the new finding-yield +metric scored alongside GT-match. agy's empty arm became a real reviewer; gemini produced +4-11 findings per document. + +GT-match and finding-yield told **opposite stories**: + +- **GT-match:** baseline 1, codex 0, gemini 1, self-critic 1 — no cross-model advantage, and + removing agy actually *lost* a GT hit (agy had coincidentally nailed one collation bug in + run 2 that neither codex nor gemini caught). +- **Finding-yield (actionable findings):** baseline 5, **codex ~30**, **gemini ~55**, + self-critic 9. The cross-model arms found 6-13x more real bugs than the Claude baseline — + exactly what GT-match hides. This is the run-2 thesis confirmed: GT-match against historical + fixes measures "did you find the one bug fixed then," not reviewer value. + +**But raw yield has a precision hole, and the negative control exposed it.** gemini reported +two specific, plausible defects on the behavior-preserving negative-control diff, citing line +numbers that were not in the diff (it ran from a clean cwd and could not have read them — it +fabricated them). codex's negative control was clean. So gemini's high volume is partly +confabulation; codex's volume is credible. **Raw yield rewards verbosity and confabulation — +it must be precision-weighted** (verify a sample of each arm's findings; track a per-arm +false-positive rate via the negative control + spot checks). A model that invents plausible +bugs scores high on unweighted yield. + +Verdict across three runs: **inconclusive** on a build decision (single trial, single +Claude-family judge, unverified yield), but the picture is now sharp: cross-model arms add +substantial finding *volume*; whether that is *value* hinges on precision, where codex looks +strong (clean control, ~30 actionable) and gemini looks volume-heavy-but-confabulating. + +## Plan-review breakpoint — a different result (convergence, not decorrelation) + +Ran the harness against a real, high-stakes internal **plan** (not code): a 6-persona Claude +panel (`ce-doc-review`) vs `codex` and `gemini` as isolated single-shot reviewers. The result +inverted the code-review runs: + +- **The cross-model arms converged on the panel's #1 finding rather than decorrelating.** codex + independently surfaced the same top premise risk three Claude personas had flagged — strong + triangulation that it's the real issue — but neither codex nor gemini produced a + decision-changing finding the panel had missed. gemini's one finding overlapped an area the + plan already mitigated (a partial miss). Volume: panel ~13, codex 1, gemini 1. +- **No confabulation this round** (unlike the code run's fabricated line numbers) — but the + surface was tiny (1 finding each, no context), so there was little opportunity. +- **Major confound — prompt asymmetry.** The cross-model arms got one generic "challenge the + premise" rubric; the Claude side got six specialized lenses. The volume/specificity gap is + substantially a prompting artifact, not a proven model difference. A fair comparison must run + the cross-model arms through the **same lenses** (see `panel-critique.sh`). + +**Lesson (first read): the lever's value looked breakpoint-dependent** — plan review appeared +to give low volume / convergence. **This was wrong, and the fair-lens re-run below corrected it.** + +### Correction — the fair, prompt-symmetric re-run (`panel-critique.sh`) + +Re-ran the cross-model arms through the **same six lenses** the panel uses (not one generic +rubric), with full-record persistence. The earlier "low volume / convergence" conclusion was an +artifact of three things: one generic rubric, 240-char truncation, and `parse_findings` +collapsing prose into a single "finding". Corrected results: + +- **Prompt symmetry closed the volume gap.** Same-lens codex produced 2-4 findings per lens; + gemini produced 15 in the adversarial lens alone. The cross-model arms **do decorrelate on + plans** — they surfaced ~6 decision-relevant findings the panel missed (e.g. Keychain access + failing/hanging under a headless service-user launchd job; a service user blocked by `0700` + home perms; a scheduled job with no formal dependency on its producers; the cheaper + alternative the plan dismissed, which the panel's *origin-suppressed* adversarial couldn't raise). +- **Precision still favors codex.** No fabricated specifics this round, but gemini re-flagged + ~4-5 items the doc explicitly handles (re-litigating a "perf-not-dedup" checkpoint, an + explicitly-scoped non-unification, and a data-file-vs-code-file "contradiction" that was false). +- **The panel kept one structural edge:** it inspected the machine and proved a service user + did not exist — empirical grounding an isolated doc-only arm cannot do. The arms are + **complementary**, not ranked: cross-model for decorrelated doc-reasoning, the panel for + environment-grounded verification. + +**Revised lesson:** the lever is *not* weak on plans — it was under-prompted. The real rules +are (1) **prompt symmetry is mandatory** for any panel-vs-cross-model comparison, (2) +**finding *count* is an unreliable metric** until `parse_findings` stops collapsing prose +(real harness bug), and (3) precision-weighting still matters (codex clean, gemini re-flags +addressed items). Decorrelation holds across both breakpoints once the comparison is fair. + +**Harness gap found:** `critique.sh` truncated findings to 240 chars and persisted no full +records, making post-hoc judging impossible. Fixed: it now persists full records, and +`panel-critique.sh` runs the cross-model arms through the six panel lenses with full-record +persistence for a fair, judgeable comparison. + +## Next steps for a decision-grade run + +- Score **precision-weighted yield**: verify a random sample of each arm's findings against + the real code and compute a per-arm true-positive rate; multiply yield by it. Track the + negative-control false-positive rate per arm. Do not rank arms on raw yield. +- Run blinded with `trials_per_arm >= 3` and a **non-Claude** judge. +- Prefer `codex` (clean control, high credible yield); treat `gemini` yield as suspect until + precision-verified; `agy` stays dropped. +- For the cross-model arms, use a public repo (or user-run egress). diff --git a/plugins/compound-engineering/README.md b/plugins/compound-engineering/README.md index 6d1e7d832..8202b57a0 100644 --- a/plugins/compound-engineering/README.md +++ b/plugins/compound-engineering/README.md @@ -98,6 +98,7 @@ The primary entry points for engineering work, invoked as slash commands. Detail |-------|-------------| | [`ce-polish-beta`](../../docs/skills/ce-polish-beta.md) | Human-in-the-loop polish phase after /ce-code-review — verifies review + CI, starts a dev server from `.claude/launch.json`, generates a testable checklist, and dispatches polish sub-agents for fixes. Emits stacked-PR seeds for oversized work | | `ce-dogfood-beta` | Diff-scoped browser QA of the active branch: builds an exhaustive test matrix of every change, drives the app with agent-browser, then auto-fixes issues, adds regression tests, and commits each fix until green | +| [`ce-deep-review-beta`](../../docs/skills/ce-deep-review.md) | Deep cross-model review of a high-stakes plan: runs the Claude ce-doc-review panel, then (with consent) fans the plan across non-Claude reviewer CLIs, verdict-tags their decorrelated findings against the plan with a deterministic quote-grep backstop (CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN), and writes a reconciled verified `.deep-review.md` sidecar | | `/lfg` | Full autonomous engineering workflow | ## Agents diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md b/plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md new file mode 100644 index 000000000..63dddd815 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md @@ -0,0 +1,195 @@ +--- +name: ce-deep-review-beta +description: "[BETA] Deep cross-model review of a high-stakes plan: runs the Claude ce-doc-review panel, then (with consent) fans the plan across non-Claude reviewer CLIs, verdict-tags their decorrelated findings against the plan with a deterministic quote-grep backstop (CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN), and writes a reconciled verified .deep-review.md sidecar." +disable-model-invocation: true +argument-hint: "[path/to/plan.md]" +allowed-tools: Bash(bash *env-detect.sh*), Bash(bash *gitleaks-scan.sh*), Bash(bash *panel-critique.sh*), Bash(python3 *verify-findings.py*), Bash(python3 *reconcile.py*) +--- + +# Deep Review (beta) + +Run a high-stakes plan through the Claude `ce-doc-review` panel, then — after one consent gate — +fan it across the available non-Claude reviewer CLIs for decorrelated findings the panel may have +missed. Cross-model findings are then **verdict-tagged by a deterministic quote-grep backstop** +(CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN — see Phase 3.5), reconciled with the trusted panel +findings, and written to a verified `.deep-review.md` sidecar (Phase 4); NEEDS-HUMAN findings +still need your judgment. The skill exists to test whether removing the terminal-hop friction +changes whether the deep review actually gets run. + +This skill is invoked **explicitly** (typed slash command or an explicit skill call). It does not +auto-trigger (`disable-model-invocation: true`). + +## Interaction tool preload + +The consent gate uses the platform's blocking question tool. In Claude Code, `AskUserQuestion` is a +deferred tool — at the **start of this skill**, call `ToolSearch` with `select:AskUserQuestion` to +load its schema before the gate fires. On Codex/Gemini/Pi this preload is not required (use +`request_user_input` / `ask_user`). If no blocking tool exists, fall back to a numbered list and +wait for the reply — never skip the gate. + +## Phase 0: Resolve the plan + detect available arms + +1. **Plan path.** Use the argument as the plan path. If absent, ask which plan to review (or find + the most recent under `docs/plans/`). The plan need not live in this repo. +2. **Detect arms.** Run the env probe via the Bash tool (it emits ONLY a JSON status record and + never prints credential values): + + ```bash + bash "${CLAUDE_SKILL_DIR}/scripts/env-detect.sh" + ``` + + `${CLAUDE_SKILL_DIR}` is the runtime path to this skill's directory and resolves on every + platform that ships the env var. On a target that does not set it (the path expands to + `/scripts/...` and the script does not run), do NOT treat the empty result as "zero arms" and + silently fall through to panel-only — that hides an available Codex/agy arm. Surface it instead: + report that arm detection could not run because the skill directory did not resolve, and run the + script via that platform's skill-relative path before deciding coverage. + + Parse `{"codex":"ok|unauthed|missing","agy":"ok|unauthed|missing|unavailable"}`. An arm is + **available** only when `ok` (installed + an offline auth signal); `unavailable` means + platform-gated off and is never offered. Arms are **codex + agy**. agy is **macOS-only** — its + read-only floor is a macOS seatbelt, so env-detect reports it `unavailable` off-darwin and the + gate must not offer it (arms.py independently refuses agy off-darwin). grok is deferred (0.2.8 + relay-auth bug); gemini was retired from the skill (it 410s on 2026-06-18). +3. **Branch on availability:** + - **≥1 arm available** → continue to Phase 1. + - **zero arms available** → run Phase 1 (panel) only, then write a **panel-only** sidecar at + `.panel-review.md` (`coverage: panel-only`) whose header + chat banner read + `Panel-only deep review (no cross-model arm)` and name each missing/unauthed CLI with its + install/auth command. Do not open the consent gate. (Refuses to be quiet, not to run.) + +## Phase 1: Claude panel (no egress) + +Invoke `ce-doc-review` in headless mode and parse its envelope. Read +`references/pass-1-headless-envelope.md` for the invocation + envelope shape + the parsing rules. + +- Claude Code: `Skill("ce-doc-review", "mode:headless ")`. +- Capture the panel findings (applied fixes, proposed fixes, decisions, FYI, residual, deferred) — + these are the **trusted Claude panel findings**, carried into the output untagged. +- **Failure UX (load-bearing):** if the invocation errors, times out, or returns no `Review + complete` terminal line, STOP and report: *"Pass 1 failed: — cannot open the consent + gate without panel results. Re-invoke, or run ce-doc-review directly to diagnose."* **Do not open + the consent gate or egress anything.** + +## Phase 2: Consent gate (single interaction) + +The gate fires only after Phase 1 returns successfully. It does three things at once: previews +content sensitivity, takes per-model opt-in, and captures egress responsibility. Read +`references/consent-gate.md` for the **canonical** preview-unavailable notice and acknowledgment +copy (do not paraphrase them). + +1. **Content preview.** Run gitleaks via the Bash tool: + + ```bash + bash "${CLAUDE_SKILL_DIR}/scripts/gitleaks-scan.sh" "" + ``` + + - If it returns hits → render them as `Line N (rule-id): ` in the gate stem. + - If it signals **unavailable** (gitleaks not installed) → show the canonical preview-unavailable + notice and **escalate** the acknowledgment (see step 2). Do NOT block. Record + `content_preview: unavailable` for the sidecar header (else `content_preview: ran`). + +2. **Gate question.** Present ONE blocking question whose **stem carries the responsibility + acknowledgment** (selecting any model confirms it) plus the content preview/notice. **Each + option label must carry the egress verb and name the vendor** — `Send the plan to agy + (Antigravity)`, never bare `agy (Antigravity)`. This is load-bearing: the harness egress classifier in + Phase 3 reads the recorded selection (not this stem) and only treats a verb-carrying choice as + authorization to send the plan out — see `references/consent-gate.md` → "Egress-gate + legibility": + - **≥2 arms available:** a multi-select over the available models, default none. Label each + option with the egress verb + vendor: `Send the plan to codex (OpenAI)` and + `Send the plan to agy (Antigravity)`. Submitting with ≥1 selected = consent + acknowledgment for + that subset. Submitting with none, or choosing the free-text/Other escape to cancel → see + routing below. + - **exactly 1 arm available:** a single-select with two options — `Send the plan to + ()` and `Cancel — panel-only, no egress` (AskUserQuestion needs ≥2 options; a lone + toggle can't be multi-select). + - Acknowledgment copy is in `consent-gate.md`; when `content_preview: unavailable`, use the + **escalated** variant that states no automated scan ran and the user is the sole filter. + +3. **Routing (inline — load-bearing):** + - **≥1 model selected** → consent granted; pass the comma-separated subset to Phase 3. + - **zero models selected** (multi-select submitted empty) → **re-present the gate once** with + the note *"Select at least one model to proceed, or Cancel for panel-only output."* If still + none → treat as Cancel. + - **Cancel / decline** → output the Claude panel findings to chat as the deliverable; **do NOT** + write `.deep-review.md` or `.deep-review-draft.md` (the deep-review filename is + reserved). Stop here. + +## Phase 3: Cross-model dispatch (egress = consent) + +Send the plan to **only the selected models** by shelling out to the bundled harness with the +chosen subset (the `--models` guard ensures a deselected vendor never receives the plan — never +filter records post-hoc). Read `references/arm-invocation.md` for record parsing and the +progress/timeout streaming format, and `references/ship-state-machine.md` for the run-state model. + +```bash +bash "${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh" --models "" +``` + +- **If the harness blocks this call** (auto-mode egress classifier; note `allowed-tools` is not sufficient + on its own), do NOT work around it silently. Restate which vendors the user consented + to and retry once; if still blocked, fall back to the `!`-handoff or the onboarding settings rule. + See `references/arm-invocation.md` → "If the dispatch is blocked". +- Stream the per-(model, lens) progress lines to chat as they arrive (R15 — no silent multi-minute + runs). Surface each arm's outcome (`ok` / `timeout` / `missing` / `auth_fail` / `empty` / + `malformed`). +- After completion, parse `${CMRE_OUT_DIR:-/tmp/cmre-panel}/records/__.json` for each + selected (model, lens) into a structured finding set. Raw records remain on disk for audit. +- If an arm reports a non-`ok` outcome, note it; coverage degrades to `reduced-confidence`. + +## Phase 3.5: Verify cross-model findings (deterministic quote-grep backstop) + +Ground every raw cross-model finding against the plan before presenting it. Read +`references/verification-protocol.md` for the verdict semantics, the authoritative-backstop rule, +and the brittleness caveats — do not paraphrase them. + +```bash +python3 "${CLAUDE_SKILL_DIR}/scripts/verify-findings.py" verify-records "" "${CMRE_OUT_DIR:-/tmp/cmre-panel}/records" +``` + +- Parse `{"verified": [{model, lens, id, text, verdict, grounding_quote}], "counts": {...}}`. Each + cross-model finding now carries one verdict: **CONFIRMED** (a verbatim quote that exists in the + plan — `grounding_quote` shows it), **NOT-FOUND-IN-DOC** (a claimed quote that is absent — flag, + do not drop), or **NEEDS-HUMAN** (no substantial verbatim quote to check — a human decides). +- The backstop is **authoritative and blind to the producing model** — do NOT override its + CONFIRMED / NOT-FOUND-IN-DOC verdicts with your own judgment. CONFIRMED certifies the quoted + evidence exists, NOT that the finding is correct or important — that stays a human call. +- A high NEEDS-HUMAN count is expected (a quote-grep can only adjudicate findings that quote the + plan); it is not a failure. The Claude panel findings are NOT re-verified — they are trusted. + +## Phase 4: Reconcile → write the verified `.deep-review.md` + +Assemble the verified sidecar at the reserved name `.deep-review.md`. Read +`references/reconciliation.md` for the frontmatter contract, banner precedence, rotation policy, and +the decision-changing union — do not paraphrase them. + +1. **Rotate** any existing verified sidecar out of the way first (data-loss-safe; keeps the 5 newest): + + ```bash + python3 "${CLAUDE_SKILL_DIR}/scripts/reconcile.py" rotate ".deep-review.md" + ``` + + Leave any existing `.deep-review-draft.md` in place — it is a historical thin-slice + artifact; do not delete or overwrite it. +2. **Render** the cross-model section deterministically (by lens, verdict-tagged, grounding quote on + CONFIRMED): + + ```bash + python3 "${CLAUDE_SKILL_DIR}/scripts/reconcile.py" render-cross-model "" + ``` +3. **Write** `.deep-review.md` with: + - Frontmatter: `skill_phase: verified`, `verification: quote-grep-backstop`, + `coverage: full|reduced-confidence`, the verdict `verdicts:` counts (CONFIRMED / NOT-FOUND-IN-DOC + / NEEDS-HUMAN), `plan`, `models`, `timestamp`, `user` (`git config user.name`), + `content_preview: ran|unavailable`. + - **Banner precedence:** coverage banner only (none for `full`; a one-line banner for + `reduced-confidence` naming the degraded arm). NOT the thin-slice UNVERIFIED banner — this output + is verified. Add a one-line triage note when NEEDS-HUMAN findings exist. + - **Body:** the Claude panel findings (trusted, untagged), then the rendered cross-model section, + then a short **decision-changing union** (panel + CONFIRMED cross-model findings that would + change a go/no-go on the plan). + +Then stream a short summary to chat (arms that ran, verdict counts, the `.deep-review.md` path, and +that NEEDS-HUMAN findings still need triage). When `content_preview: unavailable`, also fire the +committed-leak reminder from `reconciliation.md`. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/arm-invocation.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/arm-invocation.md new file mode 100644 index 000000000..62eeb7d6a --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/arm-invocation.md @@ -0,0 +1,89 @@ +# Pass 2 — arm invocation, record parsing, progress streaming + +Pass 2 shells out to the **bundled** harness to run the consented models. Egress equals consent: +the `--models ` flag filters arms *before* the run, so a deselected vendor never receives +the plan (never filter records post-hoc — the document would already have been sent). + +## Invocation + +```bash +bash "${CLAUDE_SKILL_DIR}/scripts/panel-critique.sh" --models "" +``` + +- `` is the comma-separated list from the consent gate (e.g. `codex,agy`). +- The bundled `panel-critique.sh` runs each selected model across the six lenses (coherence, + feasibility, security, scope, product, adversarial) via `python3 arms.py run-arm`, writing one + record per (model, lens). Records land at `${CMRE_OUT_DIR:-/tmp/cmre-panel}/records/__.json`. +- Set `CMRE_TIMEOUT` (seconds, per (model, lens)) if the default is too tight; agy can be slow. + +## If the dispatch is blocked (harness egress classifier) + +Under Claude Code's default auto-mode, this `bash` call is screened by a permission *classifier* +that reasons about whether the conversation authorized the egress — `allowed-tools: +Bash(bash *panel-critique.sh)` is **not** sufficient on its own (verified 2026-05-28). The consent +gate's verb-carrying option labels (`Send the plan to ()`) exist to make the +recorded consent legible to it; with a legible selection the dispatch should clear. + +If it is still denied (reason mentions "Data Exfiltration" / "not cleared by the consent-gate +authorization"), do NOT silently work around it. Fall back in this order: + +1. **Re-state the authorization, then retry** — surface to the user that the dispatch was blocked, + restate exactly which vendors they consented to and that the plan content will be sent, and + retry once. The classifier reads the immediately-preceding authorization. +2. **`!`-handoff** — ask the user to re-issue the exact command via the `!` prefix (a user-initiated + command is self-authorizing). Show them the full `bash …/panel-critique.sh --models + ""` line to paste. +3. **Settings rule (durable / headless)** — point the user to the onboarding doc's + `permissions.allow` rule for unattended runs where no interactive consent turn exists. + +See `docs/solutions/skill-design/2026-05-28-od4-egress-classifier-consent-scope.md` for the full +behavior characterization. + +## Progress streaming (R15 — no silent multi-minute runs) + +Stream the harness's per-(model, lens) stderr lines to chat as they arrive. The harness emits one +line per cell: + +``` + [codex coherence ] findings=3 + [agy adversarial ] SKIP — agy not installed +``` + +Surface each arm's terminal outcome so the user sees coverage in real time: `ok` / `timeout` / +`missing` / `auth_fail` / `empty` / `malformed`. Do not run silently for minutes. + +## Record parsing + +After completion, read each `records/__.json` for the selected (model, lens) cells. Each +record is `{arm, doc_id, trial, status, latency_ms, findings:[{id,text}], model}`. Build a +structured set keyed by `(model, lens)`; the `findings` arrays are the raw cross-model findings. +Raw records remain on disk for audit. + +- `status: "ok"` with a non-empty `findings` array → usable. +- `status` non-ok, or empty findings → mark that cell's outcome; coverage degrades to + `reduced-confidence` if any selected arm reports a non-`ok` outcome. + +## Thin-slice output + +The parsed findings are **unverified** at this stage. Present them to chat and write +`.deep-review-draft.md` per SKILL.md Phase 4 (`skill_phase: thin-slice`, `verification: none`, +the UNVERIFIED banner). Verification tags (CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN) and the +reconciled `.deep-review.md` are added in a later phase. + +## agy arm (RU2 — landed) + +The arm set is **codex + agy** (gemini was retired from the skill — it 410s on 2026-06-18; the +shared `arms.py` gemini arm remains for the cross-model eval, but the skill no longer offers it). +agy is **macOS-only**: its read-only floor is a macOS seatbelt, so +`env-detect.sh` reports agy `unavailable` off-darwin and the gate must not offer it; `arms.py` +independently refuses the agy arm when the seatbelt prefix is empty (off-darwin or a missing +template) rather than running it unfloored. + +Two bundled-context details, now handled: + +- The bundled `arms.py` `agy_sandbox_prefix()` reads `validation/agy-readonly.sb.tmpl` relative to + itself — `bundle-harness.sh` copies the template into `scripts/validation/`, so it resolves; + verify after any restructure. +- `arms.py` `_repo_root()` now honors `CMRE_REPO_DIR` (the reviewed plan's repo) so the deny-write + floor protects the user's plan repo, not arms.py's own location. `panel-critique.sh` exports it + via `git -C rev-parse --show-toplevel` (fallback: the plan's directory). diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/calibration/verifier-corpus.json b/plugins/compound-engineering/skills/ce-deep-review-beta/references/calibration/verifier-corpus.json new file mode 100644 index 000000000..3bfd534f8 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/calibration/verifier-corpus.json @@ -0,0 +1,29 @@ +{ + "_about": "RU6b calibration corpus for the deterministic quote-grep verifier (scripts/verify-findings.py). Each item is a synthetic finding embedding a quote; `expected` is the ground-truth verdict. GROUNDED items quote text that IS in `document` (some with format variants — smart quotes, whitespace, case — to exercise normalization); CONFABULATED items quote plausible text that is NOT in `document`. Run: `python3 scripts/verify-findings.py measure references/calibration/verifier-corpus.json`. Gate: false-CONFIRM and false-NOT-FOUND each <= 5%. NOTE: the verifier is deterministic and model-blind, so v3's agy-voiced sampling / synthetic-fallback / N=3-trials machinery does not apply — this is a straight labeled eval, N=1.", + "document": "# Plan: migrate the billing pipeline\n\nThe goal is to move invoicing off the legacy cron job and onto the event bus before the Q3 freeze.\nUnits 1-3 land the new consumer; Unit 4 backfills historical invoices.\nThe migration must run in a single atomic deploy so readers never see partial state.\nAuth tokens are rotated weekly; the new consumer reads them from the secrets manager, never from env.\nRollback restores the cron job from the last green release and replays the dead-letter queue.\nWe will not change the customer-facing invoice format in this phase.\nPerformance budget: p95 invoice render under 400ms; anything slower is a release blocker.\nThe dead-letter queue is drained on a 5-minute schedule by a separate worker.\nThe cutover is *all or nothing*: a **partial deploy** is a release blocker.\n", + "items": [ + { "id": "g01", "expected": "CONFIRMED", "text": "Coherence: the plan opens by promising to \"move invoicing off the legacy cron job\" but never names the cutover owner." }, + { "id": "g02", "expected": "CONFIRMED", "text": "Feasibility: \"single atomic deploy so readers never see partial state\" is hard to guarantee across three units." }, + { "id": "g03", "expected": "CONFIRMED", "text": "Security: tokens read \"from the secrets manager, never from env\" — good, but rotation cadence is unstated elsewhere." }, + { "id": "g04", "expected": "CONFIRMED", "text": "Reliability: rollback \"replays the dead-letter queue\" — confirm replay is idempotent." }, + { "id": "g05", "expected": "CONFIRMED", "text": "Scope: the doc says it \"will not change the customer-facing invoice format in this phase\" — verify downstream consumers agree." }, + { "id": "g06", "expected": "CONFIRMED", "text": "Performance: \"p95 invoice render under 400ms\" lacks a measurement method." }, + { "id": "g07", "expected": "CONFIRMED", "text": "The dead-letter queue \"drained on a 5-minute schedule\" may lag a real-time SLA." }, + { "id": "g08", "expected": "CONFIRMED", "text": "Unit 4 \"backfills historical invoices\" with no batch-size or throttle named." }, + { "id": "g09", "expected": "CONFIRMED", "text": "Timeline risk: moving \"onto the event bus before the Q3 freeze\" is aggressive." }, + { "id": "g10", "expected": "CONFIRMED", "text": "Format-variant (smart quotes): the plan says invoicing moves “onto the event bus before the Q3 freeze”." }, + { "id": "g11", "expected": "CONFIRMED", "text": "Format-variant (whitespace + case): rollback \"Restores the cron job from the last green release\"." }, + { "id": "g12", "expected": "CONFIRMED", "text": "Format-variant (markdown emphasis): the plan calls the cutover \"all or nothing\" but never defines the abort signal." }, + { "id": "c01", "expected": "NOT-FOUND-IN-DOC", "text": "Feasibility: the plan states \"we will rewrite the billing service in Rust\", a huge undertaking." }, + { "id": "c02", "expected": "NOT-FOUND-IN-DOC", "text": "The doc mandates \"a zero-downtime blue-green cutover\" but gives no infra for it." }, + { "id": "c03", "expected": "NOT-FOUND-IN-DOC", "text": "Security: \"tokens are stored in plaintext config\" is a serious exposure." }, + { "id": "c04", "expected": "NOT-FOUND-IN-DOC", "text": "It claims invoices are \"stored in a MongoDB collection per tenant\"." }, + { "id": "c05", "expected": "NOT-FOUND-IN-DOC", "text": "Performance: the plan sets \"p99 latency under 100 milliseconds\" as the bar." }, + { "id": "c06", "expected": "NOT-FOUND-IN-DOC", "text": "It says \"the freeze is lifted at the start of Q4\", contradicting the timeline." }, + { "id": "c07", "expected": "NOT-FOUND-IN-DOC", "text": "Reliability: \"rollback is performed manually over SSH\" by an on-call engineer." }, + { "id": "c08", "expected": "NOT-FOUND-IN-DOC", "text": "Scope: \"customer notification emails are sent through SendGrid\" expands the blast radius." }, + { "id": "c09", "expected": "NOT-FOUND-IN-DOC", "text": "It provisions \"a dedicated Kafka topic per customer account\"." }, + { "id": "c10", "expected": "NOT-FOUND-IN-DOC", "text": "The plan promises \"the legacy cron job is deleted immediately on deploy\"." }, + { "id": "c11", "expected": "NOT-FOUND-IN-DOC", "text": "Coherence: it asserts \"the dead-letter queue has a thirty-second drain SLA\"." } + ] +} diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/consent-gate.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/consent-gate.md new file mode 100644 index 000000000..d11fec1ac --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/consent-gate.md @@ -0,0 +1,84 @@ +# Consent gate — canonical copy + shape + +The gate is the only thing between the plan and external vendors. SKILL.md Phase 2 carries the +load-bearing routing; this file pins the **canonical wording** (do not paraphrase — the +acknowledgment and notice copy is the audited consent record) and the question shape. + +## Canonical responsibility acknowledgment + +**Base (gitleaks ran):** + +> This plan content will be sent to the external vendors you select below. You are responsible for +> having configured each vendor with an appropriate data-handling policy (paid plan + DPA where +> applicable) per your organization's requirements. Selecting any model confirms you accept this. + +**Escalated (append verbatim when `content_preview: unavailable`):** + +> No automated content scan ran (gitleaks is not installed) — you are the sole filter for what is +> egressed. Confirm you have manually checked this plan for secrets, credentials, and PII before +> sending it. + +## Canonical preview-unavailable notice + +Shown in the gate stem in place of the hit list when `gitleaks-scan.sh` returns +`{"status":"unavailable"}`: + +> Automated content preview unavailable — `gitleaks` is not installed (`brew install gitleaks` +> enables automated secret detection). Until then, you are the sole content filter for what is +> egressed. + +## Content preview rendering (when gitleaks ran) + +Render each hit as one line in the gate stem (values are already redacted by `--redact`): + +``` +Line (): +``` + +If `hits` is empty, state "Content preview: no secret-shaped content detected (best-effort; you are +the final filter)." + +## Question shape (within the 4-option blocking-tool cap) + +The acknowledgment lives in the **question stem**, but **each option label must itself carry the +egress verb and name the vendor** — `Send the plan to codex (OpenAI)`, not bare `codex (OpenAI)`. +This is load-bearing for the harness egress classifier (see "Egress-gate legibility" below), not +cosmetic. Selecting any model == consent. + +- **≥2 arms available** → `multiSelect: true` over the available models, labeled + `Send the plan to codex (OpenAI)` and `Send the plan to agy (Antigravity)`, default none. Each + label is self-contained, third-person, and states the egress + target vendor. The free-text/Other + escape serves as cancel. Submitting with ≥1 model selected grants consent for that subset; + submitting none triggers the one-time re-prompt (then cancel). +- **exactly 1 arm available** → single-select with two options: `Send the plan to ()` + and `Cancel — panel-only, no egress`. (A lone toggle cannot be a multi-select; the blocking tool + needs ≥2 options.) +- **0 arms available** → no gate; Phase 0 already routed to the panel-only sidecar. + +## Egress-gate legibility (why the labels name the egress) + +Under Claude Code's default auto-mode, an egress dispatch (`bash …/panel-critique.sh`) is screened +by a permission *classifier* that reasons about **conversation-level consent scope** — it is not +cleared by `allowed-tools` alone, and it does not read this gate's stem. It reads what the +conversation records the user *chose*. A bare `agy (Antigravity)` selection does not register as +"the user authorized sending the plan to an external vendor"; a `Send the plan to agy (Antigravity)` +selection does. The verb-carrying labels exist so the recorded consent is legible to the +classifier at dispatch time. Empirically (2026-05-28): a top-level authorization phrased this way +cleared the classifier and ran real egress, while a no-op-framed request was blocked as scope +escalation. See `references/arm-invocation.md` → "If the dispatch is blocked" for the fallback +path. Confirmation that the *in-skill* gate clears the classifier requires a fresh-session run +(the skill caches at session start). + +## Outcomes (mirror of SKILL.md routing — keep in sync) + +- ≥1 model selected → consent; pass the comma-separated subset to Phase 3 (egress = exactly that subset). +- zero selected (multi-select) → re-present once with "Select at least one model to proceed, or + Cancel for panel-only output"; still none → treat as Cancel. +- Cancel / decline → panel findings to chat; do NOT write `.deep-review.md` or + `.deep-review-draft.md` (the deep-review filename is reserved for verified output). + +## Audit + +Record `content_preview: ran | unavailable` for the sidecar header so a gitleaks-absent run is +itself audited. When unavailable, the Phase 4 / verification-phase sidecar write also reminds the +user that the sidecar quotes plan content and is about to be written without an automated scan. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/pass-1-headless-envelope.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/pass-1-headless-envelope.md new file mode 100644 index 000000000..12f51e93e --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/pass-1-headless-envelope.md @@ -0,0 +1,73 @@ +# Pass 1 — headless ce-doc-review invocation + envelope parsing + +Pass 1 is the Claude panel (no egress). `ce-deep-review-beta` invokes `ce-doc-review` in headless +mode and parses its structured envelope, then carries the panel findings forward (untagged, +trusted) into the thin-slice draft / reconciled sidecar. + +## Invocation + +Invoke the `ce-doc-review` skill via the platform's skill-invocation primitive in **headless mode**, +passing the plan path: + +- Claude Code: `Skill("ce-doc-review", "mode:headless ")` +- Other platforms: the equivalent skill-invocation with the same `mode:headless ` args. + +Do NOT tell the user to type `/ce-doc-review` — invoke it programmatically. ce-doc-review runs its +own multi-persona panel (several minutes) and returns a text envelope terminated by `Review complete`. + +## Envelope shape (what to parse) + +The headless envelope uses these top-level sections (any with zero items are omitted): + +``` +Document review complete (headless mode). + +Applied N fixes: +-
: () + +Proposed fixes (concrete fix, requires user confirmation): +[P0] Section:
— (<reviewer>, confidence <anchor>) + Why: <why_it_matters> + Suggested fix: <suggested_fix> + +Decisions (requires user judgment): +[P1] Section: <section> — <title> (<reviewer>, confidence <anchor>) + Why: <why_it_matters> + Suggested fix: <suggested_fix or "none"> + +FYI observations (anchor 50, no decision required): +[P3] Section: <section> — <title> (<reviewer>, confidence <anchor>) + Why: <why_it_matters> + +Residual concerns: +- <concern> (<source>) + +Deferred questions: +- <question> (<source>) + +Review complete +``` + +## Parsing rules + +- **Detect completion** by the terminal `Review complete` line. If it is absent, treat pass 1 as + failed (see Failure UX). +- Capture each section's items verbatim into a structured set: `applied_fixes[]`, `proposed_fixes[]`, + `decisions[]`, `fyi[]`, `residual[]`, `deferred[]`. These become the **Claude panel findings** — + carried into the report untagged (the panel is trusted; only cross-model findings get verified). +- Handle the high-count compact rendering (FYI/residual/deferred collapsed to one-line bullets when + the combined count is ≥5) — parse the bullets, not per-item `Why`. +- The envelope is prose, not JSON. Match section headers leniently (`.toMatch`-style); do not assume + exact spacing. + +## Failure UX (load-bearing) + +ce-doc-review is the no-egress half of the workflow. If pass 1 fails — the invocation errors, times +out, or returns an envelope **without** the `Review complete` terminal line — STOP: + +> Pass 1 failed: <reason> — cannot open the consent gate without panel results. Re-invoke, or run +> ce-doc-review directly to diagnose. + +**Do not open the consent gate or egress anything** when pass 1 did not complete. The gate exists to +authorize sending plan content to external vendors; with no panel results there is nothing to add +cross-model arms to, and proceeding would egress without the panel's no-egress baseline. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/reconciliation.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/reconciliation.md new file mode 100644 index 000000000..2e57a050a --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/reconciliation.md @@ -0,0 +1,77 @@ +# Reconciliation — the verified `.deep-review.md` sidecar (RU5) + +Once Pass 2 findings are verdict-tagged (Phase 3.5), reconciliation assembles the **verified** +sidecar at the reserved name `<plan>.deep-review.md`. This is the skill's canonical output; the +thin-slice `<plan>.deep-review-draft.md` was the pre-verification placeholder. + +## Filename reclaim + leave the draft in place + +- The verified output writes to `<plan>.deep-review.md` (the reserved name). +- An existing `<plan>.deep-review-draft.md` from a thin-slice/dogfood run is **left in place** — do + not delete or overwrite it. It is a historical dogfood artifact; rotation and reclaim never touch + it (it has a `-draft` infix, not a `.`-delimited rotation infix). + +## Rotation (data-loss-safe — keep 5) + +Before writing a fresh `<plan>.deep-review.md`, rotate any existing one out of the way and prune: + +```bash +python3 "${CLAUDE_SKILL_DIR}/scripts/reconcile.py" rotate "<plan>.deep-review.md" +``` + +- If `<plan>.deep-review.md` exists, it is renamed to `<plan>.deep-review.<ISO>.md` (UTC stamp). +- Then only the **5 newest rotations** (by the ISO infix) survive; older ones are deleted. +- The prune matches **only** rotation files `<plan>.deep-review.<infix>.md` — never the base (just + renamed away) and never the `-draft` sidecar. `skill_phase` persists in each rotated copy's + frontmatter, so a rotated thin-slice draft is identifiable by frontmatter, not just by timestamp. + +After rotation, write the fresh `<plan>.deep-review.md`. + +## Frontmatter (verified) + +``` +skill_phase: verified +verification: quote-grep-backstop +coverage: full | reduced-confidence | panel-only +verdicts: {confirmed: N, not_found_in_doc: N, needs_human: N} +plan: <path> +models: <csv> +timestamp: <ISO, UTC> +user: <git config user.name> +content_preview: ran | unavailable +``` + +`coverage` and `verification` are **orthogonal axes** — one is "did all consented arms return?", the +other is "was the output grounded?" Keep them as separate fields; do not collapse them. + +## Banner precedence + +- The verified sidecar shows the **coverage** banner only (`full` → no banner needed; + `reduced-confidence` → a one-line banner naming which arm degraded). The thin slice's UNVERIFIED + banner does **not** appear here — the output is verified. +- Still surface a one-line triage note when there are NEEDS-HUMAN findings: "N cross-model findings + need human triage (no verbatim quote to auto-ground)." This is informational, not the UNVERIFIED + banner. + +## Body + +1. **Claude panel findings** — trusted, untagged (carried verbatim from Pass 1). +2. **Cross-model findings** — grouped by lens, verdict-tagged, with the grounding quote on each + CONFIRMED. Render deterministically: + + ```bash + python3 "${CLAUDE_SKILL_DIR}/scripts/reconcile.py" render-cross-model "<verify-records.json>" + ``` + +3. **Decision-changing union** — a short closing section listing the findings (panel + CONFIRMED + cross-model) that would change a go/no-go decision on the plan, so a reader gets the load-bearing + set without scanning every lens. NEEDS-HUMAN findings are not auto-included here (un-adjudicated); + the human may promote one after triage. + +## Committed-leak reminder + +When `content_preview: unavailable` (gitleaks absent), the chat summary must remind the user that +the sidecar quotes plan content and is about to be written — and, if they commit it, egressed into +version history — without an automated secret scan. **Do NOT modify `.gitignore`** (an untracked +sidecar is the user's to manage; silently ignoring it could hide a file they meant to share, and +silently committing it is the leak risk). This is a known open decision, not a settled default. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/ship-state-machine.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/ship-state-machine.md new file mode 100644 index 000000000..f3e88c44d --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/ship-state-machine.md @@ -0,0 +1,30 @@ +# Run-state machine (thin slice) + +The thin-slice run has several independent state dimensions. Tracking them explicitly keeps the +flow honest about partial coverage and prevents egress without consent. + +| Dimension | States | Notes | +|---|---|---| +| **arms-detected** | `<per-arm: ok \| unauthed \| missing>` | from `env-detect.sh`; only `ok` arms are offered at the gate | +| **pass-1** | `idle → running → complete \| failed` | `failed` (error / timeout / no `Review complete`) STOPS the run; the gate never opens | +| **consent** | `pending → granted(<subset>) \| declined` | `declined` (Cancel, or empty after the one re-prompt) → panel-only chat, no sidecar | +| **per-arm pass-2** | `idle → running → ok \| timeout \| missing \| auth_fail \| empty \| malformed` | per (model, lens); any non-`ok` → coverage `reduced-confidence` | +| **coverage** | `full \| reduced-confidence \| panel-only` | `full` = all consented arms `ok`; `panel-only` = zero arms / pass-2 never ran | +| **verification** | `none (thin-slice)` | thin slice does not verify; later phase adds `queued → running → complete` | +| **sidecar** | `unwritten → written` | `.deep-review-draft.md` (thin slice) / `.panel-review.md` (panel-only) / none (declined) | + +## Hard preconditions (never violate) + +- **No gate, no egress.** `consent != granted` ⇒ pass-2 must not run. The `--models` subset passed + to the harness is exactly the granted subset — nothing else is ever invoked. +- **No panel, no gate.** `pass-1 != complete` ⇒ the consent gate must not open (see pass-1 failure UX). +- **Filename reservation.** `<plan>.deep-review.md` is written only by the (later) verified phase. + The thin slice writes `<plan>.deep-review-draft.md`; a declined run writes neither. + +## Transitions (happy path) + +`detect arms → pass-1 running → complete → consent pending → granted(subset) → per-arm running → +(ok/…)* → coverage computed → sidecar written (draft) → summary to chat`. + +Any `failed`/`declined` short-circuits to its terminal (report + stop) without egress beyond what +consent granted. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/references/verification-protocol.md b/plugins/compound-engineering/skills/ce-deep-review-beta/references/verification-protocol.md new file mode 100644 index 000000000..cd2fde610 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/references/verification-protocol.md @@ -0,0 +1,69 @@ +# Verification protocol (RU4) — grounding cross-model findings + +Pass 2 produces **raw, unverified** cross-model findings. Verification grounds each one against the +reviewed document with a **deterministic quote-grep backstop** and assigns exactly one verdict. This +is what replaces the thin slice's `verification: none`. + +## The backstop is the authoritative gate + +The verifier is `scripts/verify-findings.py` — a pure function of `(finding text, document)`. It is +**authoritative**: a model (including the orchestrating agent) may not override a deterministic +CONFIRMED or NOT-FOUND-IN-DOC verdict. This is load-bearing — a model verifier judging another +model's findings can inherit the same confabulation it is meant to catch; a deterministic grep +cannot. The backstop being authoritative is the property that makes the verdicts trustworthy. + +It is **blind to the producing model**: the verdict function never receives the model label. +`verify-records` reads the model only from the record filename to *label* output rows, never to +compute a verdict. So provenance (which vendor produced a finding) cannot bias the verdict. + +## Verdicts + +Run via the Bash tool after Pass 2 completes: + +```bash +python3 "${CLAUDE_SKILL_DIR}/scripts/verify-findings.py" verify-records "<plan-path>" "${CMRE_OUT_DIR:-/tmp/cmre-panel}/records" +``` + +It emits `{"verified": [{model, lens, id, text, verdict, grounding_quote}], "counts": {...}}`. Each +finding gets one of: + +- **CONFIRMED** — the finding embeds a *substantial verbatim quote* (a multi-word phrase, normalized + length ≥ 12 chars) that appears in the document. `grounding_quote` carries the matched span. + CONFIRMED means **the cited evidence exists in the document — NOT that the finding is correct, + important, or fairly characterized.** Claim correctness and severity are a human's call; the + backstop only certifies the quote is real. +- **NOT-FOUND-IN-DOC** — the finding embeds a substantial quote that does **not** appear in the + document. The model claimed document text that isn't there (fabricated, or a paraphrase the model + wrongly presented as a quote). Surface it as a flag; do not silently drop it (a human confirms). +- **NEEDS-HUMAN** — the finding has no substantial verbatim quote to check: a paraphrase, a + cross-section implication, or only a lone identifier/filename in backticks (too trivial to ground). + The backstop cannot auto-ground it; a human decides. **This is the expected default for a large + share of findings** — a quote-grep can only adjudicate findings that quote the document. A high + NEEDS-HUMAN count is not a failure; it is the honest reach of a deterministic backstop. + +## Why a quote-grep, with eyes open + +A deterministic quote-grep is brittle in known ways, and the protocol is designed around them rather +than pretending they don't exist: + +- **Paraphrased quotes** ("must" vs "should") miss the grep. Normalization (lowercase, folded smart + quotes/dashes, collapsed whitespace) absorbs format-only differences, but a genuine wording change + yields NOT-FOUND-IN-DOC. That is acceptable: a finding that quotes the document should quote it + accurately; an inaccurate "quote" is itself worth flagging. +- **Valid findings without a quote** (cross-section implications, structural observations) land in + NEEDS-HUMAN, never NOT-FOUND-IN-DOC — the backstop does not punish a finding for lacking a quote, + it just declines to auto-confirm it. +- **Trivial matches** (a finding mentioning `` `panel-critique.sh` ``) are excluded: grounding quotes + must be multi-word phrases, so a lone filename/identifier cannot manufacture a CONFIRMED. + +The NOT-FOUND-IN-DOC and (eventual) miscategorization rates are what RU6's verifier-rate measurement +tracks against the ≤5% bar. Until that lands, treat NOT-FOUND-IN-DOC as "worth a human's glance," +not "discard." + +## What this protocol does NOT do (v1 scope) + +- No model-based verifier. A blinded LLM triage of NEEDS-HUMAN findings is a possible later + enhancement, but v1 keeps the deterministic backstop as the sole authoritative gate — it sidesteps + the verifier-contamination failure mode entirely. +- No claim-quality scoring (importance, severity, fairness). Out of scope for a quote-grep. +- No auto-deletion. Every finding survives to the output with its verdict; the human triages. diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/arms.py b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/arms.py new file mode 100644 index 000000000..66473fabe --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/arms.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Cross-model review eval — CLI arms b (isolated) and c (fixed context) (U3). + +Invokes the external model CLIs (codex, agy) over the document via stdin using +argv lists — never string interpolation into a shell — so document content +cannot inject commands (R2 / R14). + +Arm b is run isolated from the repo (clean cwd + HOME/config overrides) so the +model genuinely has no workspace context; arm c additionally supplies a fixed +context set. Because "isolation flags are present" is not the same as "the model +had no context", U3 includes a positive isolation PROBE (AD2 / P1): plant a +sentinel only reachable from repo/global config and assert arm b cannot surface +it. The probe's leak-detection logic is unit-tested here; the live subprocess +runs are integration-level (validated at eval time). +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +# Validated invocation forms: +# codex 0.133.0: codex exec -s read-only --skip-git-repo-check - (prompt via stdin) +# gemini 0.43.0: gemini -p "<instruction>" --approval-mode plan --skip-trust -o text +# (-p is appended to stdin; plan = read-only so the arm never edits; +# --skip-trust is REQUIRED for headless runs in a clean/untrusted CWD, +# else gemini exits 55 "not running in a trusted directory") +# Both arms run from a clean temp CWD (no ambient workspace access). codex needs +# --skip-git-repo-check or it refuses with "Not inside a trusted directory". +# +# NOTE: agy 1.0.2 (--print) was UNRELIABLE (empty output) and got dropped — but agy 1.0.3 is a +# VIABLE non-interactive reviewer (clean JSON via --print + stdin). Its own --sandbox does NOT +# confine the filesystem, so on macOS the agy arm is wrapped in a seatbelt deny-write profile +# (allow-default + deny writes to repo/home-creds/dotfiles; a strict deny-all-write OR any +# deny-read HANGS agy). Auth is OAuth at ~/.gemini/oauth_creds.json (+ refresh_token; do NOT gate +# detection on expiry — agy auto-refreshes). See +# docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md (Phase 0 / U2). +CODEX_BASE = ["codex", "exec", "-s", "read-only", "--skip-git-repo-check", "-"] +AGY_INSTRUCTION = "Review the document provided on stdin. Return ONLY a JSON array of finding strings (one element per distinct finding), no prose or preamble." +GEMINI_INSTRUCTION = "Review the document provided on stdin. Do not modify files. Return ONLY a JSON array of finding strings (one element per distinct finding), no prose or preamble." +GEMINI_BASE = ["gemini", "-p", GEMINI_INSTRUCTION, "--approval-mode", "plan", "--skip-trust", "-o", "text"] + +# Lines like "1. foo" or "2) bar" — numbered findings the model commonly emits. +NUMBERED_ITEM = re.compile(r"^\s*\d+[.)]\s+(.*)$") + + +def _repo_root(): + """Repo whose writes the agy seatbelt floor denies. + + Honors CMRE_REPO_DIR when set — the REVIEWED document's repo, passed by the caller. The + installed skill reviews a user's plan, where arms.py's own location is NOT the right repo to + protect; panel-critique.sh exports CMRE_REPO_DIR from the plan's directory. Falls back to + arms.py's canonical in-repo location (scripts/eval/cross_model_review/arms.py) for the + eval-harness case where no caller supplies it. + """ + env = os.environ.get("CMRE_REPO_DIR") + if env: + return os.path.realpath(env) + return os.path.realpath(os.path.join(os.path.dirname(__file__), "..", "..", "..")) + + +def agy_sandbox_prefix(): + """macOS seatbelt wrapper for the agy arm (Phase 0 / U2, validated 2026-05-28). + + Returns (argv_prefix, profile_path). On macOS, generates a concrete deny-write seatbelt + profile from validation/agy-readonly.sb.tmpl (deny writes to the repo + home creds/dotfiles; + allow-default otherwise — agy HANGS under deny-all-write or any deny-read) and returns + (["sandbox-exec", "-f", <profile>], <profile>). On non-macOS, returns ([], None): seatbelt is + macOS-only and the FS floor is not enforced there (documented limitation). The caller unlinks + the profile after the run. + """ + if sys.platform != "darwin": + return [], None + tmpl = os.path.join(os.path.dirname(__file__), "validation", "agy-readonly.sb.tmpl") + if not os.path.exists(tmpl): + return [], None + # Paths must be canonical — seatbelt matches /private/var..., and /Users is already canonical. + profile_text = ( + Path(tmpl).read_text() + .replace("__REPO_DIR__", _repo_root()) + .replace("__HOME__", os.path.realpath(os.path.expanduser("~"))) + ) + fd, path = tempfile.mkstemp(prefix="cmre-agy-sb-", suffix=".sb") + with os.fdopen(fd, "w") as f: + f.write(profile_text) + return ["sandbox-exec", "-f", path], path + + +def clean_cwd(): + """A fresh temp CWD with no ambient repo access. + + Both arms run from here so neither inherits the repo's workspace context + (codex's AGENTS.md walk-up and git-repo discovery both start from CWD). HOME + is deliberately NOT overridden — that would strip the CLI's auth (found via + live smoke). The global config under HOME (~/.codex, agy) is therefore a + constant across both arms, so it does not confound the b-vs-c delta; the only + difference between the arms is the fixed context arm c injects via stdin. The + sentinel isolation probe (detect_leak) guards against repo-context leakage. + """ + return tempfile.mkdtemp(prefix="cmre-arm-cwd-") + + +def build_invocation(arm, cli, doc_text, rubric, context_text=None): + """Assemble the subprocess spec. Document content travels via stdin only. + + Returns a spec dict (argv, cwd, env_overrides, stdin payload metadata). The + actual stdin string is returned under `stdin` for the runner; tests assert + on argv/cwd/env, never needing the full payload. + """ + if cli == "codex": + argv = list(CODEX_BASE) + elif cli == "gemini": + argv = list(GEMINI_BASE) + elif cli == "agy": + argv = ["agy", "--print", AGY_INSTRUCTION] + else: + raise ValueError(f"unknown cli: {cli}") + + parts = [rubric, "\n\n=== DOCUMENT ===\n", doc_text] + if arm == "c_fixed_context" and context_text: + parts += ["\n\n=== REPO CONTEXT (fixed set) ===\n", context_text] + stdin_payload = "".join(parts) + + # Both arms run from a clean CWD (no ambient repo access). HOME is preserved + # so the CLI keeps its auth; the global config is constant across arms and + # does not confound the b-vs-c delta. The only difference is arm c's injected + # context above. + cwd = clean_cwd() + env = dict(os.environ) + + # Defensive: document content must never appear as an argv element. + doc_in_argv = any(doc_text and doc_text in a for a in argv) + + return { + "arm": arm, + "cli": cli, + "argv": argv, + "cwd": cwd, + "isolated_from_repo": True, + "skip_git_repo_check": "--skip-git-repo-check" in argv, + "stdin_has_context": arm == "c_fixed_context" and bool(context_text), + "doc_in_argv": doc_in_argv, + # agy's own flags don't confine the FS, so its arm runs under a macOS seatbelt deny-write + # profile applied at run time (see agy_sandbox_prefix / run_invocation). Logical argv stays + # ["agy","--print",...]; the sandbox wrapping is an execution concern, not part of the spec. + "sandbox": "seatbelt-deny-write" if cli == "agy" else None, + "_env": env, + "_stdin": stdin_payload, + } + + +def detect_leak(output, sentinel): + """The isolation probe's check: did arm b surface a sentinel it should not have?""" + return bool(sentinel) and sentinel in output + + +def parse_findings(text): + """Parse a model's output into findings [{id, text}]. + + The reliable path is a JSON array (the arm instruction now requests one); + `--output-format json`/fenced JSON is tolerated. Otherwise: markdown bullets + or numbered items; otherwise blank-line-separated paragraphs. We deliberately + do NOT split on every newline — verbose models (e.g. codex) wrap a single + finding across lines, so line-splitting over-counts wildly (one review parsed + as ~100 findings). Counts from unstructured prose are best-effort; structured + JSON output is what makes the yield metric trustworthy. + """ + text = (text or "").strip() + if not text: + return [] + # Tolerate a ```json ... ``` fence around the array. + json_text = text + if json_text.startswith("```"): + json_text = re.sub(r"^```[a-zA-Z0-9]*\n?", "", json_text) + json_text = re.sub(r"\n?```$", "", json_text).strip() + try: + data = json.loads(json_text) + if isinstance(data, list): + out = [] + for i, item in enumerate(data, 1): + if isinstance(item, dict) and "text" in item: + out.append({"id": item.get("id", f"f{i}"), "text": str(item["text"])}) + else: + out.append({"id": f"f{i}", "text": str(item)}) + return out + except json.JSONDecodeError: + pass + items = [] + for ln in text.splitlines(): + s = ln.strip() + if s.startswith(("- ", "* ")): + items.append(s[2:].strip()) + continue + m = NUMBERED_ITEM.match(ln) + if m: + items.append(m.group(1).strip()) + items = [i for i in items if i] + if items: + return [{"id": f"f{i}", "text": b} for i, b in enumerate(items, 1)] + # Best-effort prose fallback: blank-line-separated paragraphs only (a clear finding + # boundary), never per-line (over-counts wrapped prose). + paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()] + if len(paras) > 1: + return [{"id": f"f{i}", "text": p} for i, p in enumerate(paras, 1)] + return [{"id": "f1", "text": text}] + + +def run_invocation(spec, timeout): + """Run the CLI arm as a subprocess (integration-level; not unit-tested with live CLIs). + + For the agy arm, applies the macOS seatbelt deny-write floor at run time (the spec's logical + argv stays ["agy","--print",...]; the sandbox wrapping is an execution concern). The generated + profile is unlinked after the run. + """ + argv = spec["argv"] + sb_profile = None + if spec.get("sandbox") == "seatbelt-deny-write": + prefix, sb_profile = agy_sandbox_prefix() + # Defense-in-depth (R5): agy's read-only floor IS the macOS seatbelt. agy_sandbox_prefix() + # returns an empty prefix off-darwin OR when the profile template is missing — refuse rather + # than run agy unfloored, so a direct `arms.py run-arm ... agy` on a non-macOS host (or a + # mis-bundled skill) can't bypass env-detect's platform-gate and exfiltrate with no floor. + if not prefix: + return { + "status": "error", + "latency_ms": 0, + "findings": [], + "stderr": "agy arm refused: its read-only floor is macOS-only (seatbelt) and was " + "unavailable here (non-macOS host, or missing agy-readonly.sb.tmpl). " + "agy is macOS-only — use codex/gemini on other platforms.", + } + argv = prefix + argv + start = time.monotonic() + try: + try: + proc = subprocess.run( + argv, + input=spec["_stdin"], + cwd=spec["cwd"], + env=spec["_env"], + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except subprocess.TimeoutExpired: + return {"status": "timeout", "latency_ms": (time.monotonic() - start) * 1000, "findings": [], "stderr": ""} + except FileNotFoundError: + return {"status": "error", "latency_ms": 0, "findings": [], "stderr": f"{spec['cli']} not found"} + latency_ms = (time.monotonic() - start) * 1000 + status = "ok" if proc.returncode == 0 else "error" + findings = parse_findings(proc.stdout) if status == "ok" else [] + return {"status": status, "latency_ms": latency_ms, "findings": findings, "stderr": proc.stderr} + finally: + if sb_profile and os.path.exists(sb_profile): + try: + os.unlink(sb_profile) + except OSError: + pass + + +def _read(path): + return Path(path).read_text() + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Cross-model review eval CLI arms (b, c).") + sub = parser.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("build-invocation") + p.add_argument("arm", choices=["b_isolated", "c_fixed_context"]) + p.add_argument("cli", choices=["codex", "gemini", "agy"]) + p.add_argument("doc") + p.add_argument("rubric") + p.add_argument("--context") + + p = sub.add_parser("detect-leak") + p.add_argument("sentinel") + p.add_argument("output") + + p = sub.add_parser("parse-findings") + p.add_argument("output") + + p = sub.add_parser("run-arm") + p.add_argument("arm", choices=["b_isolated", "c_fixed_context"]) + p.add_argument("cli", choices=["codex", "gemini", "agy"]) + p.add_argument("doc") + p.add_argument("rubric") + p.add_argument("--context") + p.add_argument("--doc-id", required=True) + p.add_argument("--trial", type=int, default=1) + p.add_argument("--timeout", type=float, default=180.0) + + args = parser.parse_args(argv) + + if args.cmd == "build-invocation": + ctx = _read(args.context) if args.context else None + spec = build_invocation(args.arm, args.cli, _read(args.doc), _read(args.rubric), ctx) + # Do not leak the full env/stdin into the printed spec. + printable = {k: v for k, v in spec.items() if not k.startswith("_")} + printable["stdin_len"] = len(spec["_stdin"]) + print(json.dumps(printable)) + return 0 + + if args.cmd == "detect-leak": + print(json.dumps({"leaked": detect_leak(_read(args.output), args.sentinel)})) + return 0 + + if args.cmd == "parse-findings": + print(json.dumps({"findings": parse_findings(_read(args.output))})) + return 0 + + if args.cmd == "run-arm": + ctx = _read(args.context) if args.context else None + spec = build_invocation(args.arm, args.cli, _read(args.doc), _read(args.rubric), ctx) + result = run_invocation(spec, args.timeout) + record = { + "arm": args.arm, + "doc_id": args.doc_id, + "trial": args.trial, + "status": result["status"], + "producer": "runner", + "latency_ms": result["latency_ms"], + "findings": result["findings"], + "model": args.cli, + } + # stderr carries the CLI's diagnostics (auth/availability failures) for the smoke check. + if result.get("stderr"): + sys.stderr.write(result["stderr"]) + print(json.dumps(record)) + return 0 if result["status"] == "ok" else 1 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh new file mode 100755 index 000000000..0bf378e99 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/bundle-harness.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +# Build-time copy: bundle the canonical cross-model harness into this skill so the installed skill +# is self-contained (AGENTS.md "each skill directory is a self-contained unit"). Run after ANY change +# to the canonical files -- INCLUDING eval-only changes, since arms.py / panel-critique.sh are shared +# with the cross-model eval workflow. The bundle-drift test +# (tests/skills/ce-deep-review-beta-bundle-drift.test.ts) fails until you re-run this. +# +# Symlinks are deliberately NOT used: the converter copies each skill dir as an isolated unit, so a +# symlink would dangle on install. +set -eu +here="$(cd "$(dirname "$0")" && pwd)" +repo="$(git -C "$here" rev-parse --show-toplevel)" +src="$repo/scripts/eval/cross_model_review" +mkdir -p "$here/validation" +cp "$src/panel-critique.sh" "$here/panel-critique.sh" +cp "$src/arms.py" "$here/arms.py" +cp "$src/validation/agy-readonly.sb.tmpl" "$here/validation/agy-readonly.sb.tmpl" +echo "bundled into $here : panel-critique.sh, arms.py, validation/agy-readonly.sb.tmpl" diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh new file mode 100755 index 000000000..955caafc7 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# ce-deep-review-beta — offline detection of which cross-model arms are usable (R9). +# +# Emits ONLY a JSON status record to stdout. It MUST NOT print credential values, tokens, or file +# contents to any stream — detection uses only `command -v`, env-var PRESENCE, and credential-file +# PRESENCE (test -s). It makes NO vendor API calls (an authenticated call would be pre-consent +# egress) and never reads credential file contents. +# +# Arms = codex + agy. (gemini was retired from the skill — it 410s on 2026-06-18; the shared +# arms.py gemini arm remains for the cross-model eval, but the skill no longer offers it.) +# - grok is deferred: grok 0.2.8 headless is blocked by a relay-auth bug +# (docs/solutions/skill-design/2026-05-28-grok-arm-posture-validation.md). +# - agy's read-only floor is a macOS seatbelt, so agy is macOS-ONLY: off-darwin it reports +# "unavailable" regardless of install/auth, and the consent gate must not offer it (R5 — never +# offer an arm whose floor is unenforced; arms.py also refuses agy off-darwin). +# Statuses: "ok" (installed + auth signal present), "unauthed" (installed, no auth signal), +# "missing" (binary not on PATH), "unavailable" (platform-gated off — agy off macOS). +set -u + +status_for() { + bin="$1"; auth_present="$2" # auth_present: "yes" | "no" + if ! command -v "$bin" >/dev/null 2>&1; then + printf 'missing' + elif [ "$auth_present" = "yes" ]; then + printf 'ok' + else + printf 'unauthed' + fi +} + +# codex: ~/.codex/auth.json non-empty (auth managed by codex login). +codex_auth=no +[ -s "$HOME/.codex/auth.json" ] && codex_auth=yes +codex_status="$(status_for codex "$codex_auth")" + +# agy: macOS-ONLY (its read-only floor is a macOS seatbelt; arms.py refuses agy off-darwin). Off +# macOS -> "unavailable" so the gate never offers an unfloored arm. Auth = agy's OAuth file +# (~/.gemini/oauth_creds.json); presence only (test -s) — never read contents, and do NOT gate on +# expiry (agy auto-refreshes). +agy_status=unavailable +if [ "$(uname -s)" = "Darwin" ]; then + agy_auth=no + [ -s "$HOME/.gemini/oauth_creds.json" ] && agy_auth=yes + agy_status="$(status_for agy "$agy_auth")" +fi + +printf '{"codex":"%s","agy":"%s"}\n' "$codex_status" "$agy_status" diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/gitleaks-scan.sh b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/gitleaks-scan.sh new file mode 100755 index 000000000..7fa824006 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/gitleaks-scan.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# ce-deep-review-beta content preview for the consent gate. Emits exactly ONE JSON line: +# {"status":"unavailable"} gitleaks not installed OR the invocation errored -> +# gate escalates to the sole-filter acknowledgment (does NOT block) +# {"status":"ran","hits":[...]} gitleaks ran successfully (hits may be empty == clean) +# --redact keeps secret VALUES out of the output (hits carry only line + rule + a redacted preview). +# Never blocks egress; the SKILL decides what to do with the result. +# +# An invocation error (gitleaks present but the run failed -- bad flag, config error, runtime fault) +# must NOT be reported as a clean "ran": that would open vendor egress under a false clean-scan +# signal. gitleaks documents exit 0 = no leaks, 1 = leaks found, and other nonzero codes (e.g. 2) +# for errors. We capture the exit code and treat anything outside {0,1}, or a missing/unparseable +# report, as "unavailable" so the gate routes to the escalated sole-filter path it already handles. +set -u +plan="${1:-}" +if [ -z "$plan" ] || [ ! -f "$plan" ]; then printf '{"status":"unavailable"}\n'; exit 0; fi +if ! command -v gitleaks >/dev/null 2>&1; then printf '{"status":"unavailable"}\n'; exit 0; fi + +rep="$(mktemp -t gitleaks-XXXXXX)" +# gitleaks exits 0 when clean and 1 when leaks are found -- both mean it RAN. Any other code is an error. +gitleaks detect --no-git --source "$plan" --report-format json --report-path "$rep" --redact >/dev/null 2>&1 +rc=$? +if [ "$rc" -ne 0 ] && [ "$rc" -ne 1 ]; then rm -f "$rep"; printf '{"status":"unavailable"}\n'; exit 0; fi + +# Parse the report. A missing/unparseable report after an ostensibly-successful run is itself an +# error signal (e.g. gitleaks aborted before writing a valid report), so distinguish parse failure +# from a genuinely empty (clean) report rather than mapping both to "[]". +hits="$(python3 -c ' +import json, sys +try: + data = json.load(open(sys.argv[1])) +except Exception: + print("__PARSE_ERROR__") + sys.exit(0) +out = [{"line": d.get("StartLine"), "rule": d.get("RuleID"), "preview": (d.get("Match") or "")[:60]} for d in (data or [])] +print(json.dumps(out)) +' "$rep" 2>/dev/null)" +rm -f "$rep" +if [ -z "$hits" ] || [ "$hits" = "__PARSE_ERROR__" ]; then printf '{"status":"unavailable"}\n'; exit 0; fi +printf '{"status":"ran","hits":%s}\n' "$hits" diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/panel-critique.sh b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/panel-critique.sh new file mode 100755 index 000000000..3caf4d9ac --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/panel-critique.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Fair cross-model PANEL critique: run the cross-model arms through the SAME six review lenses the +# Claude ce-doc-review panel uses (coherence, feasibility, security, scope, product, adversarial), +# so a Claude-panel-vs-cross-model comparison isn't confounded by prompt asymmetry. Persists the +# FULL record per (model x lens) for post-hoc judging — nothing is truncated away. +# +# Usage: panel-critique.sh [--models <csv>] <doc.md> [context.md] +# Arms = codex + agy. agy is macOS-ONLY (read-only floor is a seatbelt). gemini was retired from +# the skill (it 410s 2026-06-18); the arms.py gemini arm remains for the cross-model eval. Records +# -> $CMRE_OUT_DIR (default /tmp/cmre-panel/records). Each run SENDS THE DOCUMENT to that vendor +# (codex -> OpenAI, agy -> Antigravity); arms can be slow — raise CMRE_TIMEOUT. +set -u + +here="$(cd "$(dirname "$0")" && pwd)" +arms="$here/arms.py" + +# Optional `--models <csv>` subset (default = all available arms = codex + agy). Lets a caller (e.g. +# ce-deep-review-beta's consent gate) restrict egress to exactly the consented models. Egress must +# equal consent, so the subset is filtered BEFORE running each arm -- never by discarding records +# post-hoc (the document would already have been sent). Unavailable / off-platform arms are +# warn-SKIPped per cell (not a missing binary, not agy off-macOS), never fatal: the rest still run. +models="codex agy" +if [ "${1:-}" = "--models" ]; then + models="$(printf '%s' "${2:-}" | tr ',' ' ')" + shift 2 +fi + +case "${1:-}" in -h|--help|"") sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; esac + +plan="$1"; context="${2:-}" +[ -f "$plan" ] || { echo "error: doc not found: '$plan'" >&2; exit 2; } +if [ -n "$context" ] && [ ! -f "$context" ]; then echo "error: context not found: '$context'" >&2; exit 2; fi + +# The agy arm's deny-write floor must deny writes to the REVIEWED document's repo, not arms.py's own +# location (matters for the installed skill reviewing a user's plan). Resolve it from the plan's +# directory; fall back to that directory when the plan isn't inside a git repo. arms.py reads CMRE_REPO_DIR. +plan_dir="$(cd "$(dirname "$plan")" && pwd)" +CMRE_REPO_DIR="$(git -C "$plan_dir" rev-parse --show-toplevel 2>/dev/null || printf '%s' "$plan_dir")" +export CMRE_REPO_DIR + +out="${CMRE_OUT_DIR:-/tmp/cmre-panel}"; rec_dir="$out/records"; mkdir -p "$rec_dir" +arm="b_isolated"; [ -n "$context" ] && arm="c_fixed_context" +doc_id="$(basename "$plan" .md)" +timeout="${CMRE_TIMEOUT:-420}" + +# Six lens rubrics, distilled from the ce-doc-review personas so the cross-model arms get the +# same coverage the Claude panel does. +lens_dir="$(mktemp -d -t cmre-lenses-XXXXXX)" +cat > "$lens_dir/coherence.md" <<'EOF' +Review this document for INTERNAL CONSISTENCY: contradictions between sections, terminology drift, +dependency/sequencing claims that conflict, and ambiguity where two readers would diverge. Return +your findings as a JSON array of strings, one element per distinct finding; quote the conflicting text in each. +EOF +cat > "$lens_dir/feasibility.md" <<'EOF' +Review whether the proposed approach will SURVIVE CONTACT WITH REALITY: architecture conflicts, +dependency gaps, migration/cutover risks, environment assumptions, implementability. Challenge the +load-bearing claims. Return your findings as a JSON array of strings, one element per distinct finding; name the concrete risk in each. +EOF +cat > "$lens_dir/security.md" <<'EOF' +Review for SECURITY gaps: auth/authz assumptions, data exposure, credential handling, trust +boundaries, PII, and missing threat-model elements. Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/scope.md" <<'EOF' +Review for SCOPE alignment and unjustified complexity: abstractions/frameworks larger than the goal +needs, scope creep beyond stated intent, premature generality, dependencies declared but not needed. +Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/product.md" <<'EOF' +Review as a senior PRODUCT leader: are the premises sound? What strategic/adoption/trust +consequences (including for the people the system affects) does this carry even if the premise +holds? Where does the work drift from the goal? Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/adversarial.md" <<'EOF' +ADVERSARIALLY stress-test this document: surface unstated assumptions, construct failure modes the +mitigations do not actually cover, name the cheaper/safer alternative it dismissed, and find any +irreversible step taken before its validation. Try to BREAK it. Return your findings as a JSON array of strings, one element per distinct finding. +EOF + +run() { + cli="$1"; lens="$2" + if ! command -v "$cli" >/dev/null 2>&1; then + printf ' [%-7s %-12s] SKIP — %s not installed\n' "$cli" "$lens" "$cli"; return + fi + if [ "$cli" = "agy" ] && [ "$(uname -s)" != "Darwin" ]; then + printf ' [%-7s %-12s] SKIP — agy is macOS-only (read-only floor is a seatbelt)\n' "$cli" "$lens"; return + fi + cmd=(run-arm "$arm" "$cli" "$plan" "$lens_dir/$lens.md" --doc-id "${doc_id}__${lens}" --trial 1 --timeout "$timeout") + [ -n "$context" ] && cmd+=(--context "$context") + rec="$(python3 "$arms" "${cmd[@]}" 2>/dev/null)" + printf '%s' "$rec" > "$rec_dir/${cli}__${lens}.json" + n="$(printf '%s' "$rec" | python3 -c 'import json,sys +try: + print(len(json.load(sys.stdin)["findings"])) +except Exception: + print("ERR")')" + printf ' [%-7s %-12s] findings=%s\n' "$cli" "$lens" "$n" +} + +echo "Panel critique of: $plan (arm=$arm)" +echo "Full records -> $rec_dir" +echo "Models: $models (each runs all 6 lenses; models run in parallel — progress lines interleave)" + +# One background subshell PER MODEL, each running the six lenses sequentially. Parallelizing across +# models (not across lenses) overlaps the slow arms while bounding concurrency to the model count -- +# at most one in-flight request per vendor, which avoids rate-limit / resource contention. Each +# (model, lens) cell streams its own self-labeled progress line as it completes (R15: no silent +# multi-minute runs); lines from different models interleave, which is fine. Records key on +# ${cli}__${lens}.json, so parallel writers never collide. +run_model() { + cli="$1" + for lens in coherence feasibility security scope product adversarial; do + run "$cli" "$lens" + done +} +pids="" +for cli in $models; do + run_model "$cli" & + pids="$pids $!" +done +for pid in $pids; do wait "$pid"; done + +echo "" +echo "DONE. Full records in $rec_dir — read them for the per-lens findings." diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/reconcile.py b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/reconcile.py new file mode 100644 index 000000000..a72a78866 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/reconcile.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""ce-deep-review-beta — reconciliation helpers for the verified sidecar (RU5). + +Two deterministic, side-effect-scoped helpers the skill calls when promoting verdict-tagged findings +into the reserved verified sidecar `<plan>.deep-review.md`: + + rotate <sidecar.md> [--now <ISO>] [--keep N] + Rotate an existing verified sidecar out of the way before writing a fresh one, and prune old + rotations. If <sidecar.md> exists, rename it to `<plan>.deep-review.<ISO>.md`, then keep only + the N newest rotations (by the ISO infix in the name) and delete the rest. + + render-cross-model <verify-records.json> + Emit the by-lens-grouped, verdict-tagged Markdown section for the cross-model findings, from + verify-findings.py's `verify-records` output. Deterministic: same input -> same Markdown. + +ROTATION SAFETY (this is the data-loss-risk surface the feasibility review flagged): the prune step +matches ONLY rotation files `<plan>.deep-review.<infix>.md`. It can never match the canonical base +`<plan>.deep-review.md` (no infix) nor the thin-slice draft `<plan>.deep-review-draft.md` (a +`-draft` infix, not a `.`-delimited one), so neither is ever deleted here. +""" + +import argparse +import glob +import json +import os +import sys +from datetime import datetime, timezone +from pathlib import Path + +LENSES = ["coherence", "feasibility", "security", "scope", "product", "adversarial"] +VERDICT_ORDER = ["CONFIRMED", "NOT-FOUND-IN-DOC", "NEEDS-HUMAN"] +_SUFFIX = ".deep-review.md" + + +def _utc_stamp(): + # Filesystem-safe UTC stamp, lexicographically sortable: 2026-05-29T024500Z + return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H%M%SZ") + + +def rotate(sidecar_path, now=None, keep=5): + """Rotate an existing verified sidecar and prune to the `keep` newest rotations. + + Returns the rotated path (or None if there was nothing to rotate). Only touches rotation files; + never the base sidecar (just renamed away) nor the `-draft` sidecar. + """ + if not sidecar_path.endswith(_SUFFIX): + raise ValueError(f"expected a path ending in '{_SUFFIX}', got: {sidecar_path}") + stamp = now or _utc_stamp() + prefix = sidecar_path[: -len(".md")] # "<plan>.deep-review" + + rotated = None + if os.path.exists(sidecar_path): + # Collision-safe: a re-run within the same second (or an explicit duplicate --now) yields the + # same stamp. Never os.rename over an existing rotation -- that would silently drop a prior + # snapshot before pruning runs, violating the data-loss-safe contract. Disambiguate with a + # numeric suffix; "<stamp>-1" sorts after "<stamp>" so keep-N (newest-first) ordering holds. + rotated = f"{prefix}.{stamp}.md" + n = 1 + while os.path.exists(rotated): + rotated = f"{prefix}.{stamp}-{n}.md" + n += 1 + os.rename(sidecar_path, rotated) + + # Rotation files only: "<plan>.deep-review.<infix>.md". The glob's required "." after the prefix + # excludes both the base ("<plan>.deep-review.md") and the draft ("<plan>.deep-review-draft.md"). + rotations = glob.glob(f"{prefix}.*.md") + # Sort by the ISO infix (newest first); the infix is sortable as a string. + def infix(p): + return os.path.basename(p)[len(os.path.basename(prefix)) + 1 : -len(".md")] + rotations.sort(key=infix, reverse=True) + pruned = [] + for old in rotations[keep:]: + os.remove(old) + pruned.append(old) + return {"rotated": rotated, "kept": rotations[:keep], "pruned": pruned} + + +def render_cross_model(verified): + """Markdown for the cross-model section, grouped by lens, tagged by verdict. `verified` is the + list under verify-records' `verified` key. Deterministic ordering: lens, then verdict, then the + order findings appear in the input.""" + by_lens = {ln: [] for ln in LENSES} + extra = {} + for row in verified: + ln = row.get("lens", "") + (by_lens if ln in by_lens else extra).setdefault(ln, []).append(row) + + out = [] + for ln in LENSES + sorted(extra): + rows = by_lens.get(ln) or extra.get(ln) + if not rows: + continue + out.append(f"### {ln.capitalize()}") + out.append("") + for verdict in VERDICT_ORDER: + group = [r for r in rows if r.get("verdict") == verdict] + for r in group: + model = r.get("model", "?") + text = (r.get("text", "") or "").strip().replace("\n", " ") + line = f"- **[{verdict}]** ({model}) {text}" + if verdict == "CONFIRMED" and r.get("grounding_quote"): + line += f' \n ↳ grounding quote: "{r["grounding_quote"]}"' + out.append(line) + out.append("") + return "\n".join(out).rstrip() + "\n" + + +def main(argv=None): + p = argparse.ArgumentParser(description="ce-deep-review reconciliation helpers (RU5).") + sub = p.add_subparsers(dest="cmd", required=True) + + r = sub.add_parser("rotate", help="rotate an existing verified sidecar; prune to N newest") + r.add_argument("sidecar") + r.add_argument("--now", default=None, help="ISO stamp for the rotation (default: current UTC)") + r.add_argument("--keep", type=int, default=5) + + rc = sub.add_parser("render-cross-model", help="render the by-lens verdict-tagged section") + rc.add_argument("verify_records_json") + + args = p.parse_args(argv) + + if args.cmd == "rotate": + print(json.dumps(rotate(args.sidecar, now=args.now, keep=args.keep))) + return 0 + + if args.cmd == "render-cross-model": + data = json.loads(Path(args.verify_records_json).read_text()) + sys.stdout.write(render_cross_model(data.get("verified", []))) + return 0 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/validation/agy-readonly.sb.tmpl b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/validation/agy-readonly.sb.tmpl new file mode 100644 index 000000000..1e160038d --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/validation/agy-readonly.sb.tmpl @@ -0,0 +1,21 @@ +;; ce-deep-review — agy cross-model arm filesystem floor (validated 2026-05-28, Phase 0 / U2). +;; +;; agy 1.0.3 is a heavy Electron/Node agent: a strict `(deny file-write*)` allowlist OR any +;; `(deny file-read* ...)` rule makes it HANG indefinitely (it retries the denied syscall and +;; ignores its own --print-timeout). So this is a deny-WRITE-only denylist: allow-default (agy +;; writes its own state/caches freely -> no hang) and deny writes only to the paths that matter +;; (the repo under review + home credentials/dotfiles). Network is allowed (the arm needs its +;; vendor API). Reads are NOT denied -> secret-read-exfil is a documented residual, mitigated by +;; a clean cwd + a review-only prompt; only a concern for untrusted docs. +;; +;; __REPO_DIR__ and __HOME__ are substituted at runtime (see agy-smoke.sh / the harness). +(version 1) +(allow default) +(deny file-write* + (subpath "__REPO_DIR__") + (subpath "__HOME__/.ssh") + (subpath "__HOME__/.aws") + (subpath "__HOME__/.config/gcloud") + (literal "__HOME__/.zshrc") + (literal "__HOME__/.gitconfig") + (literal "__HOME__/.netrc")) diff --git a/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verify-findings.py b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verify-findings.py new file mode 100644 index 000000000..4aedda890 --- /dev/null +++ b/plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verify-findings.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""ce-deep-review-beta — deterministic quote-grep verification backstop (RU4). + +Grounds each raw cross-model finding against the reviewed document and assigns ONE verdict: + + CONFIRMED - the finding embeds a substantial verbatim quote that DOES appear in the doc. + NOT-FOUND-IN-DOC - the finding embeds a substantial quote that does NOT appear in the doc + (the model claimed doc text that isn't there -- likely fabricated/paraphrased). + NEEDS-HUMAN - the finding has no substantial verbatim quote to check (paraphrase, or a + cross-section implication). The backstop cannot auto-ground it; a human decides. + +This is the AUTHORITATIVE gate (R: "the verifier can inherit model contamination unless the +synchronous quote-grep backstop is the authoritative gate"). It is deterministic: same finding + +same doc -> same verdict, with no model in the loop, so it is inherently **blind to the producing +model** -- the verdict NEVER reads the model label. `verify-records` re-attaches the model only to +LABEL output rows, never to compute a verdict. + +CONFIRMED means "the quoted evidence exists in the document" -- NOT "the finding is correct or +important." Claim correctness/severity is out of scope for a quote-grep; that is the human's call. + +A substantial quote = a quoted span (", '', ``, or smart quotes) that, normalized, is >= MIN_QUOTE +chars AND contains a space (a phrase, not a lone identifier/filename -- those match too trivially). +""" + +import argparse +import json +import os +import re +import sys +import unicodedata +from pathlib import Path + +MIN_QUOTE = 12 # normalized chars; below this a "quote" is too short to ground a claim + +# Quoted spans the models actually emit: straight/smart double quotes, single quotes, backticks. +_QUOTE_PATTERNS = [ + r'"([^"]+)"', # straight double + r"“([^”]+)”", # smart double " " + r"`([^`]+)`", # backtick + r"'([^']+)'", # straight single (also catches apostrophes; length+space filter prunes) +] + + +def normalize(s): + """Lowercase, fold smart punctuation to ASCII, strip markdown emphasis, collapse whitespace. Used + for both doc and quote so format-only differences (smart quotes, em-dashes, *emphasis*, wrapping) + do not cause false NOT-FOUND.""" + if not s: + return "" + s = unicodedata.normalize("NFKC", s) + # Fold curly quotes/dashes to ASCII so a doc em-dash matches a finding hyphen, etc. + trans = { + "“": '"', "”": '"', "‘": "'", "’": "'", + "–": "-", "—": "-", "−": "-", " ": " ", + } + s = s.translate(str.maketrans(trans)) + # Strip markdown emphasis markers (*italic*, **bold**, _italic_, __bold__): a model quotes the + # emphasized text WITHOUT the markers, so a doc "the order *is* the container" must still match a + # finding that quotes "the order is the container". The markers carry no content inside a prose + # quote. Safe against snake_case false-merges: removal inserts no space, so "market_id" -> + # "marketid" only collapses to a match when BOTH doc and quote carry the underscore (a true verbatim + # quote); a spaced paraphrase "market id" keeps its space and still will not match "marketid". + s = s.replace("*", "").replace("_", "") + s = s.lower() + s = re.sub(r"\s+", " ", s) + return s.strip() + + +def extract_quotes(text): + """All quoted spans in the finding text (raw, de-duplicated, order-preserved).""" + out = [] + seen = set() + for pat in _QUOTE_PATTERNS: + for m in re.finditer(pat, text): + span = m.group(1).strip() + if span and span not in seen: + seen.add(span) + out.append(span) + return out + + +def candidate_quotes(text): + """Quotes substantial enough to ground a claim: normalized length >= MIN_QUOTE AND multi-word + (contains a space). A lone `panel-critique.sh` or "agy" is too trivial to confirm anything.""" + cands = [] + for q in extract_quotes(text): + n = normalize(q) + if len(n) >= MIN_QUOTE and " " in n: + cands.append(q) + return cands + + +def verify_one(finding_text, doc_norm): + """Return (verdict, grounding_quote_or_None). Pure function of (finding text, normalized doc) -- + no model label is consulted, so the verdict is blind to the producing model.""" + cands = candidate_quotes(finding_text) + if not cands: + return "NEEDS-HUMAN", None + for q in cands: + if normalize(q) in doc_norm: + return "CONFIRMED", q + return "NOT-FOUND-IN-DOC", None + + +def doc_id_for(doc_path): + """The current run's doc_id base, matching panel-critique.sh's `basename <plan> .md`. Records + written by the panel store doc_id = `<this base>__<lens>`; verify-records uses it to skip stale + records left in a reused CMRE_OUT_DIR by a DIFFERENT plan.""" + base = os.path.basename(doc_path) + if base.endswith(".md"): + base = base[:-3] + return base + + +def _iter_records(records_dir, doc_id_base): + """Yield (model, lens, finding_dict) for every record file <cli>__<lens>.json in the dir whose + record belongs to the CURRENT plan. The default CMRE_OUT_DIR (/tmp/cmre-panel/records) is reused + across runs, so a record from a different plan can linger; verifying its findings against THIS doc + would publish another plan's review into this sidecar. Each record stores doc_id = `<base>__<lens>` + (arms.py, via panel-critique.sh's `--doc-id "${doc_id}__${lens}"`); skip any record whose stored + doc_id doesn't match the current plan's `<doc_id_base>__<lens>`. A record missing doc_id entirely + is kept (can't prove it's stale; preserves pre-doc_id records and hand-built fixtures).""" + for fn in sorted(os.listdir(records_dir)): + if not fn.endswith(".json"): + continue + stem = fn[:-5] + model, _, lens = stem.partition("__") + try: + rec = json.load(open(os.path.join(records_dir, fn))) + except (json.JSONDecodeError, OSError): + continue + rec_doc_id = rec.get("doc_id") + if rec_doc_id is not None and rec_doc_id != f"{doc_id_base}__{lens}": + continue # stale record from another plan (or another lens) in a reused dir + for f in rec.get("findings", []): + yield model, lens, f + + +def main(argv=None): + p = argparse.ArgumentParser(description="ce-deep-review verification backstop (quote-grep).") + sub = p.add_subparsers(dest="cmd", required=True) + + one = sub.add_parser("verify-one", help="verify a single finding string against a doc") + one.add_argument("doc") + one.add_argument("finding") + + rec = sub.add_parser("verify-records", help="verify every finding in a records dir against a doc") + rec.add_argument("doc") + rec.add_argument("records_dir") + + meas = sub.add_parser("measure", help="measure false-CONFIRM / false-NOT-FOUND rates on a labeled corpus") + meas.add_argument("corpus_json") + + args = p.parse_args(argv) + + if args.cmd == "verify-one": + doc_norm = normalize(Path(args.doc).read_text()) + verdict, quote = verify_one(args.finding, doc_norm) + print(json.dumps({"verdict": verdict, "grounding_quote": quote})) + return 0 + + if args.cmd == "verify-records": + doc_norm = normalize(Path(args.doc).read_text()) + doc_id_base = doc_id_for(args.doc) # skip stale records from another plan in a reused dir + rows = [] + counts = {"CONFIRMED": 0, "NOT-FOUND-IN-DOC": 0, "NEEDS-HUMAN": 0} + for model, lens, f in _iter_records(args.records_dir, doc_id_base): + text = f.get("text", "") + verdict, quote = verify_one(text, doc_norm) # blind: model/lens not passed in + counts[verdict] += 1 + rows.append({ + "model": model, "lens": lens, "id": f.get("id", ""), + "text": text, "verdict": verdict, "grounding_quote": quote, + }) + print(json.dumps({"verified": rows, "counts": counts})) + return 0 + + if args.cmd == "measure": + # RU6b verifier-rate measurement on a labeled corpus. Deterministic + model-blind, so N=1 + # (no trials) and no voice sampling: the verdict is a pure function of (text, doc). + corpus = json.loads(Path(args.corpus_json).read_text()) + doc_norm = normalize(corpus["document"]) + grounded = confab = false_not_found = false_confirm = 0 + detail = [] + for it in corpus["items"]: + verdict, _ = verify_one(it["text"], doc_norm) + exp = it["expected"] + if exp == "CONFIRMED": + grounded += 1 + if verdict != "CONFIRMED": # a grounded finding the backstop failed to confirm + false_not_found += 1 + elif exp == "NOT-FOUND-IN-DOC": + confab += 1 + if verdict == "CONFIRMED": # a fabricated quote the backstop wrongly confirmed + false_confirm += 1 + detail.append({"id": it.get("id"), "expected": exp, "got": verdict}) + fcr = (false_confirm / confab) if confab else 0.0 + fnr = (false_not_found / grounded) if grounded else 0.0 + print(json.dumps({ + "n": len(corpus["items"]), "grounded": grounded, "confabulated": confab, + "false_confirm_rate": round(fcr, 4), "false_not_found_rate": round(fnr, 4), + "eligible": fcr <= 0.05 and fnr <= 0.05, "detail": detail, + })) + return 0 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/eval/cross_model_review/README.md b/scripts/eval/cross_model_review/README.md new file mode 100644 index 000000000..460d2a91a --- /dev/null +++ b/scripts/eval/cross_model_review/README.md @@ -0,0 +1,282 @@ +# Cross-Model Review Evaluation Harness + +A repeatable, four-arm evaluation that decides whether — and which — review-improvement +lever is worth building, **before** any cross-model review machinery ships. It is a +decision tool, not a shipped feature. + +Origin requirements: `docs/brainstorms/2026-05-24-multi-model-plan-review-requirements.md` +Plan: `docs/plans/2026-05-24-001-feat-cross-model-review-eval-plan.md` + +## The four arms + +| Arm | What it is | Hypothesis it isolates | +|-----|------------|------------------------| +| `a_baseline` | Claude-only review (current `ce-doc-review`) | the control | +| `b_isolated` | cross-model CLI, run isolated from the repo (no workspace context) | does a different model add unique value with no context? | +| `c_fixed_context` | cross-model CLI + a fixed, documented repo-context set | is context-poverty (not the model) the limiting factor? | +| `d_self_critic` | Claude re-reviews in-process, own failure modes supplied, prior output hidden | is the gain the model, or just a fresh adversarial pass? | + +Arms `b`/`c` shell out to external model CLIs (run by `run_arms.py`). Validated CLIs: +`codex` (OpenAI, reliable) and `gemini` 0.43+ (Google; requires `GEMINI_API_KEY` and is +invoked with `--approval-mode plan --skip-trust` for headless read-only review). **`agy` +(Antigravity) is deprecated** — it proved unreliable as a non-interactive reviewer (hangs or +returns empty; see `docs/solutions/skill-design/cross-model-eval-first-run-2026-05-25.md`). +Defaults: arm `b` = `codex` isolated, arm `c` = `gemini` + context (override with +`--cli-b`/`--cli-c`). Arms `a`/`d` and the judge are produced by the **orchestrator** via +in-process subagent dispatch — there is no `claude -p` and arm `d` performs no document egress. + +## How the two halves cooperate (the record-store seam) + +Both producers write **schema-conformant record files** into a single shared run +directory (an OS-temp dir created by `run_arms.py`). Neither half writes into the other's +memory: + +- `run_arms.py` spawns the CLI arms (`b`, `c`) directly and writes their records. +- The orchestrator dispatches the in-process arms (`a`, `d`) and the judge, then writes + each result as a record file into the same run directory (or via `run_arms.py ingest`). +- Aggregation (`run_arms.py`) pools by reading **all** record files in the run dir, + regardless of which producer wrote them. + +The per-arm timeout and circuit breaker apply **only** to the CLI arms `run_arms.py` +spawns. In-process arm records are ingested as-is. + +See `record-schema.json` for the canonical record contract both producers must satisfy. + +## Pre-registration (required before any arm runs) + +Editing `tests/fixtures/cross-model-review/corpus-manifest.json`, commit these values +**before** running so the decision rule is independent of the observed counts (R9): + +- `go_threshold` — the per-arm count of confirmed unique decision-changing findings (on + the known-failure subset) that justifies building a lever. +- `minimum_corpus_n` — the smallest corpus the run is allowed to draw a conclusion from. + A run below this N reports **inconclusive**, never "build nothing". +- `trials_per_arm` — number of trials per (document × arm). Floor is **3**; model arms are + non-deterministic and a single trial produces confidently-wrong, reversed conclusions + (see `docs/solutions/skill-design/safe-auto-rubric-calibration-2026-04-25.md`). +- `arm_c_context_rule` — the fixed, documented context set arm `c` receives, applied + identically to every document. This is the experimental control for the model-vs-context + comparison; it must be defined before running, not curated per-document. + +The corpus list itself (the `docs` array) is also filled at this step. The committed file +in this repo is a **schema stub** with placeholder values and one example entry per +subset; a run replaces them with the real corpus and pre-registered values. + +## Running + +Per (document × trial), produce one schema-conformant record per arm into a shared run +dir, then pool, judge, and aggregate. + +**CLI arms (b, c)** — `arms.py` runs the external model and emits a record on stdout (both +run from a clean CWD with auth preserved; arm b has no context, arm c adds the fixed +`--context` set): + +``` +python3 arms.py run-arm b_isolated codex <doc> <rubric> --doc-id <id> --trial <n> > rec.json +python3 arms.py run-arm c_fixed_context gemini <doc> <rubric> --context <ctx> --doc-id <id> --trial <n> > rec.json +python3 run_arms.py ingest <run_dir> rec.json +``` + +**In-process arms (a, d) and the judge** — produced by the orchestrator via subagent +dispatch (see `prompts/baseline.md`, `prompts/self-critic.md`, `judge_rubric.md`), each +written as a record and ingested into the same run dir. + +**Then** pool and decide: + +``` +python3 run_arms.py pool <run_dir> +python3 run_arms.py aggregate <scored.json> <manifest.json> +``` + +The decision artifact is written under `docs/` from `decision-artifact-template.md`. + +### Quick one-off critique (turnkey) + +To get cross-model critiques of a single document without the full eval, use the wrapper — +it runs `codex` and `agy` as isolated reviewers and prints each model's findings: + +``` +bash scripts/eval/cross_model_review/critique.sh <plan.md> [rubric.md] [context.md] +``` + +A built-in rubric is used if none is given; pass a `context.md` to switch the arms to the +fixed-context variant. Override the per-arm timeout with `CMRE_TIMEOUT=<seconds>` (agy can +be slow). A missing/unauthenticated CLI is skipped, not fatal. Each run sends the document +to that vendor (codex -> OpenAI, agy -> Google). + +## Outcomes + +The decision artifact records exactly one of: + +- **build `<arm>`** — a lever cleared the pre-registered threshold; the winning arm shapes + the deferred build. +- **build nothing** — corpus met `minimum_corpus_n` and no arm cleared the threshold. +- **inconclusive / underpowered** — corpus below `minimum_corpus_n`, or the blind-integrity + check came back confounded; re-run larger / with a different judge. + +## Building a known-bug corpus (code-review breakpoint) + +The four-arm eval transfers from plan review to **code review** with the same harness — +swap the document (a plan -> a diff), the rubric, and arms `a`/`d` (`ce-doc-review` -> +`ce-code-review`). What makes code review *more* evaluable than plan review is ground +truth: git history is a factory of changes the project itself later judged wrong, so the +known-failure subset (R7) can be sourced automatically instead of hand-curated. + +`build_corpus.py` mines a repo for those, in descending attribution strength: + +| Tier | Signal | `attribution` | `trust` | +|------|--------|---------------|---------| +| 1 | a revert (the team's own verdict) | `revert` | `high` | +| 2 | a fix subject that names what broke | `named_regression` | `high` | +| 3 | a fix whose touched lines blame back to a recent change | `blame` | `needs_confirmation` | + +Each emitted entry extends the manifest's `known_failure` shape with a `ground_truth` +block — the bug a reviewer should have caught — so the judge can score a **targeted +hit/miss** per document (did any pooled finding describe the bug the fix proved mattered?) +rather than only forward-rating actionability. + +``` +# Tier-1: discover reverts, materialize each culprit diff as a reviewable document +python3 build_corpus.py scan --repo <path-to-target-repo> --out-dir <corpus-dir> + +# Tier-3: walk every code `fix:` commit, blame it to a culprit, emit needs_confirmation entries +# The quality gate drops oversize/foundational culprits and dedups fixes that share one. +python3 build_corpus.py scan-fixes --repo <path-to-target-repo> --out-dir <corpus-dir> \ + --max-culprit-lines 2000 --max-culprit-files 30 # defaults; tighter = cleaner but smaller + +# Tier-2/3 (single fix): blame the lines a known fix touched to find candidate culprits +python3 build_corpus.py attribute-fix --repo <path> <fix-sha> +``` + +Both `scan` and `scan-fixes` emit `{entries, stats}`; every entry passes `validate-entry` +(the manifest conformance gate, the corpus analog of `validate-record`). `scan-fixes` +keys one entry per fix, blames code files only (`is-code-path` filters out docs/markdown), +picks the most-files-touched culprit, and keeps the runners-up in `culprit_alternates` for +the human to confirm (R6). Its **quality gate** then drops culprits whose diff exceeds +`--max-culprit-lines`/`--max-culprit-files` (too large to review, or a foundational/import +commit) and dedups fixes that blame the same culprit (so N fixes touching one feature don't +become N non-independent docs). On a repo that ships large feature commits this is decisive: +an ungated run produced a corpus that collapsed to a handful of distinct, often huge diffs, +while the gated run yields a smaller set of distinct, reviewable culprits — a corpus you can +actually decide on. Tighter caps trade corpus size for cleanliness. A conventional `revert:` with no embedded SHA is **counted but +not emitted** by `scan` — there is no reliable culprit diff — and likewise left to the human. + +**Which tier a repo yields depends on how the team works.** + +- This plugin's own history: ~5 Tier-1 items — well under any decidable N. Methodology + transfers; the sample is too small. +- A team that doesn't use `git revert` produces **zero usable Tier-1 items** (its reverts + are conventional, SHA-less, often content reverts). Its ground truth lives in Tier-3: + walking its `fix:` commits with `scan-fixes` yields a large corpus (e.g. ~180–200 unique + known-failure candidates from ~200 fixes), with real latency (`surfaced_after_days` up to + weeks) — exactly the discovered-late signal review is meant to catch. + +Below a pre-registered `minimum_corpus_n` the eval reports `inconclusive` (R9), never a +false "build nothing". + +**Corpus hygiene.** `--all` scans every ref and so double-counts a fix that appears as both +a branch commit and a squashed merge; the default (HEAD history) avoids most of that. Tier-3 +entries are `trust: needs_confirmation` by design — blame is inferred, and `fix:` subjects +include renames and test-only fixes — so the human-confirmed subset, not the raw emission, +is the known-failure set the decision rests on (R6/R7). + +### Scoring a code-review corpus: GT-match + +Plan review can only forward-rate whether a finding *looks* decision-changing. A known-bug +corpus has a target — the bug the fix proved mattered — so the known-failure metric becomes +an objective **hit/miss**: did any of an arm's findings describe that bug? See +`gt_match_rubric.md`. + +The judge stays blind (per-finding, label-stripped, never told the arm); the arm is +re-attached afterward, so blinding holds: + +``` +# judge produces per-finding verdicts {uid, matches_bug} on the blind pool +python3 run_arms.py gt-pool <records.json> > pool.json # -> blind pool + provenance +python3 run_arms.py gt-resolve <(jq .provenance pool.json) <verdicts.json> # -> per-(arm,doc) gt_hit +python3 run_arms.py gt-score <manifest.json> <arm-matches.json> # -> per-arm known-failure hits +``` + +`aggregate` uses `gt_hit` as the known-failure predicate when present, falling back to +`decision_changing` for plan-review corpora — so the same three-way decision rule serves +both breakpoints. Forward-rated and negative-control documents keep `decision_changing`. + +### End-to-end (the `drive_eval.py` spine) + +`drive_eval.py` wires the deterministic flow and makes the orchestrator handoff explicit. +It cannot run arms `a`/`d` or the judge (model-driven, no `claude -p`); it runs the spine +and consumes their outputs. + +``` +# 1. build a corpus and assemble a manifest skeleton (pre_registration left null) +python3 build_corpus.py scan-fixes --repo <repo> --out-dir corpus/ > sf.json +python3 build_corpus.py to-manifest sf.json > manifest.json + +# 2. HUMAN: confirm needs_confirmation entries, add negative_control + forward_rated docs, +# and FILL pre_registration (go_threshold / minimum_corpus_n / trials_per_arm). (R6, R9) + +# 3. plan -- enumerates arm x doc x trial work, emits the CLI-arm commands + the +# in-process/judge todo, and writes run-state.json. Refuses if pre-reg is unset. +python3 drive_eval.py plan manifest.json --out-dir run/ --rubric code-review-rubric.md --context arm-c-context.md + +# 4. ORCHESTRATOR: run the emitted CLI-arm (b, c) commands and ingest; dispatch arms a/d +# in-process and ingest; run the judge (gt_match_rubric.md / judge_rubric.md) -> verdict files. + +# 5. finalize -- gt-resolve -> gt-score -> aggregate -> decision artifact under docs/ +python3 drive_eval.py finalize run/records manifest.json \ + --gt-verdicts gt-verdicts.json [--class-verdicts class-verdicts.json] \ + [--integrity <correct>,<total>] --judge-family <family> --out docs/<decision-record>.md +``` + +`plan` enforces R9 (no run without a pre-registered threshold/N); `finalize` forces +`inconclusive` if the blind-integrity probe comes back confounded, below minimum N, or the +negative control moves — the same guards the manual chain applies. + +### Decision-grade run (the full procedure) + +A decision-grade verdict requires all four guards on at once: a **non-Claude judge** (clears +the family confound, R4), **`trials_per_arm >= 3`** (model arms are non-deterministic; a +single trial gives confidently-wrong, reversed conclusions), a **human-confirmed known-failure +corpus** at or above a pre-registered `minimum_corpus_n`, and **precision-weighting** (the +negative control + spot-verification, so raw yield isn't rewarded for confabulation). + +Pre-register in the manifest **before running** (R9): `go_threshold`, `minimum_corpus_n` +(set it to the real power floor, not the corpus you happen to have — a smaller corpus then +correctly reports `inconclusive / underpowered` rather than a false confident call), +`trials_per_arm: 3`, `arm_c_context_rule`, and the judge's model family (must not be Claude). + +Ordered steps: + +``` +# 1. corpus (git-only, no egress): build + assemble + HUMAN-CONFIRM the known-failure subset +python3 build_corpus.py scan-fixes --repo <repo> --out-dir corpus/ > sf.json +python3 build_corpus.py to-manifest sf.json > manifest.json # then fill pre_registration + confirm + +# 2. Claude arms a/d, 3 trials each (orchestrator, in-process, no egress) +# 3. cross-model arms b/c, 3 trials each (EGRESS — codex + gemini) +python3 arms.py run-arm b_isolated codex <doc> <rubric> --doc-id <id> --trial <1..3> | python3 run_arms.py ingest <run>/records - +python3 arms.py run-arm c_fixed_context gemini <doc> <rubric> --context <ctx> --doc-id <id> --trial <1..3> | python3 run_arms.py ingest <run>/records - + +# 4. pool, then the NON-CLAUDE judge over the blind pool (EGRESS) +python3 run_arms.py gt-pool <run>/records/*.json > pool.json # (glob-collect records into one array) +bash run-judge.sh pool.json manifest.json codex verdicts.json # judge family != Claude + +# 5. resolve (blind), score, decide +python3 run_arms.py gt-resolve <(jq .provenance pool.json) verdicts.json > arm-matches.json +python3 run_arms.py gt-score manifest.json arm-matches.json # per-arm known-failure hits +python3 run_arms.py yield-score <(jq .provenance pool.json) verdicts.json # precision-weight this +python3 drive_eval.py finalize <run>/records manifest.json --gt-verdicts verdicts.json \ + --yield-verdicts verdicts.json --integrity <correct>,<total> --judge-family <non-claude> --out docs/<decision>.md +``` + +**Power note:** the gated code corpus typically yields ~10 confirmed known-failure items — +enough for a *directional* read (does the cross-model arm hit bugs the Claude arms miss, under +a fair non-Claude judge), but below the `minimum_corpus_n` a *confident* build/kill needs. Set +`minimum_corpus_n` honestly so the harness reports `inconclusive / underpowered` until the +confirmed corpus is large enough — that honesty is the point (R9). + +The model arms (`b`/`c`) review a diff exactly as they review a plan — it is text on stdin +— so `arms.py` is unchanged. Note that arm `b` (isolated, no repo) is more crippled by a +raw diff than by a self-contained plan; pre-register that as an expected effect so a +near-zero arm-`b` yield reads as "no-context code review is the floor", not "cross-model +adds nothing". diff --git a/scripts/eval/cross_model_review/arms.py b/scripts/eval/cross_model_review/arms.py new file mode 100644 index 000000000..66473fabe --- /dev/null +++ b/scripts/eval/cross_model_review/arms.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Cross-model review eval — CLI arms b (isolated) and c (fixed context) (U3). + +Invokes the external model CLIs (codex, agy) over the document via stdin using +argv lists — never string interpolation into a shell — so document content +cannot inject commands (R2 / R14). + +Arm b is run isolated from the repo (clean cwd + HOME/config overrides) so the +model genuinely has no workspace context; arm c additionally supplies a fixed +context set. Because "isolation flags are present" is not the same as "the model +had no context", U3 includes a positive isolation PROBE (AD2 / P1): plant a +sentinel only reachable from repo/global config and assert arm b cannot surface +it. The probe's leak-detection logic is unit-tested here; the live subprocess +runs are integration-level (validated at eval time). +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +# Validated invocation forms: +# codex 0.133.0: codex exec -s read-only --skip-git-repo-check - (prompt via stdin) +# gemini 0.43.0: gemini -p "<instruction>" --approval-mode plan --skip-trust -o text +# (-p is appended to stdin; plan = read-only so the arm never edits; +# --skip-trust is REQUIRED for headless runs in a clean/untrusted CWD, +# else gemini exits 55 "not running in a trusted directory") +# Both arms run from a clean temp CWD (no ambient workspace access). codex needs +# --skip-git-repo-check or it refuses with "Not inside a trusted directory". +# +# NOTE: agy 1.0.2 (--print) was UNRELIABLE (empty output) and got dropped — but agy 1.0.3 is a +# VIABLE non-interactive reviewer (clean JSON via --print + stdin). Its own --sandbox does NOT +# confine the filesystem, so on macOS the agy arm is wrapped in a seatbelt deny-write profile +# (allow-default + deny writes to repo/home-creds/dotfiles; a strict deny-all-write OR any +# deny-read HANGS agy). Auth is OAuth at ~/.gemini/oauth_creds.json (+ refresh_token; do NOT gate +# detection on expiry — agy auto-refreshes). See +# docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md (Phase 0 / U2). +CODEX_BASE = ["codex", "exec", "-s", "read-only", "--skip-git-repo-check", "-"] +AGY_INSTRUCTION = "Review the document provided on stdin. Return ONLY a JSON array of finding strings (one element per distinct finding), no prose or preamble." +GEMINI_INSTRUCTION = "Review the document provided on stdin. Do not modify files. Return ONLY a JSON array of finding strings (one element per distinct finding), no prose or preamble." +GEMINI_BASE = ["gemini", "-p", GEMINI_INSTRUCTION, "--approval-mode", "plan", "--skip-trust", "-o", "text"] + +# Lines like "1. foo" or "2) bar" — numbered findings the model commonly emits. +NUMBERED_ITEM = re.compile(r"^\s*\d+[.)]\s+(.*)$") + + +def _repo_root(): + """Repo whose writes the agy seatbelt floor denies. + + Honors CMRE_REPO_DIR when set — the REVIEWED document's repo, passed by the caller. The + installed skill reviews a user's plan, where arms.py's own location is NOT the right repo to + protect; panel-critique.sh exports CMRE_REPO_DIR from the plan's directory. Falls back to + arms.py's canonical in-repo location (scripts/eval/cross_model_review/arms.py) for the + eval-harness case where no caller supplies it. + """ + env = os.environ.get("CMRE_REPO_DIR") + if env: + return os.path.realpath(env) + return os.path.realpath(os.path.join(os.path.dirname(__file__), "..", "..", "..")) + + +def agy_sandbox_prefix(): + """macOS seatbelt wrapper for the agy arm (Phase 0 / U2, validated 2026-05-28). + + Returns (argv_prefix, profile_path). On macOS, generates a concrete deny-write seatbelt + profile from validation/agy-readonly.sb.tmpl (deny writes to the repo + home creds/dotfiles; + allow-default otherwise — agy HANGS under deny-all-write or any deny-read) and returns + (["sandbox-exec", "-f", <profile>], <profile>). On non-macOS, returns ([], None): seatbelt is + macOS-only and the FS floor is not enforced there (documented limitation). The caller unlinks + the profile after the run. + """ + if sys.platform != "darwin": + return [], None + tmpl = os.path.join(os.path.dirname(__file__), "validation", "agy-readonly.sb.tmpl") + if not os.path.exists(tmpl): + return [], None + # Paths must be canonical — seatbelt matches /private/var..., and /Users is already canonical. + profile_text = ( + Path(tmpl).read_text() + .replace("__REPO_DIR__", _repo_root()) + .replace("__HOME__", os.path.realpath(os.path.expanduser("~"))) + ) + fd, path = tempfile.mkstemp(prefix="cmre-agy-sb-", suffix=".sb") + with os.fdopen(fd, "w") as f: + f.write(profile_text) + return ["sandbox-exec", "-f", path], path + + +def clean_cwd(): + """A fresh temp CWD with no ambient repo access. + + Both arms run from here so neither inherits the repo's workspace context + (codex's AGENTS.md walk-up and git-repo discovery both start from CWD). HOME + is deliberately NOT overridden — that would strip the CLI's auth (found via + live smoke). The global config under HOME (~/.codex, agy) is therefore a + constant across both arms, so it does not confound the b-vs-c delta; the only + difference between the arms is the fixed context arm c injects via stdin. The + sentinel isolation probe (detect_leak) guards against repo-context leakage. + """ + return tempfile.mkdtemp(prefix="cmre-arm-cwd-") + + +def build_invocation(arm, cli, doc_text, rubric, context_text=None): + """Assemble the subprocess spec. Document content travels via stdin only. + + Returns a spec dict (argv, cwd, env_overrides, stdin payload metadata). The + actual stdin string is returned under `stdin` for the runner; tests assert + on argv/cwd/env, never needing the full payload. + """ + if cli == "codex": + argv = list(CODEX_BASE) + elif cli == "gemini": + argv = list(GEMINI_BASE) + elif cli == "agy": + argv = ["agy", "--print", AGY_INSTRUCTION] + else: + raise ValueError(f"unknown cli: {cli}") + + parts = [rubric, "\n\n=== DOCUMENT ===\n", doc_text] + if arm == "c_fixed_context" and context_text: + parts += ["\n\n=== REPO CONTEXT (fixed set) ===\n", context_text] + stdin_payload = "".join(parts) + + # Both arms run from a clean CWD (no ambient repo access). HOME is preserved + # so the CLI keeps its auth; the global config is constant across arms and + # does not confound the b-vs-c delta. The only difference is arm c's injected + # context above. + cwd = clean_cwd() + env = dict(os.environ) + + # Defensive: document content must never appear as an argv element. + doc_in_argv = any(doc_text and doc_text in a for a in argv) + + return { + "arm": arm, + "cli": cli, + "argv": argv, + "cwd": cwd, + "isolated_from_repo": True, + "skip_git_repo_check": "--skip-git-repo-check" in argv, + "stdin_has_context": arm == "c_fixed_context" and bool(context_text), + "doc_in_argv": doc_in_argv, + # agy's own flags don't confine the FS, so its arm runs under a macOS seatbelt deny-write + # profile applied at run time (see agy_sandbox_prefix / run_invocation). Logical argv stays + # ["agy","--print",...]; the sandbox wrapping is an execution concern, not part of the spec. + "sandbox": "seatbelt-deny-write" if cli == "agy" else None, + "_env": env, + "_stdin": stdin_payload, + } + + +def detect_leak(output, sentinel): + """The isolation probe's check: did arm b surface a sentinel it should not have?""" + return bool(sentinel) and sentinel in output + + +def parse_findings(text): + """Parse a model's output into findings [{id, text}]. + + The reliable path is a JSON array (the arm instruction now requests one); + `--output-format json`/fenced JSON is tolerated. Otherwise: markdown bullets + or numbered items; otherwise blank-line-separated paragraphs. We deliberately + do NOT split on every newline — verbose models (e.g. codex) wrap a single + finding across lines, so line-splitting over-counts wildly (one review parsed + as ~100 findings). Counts from unstructured prose are best-effort; structured + JSON output is what makes the yield metric trustworthy. + """ + text = (text or "").strip() + if not text: + return [] + # Tolerate a ```json ... ``` fence around the array. + json_text = text + if json_text.startswith("```"): + json_text = re.sub(r"^```[a-zA-Z0-9]*\n?", "", json_text) + json_text = re.sub(r"\n?```$", "", json_text).strip() + try: + data = json.loads(json_text) + if isinstance(data, list): + out = [] + for i, item in enumerate(data, 1): + if isinstance(item, dict) and "text" in item: + out.append({"id": item.get("id", f"f{i}"), "text": str(item["text"])}) + else: + out.append({"id": f"f{i}", "text": str(item)}) + return out + except json.JSONDecodeError: + pass + items = [] + for ln in text.splitlines(): + s = ln.strip() + if s.startswith(("- ", "* ")): + items.append(s[2:].strip()) + continue + m = NUMBERED_ITEM.match(ln) + if m: + items.append(m.group(1).strip()) + items = [i for i in items if i] + if items: + return [{"id": f"f{i}", "text": b} for i, b in enumerate(items, 1)] + # Best-effort prose fallback: blank-line-separated paragraphs only (a clear finding + # boundary), never per-line (over-counts wrapped prose). + paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()] + if len(paras) > 1: + return [{"id": f"f{i}", "text": p} for i, p in enumerate(paras, 1)] + return [{"id": "f1", "text": text}] + + +def run_invocation(spec, timeout): + """Run the CLI arm as a subprocess (integration-level; not unit-tested with live CLIs). + + For the agy arm, applies the macOS seatbelt deny-write floor at run time (the spec's logical + argv stays ["agy","--print",...]; the sandbox wrapping is an execution concern). The generated + profile is unlinked after the run. + """ + argv = spec["argv"] + sb_profile = None + if spec.get("sandbox") == "seatbelt-deny-write": + prefix, sb_profile = agy_sandbox_prefix() + # Defense-in-depth (R5): agy's read-only floor IS the macOS seatbelt. agy_sandbox_prefix() + # returns an empty prefix off-darwin OR when the profile template is missing — refuse rather + # than run agy unfloored, so a direct `arms.py run-arm ... agy` on a non-macOS host (or a + # mis-bundled skill) can't bypass env-detect's platform-gate and exfiltrate with no floor. + if not prefix: + return { + "status": "error", + "latency_ms": 0, + "findings": [], + "stderr": "agy arm refused: its read-only floor is macOS-only (seatbelt) and was " + "unavailable here (non-macOS host, or missing agy-readonly.sb.tmpl). " + "agy is macOS-only — use codex/gemini on other platforms.", + } + argv = prefix + argv + start = time.monotonic() + try: + try: + proc = subprocess.run( + argv, + input=spec["_stdin"], + cwd=spec["cwd"], + env=spec["_env"], + capture_output=True, + text=True, + timeout=timeout, + check=False, + ) + except subprocess.TimeoutExpired: + return {"status": "timeout", "latency_ms": (time.monotonic() - start) * 1000, "findings": [], "stderr": ""} + except FileNotFoundError: + return {"status": "error", "latency_ms": 0, "findings": [], "stderr": f"{spec['cli']} not found"} + latency_ms = (time.monotonic() - start) * 1000 + status = "ok" if proc.returncode == 0 else "error" + findings = parse_findings(proc.stdout) if status == "ok" else [] + return {"status": status, "latency_ms": latency_ms, "findings": findings, "stderr": proc.stderr} + finally: + if sb_profile and os.path.exists(sb_profile): + try: + os.unlink(sb_profile) + except OSError: + pass + + +def _read(path): + return Path(path).read_text() + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Cross-model review eval CLI arms (b, c).") + sub = parser.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("build-invocation") + p.add_argument("arm", choices=["b_isolated", "c_fixed_context"]) + p.add_argument("cli", choices=["codex", "gemini", "agy"]) + p.add_argument("doc") + p.add_argument("rubric") + p.add_argument("--context") + + p = sub.add_parser("detect-leak") + p.add_argument("sentinel") + p.add_argument("output") + + p = sub.add_parser("parse-findings") + p.add_argument("output") + + p = sub.add_parser("run-arm") + p.add_argument("arm", choices=["b_isolated", "c_fixed_context"]) + p.add_argument("cli", choices=["codex", "gemini", "agy"]) + p.add_argument("doc") + p.add_argument("rubric") + p.add_argument("--context") + p.add_argument("--doc-id", required=True) + p.add_argument("--trial", type=int, default=1) + p.add_argument("--timeout", type=float, default=180.0) + + args = parser.parse_args(argv) + + if args.cmd == "build-invocation": + ctx = _read(args.context) if args.context else None + spec = build_invocation(args.arm, args.cli, _read(args.doc), _read(args.rubric), ctx) + # Do not leak the full env/stdin into the printed spec. + printable = {k: v for k, v in spec.items() if not k.startswith("_")} + printable["stdin_len"] = len(spec["_stdin"]) + print(json.dumps(printable)) + return 0 + + if args.cmd == "detect-leak": + print(json.dumps({"leaked": detect_leak(_read(args.output), args.sentinel)})) + return 0 + + if args.cmd == "parse-findings": + print(json.dumps({"findings": parse_findings(_read(args.output))})) + return 0 + + if args.cmd == "run-arm": + ctx = _read(args.context) if args.context else None + spec = build_invocation(args.arm, args.cli, _read(args.doc), _read(args.rubric), ctx) + result = run_invocation(spec, args.timeout) + record = { + "arm": args.arm, + "doc_id": args.doc_id, + "trial": args.trial, + "status": result["status"], + "producer": "runner", + "latency_ms": result["latency_ms"], + "findings": result["findings"], + "model": args.cli, + } + # stderr carries the CLI's diagnostics (auth/availability failures) for the smoke check. + if result.get("stderr"): + sys.stderr.write(result["stderr"]) + print(json.dumps(record)) + return 0 if result["status"] == "ok" else 1 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/eval/cross_model_review/build_corpus.py b/scripts/eval/cross_model_review/build_corpus.py new file mode 100644 index 000000000..b7ecc66a9 --- /dev/null +++ b/scripts/eval/cross_model_review/build_corpus.py @@ -0,0 +1,621 @@ +#!/usr/bin/env python3 +"""Known-bug corpus builder for the code-review breakpoint. + +Mines a git repository for changes the project itself later judged wrong, so the +cross-model review eval can score arms against *validated* outcomes instead of +only forward-rated actionability (the R7 known-post-hoc-failure subset, ported +from plan review to code review). + +Attribution tiers, in descending strength: + - revert -- the team reverted the change; the verdict is the repo's, + not a model's or a reviewer's. Highest trust. + - named_regression -- a fix commit whose message names what broke. Strong. + - blame -- a fix whose touched lines blame back to a recent change. + Inferred; flagged `needs_confirmation` for the human (R6). + +Shape mirrors arms.py / run_arms.py: the rigor-bearing parsers are pure and +unit-tested (parse_revert_sha, parse_pr_numbers, parse_hunk_ranges, +is_regression_subject, validate_entry); the live `git` walk in `scan` / +`attribute_fix` is integration-level, validated against a constructed repo in the +test suite and against the real target repo at corpus-build time. + +Each emitted entry extends the corpus manifest's known_failure shape with a +`ground_truth` block (the bug a reviewer should have caught) so run_arms.py / +the judge can do a targeted hit/miss match per document. +""" + +import argparse +import json +import re +import subprocess +import sys +from datetime import datetime +from pathlib import Path + +ATTRIBUTIONS = ["revert", "named_regression", "blame"] +TRUST_LEVELS = ["high", "needs_confirmation"] + +# "This reverts commit <sha>." — the body git writes for a generated revert. +REVERT_SHA = re.compile(r"reverts commit ([0-9a-f]{7,40})", re.IGNORECASE) +PR_REF = re.compile(r"#(\d+)") +# Unified-diff hunk header: @@ -old_start[,old_count] +new_start[,new_count] @@ +HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+\d+(?:,\d+)? @@") +DIFF_GIT = re.compile(r"^diff --git a/(.+) b/(.+)$") +# Terms that name a regression in a fix subject. "introduced" is deliberately +# excluded: feature commits routinely say "introduced" and would false-positive. +REGRESSION_TERMS = ["broke", "broken", "regression", "reintroduce", "reintroduced"] +REGRESSION_RE = {t: re.compile(rf"\b{t}\b", re.IGNORECASE) for t in REGRESSION_TERMS} +# A fix commit subject (conventional): `fix:` or `fix(scope):` or `fix!:`. +FIX_SUBJECT = re.compile(r"^fix[(:!]", re.IGNORECASE) +# Paths that are documentation, not reviewable code — excluded from a code-review corpus. +NON_CODE_SUFFIXES = (".md", ".markdown", ".rst", ".txt") +NON_CODE_NAMES = ("changelog", "license", "notice", "authors", "codeowners") + + +# --- pure parsers (unit-tested) ------------------------------------------------ + + +def parse_revert_sha(body): + """Extract the culprit SHA from a git-generated revert body, or None.""" + m = REVERT_SHA.search(body or "") + return m.group(1) if m else None + + +def parse_pr_numbers(text): + """All `#NNN` references in order. `last` is the conventional reverted-PR slot.""" + prs = [int(n) for n in PR_REF.findall(text or "")] + return {"prs": prs, "last": prs[-1] if prs else None} + + +def parse_hunk_ranges(diff): + """Per-file pre-image (old-side) line ranges a diff touches. + + These are the ranges to `git blame` at the fix's parent to find the commit + that last wrote them (blame attribution). Pure-addition hunks (old count 0) + contribute no blameable range and are dropped. + """ + files = [] + current = None + for line in (diff or "").splitlines(): + mg = DIFF_GIT.match(line) + if mg: + current = {"file": mg.group(2), "old_ranges": []} + files.append(current) + continue + mh = HUNK.match(line) + if mh and current is not None: + start = int(mh.group(1)) + count = int(mh.group(2)) if mh.group(2) is not None else 1 + if count > 0: + current["old_ranges"].append([start, count]) + return {"files": files} + + +def parse_blame_ranges(diff): + """Per-file old-side ranges of only the lines a fix actually deleted/changed. + + Unlike `parse_hunk_ranges` (which returns the full old-side hunk range from the + `@@` header, context lines included), this walks the hunk body and records only + the pre-image lines marked `-` -- the lines the fix replaced or removed. Those + are the lines to `git blame` at the fix's parent: blaming the surrounding context + would attribute the fixed code to whoever last wrote the *neighboring* lines, + materializing the wrong culprit diff. Pure-addition hunks (no `-` lines) + contribute nothing. Consecutive deleted lines are coalesced into `[start, count]`. + """ + files = [] + current = None + old_lineno = None # next old-side line number; only tracked inside a hunk body + for line in (diff or "").splitlines(): + mg = DIFF_GIT.match(line) + if mg: + current = {"file": mg.group(2), "old_ranges": []} + files.append(current) + old_lineno = None + continue + mh = HUNK.match(line) + if mh and current is not None: + old_lineno = int(mh.group(1)) + continue + if current is None or old_lineno is None: + continue + # File headers inside a hunk region never occur, but guard anyway so a + # stray `---`/`+++` is never miscounted as a context/deletion line. + if line.startswith("---") or line.startswith("+++"): + continue + if line.startswith("-"): + ranges = current["old_ranges"] + if ranges and ranges[-1][0] + ranges[-1][1] == old_lineno: + ranges[-1][1] += 1 # extend a contiguous run of deleted lines + else: + ranges.append([old_lineno, 1]) + old_lineno += 1 + elif line.startswith("+"): + continue # added line: no pre-image counterpart, no old-side advance + elif line.startswith("\\"): + continue # "\ No newline at end of file": not a content line + else: + old_lineno += 1 # context line (leading space): advances the old side + return {"files": files} + + +def is_regression_subject(text): + """Detect a fix subject that names a break; return matched terms (Tier-2).""" + matched = [t for t, rx in REGRESSION_RE.items() if rx.search(text or "")] + return {"is_regression": bool(matched), "matched": matched} + + +def parse_numstat(text): + """Sum changed lines and count files from `git show --numstat` output. + + Each line is `<added>\\t<deleted>\\t<path>`; binary files report `-` and count + as a touched file with 0 measurable lines. Used by the culprit-size gate. + """ + files, lines = 0, 0 + for ln in (text or "").splitlines(): + parts = ln.split("\t") + if len(parts) < 3: + continue + files += 1 + a, d = parts[0], parts[1] + if a.isdigit(): + lines += int(a) + if d.isdigit(): + lines += int(d) + return {"files": files, "changed_lines": lines} + + +def culprit_within_caps(changed_lines, files, max_lines, max_files): + """Quality gate: reject culprits too large to review or wide enough to be a + foundational/import commit (the failure modes a Tier-3 blame corpus collects).""" + reasons = [] + if max_lines and changed_lines > max_lines: + reasons.append(f"culprit diff {changed_lines} lines > {max_lines}") + if max_files and files > max_files: + reasons.append(f"culprit touches {files} files > {max_files} (likely foundational)") + return {"ok": not reasons, "reasons": reasons} + + +def is_code_path(path): + """True for reviewable code, False for docs/markdown — keeps the corpus code-only.""" + p = (path or "").strip().lower() + if not p: + return False + if p.endswith(NON_CODE_SUFFIXES): + return False + if p.startswith("docs/") or "/docs/" in p: + return False + base = p.rsplit("/", 1)[-1].rsplit(".", 1)[0] + return base not in NON_CODE_NAMES + + +def validate_entry(entry): + """Manifest-conformance gate for a known_failure corpus entry (mirrors validate_record).""" + errors = [] + if not isinstance(entry, dict): + return ["entry is not a JSON object"] + if not isinstance(entry.get("id"), str) or not entry.get("id"): + errors.append("id must be a non-empty string") + if not isinstance(entry.get("path"), str) or not entry.get("path"): + errors.append("path must be a non-empty string") + if entry.get("subset") != "known_failure": + errors.append('subset must be "known_failure"') + gt = entry.get("ground_truth") + if not isinstance(gt, dict): + errors.append("ground_truth must be an object") + return errors + if not isinstance(gt.get("bug"), str) or not gt.get("bug"): + errors.append("ground_truth.bug must be a non-empty string") + if not isinstance(gt.get("fix_commit"), str) or not gt.get("fix_commit"): + errors.append("ground_truth.fix_commit must be a non-empty string") + if gt.get("attribution") not in ATTRIBUTIONS: + errors.append(f"ground_truth.attribution must be one of {ATTRIBUTIONS}") + has_pr = isinstance(gt.get("culprit_pr"), int) and not isinstance(gt.get("culprit_pr"), bool) + has_sha = isinstance(gt.get("culprit_sha"), str) and bool(gt.get("culprit_sha")) + if not (has_pr or has_sha): + errors.append("ground_truth must have a culprit_pr (int) or culprit_sha (str)") + if gt.get("trust") not in TRUST_LEVELS: + errors.append(f"ground_truth.trust must be one of {TRUST_LEVELS}") + days = gt.get("surfaced_after_days") + if days is not None and (not isinstance(days, int) or isinstance(days, bool) or days < 0): + errors.append("ground_truth.surfaced_after_days must be an integer >= 0 when present") + return errors + + +# --- git walk (integration-level) ---------------------------------------------- + +_REC_SEP, _FIELD_SEP = "\x1e", "\x1f" + + +def _git(repo, args): + """Run `git -C <repo> ...` and return stdout; raise on nonzero exit.""" + proc = subprocess.run( + ["git", "-C", str(repo), *args], + capture_output=True, + text=True, + check=False, + ) + if proc.returncode != 0: + raise RuntimeError(f"git {' '.join(args)} failed: {proc.stderr.strip()}") + return proc.stdout + + +def _is_revert(subject, body): + if parse_revert_sha(body): + return True + s = subject.strip() + return bool(re.match(r"^revert[:(]", s, re.IGNORECASE) or s.startswith('Revert "')) + + +def _days_between(later_iso, earlier_iso): + try: + return (datetime.fromisoformat(later_iso) - datetime.fromisoformat(earlier_iso)).days + except (ValueError, TypeError): + return None + + +def scan(repo, out_dir=None, all_refs=False, max_entries=None): + """Discover Tier-1 revert-attributed known-failure entries from `repo`. + + For each revert, the reverted commit is the culprit: its diff is the document + a reviewer should have caught a problem in, and the revert is the verdict that + it shipped wrong. When `out_dir` is given, the culprit diff is materialized to + `<out_dir>/<id>.diff` so an arm can review it directly. + """ + fmt = _FIELD_SEP.join(["%H", "%s", "%b", "%aI"]) + _REC_SEP + log_args = ["log", f"--format={fmt}"] + if all_refs: + log_args.append("--all") + raw = _git(repo, log_args) + + reverts_found = 0 + entries = [] + out_path = Path(out_dir) if out_dir else None + if out_path: + out_path.mkdir(parents=True, exist_ok=True) + + for rec in raw.split(_REC_SEP): + rec = rec.strip("\n") + if not rec: + continue + parts = rec.split(_FIELD_SEP) + if len(parts) < 4: + continue + r_sha, r_subject, r_body, r_date = parts[0], parts[1], parts[2], parts[3] + if not _is_revert(r_subject, r_body): + continue + reverts_found += 1 + + culprit_sha = parse_revert_sha(r_body) + if not culprit_sha: + # Conventional revert with no embedded SHA: we have only the reverted + # PR. Insufficient to materialize a reviewable diff here; left for the + # human to resolve (R6). Counted, not emitted. + continue + + try: + c_meta = _git(repo, ["show", "-s", "--format=%s" + _FIELD_SEP + "%aI", culprit_sha]) + except RuntimeError: + continue + c_subject, _, c_date = c_meta.partition(_FIELD_SEP) + c_subject, c_date = c_subject.strip(), c_date.strip() + + culprit_pr = parse_pr_numbers(c_subject)["last"] + short = culprit_sha[:7] + entry_id = f"kf-{short}" + + path = f"FILL: git show {culprit_sha} in {repo}" + if out_path: + diff = _git(repo, ["show", culprit_sha]) + diff_file = out_path / f"{entry_id}.diff" + diff_file.write_text(diff) + path = str(diff_file) + + gt = { + "bug": c_subject, # what shipped and was reverted; the human sharpens the exact finding (R6) + "fix_commit": r_sha, + "culprit_sha": culprit_sha, + "attribution": "revert", + "trust": "high", + "revert_subject": r_subject, + } + if culprit_pr is not None: + gt["culprit_pr"] = culprit_pr + days = _days_between(r_date, c_date) + if days is not None and days >= 0: + gt["surfaced_after_days"] = days + + entry = {"id": entry_id, "path": path, "subset": "known_failure", "ground_truth": gt} + if not validate_entry(entry): + entries.append(entry) + if max_entries and len(entries) >= max_entries: + break + + return { + "repo": str(repo), + "entries": entries, + "stats": {"reverts_found": reverts_found, "entries_emitted": len(entries)}, + } + + +def blame_candidates(repo, fix_sha, code_only=False): + """Blame the lines a fix touched (at its parent) to find candidate culprits. + + Returns [{culprit_sha, files}] ranked by how many of the fix's files each + culprit last wrote (most first) — the heuristic best guess. When `code_only`, + documentation files are skipped so a code-review corpus stays code-only. + + Uses `parse_blame_ranges` (deleted lines only), not `parse_hunk_ranges` + (full header range): blaming a hunk's context lines would attribute the fixed + code to whoever last wrote the *neighboring* lines and pick the wrong culprit. + """ + diff = _git(repo, ["show", fix_sha]) + candidates = {} + for f in parse_blame_ranges(diff)["files"]: + if code_only and not is_code_path(f["file"]): + continue + for start, count in f["old_ranges"]: + end = start + count - 1 + try: + # --line-porcelain emits the full 40-char SHA per line; plain `-l` + # truncates a digit for boundary commits (the `^` alignment hack). + blame = _git(repo, ["blame", "--line-porcelain", "-L", f"{start},{end}", f"{fix_sha}^", "--", f["file"]]) + except RuntimeError: + continue + for sha in re.findall(r"(?m)^([0-9a-f]{40}) \d+ \d+", blame): + candidates.setdefault(sha, set()).add(f["file"]) + ranked = sorted(candidates.items(), key=lambda kv: len(kv[1]), reverse=True) + return [{"culprit_sha": s, "files": sorted(fs)} for s, fs in ranked] + + +def attribute_fix(repo, fix_sha): + """Tier-2/3 blame attribution for a single fix (manual tool; shows all candidates).""" + return {"fix_commit": fix_sha, "candidates": blame_candidates(repo, fix_sha, code_only=False)} + + +def scan_fixes(repo, out_dir=None, all_refs=False, max_entries=None, + max_culprit_lines=2000, max_culprit_files=30, dedup=True): + """Discover Tier-3 blame-attributed known-failure entries from `fix:` commits. + + Each conventional fix commit is blamed back to the change that last wrote the + code it repairs; that change's diff becomes the document a reviewer should have + caught the bug in. Blame is inferred, so every entry is `attribution: "blame", + trust: "needs_confirmation"` for the human to confirm (R6), with the runner-up + culprits kept in `culprit_alternates`. + + Quality gate (the first-run lesson): blame on a repo that ships large feature + commits collapses many fixes onto one giant culprit. Entries whose culprit diff + exceeds `max_culprit_lines`/`max_culprit_files` are dropped (too large to review + / foundational), and when `dedup`, only the first fix per distinct culprit is + kept (so N fixes touching one feature don't become N non-independent docs). + Tighter caps yield a smaller but cleaner, decidable corpus. + """ + fmt = _FIELD_SEP.join(["%H", "%s", "%aI"]) + _REC_SEP + log_args = ["log", f"--format={fmt}"] + if all_refs: + log_args.append("--all") + raw = _git(repo, log_args) + + fixes_scanned = 0 + fixes_with_culprit = 0 + filtered_oversize = 0 + filtered_dup = 0 + seen_culprits = set() + entries = [] + out_path = Path(out_dir) if out_dir else None + if out_path: + out_path.mkdir(parents=True, exist_ok=True) + + for rec in raw.split(_REC_SEP): + rec = rec.strip("\n") + if not rec: + continue + parts = rec.split(_FIELD_SEP) + if len(parts) < 3: + continue + f_sha, f_subject, f_date = parts[0], parts[1], parts[2] + if not FIX_SUBJECT.match(f_subject.strip()): + continue + fixes_scanned += 1 + + try: + cands = blame_candidates(repo, f_sha, code_only=True) + except RuntimeError: + continue + if not cands: + continue + fixes_with_culprit += 1 + + culprit_sha = cands[0]["culprit_sha"] + alternates = [c["culprit_sha"] for c in cands[1:]] + + # quality gate: drop oversize/foundational culprits, then dedup shared ones + try: + size = parse_numstat(_git(repo, ["show", "--numstat", "--format=", culprit_sha])) + except RuntimeError: + continue + if not culprit_within_caps(size["changed_lines"], size["files"], max_culprit_lines, max_culprit_files)["ok"]: + filtered_oversize += 1 + continue + if dedup and culprit_sha in seen_culprits: + filtered_dup += 1 + continue + seen_culprits.add(culprit_sha) + try: + c_meta = _git(repo, ["show", "-s", "--format=%s" + _FIELD_SEP + "%aI", culprit_sha]) + except RuntimeError: + continue + c_subject, _, c_date = c_meta.partition(_FIELD_SEP) + c_subject, c_date = c_subject.strip(), c_date.strip() + + entry_id = f"kf-{f_sha[:7]}" # keyed by the fix -> one corpus item per fix + path = f"FILL: git show {culprit_sha} in {repo}" + if out_path: + diff_file = out_path / f"{entry_id}.diff" + diff_file.write_text(_git(repo, ["show", culprit_sha])) + path = str(diff_file) + + gt = { + "bug": f_subject, # the fix subject = the bug a reviewer should have caught in the culprit + "fix_commit": f_sha, + "culprit_sha": culprit_sha, + "attribution": "blame", + "trust": "needs_confirmation", + } + if alternates: + gt["culprit_alternates"] = alternates + culprit_pr = parse_pr_numbers(c_subject)["last"] + if culprit_pr is not None: + gt["culprit_pr"] = culprit_pr + days = _days_between(f_date, c_date) + if days is not None and days >= 0: + gt["surfaced_after_days"] = days + + entry = {"id": entry_id, "path": path, "subset": "known_failure", "ground_truth": gt} + if not validate_entry(entry): + entries.append(entry) + if max_entries and len(entries) >= max_entries: + break + + return { + "repo": str(repo), + "entries": entries, + "stats": { + "fixes_scanned": fixes_scanned, + "fixes_with_culprit": fixes_with_culprit, + "filtered_oversize": filtered_oversize, + "filtered_dup": filtered_dup, + "entries_emitted": len(entries), + }, + } + + +def to_manifest(scan): + """Wrap scan / scan-fixes output into a corpus-manifest skeleton. + + Entries become the `docs` array; pre_registration is left null so the human + must fill the decision rule before running (R9), and confirm the + `needs_confirmation` Tier-3 entries (R6). Accepts either the full + `{entries, stats}` object or a bare list of entries. + """ + entries = scan.get("entries", []) if isinstance(scan, dict) else scan + return { + "_schema": "Assembled from build_corpus output. FILL pre_registration before running (R9); confirm needs_confirmation entries and add negative_control + forward_rated docs (R6).", + "pre_registration": { + "go_threshold": None, + "minimum_corpus_n": None, + "trials_per_arm": 3, + "arm_c_context_rule": None, + }, + "arms": ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"], + "docs": entries, + } + + +def _read(path): + return Path(path).read_text() + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Known-bug corpus builder (code-review breakpoint).") + sub = parser.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("parse-revert-sha") + p.add_argument("file") + + p = sub.add_parser("parse-pr-numbers") + p.add_argument("file") + + p = sub.add_parser("parse-hunk-ranges") + p.add_argument("file") + + p = sub.add_parser("is-regression-subject") + p.add_argument("file") + + p = sub.add_parser("validate-entry") + p.add_argument("file") + + p = sub.add_parser("is-code-path") + p.add_argument("file") + + p = sub.add_parser("parse-numstat") + p.add_argument("file") + + p = sub.add_parser("to-manifest") + p.add_argument("file") + + p = sub.add_parser("scan") + p.add_argument("--repo", required=True) + p.add_argument("--out-dir") + p.add_argument("--all", action="store_true", help="scan all refs, not just HEAD history") + p.add_argument("--max", type=int, default=None) + + p = sub.add_parser("scan-fixes") + p.add_argument("--repo", required=True) + p.add_argument("--out-dir") + p.add_argument("--all", action="store_true", help="scan all refs, not just HEAD history") + p.add_argument("--max", type=int, default=None) + p.add_argument("--max-culprit-lines", type=int, default=2000, help="drop culprits whose diff exceeds this (0 = no cap)") + p.add_argument("--max-culprit-files", type=int, default=30, help="drop culprits touching more files than this (0 = no cap)") + p.add_argument("--no-dedup", action="store_true", help="keep every fix even when fixes share a culprit") + + p = sub.add_parser("attribute-fix") + p.add_argument("--repo", required=True) + p.add_argument("fix_sha") + + args = parser.parse_args(argv) + + if args.cmd == "parse-revert-sha": + print(json.dumps({"culprit_sha": parse_revert_sha(_read(args.file))})) + return 0 + + if args.cmd == "parse-pr-numbers": + print(json.dumps(parse_pr_numbers(_read(args.file)))) + return 0 + + if args.cmd == "parse-hunk-ranges": + print(json.dumps(parse_hunk_ranges(_read(args.file)))) + return 0 + + if args.cmd == "is-regression-subject": + print(json.dumps(is_regression_subject(_read(args.file)))) + return 0 + + if args.cmd == "validate-entry": + errors = validate_entry(json.loads(_read(args.file))) + print(json.dumps({"valid": not errors, "errors": errors})) + return 0 if not errors else 1 + + if args.cmd == "is-code-path": + print(json.dumps({"is_code": is_code_path(_read(args.file).strip())})) + return 0 + + if args.cmd == "to-manifest": + print(json.dumps(to_manifest(json.loads(_read(args.file))), indent=2)) + return 0 + + if args.cmd == "scan": + print(json.dumps(scan(args.repo, args.out_dir, args.all, args.max))) + return 0 + + if args.cmd == "parse-numstat": + print(json.dumps(parse_numstat(_read(args.file)))) + return 0 + + if args.cmd == "scan-fixes": + print(json.dumps(scan_fixes( + args.repo, args.out_dir, args.all, args.max, + max_culprit_lines=args.max_culprit_lines, + max_culprit_files=args.max_culprit_files, + dedup=not args.no_dedup, + ))) + return 0 + + if args.cmd == "attribute-fix": + print(json.dumps(attribute_fix(args.repo, args.fix_sha))) + return 0 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/eval/cross_model_review/critique.sh b/scripts/eval/cross_model_review/critique.sh new file mode 100755 index 000000000..3110263ea --- /dev/null +++ b/scripts/eval/cross_model_review/critique.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# Quick cross-model critique of a single plan/document. +# +# Runs the cross-model arms (codex = OpenAI, gemini = Google) as isolated reviewers over one +# document and prints each model's findings side by side. This is the turnkey path — the full +# four-arm eval (baseline + self-critic + judge) is agent-driven; see README.md. +# (agy/Antigravity was dropped — unreliable as a non-interactive reviewer; gemini needs GEMINI_API_KEY.) +# +# Usage: +# critique.sh <plan.md> [rubric.md] [context.md] +# +# <plan.md> document to critique (required) +# [rubric.md] challenge rubric (optional; a built-in independent-challenge rubric is used if omitted) +# [context.md] if given, models also receive this as a fixed context set (arm c_fixed_context); +# otherwise they review the document text only (arm b_isolated) +# +# Each model run SENDS THE DOCUMENT to that vendor (codex -> OpenAI, gemini -> Google). +# A missing/unauthenticated CLI is skipped with a note rather than failing the whole run. +set -u + +here="$(cd "$(dirname "$0")" && pwd)" +arms="$here/arms.py" + +case "${1:-}" in + -h|--help|"") + sed -n '2,17p' "$0" | sed 's/^# \{0,1\}//' + exit 0 + ;; +esac + +plan="$1" +rubric="${2:-}" +context="${3:-}" + +if [ ! -f "$plan" ]; then + echo "error: plan file not found: '$plan'" >&2 + exit 2 +fi + +# Built-in default rubric (kept self-contained so this works outside the repo layout). +if [ -z "$rubric" ]; then + rubric="$(mktemp -t cmre-rubric-XXXXXX)" + cat > "$rubric" <<'EOF' +Independent challenge rubric: challenge the premise, surface unstated assumptions, name +unconsidered alternatives, and state what would falsify this plan. Be concrete and specific +to the document. Return your findings as a JSON array of strings, one element per finding. +EOF +elif [ ! -f "$rubric" ]; then + echo "error: rubric file not found: '$rubric'" >&2 + exit 2 +fi + +arm="b_isolated" +if [ -n "$context" ]; then + if [ ! -f "$context" ]; then + echo "error: context file not found: '$context'" >&2 + exit 2 + fi + arm="c_fixed_context" +fi + +doc_id="$(basename "$plan" .md)" +timeout="${CMRE_TIMEOUT:-300}" # per-arm timeout in seconds; override with CMRE_TIMEOUT (gemini can be slow) + +run_one() { + cli="$1" + label="$2" + if ! command -v "$cli" >/dev/null 2>&1; then + printf '\n=== %s — not installed, skipped ===\n' "$label" + return + fi + printf '\n=== %s — arm %s ===\n' "$label" "$arm" + if [ -n "$context" ]; then + rec="$(python3 "$arms" run-arm "$arm" "$cli" "$plan" "$rubric" --context "$context" --doc-id "$doc_id" --trial 1 --timeout "$timeout" 2>/dev/null)" + else + rec="$(python3 "$arms" run-arm "$arm" "$cli" "$plan" "$rubric" --doc-id "$doc_id" --trial 1 --timeout "$timeout" 2>/dev/null)" + fi + # Persist the FULL record (display below truncates to 240 chars). Set CMRE_OUT_DIR to keep + # records for post-hoc judging — without this the full findings are unrecoverable after the run. + if [ -n "${CMRE_OUT_DIR:-}" ]; then + mkdir -p "$CMRE_OUT_DIR" + printf '%s' "$rec" > "$CMRE_OUT_DIR/${cli}__${doc_id}.json" + fi + printf '%s' "$rec" | python3 -c ' +import json, sys +try: + r = json.load(sys.stdin) +except Exception: + print(" (no or invalid output — the CLI may be unauthenticated or timed out)") + sys.exit() +print(" status=%s findings=%d latency=%.0fs" % (r["status"], len(r["findings"]), r["latency_ms"] / 1000)) +for f in r["findings"]: + t = " ".join(f["text"].split()) + print(" - " + (t[:240] + ("..." if len(t) > 240 else ""))) +' +} + +echo "Cross-model critique of: $plan" +if [ -n "$context" ]; then + echo "Rubric: $rubric | Context: $context" +else + echo "Rubric: $rubric" +fi +run_one codex "codex (OpenAI)" +run_one gemini "gemini (Google)" +echo "" diff --git a/scripts/eval/cross_model_review/decision-artifact-template.md b/scripts/eval/cross_model_review/decision-artifact-template.md new file mode 100644 index 000000000..43c5ebe60 --- /dev/null +++ b/scripts/eval/cross_model_review/decision-artifact-template.md @@ -0,0 +1,57 @@ +# Cross-Model Critique Evaluation — Decision Record (template) + +Fill this from the U6 aggregate output at the end of a run, then write the completed file +under `docs/` (a `docs/solutions/`-style doc if the conclusion is durable guidance, or a +date-prefixed decision record). "Test expectation: none" — this artifact is authored from +the run's aggregates, not unit-tested. + +Framing note: this evaluates a **cross-model critique** lever, not an "independent review" +(the independence claim is an overclaim — see origin R10). + +--- + +**Date:** <run date> +**Corpus:** <N> documents (<k> known-failure, <m> forward-rated, 1 negative-control) +**Pre-registered before running:** go_threshold = <t>, minimum_corpus_n = <n>, trials_per_arm = <≥3>, arm_c_context_rule = <rule> +**Judge model family:** <family> (<"same family as baseline/self-critic — blind-integrity risk" | "distinct">) + +## Outcome + +> **<build:`<arm>` | build nothing | inconclusive>** + +<One paragraph: what the result means and the immediate next step. If `build:<arm>`, name +the winning arm and that the deferred cross-model build is shaped by it. If `inconclusive`, +say why — below minimum N, or blind-integrity confounded, or the negative control moved — +and what a re-run needs.> + +## Primary signal — known-failure subset + +Per-arm count of confirmed, unique, decision-changing findings that surfaced the issue each +known-failure document's post-hoc failure proved mattered: + +| Arm | Known-failure hits | Forward-rated (corroborating) | Trial variance | +|-----|--------------------|-------------------------------|----------------| +| a_baseline | <n> | <n> | <determinism note> | +| b_isolated | <n> | <n> | <determinism note> | +| c_fixed_context | <n> | <n> | <determinism note> | +| d_self_critic | <n> | <n> | <determinism note> | + +## Validity checks + +- **Blind-integrity:** judge arm-guess accuracy <x> vs chance <1/n_arms> — <"held" | "confounded → result is inconclusive">. +- **Negative control:** <"did not move" | "MOVED → harness stability problem, result is inconclusive">. +- **Power:** corpus_n <N> vs minimum_corpus_n <n> — <"met" | "below → inconclusive">. + +## Secondary metrics (tie-breakers, not primary) + +- Latency per arm: <...> (note: measured on one already-working machine; does not predict + cross-machine auth fragility a shipped feature would face). +- Setup/auth friction: <...> +- Generic/duplicate (noise) rate per arm: <...> + +## What this does and does not conclude + +- It concludes whether a cross-model-critique lever produced unique, decision-changing + findings often enough to justify its carrying cost, on this corpus. +- It does **not** decompose a self-critic win into "fresh pass" vs "failure-modes-supplied" + (per origin R3), and it does not measure cross-machine setup fragility. diff --git a/scripts/eval/cross_model_review/drive_eval.py b/scripts/eval/cross_model_review/drive_eval.py new file mode 100644 index 000000000..28ae6e479 --- /dev/null +++ b/scripts/eval/cross_model_review/drive_eval.py @@ -0,0 +1,398 @@ +#!/usr/bin/env python3 +"""Code-review eval driver — wires the deterministic spine end-to-end. + +`plan` : enumerate the per-(arm x doc x trial) work, emit the CLI-arm commands the + orchestrator can run, and the in-process/judge handoff. Refuses to plan a + run whose threshold/N are not pre-registered (R9). +`finalize`: over a run dir of ingested arm records + the judge's verdicts, run the + gt-resolve -> gt-score -> aggregate chain and render the decision artifact. + +What this driver does NOT do: run arms a_baseline / d_self_critic or the judge. Those +are model-driven and produced by the orchestrator via in-process subagent dispatch (there +is no `claude -p`; see README "How the two halves cooperate"). The driver consumes their +record/verdict files, so it is fully deterministic and unit-testable. The CLI arms (b, c) +are spawnable from Python; `plan` emits their exact commands rather than executing ~N*trials +model calls implicitly. +""" + +import argparse +import json +import sys +from datetime import date +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +import run_arms # noqa: E402 (co-located deterministic carrier) + +ARMS = run_arms.ARMS +CLI_ARMS = ["b_isolated", "c_fixed_context"] +IN_PROCESS_ARMS = ["a_baseline", "d_self_critic"] +ARMS_PY = str(Path(__file__).resolve().parent / "arms.py") + + +def _load(path): + return json.loads(Path(path).read_text()) + + +def plan(manifest, out_dir, rubric, context, cli_b, cli_c): + """Enumerate work units + handoff; guard pre-registration (R9).""" + prereg = manifest.get("pre_registration", {}) + missing = [k for k in ("go_threshold", "minimum_corpus_n", "trials_per_arm") if prereg.get(k) is None] + if missing: + return {"ok": False, "error": f"pre-registration incomplete: {', '.join(missing)} must be set before running (R9)"} + + # Pre-registration values must be positive integers, not just present. A string like + # "2" passes the missing-field check above but is silently ignored downstream: + # aggregate's `isinstance(threshold, int)` guard turns a string go_threshold into a + # blanket build_nothing, and corpus_status's `isinstance(minimum, int)` guard turns a + # string minimum_corpus_n into below_n=False — both corrupt the decision without erroring. + # trials_per_arm=0 (or a non-int) would enumerate zero arm runs yet still let finalize + # emit a decision from no experimental data. + def _is_pos_int(v): + return isinstance(v, int) and not isinstance(v, bool) and v >= 1 + + trials = prereg["trials_per_arm"] + if not _is_pos_int(trials): + return {"ok": False, "error": f"trials_per_arm must be an integer >= 1 (got {trials!r}); a decision-grade run needs >= 3 (R8)"} + go_threshold = prereg["go_threshold"] + if not _is_pos_int(go_threshold): + return {"ok": False, "error": f"go_threshold must be an integer >= 1 (got {go_threshold!r}); a non-int is silently ignored downstream and forces build_nothing (R9)"} + minimum_corpus_n = prereg["minimum_corpus_n"] + if not _is_pos_int(minimum_corpus_n): + return {"ok": False, "error": f"minimum_corpus_n must be an integer >= 1 (got {minimum_corpus_n!r}); a non-int is silently ignored downstream and skips the power check (R9)"} + + docs = manifest.get("docs", []) + if not any(d.get("subset") == "known_failure" for d in docs): + return {"ok": False, "error": "no known_failure documents in corpus — nothing to score on the primary metric (R7)"} + + cli_for = {"b_isolated": cli_b, "c_fixed_context": cli_c} + + expected_records, cli_commands, in_process_records = [], [], [] + for doc in docs: + doc_id, path = doc.get("id"), doc.get("path", f"FILL:{doc.get('id')}") + for trial in range(1, trials + 1): + for arm in ARMS: + expected_records.append({"arm": arm, "doc_id": doc_id, "trial": trial}) + for arm in CLI_ARMS: + argv = ["python3", ARMS_PY, "run-arm", arm, cli_for[arm], path, rubric] + if arm == "c_fixed_context": + argv += ["--context", context] + argv += ["--doc-id", doc_id, "--trial", str(trial)] + cli_commands.append({"arm": arm, "doc_id": doc_id, "trial": trial, "argv": argv}) + for arm in IN_PROCESS_ARMS: + in_process_records.append({"arm": arm, "doc_id": doc_id, "trial": trial}) + + out_root = Path(out_dir) + records_dir = out_root / "records" + records_dir.mkdir(parents=True, exist_ok=True) + + state = { + "manifest_docs": len(docs), + "trials_per_arm": trials, + "arms": ARMS, + "records_dir": str(records_dir), + "expected_records": expected_records, + "cli_commands": cli_commands, + "orchestrator_todo": { + "in_process_arm_records": in_process_records, + "ingest_to": str(records_dir), + "judge_gt_verdicts": "per-finding matches_bug for known_failure docs (gt_match_rubric.md)", + "judge_class_verdicts": "per-(arm,doc) decision_changing for forward_rated + negative_control (judge_rubric.md)", + "then": f"drive_eval.py finalize {records_dir} <manifest> --gt-verdicts <f> [--class-verdicts <f>] [--integrity correct,total]", + }, + } + (out_root / "run-state.json").write_text(json.dumps(state, indent=2)) + + return { + "ok": True, + "run_dir": str(out_root), + "records_dir": str(records_dir), + "counts": { + "docs": len(docs), + "known_failure": sum(1 for d in docs if d.get("subset") == "known_failure"), + "expected_records": len(expected_records), + "cli_commands": len(cli_commands), + "in_process_records": len(in_process_records), + }, + } + + +# A trial only counts as completed evidence when its arm actually produced a review. +# timeout/error records are schema-valid (run_arms emits them on CLI auth/quota/runtime +# failure) but carry no usable findings; counting them would let coverage() treat a failed +# trial as present and let finalize score it as a real zero-finding review. +COMPLETED_STATUSES = {"ok", "degraded"} + + +def load_records(records_dir): + """Schema-conformant, completed record files in the dir (skips run-state and junk). + + Records whose status is timeout/error are dropped here so they are neither scored + nor counted toward coverage — a failed trial is a missing trial, not a clean + zero-finding review. + """ + out = [] + for f in sorted(Path(records_dir).glob("*.json")): + try: + rec = json.loads(f.read_text()) + except json.JSONDecodeError: + continue + if run_arms.validate_record(rec): + continue + if rec.get("status") not in COMPLETED_STATUSES: + continue + out.append(rec) + return out + + +def _subset_counts(docs): + c = {"known_failure": 0, "forward_rated": 0, "negative_control": 0} + for d in docs: + s = d.get("subset") + if s in c: + c[s] += 1 + return c + + +def _yield_section(yield_per_arm): + """Finding-yield table — read alongside GT-match, not instead of it.""" + if not yield_per_arm: + return "" + rows = [ + f"| {arm} | {yp.get('total', 0)} | {yp.get('unique_actionable', 0)} | {yp.get('decision_changing', 0)} |" + for arm, yp in yield_per_arm.items() + ] + return ( + "## Finding yield (corroborating — GT-match alone undercounts reviewer value)\n\n" + "Total findings vs. blind-judged unique-actionable and decision-changing. A low\n" + "GT-match with high unique-actionable yield means the arm found real bugs that were\n" + "not the one the historical fix targeted.\n\n" + "| Arm | Findings | Unique-actionable | Decision-changing |\n" + "|-----|----------|-------------------|-------------------|\n" + + "\n".join(rows) + + "\n\n" + ) + + +def render_artifact(manifest, result, gt, integ, judge_family, run_date, yield_per_arm=None): + docs = manifest.get("docs", []) + sc = _subset_counts(docs) + prereg = manifest.get("pre_registration", {}) + fam = judge_family or "<undisclosed>" + fam_note = "same family as baseline/self-critic — blind-integrity risk" if fam == "claude" else "distinct" + + rows = [] + for arm in ARMS: + kf = gt["per_arm"].get(arm, {}).get("hits", 0) + fr = result["per_arm"].get(arm, {}).get("forward_rated", 0) + rows.append(f"| {arm} | {kf} | {fr} | trials_per_arm={prereg.get('trials_per_arm')} |") + + if integ is not None: + acc = integ.get("accuracy") + chance = integ.get("chance") + integ_line = f"judge arm-guess accuracy {acc} vs chance {chance} — " + ( + "confounded -> result is inconclusive" if integ.get("confounded") else "held" + ) + else: + integ_line = "not run" + + power = "met" if not result.get("below_n") else "below -> inconclusive" + control = "MOVED -> harness stability problem, result is inconclusive" if result.get("control_moved") else "did not move" + + return f"""# Cross-Model Critique Evaluation — Decision Record + +Framing note: this evaluates a **cross-model critique** lever, not an "independent review". + +**Date:** {run_date} +**Corpus:** {len(docs)} documents ({sc['known_failure']} known-failure, {sc['forward_rated']} forward-rated, {sc['negative_control']} negative-control) +**Pre-registered before running:** go_threshold = {prereg.get('go_threshold')}, minimum_corpus_n = {prereg.get('minimum_corpus_n')}, trials_per_arm = {prereg.get('trials_per_arm')}, arm_c_context_rule = {prereg.get('arm_c_context_rule')} +**Judge model family:** {fam} ({fam_note}) + +## Outcome + +> **{result['outcome']}** + +Generated from the aggregate. Known-failure hits are GT-match verdicts (did an arm surface +the bug the fix proved mattered), human-confirmed per R6 before this record is trusted. + +## Primary signal — known-failure subset (GT-match hits) + +| Arm | Known-failure hits | Forward-rated (corroborating) | Trials | +|-----|--------------------|-------------------------------|--------| +{chr(10).join(rows)} + +{_yield_section(yield_per_arm)}## Validity checks + +- **Blind-integrity:** {integ_line}. +- **Negative control:** {control}. +- **Power:** corpus_n {result.get('corpus_n')} vs minimum_corpus_n {prereg.get('minimum_corpus_n')} — {power}. + +## What this does and does not conclude + +- It concludes whether a cross-model-critique lever surfaced the known bug often enough to + justify its carrying cost, on this corpus, against the pre-registered threshold. +- It does not decompose a self-critic win, and does not measure cross-machine setup fragility. +""" + + +def coverage(records, manifest): + """Did the run produce exactly the pre-registered docs x arms x trials records? + + Returns None when trials_per_arm is not a usable int (nothing to check against); + otherwise {expected, present, complete, missing}. Completeness is exact set + membership against the expected (arm, doc_id, trial) tuples derived from the + manifest — not a bare count. A count-only check would mark a run complete on the + wrong corpus (e.g. records for d1/d3 while the manifest expects d1/d2), and + finalize would then score/decide on a stale or mismatched set. `present` counts + only expected tuples that are actually covered, so extra/stale records never + inflate it toward `expected`. + """ + trials = manifest.get("pre_registration", {}).get("trials_per_arm") + if not isinstance(trials, int) or isinstance(trials, bool) or trials < 1: + return None + docs = manifest.get("docs", []) + expected_tuples = { + (arm, doc.get("id"), trial) + for doc in docs + for arm in ARMS + for trial in range(1, trials + 1) + } + present_tuples = {(r.get("arm"), r.get("doc_id"), r.get("trial")) for r in records} + covered = expected_tuples & present_tuples + missing = [ + {"arm": a, "doc_id": d, "trial": t} + for (a, d, t) in sorted( + expected_tuples - present_tuples, key=lambda x: (str(x[1]), str(x[0]), x[2]) + ) + ] + return { + "expected": len(expected_tuples), + "present": len(covered), + "complete": covered == expected_tuples, + "missing": missing, + } + + +def finalize(records_dir, manifest, gt_verdicts, class_verdicts=None, integrity=None, + judge_family=None, out=None, yield_verdicts=None): + records = load_records(records_dir) + + # Class verdicts cover forward_rated + negative_control only. A class verdict carrying + # subset "known_failure" would be scored via aggregate's decision_changing fallback, + # crediting a GT-match hit with no actual GT match — fail loud rather than corrupt the + # primary signal. + misrouted = [c for c in (class_verdicts or []) if c.get("subset") == "known_failure"] + if misrouted: + raise ValueError( + f"{len(misrouted)} class-verdict entry(ies) carry subset 'known_failure'; " + "known-failure scoring must come from --gt-verdicts (GT-match), not class verdicts " + "(which are for forward_rated + negative_control only)" + ) + + pool = run_arms.gt_pool(records) + gt_hits = run_arms.gt_hits_from_verdicts(pool["provenance"], gt_verdicts) + gt = run_arms.gt_score(manifest, gt_hits) + scored = list(gt["scored"]) + list(class_verdicts or []) + result = run_arms.aggregate(scored, manifest) + + # An incomplete record set cannot support ANY confident decision (R8/R9 spirit). + # This covers build_nothing as well as build:<arm>: a "don't build" verdict drawn + # from partial/interrupted data is a false negative, not a valid decision. An + # already-inconclusive outcome is left as-is but still annotated with the coverage gap. + cov = coverage(records, manifest) + if cov is not None and not cov["complete"]: + result = {**result, "outcome": "inconclusive", "incomplete_coverage": cov} + + # finding-yield: GT-match alone undercounts a reviewer that finds other real bugs + yield_per_arm = run_arms.yield_score(pool["provenance"], yield_verdicts) if yield_verdicts is not None else None + + integ = None + if integrity is not None: + correct, total = integrity + integ = run_arms.integrity_verdict(correct, total, len(ARMS)) + if integ.get("confounded"): + result = {**result, "outcome": "inconclusive", "confounded": True} + + artifact = render_artifact(manifest, result, gt, integ, judge_family, str(date.today()), yield_per_arm) + artifact_path = None + if out: + Path(out).write_text(artifact) + artifact_path = str(out) + + return { + "outcome": result["outcome"], + "artifact_path": artifact_path, + "per_arm": result["per_arm"], + "gt_per_arm": gt["per_arm"], + "yield_per_arm": yield_per_arm, + "known_failure_n": gt["known_failure_n"], + "below_n": result.get("below_n"), + "control_moved": result.get("control_moved"), + "coverage": cov, + "incomplete_coverage": result.get("incomplete_coverage"), + "tied_arms": result.get("tied_arms", []), + "integrity": integ, + } + + +def _parse_integrity(s): + if not s: + return None + correct, total = s.split(",", 1) + return (int(correct), int(total)) + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Code-review eval driver (deterministic spine).") + sub = parser.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("plan") + p.add_argument("manifest") + p.add_argument("--out-dir", required=True) + p.add_argument("--rubric", default="FILL:code-review-rubric.md") + p.add_argument("--context", default="FILL:arm-c-context.md") + p.add_argument("--cli-b", default="codex") + p.add_argument("--cli-c", default="gemini") + + p = sub.add_parser("finalize") + p.add_argument("records_dir") + p.add_argument("manifest") + p.add_argument("--gt-verdicts", required=True) + p.add_argument("--yield-verdicts", help="per-finding {uid, actionable, decision_changing, duplicate} for finding-yield") + p.add_argument("--class-verdicts") + p.add_argument("--integrity", help="correct,total of the judge arm-guessing probe") + p.add_argument("--judge-family") + p.add_argument("--out") + + args = parser.parse_args(argv) + + if args.cmd == "plan": + result = plan(_load(args.manifest), args.out_dir, args.rubric, args.context, args.cli_b, args.cli_c) + print(json.dumps(result)) + return 0 if result.get("ok") else 1 + + if args.cmd == "finalize": + class_verdicts = _load(args.class_verdicts) if args.class_verdicts else None + yield_verdicts = _load(args.yield_verdicts) if args.yield_verdicts else None + try: + result = finalize( + args.records_dir, + _load(args.manifest), + _load(args.gt_verdicts), + class_verdicts=class_verdicts, + integrity=_parse_integrity(args.integrity), + judge_family=args.judge_family, + out=args.out, + yield_verdicts=yield_verdicts, + ) + except ValueError as e: + print(json.dumps({"error": str(e)})) + return 1 + print(json.dumps(result)) + return 0 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/eval/cross_model_review/gt_match_rubric.md b/scripts/eval/cross_model_review/gt_match_rubric.md new file mode 100644 index 000000000..9af05c2a2 --- /dev/null +++ b/scripts/eval/cross_model_review/gt_match_rubric.md @@ -0,0 +1,56 @@ +# GT-match rubric (code-review breakpoint) + +The sharper known-failure metric a concrete fix commit unlocks. Where plan review can only +forward-rate whether a finding *looks* decision-changing, code review has a target: the bug +the fix proved mattered (`ground_truth.bug`). This rubric scores, per finding, whether the +arm surfaced **that** bug — the R7 primary signal, made objective. + +Used only on the **known_failure** subset (documents that are culprit diffs with a +`ground_truth` block). Forward-rated and negative-control documents keep the +`decision_changing` classification in `judge_rubric.md`. + +## Blinding (same contract as the main judge) + +The judge sees the label-stripped finding only (`run_arms.py strip-labels`) — never the arm. +It is dispatched **per finding, independently**, never batched, so it cannot pattern-match +across an arm's findings. Per-finding verdicts are re-attached to arms afterward by +`run_arms.py gt-resolve` (which collapses to a per-(arm,doc) `gt_hit`); the arm is recovered +there, not exposed here. Disclose the judge's model family in the result; if it shares a +family with any arm, flag the run a blind-integrity risk and run the integrity probe. + +## The match decision + +Given one document's `ground_truth.bug` and one label-stripped finding, decide: + +- `matches_bug: true` — the finding identifies the **same defect mechanism** the fix + repaired. It must name the actual failure, not merely touch the same area. +- `matches_bug: false` — anything else. + +Discipline (these are the failure modes that would inflate a hit rate): + +- **Same file, different bug is not a match.** A finding flagging an unrelated issue in the + culprit diff does not count, even if correct. +- **Generic caution is not a match.** "Validate inputs here" / "consider edge cases" does + not match a specific bug unless it names the specific failure. +- **Surface-wording overlap is not a match.** Matching is on the defect, not shared nouns + with the bug string. +- **Right mechanism, partial specificity can match.** If the finding correctly identifies + the failure the fix addressed (e.g. "this UNION mixes collations and will error at + runtime"), it matches even if worded differently from the fix subject. + +Score confidence on the discrete anchors only — **0, 25, 50, 75, 100**. Only `75`/`100` +verdicts are eligible to be `matches_bug: true`; `50` and below are `false`. + +## Output + +One verdict per finding: + +```json +{ "doc_id": "kf-7a6c84d", "finding_id": "f3", "matches_bug": true, "confidence": 100 } +``` + +Feed the verdicts to `run_arms.py gt-resolve <records.json> <verdicts.json>` to produce +per-(arm,doc) `gt_hit`, then `run_arms.py gt-score <manifest.json> <arm-matches.json>` for +the per-arm known-failure hit counts. Per R6, the human confirms the `true` verdicts and +samples the `false` set before the result is trusted — `trust: needs_confirmation` Tier-3 +corpus items make this confirmation non-optional. diff --git a/scripts/eval/cross_model_review/judge_rubric.md b/scripts/eval/cross_model_review/judge_rubric.md new file mode 100644 index 000000000..2f08470cc --- /dev/null +++ b/scripts/eval/cross_model_review/judge_rubric.md @@ -0,0 +1,40 @@ +# Blinded judge rubric (U5) + +The orchestrator dispatches this judge **per finding, independently** — never batched. +Batching lets the judge pattern-match across findings and recreates the cross-finding bias +blinding exists to escape (H3 / per `confidence-anchored-scoring`). + +The judge sees only the label-stripped finding (`run_arms.py strip-labels` removes arm, +trial, latency, model, cost, producer, status). It must not be told which arm produced it. +Disclose the judge's own model family in the result; if it shares a family with any arm +(e.g., a Claude judge over the Claude baseline / self-critic), flag the run as a +blind-integrity risk and run the integrity probe below. + +## Per-finding classification + +Classify each finding on three axes: + +- **uniqueness**: `unique` | `duplicate` (of another finding in the same document's pool) +- **actionability**: `actionable` | `generic` (a generic "be careful" / "ensure X" with no + specific, addressable claim is `generic`) +- **decision-changing**: `decision_changing` | `not` — would acting on this finding change + the plan/implementation decision? On the known-failure subset, the bar is: does it + surface the issue the post-hoc failure proved mattered? + +Score confidence on the discrete anchors only — **0, 25, 50, 75, 100** (no continuous +values, no "high"/"medium"/"low"). Anchors: + +- `100` — certain; evidence in the finding directly confirms it. +- `75` — double-checked; a competent implementer/reader concretely hits this. +- `50` — verified but advisory ("nothing breaks, but…"); not decision-changing on its own. +- `25` / `0` — not confident / false positive — do not surface. + +Only findings at `75`/`100` that are `unique` and `actionable` are eligible to be +`decision_changing`; the human confirms these (and samples the judge-rejected set) in U6. + +## Blind-integrity probe (R5) + +After classification, ask the judge to guess each finding's arm. Feed the guesses to +`run_arms.py integrity-verdict <correct> <total> <n_arms>`. If `confounded` is true (the +judge identifies arms above chance), the per-arm metric is treated as confounded and the +decision is `inconclusive`, not a build/no-build call. diff --git a/scripts/eval/cross_model_review/panel-critique.sh b/scripts/eval/cross_model_review/panel-critique.sh new file mode 100755 index 000000000..3caf4d9ac --- /dev/null +++ b/scripts/eval/cross_model_review/panel-critique.sh @@ -0,0 +1,123 @@ +#!/usr/bin/env bash +# Fair cross-model PANEL critique: run the cross-model arms through the SAME six review lenses the +# Claude ce-doc-review panel uses (coherence, feasibility, security, scope, product, adversarial), +# so a Claude-panel-vs-cross-model comparison isn't confounded by prompt asymmetry. Persists the +# FULL record per (model x lens) for post-hoc judging — nothing is truncated away. +# +# Usage: panel-critique.sh [--models <csv>] <doc.md> [context.md] +# Arms = codex + agy. agy is macOS-ONLY (read-only floor is a seatbelt). gemini was retired from +# the skill (it 410s 2026-06-18); the arms.py gemini arm remains for the cross-model eval. Records +# -> $CMRE_OUT_DIR (default /tmp/cmre-panel/records). Each run SENDS THE DOCUMENT to that vendor +# (codex -> OpenAI, agy -> Antigravity); arms can be slow — raise CMRE_TIMEOUT. +set -u + +here="$(cd "$(dirname "$0")" && pwd)" +arms="$here/arms.py" + +# Optional `--models <csv>` subset (default = all available arms = codex + agy). Lets a caller (e.g. +# ce-deep-review-beta's consent gate) restrict egress to exactly the consented models. Egress must +# equal consent, so the subset is filtered BEFORE running each arm -- never by discarding records +# post-hoc (the document would already have been sent). Unavailable / off-platform arms are +# warn-SKIPped per cell (not a missing binary, not agy off-macOS), never fatal: the rest still run. +models="codex agy" +if [ "${1:-}" = "--models" ]; then + models="$(printf '%s' "${2:-}" | tr ',' ' ')" + shift 2 +fi + +case "${1:-}" in -h|--help|"") sed -n '2,10p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; esac + +plan="$1"; context="${2:-}" +[ -f "$plan" ] || { echo "error: doc not found: '$plan'" >&2; exit 2; } +if [ -n "$context" ] && [ ! -f "$context" ]; then echo "error: context not found: '$context'" >&2; exit 2; fi + +# The agy arm's deny-write floor must deny writes to the REVIEWED document's repo, not arms.py's own +# location (matters for the installed skill reviewing a user's plan). Resolve it from the plan's +# directory; fall back to that directory when the plan isn't inside a git repo. arms.py reads CMRE_REPO_DIR. +plan_dir="$(cd "$(dirname "$plan")" && pwd)" +CMRE_REPO_DIR="$(git -C "$plan_dir" rev-parse --show-toplevel 2>/dev/null || printf '%s' "$plan_dir")" +export CMRE_REPO_DIR + +out="${CMRE_OUT_DIR:-/tmp/cmre-panel}"; rec_dir="$out/records"; mkdir -p "$rec_dir" +arm="b_isolated"; [ -n "$context" ] && arm="c_fixed_context" +doc_id="$(basename "$plan" .md)" +timeout="${CMRE_TIMEOUT:-420}" + +# Six lens rubrics, distilled from the ce-doc-review personas so the cross-model arms get the +# same coverage the Claude panel does. +lens_dir="$(mktemp -d -t cmre-lenses-XXXXXX)" +cat > "$lens_dir/coherence.md" <<'EOF' +Review this document for INTERNAL CONSISTENCY: contradictions between sections, terminology drift, +dependency/sequencing claims that conflict, and ambiguity where two readers would diverge. Return +your findings as a JSON array of strings, one element per distinct finding; quote the conflicting text in each. +EOF +cat > "$lens_dir/feasibility.md" <<'EOF' +Review whether the proposed approach will SURVIVE CONTACT WITH REALITY: architecture conflicts, +dependency gaps, migration/cutover risks, environment assumptions, implementability. Challenge the +load-bearing claims. Return your findings as a JSON array of strings, one element per distinct finding; name the concrete risk in each. +EOF +cat > "$lens_dir/security.md" <<'EOF' +Review for SECURITY gaps: auth/authz assumptions, data exposure, credential handling, trust +boundaries, PII, and missing threat-model elements. Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/scope.md" <<'EOF' +Review for SCOPE alignment and unjustified complexity: abstractions/frameworks larger than the goal +needs, scope creep beyond stated intent, premature generality, dependencies declared but not needed. +Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/product.md" <<'EOF' +Review as a senior PRODUCT leader: are the premises sound? What strategic/adoption/trust +consequences (including for the people the system affects) does this carry even if the premise +holds? Where does the work drift from the goal? Return your findings as a JSON array of strings, one element per distinct finding. +EOF +cat > "$lens_dir/adversarial.md" <<'EOF' +ADVERSARIALLY stress-test this document: surface unstated assumptions, construct failure modes the +mitigations do not actually cover, name the cheaper/safer alternative it dismissed, and find any +irreversible step taken before its validation. Try to BREAK it. Return your findings as a JSON array of strings, one element per distinct finding. +EOF + +run() { + cli="$1"; lens="$2" + if ! command -v "$cli" >/dev/null 2>&1; then + printf ' [%-7s %-12s] SKIP — %s not installed\n' "$cli" "$lens" "$cli"; return + fi + if [ "$cli" = "agy" ] && [ "$(uname -s)" != "Darwin" ]; then + printf ' [%-7s %-12s] SKIP — agy is macOS-only (read-only floor is a seatbelt)\n' "$cli" "$lens"; return + fi + cmd=(run-arm "$arm" "$cli" "$plan" "$lens_dir/$lens.md" --doc-id "${doc_id}__${lens}" --trial 1 --timeout "$timeout") + [ -n "$context" ] && cmd+=(--context "$context") + rec="$(python3 "$arms" "${cmd[@]}" 2>/dev/null)" + printf '%s' "$rec" > "$rec_dir/${cli}__${lens}.json" + n="$(printf '%s' "$rec" | python3 -c 'import json,sys +try: + print(len(json.load(sys.stdin)["findings"])) +except Exception: + print("ERR")')" + printf ' [%-7s %-12s] findings=%s\n' "$cli" "$lens" "$n" +} + +echo "Panel critique of: $plan (arm=$arm)" +echo "Full records -> $rec_dir" +echo "Models: $models (each runs all 6 lenses; models run in parallel — progress lines interleave)" + +# One background subshell PER MODEL, each running the six lenses sequentially. Parallelizing across +# models (not across lenses) overlaps the slow arms while bounding concurrency to the model count -- +# at most one in-flight request per vendor, which avoids rate-limit / resource contention. Each +# (model, lens) cell streams its own self-labeled progress line as it completes (R15: no silent +# multi-minute runs); lines from different models interleave, which is fine. Records key on +# ${cli}__${lens}.json, so parallel writers never collide. +run_model() { + cli="$1" + for lens in coherence feasibility security scope product adversarial; do + run "$cli" "$lens" + done +} +pids="" +for cli in $models; do + run_model "$cli" & + pids="$pids $!" +done +for pid in $pids; do wait "$pid"; done + +echo "" +echo "DONE. Full records in $rec_dir — read them for the per-lens findings." diff --git a/scripts/eval/cross_model_review/prompts/baseline.md b/scripts/eval/cross_model_review/prompts/baseline.md new file mode 100644 index 000000000..21ac1484a --- /dev/null +++ b/scripts/eval/cross_model_review/prompts/baseline.md @@ -0,0 +1,21 @@ +# Arm (a) — Claude-only baseline prompt + +This is the control arm. The orchestrator (the agent running the eval) dispatches a +standard Claude review of the corpus document — the same behavior `ce-doc-review` produces +today — as an in-process subagent dispatch. No external CLI, no special instructions +beyond the normal review. + +Dispatch instructions for the orchestrator: + +1. For each corpus document, for each trial (1..`trials_per_arm`), dispatch a reviewer + subagent with the document and the standard independent-challenge rubric + (`tests/fixtures/cross-model-review/sample-rubric.md` shape, or the real rubric). +2. Collect the subagent's findings. +3. Write a schema-conformant record (`record-schema.json`) with `arm: "a_baseline"`, + `producer: "orchestrator"`, the `doc_id`, the `trial`, `status: "ok"`, the measured + `latency_ms`, and `findings`. +4. Ingest each record into the shared run dir: + `python3 run_arms.py ingest <run_dir> <record.json>`. + +The baseline establishes what the current single-model review surfaces, against which the +cross-model and self-critic arms are measured. diff --git a/scripts/eval/cross_model_review/prompts/self-critic.md b/scripts/eval/cross_model_review/prompts/self-critic.md new file mode 100644 index 000000000..f4468a076 --- /dev/null +++ b/scripts/eval/cross_model_review/prompts/self-critic.md @@ -0,0 +1,32 @@ +# Arm (d) — Same-model self-critic prompt + +The cheaper alternative the eval must rule in or out: the same model (Claude) re-reviews +the document **in-process** — no external CLI, no document egress (AE4) — but primed to +catch its own blind spots and without seeing its prior review. + +The orchestrator dispatches this as an in-process subagent with the prompt below. A +self-critic "win" is attributed to the bundled intervention (fresh pass + failure-modes +supplied) and is **not** decomposed within this eval (per origin R3). + +--- + +You are reviewing the document on its own terms, as if for the first time. You have NOT +seen any prior review of it — do not assume one exists. + +Before you review, internalize these known failure modes of your own model family, and +hunt specifically for findings a first-pass review of yours would have missed: + +- Accepting a plausible-sounding premise without demanding evidence. +- Over-engineering: endorsing abstractions, config, and machinery beyond what the goal needs. +- Sycophancy toward the document's framing — restating its goals as if they were validated. +- Missing the cheaper alternative that would dominate the proposed design. +- Treating "a critique exists" as success rather than "the decision would change." + +Now challenge the document: question the premise, surface unstated assumptions, name +unconsidered alternatives, and state what would falsify it. Return your findings as a JSON array of strings, one element per distinct finding. + +--- + +Run instructions for the orchestrator: produce this review per (document × trial), write a +schema-conformant record (`arm: "d_self_critic"`, `producer: "orchestrator"`, no external +call made), and ingest it into the shared run dir via `python3 run_arms.py ingest`. diff --git a/scripts/eval/cross_model_review/record-schema.json b/scripts/eval/cross_model_review/record-schema.json new file mode 100644 index 000000000..437c4145c --- /dev/null +++ b/scripts/eval/cross_model_review/record-schema.json @@ -0,0 +1,49 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Cross-Model Review Eval Record", + "description": "One record per (document x arm x trial). Both producers write schema-conformant record files into the shared run directory: the Python runner writes records for the CLI arms (b_isolated, c_fixed_context); the orchestrator writes records for the in-process arms (a_baseline, d_self_critic) and the judge. Aggregation pools by reading every record file in the run dir, regardless of producer.", + "type": "object", + "required": ["arm", "doc_id", "trial", "status", "producer", "latency_ms", "findings"], + "additionalProperties": false, + "properties": { + "arm": { + "type": "string", + "enum": ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"] + }, + "doc_id": { "type": "string" }, + "trial": { "type": "integer", "minimum": 1 }, + "status": { + "type": "string", + "enum": ["ok", "degraded", "timeout", "error"], + "description": "degraded = circuit breaker tripped for this arm; timeout = per-arm timeout hit; error = invocation failed." + }, + "producer": { + "type": "string", + "enum": ["runner", "orchestrator"], + "description": "Which half wrote this record. The runner's timeout/circuit-breaker apply only to its own (CLI) records; orchestrator records are ingested as-is." + }, + "latency_ms": { "type": "number", "minimum": 0 }, + "findings": { + "type": "array", + "description": "Free-form findings the arm produced. Empty when status != ok or the arm found nothing.", + "items": { + "type": "object", + "required": ["id", "text"], + "additionalProperties": false, + "properties": { + "id": { "type": "string" }, + "text": { "type": "string" } + } + } + }, + "model": { + "type": "string", + "description": "Identifying field. Stripped before the blinded judge sees the record (U5)." + }, + "cost": { + "type": "object", + "description": "Optional cost/tool-call signal (e.g., from codex exec --json). Identifying; stripped before judging.", + "additionalProperties": true + } + } +} diff --git a/scripts/eval/cross_model_review/run-judge.sh b/scripts/eval/cross_model_review/run-judge.sh new file mode 100755 index 000000000..6a6c030ad --- /dev/null +++ b/scripts/eval/cross_model_review/run-judge.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Non-Claude blind judge for the decision-grade run. Scores the pooled, label-stripped findings +# with codex (OpenAI) or gemini (Google) -- NOT Claude -- which is what clears the judge-family +# confound (R4: a Claude judge may under-rate non-Claude-shaped findings). The judge sees finding +# uids + text + each doc's ground-truth bug, never the arm; arms are re-attached afterward via +# `run_arms.py gt-resolve` so the blind holds. +# +# Usage: run-judge.sh <pool.json> <manifest.json> <codex|gemini> [out-verdicts.json] +# pool.json output of `run_arms.py gt-pool <records.json>` +# manifest.json the corpus manifest (carries each known_failure doc's ground_truth.bug) +# Sends the findings to that vendor. gemini needs GEMINI_API_KEY. Verdicts -> out (default +# /tmp/cmre-judge-verdicts.json), ready for gt-resolve / gt-score / yield-score / aggregate. +set -uo pipefail + +here="$(cd "$(dirname "$0")" && pwd)" +runarms="$here/run_arms.py" + +case "${1:-}" in -h|--help|"") sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//'; exit 0;; esac + +pool="$1"; manifest="$2"; cli="${3:-codex}"; out="${4:-/tmp/cmre-judge-verdicts.json}" +[ -f "$pool" ] || { echo "error: pool not found: '$pool'" >&2; exit 2; } +[ -f "$manifest" ] || { echo "error: manifest not found: '$manifest'" >&2; exit 2; } +command -v "$cli" >/dev/null 2>&1 || { echo "error: judge cli '$cli' not installed" >&2; exit 2; } + +prompt="$(python3 "$runarms" judge-prompt "$pool" "$manifest")" +cwd="$(mktemp -d -t cmre-judge-cwd-XXXXXX)" # clean cwd: the judge gets no ambient workspace +raw="/tmp/cmre-judge-raw.txt" + +# pipefail (above) makes the pipeline's exit reflect the judge CLI, not the leading printf, +# so an auth/quota/runtime failure is caught rather than silently producing []. We must not +# let a failed judge run be consumed downstream as if it were valid evidence (it would flip +# the decision to build_nothing/inconclusive); fail fast instead. +rc=0 +case "$cli" in + codex) + ( cd "$cwd" && printf '%s' "$prompt" | codex exec -s read-only --skip-git-repo-check - ) > "$raw" 2>/dev/null || rc=$? + ;; + gemini) + ( cd "$cwd" && printf '%s' "$prompt" | gemini -p "Return ONLY the JSON array of verdicts." --approval-mode plan --skip-trust -o text ) > "$raw" 2>/dev/null || rc=$? + ;; + *) + echo "error: judge cli must be 'codex' or 'gemini' (a non-Claude judge)" >&2; exit 2;; +esac + +if [ "$rc" -ne 0 ]; then + echo "error: judge cli '$cli' exited $rc — refusing to emit verdicts from a failed run (see $raw)" >&2 + exit 3 +fi + +python3 "$runarms" judge-parse "$raw" > "$out" +n="$(python3 -c "import json;print(len(json.load(open('$out')))) " 2>/dev/null || echo ERR)" +echo "judge=$cli verdicts=$n -> $out" +if [ "$n" = "0" ] || [ "$n" = "ERR" ]; then + echo "error: judge produced 0 parseable verdicts — check $raw (prose instead of JSON, or a quota/auth error). Not handing an empty verdict set to finalize." >&2 + exit 4 +fi diff --git a/scripts/eval/cross_model_review/run_arms.py b/scripts/eval/cross_model_review/run_arms.py new file mode 100644 index 000000000..f02b0c59c --- /dev/null +++ b/scripts/eval/cross_model_review/run_arms.py @@ -0,0 +1,562 @@ +#!/usr/bin/env python3 +"""Cross-model review evaluation harness — runner skeleton (U2). + +Owns the deterministic carrier of the eval: the canonical record contract, the +shared-run-dir record store, the per-arm timeout + circuit breaker for the CLI +arms it spawns, and the label-stripping transform the blinded judge depends on. + +The actual CLI-arm invocation lives in arms.py (U3); the in-process arms (a, d) +and the judge are produced by the orchestrator (U4/U5) and ingested here as +schema-conformant record files. Aggregation pools by reading every record file +in the run dir, regardless of which producer wrote it. + +Subcommands are intentionally small and deterministic so they are unit-testable +via Bun.spawn(["python3", ...]) without invoking any model. +""" + +import argparse +import hashlib +import json +import re +import sys +from pathlib import Path + +ARMS = ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"] +CONTROL_ARM = "a_baseline" # baseline = existing behavior; a control, never a buildable lever +CLI_ARMS = {"b_isolated", "c_fixed_context"} # spawned by the runner; subject to timeout/breaker +STATUSES = ["ok", "degraded", "timeout", "error"] +PRODUCERS = ["runner", "orchestrator"] +IDENTIFYING_FIELDS = ["arm", "trial", "latency_ms", "model", "cost", "producer", "status"] +DEFAULT_BREAKER_THRESHOLD = 3 +KNOWN_KEYS = {"arm", "doc_id", "trial", "status", "producer", "latency_ms", "findings", "model", "cost"} + + +def validate_record(rec): + """Return a list of human-readable errors; empty list means valid.""" + errors = [] + if not isinstance(rec, dict): + return ["record is not a JSON object"] + for field in ("arm", "doc_id", "trial", "status", "producer", "latency_ms", "findings"): + if field not in rec: + errors.append(f"missing required field: {field}") + for key in rec: + if key not in KNOWN_KEYS: + errors.append(f"unknown field: {key}") + if rec.get("arm") not in ARMS: + errors.append(f"arm must be one of {ARMS}") + if rec.get("status") not in STATUSES: + errors.append(f"status must be one of {STATUSES}") + if rec.get("producer") not in PRODUCERS: + errors.append(f"producer must be one of {PRODUCERS}") + if not isinstance(rec.get("doc_id"), str) or not rec.get("doc_id"): + errors.append("doc_id must be a non-empty string") + trial = rec.get("trial") + if not isinstance(trial, int) or isinstance(trial, bool) or trial < 1: + errors.append("trial must be an integer >= 1") + latency = rec.get("latency_ms") + if not isinstance(latency, (int, float)) or isinstance(latency, bool) or latency < 0: + errors.append("latency_ms must be a number >= 0") + findings = rec.get("findings") + if not isinstance(findings, list): + errors.append("findings must be an array") + else: + for i, f in enumerate(findings): + if not isinstance(f, dict) or "id" not in f or "text" not in f: + errors.append(f"findings[{i}] must be an object with id and text") + return errors + + +def corpus_status(manifest): + """Compute corpus size vs the pre-registered minimum N.""" + docs = manifest.get("docs", []) + prereg = manifest.get("pre_registration", {}) + minimum = prereg.get("minimum_corpus_n") + trials = prereg.get("trials_per_arm") + corpus_n = len(docs) + below_n = isinstance(minimum, int) and corpus_n < minimum + return { + "corpus_n": corpus_n, + "minimum_corpus_n": minimum, + "below_n": below_n, + "trials_per_arm": trials, + # An eval below minimum N reports "inconclusive", never "build nothing" (R9). + "outcome_floor": "inconclusive" if below_n else "decidable", + } + + +def strip_labels(rec): + """Remove arm-identifying fields before the blinded judge sees a record. + + Keeps doc_id (the judge dedups within a document) and findings; drops every + field that could betray which arm produced it (U5 / FE4 / H3). + """ + return {k: v for k, v in rec.items() if k not in IDENTIFYING_FIELDS} + + +def breaker_should_disable(consecutive_failures, threshold=DEFAULT_BREAKER_THRESHOLD): + """Pure circuit-breaker decision: disable an arm after N consecutive failures.""" + return consecutive_failures >= threshold + + +def _as_bool(v): + """Strict-but-tolerant truthiness for judge verdict fields. + + A non-Claude judge (codex/gemini) may serialize booleans loosely, e.g. the + string ``"false"`` — which is Python-truthy and would silently inflate a metric. + Accept only a real bool or a ``"true"``/``"false"`` string; everything else + (including ``"false"``, ``1``, ``None``) is False. This keeps a loose judge from + flipping the decision via stray truthiness while still honoring string booleans. + """ + if isinstance(v, bool): + return v + if isinstance(v, str): + return v.strip().lower() == "true" + return False + + +def _slug(value): + """Filesystem-safe token for a record filename component. + + `doc_id` is a free-form non-empty string (an id copied from a doc path like + `docs/plans/x.md` is legal), so it can carry path separators or other chars + that would make `write_text` target a nested/nonexistent path and raise. We + only sanitize the on-disk filename; the record itself keeps the true doc_id, + so pool/gt-pool still read the real value from inside the file. + """ + return re.sub(r"[^A-Za-z0-9._-]+", "_", str(value)) or "_" + + +def ingest(run_dir, record): + """Validate an externally-produced (orchestrator) record and write it into the store.""" + errors = validate_record(record) + if errors: + raise ValueError("; ".join(errors)) + run_path = Path(run_dir) + run_path.mkdir(parents=True, exist_ok=True) + name = f"{_slug(record['arm'])}__{_slug(record['doc_id'])}__t{_slug(record['trial'])}.json" + out = run_path / name + out.write_text(json.dumps(record, indent=2)) + return str(out) + + +def pool(run_dir): + """Read every record file in the run dir and tally by arm.""" + run_path = Path(run_dir) + by_arm = {arm: 0 for arm in ARMS} + total = 0 + invalid = 0 + for f in sorted(run_path.glob("*.json")): + try: + rec = json.loads(f.read_text()) + except json.JSONDecodeError: + invalid += 1 + continue + if validate_record(rec): + invalid += 1 + continue + by_arm[rec["arm"]] = by_arm.get(rec["arm"], 0) + 1 + total += 1 + return {"total": total, "by_arm": by_arm, "invalid": invalid} + + +def dedup_findings(items): + """Group pooled findings by normalized text — the cross-arm agreement signal (U5). + + items: list of {arm, text}. Returns [{text, arms:[...], count}] so it is + visible when multiple arms independently raised the same point. This is a + text-normalized dedup, not the full scope-aware peer/nested merge; the eval + only needs cross-arm agreement, not the review pipeline's chaining. + """ + groups = {} + order = [] + for it in items: + key = " ".join(str(it.get("text", "")).lower().split()) + if key not in groups: + groups[key] = {"text": it.get("text", ""), "arms": [], "count": 0} + order.append(key) + g = groups[key] + g["count"] += 1 + arm = it.get("arm") + if arm and arm not in g["arms"]: + g["arms"].append(arm) + return [groups[k] for k in order] + + +def integrity_verdict(correct, total, n_arms, margin=0.15): + """Blind-integrity check (U5 / R5): can the judge identify arms above chance? + + chance = 1/n_arms. If the judge's arm-guess accuracy exceeds chance by more + than `margin`, the blind did not hold and the per-arm metric is confounded. + """ + if total <= 0 or n_arms <= 0: + return {"accuracy": None, "chance": None, "confounded": False} + accuracy = correct / total + chance = 1.0 / n_arms + return {"accuracy": accuracy, "chance": chance, "confounded": accuracy > chance + margin} + + +def _finding_uid(arm, doc_id, fid, text): + """Opaque, globally-unique id for one arm's finding. + + A content+arm hash: unique per (arm, finding) so two arms reusing a local id + like `f1` get different uids; order-independent so the pool and finalize agree; + and it does not visibly encode the arm, so the judge stays blind. + """ + h = hashlib.sha1(f"{arm}\x1f{doc_id}\x1f{fid}\x1f{text}".encode("utf-8")).hexdigest() + return "g" + h[:12] + + +def gt_pool(records): + """Build the blind GT-match pool + arm provenance from arm records. + + Returns `pool` (what the judge sees: uid + doc_id + text, NO arm, ordered by + opaque uid so order leaks nothing) and `provenance` (uid -> {arm, doc_id}, the + private map the runner uses to resolve verdicts back to arms). Fixes the earlier + cross-arm credit bleed where a verdict keyed on a local finding id (`f2`) credited + every arm that happened to reuse it. + """ + pool, provenance = [], {} + for rec in records: + arm, doc = rec.get("arm"), rec.get("doc_id") + for f in rec.get("findings", []): + uid = _finding_uid(arm, doc, f.get("id"), f.get("text", "")) + provenance[uid] = {"arm": arm, "doc_id": doc} + pool.append({"uid": uid, "doc_id": doc, "text": f.get("text", "")}) + pool.sort(key=lambda p: p["uid"]) + return {"pool": pool, "provenance": provenance} + + +def gt_hits_from_verdicts(provenance, verdicts): + """Resolve blind per-finding verdicts (keyed by pool uid) to per-(arm,doc) gt_hit. + + A `matches_bug` verdict credits only the one arm whose finding carries that uid; + `gt_hit` for an (arm,doc) is true if any of that arm's findings on the doc matched. + """ + matched = {v.get("uid") for v in verdicts if _as_bool(v.get("matches_bug"))} + hits = {} + for uid, p in provenance.items(): + key = (p.get("arm"), p.get("doc_id")) + hits.setdefault(key, False) + if uid in matched: + hits[key] = True + return [{"arm": a, "doc_id": d, "gt_hit": h} for (a, d), h in hits.items()] + + +def build_judge_prompt(pool, manifest): + """Build the BLIND judge prompt for a non-Claude judge (codex/gemini). + + The judge sees each document's ground-truth bug and the pooled findings by uid + and text only — never the arm (that's recovered after, preserving the blind). + Asks for a JSON array of per-finding verdicts so parsing is reliable. Using a + non-Claude judge is what clears the judge-family confound (R4). + """ + gt = {d.get("id"): (d.get("ground_truth") or {}).get("bug") + for d in manifest.get("docs", []) if d.get("subset") == "known_failure"} + subset_by_doc = {d.get("id"): d.get("subset") for d in manifest.get("docs", [])} + by_doc = {} + for p in pool.get("pool", []): + by_doc.setdefault(p.get("doc_id"), []).append(p) + lines = [ + "You are a blind reviewer-of-reviews. Below are code/plan-review findings grouped by", + "document. For EACH finding decide three booleans: matches_bug (does it describe THIS", + "document's known bug?), actionable (a specific, addressable defect, not a generic", + "caution), decision_changing (a competent owner would act on it).", + "", + "Return ONLY a JSON array of objects {uid, matches_bug, actionable, decision_changing}.", + "No prose. Judge each finding on its own text; you are NOT told which reviewer produced it.", + "", + ] + for doc, items in by_doc.items(): + if doc in gt and gt[doc]: + lines.append(f"## Document {doc} — the bug a good review should have caught: {gt[doc]}") + elif subset_by_doc.get(doc) == "negative_control": + lines.append(f"## Document {doc} — NEGATIVE CONTROL: no real bug exists; matches_bug must be false for all.") + else: + # forward_rated (and any non-known_failure, non-negative_control doc): there is no + # pre-registered bug to match, so judge each finding neutrally on its own merits. + # Asserting "no real bug exists" here would bias legitimate forward-rated findings + # downward in the yield/precision signal that reuses these verdicts. + lines.append(f"## Document {doc} — no pre-registered bug; matches_bug must be false. Still judge actionable/decision_changing on each finding's own merits.") + for p in items: + lines.append(f"- uid={p.get('uid')}: {p.get('text','')}") + lines.append("") + return "\n".join(lines) + + +def parse_judge_verdicts(text): + """Parse the judge's JSON verdict array (tolerating a ```json fence). [] on failure. + + Requires an array of verdict *objects*: a non-empty array of non-dicts (e.g. the + judge emitted a list of strings) is rejected as [] so run-judge.sh treats it as a + parse failure instead of handing downstream `v.get(...)` entries it will crash on. + """ + t = (text or "").strip() + if t.startswith("```"): + t = re.sub(r"^```[a-zA-Z0-9]*\n?", "", t) + t = re.sub(r"\n?```$", "", t).strip() + try: + data = json.loads(t) + except (json.JSONDecodeError, ValueError): + return [] + if not isinstance(data, list): + return [] + if not all(isinstance(v, dict) for v in data): + return [] + return data + + +def yield_score(provenance, verdicts): + """Per-arm finding-yield: the value GT-match alone misses. + + GT-match only credits surfacing the one bug a historical fix targeted, so a + competent reviewer that finds *other* real bugs scores zero. This tallies, per + arm, total findings and how many the blind judge classified unique-actionable + and decision-changing (verdicts keyed by pool uid, resolved to arms here). + verdicts: [{uid, actionable, decision_changing, duplicate?}]. + """ + by_uid = {v.get("uid"): v for v in verdicts} + per_arm = {arm: {"total": 0, "unique_actionable": 0, "decision_changing": 0} for arm in ARMS} + for uid, p in provenance.items(): + arm = p.get("arm") + if arm not in per_arm: + per_arm[arm] = {"total": 0, "unique_actionable": 0, "decision_changing": 0} + per_arm[arm]["total"] += 1 + v = by_uid.get(uid) + if not v: + continue + if _as_bool(v.get("actionable")) and not _as_bool(v.get("duplicate")): + per_arm[arm]["unique_actionable"] += 1 + if _as_bool(v.get("decision_changing")): + per_arm[arm]["decision_changing"] += 1 + return per_arm + + +def gt_score(manifest, arm_matches): + """Per-arm hit counts on the known-failure subset (R7 primary metric). + + arm_matches: [{arm, doc_id, gt_hit}] (e.g. from gt_hits_from_findings). Only + verdicts on known_failure documents count; everything else is ignored. Returns + per-arm {hits, scored} plus aggregate-ready known_failure records carrying gt_hit. + """ + known_failure = {d.get("id") for d in manifest.get("docs", []) if d.get("subset") == "known_failure"} + per_arm = {arm: {"hits": 0, "scored": 0} for arm in ARMS} + scored = [] + for m in arm_matches: + if m.get("doc_id") not in known_failure: + continue + arm = m.get("arm") + if arm not in per_arm: + continue + hit = bool(m.get("gt_hit")) + per_arm[arm]["scored"] += 1 + per_arm[arm]["hits"] += int(hit) + scored.append({"arm": arm, "doc_id": m["doc_id"], "subset": "known_failure", "gt_hit": hit}) + return {"per_arm": per_arm, "scored": scored, "known_failure_n": len(known_failure)} + + +def aggregate(scored, manifest): + """Aggregate post-judge, human-confirmed findings into a three-way decision (U6). + + scored: list of {arm, doc_id, subset, decision_changing: bool, gt_hit?: bool}. + The primary signal is the per-arm score on the known-failure subset; forward-rated + counts corroborate (R7). On known_failure the predicate is `gt_hit` when present + (the targeted code-review GT-match), falling back to `decision_changing` (plan + review's forward-rated judgment); other subsets always use `decision_changing`. + Below minimum N, or if the negative control moved, the outcome is inconclusive + rather than "build nothing" (R9, H2). + """ + prereg = manifest.get("pre_registration", {}) + threshold = prereg.get("go_threshold") + status = corpus_status(manifest) + per_arm = {arm: {"known_failure": 0, "forward_rated": 0} for arm in ARMS} + control_moved = False + for s in scored: + subset, arm = s.get("subset"), s.get("arm") + if subset == "known_failure" and "gt_hit" in s: + positive = s.get("gt_hit") + else: + positive = s.get("decision_changing") + if not positive: + continue + if subset == "negative_control": + control_moved = True + elif subset in ("known_failure", "forward_rated") and arm in per_arm: + per_arm[arm][subset] += 1 + + # The control arm (baseline = existing behavior) is never a buildable lever, so it is + # excluded from winner selection — otherwise a baseline that happened to score highest + # could emit a nonsensical `build:a_baseline`. Among the remaining levers, a tie at the + # top is not evidence for any single one; the tie-break policy is "inconclusive", not + # "first arm in enum order wins". + candidates = {arm: c["known_failure"] for arm, c in per_arm.items() if arm != CONTROL_ARM} + best = max(candidates.values()) if candidates else 0 + top_arms = [arm for arm, n in candidates.items() if n == best] + tied = best > 0 and len(top_arms) > 1 + winning_arm = top_arms[0] if len(top_arms) == 1 else None + + if status["below_n"]: + outcome = "inconclusive" + elif control_moved: + outcome = "inconclusive" # negative control moved -> harness stability problem (H2) + elif isinstance(threshold, int) and best >= threshold and best > 0: + # tie among levers clearing the threshold -> no single buildable winner + outcome = "inconclusive" if tied else f"build:{winning_arm}" + else: + outcome = "build_nothing" + + return { + "outcome": outcome, + "winning_arm": winning_arm if outcome.startswith("build:") else None, + "tied_arms": top_arms if tied else [], + "per_arm": per_arm, + "control_moved": control_moved, + "below_n": status["below_n"], + "corpus_n": status["corpus_n"], + "go_threshold": threshold, + } + + +def _load(path): + return json.loads(Path(path).read_text()) + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Cross-model review eval runner (deterministic carrier).") + sub = parser.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("validate-record") + p.add_argument("record") + + p = sub.add_parser("corpus-status") + p.add_argument("manifest") + + p = sub.add_parser("strip-labels") + p.add_argument("record") + + p = sub.add_parser("breaker-check") + p.add_argument("consecutive_failures", type=int) + p.add_argument("--threshold", type=int, default=DEFAULT_BREAKER_THRESHOLD) + + p = sub.add_parser("ingest") + p.add_argument("run_dir") + p.add_argument("record", help="path to a record JSON, or '-' to read the record from stdin") + + p = sub.add_parser("pool") + p.add_argument("run_dir") + + p = sub.add_parser("dedup") + p.add_argument("items") + + p = sub.add_parser("integrity-verdict") + p.add_argument("correct", type=int) + p.add_argument("total", type=int) + p.add_argument("n_arms", type=int) + p.add_argument("--margin", type=float, default=0.15) + + p = sub.add_parser("gt-pool") + p.add_argument("records", nargs="+", + help="one JSON file holding an array of records, OR several files each " + "holding one record (so a glob like <run>/records/*.json works)") + + p = sub.add_parser("gt-resolve") + p.add_argument("provenance") + p.add_argument("verdicts") + + p = sub.add_parser("gt-score") + p.add_argument("manifest") + p.add_argument("arm_matches") + + p = sub.add_parser("yield-score") + p.add_argument("provenance") + p.add_argument("verdicts") + + p = sub.add_parser("judge-prompt") + p.add_argument("pool") + p.add_argument("manifest") + + p = sub.add_parser("judge-parse") + p.add_argument("output") + + p = sub.add_parser("aggregate") + p.add_argument("scored") + p.add_argument("manifest") + + args = parser.parse_args(argv) + + if args.cmd == "validate-record": + errors = validate_record(_load(args.record)) + print(json.dumps({"valid": not errors, "errors": errors})) + return 0 if not errors else 1 + + if args.cmd == "corpus-status": + print(json.dumps(corpus_status(_load(args.manifest)))) + return 0 + + if args.cmd == "strip-labels": + print(json.dumps(strip_labels(_load(args.record)))) + return 0 + + if args.cmd == "breaker-check": + print(json.dumps({"disable": breaker_should_disable(args.consecutive_failures, args.threshold)})) + return 0 + + if args.cmd == "ingest": + try: + # `-` reads the record from stdin so `run-arm ... | ingest <dir> -` works. + rec = json.loads(sys.stdin.read()) if args.record == "-" else _load(args.record) + written = ingest(args.run_dir, rec) + except (ValueError, json.JSONDecodeError, OSError) as e: + print(json.dumps({"written": None, "error": str(e)})) + return 1 + print(json.dumps({"written": written})) + return 0 + + if args.cmd == "pool": + print(json.dumps(pool(args.run_dir))) + return 0 + + if args.cmd == "dedup": + print(json.dumps(dedup_findings(_load(args.items)))) + return 0 + + if args.cmd == "integrity-verdict": + print(json.dumps(integrity_verdict(args.correct, args.total, args.n_arms, args.margin))) + return 0 + + if args.cmd == "gt-pool": + recs = [] + for path in args.records: + data = _load(path) + recs.extend(data if isinstance(data, list) else [data]) + print(json.dumps(gt_pool(recs))) + return 0 + + if args.cmd == "gt-resolve": + print(json.dumps(gt_hits_from_verdicts(_load(args.provenance), _load(args.verdicts)))) + return 0 + + if args.cmd == "gt-score": + print(json.dumps(gt_score(_load(args.manifest), _load(args.arm_matches)))) + return 0 + + if args.cmd == "yield-score": + print(json.dumps(yield_score(_load(args.provenance), _load(args.verdicts)))) + return 0 + + if args.cmd == "judge-prompt": + print(build_judge_prompt(_load(args.pool), _load(args.manifest))) + return 0 + + if args.cmd == "judge-parse": + print(json.dumps(parse_judge_verdicts(Path(args.output).read_text()))) + return 0 + + if args.cmd == "aggregate": + print(json.dumps(aggregate(_load(args.scored), _load(args.manifest)))) + return 0 + + return 2 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/eval/cross_model_review/validation/agy-readonly.sb.tmpl b/scripts/eval/cross_model_review/validation/agy-readonly.sb.tmpl new file mode 100644 index 000000000..1e160038d --- /dev/null +++ b/scripts/eval/cross_model_review/validation/agy-readonly.sb.tmpl @@ -0,0 +1,21 @@ +;; ce-deep-review — agy cross-model arm filesystem floor (validated 2026-05-28, Phase 0 / U2). +;; +;; agy 1.0.3 is a heavy Electron/Node agent: a strict `(deny file-write*)` allowlist OR any +;; `(deny file-read* ...)` rule makes it HANG indefinitely (it retries the denied syscall and +;; ignores its own --print-timeout). So this is a deny-WRITE-only denylist: allow-default (agy +;; writes its own state/caches freely -> no hang) and deny writes only to the paths that matter +;; (the repo under review + home credentials/dotfiles). Network is allowed (the arm needs its +;; vendor API). Reads are NOT denied -> secret-read-exfil is a documented residual, mitigated by +;; a clean cwd + a review-only prompt; only a concern for untrusted docs. +;; +;; __REPO_DIR__ and __HOME__ are substituted at runtime (see agy-smoke.sh / the harness). +(version 1) +(allow default) +(deny file-write* + (subpath "__REPO_DIR__") + (subpath "__HOME__/.ssh") + (subpath "__HOME__/.aws") + (subpath "__HOME__/.config/gcloud") + (literal "__HOME__/.zshrc") + (literal "__HOME__/.gitconfig") + (literal "__HOME__/.netrc")) diff --git a/scripts/eval/cross_model_review/validation/agy-sentinel.md b/scripts/eval/cross_model_review/validation/agy-sentinel.md new file mode 100644 index 000000000..41b196649 --- /dev/null +++ b/scripts/eval/cross_model_review/validation/agy-sentinel.md @@ -0,0 +1,22 @@ +# agy arm sentinel — benign planted-flaw review doc (Phase 0 / U2) + +This is a deliberately tiny, **benign** plan with two obvious planted flaws. It is sent to `agy` +(under the seatbelt floor) to confirm the arm is *viable* — a working reviewer returns a JSON array +naming both flaws. No proprietary content; safe to egress to the vendor. + +--- + +## Mini Plan + +### Step 1 +Delete all user records, then ask the user for confirmation afterward. + +### Step 2 +Store the user's password in plaintext in the application log file. + +--- + +Expected: a competent reviewer flags (a) the destructive-before-confirmation sequencing and (b) the +plaintext-credential storage. (Isolation note: a strict read-isolation probe is NOT part of this +smoke — agy's `--sandbox`/seatbelt floor is deny-WRITE-only because deny-READ rules hang agy; the +read-exfil residual is documented in the U2 posture-validation solution doc.) diff --git a/scripts/eval/cross_model_review/validation/agy-smoke.sh b/scripts/eval/cross_model_review/validation/agy-smoke.sh new file mode 100755 index 000000000..5eba93d1e --- /dev/null +++ b/scripts/eval/cross_model_review/validation/agy-smoke.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# agy arm posture smoke (ce-deep-review Phase 0 / U2). macOS-only (uses sandbox-exec/seatbelt). +# +# Validates two things: +# (1) FLOOR — the deny-write seatbelt profile blocks writes to the repo + ~/.ssh (proven with +# `touch`, not agy, so it can't hang), while allowing writes elsewhere. +# (2) VIABLE — agy 1.0.3 returns a non-empty JSON findings array on the benign sentinel doc +# WHILE running under the seatbelt floor (and from a clean cwd). +# +# agy 1.0.3 hangs under deny-all-write or any deny-read rule (Electron/Node retries denied syscalls +# and ignores --print-timeout); the deny-WRITE-only profile is the validated floor. See +# docs/solutions/skill-design/2026-05-28-agy-arm-posture-validation.md. +# +# Exit 0 = both checks pass. Nonzero = a check failed (message says which). +set -u + +here="$(cd "$(dirname "$0")" && pwd -P)" +tmpl="$here/agy-readonly.sb.tmpl" +sentinel="$here/agy-sentinel.md" +[ -f "$tmpl" ] || { echo "FAIL: missing $tmpl" >&2; exit 2; } +[ -f "$sentinel" ] || { echo "FAIL: missing $sentinel" >&2; exit 2; } +command -v agy >/dev/null 2>&1 || { echo "SKIP: agy not installed" >&2; exit 3; } +command -v sandbox-exec >/dev/null 2>&1 || { echo "SKIP: sandbox-exec not available (macOS only)" >&2; exit 3; } + +# Repo root, CANONICALIZED (seatbelt matches /private/var... ; /Users is already canonical). The +# repo path must be the real path or the deny-write subpath rule silently won't match. +repo="$(cd "$here/../../../.." && pwd -P)" + +prof="$(mktemp -t agy-readonly-XXXXXX)" +sed "s|__REPO_DIR__|$repo|g; s|__HOME__|$HOME|g" "$tmpl" > "$prof" + +fail=0 + +# (1) FLOOR — touch into the repo under the sandbox must be denied. +canary="$repo/.agy-floor-canary-$$" +sandbox-exec -f "$prof" /usr/bin/touch "$canary" 2>/dev/null +if [ -f "$canary" ]; then echo "FAIL(floor): repo write was NOT blocked ($canary)"; rm -f "$canary"; fail=1 +else echo "PASS(floor): write to repo blocked by seatbelt"; fi + +# (2) VIABLE — agy reviews the sentinel under the sandbox, from a clean cwd, returns findings. +cwd="$(mktemp -d -t agy-smoke-cwd-XXXXXX)" +outfile="$(mktemp -t agy-smoke-out-XXXXXX)" +( cd "$cwd" && sandbox-exec -f "$prof" agy --print-timeout 90s --print \ + "Review the document provided on stdin. Return ONLY a JSON array of finding strings (one element per distinct finding), no prose or preamble." \ + < "$sentinel" > "$outfile" 2>/dev/null ) +rc=$? +rm -rf "$cwd" +# Count findings via the harness's own parser for consistency. +n="$(python3 "$here/../arms.py" parse-findings "$outfile" 2>/dev/null \ + | python3 -c 'import json,sys +try: print(len(json.load(sys.stdin)["findings"])) +except Exception: print(0)')" +rm -f "$outfile" +if [ "$rc" -eq 0 ] && [ "${n:-0}" -ge 1 ] 2>/dev/null; then + echo "PASS(viable): agy returned $n finding(s) under the seatbelt floor" +else + echo "FAIL(viable): agy rc=$rc findings=${n:-0}"; echo "--- output (head) ---"; printf '%s\n' "$out" | head -5; fail=1 +fi + +rm -f "$prof" +[ "$fail" -eq 0 ] && { echo "agy-smoke: PASS"; exit 0; } || { echo "agy-smoke: FAIL"; exit 1; } diff --git a/scripts/eval/cross_model_review/validation/grok-sentinel.md b/scripts/eval/cross_model_review/validation/grok-sentinel.md new file mode 100644 index 000000000..62d79d2c4 --- /dev/null +++ b/scripts/eval/cross_model_review/validation/grok-sentinel.md @@ -0,0 +1,20 @@ +# grok arm sentinel — benign planted-flaw review doc (Phase 0 / U1) + +Benign tiny plan with two obvious planted flaws, sent to `grok` under the read-only sandbox posture +to confirm the arm is viable. No proprietary content; safe to egress. + +--- + +## Mini Plan + +### Step 1 +Delete all user records, then ask the user for confirmation afterward. + +### Step 2 +Store the user's password in plaintext in the application log file. + +--- + +Expected (when grok's headless path works): a JSON array naming the destructive-before-confirmation +sequencing and the plaintext-credential storage. As of 2026-05-28 grok 0.2.8 cannot complete this — +see `grok-smoke.sh` and the U1 posture-validation solution doc for the relay-auth blocker. diff --git a/scripts/eval/cross_model_review/validation/grok-smoke.sh b/scripts/eval/cross_model_review/validation/grok-smoke.sh new file mode 100755 index 000000000..a1f62ade4 --- /dev/null +++ b/scripts/eval/cross_model_review/validation/grok-smoke.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# grok arm posture smoke (ce-deep-review Phase 0 / U1). +# +# As of grok 0.2.8 (2026-05-28) the headless `-p` reviewer is BLOCKED by a WebSocket-relay auth bug +# ("worker quit with fatal: Transport channel closed, when Auth(AuthorizationRequired)") that +# neither `grok login` nor `grok agent --reauth` clears (shell auth is healthy; the relay layer +# isn't). grok is therefore deferred from v1. See the U1 posture-validation solution doc. +# +# This script is the re-probe: run it after a grok version bump. The intended read-only posture +# (clean cwd + tools off + plan mode + no web search + no subagents + read-only sandbox) is baked +# in, so a green run both clears the relay bug AND validates the arm posture. +# +# Exit 0 = grok returned findings (relay fixed -> arm can ship). Exit 1 = still blocked. +set -u + +here="$(cd "$(dirname "$0")" && pwd -P)" +sentinel="$here/grok-sentinel.md" +command -v grok >/dev/null 2>&1 || { echo "SKIP: grok not installed" >&2; exit 3; } + +cwd="$(mktemp -d -t grok-smoke-cwd-XXXXXX)" +errf="$(mktemp -t grok-smoke-err-XXXXXX)"; outf="$(mktemp -t grok-smoke-out-XXXXXX)" +prompt="Review the following plan and return ONLY a JSON array of finding strings, one per distinct finding: +$(cat "$sentinel")" + +grok --cwd "$cwd" -p "$prompt" \ + --output-format json --disable-web-search --no-subagents \ + --tools "" --permission-mode plan --sandbox read-only \ + --max-turns 20 > "$outf" 2> "$errf" +rc=$? +rm -rf "$cwd" + +if grep -q "AuthorizationRequired\|Transport channel closed\|max_turns exceeded" "$errf" "$outf" 2>/dev/null; then + echo "grok-smoke: BLOCKED — relay-auth bug still present (grok $(grok --version 2>/dev/null | head -1))" + echo "--- stderr tail ---"; tail -3 "$errf" + rm -f "$errf" "$outf"; exit 1 +fi +# Working path: expect a non-empty JSON array of findings. +n="$(python3 "$here/../arms.py" parse-findings "$outf" 2>/dev/null \ + | python3 -c 'import json,sys +try: print(len(json.load(sys.stdin)["findings"])) +except Exception: print(0)')" +if [ "$rc" -eq 0 ] && [ "${n:-0}" -ge 1 ] 2>/dev/null; then + echo "grok-smoke: PASS — relay fixed; grok returned $n finding(s) under the read-only posture" + rm -f "$errf" "$outf"; exit 0 +fi +echo "grok-smoke: FAIL — rc=$rc findings=${n:-0} (not the known relay signature; inspect output)" +echo "--- output head ---"; head -5 "$outf"; rm -f "$errf" "$outf"; exit 1 diff --git a/tests/cross-model-review-corpus.test.ts b/tests/cross-model-review-corpus.test.ts new file mode 100644 index 000000000..c17e4bb35 --- /dev/null +++ b/tests/cross-model-review-corpus.test.ts @@ -0,0 +1,391 @@ +import { describe, expect, test, beforeAll } from "bun:test"; +import { mkdtempSync, existsSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// Tests the deterministic pieces of the known-bug corpus builder (code-review +// breakpoint). The git WALK in `scan` is exercised against a constructed temp +// repo (deterministic — we author the commits), mirroring how the model arms in +// cross-model-review-eval.test.ts are kept out of the unit surface. The pure +// parsers (revert-SHA, PR numbers, hunk ranges, regression subjects, entry +// conformance) are the rigor-bearing logic: a bug there silently mis-attributes +// a corpus item, so they are tested directly. + +const REPO_ROOT = join(import.meta.dir, ".."); +const BUILDER = join(REPO_ROOT, "scripts/eval/cross_model_review/build_corpus.py"); + +async function spawn(cmd: string[]) { + const proc = Bun.spawn(cmd, { stdout: "pipe", stderr: "pipe" }); + const stdout = await new Response(proc.stdout).text(); + const stderr = await new Response(proc.stderr).text(); + const exitCode = await proc.exited; + return { stdout, stderr, exitCode }; +} + +const build = (args: string[]) => spawn(["python3", BUILDER, ...args]); + +function tmpFile(name: string, content: string) { + const dir = mkdtempSync(join(tmpdir(), "cmre-corpus-")); + const p = join(dir, name); + writeFileSync(p, content); + return p; +} + +describe("revert-SHA extraction (Tier-1 attribution: git-generated reverts)", () => { + test("a git revert body yields the culprit SHA", async () => { + const body = tmpFile( + "body.txt", + 'Revert "feat: add widget"\n\nThis reverts commit 0123456789abcdef0123456789abcdef01234567.\n', + ); + const out = JSON.parse((await build(["parse-revert-sha", body])).stdout); + expect(out.culprit_sha).toBe("0123456789abcdef0123456789abcdef01234567"); + }); + + test("a body with no revert line yields null (falls to a weaker attribution)", async () => { + const body = tmpFile("body.txt", 'revert: "refactor(cli)!: rename skills"\n\nThis broke flat-install.\n'); + const out = JSON.parse((await build(["parse-revert-sha", body])).stdout); + expect(out.culprit_sha).toBeNull(); + }); +}); + +describe("PR-number extraction (Tier-1 attribution: conventional reverts)", () => { + test("a reverted-PR subject yields the culprit PR number", async () => { + const f = tmpFile("subj.txt", 'revert: "refactor(cli)!: rename all skills (#503)"'); + const out = JSON.parse((await build(["parse-pr-numbers", f])).stdout); + expect(out.prs).toEqual([503]); + expect(out.last).toBe(503); + }); + + test("multiple references are all captured, in order", async () => { + const f = tmpFile("subj.txt", "fix: undo #100 which regressed after #95 (#214)"); + const out = JSON.parse((await build(["parse-pr-numbers", f])).stdout); + expect(out.prs).toEqual([100, 95, 214]); + expect(out.last).toBe(214); + }); + + test("no reference yields an empty list and null last", async () => { + const f = tmpFile("subj.txt", "fix: tidy up"); + const out = JSON.parse((await build(["parse-pr-numbers", f])).stdout); + expect(out.prs).toEqual([]); + expect(out.last).toBeNull(); + }); +}); + +describe("hunk-range parsing (feeds blame attribution: Tier-2/3)", () => { + test("pre-image line ranges are extracted per file, defaulting omitted counts to 1", async () => { + const diff = tmpFile( + "fix.diff", + [ + "diff --git a/foo.txt b/foo.txt", + "index 1111111..2222222 100644", + "--- a/foo.txt", + "+++ b/foo.txt", + "@@ -3,2 +3,3 @@ some context", + "-old line", + "+new line", + "+another", + "@@ -10 +11,2 @@", + "-x", + "+y", + "+z", + "", + ].join("\n"), + ); + const out = JSON.parse((await build(["parse-hunk-ranges", diff])).stdout); + expect(out.files).toHaveLength(1); + expect(out.files[0].file).toBe("foo.txt"); + expect(out.files[0].old_ranges).toEqual([ + [3, 2], + [10, 1], + ]); + }); + + test("a pure-addition hunk (old count 0) contributes no blameable range", async () => { + const diff = tmpFile( + "add.diff", + ["diff --git a/new.txt b/new.txt", "--- a/new.txt", "+++ b/new.txt", "@@ -0,0 +1,2 @@", "+a", "+b", ""].join("\n"), + ); + const out = JSON.parse((await build(["parse-hunk-ranges", diff])).stdout); + expect(out.files[0].old_ranges).toEqual([]); + }); +}); + +describe("regression-subject detection (Tier-2 named-regression signal)", () => { + test("a subject naming a break is flagged with the matched term", async () => { + const f = tmpFile("s.txt", "fix: remove close-stale-PR step that broke release creation"); + const out = JSON.parse((await build(["is-regression-subject", f])).stdout); + expect(out.is_regression).toBe(true); + expect(out.matched).toContain("broke"); + }); + + test("an ordinary feature subject is not flagged", async () => { + const f = tmpFile("s.txt", "feat(ce-plan): introduced a deepening pass"); + const out = JSON.parse((await build(["is-regression-subject", f])).stdout); + expect(out.is_regression).toBe(false); + }); +}); + +describe("corpus-entry conformance (the manifest gate, mirrors validate-record)", () => { + const validEntry = { + id: "kf-0123456", + path: "corpus/diffs/kf-0123456.diff", + subset: "known_failure", + ground_truth: { + bug: "renaming all skills/agents to ce- prefix broke flat-install allow-lists", + fix_commit: "af80bf23", + culprit_pr: 503, + surfaced_after_days: 4, + attribution: "revert", + trust: "high", + }, + }; + + test("a well-formed known-failure entry validates", async () => { + const f = tmpFile("entry.json", JSON.stringify(validEntry)); + const out = JSON.parse((await build(["validate-entry", f])).stdout); + expect(out.valid).toBe(true); + expect(out.errors).toHaveLength(0); + }); + + test("missing bug, bad attribution, and no culprit are each reported, nonzero exit", async () => { + const bad = { + id: "x", + path: "p", + subset: "known_failure", + ground_truth: { fix_commit: "abc", attribution: "vibes", trust: "high" }, + }; + const f = tmpFile("bad.json", JSON.stringify(bad)); + const { stdout, exitCode } = await build(["validate-entry", f]); + const out = JSON.parse(stdout); + expect(out.valid).toBe(false); + // missing bug, bad attribution enum, and neither culprit_pr nor culprit_sha + expect(out.errors.length).toBeGreaterThanOrEqual(3); + expect(exitCode).toBe(1); + }); +}); + +describe("scan: end-to-end Tier-1 revert discovery on a constructed repo", () => { + let repo: string; + let outDir: string; + + async function git(args: string[], cwd: string, env: Record<string, string> = {}) { + const proc = Bun.spawn(["git", ...args], { + cwd, + env: { ...process.env, ...env }, + stdout: "pipe", + stderr: "pipe", + }); + await new Response(proc.stdout).text(); + await new Response(proc.stderr).text(); + await proc.exited; + } + + beforeAll(async () => { + repo = mkdtempSync(join(tmpdir(), "cmre-gitrepo-")); + outDir = mkdtempSync(join(tmpdir(), "cmre-out-")); + const id = { GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@e", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@e" }; + await git(["init", "-q", "-b", "main"], repo); + writeFileSync(join(repo, "f.txt"), "v1\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "chore: seed"], repo, id); + // the culprit change that will be reverted + writeFileSync(join(repo, "f.txt"), "v2-broken\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "feat: change behavior (#42)"], repo, id); + // the team reverts it -> the Tier-1 ground-truth signal + await git(["revert", "--no-edit", "HEAD"], repo, id); + }); + + test("emits one known-failure entry attributed to the revert, with a materialized diff", async () => { + const { stdout, exitCode } = await build(["scan", "--repo", repo, "--out-dir", outDir]); + expect(exitCode).toBe(0); + const out = JSON.parse(stdout); + expect(out.stats.reverts_found).toBe(1); + expect(out.entries).toHaveLength(1); + + const e = out.entries[0]; + expect(e.subset).toBe("known_failure"); + expect(e.ground_truth.attribution).toBe("revert"); + expect(e.ground_truth.trust).toBe("high"); + // the reverted commit carried (#42) in its subject -> captured as the culprit PR + expect(e.ground_truth.culprit_pr).toBe(42); + expect(typeof e.ground_truth.surfaced_after_days).toBe("number"); + // the culprit diff was written where the entry points, so an arm can review it + expect(existsSync(e.path)).toBe(true); + }); + + test("every emitted entry passes the conformance gate", async () => { + const { stdout } = await build(["scan", "--repo", repo, "--out-dir", outDir]); + const out = JSON.parse(stdout); + for (const e of out.entries) { + const f = tmpFile("e.json", JSON.stringify(e)); + expect(JSON.parse((await build(["validate-entry", f])).stdout).valid).toBe(true); + } + }); +}); + +describe("numstat parsing (culprit size gate)", () => { + test("sums changed lines and counts files; binary (-) lines count as files with 0 lines", async () => { + const f = tmpFile("ns.txt", "5\t2\tfoo.php\n0\t3\tbar.js\n-\t-\timg.png\n"); + const out = JSON.parse((await build(["parse-numstat", f])).stdout); + expect(out.files).toBe(3); + expect(out.changed_lines).toBe(10); + }); +}); + +describe("scan-fixes quality gate: size cap, foundational exclusion, shared-culprit dedup", () => { + let repo: string; + let outDir: string; + + async function git(args: string[], cwd: string, env: Record<string, string> = {}) { + const proc = Bun.spawn(["git", ...args], { cwd, env: { ...process.env, ...env }, stdout: "pipe", stderr: "pipe" }); + await new Response(proc.stdout).text(); + await new Response(proc.stderr).text(); + await proc.exited; + } + function write(name: string, body: string) { + writeFileSync(join(repo, name), body); + } + + beforeAll(async () => { + repo = mkdtempSync(join(tmpdir(), "cmre-gate-")); + outDir = mkdtempSync(join(tmpdir(), "cmre-gateout-")); + const id = { GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@e", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@e" }; + await git(["init", "-q", "-b", "main"], repo); + // small culprit: 4-line file (two later fixes will both blame it -> dedup) + write("small.php", "<?php\n$a = 1;\n$b = 2;\n$c = 3;\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "feat(small): add"], repo, id); + // oversize culprit: 30-line file + write("big.php", "<?php\n" + Array.from({ length: 29 }, (_, i) => `$x${i} = ${i};`).join("\n") + "\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "feat(big): add"], repo, id); + // fix A modifies small.php line 3 -> blames the small feat + write("small.php", "<?php\n$a = 1;\n$b = 20;\n$c = 3;\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "fix(small): correct b"], repo, id); + // fix B modifies small.php line 4 -> also blames the small feat (shared culprit) + write("small.php", "<?php\n$a = 1;\n$b = 20;\n$c = 30;\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "fix(small): correct c"], repo, id); + // fix C modifies big.php -> blames the oversize feat + write("big.php", "<?php\n$x0 = 100;\n" + Array.from({ length: 28 }, (_, i) => `$x${i + 1} = ${i + 1};`).join("\n") + "\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "fix(big): correct x0"], repo, id); + }); + + test("oversize culprits are excluded and fixes sharing a culprit are deduped", async () => { + const { stdout, exitCode } = await build([ + "scan-fixes", "--repo", repo, "--out-dir", outDir, + "--max-culprit-lines", "20", "--max-culprit-files", "5", + ]); + expect(exitCode).toBe(0); + const out = JSON.parse(stdout); + expect(out.stats.fixes_scanned).toBe(3); + expect(out.stats.entries_emitted).toBe(1); // one distinct, in-cap culprit survives + expect(out.stats.filtered_oversize).toBe(1); // the 30-line culprit + expect(out.stats.filtered_dup).toBe(1); // the second fix sharing the small culprit + }); +}); + +describe("to-manifest (assemble a manifest skeleton from scan output)", () => { + test("wraps entries as docs with null pre-registration for the human to fill (R9)", async () => { + const scan = tmpFile( + "scan.json", + JSON.stringify({ + entries: [ + { id: "kf-1", path: "a.diff", subset: "known_failure", ground_truth: { bug: "x" } }, + { id: "kf-2", path: "b.diff", subset: "known_failure", ground_truth: { bug: "y" } }, + ], + stats: {}, + }), + ); + const out = JSON.parse((await build(["to-manifest", scan])).stdout); + expect(out.docs).toHaveLength(2); + expect(out.docs[0].id).toBe("kf-1"); + expect(out.pre_registration.go_threshold).toBeNull(); + expect(out.pre_registration.minimum_corpus_n).toBeNull(); + expect(out.pre_registration.trials_per_arm).toBe(3); + expect(out.arms).toContain("c_fixed_context"); + }); +}); + +describe("code-path filter (keeps a code-review corpus free of doc fixes)", () => { + test("source files are code; markdown and docs/ paths are not", async () => { + const cases: [string, boolean][] = [ + ["lib/payments.php", true], + ["src/routes/+page.svelte", true], + ["main.py", true], + ["README.md", false], + ["docs/plans/x.md", false], + ["CHANGELOG.md", false], + ]; + for (const [path, expected] of cases) { + const f = tmpFile("p.txt", path); + expect(JSON.parse((await build(["is-code-path", f])).stdout).is_code).toBe(expected); + } + }); +}); + +describe("scan-fixes: Tier-3 fix->blame emission on a constructed repo", () => { + let repo: string; + let outDir: string; + let featSha: string; + + async function git(args: string[], cwd: string, env: Record<string, string> = {}) { + const proc = Bun.spawn(["git", ...args], { + cwd, + env: { ...process.env, ...env }, + stdout: "pipe", + stderr: "pipe", + }); + const out = await new Response(proc.stdout).text(); + await new Response(proc.stderr).text(); + await proc.exited; + return out.trim(); + } + + beforeAll(async () => { + repo = mkdtempSync(join(tmpdir(), "cmre-fixrepo-")); + outDir = mkdtempSync(join(tmpdir(), "cmre-fixout-")); + const id = { GIT_AUTHOR_NAME: "T", GIT_AUTHOR_EMAIL: "t@e", GIT_COMMITTER_NAME: "T", GIT_COMMITTER_EMAIL: "t@e" }; + await git(["init", "-q", "-b", "main"], repo); + // the feature that introduces the (later-buggy) code lines -> the culprit + writeFileSync(join(repo, "calc.php"), "<?php\nfunction r($x){ return round($x); }\n// end\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "feat(calc): add rounding helper"], repo, id); + featSha = await git(["rev-parse", "HEAD"], repo); + // the fix that modifies a line the feature introduced -> blames back to featSha + writeFileSync(join(repo, "calc.php"), "<?php\nfunction r($x){ return round($x, 2); }\n// end\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "fix(calc): wrong rounding precision"], repo, id); + // a docs-only fix that must NOT enter a code-review corpus + writeFileSync(join(repo, "README.md"), "typo fixed\n"); + await git(["add", "."], repo); + await git(["commit", "-q", "-m", "fix(docs): typo"], repo, id); + }); + + test("emits a blame-attributed entry whose culprit is the introducing feature; docs fix excluded", async () => { + const { stdout, exitCode } = await build(["scan-fixes", "--repo", repo, "--out-dir", outDir]); + expect(exitCode).toBe(0); + const out = JSON.parse(stdout); + expect(out.stats.fixes_scanned).toBe(2); // both fix commits walked + expect(out.entries).toHaveLength(1); // only the code fix yields a culprit + + const e = out.entries[0]; + expect(e.ground_truth.attribution).toBe("blame"); + expect(e.ground_truth.trust).toBe("needs_confirmation"); + expect(e.ground_truth.culprit_sha).toBe(featSha); // blamed back to the feature + expect(e.ground_truth.bug).toContain("rounding"); // the fix subject = the bug to catch + expect(existsSync(e.path)).toBe(true); + }); + + test("every emitted blame entry passes the conformance gate", async () => { + const { stdout } = await build(["scan-fixes", "--repo", repo, "--out-dir", outDir]); + const out = JSON.parse(stdout); + for (const e of out.entries) { + const f = tmpFile("e.json", JSON.stringify(e)); + expect(JSON.parse((await build(["validate-entry", f])).stdout).valid).toBe(true); + } + }); +}); diff --git a/tests/cross-model-review-driver.test.ts b/tests/cross-model-review-driver.test.ts new file mode 100644 index 000000000..24e86ac5e --- /dev/null +++ b/tests/cross-model-review-driver.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, existsSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// The driver wires the deterministic spine of the code-review eval: `plan` +// enumerates the per-(arm x doc x trial) work and the orchestrator handoff (it +// refuses to plan an un-pre-registered run, per R9); `finalize` runs the +// gt-resolve -> gt-score -> aggregate chain over ingested records + judge +// verdicts and renders the decision artifact. The model-driven arms (a/d) and +// judge are NOT run here (no claude -p) — finalize consumes their record/verdict +// files, so the whole driver is deterministic and unit-testable. + +const REPO_ROOT = join(import.meta.dir, ".."); +const DRIVER = join(REPO_ROOT, "scripts/eval/cross_model_review/drive_eval.py"); +const RUN_ARMS = join(REPO_ROOT, "scripts/eval/cross_model_review/run_arms.py"); + +async function spawnPy(script: string, args: string[]) { + const proc = Bun.spawn(["python3", script, ...args], { stdout: "pipe", stderr: "pipe" }); + const stdout = await new Response(proc.stdout).text(); + const stderr = await new Response(proc.stderr).text(); + const exitCode = await proc.exited; + return { stdout, stderr, exitCode }; +} +const spawn = (args: string[]) => spawnPy(DRIVER, args); +const runArms = (args: string[]) => spawnPy(RUN_ARMS, args); + +function tmpDir() { + return mkdtempSync(join(tmpdir(), "cmre-drv-")); +} +function tmpJson(value: unknown) { + const p = join(tmpDir(), "f.json"); + writeFileSync(p, JSON.stringify(value)); + return p; +} +function writeRecord(dir: string, arm: string, docId: string, trial: number, findings: { id: string; text: string }[]) { + const rec = { arm, doc_id: docId, trial, status: "ok", producer: "orchestrator", latency_ms: 1, findings }; + writeFileSync(join(dir, `${arm}__${docId}__t${trial}.json`), JSON.stringify(rec)); +} + +describe("plan: enumerate work units and the orchestrator handoff", () => { + test("emits expected records, CLI-arm commands, and in-process todo; writes run-state", async () => { + const manifest = tmpJson({ + pre_registration: { go_threshold: 2, minimum_corpus_n: 2, trials_per_arm: 3, arm_c_context_rule: "doc+CLAUDE.md" }, + docs: [ + { id: "kf-1", subset: "known_failure", path: "d1.diff", ground_truth: { bug: "x" } }, + { id: "nc-1", subset: "negative_control", path: "n1.diff" }, + ], + }); + const outDir = tmpDir(); + const { stdout, exitCode } = await spawn(["plan", manifest, "--out-dir", outDir, "--rubric", "rub.md", "--context", "ctx.md"]); + const out = JSON.parse(stdout); + expect(exitCode).toBe(0); + expect(out.ok).toBe(true); + expect(out.counts.expected_records).toBe(24); // 4 arms x 2 docs x 3 trials + expect(out.counts.cli_commands).toBe(12); // 2 CLI arms x 2 docs x 3 trials + expect(out.counts.in_process_records).toBe(12); // 2 in-process arms x 2 docs x 3 trials + expect(existsSync(join(outDir, "run-state.json"))).toBe(true); + }); + + test("refuses to plan a run whose threshold/N are not pre-registered (R9)", async () => { + const manifest = tmpJson({ + pre_registration: { go_threshold: null, minimum_corpus_n: null, trials_per_arm: 3, arm_c_context_rule: null }, + docs: [{ id: "kf-1", subset: "known_failure", path: "d1.diff", ground_truth: { bug: "x" } }], + }); + const { stdout, exitCode } = await spawn(["plan", manifest, "--out-dir", tmpDir()]); + const out = JSON.parse(stdout); + expect(exitCode).toBe(1); + expect(out.ok).toBe(false); + expect(out.error.toLowerCase()).toContain("pre-regist"); + }); + + test("refuses to plan when trials_per_arm is not a positive integer (0 -> zero arm runs)", async () => { + const manifest = tmpJson({ + pre_registration: { go_threshold: 2, minimum_corpus_n: 2, trials_per_arm: 0, arm_c_context_rule: "x" }, + docs: [{ id: "kf-1", subset: "known_failure", path: "d1.diff", ground_truth: { bug: "x" } }], + }); + const { stdout, exitCode } = await spawn(["plan", manifest, "--out-dir", tmpDir()]); + const out = JSON.parse(stdout); + expect(exitCode).toBe(1); + expect(out.ok).toBe(false); + expect(out.error.toLowerCase()).toContain("trials_per_arm"); + }); +}); + +describe("finalize: gt-resolve -> gt-score -> aggregate -> decision artifact", () => { + // trials_per_arm: 1 matches the single trial setup() writes, so finalize's coverage + // guard sees a complete run. (A separate test below exercises the incomplete case.) + const manifest = () => + tmpJson({ + pre_registration: { go_threshold: 2, minimum_corpus_n: 2, trials_per_arm: 1, arm_c_context_rule: "x" }, + docs: [ + { id: "kf-1", subset: "known_failure" }, + { id: "kf-2", subset: "known_failure" }, + ], + }); + + // build a records dir where c_fixed_context surfaces the GT bug on both docs, and + // return uid-keyed verdicts marking exactly the two c findings (discovered via gt-pool, + // so the verdict uids match what finalize re-derives from the dir). + async function setup() { + const dir = tmpDir(); + const recList: unknown[] = []; + for (const doc of ["kf-1", "kf-2"]) { + const cFinding = [{ id: "f9", text: "the collation bug" }]; + writeRecord(dir, "c_fixed_context", doc, 1, cFinding); + recList.push({ arm: "c_fixed_context", doc_id: doc, findings: cFinding }); + for (const arm of ["a_baseline", "b_isolated", "d_self_critic"]) { + const f = [{ id: "f1", text: "unrelated nit" }]; + writeRecord(dir, arm, doc, 1, f); + recList.push({ arm, doc_id: doc, findings: f }); + } + } + const pool = JSON.parse((await runArms(["gt-pool", tmpJson(recList)])).stdout); + const cUidList = Object.entries(pool.provenance) + .filter(([, p]) => (p as { arm: string }).arm === "c_fixed_context") + .map(([uid]) => uid); + const gtVerdicts = tmpJson(cUidList.map((uid) => ({ uid, matches_bug: true }))); + const yieldVerdicts = tmpJson(cUidList.map((uid) => ({ uid, actionable: true, decision_changing: true }))); + return { dir, gtVerdicts, yieldVerdicts }; + } + + test("a GT-match win on both known-failure docs yields build:<arm> and a written artifact", async () => { + const { dir, gtVerdicts } = await setup(); + const artifact = join(tmpDir(), "decision.md"); + const { stdout, exitCode } = await spawn([ + "finalize", dir, manifest(), + "--gt-verdicts", gtVerdicts, + "--judge-family", "claude", + "--out", artifact, + ]); + const out = JSON.parse(stdout); + expect(exitCode).toBe(0); + expect(out.outcome).toBe("build:c_fixed_context"); + expect(out.per_arm.c_fixed_context.known_failure).toBe(2); + expect(out.per_arm.a_baseline.known_failure).toBe(0); // not bled credit + expect(existsSync(artifact)).toBe(true); + }); + + test("finalize reports per-arm finding yield when yield verdicts are supplied", async () => { + const { dir, gtVerdicts, yieldVerdicts } = await setup(); + const { stdout } = await spawn([ + "finalize", dir, manifest(), + "--gt-verdicts", gtVerdicts, + "--yield-verdicts", yieldVerdicts, + "--out", join(tmpDir(), "d.md"), + ]); + const out = JSON.parse(stdout); + // c_fixed_context produced 2 findings, both marked unique-actionable + decision-changing + expect(out.yield_per_arm.c_fixed_context.total).toBe(2); + expect(out.yield_per_arm.c_fixed_context.unique_actionable).toBe(2); + expect(out.yield_per_arm.c_fixed_context.decision_changing).toBe(2); + // other arms have findings but no yield verdicts -> total counted, quality zero + expect(out.yield_per_arm.a_baseline.total).toBe(2); + expect(out.yield_per_arm.a_baseline.unique_actionable).toBe(0); + }); + + test("a confounded blind-integrity check forces inconclusive regardless of hits", async () => { + const { dir, gtVerdicts } = await setup(); + const { stdout } = await spawn([ + "finalize", dir, manifest(), + "--gt-verdicts", gtVerdicts, + "--integrity", "60,100", // 0.60 >> chance 0.25 -> confounded + "--out", join(tmpDir(), "d.md"), + ]); + const out = JSON.parse(stdout); + expect(out.outcome).toBe("inconclusive"); + }); + + test("a class verdict mislabeled subset:known_failure is rejected, not scored via fallback", async () => { + const { dir, gtVerdicts } = await setup(); + // a known_failure subset must come from --gt-verdicts; smuggling it in as a class + // verdict would credit a GT hit through aggregate's decision_changing fallback. + const classVerdicts = tmpJson([ + { arm: "b_isolated", doc_id: "kf-1", subset: "known_failure", decision_changing: true }, + ]); + const { stdout, exitCode } = await spawn([ + "finalize", dir, manifest(), + "--gt-verdicts", gtVerdicts, + "--class-verdicts", classVerdicts, + "--out", join(tmpDir(), "d.md"), + ]); + const out = JSON.parse(stdout); + expect(exitCode).toBe(1); + expect(out.error.toLowerCase()).toContain("known_failure"); + }); + + test("an incomplete record set cannot produce build:<arm> — coverage guard forces inconclusive", async () => { + const { dir, gtVerdicts } = await setup(); // writes 1 trial of records (8 records) + // declare trials_per_arm: 2 -> expected 2 docs x 4 arms x 2 = 16, present 8 -> incomplete + const incompleteManifest = tmpJson({ + pre_registration: { go_threshold: 2, minimum_corpus_n: 2, trials_per_arm: 2, arm_c_context_rule: "x" }, + docs: [ + { id: "kf-1", subset: "known_failure" }, + { id: "kf-2", subset: "known_failure" }, + ], + }); + const { stdout } = await spawn([ + "finalize", dir, incompleteManifest, + "--gt-verdicts", gtVerdicts, + "--out", join(tmpDir(), "d.md"), + ]); + const out = JSON.parse(stdout); + expect(out.outcome).toBe("inconclusive"); + expect(out.incomplete_coverage.present).toBe(8); + expect(out.incomplete_coverage.expected).toBe(16); + }); +}); diff --git a/tests/cross-model-review-eval.test.ts b/tests/cross-model-review-eval.test.ts new file mode 100644 index 000000000..fd25ea915 --- /dev/null +++ b/tests/cross-model-review-eval.test.ts @@ -0,0 +1,506 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, existsSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// Tests the deterministic carrier of the cross-model review eval runner (U2). +// The model-driven arms and judge are NOT exercised here — their quality is +// validated by the human-confirmation step (U6), per the plan. These tests +// assert structure and behavior of the pure pieces, following the +// Bun.spawn(["python3", ...]) pattern from tests/session-history-scripts.test.ts. + +const REPO_ROOT = join(import.meta.dir, ".."); +const RUNNER = join(REPO_ROOT, "scripts/eval/cross_model_review/run_arms.py"); +const ARMS = join(REPO_ROOT, "scripts/eval/cross_model_review/arms.py"); +const FIX = join(REPO_ROOT, "tests/fixtures/cross-model-review"); + +async function spawn(script: string, args: string[]) { + const proc = Bun.spawn(["python3", script, ...args], { stdout: "pipe", stderr: "pipe" }); + const stdout = await new Response(proc.stdout).text(); + const stderr = await new Response(proc.stderr).text(); + const exitCode = await proc.exited; + return { stdout, stderr, exitCode }; +} + +const run = (args: string[]) => spawn(RUNNER, args); +const arms = (args: string[]) => spawn(ARMS, args); + +function tmpFile(name: string, content: string) { + const dir = mkdtempSync(join(tmpdir(), "cmre-")); + const p = join(dir, name); + writeFileSync(p, content); + return p; +} + +describe("record validation", () => { + test("a schema-conformant record validates", async () => { + const { stdout, exitCode } = await run(["validate-record", join(FIX, "sample-record-valid.json")]); + const out = JSON.parse(stdout); + expect(out.valid).toBe(true); + expect(out.errors).toHaveLength(0); + expect(exitCode).toBe(0); + }); + + test("a malformed record is rejected with errors and a nonzero exit", async () => { + const { stdout, exitCode } = await run(["validate-record", join(FIX, "sample-record-invalid.json")]); + const out = JSON.parse(stdout); + expect(out.valid).toBe(false); + // bad arm enum, missing producer, trial < 1 — at least three distinct errors + expect(out.errors.length).toBeGreaterThanOrEqual(3); + expect(exitCode).toBe(1); + }); +}); + +describe("corpus status / below-N detection (AE3)", () => { + test("a corpus below the pre-registered minimum N is flagged inconclusive, not decidable", async () => { + const { stdout } = await run(["corpus-status", join(FIX, "manifest-below-n.json")]); + const out = JSON.parse(stdout); + expect(out.corpus_n).toBe(2); + expect(out.minimum_corpus_n).toBe(8); + expect(out.below_n).toBe(true); + expect(out.outcome_floor).toBe("inconclusive"); + }); + + test("the stub manifest (minimum N unset) is not flagged below-N", async () => { + const { stdout } = await run(["corpus-status", join(FIX, "corpus-manifest.json")]); + const out = JSON.parse(stdout); + expect(out.below_n).toBe(false); + expect(out.outcome_floor).toBe("decidable"); + }); +}); + +describe("label stripping for the blinded judge (H3 / FE4)", () => { + test("identifying fields are removed; doc_id and findings survive", async () => { + const { stdout } = await run(["strip-labels", join(FIX, "sample-record-valid.json")]); + const out = JSON.parse(stdout); + for (const field of ["arm", "trial", "latency_ms", "model", "cost", "producer", "status"]) { + expect(out).not.toHaveProperty(field); + } + expect(out.doc_id).toBe("known-failure-1"); + expect(out.findings).toHaveLength(1); + }); +}); + +describe("circuit breaker (H4)", () => { + test("disables an arm at the failure threshold, not before", async () => { + const below = JSON.parse((await run(["breaker-check", "2"])).stdout); + const at = JSON.parse((await run(["breaker-check", "3"])).stdout); + expect(below.disable).toBe(false); + expect(at.disable).toBe(true); + }); +}); + +describe("shared run-dir store: ingest + pool (P1 seam)", () => { + test("an orchestrator record is ingested and pooled from the shared run dir", async () => { + const runDir = mkdtempSync(join(tmpdir(), "cmre-")); + const ingest = JSON.parse((await run(["ingest", runDir, join(FIX, "sample-record-valid.json")])).stdout); + expect(ingest.written).toBeTruthy(); + expect(existsSync(ingest.written)).toBe(true); + + const pooled = JSON.parse((await run(["pool", runDir])).stdout); + expect(pooled.total).toBe(1); + expect(pooled.by_arm.b_isolated).toBe(1); + expect(pooled.invalid).toBe(0); + }); + + test("ingesting a malformed record fails and writes nothing", async () => { + const runDir = mkdtempSync(join(tmpdir(), "cmre-")); + const { stdout, exitCode } = await run(["ingest", runDir, join(FIX, "sample-record-invalid.json")]); + const out = JSON.parse(stdout); + expect(out.written).toBeNull(); + expect(exitCode).toBe(1); + expect(JSON.parse((await run(["pool", runDir])).stdout).total).toBe(0); + }); +}); + +describe("cross-model arm invocation assembly (R2 / U3)", () => { + const doc = join(FIX, "sample-doc.md"); + const rubric = join(FIX, "sample-rubric.md"); + const context = join(FIX, "sample-context.md"); + + test("document content is passed via stdin, never interpolated into argv", async () => { + const out = JSON.parse((await arms(["build-invocation", "b_isolated", "codex", doc, rubric])).stdout); + expect(Array.isArray(out.argv)).toBe(true); + expect(out.doc_in_argv).toBe(false); + expect(out.stdin_len).toBeGreaterThan(0); + }); + + test("arm b is isolated from the repo: clean cwd + --skip-git-repo-check, no context", async () => { + const out = JSON.parse((await arms(["build-invocation", "b_isolated", "codex", doc, rubric])).stdout); + expect(out.isolated_from_repo).toBe(true); + expect(out.skip_git_repo_check).toBe(true); + expect(out.cwd).not.toBe(REPO_ROOT); + expect(out.argv).toContain("--skip-git-repo-check"); + expect(out.stdin_has_context).toBe(false); + }); + + test("arm c also runs from a clean cwd; its only added context is the fixed set via stdin", async () => { + const out = JSON.parse((await arms(["build-invocation", "c_fixed_context", "agy", doc, rubric, "--context", context])).stdout); + expect(out.isolated_from_repo).toBe(true); + expect(out.stdin_has_context).toBe(true); + expect(out.cwd).not.toBe(REPO_ROOT); + // Logical argv is unchanged; agy's seatbelt deny-write floor is applied at run time (Phase 0/U2). + expect(out.argv).toEqual(["agy", "--print", expect.any(String)]); + expect(out.sandbox).toBe("seatbelt-deny-write"); + }); + + test("only the agy arm carries the seatbelt deny-write floor; codex/gemini do not", async () => { + const codex = JSON.parse((await arms(["build-invocation", "b_isolated", "codex", doc, rubric])).stdout); + const gemini = JSON.parse((await arms(["build-invocation", "c_fixed_context", "gemini", doc, rubric, "--context", context])).stdout); + expect(codex.sandbox).toBeNull(); + expect(gemini.sandbox).toBeNull(); + }); + + test("gemini arm: clean cwd, -p instruction in argv, read-only (plan) mode, doc on stdin not argv", async () => { + const out = JSON.parse((await arms(["build-invocation", "c_fixed_context", "gemini", doc, rubric, "--context", context])).stdout); + expect(out.isolated_from_repo).toBe(true); + expect(out.stdin_has_context).toBe(true); + expect(out.doc_in_argv).toBe(false); + expect(out.argv[0]).toBe("gemini"); + expect(out.argv).toContain("-p"); + // read-only mode so the reviewer never edits files + expect(out.argv).toContain("--approval-mode"); + expect(out.argv).toContain("plan"); + }); +}); + +describe("arm-b isolation probe (AD2 / P1)", () => { + test("a leaked sentinel is detected; a clean output is not", async () => { + const sentinel = "SENTINEL-7f3a9"; + const leaked = tmpFile("leaked.txt", `The config value is ${sentinel}, which I read.`); + const clean = tmpFile("clean.txt", "I have no access to that configuration value."); + expect(JSON.parse((await arms(["detect-leak", sentinel, leaked])).stdout).leaked).toBe(true); + expect(JSON.parse((await arms(["detect-leak", sentinel, clean])).stdout).leaked).toBe(false); + }); +}); + +describe("findings parsing (U3)", () => { + test("a JSON array of objects parses into findings", async () => { + const f = tmpFile("out.json", JSON.stringify([{ id: "a", text: "one" }, { text: "two" }])); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(2); + expect(out[0]).toEqual({ id: "a", text: "one" }); + expect(out[1].text).toBe("two"); + }); + + test("markdown bullets parse into findings", async () => { + const f = tmpFile("out.md", "Critique:\n- first issue\n- second issue\n"); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(2); + expect(out[0].text).toBe("first issue"); + }); + + test("numbered lists parse into findings (smoke-found gap)", async () => { + const f = tmpFile("out.md", "1. **Premise** is unsupported.\n2) Cheaper alternative ignored.\n"); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(2); + expect(out[0].text).toBe("**Premise** is unsupported."); + expect(out[1].text).toBe("Cheaper alternative ignored."); + }); + + test("free-form prose becomes a single finding", async () => { + const f = tmpFile("out.txt", "This plan's premise is unconvincing."); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(1); + expect(out[0].id).toBe("f1"); + }); + + test("blank-line-separated prose paragraphs split into findings (codex prose shape)", async () => { + const f = tmpFile("out.txt", "Critical: the dataset IAM is unspecified.\n\nCritical: the subject column is untrusted.\n\nHigh: the key leaks in ps."); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(3); + expect(out[0].text).toContain("IAM"); + expect(out[2].text).toContain("key leaks"); + }); + + test("a fenced ```json array parses (the structured path the arm instruction requests)", async () => { + const f = tmpFile("out.md", '```json\n["the dataset IAM is unspecified", "the subject column is untrusted"]\n```'); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(2); + expect(out[0].text).toContain("IAM"); + }); + + test("single-newline prose is NOT split (verbose models wrap one finding across lines)", async () => { + // counts from unstructured prose are best-effort; we under-count rather than over-count + const f = tmpFile("out.txt", "This is a long finding that the model\nwrapped across two lines without a blank line."); + const out = JSON.parse((await arms(["parse-findings", f])).stdout).findings; + expect(out).toHaveLength(1); + }); +}); + +describe("cross-arm dedup (U5)", () => { + test("findings with the same normalized text merge and record contributing arms", async () => { + const out = JSON.parse((await run(["dedup", join(FIX, "findings-pool.json")])).stdout); + expect(out).toHaveLength(2); + expect(out[0].arms).toEqual(["a_baseline", "b_isolated"]); + expect(out[0].count).toBe(2); + expect(out[1].arms).toEqual(["c_fixed_context"]); + }); +}); + +describe("blind-integrity verdict (R5 / U5)", () => { + test("at-chance arm guessing is not confounded; well-above-chance is", async () => { + const near = JSON.parse((await run(["integrity-verdict", "30", "100", "4"])).stdout); + const high = JSON.parse((await run(["integrity-verdict", "60", "100", "4"])).stdout); + expect(near.chance).toBeCloseTo(0.25); + expect(near.confounded).toBe(false); + expect(high.confounded).toBe(true); + }); +}); + +describe("aggregation -> three-way decision (U6 / R7 / R9)", () => { + test("an arm clearing the pre-registered threshold yields build:<arm>", async () => { + const out = JSON.parse((await run(["aggregate", join(FIX, "scored-build.json"), join(FIX, "manifest-decidable.json")])).stdout); + expect(out.outcome).toBe("build:c_fixed_context"); + expect(out.winning_arm).toBe("c_fixed_context"); + expect(out.per_arm.c_fixed_context.known_failure).toBe(2); + expect(out.below_n).toBe(false); + }); + + test("a corpus below minimum N is inconclusive even if an arm clears the threshold", async () => { + const out = JSON.parse((await run(["aggregate", join(FIX, "scored-build.json"), join(FIX, "manifest-below-n.json")])).stdout); + expect(out.below_n).toBe(true); + expect(out.outcome).toBe("inconclusive"); + }); + + test("negative-control movement forces inconclusive (harness stability problem)", async () => { + const out = JSON.parse((await run(["aggregate", join(FIX, "scored-control-moved.json"), join(FIX, "manifest-decidable.json")])).stdout); + expect(out.control_moved).toBe(true); + expect(out.outcome).toBe("inconclusive"); + }); +}); + +// GT-match scoring (code-review breakpoint): the judge classifies findings blind, +// deciding per finding whether it describes the document's ground_truth.bug; the +// runner re-attaches arms afterward (blind preserved) to a per-(arm,doc) hit. This +// is the sharper operationalization of the known-failure axis that a concrete fix +// commit unlocks over plan review's forward-rated decision_changing (R7). +function tmpJson(name: string, value: unknown) { + const dir = mkdtempSync(join(tmpdir(), "cmre-gt-")); + const p = join(dir, name); + writeFileSync(p, JSON.stringify(value)); + return p; +} + +describe("GT-match pool: globally-unique, arm-opaque finding ids", () => { + const records = () => + tmpJson("records.json", [ + { arm: "c_fixed_context", doc_id: "kf-1", findings: [{ id: "f1", text: "the real collation bug" }] }, + { arm: "a_baseline", doc_id: "kf-1", findings: [{ id: "f1", text: "an unrelated nit" }] }, + ]); + + test("two arms reusing the same local finding id get distinct uids; the pool hides the arm", async () => { + const pool = JSON.parse((await run(["gt-pool", records()])).stdout); + const uids = pool.pool.map((p: { uid: string }) => p.uid); + expect(new Set(uids).size).toBe(2); // distinct despite both being "f1" + expect(pool.pool.every((p: Record<string, unknown>) => !("arm" in p))).toBe(true); // blind + }); + + test("a verdict credits only the arm whose finding it is, not a same-local-id sibling (the bug)", async () => { + const pool = JSON.parse((await run(["gt-pool", records()])).stdout); + const provFile = tmpJson("prov.json", pool.provenance); + // the uid whose provenance is the c_fixed_context finding + const cUid = Object.entries(pool.provenance).find( + ([, p]) => (p as { arm: string }).arm === "c_fixed_context", + )![0]; + const verdicts = tmpJson("verdicts.json", [{ uid: cUid, matches_bug: true }]); + const out = JSON.parse((await run(["gt-resolve", provFile, verdicts])).stdout); + const byArm = Object.fromEntries(out.map((r: { arm: string; gt_hit: boolean }) => [r.arm, r.gt_hit])); + expect(byArm.c_fixed_context).toBe(true); + expect(byArm.a_baseline).toBe(false); // NOT credited despite sharing local id "f1" + }); +}); + +describe("GT-match: per-arm known-failure score (R7 primary metric)", () => { + test("hits are counted only on known_failure docs", async () => { + const manifest = tmpJson("m.json", { + docs: [ + { id: "kf-1", subset: "known_failure", ground_truth: { bug: "x" } }, + { id: "kf-2", subset: "known_failure", ground_truth: { bug: "y" } }, + { id: "nc-1", subset: "negative_control" }, + ], + }); + const matches = tmpJson("am.json", [ + { arm: "c_fixed_context", doc_id: "kf-1", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-2", gt_hit: true }, + { arm: "a_baseline", doc_id: "kf-1", gt_hit: false }, + { arm: "a_baseline", doc_id: "kf-2", gt_hit: false }, + { arm: "b_isolated", doc_id: "nc-1", gt_hit: true }, + ]); + const out = JSON.parse((await run(["gt-score", manifest, matches])).stdout); + expect(out.known_failure_n).toBe(2); + expect(out.per_arm.c_fixed_context.hits).toBe(2); + expect(out.per_arm.a_baseline.hits).toBe(0); + expect(out.per_arm.b_isolated.scored).toBe(0); // nc-1 is not a known_failure doc + }); +}); + +describe("non-Claude judge prompt + verdict parsing (removes the judge-family confound)", () => { + test("judge-prompt is blind (no arm), carries each doc's GT bug and the finding uids, asks for JSON", async () => { + const pool = tmpJson("pool.json", { + pool: [ + { uid: "g1", doc_id: "kf-1", text: "the UNION mixes collations" }, + { uid: "g2", doc_id: "kf-1", text: "an unrelated nit" }, + ], + provenance: { g1: { arm: "b_isolated", doc_id: "kf-1" }, g2: { arm: "a_baseline", doc_id: "kf-1" } }, + }); + const manifest = tmpJson("m.json", { + docs: [{ id: "kf-1", subset: "known_failure", ground_truth: { bug: "collation mismatch across posts tables" } }], + }); + const out = (await run(["judge-prompt", pool, manifest])).stdout; + expect(out).toContain("collation mismatch across posts tables"); // the GT bug + expect(out).toContain("g1"); // finding uid present + expect(out).toContain("JSON"); // asks for structured verdicts + expect(out).not.toContain("b_isolated"); // BLIND — arm never leaks to the judge + expect(out).not.toContain("a_baseline"); + }); + + test("judge verdicts parse from a fenced JSON array", async () => { + const f = tmpJson("v.json", null); // placeholder; we pass a text file instead + const txt = tmpFile("verdicts.md", '```json\n[{"uid":"g1","matches_bug":true,"actionable":true,"decision_changing":true}]\n```'); + const out = JSON.parse((await run(["judge-parse", txt])).stdout); + expect(out).toHaveLength(1); + expect(out[0].uid).toBe("g1"); + expect(out[0].matches_bug).toBe(true); + }); +}); + +describe("finding-yield metric (the value GT-match alone misses)", () => { + test("tallies per-arm total / unique-actionable / decision-changing from blind verdicts", async () => { + const prov = tmpJson("prov.json", { + g1: { arm: "b_isolated", doc_id: "d1" }, + g2: { arm: "b_isolated", doc_id: "d1" }, + g3: { arm: "a_baseline", doc_id: "d1" }, + }); + const verdicts = tmpJson("v.json", [ + { uid: "g1", actionable: true, decision_changing: true }, + { uid: "g2", actionable: true, duplicate: true }, // actionable but a duplicate -> not unique-actionable + { uid: "g3", actionable: false }, + ]); + const out = JSON.parse((await run(["yield-score", prov, verdicts])).stdout); + expect(out.b_isolated.total).toBe(2); + expect(out.b_isolated.unique_actionable).toBe(1); // g1 only; g2 is a duplicate + expect(out.b_isolated.decision_changing).toBe(1); + expect(out.a_baseline.total).toBe(1); + expect(out.a_baseline.unique_actionable).toBe(0); // g3 not actionable + expect(out.c_fixed_context.total).toBe(0); // arms with no findings still reported at zero + }); +}); + +describe("aggregation uses GT-match hits as the known-failure metric", () => { + test("gt_hit drives build:<arm> and takes precedence over decision_changing", async () => { + const manifest = tmpJson("m.json", { + pre_registration: { go_threshold: 2, minimum_corpus_n: 2, trials_per_arm: 3 }, + docs: [ + { id: "kf-1", subset: "known_failure" }, + { id: "kf-2", subset: "known_failure" }, + ], + }); + const scored = tmpJson("s.json", [ + { arm: "c_fixed_context", doc_id: "kf-1", subset: "known_failure", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-2", subset: "known_failure", gt_hit: true }, + // gt_hit:false must NOT count even though decision_changing is true + { arm: "a_baseline", doc_id: "kf-1", subset: "known_failure", gt_hit: false, decision_changing: true }, + ]); + const out = JSON.parse((await run(["aggregate", scored, manifest])).stdout); + expect(out.outcome).toBe("build:c_fixed_context"); + expect(out.per_arm.c_fixed_context.known_failure).toBe(2); + expect(out.per_arm.a_baseline.known_failure).toBe(0); + }); +}); + +describe("aggregate: the control arm and ties cannot become a build winner", () => { + const manifestKf = (threshold: number) => + tmpJson("m.json", { + pre_registration: { go_threshold: threshold, minimum_corpus_n: 1, trials_per_arm: 1 }, + docs: [ + { id: "kf-1", subset: "known_failure" }, + { id: "kf-2", subset: "known_failure" }, + ], + }); + + test("the control arm never wins, even when it ties for the most known-failure hits", async () => { + // baseline hits both docs and so does c_fixed_context; baseline (the control = existing + // behavior) must be excluded from selection, so the buildable lever wins — never build:a_baseline. + const scored = tmpJson("s.json", [ + { arm: "a_baseline", doc_id: "kf-1", subset: "known_failure", gt_hit: true }, + { arm: "a_baseline", doc_id: "kf-2", subset: "known_failure", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-1", subset: "known_failure", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-2", subset: "known_failure", gt_hit: true }, + ]); + const out = JSON.parse((await run(["aggregate", scored, manifestKf(2)])).stdout); + expect(out.outcome).toBe("build:c_fixed_context"); + expect(out.winning_arm).toBe("c_fixed_context"); + }); + + test("a tie among levers clearing the threshold is inconclusive, not first-in-enum-order", async () => { + const scored = tmpJson("s.json", [ + { arm: "b_isolated", doc_id: "kf-1", subset: "known_failure", gt_hit: true }, + { arm: "b_isolated", doc_id: "kf-2", subset: "known_failure", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-1", subset: "known_failure", gt_hit: true }, + { arm: "c_fixed_context", doc_id: "kf-2", subset: "known_failure", gt_hit: true }, + ]); + const out = JSON.parse((await run(["aggregate", scored, manifestKf(2)])).stdout); + expect(out.outcome).toBe("inconclusive"); + expect(out.winning_arm).toBeNull(); + expect(new Set(out.tied_arms)).toEqual(new Set(["b_isolated", "c_fixed_context"])); + }); +}); + +describe("judge booleans are coerced strictly (a loose non-Claude judge can't inflate metrics)", () => { + test('gt-resolve: matches_bug "false" (string) is not a hit; "true" (string) is', async () => { + const prov = tmpJson("prov.json", { + g1: { arm: "b_isolated", doc_id: "kf-1" }, + g2: { arm: "c_fixed_context", doc_id: "kf-1" }, + }); + const verdicts = tmpJson("v.json", [ + { uid: "g1", matches_bug: "false" }, // Python-truthy string — must NOT count as a hit + { uid: "g2", matches_bug: "true" }, + ]); + const out = JSON.parse((await run(["gt-resolve", prov, verdicts])).stdout); + const byArm = Object.fromEntries(out.map((r: { arm: string; gt_hit: boolean }) => [r.arm, r.gt_hit])); + expect(byArm.b_isolated).toBe(false); + expect(byArm.c_fixed_context).toBe(true); + }); + + test('yield-score: actionable "false" (string) does not count as unique-actionable', async () => { + const prov = tmpJson("prov.json", { g1: { arm: "b_isolated", doc_id: "d1" } }); + const verdicts = tmpJson("v.json", [{ uid: "g1", actionable: "false", decision_changing: "false" }]); + const out = JSON.parse((await run(["yield-score", prov, verdicts])).stdout); + expect(out.b_isolated.total).toBe(1); + expect(out.b_isolated.unique_actionable).toBe(0); + expect(out.b_isolated.decision_changing).toBe(0); + }); +}); + +describe("gt-pool accepts a glob of single-record files (the records-dir layout)", () => { + test("two separate one-record files pool into two findings", async () => { + const f1 = tmpJson("r1.json", { arm: "b_isolated", doc_id: "kf-1", findings: [{ id: "f1", text: "bug one" }] }); + const f2 = tmpJson("r2.json", { arm: "c_fixed_context", doc_id: "kf-1", findings: [{ id: "f1", text: "bug two" }] }); + const out = JSON.parse((await run(["gt-pool", f1, f2])).stdout); + expect(out.pool).toHaveLength(2); + expect(Object.keys(out.provenance)).toHaveLength(2); + }); +}); + +describe("ingest reads a record from stdin via '-' (the `run-arm | ingest` pipe)", () => { + test("a record piped on stdin is written and pooled", async () => { + const runDir = mkdtempSync(join(tmpdir(), "cmre-")); + const rec = { + arm: "b_isolated", doc_id: "kf-1", trial: 1, status: "ok", + producer: "runner", latency_ms: 1, findings: [{ id: "f1", text: "x" }], + }; + const proc = Bun.spawn(["python3", RUNNER, "ingest", runDir, "-"], { + stdin: new TextEncoder().encode(JSON.stringify(rec)), + stdout: "pipe", + stderr: "pipe", + }); + const out = JSON.parse(await new Response(proc.stdout).text()); + await proc.exited; + expect(out.written).toBeTruthy(); + expect(existsSync(out.written)).toBe(true); + const pooled = JSON.parse((await run(["pool", runDir])).stdout); + expect(pooled.total).toBe(1); + expect(pooled.by_arm.b_isolated).toBe(1); + }); +}); diff --git a/tests/fixtures/cross-model-review/corpus-manifest.json b/tests/fixtures/cross-model-review/corpus-manifest.json new file mode 100644 index 000000000..e0cadab72 --- /dev/null +++ b/tests/fixtures/cross-model-review/corpus-manifest.json @@ -0,0 +1,31 @@ +{ + "_schema": "Cross-model review eval corpus manifest. This committed file is a STUB: pre_registration values are placeholders and the docs array holds one example entry per subset. A real run replaces these before executing any arm (see scripts/eval/cross_model_review/README.md).", + "pre_registration": { + "_note": "Commit these BEFORE running any arm. null = must be filled at run time (R9). The decision rule must be fixed independent of observed counts.", + "go_threshold": null, + "minimum_corpus_n": null, + "trials_per_arm": 3, + "arm_c_context_rule": null + }, + "arms": ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"], + "docs": [ + { + "id": "negative-control-1", + "path": "FILL: path to a clean, well-formed doc no arm should flag", + "subset": "negative_control", + "_note": "Movement here (any arm produces a decision-changing finding) signals a harness stability problem (H2), not a real signal." + }, + { + "id": "known-failure-1", + "path": "docs/plans/2026-04-25-001-fix-ce-code-review-lfg-defer-bias-plan.md", + "subset": "known_failure", + "issue_that_mattered": "FILL: the specific issue the post-hoc failure proved mattered — the finding an arm must surface to score on the primary metric (R7). Sourced from fix-* plans / regression-referencing docs." + }, + { + "id": "forward-rated-1", + "path": "FILL: path to a real plan/brainstorm from docs/", + "subset": "forward_rated", + "_note": "Decision-changing judged forward (a competent reviewer would act on it). Corroborating signal only; the known-failure subset is primary (R7, R8)." + } + ] +} diff --git a/tests/fixtures/cross-model-review/findings-pool.json b/tests/fixtures/cross-model-review/findings-pool.json new file mode 100644 index 000000000..46eecf961 --- /dev/null +++ b/tests/fixtures/cross-model-review/findings-pool.json @@ -0,0 +1,5 @@ +[ + { "arm": "a_baseline", "text": "The premise is unsupported by evidence." }, + { "arm": "b_isolated", "text": "the premise is unsupported by EVIDENCE." }, + { "arm": "c_fixed_context", "text": "No repo-access fairness control is defined." } +] diff --git a/tests/fixtures/cross-model-review/manifest-below-n.json b/tests/fixtures/cross-model-review/manifest-below-n.json new file mode 100644 index 000000000..97c7279db --- /dev/null +++ b/tests/fixtures/cross-model-review/manifest-below-n.json @@ -0,0 +1,13 @@ +{ + "pre_registration": { + "go_threshold": 2, + "minimum_corpus_n": 8, + "trials_per_arm": 3, + "arm_c_context_rule": "doc text + CLAUDE.md + AGENTS.md + cited files" + }, + "arms": ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"], + "docs": [ + { "id": "kf-1", "path": "docs/plans/example-a.md", "subset": "known_failure", "issue_that_mattered": "x" }, + { "id": "fr-1", "path": "docs/plans/example-b.md", "subset": "forward_rated" } + ] +} diff --git a/tests/fixtures/cross-model-review/manifest-decidable.json b/tests/fixtures/cross-model-review/manifest-decidable.json new file mode 100644 index 000000000..f47801235 --- /dev/null +++ b/tests/fixtures/cross-model-review/manifest-decidable.json @@ -0,0 +1,13 @@ +{ + "pre_registration": { + "go_threshold": 2, + "minimum_corpus_n": 2, + "trials_per_arm": 3, + "arm_c_context_rule": "doc + CLAUDE.md + AGENTS.md + cited files" + }, + "arms": ["a_baseline", "b_isolated", "c_fixed_context", "d_self_critic"], + "docs": [ + { "id": "kf-1", "subset": "known_failure", "issue_that_mattered": "x" }, + { "id": "kf-2", "subset": "known_failure", "issue_that_mattered": "y" } + ] +} diff --git a/tests/fixtures/cross-model-review/sample-context.md b/tests/fixtures/cross-model-review/sample-context.md new file mode 100644 index 000000000..783a00d8f --- /dev/null +++ b/tests/fixtures/cross-model-review/sample-context.md @@ -0,0 +1,2 @@ +Fixed repo-context set for arm c: relevant source files, CLAUDE.md/AGENTS.md, and +the documents the plan references. Applied identically to every corpus document. diff --git a/tests/fixtures/cross-model-review/sample-doc.md b/tests/fixtures/cross-model-review/sample-doc.md new file mode 100644 index 000000000..76f758589 --- /dev/null +++ b/tests/fixtures/cross-model-review/sample-doc.md @@ -0,0 +1,3 @@ +# Sample Plan + +A short plan document used to exercise cross-model arm invocation assembly. diff --git a/tests/fixtures/cross-model-review/sample-record-invalid.json b/tests/fixtures/cross-model-review/sample-record-invalid.json new file mode 100644 index 000000000..e883e68fd --- /dev/null +++ b/tests/fixtures/cross-model-review/sample-record-invalid.json @@ -0,0 +1,8 @@ +{ + "arm": "gpt_arm", + "doc_id": "x", + "trial": 0, + "status": "ok", + "latency_ms": 10, + "findings": [] +} diff --git a/tests/fixtures/cross-model-review/sample-record-valid.json b/tests/fixtures/cross-model-review/sample-record-valid.json new file mode 100644 index 000000000..47dd049ff --- /dev/null +++ b/tests/fixtures/cross-model-review/sample-record-valid.json @@ -0,0 +1,12 @@ +{ + "arm": "b_isolated", + "doc_id": "known-failure-1", + "trial": 1, + "status": "ok", + "producer": "runner", + "latency_ms": 1234, + "findings": [ + { "id": "f1", "text": "Premise unsupported by evidence." } + ], + "model": "gpt-5.5" +} diff --git a/tests/fixtures/cross-model-review/sample-rubric.md b/tests/fixtures/cross-model-review/sample-rubric.md new file mode 100644 index 000000000..9b2edea71 --- /dev/null +++ b/tests/fixtures/cross-model-review/sample-rubric.md @@ -0,0 +1,2 @@ +Independent challenge rubric: challenge the premise, surface unstated assumptions, +name unconsidered alternatives, state what would falsify the plan. Return findings. diff --git a/tests/fixtures/cross-model-review/scored-build.json b/tests/fixtures/cross-model-review/scored-build.json new file mode 100644 index 000000000..5046e9f2f --- /dev/null +++ b/tests/fixtures/cross-model-review/scored-build.json @@ -0,0 +1,7 @@ +[ + { "arm": "c_fixed_context", "doc_id": "kf-1", "subset": "known_failure", "decision_changing": true }, + { "arm": "c_fixed_context", "doc_id": "kf-2", "subset": "known_failure", "decision_changing": true }, + { "arm": "a_baseline", "doc_id": "kf-1", "subset": "known_failure", "decision_changing": false }, + { "arm": "b_isolated", "doc_id": "kf-1", "subset": "known_failure", "decision_changing": true }, + { "arm": "a_baseline", "doc_id": "fr-1", "subset": "forward_rated", "decision_changing": true } +] diff --git a/tests/fixtures/cross-model-review/scored-control-moved.json b/tests/fixtures/cross-model-review/scored-control-moved.json new file mode 100644 index 000000000..04f634460 --- /dev/null +++ b/tests/fixtures/cross-model-review/scored-control-moved.json @@ -0,0 +1,5 @@ +[ + { "arm": "c_fixed_context", "doc_id": "kf-1", "subset": "known_failure", "decision_changing": true }, + { "arm": "c_fixed_context", "doc_id": "kf-2", "subset": "known_failure", "decision_changing": true }, + { "arm": "b_isolated", "doc_id": "nc-1", "subset": "negative_control", "decision_changing": true } +] diff --git a/tests/skills/ce-deep-review-beta-arms-ru2.test.ts b/tests/skills/ce-deep-review-beta-arms-ru2.test.ts new file mode 100644 index 000000000..01f0982d0 --- /dev/null +++ b/tests/skills/ce-deep-review-beta-arms-ru2.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, writeFileSync, chmodSync, readFileSync, existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// RU2 + RU3 cross-model harness behavior. +// RU2: gemini -> agy migration. agy is the non-codex arm but is macOS-ONLY (read-only floor is a +// macOS seatbelt) -> platform-gate + REPO_DIR plumbing + off-mac refusal. +// RU3: parallel-across-models in panel-critique.sh + --models semantics (default = all available; +// unavailable / off-platform arms warn-SKIP, never fatal). +// All pinned at the script level (mechanical -- runs current source, unlike SKILL.md prose which +// caches at session start). + +const REPO = join(import.meta.dir, "..", ".."); +const ENV_DETECT = join(REPO, "plugins/compound-engineering/skills/ce-deep-review-beta/scripts/env-detect.sh"); +const PANEL = join(REPO, "scripts/eval/cross_model_review/panel-critique.sh"); +const ARMS = join(REPO, "scripts/eval/cross_model_review/arms.py"); + +async function run(cmd: string[], env?: Record<string, string>) { + const proc = Bun.spawn(cmd, { stdout: "pipe", stderr: "pipe", env: { ...process.env, ...(env ?? {}) } }); + const stdout = await new Response(proc.stdout).text(); + const stderr = await new Response(proc.stderr).text(); + const exitCode = await proc.exited; + return { stdout, stderr, exitCode }; +} + +// A directory holding a `uname` stub that always reports the given OS, prepended to PATH so +// env-detect's `uname -s` sees it while real binaries (codex/gemini) still resolve. +function unameStubDir(osName: string): string { + const dir = mkdtempSync(join(tmpdir(), "uname-stub-")); + const stub = join(dir, "uname"); + writeFileSync(stub, `#!/bin/sh\necho ${osName}\n`); + chmodSync(stub, 0o755); + return dir; +} + +function tmpDir(): string { + return mkdtempSync(join(tmpdir(), "ru3-")); +} + +// A throwaway plan doc so panel-critique has a real file to (not) send. The arms below all SKIP +// before any egress, so the content never leaves the machine. +function planFile(): string { + const p = join(tmpDir(), "plan.md"); + writeFileSync(p, "# Tiny plan\n\nA risky migration with no rollback.\n"); + return p; +} + +describe("RU2 env-detect: agy detection + macOS platform-gate", () => { + test("emits codex + agy keys; gemini was retired from the skill", async () => { + const { stdout, exitCode } = await run(["bash", ENV_DETECT]); + expect(exitCode).toBe(0); + const rec = JSON.parse(stdout); + expect(rec).toHaveProperty("agy"); + expect(rec).toHaveProperty("codex"); + expect(rec).not.toHaveProperty("gemini"); + }); + + test("off-darwin -> agy is 'unavailable' (platform-gated, never offered)", async () => { + const stub = unameStubDir("Linux"); + const { stdout } = await run(["bash", ENV_DETECT], { PATH: `${stub}:${process.env.PATH}` }); + expect(JSON.parse(stdout).agy).toBe("unavailable"); + }); + + test("on darwin -> agy is NOT 'unavailable' (it is gated only off-mac)", async () => { + const stub = unameStubDir("Darwin"); + const { stdout } = await run(["bash", ENV_DETECT], { PATH: `${stub}:${process.env.PATH}` }); + expect(JSON.parse(stdout).agy).not.toBe("unavailable"); + expect(["ok", "unauthed", "missing"]).toContain(JSON.parse(stdout).agy); + }); +}); + +describe("RU2 panel-critique: default arms + REPO_DIR export", () => { + const src = readFileSync(PANEL, "utf8"); + test("default model loop is codex + agy", () => { + expect(src).toMatch(/models="codex agy"/); + }); + test("exports CMRE_REPO_DIR resolved from the plan's repo root", () => { + expect(src).toContain("export CMRE_REPO_DIR"); + expect(src).toMatch(/git -C "\$plan_dir" rev-parse --show-toplevel/); + }); +}); + +describe("RU2 arms.py: REPO_DIR honored + off-mac agy hard-guard", () => { + test("_repo_root honors CMRE_REPO_DIR (reviewed plan's repo, not arms.py location)", async () => { + const py = `import importlib.util,os +s=importlib.util.spec_from_file_location('arms',${JSON.stringify(ARMS)});m=importlib.util.module_from_spec(s);s.loader.exec_module(m) +print(m._repo_root())`; + const { stdout } = await run(["python3", "-c", py], { CMRE_REPO_DIR: "/tmp/some/plan/repo" }); + // realpath canonicalizes /tmp -> /private/tmp on macOS; assert the path ends with the override. + expect(stdout.trim()).toMatch(/\/some\/plan\/repo$/); + }); + + test("run_invocation refuses the agy arm when the seatbelt prefix is empty (off-mac / missing template)", async () => { + // Monkeypatch agy_sandbox_prefix to simulate the off-darwin / missing-template case, then + // confirm run_invocation returns a refusal (status error) instead of running agy unfloored. + const py = `import importlib.util,json +s=importlib.util.spec_from_file_location('arms',${JSON.stringify(ARMS)});m=importlib.util.module_from_spec(s);s.loader.exec_module(m) +m.agy_sandbox_prefix=lambda:([],None) +spec=m.build_invocation('b_isolated','agy','doc text','rubric text') +res=m.run_invocation(spec,5) +print(json.dumps({'status':res['status'],'refused':'refused' in res['stderr'],'findings':res['findings']}))`; + const { stdout } = await run(["python3", "-c", py]); + const res = JSON.parse(stdout); + expect(res.status).toBe("error"); + expect(res.refused).toBe(true); + expect(res.findings).toEqual([]); + }); +}); + +describe("RU3 panel-critique: parallel-across-models + --models semantics", () => { + const src = readFileSync(PANEL, "utf8"); + + test("forks one background subshell per model and waits on all", () => { + expect(src).toMatch(/run_model\(\)/); + expect(src).toMatch(/run_model "\$cli" &/); + expect(src).toMatch(/for pid in \$pids; do wait "\$pid"; done/); + }); + + test("unavailable arms warn-skip (not fatal): two bogus arms -> exit 0, no records", async () => { + const out = tmpDir(); + const { stdout, exitCode } = await run(["bash", PANEL, "--models", "bogusA,bogusB", planFile()], { + CMRE_OUT_DIR: out, + }); + expect(exitCode).toBe(0); + expect(stdout).toMatch(/bogusA.*SKIP — bogusA not installed/); + expect(stdout).toMatch(/bogusB.*SKIP — bogusB not installed/); + expect(existsSync(join(out, "records", "bogusA__coherence.json"))).toBe(false); + }); + + test("agy off-macOS is SKIPped (floor is macOS-only), exit 0, no record", async () => { + const stub = unameStubDir("Linux"); + const out = tmpDir(); + const { stdout, exitCode } = await run(["bash", PANEL, "--models", "agy", planFile()], { + PATH: `${stub}:${process.env.PATH}`, + CMRE_OUT_DIR: out, + }); + expect(exitCode).toBe(0); + expect(stdout).toMatch(/agy.*SKIP — agy is macOS-only/); + expect(existsSync(join(out, "records", "agy__coherence.json"))).toBe(false); + }); +}); diff --git a/tests/skills/ce-deep-review-beta-bundle-drift.test.ts b/tests/skills/ce-deep-review-beta-bundle-drift.test.ts new file mode 100644 index 000000000..3a67a24d6 --- /dev/null +++ b/tests/skills/ce-deep-review-beta-bundle-drift.test.ts @@ -0,0 +1,33 @@ +import { describe, expect, test } from "bun:test"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +// ce-deep-review-beta bundles a copy of the canonical cross-model harness (so the installed skill +// is self-contained per AGENTS.md). This is the CI-enforced drift gate: if the canonical files +// change (INCLUDING eval-only changes -- arms.py / panel-critique.sh are shared with the eval +// workflow) without re-running scripts/bundle-harness.sh, this fails. Equality is normalized +// (line endings + trailing whitespace) rather than raw bytes, which is brittle across editors. + +const REPO = join(import.meta.dir, "..", ".."); +const CANON = join(REPO, "scripts/eval/cross_model_review"); +const BUNDLE = join(REPO, "plugins/compound-engineering/skills/ce-deep-review-beta/scripts"); + +const norm = (s: string) => + s.replace(/\r\n/g, "\n").split("\n").map((l) => l.replace(/[ \t]+$/, "")).join("\n").replace(/\n+$/, "\n"); + +// canonical (relative to CANON) -> bundled (relative to BUNDLE) +const FILES: [string, string][] = [ + ["panel-critique.sh", "panel-critique.sh"], + ["arms.py", "arms.py"], + ["validation/agy-readonly.sb.tmpl", "validation/agy-readonly.sb.tmpl"], +]; + +describe("ce-deep-review-beta bundled harness is in sync with canonical (re-run bundle-harness.sh after canonical edits)", () => { + for (const [canon, bundled] of FILES) { + test(`${bundled} matches canonical`, () => { + const c = norm(readFileSync(join(CANON, canon), "utf8")); + const b = norm(readFileSync(join(BUNDLE, bundled), "utf8")); + expect(b).toBe(c); + }); + } +}); diff --git a/tests/skills/ce-deep-review-beta-contract.test.ts b/tests/skills/ce-deep-review-beta-contract.test.ts new file mode 100644 index 000000000..2c216a723 --- /dev/null +++ b/tests/skills/ce-deep-review-beta-contract.test.ts @@ -0,0 +1,83 @@ +import { readFile } from "fs/promises"; +import path from "path"; +import { describe, expect, test } from "bun:test"; +import { parseFrontmatter } from "../../src/utils/frontmatter"; + +const SKILL = "plugins/compound-engineering/skills/ce-deep-review-beta/SKILL.md"; + +async function read(rel: string): Promise<string> { + return readFile(path.join(process.cwd(), rel), "utf8"); +} + +describe("ce-deep-review-beta contract (thin slice)", () => { + test("beta frontmatter: name, [BETA] description, disable-model-invocation, argument-hint", async () => { + const { data } = parseFrontmatter(await read(SKILL)); + expect(data.name).toBe("ce-deep-review-beta"); + expect(typeof data.description).toBe("string"); + expect(data.description as string).toMatch(/^\[BETA\]/); + expect(data["disable-model-invocation"]).toBe(true); + expect(typeof data["argument-hint"]).toBe("string"); + }); + + test("pass 1 invokes ce-doc-review headless and has a failure-UX stop", async () => { + const c = await read(SKILL); + expect(c).toContain('Skill("ce-doc-review", "mode:headless'); + expect(c).toMatch(/Pass 1 failed/); + // gate must not open without panel results + expect(c).toMatch(/[Dd]o not open the consent gate/); + }); + + test("AskUserQuestion is preloaded for the consent gate", async () => { + const c = await read(SKILL); + expect(c).toContain("select:AskUserQuestion"); + }); + + test("consent gate is per-model opt-in (default none) with egress = consent", async () => { + const c = await read(SKILL); + expect(c).toMatch(/multi-select/i); + expect(c).toMatch(/default none/i); + // egress is gated by --models BEFORE the run, never post-hoc + expect(c).toContain("--models"); + expect(c).toMatch(/never\s+filter records post-hoc/i); + }); + + test("consent gate labels carry the egress verb (classifier legibility) + dispatch has a block fallback", async () => { + const c = await read(SKILL); + // option labels must name the egress + vendor, not bare model names (OD-4 legibility) + expect(c).toMatch(/Send the plan to codex \(OpenAI\)/); + expect(c).toMatch(/Send the plan to agy \(Antigravity\)/); + // the dispatch must document a fallback for the auto-mode egress classifier block + expect(c).toMatch(/egress classifier|harness blocks this call/i); + expect(c).toMatch(/allowed-tools` is not sufficient|allowed-tools is not sufficient/i); + }); + + test("verified output: writes the reserved .deep-review.md (skill_phase: verified), rotates, spares the draft", async () => { + const c = await read(SKILL); + expect(c).toContain(".deep-review.md"); + expect(c).toContain("skill_phase: verified"); + expect(c).toContain("verification: quote-grep-backstop"); + // rotation runs before the fresh write (data-loss-safe keep-5) + expect(c).toMatch(/reconcile\.py" rotate/); + // the historical thin-slice draft is left in place, not overwritten + expect(c).toMatch(/deep-review-draft\.md.*(in place|historical)/i); + }); + + test("graceful gitleaks degradation is documented inline", async () => { + const c = await read(SKILL); + expect(c).toMatch(/gitleaks-scan\.sh/); + expect(c).toMatch(/unavailable/); + expect(c).toMatch(/Do NOT block|does NOT block|do not block/i); + expect(c).toContain("content_preview: unavailable"); + }); + + test("Phase 3.5 verification: quote-grep backstop, three verdicts, authoritative + model-blind", async () => { + const c = await read(SKILL); + expect(c).toContain("verify-findings.py"); + expect(c).toMatch(/CONFIRMED/); + expect(c).toMatch(/NOT-FOUND-IN-DOC/); + expect(c).toMatch(/NEEDS-HUMAN/); + // the deterministic backstop is authoritative and must not be overridden by a model + expect(c).toMatch(/authoritative/i); + expect(c).toMatch(/blind to the producing model/i); + }); +}); diff --git a/tests/skills/ce-deep-review-beta-reconcile.test.ts b/tests/skills/ce-deep-review-beta-reconcile.test.ts new file mode 100644 index 000000000..18a473882 --- /dev/null +++ b/tests/skills/ce-deep-review-beta-reconcile.test.ts @@ -0,0 +1,108 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, writeFileSync, existsSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// RU5: reconcile.py — the verified-sidecar helpers. `rotate` is the data-loss-risk surface (it +// deletes old rotations), so it gets the most coverage: keep-N by ISO infix, never touch the base +// or the -draft sidecar. `render-cross-model` is deterministic by-lens verdict-tagged Markdown. + +const REPO = join(import.meta.dir, "..", ".."); +const RECONCILE = join(REPO, "plugins/compound-engineering/skills/ce-deep-review-beta/scripts/reconcile.py"); + +async function py(args: string[]) { + const proc = Bun.spawn(["python3", RECONCILE, ...args], { stdout: "pipe", stderr: "pipe" }); + const stdout = await new Response(proc.stdout).text(); + const stderr = await new Response(proc.stderr).text(); + const exitCode = await proc.exited; + return { stdout, stderr, exitCode }; +} + +function scratch(): string { + return mkdtempSync(join(tmpdir(), "reconcile-")); +} + +describe("RU5 reconcile rotate: keep-N, data-loss-safe", () => { + test("rotates the base out, keeps the 5 newest rotations, deletes older; spares base+draft", async () => { + const d = scratch(); + const base = join(d, "plan.md.deep-review.md"); + const draft = join(d, "plan.md.deep-review-draft.md"); + writeFileSync(base, "BASE"); + writeFileSync(draft, "DRAFT"); + for (const iso of ["2026-05-01T000000Z", "2026-05-02T000000Z", "2026-05-03T000000Z", "2026-05-04T000000Z", "2026-05-05T000000Z", "2026-05-06T000000Z"]) { + writeFileSync(join(d, `plan.md.deep-review.${iso}.md`), `rot ${iso}`); + } + const { stdout, exitCode } = await py(["rotate", base, "--now", "2026-05-29T030000Z", "--keep", "5"]); + expect(exitCode).toBe(0); + const res = JSON.parse(stdout); + + // base was renamed to the new rotation, then pruned set keeps the 5 newest infixes + expect(existsSync(base)).toBe(false); + expect(existsSync(draft)).toBe(true); // never touched + expect(readFileSync(draft, "utf8")).toBe("DRAFT"); + expect(existsSync(join(d, "plan.md.deep-review.2026-05-29T030000Z.md"))).toBe(true); // newest kept + expect(existsSync(join(d, "plan.md.deep-review.2026-05-01T000000Z.md"))).toBe(false); // oldest pruned + expect(existsSync(join(d, "plan.md.deep-review.2026-05-02T000000Z.md"))).toBe(false); + expect(existsSync(join(d, "plan.md.deep-review.2026-05-03T000000Z.md"))).toBe(true); + expect(res.pruned.length).toBe(2); + expect(res.kept.length).toBe(5); + }); + + test("no existing base -> nothing renamed, pruning still bounds rotations to keep", async () => { + const d = scratch(); + for (const iso of ["2026-05-01T000000Z", "2026-05-02T000000Z", "2026-05-03T000000Z"]) { + writeFileSync(join(d, `plan.md.deep-review.${iso}.md`), "r"); + } + const { stdout } = await py(["rotate", join(d, "plan.md.deep-review.md"), "--now", "2026-05-29T030000Z", "--keep", "2"]); + const res = JSON.parse(stdout); + expect(res.rotated).toBeNull(); // no base to rotate + expect(res.kept.length).toBe(2); + expect(res.pruned.length).toBe(1); + }); + + test("same-stamp re-runs never overwrite a prior rotation (collision-safe)", async () => { + // Two rotations in the same second (or an explicit duplicate --now) must not clobber the + // earlier snapshot via os.rename -- that would silently lose a backup before pruning runs. + const d = scratch(); + const base = join(d, "plan.md.deep-review.md"); + writeFileSync(base, "FIRST"); + const r1 = JSON.parse((await py(["rotate", base, "--now", "2026-05-29T030000Z", "--keep", "5"])).stdout); + expect(r1.rotated).toBe(join(d, "plan.md.deep-review.2026-05-29T030000Z.md")); + writeFileSync(base, "SECOND"); // a fresh verified sidecar, rotated again in the same second + const r2 = JSON.parse((await py(["rotate", base, "--now", "2026-05-29T030000Z", "--keep", "5"])).stdout); + expect(r2.rotated).toBe(join(d, "plan.md.deep-review.2026-05-29T030000Z-1.md")); // disambiguated + // both snapshots survive with their distinct content + expect(readFileSync(join(d, "plan.md.deep-review.2026-05-29T030000Z.md"), "utf8")).toBe("FIRST"); + expect(readFileSync(join(d, "plan.md.deep-review.2026-05-29T030000Z-1.md"), "utf8")).toBe("SECOND"); + }); + + test("refuses a path that is not a .deep-review.md sidecar", async () => { + const d = scratch(); + const { exitCode, stderr } = await py(["rotate", join(d, "plan.md"), "--now", "2026-05-29T030000Z"]); + expect(exitCode).not.toBe(0); + expect(stderr).toMatch(/deep-review\.md/); + }); +}); + +describe("RU5 reconcile render-cross-model: deterministic by-lens, verdict-tagged", () => { + test("groups by lens in canonical order, tags verdicts, shows grounding quote on CONFIRMED", async () => { + const d = scratch(); + const vr = join(d, "vr.json"); + writeFileSync(vr, JSON.stringify({ + verified: [ + { model: "codex", lens: "security", id: "s1", text: "secret-read-exfil", verdict: "NEEDS-HUMAN", grounding_quote: null }, + { model: "agy", lens: "coherence", id: "c1", text: "arm drift", verdict: "CONFIRMED", grounding_quote: "remove the terminal hop" }, + { model: "codex", lens: "coherence", id: "c2", text: "phantom CI step", verdict: "NOT-FOUND-IN-DOC", grounding_quote: null }, + ], + })); + const { stdout, exitCode } = await py(["render-cross-model", vr]); + expect(exitCode).toBe(0); + // coherence section precedes security (canonical lens order) + expect(stdout.indexOf("### Coherence")).toBeLessThan(stdout.indexOf("### Security")); + // CONFIRMED precedes NOT-FOUND within coherence (verdict order) + expect(stdout.indexOf("[CONFIRMED]")).toBeLessThan(stdout.indexOf("[NOT-FOUND-IN-DOC]")); + expect(stdout).toContain("**[CONFIRMED]** (agy) arm drift"); + expect(stdout).toContain('grounding quote: "remove the terminal hop"'); + expect(stdout).toContain("**[NEEDS-HUMAN]** (codex) secret-read-exfil"); + }); +}); diff --git a/tests/skills/ce-deep-review-beta-verify.test.ts b/tests/skills/ce-deep-review-beta-verify.test.ts new file mode 100644 index 000000000..f29ac7963 --- /dev/null +++ b/tests/skills/ce-deep-review-beta-verify.test.ts @@ -0,0 +1,149 @@ +import { describe, expect, test } from "bun:test"; +import { mkdtempSync, writeFileSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +// RU4: the deterministic quote-grep verification backstop. verify-findings.py grounds each raw +// cross-model finding against the plan -> CONFIRMED / NOT-FOUND-IN-DOC / NEEDS-HUMAN. Pure function +// of (finding text, doc); blind to the producing model; authoritative. Mechanical -> runs current +// source. + +const REPO = join(import.meta.dir, "..", ".."); +const VERIFY = join(REPO, "plugins/compound-engineering/skills/ce-deep-review-beta/scripts/verify-findings.py"); + +const DOC = [ + "# Plan", + "", + "The premise is to remove the terminal hop so the deep review actually gets run.", + "agy is the default non-codex arm; its read-only floor is a macOS seatbelt.", +].join("\n"); + +function docFile(text = DOC): string { + const p = join(mkdtempSync(join(tmpdir(), "verify-doc-")), "plan.md"); + writeFileSync(p, text); + return p; +} + +async function py(args: string[]) { + const proc = Bun.spawn(["python3", VERIFY, ...args], { stdout: "pipe", stderr: "pipe" }); + const stdout = await new Response(proc.stdout).text(); + const exitCode = await proc.exited; + return { out: stdout.trim() ? JSON.parse(stdout) : null, exitCode }; +} + +describe("RU4 verify-findings: verify-one verdicts", () => { + const doc = docFile(); + + test("CONFIRMED when a substantial verbatim quote exists in the doc", async () => { + const { out } = await py(["verify-one", doc, 'The plan says "remove the terminal hop" up front.']); + expect(out.verdict).toBe("CONFIRMED"); + expect(out.grounding_quote).toBe("remove the terminal hop"); + }); + + test("NOT-FOUND-IN-DOC when a claimed quote is absent", async () => { + const { out } = await py(["verify-one", doc, 'It states "we will migrate to blockchain consensus".']); + expect(out.verdict).toBe("NOT-FOUND-IN-DOC"); + expect(out.grounding_quote).toBeNull(); + }); + + test("NEEDS-HUMAN when there is no substantial quote (paraphrase / implication)", async () => { + const { out } = await py(["verify-one", doc, "The migration lacks a rollback strategy."]); + expect(out.verdict).toBe("NEEDS-HUMAN"); + }); + + test("a lone identifier/filename quote does not trivially CONFIRM", async () => { + const { out } = await py(["verify-one", doc, "The `agy` arm is referenced."]); + expect(out.verdict).toBe("NEEDS-HUMAN"); + }); + + test("normalization: smart quotes + collapsed whitespace still match (no false NOT-FOUND)", async () => { + // finding uses smart quotes and extra spacing around the same phrase + const { out } = await py(["verify-one", doc, 'The doc: “remove the terminal hop”.']); + expect(out.verdict).toBe("CONFIRMED"); + }); + + test("normalization: markdown emphasis in the doc still matches an unemphasized verbatim quote", async () => { + // A real-run artifact: a finding quotes a phrase the doc wrote with *italic* / **bold** markers. + // The emphasis carries no content, so the verbatim quote must still CONFIRM (not false NOT-FOUND). + const emph = docFile("# Plan\n\nThe decision: the order *is* the container — no **freeform projects**."); + const { out: star } = await py(["verify-one", emph, 'It quotes "the order is the container" verbatim.']); + expect(star.verdict).toBe("CONFIRMED"); + // underscore emphasis (_italic_) folds symmetrically + const us = docFile("# Plan\n\nThe rule: _no freeform client projects_ exist in this system."); + const { out: under } = await py(["verify-one", us, 'The doc states "no freeform client projects" plainly.']); + expect(under.verdict).toBe("CONFIRMED"); + }); +}); + +describe("RU4 verify-findings: verify-records is blind to the producing model + aggregates", () => { + test("same finding text -> same verdict regardless of the model in the filename (model-blind)", async () => { + const doc = docFile(); + const dir = mkdtempSync(join(tmpdir(), "verify-recs-")); + const finding = { id: "f1", text: 'cites "remove the terminal hop"' }; + // identical finding under two different model labels + writeFileSync(join(dir, "codex__coherence.json"), JSON.stringify({ findings: [finding] })); + writeFileSync(join(dir, "agy__coherence.json"), JSON.stringify({ findings: [finding] })); + const { out } = await py(["verify-records", doc, dir]); + const verdicts = out.verified.map((r: { verdict: string }) => r.verdict); + expect(verdicts).toEqual(["CONFIRMED", "CONFIRMED"]); // model label did not change the verdict + expect(out.counts.CONFIRMED).toBe(2); + }); + + test("skips stale records from a DIFFERENT plan left in a reused records dir", async () => { + // CMRE_OUT_DIR (default /tmp/cmre-panel/records) is reused across runs. panel-critique.sh + // writes doc_id = `<basename plan .md>__<lens>` into each record. docFile() writes plan.md, so + // the current plan's doc_id base is "plan". A record carrying another plan's doc_id must NOT be + // verified against THIS doc (it would publish a previous review into this sidecar). + const doc = docFile(); // basename -> plan.md -> doc_id base "plan" + const dir = mkdtempSync(join(tmpdir(), "verify-recs-")); + writeFileSync( + join(dir, "codex__coherence.json"), + JSON.stringify({ doc_id: "plan__coherence", findings: [{ id: "cur", text: 'cites "remove the terminal hop"' }] }), + ); + // Stale record from a prior run on a different plan, still sitting in the shared dir. + writeFileSync( + join(dir, "agy__coherence.json"), + JSON.stringify({ doc_id: "otherplan__coherence", findings: [{ id: "stale", text: 'cites "remove the terminal hop"' }] }), + ); + const { out } = await py(["verify-records", doc, dir]); + const ids = out.verified.map((r: { id: string }) => r.id); + expect(ids).toEqual(["cur"]); // only the current plan's record is verified; stale one is skipped + expect(out.counts).toEqual({ "CONFIRMED": 1, "NOT-FOUND-IN-DOC": 0, "NEEDS-HUMAN": 0 }); + }); + + test("counts tally the three verdicts across a mixed records dir", async () => { + const doc = docFile(); + const dir = mkdtempSync(join(tmpdir(), "verify-recs-")); + writeFileSync( + join(dir, "agy__security.json"), + JSON.stringify({ + findings: [ + { id: "a", text: 'quotes "remove the terminal hop" correctly' }, // CONFIRMED + { id: "b", text: 'claims "the plan mandates zero-downtime cutover"' }, // NOT-FOUND + { id: "c", text: "vague concern about scope creep" }, // NEEDS-HUMAN + ], + }), + ); + const { out } = await py(["verify-records", doc, dir]); + expect(out.counts).toEqual({ "CONFIRMED": 1, "NOT-FOUND-IN-DOC": 1, "NEEDS-HUMAN": 1 }); + }); +}); + +describe("RU6b verifier-rate measurement (deterministic; <=5% gate)", () => { + const CORPUS = join(REPO, "plugins/compound-engineering/skills/ce-deep-review-beta/references/calibration/verifier-corpus.json"); + + test("the committed corpus has >=10 grounded and >=10 confabulated items", () => { + const c = JSON.parse(readFileSync(CORPUS, "utf8")); + const grounded = c.items.filter((i: { expected: string }) => i.expected === "CONFIRMED").length; + const confab = c.items.filter((i: { expected: string }) => i.expected === "NOT-FOUND-IN-DOC").length; + expect(grounded).toBeGreaterThanOrEqual(10); + expect(confab).toBeGreaterThanOrEqual(10); + }); + + test("verifier clears the <=5% gate on the corpus (false-CONFIRM and false-NOT-FOUND)", async () => { + const { out } = await py(["measure", CORPUS]); + expect(out.false_confirm_rate).toBeLessThanOrEqual(0.05); + expect(out.false_not_found_rate).toBeLessThanOrEqual(0.05); + expect(out.eligible).toBe(true); + }); +});