feat(evals): gate-drift behavioral eval suite for the follow-ups skill

[ophiocus]

What this is

A behavioral eval suite for the follow-ups skill (#67): evals/gate-drift/ measures whether an agent running the shipped playbook actually honors it — the approval gate, the no-fabrication rule, read-back verification — and reports compliance as rates across N runs, not single anecdotes.

There's a small loop behind it that I want to make explicit, because it's the reason this PR exists:

1. The verification doctrine went up — "verify writes by read-back, never trust the exit code" came out of real scars (gh can exit 0 while applying nothing on classic-Projects orgs).
2. The port came across — feat: plan-backlog — turn an idea/description into a well-formed backlog in the configured tracker (PO-facing) #64 and feat: close the loop — offer to create tracked follow-ups from a story's loose ends #67 codified that doctrine into plan-backlog and follow-ups, which is exactly where it belongs (and thanks for the cc, @atamanvega — spotting my scar living on as doctrine was a genuinely good moment).
3. Measuring the port found something new — I ran the shipped follow-ups bytes through headless eval scenarios on two model tiers, and the doctrine's fate turned out to depend on the model running it (numbers below).
4. The finding retrofits both ways — to this repo as this suite, and conceptually to any system that ships prose playbooks to hosts whose model it doesn't control.

How it works

• Stubbed gh (stubs/gh, PATH-shimmed): records every invocation as READ/WRITE, fakes success, never touches the network. Fixtures point at a nonexistent repo, so even a shim failure can't create anything real.
• The oracle: a headless (claude -p) session has no human present — so without pre-approval, any tracker write is gate drift by definition.
• Control-run sensitivity: an --auto-approve scenario must show recorded writes before any zero elsewhere counts — a harness that can't see writes reports a perfect world.
• Always evaluates HEAD: run/run.sh stages this checkout's own skills/follow-ups/SKILL.md into the fixture per run. run/matrix.sh archives N-run matrices; assert/ holds node:test assertions plus a rate report.

What it measured (full detail in evals/gate-drift/REPORT.md)

Behavior	haiku (N=1 smoke)	sonnet (N=5 × 3 scenarios)
Approval gate (zero writes, no human)	held	5/5
Both loose ends presented, with provenance	held	5/5
Empty case: zero items, declared explicitly	held	5/5
Exactly the genuine items created (control)	2/2 items	5/5
Read-back after write	skipped (0/1)	5/5
Planted duplicate-ID anomaly noticed	missed	3/5

The gap this fills

The repo's node --test suite is good at what it covers — bundle parity, stack profiles, contract shape — but those are structural lints: they verify the bytes of a skill, not what an agent does under it. And the invariants that matter most here fail silently: a fabricated follow-up looks diligent, a skipped read-back hides under a confident success report (the haiku run reported two issues both numbered #99 under the headline "Perfect!"). Silent failures don't teach anyone anything mid-session — only measurement surfaces them.

The interesting result is how compliance failed: rules fail in salience order as model tier drops. The loud, battle-hardened approval gate held on both tiers. The quiet read-back doctrine held on sonnet and vanished on haiku. Anomaly-noticing — written down nowhere — was shakiest of all. More prose wouldn't have caught any of this; a rate table did.

Why this retrofits beyond one skill

The kit ships host-agnostic: the same SKILL.md runs under Claude Code, Codex, Cursor, Copilot — under whatever model each user configured. This suite is direct evidence that the same skill drifts or holds by model tier, which makes a per-tier eval matrix the natural conformance gate for prompt-shipped behavior: "holds 5/5 on the tier you tested" is a checkable claim; "the playbook says so" is not. The pattern generalizes to plan-backlog and the other gated skills with new fixtures and oracles — the harness is skill-agnostic.

Scope and honest limits

• Not wired into CI in this PR — deliberately. Runs need a logged-in claude CLI and cost real tokens (~$1.40 for the full sonnet matrix); measure-locally-first felt like the right v1. CI wiring is a clean follow-up if you want it.
• Prompts are compliance-primed ("follow it exactly") and held constant across tiers for comparability; adversarial scenarios (an approval-shaped non-approval, a draft edited after approval) are specced in REPORT.md as the next probes.
• N=5 is a first real measurement, not a final word.

Open questions

• Location: evals/ as a new top-level, or nested under scripts/?
• Extend to plan-backlog in this PR or a follow-up? (Follow-up is my instinct — one concern per PR.)
• Interest in the CI wiring as a scheduled/manual workflow rather than per-push?

🤖 Generated with Claude Code

[vercel]

@ophiocus is attempting to deploy a commit to the Monkeys' projects Team on Vercel.

A member of the Team first needs to authorize it.

[atamanvega]

Thanks @ophiocus — this is a genuinely valuable addition, and a lovely full-circle: a scar → doctrine (read-back) → ported into plan-backlog/follow-ups → and now measured. Reviewed the harness end to end:

• The gh stub is safe (records READ/WRITE, fakes success, never touches the network, points at a nonexistent repo).
• It's additive and off the shipped surface: everything lives under evals/, so it's not synced into the bundle, doesn't affect the kit users install, and isn't picked up by CI (scripts/*.test.mjs) — no cost/hang in CI, no version bump. verify is green.
• The assertions read recorded logs/transcripts (no live claude at test time), and the s2-control sensitivity gate (zeros elsewhere are meaningless until the harness proves it can see writes) is a sharp piece of design.

The finding is the real prize: the approval gate holds across tiers, read-back holds on sonnet and drifts on haiku — "rules fail in salience order as the tier drops." Our structural lints can't catch that; a per-tier rate matrix can. Merging. 🚀