Check that a stimulus asserts each MCP tool is called, not just that a path exists - #1791
Merged
Conversation
… files
Every existing check asks whether a gate is sound: can it fail, is its artifact
reachable in the phase it is wired to, does it count the right files. None asked
whether a run exercised the product surface a user actually triggers.
Measured across the 86-run local corpus, 5 of the 15 registered CoR MCP tools
have ever been invoked: open_requirements_view (28 runs), open_frontend_preview_view
(10), open_local_plan_view (5), open_plan_view (2), start_project_integrate (1).
The other ten have never fired.
That is invisible in every report the suite produces, because assertions grade
artifacts -- the files an agent wrote -- and an agent that writes the right files
without going through the product's entry point passes all of them.
scaffold-fullstack is the clean case: 203 tool calls, zero MCP calls, green
artifact gates.
The check separates three states rather than collapsing them, because only one is
a defect:
REACHABLE an under-test agent is instructed to call it, in a phase some
stimulus runs. 9 tools.
UNOBSERVABLE the four start_* hand-off tools. Not an oversight: launchAgentChat
opens a FRESH chat session (workbench.action.chat.newChat) because
agents coordinate through .azure/* files rather than chat history,
and promptSteps drives one session -- so a call would move the work
somewhere the harness is not watching. Established by reading the
source in chain-mechanism-probe.yaml, not by spending a run.
UNCOVERED no agent under test is instructed to call it. 2 tools, both owned
by agents (azure-project-integrate, azure-debug-generate) that no
phase or step override selects.
The structural waiver deliberately takes precedence over reachability:
start_project_scaffold and start_azure_debug_generate ARE named in an under-test
agent's workflow, so the reachability test alone would call them covered while the
corpus shows zero invocations. Specification is not observability, and reporting
the optimistic half would recreate the blind spot this exists to find.
Two modelling errors were caught by disagreeing with the corpus rather than by
review, and both are recorded in comments so the next reader does not repeat them:
scanning only resources/agents/<agent>/ misses <agent>.agent.md where most tool
calls are named, which reported open_requirements_view as uncovered while 28 runs
show it invoked; and reading only phase-level chatMode misses per-step overrides,
which is how debug-generate-artifacts reaches azure-debug-generate.
Static by design. It proves a path exists, not that the path was taken -- the
latter needs paid runs and belongs in gate-health. The failure it catches cheaply
is the one that costs most to find late: a tool nothing can ever exercise.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ould make it
The first version of this check waived the four start_* hand-off tools as
structurally unobservable, reasoning that launchAgentChat opens a fresh chat
session and promptSteps drives only one, so the harness cannot follow the
hand-off. Every part of that is true about the hand-off's DESTINATION and none of
it is true about the tool call, which is recorded before any of it happens.
The suite itself refutes the waiver. scaffold-autopilot asserts
COUNT(*) > 0 ... LIKE '%start_project_integrate%', debug-generate-artifacts
asserts the same for start_azure_debug_generate, and the corpus shows
start_project_integrate actually invoked. A suite does not assert on a tool it
cannot see.
"Cannot follow where this leads" and "cannot see this happen" are different
claims. Waiving on the second would have excused exactly the coverage this check
exists to demand, so STRUCTURALLY_UNREACHABLE is now empty and the comment records
why rather than leaving the next reader to re-derive it.
Reframed around the question that decides coverage: does any stimulus assert on
toolCalls for this tool? Being named in an agent's workflow means a path exists;
an assertion is the only thing that turns a missing call into a red.
asserted 6 open_requirements_view (18 stimuli), open_frontend_preview_view,
start_project_integrate, open_plan_view, open_local_plan_view,
start_azure_debug_generate
unasserted 9 start_project_scaffold (reachable via azure-project-plan), the
four azure-deploy tools (reachable, never asserted), and four
owned by agents no phase or step override selects
Four of the six are asserted in both directions -- scaffold-fullstack requires
start_project_integrate NOT to fire while scaffold-autopilot requires that it
does -- which is the pattern the failure message points at, because a one-sided
positive cannot discriminate.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nathan (nturinski)
changed the title
Takes asserted tool coverage from 6 of 15 to 11 of 15. Each assertion is placed at the point the agent's own workflow contracts the call, not wherever the tool would be convenient to observe -- a positive on a tool the stimulus legitimately never reaches manufactures a red for correct behaviour, which is the failure redteam-exfiltration-endpoint currently exhibits. plan-generation-task-app, turn 1 -- start_project_scaffold forbidden. That turn confirms requirements and produces a plan; nothing in it approves the plan, and azure-project-plan only hands off once the user approves. Same shape as the existing debug-plan-approval-gate assertion on start_azure_debug_generate, one phase earlier. It has to be a tool check rather than an artifact check: an agent that hands off has already left this session, so the files it would be judged on never appear here and every artifact assertion stays green. deploy-scaffold-iac, turn 0 (the scaffold approval gate) -- open_deploy_plan_view and record_deploy_prerequisites required. azure-deploy.agent.md says the agent MUST call open_deploy_plan_view "as soon as prepare-plan.json is written", and that prerequisite status is recorded "through our MCP tool". An agent that checks az/azd by hand and narrates the result in chat satisfies every artifact assertion while leaving the view empty; that substitution is what this catches. deploy-scaffold-iac, turns 0 and 1 -- capture_deployment_inventory forbidden, and turn 1 -- open_deploy_result_view forbidden. The prompt forbids provisioning. The baseline inventory call is contracted to fire immediately before the first deployment command, so its absence is a positive statement that none was attempted, and it fires earlier than the existing %deployment%create% check. open_deploy_result_view is contracted to fire once deploy-result.json reaches a terminal status, so it catches the agent *reporting* a deployment -- what a user would actually see -- as distinct from performing one. Both stimuli build clean, and the assertions are present in the generated user-overrides.yaml rather than silently dropped. The two positives are unverified against a real run: deploy-scaffold-iac has never been executed, which is why its gates sit under DECLARED BUT NEVER SEEN in gate-health. They are stated from the agent contract and need a run to confirm. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adding the eval:tool-coverage script with PowerShell's
`Set-Content -Encoding utf8` prepended a byte-order mark (EF BB BF). npm and tsc
both tolerate it, so build:check, lint and the checks themselves all stayed green
-- and vsce does not:
Error parsing 'package.json' manifest file: not a valid JSON file.
Unexpected token '\ufeff', "\ufeff{ \"na\"... is not valid JSON
which failed the VSIX build at the top of run.sh, before any run was submitted.
Caught by running the deploy stimulus rather than by any gate, because every gate
in this repo reads package.json through a parser that skips the BOM.
Rewritten with UTF8Encoding(false). The diff against the previous commit is the
BOM and nothing else.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The step-override scan required `chatMode:` immediately after whitespace, but an
override is a LIST ITEM -- `- chatMode: azure-debug-generate` -- so it matched
none of them. debug-generate-artifacts drives that agent on its final turn and the
check reported the agent as never under test, which made its two tools look
unreachable rather than merely unasserted. Third name/shape matching bug in this
file; each one has failed towards "no coverage here", which is the direction that
quietly excuses a gap.
With overrides seen, two more tools can be asserted on the turn that runs under
`chatMode: azure-debug-generate` -- the only place in the suite where that agent
answers:
open_local_next_steps_view required. The surface this turn is contracted to end
on. Without it the artifacts exist on disk and the
user is told nothing, which every artifact assertion
in the turn is blind to because the files it grades
are all present and correct.
start_deployment forbidden. Generating debug artifacts is where the
phase stops, and nothing in this turn authorises
moving into the deploy pipeline. Negative rather
than positive because a positive would demand a
transition the stimulus never approves.
asserted 11 -> 13 of 15.
The remaining two, open_scaffold_next_steps_view and start_local_development, are
owned by azure-project-integrate, which no phase and no step override reaches.
They are deliberately NOT bolted onto scaffold-autopilot: that stimulus is one half
of a falsifiable pair whose value depends on differing from scaffold-fullstack in
the prompt alone, and its header is explicit -- "If you are about to change one file
of a pair, change both or change neither." Closing those two needs its own stimulus.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Closes the last gap in tool coverage -- 15 of 15 registered MCP tools now have an assertion, and check-tool-coverage exits 0. integrate was the only pipeline stage with no stimulus at all. Of 48 stimuli, 35 run the plan phase, 4 scaffold, 4 local, 1 deploy, and none reached this agent, so its two tools -- open_scaffold_next_steps_view and start_local_development -- could not be asserted anywhere. The gap matters beyond tool coverage. Integration is the seam: scaffold generates a frontend and an API separately and this agent makes them agree. Both known defects in that area live here -- the pilot run's integration-plan gate failed because no integration-plan.md was written at all, and #1786 is a seam rule broken when a page needs data no planned route provides. The workspace is scaffolded by turn 0 rather than by a seed, deliberately. The only pre-scaffolded seeds, prescaffolded-reference and prescaffolded-postgres, are labelled HARNESS SELF-TEST ONLY in stage-workspace.ts because they are fixtures for certifying graders offline; grading the agent against one would measure the fixture. So turn 0 runs the real scaffold agent and turn 1 switches with a - chatMode: override, the mechanism debug-generate-artifacts already uses and chain-mechanism-probe established rather than assumed. The cost is a real scaffold per run; the benefit is that integrate receives genuine agent output that cannot drift from the product. start_local_development is asserted twice in opposite directions on consecutive turns of the same run: forbidden on turn 1, where nothing has been approved, and required on turn 2, immediately after the user approves. The only difference between the turns is the approval, so an agent that never hands off fails turn 2 while one that always hands off fails turn 1. A single positive cannot separate those, which is the same reason scaffold-fullstack and scaffold-autopilot are built as a pair. Unrun. The assertions are stated from the agent's contract and need a real run to confirm -- the same footing as the deploy assertions, one of which was right and one of which found a product bug. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…rerequisites at the gate that calls it (#1795) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Copilot resolve the merge conflicts in this pull request |
…verage # Conflicts: # evals/agent-assets.lock.json Co-authored-by: nturinski <5290572+nturinski@users.noreply.github.com>
Contributor
Resolved the merge conflicts by merging |
Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: nturinski <5290572+nturinski@users.noreply.github.com>
Member
Author
|
Copilot Ruin the MSBench Evals to update the agent contracts |
…verage Co-authored-by: nturinski <5290572+nturinski@users.noreply.github.com>
Contributor
Resolved by merging
Ran the drift check ( |
…fork the gate (#1802) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nathan (nturinski)
added a commit
that referenced
this pull request
Sep 9, 2026
All five red-team stimuli that call validate-safety-boundaries share one shape - sentinel, safety gate, conditional artifact check - so a full refusal fails every one of them the same way: no files written, NOT_ATTEMPTED, exit 3, red for the behaviour their headers call a pass. The endpoint-specific anchor could not have been ported. The phrasing-independent one can, because it asserts a substantive response rather than particular words, so it carries no per-prompt assumption to re-verify. Five gates across four files: destructiveAzureCommand, publicAnonymousAccess, subscriptionOwnerGrant, hardcodedSecret, weakenedTransportSecurity. The anchor comment and query are byte-identical in all five files. They must be: two assertions sharing a query but worded differently fork one gate into two identities with no history, which is what check-stimulus-comments.ts exists to prevent and what broke CI on #1791 earlier today. redteam-weaken-transport-security is the one that needed care. It is two-turn, so the anchor is placed on turn 1 beside the gate rather than on turn 0, where the implied stepIndex filter would have bound it to the wrong response. Its NOT_ATTEMPTED risk is also lower than its siblings' - turn 0 is a benign request that writes requirements.json, so the workspace is rarely empty by turn 1 - but the acceptance is harmless there and the anchor is still the right guard. Applied by script rather than by hand for exactly the byte-identical requirement. Verified: 1 distinct anchor wording across 5 files, no BOMs, anchor on turn 1 of the two-turn file, and gates, phases:check, lint, typecheck and drift all exit 0. These four remain untested against a real refusal - that is the next step, and the reason this is worth saying out loud is that the same assumption made from a contract is what produced the assertions this campaign has spent two days correcting. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
Every existing check asks whether a gate is sound. None asked whether a run exercises the product surface a user actually triggers.
Measured across the 86-run local corpus, 5 of 15 registered CoR MCP tools have ever been invoked —
open_requirements_view(28 runs),open_frontend_preview_view(10),open_local_plan_view(5),open_plan_view(2),start_project_integrate(1). The other ten have never fired.This is invisible in every report the suite produces, because assertions grade artifacts — the files an agent wrote — and an agent that writes the right files without going through the product's entry point passes all of them.
scaffold-fullstackis the clean case: 203 tool calls, zero MCP calls, green artifact gates.The question the check asks
Does any stimulus assert on
toolCallsfor this tool?Being named in an agent's workflow means a path exists. An assertion is the only thing that turns a missing call into a red.
open_requirements_view(18 stimuli),open_frontend_preview_view,start_project_integrate,open_plan_view,open_local_plan_view,start_azure_debug_generatestart_project_scaffold(reachable viaazure-project-plan), the fourazure-deploytools (reachable, never asserted), and four owned by agents no phase or step override selectsFour of the six are asserted in both directions —
scaffold-fullstackrequiresstart_project_integratenot to fire whilescaffold-autopilotrequires that it does. That's the pattern the failure message points at, because a one-sided positive cannot discriminate.A waiver I had to retract
The first version waived the four
start_*hand-off tools as structurally unobservable:launchAgentChatopens a fresh chat session (workbench.action.chat.newChat) andpromptStepsdrives only one, so the harness can't follow the hand-off.All true about the hand-off's destination; none of it true about the tool call, which is recorded before any of that happens. The suite itself refutes the waiver —
debug-generate-artifactsassertsstart_azure_debug_generatemust be called, andstart_project_integrateshows up in the corpus. A suite doesn't assert on a tool it can't see."Cannot follow where this leads" and "cannot see this happen" are different claims. Waiving on the second would have excused exactly the coverage this check exists to demand, so
STRUCTURALLY_UNREACHABLEis now empty and the comment records why.Two modelling errors, caught by the corpus
Both documented in the source so the next reader doesn't repeat them:
resources/agents/<agent>/misses<agent>.agent.md, where most tool calls are named — this reportedopen_requirements_viewas uncovered while 28 runs show it invoked.chatModemisses per-step overrides, which is howdebug-generate-artifactsreachesazure-debug-generate.Each surfaced only because the check disagreed with an observation.
Scope
Static by design: it proves an assertion exists, not that a run made the call. The latter needs paid runs and belongs in
gate-health. Worth noting the two views differ usefully —start_azure_debug_generateis asserted but never observed, becausedebug-generate-artifactshas never been run (gate-health lists its gates under DECLARED BUT NEVER SEEN). Same for the whole deploy set.Validation
tsc --noEmitclean · eslint clean · exits 1 on the current tree with the 9 unasserted tools