diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md new file mode 100644 index 0000000000..47a76235e8 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md @@ -0,0 +1,97 @@ +# Cheaper platform tools in the playground build kit + +Status: PLANNING — no implementation. +Date: 2026-07-20 · Revised 2026-07-27. + +The playground advertises 13 platform-op schemas on every turn — **18,353 tokens** measured — before +the model does anything. Every always-on tool is also a wander target. + +## Strategy + +**Diet the schemas, then make them lazy.** Two levers, that's it. + +| Phase | Lever | From → to | Cut | Effort | +| --- | --- | ---: | ---: | --- | +| 1–2 | schema diet | 18,353 → ~3,000 | 84% | 1 Python file | +| 3 | lazy schema | ~3,000 → ~500 | 97% | 1 runner function | + +Neither needs a permission change or transport work, and both land on all three harnesses at once. +Details: [design.md](design.md) · [plan.md](plan.md). + +## The two levers are different things + +| | the schema | the tool entry | +| --- | --- | --- | +| Diet | permanently shorter | present | +| Lazy schema | fetched on request | present | + +Both are token levers; the tool *count* is unchanged, so neither addresses wander. + +## Lazy activation is out of scope + +Deferring the tool entry itself is the only lever that would reduce tool count — and it is **not in +this plan** until explicitly asked for. It buys ~100 tokens over lazy schema, so its case rests +entirely on wander, which is asserted rather than measured. It is also the only lever needing +per-harness transport work: Pi has the API today, but Claude and Codex go over MCP and need +`notifications/tools/list_changed` — which the local HTTP server cannot send at all, which the stdio +shim would need re-plumbing to send, and which the pinned third-party ACP clients may ignore +mid-turn. Full record: +[alternatives.md](alternatives.md#3--lazy-activation-out-of-scope). + +## What the measurement changed + +Three ops are 88% of the bill, and one schema object embedded twice is 70% of it +([baseline.md](baseline.md)). The cost is schema *depth*, not catalog *length* — which is why the +diet, not the mechanism, is the delivery. + +## What the design review changed + +The original mechanism was a **card/menu invoker**: one `agenta_op(op, args)` tool proxying all 13 +ops. **Rejected 2026-07-27** — routing every op through one name breaks the permission ladder in +four fail-closed sites where a missed site fails open. + +Eight strategies were weighed in total. All of them, with the win each would deliver and why the +seven non-adopted ones are closed, are in one file: **[alternatives.md](alternatives.md)**. + +## Locked decisions + +- Diet ships first, alone. +- Lazy schema is the mechanism. One projection site, all harnesses, real names. +- Lazy activation is **out of scope** — not a later phase. Revisit only if asked for. +- Discovery is **not** lazy — names and one-liners stay in the prompt. +- Card/menu rejected, recorded, not to be re-proposed. +- Op-set curation dropped: ~5% of tokens for a real capability regression. +- Playground overlay only. No saved-agent change. No commits during planning. + +## Open questions + +Neither blocks the plan; both would only matter if lazy activation is ever asked for. + +1. **Wander evidence.** Is there measured evidence that tool *count* causes failures? +2. **Does a client honor `list_changed` mid-turn?** Only matters if 1 comes back positive. + +## Corrections and closed items + +- **Handler flag "default off" — wrong.** It defaults **on** (`platform_tools.py:41`, unset and + empty both enable). All 13 ops advertise; 18,353 is live. *(CodeRabbit, 2026-07-26.)* +- **"Malformed args become a server error" — overstated, then re-sharpened.** + `assertRequiredArguments` runs runner-side against the private spec (`relay.ts:327`, `:369`) + before execution, so lazy schema loses nothing. The diet does lose nested `required` checks, + because it shrinks the private spec too. *(CodeRabbit, 2026-07-27.)* +- **Direct-`call` eligibility heuristic — wrong.** It missed handler-mode `test_run`, 42% of the + bill. Moot now: real names need no eligibility rule. *(CodeRabbit, 2026-07-26.)* +- **Approval leak across ops — not real.** Grant and decision stores key on + `approvedCallKey(toolName, args)` — name **plus** args hash. +- **In-sandbox stdio MCP — already ships.** `tool-mcp-stdio.ts`, bundled and uploaded for Daytona. + Earlier notes treated it as future work. +- **"Disclosure inevitably breaks permission" — overstated.** Only the invoker did. + +## Docs + +- [baseline.md](baseline.md) — measured per-op cost and the concentration finding. +- [context.md](context.md) — scope, glossary, success criteria. +- [research.md](research.md) — how tools reach the model, with `file:line`. +- [design.md](design.md) — the two levers, in full. +- [plan.md](plan.md) — phases, each with its win and exit check. +- [alternatives.md](alternatives.md) — **all 8 strategies, win comparison, and why the rejected ones + were rejected.** diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/alternatives.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/alternatives.md new file mode 100644 index 0000000000..03936036bd --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/alternatives.md @@ -0,0 +1,195 @@ +# Alternatives — every strategy considered, and the win comparison + +The single record of what was weighed and why. Kept so nothing here is re-proposed. +Verified against the runner on `main`, 2026-07-26. Baseline is 18,353 tokens +([baseline.md](baseline.md)). + +## All strategies + +| # | Strategy | What it does | Win | Status | Why not | +| --- | --- | --- | ---: | --- | --- | +| 1 | **Schema diet** | Shrink the two fat schemas to shallow + pointer to prose that already ships | 18,353 → ~3,000 | **adopted** | — | +| 2 | **Lazy schema** | Real names + stub schemas; `load_op` returns the full schema on request | → ~500 | **adopted** | — | +| 3 | **Lazy activation** | Tool entry absent until activated | → ~400 | **out of scope** | ~100 tokens over #2; its case is wander, which is unmeasured. Only lever needing per-harness transport work — free on Pi, hard on Claude/Codex. Detail below | +| 4 | **Card/menu invoker** | One `agenta_op(op, args)` proxies all 13 ops | → ~400 | **rejected** | Breaks the permission ladder in 4 fail-closed sites; a missed site fails **open**. Full analysis below | +| 5 | **Op-set curation** | Drop the 5-op event pack from the default overlay | ~1,052 (5%) | **rejected** | Cannot be decided at run start. A user who pivots to "schedule this daily" mid-conversation finds the capability gone | +| 6 | **Turn-boundary activation** | Activate only between turns, so no mid-turn push is needed | — | **rejected** | Either stops the agent mid-task waiting for the user, or costs a full transcript replay per activation | +| 7 | **SSE + session ids** | Give the local HTTP MCP server a push channel | — | **rejected** | Large rewrite of a deliberately minimal server, to enable #3's ~100 tokens | +| 8 | **Mode-gating** | No build kit in Chat mode | — | **not a substitute** | Removes capability rather than deferring it. Can layer on top | + +**#1 and #2 are the delivery.** Both keep real tool names, touch no permission code, and land on all +three harnesses through one projection site. Detail: [design.md](design.md). + +## #3 — lazy activation, out of scope + +Ops would start **inactive** — not in the model's tool list at all — with a loader adding one on +demand. This is the only lever that reduces tool *count*, which is what the internal-tools review +tied to wander. **Not being built until explicitly asked for.** + +### Why it is out of scope + +**It buys ~100 tokens over #2.** Both leave names and one-liners in the prompt and both fetch +schemas on demand; the difference is 13 stub entries. Its case is wander, not tokens — and wander is +currently asserted, not measured. A lever whose entire justification is unmeasured should not gate +the levers whose justification is measured. + +**It is the only lever needing per-harness transport work**, and two of three harnesses make it +hard: + +| harness | delivery | activation | state | +| --- | --- | --- | --- | +| Pi | native, bundled extension | `pi.setActiveTools()` | **works today** | +| Claude — Daytona | in-sandbox stdio shim | `notifications/tools/list_changed` | possible, unverified | +| Claude — local | HTTP loopback | none | blocked | +| Codex | MCP, same as Claude | same as Claude | after PR #5509 | + +### The hard part, precisely + +**Pi is free.** `getAllTools()` / `getActiveTools()` / `setActiveTools()` exist and are already used +(`extensions/agenta.ts:215`). Mid-turn, in-process, no transport involved. + +**Claude and Codex are MCP-forwarding** (`capabilities.ts:99`), so adding a tool mid-turn requires +the server to *push* `notifications/tools/list_changed`. Three problems stack: + +1. **Local HTTP cannot push at all.** The loopback server is *"stateless JSON mode… no SSE… 405 for + `GET`"* (`tool-mcp-http.ts:22-27`). There is no server→client channel. Fixing it means either + running the stdio shim on the runner host — a security-posture change, since `tool-mcp-stdio.ts` + is written to run inside the sandbox and its docstring flags runner-host stdio as a deliberately + closed hole (#4831) — or building SSE + session ids (#7), which undoes the "simplest conformant + server" design. +2. **The shim's tool list is a startup snapshot.** `tool-mcp-stdio.ts` reads specs once from + `AGENTA_AGENT_TOOLS_PUBLIC_SPECS_FILE`. It would need a runner signal over the relay dir to + re-read and notify, plus `capabilities: {tools: {listChanged: true}}` at `initialize` — today the + shim sends `{tools: {}}` (`tool-mcp-stdio.ts:183`), as does the HTTP server + (`tool-mcp-http.ts:124`). +3. **The clients may ignore the notification mid-turn.** `@agentclientprotocol/claude-agent-acp` and + `codex-acp` are third-party and version-pinned. Whether either re-lists **mid-turn** is + unverified. If they only re-list on reconnect, activation lands at a turn boundary — the agent + must stop and wait for the user mid-task, or pay a full transcript replay per activation. Both + are unacceptable in a builder flow (that is #6). + +Problem 3 is unfixable by us if it fails. Three units of transport work, one outside our control, +for ~100 tokens and an unmeasured reliability claim. + +**If it is ever asked for:** measure wander first. Then Pi only (free, no transport work), and +Claude/Codex only after a spike confirms mid-turn re-listing. + +## #4 — card/menu invoker, in full + +**Status: rejected 2026-07-27.** This was the original proposal, so it gets the full write-up. +**None of it applies to #1 or #2**, because neither changes a tool's name. + +| | Card/menu (#4) | Lazy schema (#2) | +| --- | --- | --- | +| Model calls | proxy: `agenta_op(op, args)` | real op: `commit_revision(args)` | +| Gate sees | `agenta_op` for all 13 ops | the real op | +| `readOnly` / `allow_reads` | one value for 13 ops — breaks | the op's own — works | +| Policy rules (`remove_schedule: deny`) | stop matching, silently | keep matching | +| Arg validation | lost (opaque `args`) | kept (runner-side, private spec) | +| Round trips per new op | 2 | 2 | +| Work | 4-site permission surgery + wire marker + dispatch | 1 projection function | +| Fails toward | **too permissive, silently** | schema not loaded yet, loudly | + +Card/menu's one advantage was that it is transport-agnostic — a plain tool, identical on every +harness. That is why it was proposed first. It stopped being an advantage once the levers were +split: **#2 is transport-agnostic too**, at one function, with no permission change at all. + +### The root cause + +A tool's name does two jobs: it **resolves the private spec** the gate reads its fields from, and it +is **itself matched** by policy rules. The decision ladder (`effectivePermission`, +`permission-plan.ts:125`) uses both: + +```text +gate.specPermission (from resolved spec) -> short-circuits the rest when set +matchingRulePermission(gate, rules) -> matches on gate.toolName itself +defaultPermission(mode, gate) -> allow_reads: gate.readOnlyHint ? allow : ask +``` + +The card/menu invoker routes all 13 ops through one name, `agenta_op`. Every input above then reads +off one spec. + +The regression is not the deferral — it is **building the gate from the invoker's spec instead of +re-resolving the target**. It is fixable, but the fix has to land in four fail-closed sites, and a +missed site fails **open**. + +### What breaks + +**1. `readOnlyHint` — the `allow_reads` default.** `defaultPermission` (`permission-plan.ts:248`): + +```ts +if (mode === "allow_reads") return gate.readOnlyHint === true ? "allow" : "ask"; +``` + +`readOnlyHint` comes from `spec.readOnly`, set per op. One invoker spec covers thirteen: + +- `readOnly: true` → every write executes silently. `commit_revision`, `remove_schedule`, + `remove_subscription` lose their prompt. +- `readOnly: false` → every read prompts. Habituated click-through is worse than not prompting. + +No static value is correct. + +**2. `specPermission`.** Short-circuits the whole ladder (`:129`). One value for thirteen ops, and +it wins over any rule. + +**3. Policy rules stop matching.** `ruleMatches` (`:214`) compares `gate.toolName` exactly. A rule +written as `commit_revision: ask` or `remove_schedule: deny` stops matching once calls arrive as +`agenta_op`. It does not error — it falls through to the default, so `deny` becomes whatever +`allow_reads` decides. Rules authored before the change quietly stop being enforced. + +**4. Grant / stored-decision keys — safe.** Both `ApprovedExecutionGrants` (`responder.ts:85`) and +`ConversationDecisions.take` (`:255`) key on `approvedCallKey(toolName, args)` — name **plus** a +canonical args hash. The invoker's args carry the op name, so distinct ops produce distinct keys. +No cross-op approval leak. The only requirement is that `grant()` and `consume()` key consistently; +a mismatch fails closed (every Pi write refused), which is a bug, not a hole. + +### The four sites + +Each independently resolves a spec from a name and builds its own `GateDescriptor`. + +| # | Site | Path | +| --- | --- | --- | +| 1 | `buildRelayExecutionGuard` | `engines/sandbox_agent/relay-guard.ts:53` | +| 2 | `buildGateDescriptor` (Claude/ACP) | `engines/sandbox_agent/acp-interactions.ts:516` | +| 3 | `buildPiGateDescriptor` (Pi) | `engines/sandbox_agent/acp-interactions.ts:456` | +| 4 | `piDialogAllows` call (in-sandbox) | `extensions/agenta.ts:318` | + +PR #5509 adds an `ExecutableToolGate` seam, so this becomes five once Codex merges. + +Site 4 has an extra problem: the in-sandbox extension only sees the **advertised** set, so under +disclosure it no longer holds the target's `readOnly` and cannot decide `gateViaDialog` locally. + +### What was *not* a risk + +Routing the decision through a sandbox-supplied `args.op` is **not** new trust exposure. The tool +name is already sandbox-supplied (`acp-interactions.ts:307`: *"the envelope is sandbox-origin and +untrusted"*) and is safe because it is validated against a fixed spec map, failing closed otherwise +(`:473`). Reading `args.op` and validating it the same way is the same pattern at the same trust +level. The cost is surface area — four hand-written lookups instead of one. + +### Other costs + +- **Pre-call validation lost.** Pi registers each tool with its real schema + (`extensions/agenta.ts:305`); `assertRequiredArguments` checks required fields (`:318`, + `relay.ts:369`). A generic invoker makes `args` opaque, degrading both to the invoker's loose + schema. +- **Two execution shapes.** Endpoint-mode ops carry a direct `call`; handler-mode `test_run` carries + a `callRef` and runs through `callAgentaTool`. An invoker forwarding only `spec.call` cannot + execute `test_run` at all. +- **Eligibility has no clean rule.** Platform ops do not share one shape, and neither field + identifies them: + + | | direct `call` | `callRef` | + | --- | --- | --- | + | platform op | 12 endpoint ops | `test_run` | + | not a platform op | author's `reference` tool | gateway tool | + + A direct-`call` heuristic misses `test_run` — 7,777 tokens, 42% of the bill — and over-collapses + author tools. A `source:"platform"` wire marker would have been required. + +### Verdict + +The invoker's cost is security-critical work that fails toward "too permissive," silently. The +chosen levers cost one Python file and one projection function, touch no permission code, and fail +toward "the schema isn't loaded yet," loudly. For a system whose job is gating an agent's writes to +a user's own agents, that asymmetry decides it. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.md new file mode 100644 index 0000000000..a26d16bcf4 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.md @@ -0,0 +1,120 @@ +# Baseline — measured advertised token cost + +**Measured 2026-07-26** (tiktoken `o200k_base`) against the live catalog on `main`. This +supersedes the 2026-07-17 numbers that the earlier drafts carried; those were ~19% low because +the catalog grew since. + +## Per-op advertised cost + +Counts the advertised projection the model actually sees — `{name, description, inputSchema}`, +with `inputSchema` = `PlatformOp.resolved_input_schema()` (type-refs expanded, context bindings +stripped), matching `advertisedToolSpec()` in `services/runner/src/tools/public-spec.ts:40`. + +| op | total | schema | descr | note | +| --- | ---: | ---: | ---: | --- | +| `test_run` | 7,777 | 7,593 | 163 | handler-mode (`callRef`), advertised by default; write | +| `commit_revision` | 6,878 | 6,713 | 147 | write | +| `query_spans` | 1,578 | 1,463 | 96 | read | +| `create_schedule` | 425 | 351 | 56 | write | +| `create_subscription` | 393 | 322 | 53 | write | +| `discover_triggers` | 356 | 171 | 165 | read | +| `annotate_trace` | 333 | 208 | 100 | write | +| `discover_tools` | 204 | 124 | 61 | read | +| `test_subscription` | 188 | 134 | 36 | write | +| `remove_schedule` | 69 | 41 | 10 | write | +| `remove_subscription` | 69 | 41 | 10 | write | +| `list_deliveries` | 46 | 11 | 14 | read | +| `list_schedules` | 37 | 11 | 6 | read | +| **total (13 ops)** | **18,353** | | | | + +## The three facts that drive the plan + +1. **Three ops are 88% of the bill.** `test_run` + `commit_revision` + `query_spans` = 16,233 of + 18,353. The other **ten ops combined are 2,120 tokens** — seven of them under 400 each. +2. **One schema object is 70% of the bill.** `_build_agent_template_delta_schema()` + (`sdks/python/agenta/sdk/agents/platform/op_catalog.py:317`) is **6,441 tokens** and is embedded + **twice** — in `commit_revision` and in `test_run`. That is 12,882 of 18,353. +3. **All 13 ops are live by default — 18,353 is the real number.** `test_run` is handler-based + (`handler="tools.agenta.test_run"`, no `method`/`path`) and gated by + `AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS`, but that flag **defaults ON** + (`sdks/python/agenta/sdk/agents/platform/platform_tools.py:41`): + + ```python + if value is None: return True # unset -> ENABLED + return value.strip().lower() not in _DISABLED_ENV_VALUES + ``` + + Unset *and* empty both mean enabled; the resolver skips the op only for an explicit `off` / + `false` / `0` (`_DISABLED_ENV_VALUES`). The resolver's own log line says "explicitly disabled". + + > **Correction.** Drafts before 2026-07-26 (and the 2026-07-17 investigation they came from) + > claimed "default off" and carried a 10,576-token alternate figure. That was wrong. There is no + > alternate figure: `test_run` advertises unless someone opts out, so its 7,777 tokens — 42% of + > the bill — are real, and Phase 1's win is the full ~12,900. + +## Consequence for phasing + +| lever | est. after | cut from 18,353 | risk | +| --- | ---: | ---: | --- | +| Schema diet on `commit_revision` + `test_run` | ~5,500 | ~70% | near-zero (see below) | +| \+ trim `query_spans` filter DSL | ~2,970 | ~84% | low | +| \+ lazy schema | ~500 | ~97% | one projection function; no permission change | +| \+ lazy activation | ~400 | ~98% | **out of scope** — per-harness transport work | + +Two numbers decide the plan's shape: + +- **Lazy schema's marginal win over a completed diet is ~2,500 tokens (~13%)** at the cost of one + function in `public-spec.ts` — and it is structural: schema growth stops inflating the prompt. + (The eager discovery index still grows by ~12 tokens per new op.) +- **Lazy activation's marginal win over lazy schema is ~100 tokens**, which is why it is out of + scope ([alternatives.md](alternatives.md#3--lazy-activation-out-of-scope)). Its case is tool *count* + (wander), not tokens. Weigh its per-harness transport cost against wander evidence only. + +## Why the diet is near-zero risk + +The embedded schema is **advisory duplication of a doc that already ships and is already +mandatory reading**: + +- `references/config-schema.md` is a 3,621-token prose + example reference, shipped as a + `SkillFile` on `BUILD_AN_AGENT_SKILL` + (`sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py:740`). +- The skill body already instructs: *"Read `references/config-schema.md` before your first + `commit_revision`"* (`agenta_builtins.py:579`), and again on failure (`:711`, `:716`). +- That reference states plainly that **the commit endpoint does not validate this shape**. So the + embedded JSON Schema is not enforcing a server contract — it is model guidance that a + better-written, on-demand doc already provides. + +**Top-level required-argument checking is not lost.** The runner re-validates against the +**private** spec before executing — `assertRequiredArguments(spec, req.args)` at +`services/runner/src/tools/relay.ts:327` (client shape) and `:369` (endpoint and handler shapes). +What moves is *where* a malformed call is caught: at the relay rather than pre-call in the harness +(Pi passes `inputSchema` to `registerTool({parameters})`, +`services/runner/src/extensions/agenta.ts:305`). + +**What the diet does cost:** `missingRequiredFields` recurses through `properties`, so today a deep +schema enforces nested `required` too. The diet edits `op_catalog.py`, which shrinks the *private* +spec as well, so those nested checks go away at every layer. Bounded by the same fact that makes the +diet safe: the commit endpoint does not validate the config shape either. Lazy schema does **not** +have this cost — it stubs only the advertisement and leaves the private spec whole. + +## Reproducing + +```python +import sys, json, tiktoken +sys.path.insert(0, "sdks/python") +from agenta.sdk.agents.platform.op_catalog import get_platform_op + +OPS = ("discover_tools","commit_revision","annotate_trace","query_spans","test_run", + "discover_triggers","create_schedule","create_subscription","list_schedules", + "list_deliveries","test_subscription","remove_schedule","remove_subscription") +enc = tiktoken.get_encoding("o200k_base") +total = 0 +for name in OPS: + op = get_platform_op(name) + adv = {"name": f"tools.agenta.{name}", "description": op.description, + "inputSchema": op.resolved_input_schema()} + n = len(enc.encode(json.dumps(adv))) + total += n + print(f"{name:22} {n:>7}") +print(f"{'TOTAL':22} {total:>7}") +``` diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md new file mode 100644 index 0000000000..12ea0d329b --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md @@ -0,0 +1,77 @@ +# Context + +## Problem + +Open a playground agent that shows "Tools: None" and type "hi". The turn carries **18,353 prompt +tokens** of platform-op schema (measured 2026-07-26 — [baseline.md](baseline.md)). Nothing the +author did explains it: the cost is the playground **build kit**, which injects 13 platform ops +into the agent template and advertises every one of them on every turn. + +Two costs, not one: + +- **Tokens**, and they are *concentrated*. `test_run` (7,777) + `commit_revision` (6,878) + + `query_spans` (1,578) are **88%** of the bill; the other ten ops are 2,120 combined. One schema + object — `_build_agent_template_delta_schema()`, 6,441 tokens — is embedded in **both** + `test_run` and `commit_revision`, so **70% of the bill is one object counted twice.** +- **Reliability.** The internal-tools review + (`../builder-agent-reliability/tools-review/part-2-internal-tools.md`) found the same tools are a + *double* cost: "each unused tool is context cost plus a wander target (the capstone showed extra + visible tools derail runs)." + +**These two costs need different fixes, and that is the central finding of this workspace.** +Shrinking schemas fixes the token cost and the wander cost not at all — the tool *count* is +unchanged. Reducing the advertised tool count fixes wander but, after the schema work, buys ~100 +tokens. Conflating them is what made the first draft lead with the expensive, risky lever instead +of the cheap one. + +The tail is *cheap* — ten ops for 2,120 tokens — so catalog growth is not the current pain. Schema +*depth* is. + +## Scope (this delivery) + +- A **schema diet** for the ops carrying deep expanded schemas — `commit_revision`, `test_run`, + then `query_spans`. 84% of the token win, no runner change. +- **Lazy schema** — ops stay advertised under real names with stub schemas; `load_op` returns the + full schema on request. One projection site, all harnesses ([design.md](design.md)). +- A **token baseline** so before/after is measured, not asserted — done ([baseline.md](baseline.md)); + reliability measurement still open. + +## Out of scope + +- **Lazy activation** (deferring the tool entry itself). Out of scope until explicitly asked for — + ~100 tokens over lazy schema, an unmeasured wander case, and the only lever needing per-harness + transport work. Record: [alternatives.md](alternatives.md#3--lazy-activation-out-of-scope). + +## Product language + +- **Platform op** — an existing Agenta endpoint exposed to the agent as a tool, defined in the code + catalog `op_catalog.py` (e.g. `commit_revision`, `query_spans`). +- **Advertised spec** — the `{name, description, inputSchema, …}` projection the model sees; + distinct from the **private resolved spec** the runner executes and validates from. +- **Schema diet** — permanently shrinking a deep, type-expanded `inputSchema` to a shallow one plus + a pointer to prose. The detail is gone; the tool stays visible. +- **Lazy schema** — the full schema still exists but is not in the prompt. `load_op(name)` returns + it on request. The tool stays visible. +- **Lazy activation** — the tool entry itself is absent until activated. The only lever that reduces + tool count, and out of scope here. +- **Card/menu invoker** — the rejected alternative: one `agenta_op(op, args)` tool that both lists + and **proxies** every op. See [alternatives.md](alternatives.md). + +## Success criteria + +1. **Diet:** platform-op prompt cost drops from 18,353 to under ~3,000 tokens with no capability + loss — a lab agent still commits a valid config and runs a test. +2. **Lazy schema:** under ~500 tokens, and adding a catalog op grows the prompt by its index + entry only (name + one-liner, ~12 tokens), never by its schema. Discovery stays eager, so the + index itself still scales linearly; the target is that *schema* growth stops. +3. **Session-level, not turn-level:** total platform-op tokens across a full build session + (discover → wire → commit → test → schedule) drop materially. A no-op turn is *reported*, never + the target. +4. **No safety regression:** `$ctx` bindings and per-op approval behave exactly as today — + preserved by construction (real names reach every gate) and verified per mutating op. +5. **Discovery never regresses:** the model can always name what it is able to do. An op it cannot + see is a capability it will deny having — worse than the token bill. +6. **Cost of laziness bounded:** ≤1 extra round-trip per distinct op used, and no net regression in + session-level tokens. +7. **Wander measured, not assumed** — the input that would decide whether lazy activation is ever + asked for. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md new file mode 100644 index 0000000000..25c6b9e65a --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md @@ -0,0 +1,116 @@ +# Design + +Two levers, in order. Each ships alone and each is useful alone. + +| | Win | Leaves | Effort | Touches permissions | +| --- | ---: | ---: | --- | --- | +| **1 — schema diet** | 84% | ~3,000 | 1 Python file | no | +| **2 — lazy schema** | +13% | ~500 | 1 runner function | no | + +Baseline is 18,353 tokens ([baseline.md](baseline.md)). + +**They are different things and get confused.** What each defers: + +| | the schema | the tool entry | +| --- | --- | --- | +| Diet | permanently shorter | present | +| Lazy schema | fetched on request | present | + +Both are **token** levers — the tool *count* is unchanged, so neither addresses wander. The lever +that would (**lazy activation**, deferring the tool entry itself) is **out of scope** and is +recorded in [alternatives.md](alternatives.md#3--lazy-activation-out-of-scope). + +--- + +# Lever 1 — schema diet (ship first) + +Two ops embed the same 6,441-token object, `_build_agent_template_delta_schema()` +(`op_catalog.py:317`). Two copies = 12,882 tokens = 70% of the bill. Replace the deep expansion +with a shallow schema plus a pointer to prose that already ships. + +**Why it costs almost nothing:** `references/config-schema.md` (3,621 tokens, prose + examples) +ships as a `SkillFile` on `BUILD_AN_AGENT_SKILL` (`agenta_builtins.py:740`), and the skill already +says *"Read `references/config-schema.md` before your first `commit_revision`"* (`:579`). That +reference also states the commit endpoint does not validate the config shape. So the embedded JSON +Schema enforces nothing — it duplicates better guidance the model already has on demand. + +**Shallow, not empty.** Keep top-level keys (`instructions`, `llm`, `tools`, `mcps`, `skills`, +`harness`, `runner`, `sandbox`) with one-line descriptions; stop before expanding nested `$defs`. +A few hundred tokens instead of 6,441. Implement as a depth limit on `expand_type_refs` so it is +reusable by Lever 2. + +**Cost: nested required-field checks inside the collapsed subtree.** Be precise about this, because +the two levers differ. `missingRequiredFields` (`spec-schema.ts`) walks `properties` recursively, so +a deep schema today enforces nested `required` as well as top-level. The diet edits `op_catalog.py`, +which changes the **private** spec too, so those nested checks disappear from every layer including +the relay. Top-level required fields are unaffected. + +That is acceptable here and only here: the commit endpoint does not validate the config shape +either, so a malformed nested config already fails at the server rather than being written. It is +not a general licence — see Lever 2, where the private schema is untouched and nothing is lost. + +**The shallow schema must be permissive.** Harnesses validate against the *advertised* schema before +the relay sees the call (Pi at `extensions/agenta.ts:318`, MCP at `tool-mcp-http.ts:172`), and Pi's +own framework may apply its JSON Schema more strictly than our required-only check. So the depth +limit must not tighten anything: + +- no `additionalProperties: false` on a collapsed node, +- no `required` beyond what the deep schema already required, +- collapsed subtrees typed as bare `object`, never a narrower type or `enum`. + +A depth limit that only *removes* constraints cannot reject a payload the deep schema accepted. + +--- + +# Lever 2 — lazy schema + +Every op stays advertised under its real name with a one-line description and a **stub** schema. +A `load_op` tool returns the full schema on request. + +```text +model → load_op("commit_revision") returns the schema as a tool result +model → commit_revision({...}) real name, real gate, real execution +``` + +**Why it is cheap:** one site. `advertisedToolSpecs` (`public-spec.ts:57`) is the single projection +every harness shares — Pi reads it at `pi-assets.ts:353`, the MCP path at `environment.ts:721`. +Stub there and all three harnesses get it. No notification, no transport work, no per-harness code. + +**Why it is safe:** the tool name never changes, so every permission gate behaves exactly as today. +And unlike Lever 1, the **private spec is untouched** — the stub lives only in the advertisement +projection, so `assertRequiredArguments` at `relay.ts:327` / `:369` still enforces the full schema, +nested `required` included. Nothing is lost; the check simply moves from the harness to the relay. + +**The stub must be permissive too**, for the same reason as Lever 1 and more so, since it is the +only schema the harness sees: `{type: "object"}` with no `required` and no +`additionalProperties: false`. Required fields are then reported by the relay with the exact +field names, which is a usable error for the model. Do **not** stub client tools +(`request_connection`, `request_input`) — the browser fulfils them and they are cheap. + +There is precedent for exactly this rule. Codex wraps MCP calls as `{server, tool, arguments}`, which +forced `unwrapCodexMcpArgs` into `storedDecisionKeyShape` (`permission-plan.ts`, PR #5509) so +permission keys match — it normalized the **arguments** and left the **tool name real**, which is +why it needed no gate surgery. Both levers here stay on the same side of that line. + +**Cost:** one extra round-trip per distinct op used. Tool count is unchanged, so this does nothing +for wander. + +**Discovery stays eager.** Names and one-liners remain in the prompt. + +| in the prompt | cost | +| --- | --- | +| 13 names + tight one-liners | ~150 tokens | +| 13 names + today's descriptions | 917 tokens | + +If the model cannot see that `create_schedule` exists, it answers "I can't schedule things" while +holding the capability — a silent failure worse than the token bill. Carry the index in the +`load_op` description and the always-read `build-an-agent` skill. Do **not** make discovery a tool +call. + +--- + +## Everything else considered + +Eight strategies were weighed, including the **card/menu invoker** that was the original proposal +and was rejected on 2026-07-27. The comparison table, the win each would deliver, and the full +permission analysis live in one place: **[alternatives.md](alternatives.md)**. diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md new file mode 100644 index 0000000000..079bf1da0b --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/plan.md @@ -0,0 +1,99 @@ +# Plan + +**Strategy: diet, then lazy schema.** That is the whole plan — 97% of the token bill, no permission +change, no transport work, all three harnesses. + +| Phase | Lever | From → to | Cut | Effort | +| --- | --- | ---: | ---: | --- | +| 0 | baseline | — | — | done | +| 1 | diet the two fat ops | 18,353 → ~5,500 | 70% | 1 Python file | +| 2 | diet `query_spans` | ~5,500 → ~3,000 | 84% | same file | +| 3 | lazy schema | ~3,000 → ~500 | 97% | 1 runner function | +| 4 | measure | — | — | — | + +**Lazy activation is out of scope.** Not a later phase — it is not in this plan at all until +explicitly asked for. Record: [alternatives.md](alternatives.md#3--lazy-activation-out-of-scope). + +--- + +## Phase 0 — Baseline ✅ + +1. ~~Measure per-op cost.~~ [baseline.md](baseline.md): 18,353 total, top 3 ops = 88%, one + duplicated schema object = 70%. +2. ~~Confirm the live set.~~ All 13 ops advertise; `AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS` defaults + **on** (`platform_tools.py:41`). +3. Add a **resolution-path** test. `advertisedToolSpecs` is a pure `specs.map(...)`, so asserting + every input appears in its output is a tautology. Instead exercise platform resolution with the + handler flag explicitly on and off, and assert the live advertised set (13 ops / 12 ops). + +**Exit:** items 1–2 recorded; resolution-path test green on `main`. + +## Phase 1 — Diet the duplicated schema + +**Win: 18,353 → ~5,500 (70%).** The single biggest step, and the cheapest. + +1. In `op_catalog.py`, stop expanding `_build_agent_template_delta_schema()` inline into + `commit_revision` and `test_run`. Emit a shallow schema (top-level keys + one-liners, no nested + `$defs`) plus a pointer to `references/config-schema.md`. Implement as a depth limit so Phase 3 + reuses it. +2. Update the contract tests and goldens pinning those schemas. +3. Re-measure (expect ~12,900 drop). +4. Lab check: an agent still commits a valid config and `test_run` verifies it. +5. Assert the depth limit only *removes* constraints: no `additionalProperties: false` and no new + `required` on a collapsed node. Test a deeply nested valid config through **both** the Pi and MCP + paths and confirm it is not rejected pre-relay. + +**Known cost:** nested `required` inside the collapsed subtree stops being checked, because the diet +changes the private spec too. Top-level required fields still are. Accepted because the commit +endpoint does not validate the config shape either. + +**Exit:** total ~5,500; tests green; a lab run commits and tests successfully. + +## Phase 2 — Trim `query_spans` + +**Win: ~5,500 → ~3,000 (84% cumulative).** Ends the token problem without touching the runner. + +Shallow the filtering-DSL `$defs` in `_QUERY_SPANS_INPUT_SCHEMA` (1,463 → ~300), pointing at prose +for the DSL. Re-measure. + +**Exit:** under ~3,000 total. + +## Phase 3 — Lazy schema + +**Win: ~3,000 → ~500 (97% cumulative).** Also the structural fix: after this a new op costs its +index entry (~12 tokens), not its schema. Discovery stays eager, so the index still grows linearly; +what stops is *schema* growth. + +1. In `advertisedToolSpecs` (`public-spec.ts:57`), project a **stub** schema for platform ops + instead of the full one. One site — all three harnesses inherit it. +2. Add a `load_op(op)` tool that returns the full schema as its result. Its description carries the + op index (names + one-liners), so discovery is never a round trip. +3. Keep the stub permissive: `{type: "object"}`, no `required`, no `additionalProperties: false`. + Client tools (`request_connection`, `request_input`) are **not** stubbed — the browser fulfils + them and they are cheap. +4. One line in the `build-an-agent` skill about `load_op`. +5. Tests: an op executes with the **same** approval verdict as today; `$ctx` still binds + server-side; a valid deeply-nested payload passes on both the Pi and MCP paths; a call missing a + required arg errors from the relay naming the field, not from the server; an unknown `op` errors + cleanly. + +**No regression:** tool names are unchanged, so every permission gate is untouched. The private +spec is untouched too, so the relay still enforces the full schema including nested `required`. +Enforcement moves from harness-side to runner-side, both pre-execution. + +**Cost:** one extra round-trip per distinct op used. + +**Exit:** ~500 tokens; a lab run completes discover → wire → commit → schedule, loading schemas on +demand, with no permission code changed. + +## Phase 4 — Measure + +1. Session-level tokens (not a no-op turn), flag on vs off. +2. Lab / release-gate pass rate and wander failures. +3. `results.md`; decide default-on. + +## Not in this plan + +Lazy activation, card/menu invoker, op-set curation, turn-boundary activation, SSE + session ids, +mode-gating — all weighed and closed. Reasons and win comparison: +[alternatives.md](alternatives.md). diff --git a/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md b/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md new file mode 100644 index 0000000000..6e2d9aa7b9 --- /dev/null +++ b/docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md @@ -0,0 +1,173 @@ +# Research — how tools reach the model today + +Verified 2026-07-20, **re-verified 2026-07-26** against the code cited — every `file:line` below +was re-checked on `main` and holds. Paths are current (`services/runner/`; some docs still say +`services/agent/`). + +## The path a platform op takes, end to end + +1. **Overlay injects the op set (Python).** The playground build kit is a per-run overlay, not a + committed field. `build_agent_template_overlay()` (`api/oss/src/core/workflows/build_kit.py`) + emits `DEFAULT_BUILD_KIT_OPS` — exactly 13 platform ops (`build_kit.py:27`; `discover_tools`, + `commit_revision`, + `annotate_trace`, `query_spans`, `test_run`, `discover_triggers`, `create_schedule`, + `create_subscription`, `list_schedules`, `list_deliveries`, `test_subscription`, + `remove_schedule`, `remove_subscription`) plus the `request_connection` / `request_input` + client tools and the `build-an-agent` skill. It is served as the static workflow + `__ag__build_kit` and also delivered as read-only `additional_context` at + `api/oss/src/apis/fastapi/applications/router.py` (~L1916). The frontend merges it per run + (atom `workflowBuildKitOverlayReadyAtomFamily`, on by default via + `web/oss/src/lib/helpers/dynamicEnv.ts`). + +2. **Resolution (Python SDK).** `resolve_tools` + (`sdks/python/agenta/sdk/agents/platform/resolve.py` → `tools/resolver.py`) turns each + declared op into a `CallbackToolSpec` carrying a direct `call{method,path,context,args_into}`. + The op's `context_bindings` (self-targeting fields) are stripped from the model-visible schema + by `PlatformOp.resolved_input_schema()` (`.../platform/op_catalog.py`) and re-emitted as + `call.context`. No HTTP round-trip; the catalog fully describes the op. The resolved set rides + the `/run` wire as `customTools`. + +3. **Runner holds every spec privately.** `buildRunPlan` sets `plan.toolSpecs = request.customTools` + (`services/runner/src/engines/sandbox_agent/run-plan.ts:352`). The runner indexes them by name + with `toolSpecsByName(specs)` (`services/runner/src/tools/public-spec.ts:34`) — the ONE index + the relay execute loop, the internal tool-MCP server, and the ACP approval gate all key on. + +4. **Advertisement is a SEPARATE projection.** `advertisedToolSpecs(specs)` + (`public-spec.ts:57`) maps each spec to `{name, description, inputSchema, kind, render, + timeoutMs}` (`AdvertisedToolSpec`, `public-spec.ts:12`). `inputSchema` is the token weight. + It is called at exactly two sites: + - `services/runner/src/engines/sandbox_agent/pi-assets.ts:353` — Pi path; the specs become + `AGENTA_TOOL_PUBLIC_SPECS`, which the bundled extension (`extensions/agenta.ts`) reads and + registers as native Pi tools. + - `services/runner/src/engines/sandbox_agent/environment.ts:721` — Claude/ACP path; the specs + back the synthetic `agenta-tools` MCP server (`tools/mcp-bridge.ts`, `tool-mcp-http.ts`). + +5. **Execution reads the PRIVATE spec, never the advertisement.** For a platform op (direct + `call`), the relay path runs `executeRelayedTool` (`services/runner/src/tools/relay.ts:318`): + `assembleBody(spec.call, args, runContext)` (`relay.ts:384`) merges model args → static body → + `$ctx` context bindings (last, so a bound field always wins); `directCallUrl(...)` + (`relay.ts:385`, guard in `tools/direct.ts:286`) host-locks to the run's own Agenta origin and + confines to the `/api` mount; `callDirect(...)` (`relay.ts:392`) sends with the caller + credential. All keyed by the private `spec` — advertisement is irrelevant to execution. + + **Argument validation is runner-side, against the private spec.** + `assertRequiredArguments(spec, req.args)` runs in the relay loop at `relay.ts:327` (client shape) + and `relay.ts:369` (endpoint and handler shapes), before execution. The harness also validates + pre-call against the *advertised* schema (Pi at `extensions/agenta.ts:318`, MCP at + `tool-mcp-http.ts:172`), but that is a second, earlier check — not the only one. **Consequence: + stubbing the *advertisement* alone does not weaken enforcement** (lazy schema); it moves where a + malformed call is caught. Shrinking the *private* schema does (the diet), because + `missingRequiredFields` walks `properties` recursively, so nested `required` is only enforced + while the private schema still describes it. + +6. **Permission keys on the per-spec gate.** `decide(gate, plan, stored)` + (`services/runner/src/permission-plan.ts:138`) → `effectivePermission` (`:125`) resolves the + spec's own `permission` first, then rule match, then policy default (`allow_reads` → + read-only op runs, write asks). The gate is built from the resolved op's spec, so approval + fidelity lives with the private spec, not the advertised name. + +## Advertisement consumers + +| Harness | Delivery | Consumes `advertisedToolSpecs` at | Notes | +| --- | --- | --- | --- | +| Pi (`pi_core`/`pi_agenta`) | native, via bundled extension | `pi-assets.ts:353` → `AGENTA_TOOL_PUBLIC_SPECS` | No MCP server attached to Pi. | +| Claude (`claude`) | `agenta-tools` MCP server | `environment.ts:721` | Local: HTTP loopback (`tool-mcp-http.ts`). Daytona: in-sandbox stdio shim (`tool-mcp-stdio.ts`). | +| Codex | MCP, same channel as Claude | `environment.ts:721` | MCP-forwarding per `capabilities.ts:99`. PR #5509 open. | + +All paths share one projection function, so a change there covers every harness — which is why the +**schema diet** and **lazy schema** are single-site changes. **Activation** is the exception: it is +per-harness (seam 5). Pi mutates its active-tool list in-process; the MCP harnesses need a +`tools/list_changed` notification over a transport that can push. That asymmetry is the whole reason +lazy activation is out of scope while the schema levers ship. + +## Measured token cost + +**Re-baselined 2026-07-26** (tiktoken `o200k_base`) — full table, method, and reproduction script +in [baseline.md](baseline.md). The 2026-07-17 figures that earlier drafts carried (~15,454 total) +were ~19% low; the catalog grew. + +| Op | Tokens | Note | +| --- | ---: | --- | +| `test_run` | 7,777 | embeds the 6,441-token agent-template delta schema; handler-mode (`callRef`) | +| `commit_revision` | 6,878 | embeds the **same** 6,441-token delta schema | +| `query_spans` | 1,578 | filtering DSL `$defs` | +| others (10 ops) | 2,120 | combined; 7 of them under 400 each | +| **all 13 ops** | **18,353** | the "hi" cost | + +Two findings reorder the whole project: + +- **Concentration.** The top three ops are 88% of the bill. The catalog's *length* is not the + problem; two ops' schema *depth* is. +- **Duplication.** `_build_agent_template_delta_schema()` (`op_catalog.py:317`) is 6,441 tokens and + is embedded twice — 12,882 of 18,353, i.e. **70% of the bill is one object counted twice.** + +All 13 ops advertise by default, so 18,353 is the live figure. `test_run` is handler-based and +gated by `AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS`, but that flag **defaults ON** — unset and empty +both resolve to enabled, and the resolver skips the op only for an explicit `off`/`false`/`0` +(`platform_tools.py:41`). Earlier drafts said "default off" and carried an alternate 10,576 figure; +that was wrong (see [baseline.md](baseline.md)). + +**Handler-mode ops take a different execution shape.** `test_run` is the only handler-mode op +(`_HANDLER_CALL_REFS`, `op_catalog.py:58`). It resolves via `to_call_ref()` to a **`callRef`**, and +`to_call()` explicitly raises for it (`op_catalog.py:175`). So a platform op is *not* always a +direct-`call` spec. This sank the rejected invoker (it could neither identify nor execute +`test_run` — [alternatives.md](alternatives.md)); under the chosen levers it is a non-issue, since each op +keeps its own name and its own execution branch. + +### The diet's replacement already exists + +`references/config-schema.md` ships as a `SkillFile` on `BUILD_AN_AGENT_SKILL` +(`agenta_builtins.py:740`) — 3,621 tokens of prose and worked examples covering the same config +shape. The skill body **already** instructs the model to read it before its first `commit_revision` +(`agenta_builtins.py:579`) and again on failure (`:711`, `:716`). That reference also states the +commit endpoint does **not** validate the config shape — so the embedded JSON Schema enforces no +server contract. It is advisory duplication of better, on-demand guidance. + +## Seams + +Which lever touches which: + +| seam | diet | lazy schema | lazy activation *(out of scope)* | +| --- | --- | --- | --- | +| 1 advertisement projection | — | **yes** | — | +| 2 private spec index | — | — | active subset only | +| 3 execution | — | — | — | +| 4 permission | — | — | — | +| 5 activation | — | — | **yes, per harness** | +| 7 harness-side validation | shallower | stubbed | — | + +The diet changes `input_schema` values in `op_catalog.py` and nothing on this list. + +1. **Advertisement projection** (`advertisedToolSpecs`, `public-spec.ts:57`), consumed at + `pi-assets.ts:353` and `environment.ts:721`. It projects via `specInputSchema(spec)` + (`tools/spec-schema.ts:39`), not `spec.inputSchema` directly. **This one site is the whole of + lazy schema** — every harness reads through it. +2. **Private spec index** (`toolSpecsByName`, `public-spec.ts:34`). Stays complete under every + lever in scope; only the *active* set would change under lazy activation. +3. **Execution — two shapes.** Endpoint-mode ops carry a direct `call` + (`assembleBody` → `directCallUrl` → `callDirect`); handler-mode `test_run` carries a `callRef` + and runs through `callAgentaTool` with `applyContextBindings` (`relay.ts:361`). Both run + unchanged under every lever, because each op is still called by its own name against its own + spec. +4. **Permission.** Untouched — real names reach every gate, so `readOnlyHint`, `specPermission`, + name-matched rules (`ruleMatches`, `permission-plan.ts:214`) and the `allow_reads` default + (`:248`) all behave as today. (The rejected card/menu invoker broke all four; see + [alternatives.md](alternatives.md).) +5. **Activation, per harness.** Pi: `getAllTools`/`getActiveTools`/`setActiveTools` + (`extensions/agenta.ts:215`) — in-process, mid-turn, no transport. MCP harnesses: + `notifications/tools/list_changed`, which requires a server→client push channel — plumbable on + the stdio shim, absent on the local HTTP server (*"stateless JSON mode… no SSE… 405 for `GET`"*, + `tool-mcp-http.ts:22`). `initialize` currently advertises `capabilities: {tools: {}}` (`:124`), + so `listChanged` is not declared. Whether the pinned ACP clients honor it mid-turn is unverified. + **This seam is why lazy activation is out of scope.** +6. **Client tools stay advertised.** `request_connection` / `request_input` must remain + model-visible (the browser fulfils them) and are cheap. +7. **Harness-side schema validation.** Pi registers each tool with its real JSON Schema + (`extensions/agenta.ts:305`) and checks it at `:318`; the MCP server checks at + `tool-mcp-http.ts:172`. Both are *earlier* copies of a check the relay repeats against the + private spec (`relay.ts:327`, `:369`), so the diet shallowing this — or lazy schema stubbing it — + costs an early error message, not enforcement. +8. **Codex.** MCP-forwarding (`capabilities.ts:99`), so it inherits the MCP activation path. It + wraps calls as `{server, tool, arguments}`, already handled by `unwrapCodexMcpArgs` in + `storedDecisionKeyShape` (PR #5509). That fix normalized the **arguments** and left the **tool + name real** — the same rule every lever here follows.