[docs] Plan progressive tool disclosure for the playground build kit

docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md

@@ -0,0 +1,59 @@
+# Cheaper platform tools in the playground build kit
+
+Status: PLANNING — design workspace only, no implementation.
+Date: 2026-07-20 · **Revised 2026-07-26** after a re-baseline and a code-grounded security review.
+
+The playground advertises 13 platform-op schemas to the model on every turn — **18,353 tokens**
+measured — before the model has done anything. Every always-on tool is also a "wander target" that
+derails runs.
+
+**What the re-baseline changed.** Three ops are 88% of that bill, and a *single* schema object
+(`_build_agent_template_delta_schema`, 6,441 tokens, embedded in **two** ops) is 70% of it. The
+long tail of ten ops is 2,120 tokens combined. So the win is not spread across the catalog — it is
+concentrated in one duplicated object. See [baseline.md](baseline.md).
+
+That reordered this project. The **schema diet** — originally a side-slice — is now the headline:
+~84% of the win, no runner change, no wire change, no permission change. The **discovery
+meta-toolset** — originally the headline — is a follow-up worth ~14% more, and it carries by far
+the highest risk in the project ([security.md](security.md)).
+
+## Decisions (locked)
+
+- **Schema diet ships first, alone, and is independently valuable.** It is the bulk of the win at
+  near-zero risk. The replacement guidance (`references/config-schema.md`) already ships with the
+  skill and the skill already mandates reading it before `commit_revision`.
+- **The meta-toolset is deferred behind evidence**, not cancelled. Gate it on: a measured
+  post-diet baseline, an answer on prompt caching, and a demonstrated reliability problem that
+  fewer visible tools actually fixes.
+- **Seam for the meta-toolset = runner advertisement layer.** `advertisedToolSpecs` in
+  `services/runner/src/tools/public-spec.ts:57`, consumed at exactly two sites
+  (`pi-assets.ts:353`, `environment.ts:721`). Verified 2026-07-26.
+- **Permission fidelity is a four-site problem, not a one-line one.** See
+  [security.md](security.md). This is the project's dominant risk.
+- **Op-set curation is dropped, not deferred.** Hiding the 5-op event pack saves ~1,052 tokens
+  (~5%) and risks the agent being unable to schedule when a user pivots mid-conversation. Not
+  worth a capability regression. Revisit only with hard wander data.
+- **Skills are out of scope.** Already progressive (64-token announcement).
+- **Playground overlay only.** No change to any saved/committed agent.
+- **No commits during planning.** Implementation happens later on its own branch.
+
+## Deliverables
+
+- [baseline.md](baseline.md) — **measured** per-op cost, the concentration finding, the caching gap.
+- [security.md](security.md) — what collapsing 13 ops behind one name breaks; the four gate sites.
+- [context.md](context.md) — problem, scope, non-goals, product language, success criteria.
+- [research.md](research.md) — the current advertise/execute path with `file:line`, and the seams.
+- [design.md](design.md) — the diet, then the meta-toolset; execution + permission path.
+- [plan.md](plan.md) — the sliced implementation plan, each slice with an exit check.
+- [status.md](status.md) — living source of truth: locked decisions, open questions, next action.
+
+## Intended outcome
+
+**After the diet (Slices 1–2):** a playground turn carries ~2,970 tokens of platform-op schema
+instead of 18,353, with every tool still visible, every schema still authoritative enough to call
+against, and not one line of permission code touched.
+
+**If the meta-toolset later clears its evidence bar (Slice 3):** the model sees two small platform
+tools instead of thirteen, a turn costs a low-hundreds constant that stays flat as the catalog
+grows, and every op still runs with its exact self-targeting binding and approval gate — with the
+four-site permission work done properly and tested per mutating op.

docs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.md

@@ -0,0 +1,104 @@
+# Baseline — measured advertised token cost
+
+**Measured 2026-07-26** (tiktoken `o200k_base`) against the live catalog on `main`. This
+supersedes the 2026-07-17 numbers that the earlier drafts carried; those were ~19% low because
+the catalog grew since.
+
+## Per-op advertised cost
+
+Counts the advertised projection the model actually sees — `{name, description, inputSchema}`,
+with `inputSchema` = `PlatformOp.resolved_input_schema()` (type-refs expanded, context bindings
+stripped), matching `advertisedToolSpec()` in `services/runner/src/tools/public-spec.ts:40`.
+
+| op | total | schema | descr | note |
+| --- | ---: | ---: | ---: | --- |
+| `test_run` | 7,777 | 7,593 | 163 | handler-gated; write |
+| `commit_revision` | 6,878 | 6,713 | 147 | write |
+| `query_spans` | 1,578 | 1,463 | 96 | read |
+| `create_schedule` | 425 | 351 | 56 | write |
+| `create_subscription` | 393 | 322 | 53 | write |
+| `discover_triggers` | 356 | 171 | 165 | read |
+| `annotate_trace` | 333 | 208 | 100 | write |
+| `discover_tools` | 204 | 124 | 61 | read |
+| `test_subscription` | 188 | 134 | 36 | write |
+| `remove_schedule` | 69 | 41 | 10 | write |
+| `remove_subscription` | 69 | 41 | 10 | write |
+| `list_deliveries` | 46 | 11 | 14 | read |
+| `list_schedules` | 37 | 11 | 6 | read |
+| **total (13 ops)** | **18,353** | | | |
+
+## The three facts that drive the plan
+
+1. **Three ops are 88% of the bill.** `test_run` + `commit_revision` + `query_spans` = 16,233 of
+   18,353. The other **ten ops combined are 2,120 tokens** — seven of them under 400 each.
+2. **One schema object is 70% of the bill.** `_build_agent_template_delta_schema()`
+   (`sdks/python/agenta/sdk/agents/platform/op_catalog.py:317`) is **6,441 tokens** and is embedded
+   **twice** — in `commit_revision` and in `test_run`. That is 12,882 of 18,353.
+3. **`test_run` may not be live.** It is handler-based (`handler="tools.agenta.test_run"`, no
+   `method`/`path`) and gated by `AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS`, default off
+   (`sdks/python/agenta/sdk/agents/platform/platform_tools.py:36`). With handlers off the live
+   advertised cost is **10,576**, of which `commit_revision` alone is 65%.
+
+## Consequence for slicing
+
+| lever | est. after | cut from 18,353 | risk |
+| --- | ---: | ---: | --- |
+| Schema diet on `commit_revision` + `test_run` | ~5,500 | ~70% | near-zero (see below) |
+| \+ trim `query_spans` filter DSL | ~2,970 | ~84% | low |
+| \+ discovery meta-toolset | ~400 | ~98% | **high** — see [security.md](security.md) |
+
+The meta-toolset's *marginal* token win over a completed diet is ~2,600 tokens (~14% of the
+original bill). That is the number its risk must be weighed against — not the full 18,353.
+
+## Why the diet is near-zero risk
+
+The embedded schema is **advisory duplication of a doc that already ships and is already
+mandatory reading**:
+
+- `references/config-schema.md` is a 3,621-token prose + example reference, shipped as a
+  `SkillFile` on `BUILD_AN_AGENT_SKILL`
+  (`sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py:740`).
+- The skill body already instructs: *"Read `references/config-schema.md` before your first
+  `commit_revision`"* (`agenta_builtins.py:579`), and again on failure (`:711`, `:716`).
+- That reference states plainly that **the commit endpoint does not validate this shape**. So the
+  embedded JSON Schema is not enforcing a server contract — it is model guidance that a
+  better-written, on-demand doc already provides.
+
+What is genuinely lost by dieting: the harness's own pre-call JSON-Schema validation of the
+argument shape (Pi passes `inputSchema` to `registerTool({parameters})`,
+`services/runner/src/extensions/agenta.ts:305`). Malformed configs would surface as a server
+error instead of a pre-call harness error. Given the server does not validate the config shape
+anyway, this is a small change in *where* the error appears, not whether it is caught.
+
+## Reproducing
+
+```python
+import sys, json, tiktoken
+sys.path.insert(0, "sdks/python")
+from agenta.sdk.agents.platform.op_catalog import get_platform_op
+
+OPS = ("discover_tools","commit_revision","annotate_trace","query_spans","test_run",
+       "discover_triggers","create_schedule","create_subscription","list_schedules",
+       "list_deliveries","test_subscription","remove_schedule","remove_subscription")
+enc = tiktoken.get_encoding("o200k_base")
+total = 0
+for name in OPS:
+    op = get_platform_op(name)
+    adv = {"name": f"tools.agenta.{name}", "description": op.description,
+           "inputSchema": op.resolved_input_schema()}
+    n = len(enc.encode(json.dumps(adv)))
+    total += n
+    print(f"{name:22} {n:>7}")
+print(f"{'TOTAL':22} {total:>7}")
+```
+
+## Open measurement gap — prompt caching
+
+**Nothing in this project has measured whether these tokens are actually billed per turn.** Tool
+definitions are a stable prompt prefix, which is exactly what provider prompt caching targets, and
+both harnesses (Claude Code, Pi) run their own caching. `grep -ri "cache_control|prompt.cach"` over
+`services/runner/src` and `sdks/python/agenta/sdk/agents` returns nothing, so the runner neither
+sets nor inspects cache behavior — it is entirely the harness's.
+
+If the schemas are cached, the real cost is "18K once, a fraction thereafter", and the ROI of
+everything past the diet drops sharply. **Measure this before committing to the meta-toolset.**

docs/design/agent-workflows/projects/progressive-tool-disclosure/context.md

@@ -0,0 +1,85 @@
+# Context
+
+## Problem
+
+Open a playground agent that shows "Tools: None" and type "hi". The turn carries **18,353 prompt
+tokens** of platform-op schema (measured 2026-07-26, tiktoken `o200k_base` —
+[baseline.md](baseline.md)). Nothing the author did explains it: the cost is the playground
+**build kit**, which injects 13 platform ops into the agent template and advertises every one of
+them to the model on every turn.
+
+Two costs, not one:
+
+- **Tokens.** And they are *concentrated*, not spread. `test_run` (7,777) + `commit_revision`
+  (6,878) + `query_spans` (1,578) are **88%** of the bill; the other ten ops are 2,120 tokens
+  combined. One schema object — `_build_agent_template_delta_schema()`, 6,441 tokens, embedded in
+  **both** `test_run` and `commit_revision` — is **70%** on its own.
+- **Reliability.** The internal-tools review
+  (`../builder-agent-reliability/tools-review/part-2-internal-tools.md`) found the same tools are a
+  *double* cost: "each unused tool is context cost plus a wander target (the capstone showed extra
+  visible tools derail runs)."
+
+**These two costs need different fixes, and that is the central finding of this workspace.**
+Shrinking schemas fixes the token cost completely and the wander cost not at all — the tool *count*
+is unchanged. Reducing the advertised tool count fixes wander but, after the diet, buys only ~14%
+more tokens. Conflating them is what made the first draft of this plan lead with the expensive,
+risky lever instead of the cheap one.
+
+It also scales the wrong way: adding a catalog op is a one-line data change that ships to every
+playground agent unconditionally. But note the tail is *cheap* — ten ops for 2,120 tokens — so
+catalog growth is not currently the pain. Schema *depth* is.
+
+## Scope (this delivery)
+
+- A **schema diet** for the ops carrying deep expanded schemas — `commit_revision`, `test_run`,
+  then `query_spans`. This is the primary deliverable: ~84% of the token win, no runner change.
+- A **token + reliability baseline** so before/after is measured, not asserted — done for tokens
+  ([baseline.md](baseline.md)), still open for reliability and for prompt caching.
+- A **designed, risk-registered meta-toolset** (`agenta_ops` + `agenta_op`) held behind an
+  evidence gate, with its permission work scoped honestly ([security.md](security.md)).
+
+## Out of scope
+
+- **Skills.** Already progressive; a 64-token announcement. Untouched.
+- **Op-set curation** (dropping the event pack from the default overlay). *Dropped, not deferred* —
+  it saves ~1,052 tokens and risks the agent being unable to schedule when the user pivots
+  mid-conversation. Revisit only with hard wander data.
+- **External tool discovery (`discover_tools`).** It discovers *Composio* tools to wire into an
+  agent; it stays as-is and is itself one of the ops we would disclose.
+- **User / gateway / code / client tools.** The measured cost is platform ops.
+- **Committed non-playground agents.** They advertise only what their author declared.
+- **Dynamic real-name re-advertisement (M2).** A productionization option, evaluated only if the
+  meta-toolset ships.
+
+## Product language
+
+- **Platform op** — an existing Agenta endpoint exposed to the agent as a tool, defined in the code
+  catalog `op_catalog.py` (e.g. `commit_revision`, `query_spans`).
+- **Advertised spec** — the `{name, description, inputSchema, …}` projection the model sees;
+  distinct from the **private resolved spec** the runner executes from.
+- **Schema diet** — replacing a deep, type-expanded `inputSchema` with a shallow one plus a pointer
+  to on-demand prose guidance. Tool stays visible; only its manual shrinks.
+- **Discovery meta-toolset** — the small fixed set (`agenta_ops`, `agenta_op`) that would stand in
+  for the op schemas: list, describe-on-demand, invoke.
+- **Disclosure** — moving an op's full schema out of the prompt (paid every turn) into a tool
+  result (paid once, only when fetched).
+
+## Success criteria
+
+Rewritten 2026-07-26. The old criterion measured a no-op "hi" turn, which optimizes the cheapest
+case nobody pays for; and it credited the meta-toolset with a win the diet delivers.
+
+1. **Diet:** platform-op prompt cost drops from 18,353 to under ~3,000 tokens, with no capability
+   loss — a lab agent still commits a valid config and runs a test.
+2. **Session-level, not turn-level:** total platform-op tokens across a full build session
+   (discover → wire → commit → test → schedule) drop materially. A no-op turn is *reported*, never
+   the target.
+3. **Caching answered:** we know whether these tokens are billed per turn or cached after the
+   first, and the ROI of anything past the diet is restated against that answer.
+4. **No safety regression:** self-targeting `$ctx` bindings and per-op approval behave exactly as
+   today, verified per mutating op on **both** harness paths.
+5. **Reliability measured, not assumed:** if the meta-toolset proceeds, it is justified by observed
+   wander reduction on the lab matrix — the token case alone (~14%) does not carry it.
+6. **Cost of laziness bounded** (meta-toolset only): ≤1 extra round-trip per distinct op used, a
+   schema fetched at most once per op per conversation, and no net regression in session-level
+   tokens once history-resident schemas are counted.