[docs] Plan progressive tool disclosure for the playground build kit - #5405
Open
ashrafchowdury wants to merge 5 commits into
Open
[docs] Plan progressive tool disclosure for the playground build kit#5405ashrafchowdury wants to merge 5 commits into
ashrafchowdury wants to merge 5 commits into
Conversation
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (7)
📝 WalkthroughSummary by CodeRabbit
WalkthroughAdds planning documentation for progressive tool disclosure in the playground. It records an 18,353-token baseline and defines phased schema dieting followed by lazy schema loading, while excluding lazy activation and rejecting a card/menu invoker alternative. ChangesProgressive tool disclosure
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1cd91cd3-59eb-49de-9205-786764ff4f1d
📒 Files selected for processing (6)
docs/design/agent-workflows/projects/progressive-tool-disclosure/README.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/context.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/design.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/plan.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/research.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/status.md
- Revised the implementation plan for the progressive tool disclosure project, emphasizing the schema diet as the primary focus and deferring the meta-toolset. - Updated the measured token costs in baseline.md, reflecting a re-baseline on 2026-07-26, with detailed per-op costs and implications for slicing. - Added security.md to outline the risks associated with collapsing multiple ops into a single tool name, detailing permission decision impacts and required test coverage. - Enhanced research.md with insights on token costs and the necessity for a re-measurement of the live advertised set. - Created baseline.md to document the measured advertised token costs and the rationale behind the schema diet's low-risk profile. - Updated status.md to reflect the latest changes and decisions made on 2026-07-26, including corrections to earlier drafts and open questions for future consideration.
dosubot
Bot
added
size:XL
There was a problem hiding this comment.
Actionable comments posted: 7
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 31a3541b-18c7-40e5-ae51-d2d04e9f987b
📒 Files selected for processing (8)
docs/design/agent-workflows/projects/progressive-tool-disclosure/README.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/context.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/design.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/plan.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/research.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/security.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/status.md
🚧 Files skipped from review as they are similar to previous changes (3)
- docs/design/agent-workflows/projects/progressive-tool-disclosure/README.md
- docs/design/agent-workflows/projects/progressive-tool-disclosure/design.md
- docs/design/agent-workflows/projects/progressive-tool-disclosure/research.md
… token costs, correct handler mode defaults, and refine slice implementation details.
- Revised the implementation plan for the progressive tool disclosure project, focusing on a diet-first strategy to reduce token costs significantly. - Updated the research documentation to clarify argument validation processes and the implications of schema changes on permission handling. - Removed the security.md and status.md files, consolidating security considerations into the main plan and ensuring clarity on the risks associated with the meta-toolset. - Enhanced the description of seams and their interactions with the schema diet and lazy schema implementations. - Documented the decision-making process for proceeding with the meta-toolset based on evidence from the diet phase.
There was a problem hiding this comment.
Actionable comments posted: 3
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3cdffb36-3eb0-46d7-9d24-6b37c5b16e44
📒 Files selected for processing (7)
docs/design/agent-workflows/projects/progressive-tool-disclosure/README.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/alternatives.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/baseline.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/context.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/design.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/plan.mddocs/design/agent-workflows/projects/progressive-tool-disclosure/research.md
Contributor
Author
|
@mmabrouk I have iterated on this plan and made it simple and easy to implement
|
… argument handling, schema constraints, and token cost implications.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Open a playground agent that shows "Tools: None", type "hi", and the turn carries 18,353 prompt tokens of platform-op schema before the model has done anything. The cost is the build kit. It injects 13 platform ops into the agent template and advertises every one of them on every turn.
Measuring it changed the shape of the problem. The cost is not spread across the catalog, it is concentrated in two schemas.
test_run(7,777) andcommit_revision(6,878) each embed the same 6,441-token agent-template object, so 70% of the bill is one object counted twice. Addquery_spansand three ops account for 88% of the total. The remaining ten ops cost 2,120 tokens combined.The same always-on tools also hurt reliability. The internal-tools review found that extra visible tools become "wander targets" that derail runs.
What this adds
A planning workspace. No code, no runtime change. It records the measurement, the current wiring with
file:linereferences, the design, a phased plan, and every alternative that was weighed and closed.The plan: two levers
Token cost and tool count are different problems that need different fixes. Shrinking a schema does nothing for wander. Removing a tool does little for tokens once the schemas are small. Conflating them is what sent the first draft of this plan at the expensive, risky lever first.
Schema diet. Stop expanding the agent-template object inline into
commit_revisionandtest_run. Emit the top-level keys with one-line descriptions and point atreferences/config-schema.md, a prose reference that already ships as a skill file and that the skill already tells the model to read before its first commit. That reference also states the commit endpoint does not validate the config shape, so the embedded JSON Schema enforces nothing.Lazy schema. Ops stay advertised under their real names with a one-liner and a stub schema. A
load_optool returns the full schema when the model asks for it. This lands at one site,advertisedToolSpecsinservices/runner/src/tools/public-spec.ts, which all three harnesses read through.Nothing is proxied and no tool is renamed, so no permission code changes.
What changed since the first version of this PR
The original proposal was a card/menu invoker: one
agenta_op(op, args)tool routing all 13 ops. It is now rejected, and the analysis is kept inalternatives.mdso it is not re-proposed.Routing every op through one name breaks the permission ladder.
readOnlyHintcomes fromspec.readOnlyand drives theallow_readsdefault, so one invoker spec would cover thirteen ops and no static value is correct. Set it true and every write executes silently. Set it false and every read prompts. Separately,ruleMatchescompares the tool name exactly, so a policy rule written asremove_schedule: denystops matching once calls arrive asagenta_op. It does not error, it falls through to the default. Repairing this means re-resolving the target spec in four fail-closed sites, where a missed site fails open.Lazy schema gets the same token win without any of that, because the model still calls
commit_revisionby its own name.Two corrections also landed:
AGENTA_AGENT_ENABLE_PLATFORM_HANDLERSdefaults on, not off. Earlier drafts carried an alternate 10,576-token figure that does not exist. All 13 ops advertise, so 18,353 is the live number.relay.tsbefore executing. Only the location of the error moves, from the harness to the relay.For reviewers
The point of a plan-only PR is to settle these before anyone writes code:
Notes
Docs only, nothing to run. The 18,353 figure is reproducible with the script in
baseline.md(tiktokeno200k_base, live catalog onmain). Everyfile:lineinresearch.mdwas re-checked againstmainon 2026-07-26.