feat(sdk): build-an-agent reference files + hardened playbook rules by mmabrouk · Pull Request #5106 · Agenta-AI/agenta

mmabrouk · 2026-07-07T00:34:32Z

Context

Stacked on #5105. A builder agent committing "the wrong things in the wrong format" was the observed symptom: at run time nothing showed the model what parameters.agent actually looks like. commit_revision.delta.set is a free object, and the skill named four field paths and stopped. The external agenta-skills kit solved this with reference files the agent reads before acting; every merged PR there fixed a real agent mistake. This ports that knowledge to the internal skill, adapted to internal reality.

Changes

BUILD_AN_AGENT_SKILL now bundles two reference files (via SkillTemplate.files, materialized next to SKILL.md by the runner):

references/config-schema.md: the exact parameters.agent shape derived field-by-field from AgentTemplateSchema and the tool/skill/MCP models: the six tool types with per-type required fields, the skill-template limits and path rules, llm/harness/runner/sandbox enums, delta merge semantics, plus "mistakes that fail the commit" and "mistakes that commit fine but break the run" (model naming is harness-dependent: aliases on claude, provider/id on the Pi harnesses).
references/trigger-inputs.md: the inputs_fields template language, the {event, subscription, scope} fire-context shape, the synthetic schedule event, and the imperative-messages + sibling-payload pattern.

The skill body gains the externally-hardened rules: per-integration connection state is authoritative over the headline match; read the matched event's description before wiring (right integration is not enough); stop when nothing plausibly matches; all four test_run verdicts with the read-the-side-effect-back rule for external writes; prefer narrow filtered tools over list dumps; write the persona as an explicit imperative; a remediation section (validation error means fix the named fields and re-commit; re-point triggers after any commit; a resolved mismatch means silent fallback).

One SDK fix rides along: the skill parser's unresolved-embed scan flagged any string containing the literal text @ag.embed, which the new documentation legitimately contains. The string branch now checks only the @{{ snippet token; structural embeds (a mapping keyed @ag.embed) are still caught.

Scope / risk

Skill content plus the parser narrowing; no op behavior changes. Drift protection: a new test derives every top-level AgentTemplateSchema field name and every tool type discriminator from the pydantic models and asserts the reference file mentions them, so the hand-written docs fail CI when the schema grows. A string that embeds "@ag.embed" inside JSON-encoded text would no longer be caught by the parser scan; the structural and snippet checks still cover the real unresolved-embed shapes (the two existing parsing tests pass unchanged).

Tests

New test_agenta_builtins_reference_files.py (drift protection, path validation, wire serialization): part of the 549-test agents suite, all green; api test_static_catalog.py (31) and test_build_kit_overlay.py (6) green, confirming the files survive the static catalog and overlay path.
Live check: POST /api/workflows/revisions/retrieve for __ag__build_an_agent on a dev stack returned both reference files with full content.

How to QA

Prerequisites: local dev stack.

Steps:

POST /api/workflows/revisions/retrieve with the reserved slug __ag__build_an_agent.
Create an agent from the default template and start a playground run; check the run's skill directory listing (or the run trace) for references/config-schema.md.

Expected result: the skill carries files with both references; a run materializes them next to SKILL.md.

Automated tests:

cd sdks/python && uv run --no-sync python -m pytest oss/tests/pytest/unit/agents/ -q

Edge cases: a config whose skills list contains a real unresolved @ag.embed mapping must still fail parsing with the clear typed error (covered by test_parsing.py).

https://claude.ai/code/session_01N2djTMgXnpk84EqtugHDJB

vercel · 2026-07-07T00:34:38Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 7, 2026 11:19am

coderabbitai · 2026-07-07T00:34:40Z

📝 Walkthrough

Walkthrough

This PR bundles two new reference markdown files (config-schema and trigger-inputs) into the build-an-agent skill via SkillFile, revises skill instruction text across multiple sections, narrows embed-detection logic in parsing.py, and adds a dedicated drift-protection test module.

Changes

Build-an-agent skill reference bundling and instructions

Layer / File(s)	Summary
Reference text constants and skill bundling `sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py`	Adds two multiline reference-text constants (config schema/merge semantics and commit "mistakes"; inputs_fields template language and trigger context shapes), imports `SkillFile`, and attaches both as `files=[SkillFile(...)]` entries on `BUILD_AN_AGENT_SKILL`, mapped to `references/config-schema.md` and `references/trigger-inputs.md`.
Skill instruction text revisions `sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py`	Updates the module docstring on harness defaults, instructs reading the bundled reference files before `commit_revision`, revises loop/discovery/`test_run` verdict guidance, adds trigger re-pointing guidance, updates instruction-writing guidance for personas/tools, and adds a "When something fails" section.
Embed-detection narrowing `sdks/python/agenta/sdk/agents/skills/parsing.py`	Clarifies docstring and narrows `_unresolved_embed_message` string-check to detect only the `@{{` snippet marker, no longer flagging literal `@ag.embed` text in strings.
Drift-protection tests `sdks/python/oss/tests/pytest/unit/agents/test_agenta_builtins_reference_files.py`	New test module with helpers and six tests validating bundled file identity/path safety, schema field coverage, tool discriminator coverage, wire output inclusion, and trigger context shape documentation.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

Agenta-AI/agenta#5068: Both PRs modify the same BUILD_AN_AGENT_SKILL definition in agenta_builtins.py, one adding bundled files/instructions, the other establishing the underlying skill.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the main change: bundled reference files and hardened build-an-agent rules.
Description check	✅ Passed	The description is detailed and clearly matches the changeset, covering the reference files, parser fix, and drift-protection tests.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/build-an-agent-references

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

mmabrouk · 2026-07-07T00:36:10Z

-    elif isinstance(value, str) and (
-        _AG_EMBED_MARKER in value or _AG_SNIPPET_MARKER in value
-    ):
+    elif isinstance(value, str) and _AG_SNIPPET_MARKER in value:


Why this narrowing is load-bearing for this PR: the new config-schema.md documents the @ag.embed syntax, and this skill ships inlined through the static catalog, so its body/files legitimately contain that literal text. The old string branch raised SkillValidationError on it (caught by test_build_kit_overlay). Still caught after narrowing: a structural embed (mapping keyed @ag.embed) and @{{ snippet tokens, which are the two shapes a real unresolved embed takes; both existing parsing tests pass unchanged. Known one-sided piece: middlewares/running/resolver.py still substring-matches @ag.embed on strings, but there it only gates an extra resolve round-trip (a no-op on inlined configs), not an error. Follow-up candidate.

mmabrouk · 2026-07-07T00:36:11Z

+- Dropping `harness`, `runner`, or `sandbox` from a fresh full-object commit. Prefer a narrow
+  `delta.set` that touches only what you change, so the boilerplate survives the deep merge.
+
+## Mistakes that commit fine but break the run


This section exists because adversarial review caught the first draft claiming a wrong model alias "fails the commit". It does not: the model field is a free string, so the commit succeeds and the run silently falls back (the validation in #5104 checks the claude/provider pairing, not model naming). The reference now separates commit-time failures from run-time fallbacks, and points at test_run's resolved block, agreeing with the remediation section in the skill body.

mmabrouk · 2026-07-07T00:36:12Z

@@ -0,0 +1,89 @@
+"""Drift protection for the ``build-an-agent`` skill's bundled reference files.


Drift protection for reviewers: this test derives every top-level AgentTemplateSchema field name and every tool type discriminator from the pydantic models at runtime and asserts the hand-written reference mentions them. When the schema grows a field, CI fails until the doc catches up. This was the chosen alternative to generating the markdown from the schema (readability won; the doc explains semantics, not just shapes).

mmabrouk · 2026-07-07T00:36:34Z

@coderabbitai review

coderabbitai · 2026-07-07T00:36:40Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

mmabrouk · 2026-07-07T11:03:20Z

+}
+```
+
+You are a `pi_agenta` agent. Keep `harness`, `runner`, `sandbox`, and `llm` as they are unless


Why are you saying you're a Pi agent? I mean, that could be a claude agent, no?

Fixed in 23794a1 — the reference is harness-neutral now: "your own harness may be pi_agenta, claude, or pi_core. Whatever it is, keep harness, runner, sandbox, and llm as they are unless the user explicitly asks to change one." The preamble sentence names both harnesses it actually applies to and drops the Pi-only persona claim.

mmabrouk · 2026-07-07T11:04:10Z

+
+### llm
+
+- `model` — the model. How you NAME it depends on the harness (this is the trap):


I mean, you should tell it that it's unlikely that it needs to change that in the Delta, but it should keep things as they are unless the user explicitly asks for that.

Fixed in 23794a1 — the llm section now leads with: "You almost never touch llm in a delta. Keep it exactly as it is unless the user explicitly asks to change the model, provider, or connection. The rules below matter only when they do ask."

mmabrouk · 2026-07-07T11:04:23Z

+  - `claude`: an alias — `default`, `sonnet`, `opus`, or `haiku` — never a raw model id.
+- `provider` — the provider family (`openai`, `anthropic`, ...); inferred from the model string
+  when unset. The `claude` harness reaches `anthropic` only.
+- `connection` — `{ "mode": "agenta" | "self_managed", "slug": "<vault-connection>" }`. `agenta`


Covered by the same lead-in (it governs the model, provider, connection, and extras bullets).

mmabrouk · 2026-07-07T11:05:06Z

+  configure yourself with — which severs them on your next run.
+- `remove` takes dotted paths, e.g. `parameters.agent.tools`.
+
+## Mistakes that fail the commit


make sure to update this re other decissions

Updated in 23794a1 for the #5104 removal: the two sections are now one honest "Mistakes that break your agent" — the commit accepts whatever you send; each bullet states where the mistake actually bites (skill fails to load at run time / config parse rejects it on the next run / Model & Harness never resolves / silent model fallback only visible in test_run's resolved). Verified against the real failure points (parse_skill_templates, coerce_tool_configs, no schema parse anywhere on the commit path).

mmabrouk · 2026-07-07T11:05:49Z

+  `pi_core`/`pi_agenta` harness (Pi selects by provider/id). The model field is a free string, so
+  the commit succeeds — the run then silently falls back to a default model. Match the naming to
+  the harness and check `test_run`'s `resolved` block to catch a fallback.
+"""


I think it should definitely provide real examples of requests. Especially since 90% of the time the two changes that would be made are the instructions and the skills.

Done in 23794a1 — four complete example payloads, ordered instructions → skill → gateway tool → delta.remove, with the lead noting the first two cover the 90% case. The gateway example carries an explicit caveat: the list shown is shortened, resend your ENTIRE current tools list.

mmabrouk · 2026-07-07T11:06:40Z

+Passing no `inputs_fields` at all gives the agent the raw context object as inputs — fine for a
+smoke test, but a real agent should get an explicit `messages` entry so the run starts from an
+imperative instruction.
+"""


I think you need examples here, and I feel you don't mention anything about variants and how to say which variant to use, etc. latests etc.. part of our discussion

Added in 23794a1 — the reference-tool bullet now has three concrete entries: bare ref_by: variant slug = runs that variant's LATEST revision (follows every commit), version: "3" = pinned, ref_by: environment = follows what is deployed there. (Trigger revision-binding semantics are deliberately not stated yet — that's the trigger-latest-binding design doc in flight.)

mmabrouk · 2026-07-07T11:06:54Z

 Everything else is fixed unless the user explicitly asks to change it. Configure yourself with
 `commit_revision` by setting `parameters.agent` fields; do not create a separate app.

+Read `references/config-schema.md` before your first `commit_revision`: it gives the exact shape


each in new line

Reformatted in 23794a1 — one item per line, and the last item renamed to match the new section title.

mmabrouk · 2026-07-07T11:07:09Z

 `remove_subscription`) over harness builtins. Touch Terminal, RemoteTrigger, File tools, or raw
 HTTP only when your wired tools cannot do the job, and say so when you do.

+## When something fails


update given context on validation

Updated in 23794a1 — the remediation bullet no longer assumes server validation: "The commit does not validate your config: a wrong shape commits fine and surfaces at run time... verify with test_run after every commit: read resolved and the executed tool list, and when something is off, check the shape against references/config-schema.md and re-commit the fix; do not start over."

mmabrouk

reviewed pelase update and move lane to stack on correct thing

Claude-Session: https://claude.ai/code/session_01N2djTMgXnpk84EqtugHDJB

…t references Mahmoud's PR #5106 review applied: harness-neutral wording, llm keep-as-is framing, honest no-server-validation failure modes (one 'mistakes that break your agent' section grounded in the real run-time failure points), instructions+skills examples first with a resend-your-entire-list caveat, reference-tool variant/latest/pinned examples, and the remediation rewrite. Claude-Session: https://claude.ai/code/session_01N2djTMgXnpk84EqtugHDJB

mmabrouk

lgtm

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. documentation Improvements or additions to documentation enhancement New feature or request SDK labels Jul 7, 2026

mmabrouk mentioned this pull request Jul 7, 2026

docs(design): agenta-skills sync review — gaps, priorities, implementation status #5108

Closed

mmabrouk commented Jul 7, 2026

View reviewed changes

mmabrouk mentioned this pull request Jul 7, 2026

Embed detection disagrees between skill parsing and the running resolver middleware #5111

Open

mmabrouk added 2 commits July 7, 2026 13:17

feat(sdk): build-an-agent reference files + hardened playbook rules

b62fc13

Claude-Session: https://claude.ai/code/session_01N2djTMgXnpk84EqtugHDJB

mmabrouk force-pushed the feat/build-an-agent-references branch from 1c73d4b to 23794a1 Compare July 7, 2026 11:18

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 7, 2026

vercel Bot deployed to Preview July 7, 2026 11:19 View deployment

mmabrouk commented Jul 7, 2026

View reviewed changes

mmabrouk deleted the branch big-agents July 7, 2026 11:53

mmabrouk closed this Jul 7, 2026

mmabrouk reopened this Jul 7, 2026

mmabrouk changed the base branch from feat/build-kit-op-guidance to main July 7, 2026 11:55

mmabrouk changed the base branch from main to big-agents July 7, 2026 12:44

mmabrouk merged commit 8db2d42 into big-agents Jul 7, 2026
36 checks passed

mmabrouk deleted the feat/build-an-agent-references branch July 7, 2026 12:45

		@@ -0,0 +1,89 @@
		"""Drift protection for the ``build-an-agent`` skill's bundled reference files.


		### llm

		- `model` — the model. How you NAME it depends on the harness (this is the trap):

Uh oh!

Conversation

mmabrouk commented Jul 7, 2026

Context

Changes

Scope / risk

Tests

How to QA

Uh oh!

vercel Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmabrouk commented Jul 7, 2026

Uh oh!

coderabbitai Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmabrouk left a comment

Choose a reason for hiding this comment

Uh oh!

mmabrouk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jul 7, 2026 •

edited

Loading

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading

coderabbitai Bot commented Jul 7, 2026 •

edited

Loading