feat(runner,api,sdk): Complete and enable the test_run self-test tool

[mmabrouk]

Context

The playground has a "build kit": a chat where a builder agent assembles another agent by picking its tools, writing its instructions, and committing the result. The build kit hands that builder a set of platform ops. A platform op is a declarative tool that wraps one Agenta capability, such as committing a revision or querying traces. test_run is the newest op: it fires one real run of the app the builder just built and returns a pass/fail verdict, so the builder can verify its own work.

Part one (#5068) shipped the server half of test_run: the op exists and the API can execute it, but it was flag-gated off, and the runner (the process that executes an agent and relays its tool calls back to the platform) could not dispatch the new tool shape. So a builder agent still could not test the app it built. This PR completes the loop and turns the feature on.

Design doc: docs/design/agent-workflows/projects/build-kit-tools-cleanup/plan-5b.md.

What changed

Three commits.

1. Runner dispatch plus three safety properties (5efd9fa0af: runner, SDK, agent service)

New wire fields. A resolved tool spec (protocol.ts) gains contextBindings and timeoutMs, and the run context gains run.kind. callRef is the dispatch shape for server-handled ops: instead of a concrete URL, the spec carries an opaque reference that the runner posts back to Agenta's /tools/call endpoint. The SDK emits the new fields on the wire (wire.py, wire_models.py, dtos.py), and the golden request files and contract tests on both sides pin the shape.

The target is injected server-side, after the permission check. A self-test must only ever test the calling agent itself, so the model never supplies the target. The spec's contextBindings map argument paths to run-context values (for example target.workflow_variant_id comes from $ctx.workflow.variant.id). The relay (the runner component that executes a tool call once the permission layer allows it) applies these bindings after the permission verdict, on the callRef branch only. It deletes whatever the model put at a bound path, then writes the server value; if a bound value is missing from the run context, the call fails loudly instead of running unbound. The pending-approval record now strips bound paths, so before this change an approval prompt could display a forged target from the model's arguments, and after it the prompt shows only the arguments the model legitimately controls.

The 2-minute budget is honored. test_run carries timeoutMs: 120000, because a real child run takes minutes, not seconds. Before, two flat runner limits would have killed it: the 30-second host fetch (TOOL_CALL_TIMEOUT_MS) and the 60-second extension-to-host relay wait (RELAY_TIMEOUT_MS). Both layers now use the per-spec budget plus a 10-second grace window, so digest work produced right after the child's ceiling is not lost to an abort at the same deadline.

A self-test cannot recurse. The server marks the child invoke with meta.run_kind = "test". The agent service reads that into the run context, the runner forwards it on every tool callback as the x-agenta-run-kind header, and the server rejects a test_run that arrives with that header with a 400. A test run therefore cannot trigger another test run, directly or through a chain.

2. Arm the op (1220754293: API, SDK)

DEFAULT_BUILD_KIT_OPS (the op list in overlay.py; the overlay is the set of extra tools and skills the playground injects into a draft agent) gains test_run, growing the list to 13 ops. The kill-switch env var AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS flips to on by default. Before, unset meant disabled and resolving a handler-mode op raised an error. After, unset means enabled, and an explicit off value (the disabled-value set was widened) skips handler-mode ops with a warning instead of raising. The build-an-agent playbook skill replaces its "verify with query_spans" step with a test_run verify step; query_spans stays in the playbook for reading back scheduled runs after triggers fire.

3. Docs (a90fbd9323)

The plan doc, the landed status, and the runner-to-tool-callback interface page are synced to the shipped behavior.

How to review this PR

Read the runner first, then the SDK, then the service and API. Suggested stops:

1. services/runner/src/protocol.ts: the two new spec fields and run.kind. Check they are optional, so payloads from an older SDK still parse.
2. services/runner/src/tools/direct.ts (applyContextBindings): check that it deletes the model's value at each bound path before writing the server value, and that it throws when the run context lacks a bound value. The property is "never run a bound call with a model-controlled or missing target".
3. services/runner/src/tools/relay.ts: the injection sits in executeAllowedRelayedTool, which only runs after an allow verdict, and only on the callRef branch. pendingApprovalArgs redacts bound paths (and prunes emptied parent objects) from the approval record. Note the comment about Claude: its approval card is harness-rendered before this relay, so only the Pi file-relay path gets the redaction.
4. services/runner/src/tools/callback.ts and dispatch.ts: the per-spec timeout at the host fetch and the relay wait, both with the +10s grace, and the x-agenta-run-kind header on both callback and direct calls.
5. services/runner/src/tools/public-spec.ts: timeoutMs is advertised to the sandbox extension because the child relay wait needs the same budget. Check that callRef, contextBindings, and env still stay out of the advertised spec.
6. SDK: platform_tools.py (default-on flag and skip-with-warning), dtos.py / wire_models.py / wire.py (run.kind serialization), agenta_builtins.py (the playbook's new verify step).
7. Service: services/oss/src/agent/app.py and tracing.py read meta.run_kind into the run context each turn.
8. API: overlay.py is a one-line op-list addition, plus its test.
9. Tests: tool-callref-bindings.test.ts covers the injection and redaction behavior end to end; test_run_kind.py covers the service side; the wire-contract tests and goldens pin the shapes on both sides.
10. The docs commit last; it should describe exactly what the code commits do.

How to QA

Prerequisites: EE dev stack with the agent sidecar (see the subscription-sidecar recipe). No flag needed; platform handlers now default on.

Steps:

1. In the playground build kit chat, ask the builder to create a trivial agent (for example: reply "pong" to any message) and commit it.
2. Ask the builder to verify the agent. Expected: the builder calls test_run and an approval prompt appears. The prompt's arguments contain no target.* fields (bound paths are redacted).
3. Approve. Expected: the run resumes, the child run executes, and the builder reports a pass verdict with the child's output digest.
4. Try to make a test run call test_run again. Expected: the server refuses with a 400.
5. Set AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS=off and re-resolve. Expected: test_run is skipped with a warning, not an error.

Live evidence already collected on the dev stack:

1. Full loop passed: a test_run with a hidden binding paused for approval, resumed through the real playground resume mechanism (approval-responded), the child digest returned "pong", and the model reported the pass.
2. Recursion refused on both entry paths: a recursive test_run gets a 400 when called directly and via the child header chain.
3. Timeout budget holds: a 76-second child run passes (it would have died at both old deadlines), and a 130-second child fails cleanly at the 120-second per-spec budget.
4. Persistence intact: runs persist with 2xx, a real trace_id, and full tool arguments.

Automated tests:

cd services/runner && pnpm test && pnpm run typecheck   # 560 tests
cd sdks/python && uv run python -m pytest oss/tests/pytest/unit/agents -n0   # 534 tests
cd api && uv run python -m pytest oss/tests/pytest/unit/applications oss/tests/pytest/unit/tools -n0   # 108 tests

Scope and risk

Ship the runner and SDK halves together. An old runner receiving the new SDK's specs drops the bindings, so child calls fail loudly instead of silently running unbound. That failure mode is deliberate, but it means you should not deploy the SDK half without the runner half.

Kill-switch semantics changed. AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS=off now skips platform handler ops with a warning; it no longer raises. Anyone relying on the old fail-closed error should know.

Approval surface. The pending-approval record redacts bound $ctx paths, so values bound into child calls (including credentials) never reach the approval UI payload.

Deploy note. dist/ is gitignored, so deploys must run build:extension to pick up the runner changes. The subscription-sidecar recipe was updated for this; the recipe README hunk lands separately because it is entangled with the #5073 lane.

Known environment items, neither caused by this PR: the sidecar CMD does not build the Pi extension (the recipe covers it, see above), and a dev-DB session-table schema drift is tracked separately.

https://claude.ai/code/session_014iPB7HL5PjgT9npyPHaFMT

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agenta-documentation	Ready	Preview, Comment	Jul 5, 2026 4:06pm

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds test_run to the default build-kit ops, makes platform handler-mode default-on with an explicit-disable kill switch, and introduces RunContextRun/run.kind propagation across SDK DTOs, wire models, the agent service, and the runner. The runner gains contextBindings and timeoutMs support for callRef tools, applying bindings post-verdict, redacting them in approval prompts, and forwarding x-agenta-run-kind headers and per-tool timeouts through direct/callback/relay paths. Design docs and extensive tests/goldens are updated accordingly.

Changes

test_run Enablement and Run-Kind/Context-Binding Propagation

Layer / File(s)	Summary
Design docs for Plan 5b docs/design/agent-workflows/projects/build-kit-tools-cleanup/plan-5b.md, status.md, api-design.md, docs/design/agent-workflows/interfaces/cross-service/runner-to-tool-callback.md	Adds a new execution plan document and updates existing design docs to describe default-on handler-mode, contextBindings/timeoutMs propagation, and recursion-guard semantics.
SDK DTOs, wire models, and handler default-on flag sdks/python/agenta/sdk/agents/dtos.py, wire_models.py, __init__.py, utils/wire.py, platform/platform_tools.py	Adds RunContextRun DTO/export and wire model, extends RunContext.to_wire, and changes _platform_handlers_enabled() to default-on with skip-with-warning on explicit disable values.
test_run overlay enablement and skill body update api/oss/src/apis/fastapi/applications/overlay.py, tests, sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py	Adds test_run to DEFAULT_BUILD_KIT_OPS and rewrites the built-in agent skill's verification workflow to use test_run verdicts instead of query_spans.
SDK golden fixtures and wire-contract tests sdks/python/oss/tests/pytest/unit/agents/golden/*.json, test_wire_contract.py, platform/test_op_catalog.py	Updates JSON fixtures and tests to expect runContext.run.kind and contextBindings/timeoutMs on custom tools, plus new handler-flag skip tests.
Agent service run_kind stamping services/oss/src/agent/app.py, tracing.py, services/oss/tests/pytest/unit/agent/test_run_kind.py	Builds RunContext with run.kind from request.meta["run_kind"] and adds tests verifying the propagation.
Runner protocol contract services/runner/src/protocol.ts	Extends ResolvedToolSpec and RunContext with contextBindings, timeoutMs, and run.kind fields.
Runner execution wiring services/runner/src/tools/direct.ts, callback.ts, dispatch.ts, public-spec.ts, relay.ts	Adds applyContextBindings, threads runKind/timeoutMs through callDirect/callAgentaTool, forwards timeouts through relay/dispatch, redacts bound args in approval prompts, and exposes timeoutMs in advertised specs.
Runner tests services/runner/tests/unit/tool-callref-bindings.test.ts, tool-dispatch.test.ts, sandbox-agent-pi-assets.test.ts, wire-contract.test.ts	Adds/updates tests covering binding application, redaction, timeout propagation, and wire-contract assertions.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Runner
  participant ToolRelay
  participant PermissionCheck
  participant AgentaCallback

  Runner->>ToolRelay: startToolRelay(spec, args, runContext)
  ToolRelay->>PermissionCheck: request verdict
  PermissionCheck-->>ToolRelay: allow / deny / pendingApproval
  ToolRelay->>ToolRelay: pendingApprovalArgs (redact bound paths)
  ToolRelay->>ToolRelay: applyContextBindings(args, contextBindings, runContext)
  ToolRelay->>AgentaCallback: callAgentaTool(args, {timeoutMs, runKind})
  AgentaCallback-->>ToolRelay: response

Loading

Possibly related PRs

• Agenta-AI/agenta#4830: Both PRs modify the SDK /run wire-contract modeling layer, aligning run-context/run-kind changes with schema-driven contract work.
• Agenta-AI/agenta#4892: Both PRs wire per-run identity into tool dispatch via runContext-style payloads, including context binding resolution and run-kind header behavior.
• Agenta-AI/agenta#4893: Runner protocol updates for run.kind propagation and server-side context binding application complement the retrieved PR's $ctx-bound direct call spec resolver.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly matches the main change: completing and enabling the test_run self-test tool across runner, API, and SDK.
Description check	✅ Passed	The description is directly related to the changeset and accurately summarizes the runner, SDK, API, and docs updates.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/test-run-5b

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mmabrouk]

Review guide: four inline pointers to the load-bearing seams.

[mmabrouk]

Review guide: the injection seam: post-verdict, callRef branch only, bound paths redacted from the approval record.

[mmabrouk]

Review guide: per-spec budget honored at both deadlines with +10s grace; absent timeout keeps old behavior.

[mmabrouk]

Review guide: the arming flip: 13 ops, grants first; kill switch = skip, not raise.

[mmabrouk]

Review guide: playbook teaches test_run as primary self-test; query_spans stays for scheduled fires.

[mmabrouk]

@coderabbitai review

[mmabrouk]

Feedback needed on four things in particular:

1. The injection seam in relay.ts: bindings apply post-verdict, in the callRef branch only, and the approval record redacts bound paths. Is that the right trust boundary?
2. The timeout grace: per-spec budget at both deadlines with a +10s host-fetch grace window. Sane?
3. The kill-switch default flip: AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS now defaults ON, off means skip-with-warning. Any objection to default-on?
4. The ship-together deploy note: old runner + new SDK drops bindings so child calls fail loud. Acceptable, or do we need a compat shim?

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[mmabrouk]

lgtm if tested locally e2e

[mmabrouk]

Review round 1 addressed. The only actionable item was the conditional lgtm ("lgtm if tested locally e2e"). CodeRabbit finished its review with no findings, and the four inline comments are our own review guides.

I reran the e2e today (2026-07-06) on the local dev stack, at the exact code state of this PR's head (a90fbd9), against the deployed sidecar runner. Four legs, all green:

1. Full self-test loop. The agent called test_run on itself with an allow permission. The child ran a 70-second job, returned "pong-slow", and the digest came back with verdict pass and a real trace id. Total 85.6s, which is past both of the old 30s and 60s deadlines, so the per-spec timeout budget holds.
2. Approval loop end to end. With permission ask, the run paused with a pending approval whose arguments contain only inputs and expectations. No target.* field appears in the approval record, so the redaction holds. Resuming through the real approval-responded mechanism executed the call post-verdict with the server-injected target; the child ran and the verdict was pass (95.3s resume leg).
3. Recursion refusal. A direct POST /api/tools/call with header x-agenta-run-kind: test returned HTTP 400 with "test_run refused: recursive test runs are not allowed."
4. Suites at the same tip. Runner: 561 vitest tests green plus typecheck. SDK: 538 pytest green. API: 108 pytest green.

The kill-switch off path (AGENTA_AGENT_ENABLE_PLATFORM_HANDLERS=off skips with a warning) is pinned by the unit tests in test_op_catalog.py and was checked live on Jul 5; I did not flip the shared stack's env for this rerun.

One useful side finding from the rerun: calling test_run with an API key from a different project fails cleanly with "could not resolve the target workflow variant revision", which confirms the child target resolves inside the caller's project only.

No code changed this round, so the PR head is unchanged.