feat(agent): move the agent runtime into the SDK behind backend/harness ports

docs/design/agent-workflows/README.md

@@ -1,126 +1,53 @@
-# Agent Workflows
-
-Status: context draft. Research and design to follow.
-
-## Summary
-
-Add a new workflow type to the backend: **agents**. Today the backend runs
-prompt-style workflows (completion, chat, LLM-as-a-judge). Agents are different. An
-agent runs inside a sandbox, executes tools over multiple turns, returns a multi-message
-output, and is instrumented end to end. Agents run on a **pi.dev** harness by default,
-and the same harness can run locally so a configuration pulled from the server behaves
-the same on a developer machine.
-
-This document only captures context. It does not propose a solution yet. The research
-topics in [Open research topics](#open-research-topics) will be assigned to subagents and
-written up in sibling files.
-
-## What an agent is
-
-An agent is a configured, sandboxed, instrumented runtime that:
-
-- Boots a sandbox through startup hooks that lay down files and inject secrets.
-- Runs a harness (pi by default, configurable) that drives the model and its tools.
-- Produces a multi-message output rather than a single completion.
-- Carries a `session_id` so a run can be identified and, later, have its state stored.
-- Emits instrumentation through pi instruments for tracing and observability.
-
-## Agent configuration
-
-The agent configuration is what gets stored on the server, versioned as a workflow
-revision, and pulled down to run locally. It includes:
-
-- **`AGENTS.md`** — the agent's instructions.
-- **Skills** — the skills available to the agent.
-- **Model** — the model the agent runs on.
-- **Tools** — the tools the agent has access to.
-- **Files** — files that are part of the config and are laid into the sandbox by the
-  startup hook.
-- **Secrets** — for example an OpenAI key, injected into the sandbox by the startup
-  hook.
-- **Harness** — which harness runs the agent. Defaults to pi; configurable.
-
-## Runtime model
-
-- **Sandbox.** Agents run in a Daytona sandbox, or any sandbox provider that works with
-  our port. The sandbox is initialized by startup hooks: file setup, then secrets setup.
-- **Harness.** The harness (pi by default) is the layer that exposes tools and drives the
-  agent loop. It is configurable per agent.
-- **Output.** A run returns multiple messages, not one completion.
-- **Instrumentation.** Runs are instrumented with pi instruments.
-- **Sessions.** Each run has a `session_id`. Future work adds session storage alongside
-  global storage so session state can persist across runs.
-
-## Local execution parity
-
-The same harness that runs server-side must run locally on pi.dev abstractions (tools and
-the rest). A user can pull an agent's configuration from the server and run it locally
-with the same behavior. Local-server parity is a first-class requirement, not an
-afterthought.
-
-## What the research established
-
-Full write-ups live in [`research/`](research/). The load-bearing conclusions:
-
-- **pi.dev is "Pi"**, an open-source TypeScript/Node agent harness by Earendil Inc. (MIT,
-  ~v0.79.4). It is local-first (a CLI/SDK/RPC, not a hosted service) and moves fast (0.x,
-  roughly weekly releases). There is no Python SDK.
-  See [`research/pi-interaction.md`](research/pi-interaction.md),
-  [`research/open-questions.md`](research/open-questions.md).
-- **Pi can run fully diskless.** Via the SDK's `createAgentSession`, AGENTS.md
-  (`systemPromptOverride`/`agentsFilesOverride`), skills (`skillsOverride`), tools
-  (`customTools`), LLM auth (`setRuntimeApiKey` / `AuthStorage.inMemory()` / env), and
-  session/settings/model state (`*.inMemory()`) are all in-memory. The only forced disk
-  write is bash output spillover to `os.tmpdir()`, redirected with `TMPDIR` to a per-run
-  tmpfs. See [`research/diskless-in-memory-config.md`](research/diskless-in-memory-config.md).
-- **"pi instruments" is not a product.** Pi emits no OTel by itself. Instrumentation is a
-  Pi extension on the `pi.on(...)` event bus that turns lifecycle events into OTLP spans.
-  Agenta already ingests OTLP at `POST /otlp/v1/traces` with adapters for GenAI semconv
-  and OpenInference, so `gen_ai.*` spans flow with little new backend code. Watch the
-  token-attribute drift (`input_tokens`/`output_tokens` vs the mapped
-  `prompt_tokens`/`completion_tokens`). See
-  [`research/otel-instrumentation.md`](research/otel-instrumentation.md).
-- **The harness seam is ours to build.** Pi's own "harness" concept is not a swap point
-  for Codex or Claude Code. The recommended shape is a thin TypeScript wrapper that drives
-  Pi's SDK with the in-memory overrides above and exposes our own protocol on a port. That
-  wrapper is the "works with our port" contract, the swappable-harness boundary, and the
-  local/server parity point. See [`research/auth-secrets.md`](research/auth-secrets.md).
-- **One shared sandbox is viable for v1.** Daytona supports one long-lived sandbox reused
-  across runs. It does not support swapping a volume per execution (volumes mount at create
-  time only). Per-run isolation comes from process memory plus a per-run tmpfs, not a
-  volume, which the diskless finding makes clean. Concurrency is contended, so bound it.
-  See [`research/sandbox-sharing.md`](research/sandbox-sharing.md),
-  [`research/daytona-sandbox.md`](research/daytona-sandbox.md).
-
-## POC work packages
-
-The POC runs as parallel tracks. Each has its own folder with scope and a definition of
-done. WP-1 and WP-2 run against a local Pi install first (no Daytona). WP-3 takes the
-sandbox path in parallel. WP-4 and WP-5 are design tasks that feed the WP-2 interface. WP-6 registers the agent as a
-backend workflow type and template, and defines its configuration and connection to the
-running agent.
-
-- [`wp-1-pi-tracing/`](wp-1-pi-tracing/README.md) — install Pi locally and send its agent
-  telemetry to Agenta as clean, structured traces.
-- [`wp-2-agent-service/`](wp-2-agent-service/README.md) — a new service that wraps Pi and
-  exposes a completion/chat-style interface, with auth and AGENTS.md set up in memory.
-- [`wp-3-daytona-sandbox/`](wp-3-daytona-sandbox/README.md) — create a Daytona sandbox with
-  Pi installed, inject files and secrets, run an agent, and stream output back.
-- [`wp-4-multi-message-output/`](wp-4-multi-message-output/README.md) — define how an
-  agent's multi-message output is shaped, streamed, stored, and surfaced.
-- [`wp-5-chat-vs-completion/`](wp-5-chat-vs-completion/README.md) — decide the interface
-  contract; start with chat that takes a single input.
-- [`wp-6-workflow-type-and-template/`](wp-6-workflow-type-and-template/README.md) — register
-  the agent as a new backend workflow type and template; define its config (model) and the
-  connection to the running agent.
-- [`wp-7-tools/`](wp-7-tools/README.md) — make runnable tools part of the agent config; resolve
-  Composio actions into Pi tools and route tool calls back through the existing
-  `POST /tools/call`, with MCP and workflow-as-tool as future adapters.
-- [`wp-8-rivet-acp-runtime/`](wp-8-rivet-acp-runtime/README.md) — re-platform the service onto
-  `rivet-dev/sandbox-agent` so the agent is driven over ACP and the harness (Pi, Claude Code,
-  Codex) becomes a config value, running locally first; tools, Daytona, and the folder jail deferred.
-
-## Related work
-
-- [`../prompt-runtime-unification/`](../prompt-runtime-unification/README.md) — the
-  prompt-side runtime that "future agent-style services" were already anticipated against.
+# Agent workflows
+
+This folder documents a proof of concept: running a coding agent as an Agenta workflow.
+
+Agenta runs prompt workflows today (completion, chat, the LLM judge). Each calls a model
+once and returns one answer. An agent is different. It runs a loop, calls tools across many
+turns, and returns a final answer. This PoC adds the agent as a new workflow type behind the
+same `/invoke` contract, traced into the same spans, configured from the same playground.
+
+It proves one specific claim: that the **agent** and the **place it runs** are both config,
+not code. You change a dropdown to swap Pi for Claude Code, or local for a Daytona cloud
+sandbox, and nothing above the seam changes.
+
+## Read in this order
+
+1. **[Architecture](architecture.md)**. How a request flows from the playground to the model
+   and back: the relay of programs, the two containers, and the vocabulary. Start here.
+2. **[Ports and adapters](ports-and-adapters.md)**. The ports that keep the relay swappable:
+   the backend, environment, and harness layers, where they live in the SDK, the wire
+   contract, and how the service picks a backend.
+3. **[Sessions](sessions.md)**. How a multi-turn conversation holds together today (cold
+   replay), and the two paths open to us tomorrow.
+4. **[The Pi adapter](adapters/pi.md)**. The default harness, which traces itself and takes
+   tools natively through a Pi extension.
+5. **[The Claude Code adapter](adapters/claude-code.md)**. The second harness, which proves
+   the swap and is the template for any MCP-capable agent.
+6. **[The Agenta harness](adapters/agenta.md)**. Pi with an opinion: forced skills, forced
+   tools, and a base AGENTS.md preamble the author's instructions are appended to.
+
+## What this PoC includes and defers
+
+It includes the agent workflow behind `/invoke`, two harnesses (Pi and Claude Code), two
+sandboxes (local and Daytona), backend-resolved tools that keep credentials server-side, and
+tracing that nests the agent's run under the caller's span.
+
+It defers the things a production rollout will need: a warm daemon and server-owned session
+storage (see [Sessions](sessions.md)), live streaming to the client over the HTTP edge, the
+multi-tenant filesystem jail for a shared daemon, and registering the agent as a first-class
+backend workflow type with its own builtin URI. Each is called out where it belongs.
+
+The first two of those, streaming and server-owned sessions, have a proposed design:
+[Streaming and sessions](streaming-and-sessions.md) for the rationale and trade-offs, and
+the [Agent protocol RFC](agent-protocol-rfc.md) for the normative spec of the endpoints and
+the wire format. They add a new `POST /messages` endpoint (Vercel-AI-format SSE stream, an
+optional `session_id`, and `UIMessage` inputs) plus a `load-session` endpoint, sitting next
+to the existing `/invoke`, which is unchanged.
+
+## The `scratch/` folder
+
+`scratch/` holds the raw working material from the build: the original work-package folders
+(WP-1 through WP-8), the port redesign notes, the research write-ups, and the proof-of-concept
+spikes. The pages above supersede it. It stays for history and for the running POC code, and
+it is not meant to be read as the design.

docs/design/agent-workflows/adapters/agenta.md

@@ -0,0 +1,64 @@
+# The Agenta harness
+
+`AgentaHarness` is Pi with an opinion. It runs on the same engine as the [Pi
+adapter](pi.md) and produces a Pi-shaped config, so it inherits everything Pi does (native
+tools, the system-prompt layers, tracing). What it adds is a fixed set of Agenta-shipped
+extras that the agent author cannot turn off:
+
+- **Forced tools** — always unioned into the agent's resolved tools. At minimum `read`
+  (Pi only renders the skills section when `read` is enabled) and `bash` (so skills can run
+  their helper scripts).
+- **Forced skills** — Agenta-shipped Pi skills loaded on every run.
+- **A base AGENTS.md preamble** — the author's `instructions` are appended after it.
+- **A base persona** — forced onto Pi's `append_system`, with any author-supplied
+  `append_system` appended after it.
+
+Read the [architecture](../architecture.md), [ports and adapters](../ports-and-adapters.md),
+and [Pi adapter](pi.md) pages first. This page assumes them.
+
+## Where the forced bits live
+
+The forced *policy* lives in the SDK harness layer, in one editable module:
+`sdks/python/agenta/sdk/agents/adapters/agenta_builtins.py` (`AGENTA_PREAMBLE`,
+`AGENTA_FORCED_APPEND_SYSTEM`, `AGENTA_FORCED_TOOLS`, `AGENTA_FORCED_SKILLS`). `AgentaHarness`
+(`adapters/harnesses.py`) reads them in `_to_harness_config` and layers them onto the neutral
+`SessionConfig`, exactly where `PiHarness` and `ClaudeHarness` do their own translation.
+
+The forced skill *files* live with the runner that runs Pi, under
+`services/agent/skills/<name>/` (each a directory with a `SKILL.md`). Skills are real files on
+disk because they reference relative scripts and assets, so they cannot ride the wire as
+text. The contract between the two halves is the skill **name**: `AGENTA_FORCED_SKILLS` lists
+names, and each must match a committed directory under the runner's skills root.
+
+## How a skill reaches the model
+
+1. `AgentaHarness._to_harness_config` puts the forced skill names on the `skills` field of
+   the `/run` request (`AgentaAgentConfig.wire_tools`).
+2. The in-process Pi engine (`engines/pi.ts`) resolves each name against its bundled
+   `skills/` root (override with `AGENTA_AGENT_SKILLS_DIR`) and passes the directories to Pi's
+   `DefaultResourceLoader` as `additionalSkillPaths`, with `noSkills: true` so only the
+   bundled skills load (the run stays hermetic, like `noContextFiles`).
+3. Pi loads them, and because the forced `read` tool is enabled, surfaces them in the system
+   prompt. The model reads a skill's `SKILL.md` on demand (progressive disclosure).
+
+## Two prompt layers, kept distinct
+
+This follows Pi's own split (see `PiAgentConfig`): the **persona** ("who the agent is")
+belongs in `append_system`, and **project conventions** belong in `AGENTS.md`. So the Agenta
+persona is a forced `append_system`, while the Agenta base preamble plus the author's
+instructions are the `AGENTS.md`. An author's own `system` / `append_system` (via
+`AgentConfig.harness_options["pi"]`) still apply, layered after the forced persona.
+
+## Selecting it
+
+`agenta` is a harness option alongside `pi` and `claude` (the playground dropdown, the
+`harness` field). It runs on the in-process Pi backend (`InProcessPiBackend` now lists
+`HarnessType.AGENTA` as supported), so `select_backend` keeps `agenta` on the local Pi path.
+
+## Deferred
+
+Only the in-process Pi (local) path is wired. The ACP/rivet path (and therefore the Daytona
+sandbox) does not yet deliver the forced skills — it would teach `runRivet` to read the
+`skills` field and lay the bundled skill directories into the sandbox via the existing
+bundled-file provisioning. Until then, `agenta` with a non-local sandbox raises
+`UnsupportedHarnessError` rather than silently running without its skills.

docs/design/agent-workflows/adapters/claude-code.md

@@ -0,0 +1,95 @@
+# The Claude Code adapter
+
+Claude Code is the second harness. It proves the central claim of this PoC: that swapping
+the agent is one config value. Where the [Pi adapter](pi.md) does much of its work inside Pi
+through an extension, Claude does its work through standard ACP. That makes Claude the
+template for any MCP-capable harness rivet can drive.
+
+Read the [architecture](../architecture.md) and [ports and adapters](../ports-and-adapters.md)
+pages first.
+
+## Running Claude
+
+The daemon resolves the harness id `claude` to the `claude-agent-acp` adapter, which starts
+the `claude` CLI. One operational detail is worth calling out, because it caused a real bug.
+The daemon does not ship the `claude` CLI. It downloads it over HTTPS the first time a run
+asks for Claude. The sidecar image is a slim Node image with no root certificates, so that
+HTTPS download failed until we added `ca-certificates` to the image. With the certs in
+place, the download verifies and Claude runs.
+
+Auth is config, like everything else. Claude authenticates with `ANTHROPIC_API_KEY` from the
+project vault when present, or with an OAuth token (`CLAUDE_CODE_OAUTH_TOKEN`) otherwise. The
+runner turns the common failures into one clear line, so a user sees "add the project's
+Anthropic key" rather than a stack trace.
+
+## Tools over MCP
+
+Claude advertises the `mcpTools` capability, so the runner delivers tools to Claude the
+standard ACP way, over MCP. This is the branch that the [capability probe](../ports-and-adapters.md)
+chooses: deliver over MCP when the harness reports `mcpTools`, not when the harness name is
+something in particular.
+
+The mechanism is a small stdio MCP server (`tools/mcp-server.ts`) that the daemon launches
+and attaches to the session. Its tool bodies POST back to Agenta's `/tools/call` with the
+same WP-7 envelope the Pi path uses. The resolved specs and the callback endpoint reach the
+MCP server through its environment, so nothing tool-specific is written to a file the agent
+can read. The safety property is identical to Pi's: the provider key and the connection auth
+stay server-side, and the agent only ever asks Agenta to run a named tool.
+
+## Permissions
+
+Claude gates tool use behind a permission prompt. In an Agenta run there is no human at the
+keyboard to answer it, so the runner answers for it. By default it auto-approves, because the
+tools are backend-resolved and trusted. The per-run permission policy (or an env override)
+can flip this to deny, which rejects tool use instead. This is handled on
+`session.onPermissionRequest`, a hook Pi does not need because Pi does not gate tools this
+way.
+
+## Tracing from the event stream
+
+Claude does not self-instrument the way Pi does, because we do not load an Agenta extension
+into Claude. So the runner builds the trace itself, from the ACP event stream. It subscribes
+to the session's `session/update` notifications and turns them into the same span tree Pi
+produces:
+
+```
+invoke_agent            (AGENT)
+  turn 0                (CHAIN)
+    chat <model>        (LLM)
+    execute_tool <name> (TOOL)   one per ACP tool_call
+```
+
+This is the general path. Any harness rivet drives that does not bring its own
+instrumentation gets traced this way. Pi is the exception that traces itself; Claude is the
+rule.
+
+## Usage and output
+
+Claude reports usage in two places, so the runner reads both. The per-call input and output
+token split rides on the ACP `PromptResponse`, and the cost rides on the `usage_update`
+event. The runner combines them into the run total, which then rolls onto the workflow span
+the same way Pi's writeback total does.
+
+Output needs one small piece of care. Claude streams text deltas and also periodically
+streams a full cumulative snapshot of the message so far. If the runner naively appended
+everything, the answer would double. The runner detects a snapshot (a chunk that is a
+superset of what it already has) and replaces rather than appends, so the final text is
+correct whether a chunk is a delta or a snapshot.
+
+## Models
+
+Claude ignores a model id meant for another provider. Ask it for `gpt-5.5` and it keeps its
+own default. The runner handles this honestly: when the harness does not accept the requested
+model, the chat span is labelled `chat` rather than falsely claiming a model the run did not
+use.
+
+## What Claude demonstrates
+
+Claude is the proof that the seam works. Adding it took a `ClaudeHarness` (which holds its
+Pi-versus-Claude config mapping) and no change to the workflow handler above the ports; the
+same `RivetBackend` drives it. It also exercises the capability-driven branches the design is
+built on: tools over MCP because it reports `mcpTools`, a permission answer because it gates
+tools, and event-stream tracing because it does not self-instrument. A future harness that
+rivet can drive would reuse this exact path. A future harness that rivet cannot drive would
+instead get its own backend beside `RivetBackend` and `InProcessPiBackend`, behind the same
+`/run` contract.