diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md new file mode 100644 index 0000000000..598718b622 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -0,0 +1,55 @@ +# Agent Overview redesign + +**Status:** Planning (design workspace only — no implementation) +**Date:** 2026-07-21 + +Redesign the agent Overview page so it shows an agent's *work* instead of the +prompt-management views it inherited (deployments, variants, evaluations). The page is +organized around three questions — **does it need me?**, **what has it been doing?**, **is +it healthy?** — and handles the empty/no-data state, including a never-run agent, as a +first-class state, not an afterthought. This workspace names the views and data (each +grounded in a verified backend source) and slices the implementation; it does not prescribe +the visual design. + +## Decisions + +- Replace prompt-era Overview views with agent-native ones, gated by workflow kind. +- Organize around three questions; lead with work (outcomes + produced artifacts), not + config. +- Keep the charts, but drive them from the analytics gateway (`POST /analytics/query`) with + explicit specs — the current dashboard uses ~5% of the endpoint. Relabel Requests → Runs; + upgrade Latency from avg-only to percentiles. The endpoint is root-only (run-level), so + child-span breakdowns (tools, per-cause failures) stay on per-run trace reads. See + `research.md` §5–6. +- Surface resource usage as a first-class group — context usage (how full the model's + window gets), token consumption, cache savings, cost per run — mapped to three personas + (owner, builder, budget owner). Agent cost/token totals depend on a backend attribution fix + (`research.md` §6); until it lands these views degrade gracefully, never showing a false zero. +- Reuse the shipped context-budget primitive (PR #5402 + #5434) for context usage — + occupancy measure; denominator from `contextWindowForModel` (`@agenta/entities/workflow`, + sourced from the model catalog), not a hardcoded map. +- Phase 1 composes existing data only (tracing, mounts, session interactions, triggers) and + reuses `/analytics/query`; no new backend endpoints (one ingest-side attribution fix is a + Phase 2 dependency for the Cost/Token views). +- Artifacts load lazily per row; file-less runs degrade to message output. +- Three distinct zero states; a never-run agent gets plain guidance to the Playground (the + Overview reports work, it doesn't run the agent), not zeroed charts. +- Name views/data here; leave layout/visuals to design. + +## Deliverables + +- [context.md](context.md) — problem, scope, non-goals, product language, success + criteria. +- [research.md](research.md) — current wiring and the verified backend data sources, with + `file:line`. +- [design.md](design.md) — the view catalog, empty-state behavior, and never-run guidance. +- [plan.md](plan.md) — sliced implementation (Slice 0–5) with exit checks. +- [status.md](status.md) — living source of truth: locked decisions + open questions. + +## Intended outcome + +Opening an agent's Overview, a user immediately sees whether it needs them, what it has +produced (plain-language outcomes and downloadable artifacts), and whether it is healthy — +with drill-down to traces, tools, and token detail available but never forced. A +brand-new agent sees plain guidance to where it can be tried (the Playground) and a preview +of what will appear, instead of a wall of zeros. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md new file mode 100644 index 0000000000..b11c7d4820 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md @@ -0,0 +1,74 @@ +# Context — Agent Overview redesign + +## Problem + +The agent Overview page is inherited wholesale from the prompt-management era. Today it +renders `DeploymentOverview`, `VariantsOverview`, `ObservabilityOverview`, and +`LatestEvaluationRunsTable` +(`web/oss/src/pages/w/[workspace_id]/p/[project_id]/apps/[app_id]/overview/index.tsx`). + +For an agent — an autonomous worker that runs on a trigger, uses tools, produces outputs, +and sometimes needs a human — those views answer the wrong questions. Deployments (dev / +staging / prod), a prompt/variant list, and evaluation runs describe a prompt asset being +promoted through environments. They do not describe an agent's *work*. A person opening an +agent's Overview cannot currently see what it has produced, whether it needs them, or +whether it is healthy. + +There is also no considered empty state. A brand-new agent that has never run drops the +user onto the same prompt-era panels rendering zeros, with no path toward first value. + +## Scope + +- The agent Overview page: what it shows for an **active** agent that has run. +- The **empty / no-data** state for an agent that exists but has never produced activity. +- The **never-run empty state**: from "just created" to guidance on where to try the agent + (the Playground) and what the page will show once it has run. +- A named catalog of agent-workflow views/data the page can draw from, each grounded in a + real backend source (see `design.md` and `research.md`). + +## Out of scope for the first delivery + +- New backend endpoints or aggregation jobs. The first delivery composes what already + exists; anything needing new server-side aggregation is called out as Phase 2 in + `plan.md`, not built here. +- Cross-agent / workspace-level dashboards. This is the single-agent Overview only. +- Redesigning the playground, the trace/turn inspector, triggers management, or + connections management — Overview links out to those; it does not replace them. +- Prompt-management surfaces themselves. We are replacing which views the *agent* Overview + composes, not deleting the deployment/variant components (other workflow kinds still use + them via `RequireWorkflowKind`). +- Final visual/layout design. That is owned by design (Claude design / Figma). This + workspace names the views and data; it does not prescribe pixels. + +## Product language + +- **Run** — a single execution of the agent (one invocation / session). Replaces the + prompt-era "Request". +- **Outcome** — the human-readable result of a run: what the agent did, plus its state + (completed / waiting on a human / failed). +- **Artifact** — a file the agent produced during a run, stored in a session mount; + previewable and downloadable. +- **Trigger** — what started a run: a schedule, an inbound event (Slack / GitHub via + Composio), or a manual "Run now". +- **Needs-you item** — a human-in-the-loop interaction the agent is blocked on: an + approval, a question (elicitation), or a client-tool prompt. +- **Connection** — an external integration the agent authenticates to; may be healthy or + need reconnecting. + +## Success criteria + +1. Opening an active agent's Overview answers three questions above the fold: **does it + need me?**, **what has it been doing?**, **is it healthy?** +2. The page shows the agent's actual work — outcomes and produced artifacts — not its + configuration status. +3. A brand-new agent sees plain-language guidance to where it can be tried (the Playground) + and a preview of what will appear, never a wall of zeroed prompt-era panels. +4. Every view on the page maps to a named, verified backend data source (`design.md`). +5. Non-technical and technical users can both read the page: plain-language outcomes up + top, drill-down detail (traces, tokens, tool calls) available but not forced. +6. The page surfaces the agent's resource usage in terms each persona can act on — context + usage (how full the model's context window gets), token consumption, cache savings, and + cost per run — not just aggregate charts. Each of these ties to a named source in + `design.md` and maps to a persona in the "Who reads this" section. (Cost/token *totals* + for agents depend on a backend attribution fix — `research.md` §6, `plan.md` Slice 6; the + views ship degraded, never a false zero, until it lands.) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md new file mode 100644 index 0000000000..ed7590731b --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -0,0 +1,173 @@ +# Design — views, empty state, onboarding + +This names the **views/data** the Overview can show and how the three states behave. It +does not prescribe layout, components, or visuals — those are design's to decide (Claude +design / Figma). Every view cites its source from `research.md`. + +Concept reference (prior exploration artifact): +https://claude.ai/code/artifact/f2963882-117c-434d-8618-e70222c50683 + +## The three questions the page answers + +Order of priority, top to bottom: + +1. **Does it need me?** — needs-you items (blocked runs). +2. **What has it been doing?** — outcomes feed + produced artifacts. +3. **Is it healthy?** — status, success rate, cost/latency/token trends, tool reliability, + connection health. + +## View catalog (active agent) + +Each entry: **view** — one-line purpose — *source* (from `research.md`). + +### Status & health band +- **Agent status** — on / idle, is it currently running — *tracing (recent trace state)*. +- **Last run / next run** — when it last ran, when it runs next — *tracing (last trace) + + triggers (cron)*. +- **Success rate** — share of recent runs that completed — *tracing aggregate*. +- **Avg cost / run** — typical spend per run — *tracing aggregate (`gen_ai.usage.cost`)*. +- **Run now** — manual trigger — *triggers (manual)*. + +### Needs-you +- **Pending human items** — count + list of approvals / questions / client-tool prompts the + agent is blocked on, each linking to where it's answered — *session interactions, + `status = pending`*. + +### What your agent produced (outcomes feed) +Per recent run, one row that adapts to what the run produced: +- **Outcome summary** — plain-language "what it did" — *`ag.meta.final_result`*. +- **Run state** — completed / waiting on a human / failed — *trace status + interactions*. +- **Failure reason** — shown inline when failed — *`ag.exception.message`*. +- **Produced artifacts** — file chips: name, type (extension), size, preview, download — + *session mounts (`MountFile` + `MountCredentials`)*. +- **Message output** — text snippet, for agents that reply instead of writing files — + *`ag.data.outputs.completion`* (row degrades to this when no mount files). +- **Run meta** — trigger source · duration · cost · tools-used count — *tracing + + triggers*. + +### Metrics (charts) — from the analytics gateway +- **Runs** (was "Requests"), **Latency**, **Cost**, **Tokens** over time — *`POST + /analytics/query` with explicit `specs` (research.md §5–6), not the 5% the current charts + use.* The endpoint aggregates **root spans only** (one row per run), which is exactly + right for run-level charts. +- **Latency is the free upgrade**: request percentiles + histogram, not just the avg the + current chart shows (one 38 s run skews the mean). Show p50/p90/p95, keep avg as a + secondary stat. +- **Cost & Tokens are blocked on attribution** (research.md §6): agent runs don't populate + `costs.cumulative.total` (cost is derived only on LLM-type child spans; `gen_ai.usage.cost` + is unmapped) and only populate `tokens.cumulative.total` via a best-effort bridge. This is + why the live dashboard shows `-`. Depends on the backend fix (plan.md Slice 4 note + + Slice 6); until then, degrade to the run-level token count the agent reports, not a zero. + +### Reliability (drill-down, not forced) +- **Most-used tools** — which tools it calls, counts, pass/fail flag — *`ag.tool.name` / + `ag.meta.tool.call.result`, on **tool child spans** → **not** servable by the root-only + `/analytics/query`; read per-run via bounded trace reads, or add a backend roll-up + (research.md §3b/§6). Live composer equivalent (same style as the context-budget indicator, + no new fetch): tool message-parts (`part.type` `tool-*`/`dynamic-tool`, name via + `partToolName`, pass/fail via `part.state`).* +- **Where it's failing** — failures grouped by cause — *`ag.exception.type`; child-span data, + same root-only caveat as tools.* + +### Resource usage & cost (the "what is it consuming" group) +- **Context usage** — how full the model's context window gets per run (occupancy: run + tokens vs. the model's window), so a user can see runs approaching compaction/truncation — + *reuses the shipped primitive from PR #5402 + #5434. Occupancy per run = + `ag.metrics.unit.tokens.total` / `gen_ai.usage.total_tokens` from the trace store; the + denominator per run = `contextWindowForModel(capabilities, agentHarness, runModel)` + (`@agenta/entities/workflow`), sourced from the model catalog — not a hardcoded map. + Inputs (traced, `research.md` §1 / Q8): `capabilities` from the global harness-catalog + atom; `agentHarness` from agent config (`agent.harness.kind`), not the trace; `runModel` + from the run's LLM child span. Window `null` → show raw token count, no bar/percent.* Use the + **occupancy** measure (running sum was dropped in #5434). Match the composer's ambient + meter: fill bar + "Context N% used", amber `>= 75%`, red `>= 90%`. See `research.md` §1 for + the full reconciliation. +- **Token consumption** — breakdown of prompt / completion / reasoning / cached tokens per + run and over time — *`ag.metrics.unit.tokens.{prompt,completion,reasoning,cache_read,cache_creation,cached}`. + Run-level totals + distributions (mean/percentiles/histogram) come from the analytics + gateway (research.md §6) once agent tokens are attributed to the root; per-token-type + breakdown that isn't on the root reads per-run from the trace.* +- **Cache savings** — how much prompt caching is saving (cache-read share, cost avoided) — + *`ag.metrics.unit.tokens.cache_read` / `gen_ai.usage.cache_read.input_tokens`*. +- **Cost** — cost per run, avg per run, and trend; optionally cost per tool — + *`gen_ai.usage.cost`. **Attribution gap (research.md §6):** this attribute is dropped at + ingest (no semconv mapping) and cost is derived only on LLM-type child spans, so agent + runs currently have no root cost — the live dashboard shows `-`. Needs the backend fix + before this view has data.* +- **Model & provider** — which model/provider each run used (agents can switch models) — + *`ag.meta.model_name` / `ag.meta.provider`; on LLM child spans, so **not** the root-only + analytics endpoint — read per-run from the trace (same as context usage's `runModel`).* + +## Who reads this (persona → what they use) + +The catalog covers three readers without three pages — each view already maps to one of +them; the layout just orders by priority: + +- **Owner / operator** (often non-technical): *does it need me, is it working, what's it + costing?* → needs-you, outcomes feed, status band, success rate, cost. +- **Builder / developer**: *why did it fail, is it efficient, am I near the context limit?* + → context usage, token consumption, tool reliability, failures, trace drill-down. +- **Budget owner / stakeholder**: *what's the spend and is caching helping?* → cost trend, + cost per run, cache savings. + +Plain-language work stays on top for the operator; usage/efficiency detail sits lower and +behind clicks for the builder and budget reader. + +### Connections & triggers +- **Connections** — external integrations and whether any need reconnecting (Reconnect + action) — *`TriggerConnectionStatus` READY / NEEDS_AUTH / NEEDS_INPUT*. +- **Triggers** — active schedules / event subscriptions and next scheduled run — + *`TriggerSubscription` + cron*. + +## Empty / no-data state (agent exists, never produced activity) + +Not one panel. Distinguish and handle: +- **New agent, never run** — the guidance state (below): a frame of what the page tracks + + a pointer to the Playground, not zeroed charts, not a run action, not a template picker. +- **Per-section empty** — a section with no data yet shows an inviting empty state naming + what will appear there and how to make it appear, not a blank box or a zero. +- **First run failed** — treat as activity: show the failure and the reason, plus a "try + again / open trace" path. Do not show it as "no data". + +Rules: +- Never render prompt-era panels (deployments / variants / evaluations) for an agent, empty + or not. +- Never show a wall of zero charts as the primary content for a new agent. +- Every empty section states the one action that fills it. + +## Fresh-agent empty state (never run) — guidance, not a launcher + +The Overview is a **reporting** surface — it shows an agent's work; it isn't where you build +the agent or where you go to operate it. So a never-run agent gets a **guidance** state — not +a run action, not a template gallery. (We dropped both the earlier 3-step checklist and the +inline "Run now": the checklist led with configuration, and a launcher/picker makes the +Overview pretend to be a surface it isn't. Whether the *active* health band keeps a manual +"Run now" is a separate open question — see `status.md` Q5.) Structure only — wording is +design's: + +- **Frame + why it's empty.** State what this page tracks and that there is nothing yet + because the agent hasn't run. Plain, non-technical language. +- **One direction: the Playground.** A single pointer to the existing Playground (the + try/run surface) — the only affordance on the screen. No inline run, no steps. +- **Expectation-setter.** A faint preview of the sections that will appear (outcomes / + health / usage), so the empty page previews the filled one. +- **Config stays out of the empty state.** Connections/triggers are not shown here — a + connection needing auth surfaces as a needs-you item, and a trigger is added from the + triggers surface — never as an onboarding step. +- **Self-filling, nothing to dismiss.** Once a run exists (from the Playground or anywhere) + the active-agent views render normally. Empty-vs-active keys on the same first-trace signal + the rest of the page uses. + +Reuse the existing Playground route rather than inventing a launcher on Overview. Never gate +the page behind this state. + +## Cross-cutting principles + +- Lead with work, not configuration. Config status (connections, triggers) is supporting, + not the headline. +- Plain language on top, drill-down underneath. Non-technical readers get outcomes; + technical readers can open traces / token detail / tool calls. +- Calm and uncrowded. Prioritize the three questions; everything else is secondary or + behind a click. +- Reuse Agenta's existing design system and the already-built agent surfaces + (`ApprovalDock`, `TurnInspector`, agent-home onboarding). diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md new file mode 100644 index 0000000000..c47ff9473d --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -0,0 +1,142 @@ +# Plan — sliced implementation + +## Scope now + +- Frontend-only composition of data sources that **already exist** (tracing, mounts, + session interactions, triggers). No new backend endpoints in Phase 1. +- Behind agent-workflow-kind gating (`RequireWorkflowKind`) so prompt-era views are + untouched for other kinds. +- Feature-flag the new agent Overview so it can ship dark and be switched on per + environment. +- Anything needing new server-side aggregation is Phase 2 (flagged where it bites in Slice 4, + built in Slice 6), not built in Phase 1. +- **One backend dependency is surfaced, not a new endpoint:** agent cost/token + **attribution** at ingest (research.md §6). The existing `/analytics/query` endpoint is + reused as-is (it is the "gateway" — Slice 1/4); but agent runs don't populate the root + cost/token paths it reads, so the Cost/Token views degrade until that ingest fix lands + (Slice 4 note + Slice 6). This does not change "no new endpoints in Phase 1." + +Each slice leaves the tree working and testable. + +## Slice 0 — Pin the current page and gate + +1. Add an agent-kind branch in the Overview page entry so agent workflows render a new + `AgentOverview` container while all other kinds keep the current + Deployment/Variants/Observability/Evaluation composition. +2. Ship `AgentOverview` as a stub behind a feature flag (default off) that renders the + existing charts only, to prove the gate without regressing anyone. +- **Exit:** an agent workflow with the flag off shows today's page; with the flag on shows + the stub with charts. No other workflow kind changes. + +## Slice 1 — Status & health band + Runs charts + +1. Build the status/health band: agent status, last run, next run (cron), success rate, + avg cost, Run now — all from existing tracing aggregates + trigger schedule. +2. Drive the charts from the analytics gateway (`POST /analytics/query`) with **explicit + `specs`**, not the defaults (research.md §5–6). Relabel Requests → Runs. Add **latency + percentiles** (p50/p90/p95 + histogram) — a near-free upgrade over the current avg-only + chart, which one slow run skews. Cost/Tokens use the same endpoint but are gated on the + attribution fix (Slice 4 note); until then degrade to the reported token count, never a + zero. +- **Exit:** for an agent with runs, the band shows real last-run/next-run/success/cost and + the charts render (latency shows a distribution, not just avg); "Run now" triggers a run. + +## Slice 2 — Outcomes feed (with adaptive artifact/message rows) + +1. Fetch the N most recent runs (bounded `LIMIT N`, no fan-out) and render one outcome row + each: summary (`ag.meta.final_result`), state, failure reason, run meta. +2. Lazy-load produced artifacts per row on expand: list mount files, render file chips + (name / ext / size), wire preview + download through `MountCredentials`. +3. Degrade a row with no mount files to its message output + (`ag.data.outputs.completion`). +- **Exit:** recent runs show plain-language outcomes; a run that wrote files shows + previewable/downloadable chips; a text-only run shows its message; a failed run shows its + reason. No object-store LIST happens until a row is expanded. + +## Slice 3 — Needs-you, connections, triggers + +1. Needs-you: count + list of `pending` session interactions, each linking to where it's + answered (reuse `ApprovalDock` where possible). +2. Connections: list with health from `TriggerConnectionStatus`; Reconnect for NEEDS_AUTH. +3. Triggers: active schedules / subscriptions + next scheduled run. +- **Exit:** a blocked agent shows its pending items and they resolve from Overview; a + connection needing reauth shows Reconnect; schedules show next run. + +## Slice 4 — Reliability & resource-usage panels + +Two data paths here — keep them separate (research.md §6). `/analytics/query` aggregates +**root spans only** (`WHERE parent_id IS NULL`), so it serves run-level *numeric +distributions* but cannot see child-span categoricals. + +1. Reliability (**child-span data → per-run trace reads, not the gateway**): most-used tools + (counts + pass/fail from `ag.tool.name` / `ag.meta.tool.call.result`, on tool spans) and + failures grouped by cause (`ag.exception.type`). Read from bounded `LIMIT N` recent-run + traces; a backend roll-up onto the root (Slice 6) would let the gateway serve these later. +2. Resource usage: + - **Run-level numeric distributions via the gateway** (`/analytics/query` specs): + token totals and cost per run — mean/percentiles/histogram/trend — **once agent + cost/token attribution is fixed** (research.md §6; cost is dropped at ingest and + derived only on LLM-type spans, tokens ride a best-effort bridge). Until then this view + degrades to the reported token count and hides cost, never showing a false zero. + - **Cache savings** and **per-token-type breakdown / model & provider per run** are + child-span or reported-usage data → read per-run from the trace (same path as context + usage's `runModel`), not the gateway. +3. Context usage: reuse the shipped primitive (PR #5402 + #5434). Denominator per run = + `contextWindowForModel(capabilities, agentHarness, runModel)` — already exported from + `@agenta/entities/workflow`, sourced from the model catalog on the harness catalog (no map + to lift, no fork). Wire the three inputs from their real locations (traced, question #8): + `capabilities` from the global `harnessCapabilitiesAtomFamily("")`; `agentHarness` from the + agent config (`agent.harness.kind`) — NOT the trace; `runModel` from the run's LLM child + span (`ag.meta.request.model` / `ag.meta.response.model`), while the token total comes off + the workflow root span — so read root + LLM child together per run. Occupancy = latest + turn's total tokens vs. that window; degrade to a raw token count when the window is + `null`. Match the composer's ambient-meter styling (bar + "N% used", amber `>= 75%`, + red `>= 90%`). +- **Exit:** a user can see which tools an agent leans on and how reliable they are (from + per-run traces), how full the context window gets per run (occupancy, resolved from the + catalog via `contextWindowForModel`), and where tokens are going — with unknown-window runs + showing a token count rather than a broken percentage. **Cost is shown only where + attributed** (LLM-app runs today; agent runs after the Slice 6 ingest fix), never a false + zero. + +## Slice 5 — Empty states + never-run guidance + +1. Detect the three zero states (never-run / per-section-empty / first-run-failed) and + render the right one; never show zeroed prompt-era panels. +2. Never-run agent → the guidance state (structure): a frame of what the page tracks + why + it's empty, one pointer to the Playground (reuse the existing Playground route), and a + faint preview of the sections that will appear. No run action, no checklist, no template + gallery, nothing to dismiss. +3. Empty-vs-active keys on the first-trace signal (the same one the active views use); once a + run exists the active-agent views render. Keep connections/triggers out of the empty state. +- **Exit:** a brand-new agent sees guidance to the Playground plus a preview of what will + appear; once it has run, the same page shows its work; a first failed run shows the failure, + not "no data". + +## Slice 6 (Phase 2) — Backend attribution + materialized aggregates + +Two independent backend items. The first is a **correctness dependency** (agent cost/token +data), not just a performance backstop; the rest are performance/scale, done only if Phase 1 +render latency proves the per-request aggregation too costly. + +1. **Agent cost/token attribution** (unblocks the Cost/Token views, research.md §6). Pick + one: (a) map `gen_ai.usage.cost` in semconv and treat agent/workflow root spans as + cost-bearing (or bridge the harness's separate OTLP batch so its LLM spans roll onto the + root); or (b) accept the FE reading the agent's reported per-run usage from the trace and + aggregating client-side. Either makes the root cost/token paths the gateway reads + non-empty for agents. +2. **Roll child-span categoricals onto the root** (or expose a child-focused analytics + aggregation) so most-used-tools / failures-by-cause / model-mix can move from per-run + trace reads to the gateway — removes the reliability panels' fan-out. +3. Index mount file metadata in Postgres at write time (avoid object-store LIST for chips). +4. Materialize run aggregates (success rate, cost, tool pass/fail) at ingest; serve the band + + reliability panels from the materialized store, tolerating labelled staleness. +- **Exit:** agent Cost/Token views show real numbers; reliability/first-paint no longer + depend on live per-run fan-out; numbers are within the stated staleness budget. + +## Flag rollout + +Ship Slices 0–5 behind the flag, dogfood, then default on for the agent kind. Slice 6 is +mostly a performance backstop, **except item 1 (agent cost/token attribution)**, which gates +the Cost/Token views specifically — those degrade gracefully until it lands, so it does not +block the flag, only those two views' data. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md new file mode 100644 index 0000000000..d149d2d227 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md @@ -0,0 +1,322 @@ +# Research — what the code gives us today + +Everything below is verified against current code. `file:line` where a specific symbol is +cited. Anything not verified is listed as an open question, not a fact. + +## Current wiring (what Overview renders now) + +`web/oss/src/pages/w/[workspace_id]/p/[project_id]/apps/[app_id]/overview/index.tsx` +composes, via dynamic import: + +- `DeploymentOverview` — `web/oss/src/components/pages/overview/deployments/DeploymentOverview.tsx` + (dev / staging / prod environment cards). +- `VariantsOverview` — `web/oss/src/components/pages/overview/variants/VariantsOverview.tsx` + (prompt/variant list). +- `ObservabilityOverview` — `web/oss/src/components/pages/overview/observability/ObservabilityOverview.tsx` + (the Requests / Latency / Cost / Tokens charts). +- `LatestEvaluationRunsTable` — from `web/oss/src/components/EvaluationRunsTablePOC`. + +The header block resolves the current workflow from `currentWorkflowAtom` +(`@/oss/state/workflow`) and works for both apps and evaluators. The four content views are +prompt-asset-shaped: environments, variants, evaluations. Only `ObservabilityOverview` +(the charts) is arguably agent-relevant, and even it is labelled in prompt terms +("Requests"). + +## Data sources available for agent-native views + +### 1. Runs / outcomes — tracing + +Agent execution emits OTel-style traces. DTOs in `api/oss/src/core/tracing/dtos.py`: +`SimpleTrace(Link, Lifecycle)` with `SimpleTraceKind` = {ADHOC, EVAL, PLAY} and +`SimpleTraceChannel` = {OTLP, WEB, SDK, API}; `Invocation(SimpleTrace)`. Spans carry +semantic-convention attributes (`ag.*`, `gen_ai.*`) verified present in the SDK and API: + +- `ag.meta.final_result` — the run's result summary ("what it did"). **Verified present.** +- `ag.meta.agent_name`, `ag.meta.model_name`, `ag.meta.provider`. +- `ag.tool.name`, `ag.meta.tool.name`, `ag.meta.tool.call.{result,arguments,id}` — tool + calls, for "which tools it used" and pass/fail. +- `ag.data.outputs.completion` — text output (for message-output agents). +- `ag.data.multimedia.image.url` — image output. +- `ag.metrics.unit.tokens.{total,prompt,completion,reasoning,cache_read,cache_creation,cached}` + — token breakdown incl. cache savings. +- `ag.metrics.acc.duration.total` — run duration. **Verified present.** +- `ag.exception.{type,message,stacktrace,escaped}` — failure reason inline. +- `gen_ai.usage.{cost,input_tokens,output_tokens,total_tokens,cache_read.input_tokens}`. + +Per-run cost, duration, tools-used, tokens, outcome, and failure reason are all derivable +from a single trace. The existing charts already aggregate over traces. + +**Context usage** (how full the model's context window gets) — prior art exists and is now +settled across a stacked pair. Reuse it; do not reinvent: + +- **PR #5402** (`fe-feat/session-context-info`, Arda) introduced the composer's + context-budget indicator. +- **PR #5434** (`feat/extend-arda's-session-context-work`, Ashraf, stacked on #5402) + supersedes the parts below and is the current source of truth. + +Files: +- `web/oss/src/components/AgentChatSlice/assets/contextBudget.ts` — `computeContextBudget` + (occupancy only). +- `web/oss/src/components/AgentChatSlice/components/ContextBudgetIndicator.tsx` — the ambient + meter (`maxTokens: number | null` prop). +- `contextWindowForModel` — exported from `@agenta/entities/workflow`. + +What #5434 establishes (and **corrects the earlier finding in this research**): + +- **The context window IS available to the frontend — from the model catalog, not a + hardcoded map.** The SDK's `model_catalog.py` ships each model's `context_window` on the + **harness catalog** the FE already fetches — `GET /workflows/catalog/harnesses/`, a + **global, project-independent** query atom (`harnessCatalogQueryAtom` in + `web/packages/agenta-entities/src/workflow/state/inspectMeta.ts`, disk-seeded to + localStorage, background-revalidated), exposed via `harnessCapabilitiesAtomFamily("")`. + (NOT the `/inspect` response — inspect carries no behavior-changing `meta`; the file's own + header comment says so.) #5434 deleted the hardcoded `MODEL_CONTEXT_WINDOWS` map + + substring resolver and replaced them with an exact-id lookup: + `contextWindowForModel(capabilities, harness, modelId)` → + `capabilities[harness]?.model_catalog?.find(e => e.id === modelId)?.context_window ?? null` + (`@agenta/entities/workflow`). No new endpoint, every model in the picker covered + automatically, exact match instead of a guess. (This overrides the prior note that "the + backend stores only cost"; the *registry* does, but the *harness-capabilities doc* carries + `context_window`. No litellm follow-up needed.) +- **`harness` is now a required input.** `useAgentModelKeyStatus` returns `harness` (from + `agent.harness.kind`, e.g. `pi_core` / `claude`); the lookup is per-harness + per-model. +- **`context_window: null` still happens** for some catalog entries (e.g. the Claude + `default` alias) → degrade to a raw token count, no bar/percent. The fix is a data change + in `model_catalog.py`, not frontend. +- **Occupancy is the locked measure.** #5434 dropped the `Σ` running-sum from both the data + layer and the UI (it double-counted resent history and never dropped). Occupancy = the + latest turn's total tokens = how full the window is now; drops after a compaction. +- **UI is an ambient meter**, not a raw dual readout: a slim fill bar + "Context N% used", + escalating neutral → amber (`>= 0.75`) → red ("Context almost full", `>= 0.9`); exact + counts (`407k / 1M tokens`) in the tooltip. +- **Token source (FE):** per-turn usage is stamped on assistant messages + (`message.metadata.usage`, read via `getMessageUsage` in + `web/oss/src/components/AgentChatSlice/assets/trace.ts`) — no new fetch, works for live and + reloaded sessions. + +Implication for Overview: the *concept*, the *denominator selector* (`contextWindowForModel`), +and the *occupancy measure* are all shared with the composer — only the *data path differs*. +The composer reads a single live session's `messages`. Overview aggregates across many +historical runs, so it reads per-run token totals from the trace store and, for each run, +resolves the denominator with `contextWindowForModel(capabilities, harness, model)`. + +Two inputs to that call are **not co-located on a run** (confirmed for open question #8): + +- **`capabilities`** — the global harness-catalog atom above. Readable anywhere with + `useAtomValue(harnessCapabilitiesAtomFamily(""))`; no `AgentChatSlice` coupling. ✅ +- **`model`** — sits on the run's **LLM child spans** (`ag.meta.request.model` / + `ag.meta.response.model`, mapped in `api/oss/src/apis/fastapi/otlp/opentelemetry/semconv.py`), + while the token total (`gen_ai.usage.total_tokens`) is rolled onto the **workflow root** + span (`sdks/python/agenta/sdk/agents/tracing.py:record_usage`). So pairing (model, tokens) + per run means reading the root **and** its LLM child — a per-run detail read, not just the + root aggregate. A run can hold multiple LLM spans; occupancy = the latest turn's total. +- **`harness`** — **not emitted on any span.** It is an agent-config property + (`agent.harness.kind`, e.g. `pi_core` / `claude`), stable per agent. Take it from the + agent, exactly as the composer does via `useAgentModelKeyStatus`; do not expect it on the + trace. If the agent's harness changed since an old run, or the run's model is absent from + the current harness's catalog, `contextWindowForModel` returns `null` → the raw-token + degrade already designed for. + +Occupancy per run = that run's latest-turn total tokens vs. its model's window. This connects +to the known token-overhead behavior (a bare turn can cost ~15K tokens from advertised tool +schemas); context usage makes that visible on Overview. + +### 2. Produced artifacts — session mounts + +`api/oss/src/core/mounts/dtos.py` and `.../sessions/mounts/dtos.py`: + +- `Mount(Identifier, Slug, Header, Lifecycle)` — project-scoped, optional `session_id`; a + session mount is a mount viewed through a session (`SessionMount`). +- `MountFile { path: str, size: int = 0, is_folder: bool = False, mtime: Optional[int] }` + — `mtime` is object-store LastModified epoch ms + (`api/oss/src/core/mounts/dtos.py:58-65`). `MountFileList` enumerates them. +- `MountFileContent { path, content }` — preview a file. +- `MountCredentials` — short-lived, prefix-scoped signed creds; the master key never + leaves the API, scoped to `///*`, expires in minutes. This + is how the UI previews/downloads a file without proxying bytes through the API. + +So "files the agent produced" (name, type via extension, size, mtime, preview, download) +is fully supported. Not every agent writes files — message-output agents use §1's +`ag.data.outputs.completion` instead. The row must adapt (file chips vs. message snippet). + +### 3. Needs-you items — session interactions + +`api/oss/src/core/sessions/interactions/dtos.py:10-21`: + +- `SessionInteractionKind` = {`user_approval`, `user_input`, `client_tool`}. +- `SessionInteractionStatus` = {`pending`, `responded`, `resolved`, `cancelled`}. + `pending` = "awaiting a reaction" — this is the count that drives a "needs you" badge. + +`user_approval` = approve/deny gate; `user_input` = the agent asked a question; +`client_tool` = a tool the client must run. A pending interaction means the agent is +blocked on the human. + +### 3b. Tool usage — same live layer as the context budget (no new fetch) + +Tool usage is derivable from the same live `messages` array that PR #5402's +`ContextBudgetIndicator` reads — a "Tools used" readout can mirror the context-budget one +client-side, and the Overview "Most-used tools" panel has a clean live-session equivalent. +Tool calls are message *parts*: + +- A part is a tool call when `part.type.startsWith("tool-")` or `part.type === "dynamic-tool"` + (`web/oss/src/components/AgentChatSlice/AgentConversation.tsx:193-194`; also + `state/expandState.ts:39`, `components/ApprovalDock.tsx:26`). +- Tool name: `partToolName(part)` (`assets/toolDisplay.ts:79` — `dynamic-tool` carries it on + `toolName`, typed parts are `type.replace(/^tool-/, "")`); human label via + `resolveToolDisplay`. +- Outcome per call: `part.state` — `output-available` = success, `output-error` / + `output-denied` = failed (`SETTLED` set, `components/ToolActivity.tsx:35`); also + `input-available` (running), `approval-requested` / `approval-responded` (HITL). +- Each part also carries `toolCallId`, `input`, `output`, `errorText`. `ToolActivity.tsx` + already renders individual calls. + +So which-tools / call-count / pass-fail are all computable from the live session with no new +backend work — the composer parallel to context budget. **Overview** aggregates across +historical runs instead, from the trace store (`ag.tool.name` / `ag.meta.tool.call.result`). +Note (see §6): `ag.tool.name` lives on **tool child spans**, so this is **not** servable by +the root-only `/analytics/query` aggregation — Overview reads it via per-run trace reads +(bounded `LIMIT N`), or it needs a backend roll-up onto the root. Same concept as context +usage: two data paths (live messages vs. per-run trace reads), not the aggregate endpoint. + +### 4. Triggers and schedules + +`api/oss/src/core/triggers/dtos.py`: + +- `TriggerProviderKind` = {COMPOSIO} (`:41`); `TriggerAuthScheme` = {OAUTH, API_KEY} + (`:45`). +- `ConnectionStatus` values READY / NEEDS_AUTH / NEEDS_INPUT (`:117-119`); + `TriggerConnectionStatus(ConnectionStatus)` (`:187`). This drives "connection healthy vs. + needs reconnect". +- `TriggerSubscription` (`:265`) — a subscription (e.g. Composio webhook for Slack / + GitHub). Schedules are cron (5-field, UTC), validated via croniter and matched via + `croniter.match`. + +So "trigger source" (schedule / inbound event / manual), "next scheduled run" (from cron), +and "connection needs reconnect" (from `ConnectionStatus`) are all available. + +### 5. Aggregate metrics — the analytics endpoint (the gateway) + +`ObservabilityOverview` → `AnalyticsDashboard` → `useObservabilityDashboard` → +`fetchGenerationsDashboardData` → **`POST /analytics/query`** (Fern `querySpansAnalytics`, +backend `operation_id="query_analytics"`, +`api/oss/src/apis/fastapi/tracing/router.py:118`). This one endpoint is far more capable +than the four charts it currently feeds — see §6. The current wiring uses ~5% of it. +Reusable, but do **not** treat it as "relabel and ship": the charts today throw away most of +what the endpoint returns, and its default specs don't attribute agent cost/tokens (§6). + +### 6. `POST /analytics/query` — full capability, root-only scope, and the cost/token gap + +Verified against `api/oss/src/core/tracing/{service,dtos}.py`, +`api/oss/src/dbs/postgres/tracing/{dao,utils}.py`, +`web/oss/src/services/tracing/lib/helpers.ts`, and the ingest roll-up in +`api/oss/src/core/tracing/utils/trees.py`. This section supersedes the earlier "reusable +as-is, only labels change" framing. + +**Request surface.** The endpoint takes a `TracingQuery` (arbitrary `filtering` + +time-`windowing`/`interval`) and a list of **`MetricSpec`** (`type` + attribute `path`, plus +histogram opts `bins`/`vmin`/`vmax`/`edge`; `dtos.py:249`). The frontend currently sends +**no `specs`** (`web/packages/agenta-entities/src/trace/api/api.ts`), so the backend applies +`DEFAULT_ANALYTICS_SPECS` — only 6 paths (`service.py:90`): duration, errors, cost.total, +tokens.total, trace-type, span-type. Then `analyticsToGeneration` (`helpers.ts:106`) reads +only `sum`/`count` off 5 of them → the 4 charts. + +**What it computes per time bucket, per requested spec** (`dao.py:426-484`, builders in +`utils.py`): +- **Numeric** (`numeric/continuous`, `numeric/discrete`): `count`, `sum`, `mean`, `min`, + `max`, `range`, **percentiles** (p25/p50/p75/p90/p95/p99/…) with IQR/CQV/PSC spreads, and + **histograms** (bin counts + shares). +- **Categorical / binary / string** (`categorical/single|multiple`, `binary`, `string`): + **value-count frequencies** — each distinct value + its count + uniqueness. + +**Root-only scope — the load-bearing constraint.** The analytics base CTE hardcodes +`WHERE parent_id IS NULL` (`utils.py:1067`); the `focus` field does not lift this. **Every +metric is aggregated over root/workflow spans only — one row per run.** Consequences: +- **Servable directly** (one call, better specs): run count, success/failure split, and + **latency distributions** (p50/p90/p95/p99 + histogram — a real upgrade over today's + avg-only, which one 38 s run skews). Plus **run-level cost & token distributions** *once + attribution is fixed*, and any **categorical stamped on the root** (kind, channel, origin, + environment, and the run-level final exception if it lands on the root). +- **NOT servable by this endpoint**: **per-tool** usage/pass-fail (`ag.tool.name` lives on + tool child spans), **per-LLM model/provider** mix (`ag.meta.*.model` on LLM child spans), + and **failures grouped by the failing child's** `ag.exception.type`. These are child-span + data; the root-only aggregation can't see them. They need per-run trace reads (as §1/§3b + already describe) or a backend change (roll those categoricals onto the root, or expose a + child-focused aggregation). **This corrects §3b's implication that "Most-used tools" is a + trace-store aggregate the dashboard path already serves — it is not, via this endpoint.** + +**The cost/token attribution gap (why the live dashboard shows Cost/Tokens `-` for agents).** +The default specs read `costs.cumulative.total` / `tokens.cumulative.total` on the root, +populated by the ingest roll-up (`calculate_and_propagate_metrics`), which is LLM-app-shaped: +- **Cost** is *derived*, never read from the agent. `gen_ai.usage.cost` has **no semconv + mapping** (`api/oss/src/apis/fastapi/otlp/opentelemetry/semconv.py`) and is dropped at + ingest. `calculate_costs` (`trees.py:547`) computes cost only on spans whose type ∈ + `{embedding, query, completion, chat, rerank}` (`trees.py:538`) using a model-keyed price + table. An agent's root is a `workflow`/`agent` span → not cost-bearing → **cost is always + empty for agent runs.** +- **Tokens** ride a best-effort bridge: the SDK stamps `gen_ai.usage.total_tokens` on the + `/invoke` workflow span via `record_usage` (`sdks/python/agenta/sdk/agents/tracing.py:213`) + → `unit.tokens.total` → `tokens.incremental.total` → rolled to `tokens.cumulative.total` + (`cumulate_tokens`). But `record_usage` early-returns when `usage.total` is falsy, and the + harness's real LLM spans arrive in a **separate OTLP batch** that cannot roll onto the root + — so tokens land only if that bridge fires with a non-zero total. + +Implication: the Resource-usage cost/token numbers cannot come from the default dashboard +path for agents. Either (i) fix attribution at ingest, or (ii) read the agent's reported +usage per run from the trace and aggregate on the FE. Tracked as a backend dependency in +`plan.md` (Slice 4 note + Slice 6). + +**The ingest roll-up path, traced (the fix is smaller than "make spans cost-bearing").** +The `unit.* → incremental.* → cumulative.*` pipe already exists and already handles costs: +`span_data_builders.py:171-179` renames every metric key unconditionally — +`unit.costs. → costs.incremental.`, `unit.tokens. → tokens.incremental.`, +`acc.costs. → costs.cumulative.`, `acc.tokens. → tokens.cumulative.`. So the **cost pipe is +built**; it is only starved at the semconv step. `record_usage` already stamps +`gen_ai.usage.cost` on the `/invoke` root (`tracing.py:234`), but semconv has no row for it, +so `unit.costs.total` is never created and the promotion never fires. **Minimal fix: one +semconv row** — `("gen_ai.usage.cost", "ag.metrics.unit.costs.total")`. Then: +`gen_ai.usage.cost` → (semconv) `unit.costs.total` → (`span_data_builders:175`) +`costs.incremental.total` → (`cumulate_costs`, and `calculate_costs` skips the workflow root +so it isn't clobbered — root ∉ `TYPES_WITH_COSTS`) `costs.cumulative.total` → read by the +default analytics spec. Cost appears for agents. + +Caveats on that one-liner: +- **Covers cost-reporting harnesses only.** `record_usage` sets cost only when `usage.cost` + is present and early-returns when `usage.total` is falsy. A harness that reports + tokens-but-no-cost still needs the price-table derivation — the one-liner is necessary, + not complete. +- **Latent double-count.** It is safe *because* the harness's real LLM spans arrive in a + separate OTLP batch (absent from the root's tree). A future "bridge the harness batch" fix + that puts price-derived child costs in the same tree would make the root count both. The + two fixes collide — pick one. +- **Ingest-time only.** No retroactive effect; cost shows for runs ingested after the change. + +**Cost vs. tokens are different failures.** Cost is broken *at the mapping* (definitive, above). +Tokens have a *complete* pipe already (`gen_ai.usage.total_tokens` → `unit.tokens.total` → +`tokens.incremental.total` → `tokens.cumulative.total`). So if the dashboard shows tokens as +`-`, the cause is `record_usage` not firing for that agent (or the harness-batch separation), +**not** a semconv gap — confirming it needs a live root-span attribute dump (running stack). + +## FE surfaces that already consume this data (reuse, don't rebuild) + +From the platform map: `AgentChatSlice` (`AgentChatTransport`, `ClientTools`, +`ApprovalDock`, `TurnInspector`), `pages/agents`, agent-home +(templates/composer/onboarding, `useCreateAgent` ephemeral→commit). Data layer is Fern +`@agenta/api-client` via `@agenta/sdk`, with entities modules `workflow` / `gatewayTool` / +`gatewayTrigger`. Overview should link into these, and can reuse `ApprovalDock` / +`TurnInspector` for drill-down. + +## Seams where this can break (the plan pins each) + +1. **Aggregation cost.** Outcomes + artifacts + needs-you span traces, mounts, and + interactions across many sessions. Naïvely fanning out per row (object-store LIST per + run) is expensive. Cheap: already-computed chart aggregates and bounded `LIMIT N` + recent-run lookups. Expensive: windowed GROUP BYs and per-row object-store LISTs. +2. **Not every agent produces files.** The outcomes row must degrade to message output + when a run wrote no mount files, so the feed is never empty for text agents. +3. **Empty vs. new vs. failing-first-run** are three different zero states and must not + collapse into one "no data" panel. +4. **Workflow-kind gating.** Overview is shared across workflow kinds via + `RequireWorkflowKind`. Agent views must render only for the agent kind; deployment / + variant views stay for the kinds that use them. +5. **Staleness tolerance.** `mtime` / aggregates can lag. Acceptable for Overview if we + label freshness; not acceptable to block first paint on a slow fan-out. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md new file mode 100644 index 0000000000..1c361bb8ce --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -0,0 +1,130 @@ +# Status + +**Last updated:** 2026-07-21 (analytics endpoint audited: gateway capability, root-only +scope, and the agent cost/token attribution gap — research.md §5–6; cost roll-up path traced +— the fix is a one-line semconv row, Q9. Never-run empty state settled: guidance to the +Playground — not a run action, checklist, or template gallery.) + +## Current stage + +Planning workspace drafted. Problem, research, view catalog, and a sliced plan are written +and grounded in current code. No implementation has started. This is a design-workspace +only; no code, no branch, no commit. + +## Locked decisions + +- The agent Overview replaces the prompt-era composition (deployments / variants / + evaluations) with agent-native views, gated by workflow kind — prompt-era views stay for + the kinds that use them. +- The page is organized around three questions, in priority order: **does it need me?**, + **what has it been doing?**, **is it healthy?** +- Lead with the agent's *work* (outcomes + produced artifacts), not its configuration + status. +- Keep the charts, but drive them from the **analytics gateway** (`POST /analytics/query`) + with **explicit `specs`** — the current dashboard uses ~5% of the endpoint. Relabel + Requests → Runs and upgrade Latency from avg-only to **percentiles + histogram** (research.md + §5–6). +- **The analytics endpoint is the gateway for run-level aggregates, and it is root-only** + (`WHERE parent_id IS NULL`, one row per run). It serves numeric distributions + root-stamped + categoricals. It does **not** serve child-span breakdowns (per-tool usage/pass-fail, per-LLM + model mix, failures by the failing child's cause) — those stay on per-run trace reads until a + backend roll-up (Slice 6). +- **Agent cost/token attribution is a real backend gap** (why the live dashboard shows Cost + and Tokens as `-`): `gen_ai.usage.cost` is unmapped at ingest and cost is derived only on + LLM-type child spans; agent tokens ride a best-effort `record_usage` bridge. The + Cost/Token views **degrade gracefully** (reported token count, cost hidden — never a false + zero) until the Slice 6 fix lands. Full detail: research.md §6. +- Resource usage is a first-class group, not just charts: context usage (occupancy: run + tokens vs. the model's window), token consumption, cache savings, cost per run, and + model/provider. Context usage degrades to a raw token count when the window size is + unknown; cost/token totals depend on the attribution fix above. +- Context usage **reuses the shipped primitive** across a stacked pair: PR #5402 (Arda, + `fe-feat/session-context-info`) + **PR #5434** (Ashraf, + `feat/extend-arda's-session-context-work`, stacked on #5402). #5434 is the source of truth. + Overview uses the **occupancy** measure only (running sum was dropped in #5434). The + context-window denominator comes from the **model catalog on the harness-capabilities doc**, + resolved by `contextWindowForModel(capabilities, harness, modelId)` exported from + `@agenta/entities/workflow` — no hardcoded map, no litellm follow-up. `context_window: null` + entries (e.g. Claude `default` alias) degrade to a raw token count; filling them is a + `model_catalog.py` data change, not FE. Match the composer's ambient-meter UI (bar + + "Context N% used", amber `>= 75%`, red `>= 90%`). +- The view catalog serves three personas (owner/operator, builder, budget owner) from one + page via priority ordering, not three separate pages. +- Phase 1 composes only data sources that already exist (tracing, mounts, session + interactions, triggers) and reuses the existing `/analytics/query` endpoint. No new backend + *endpoints* in Phase 1 — but one backend *ingest fix* (agent cost/token attribution) is a + surfaced dependency for the Cost/Token views, tracked in Slice 6 with graceful degradation + in the meantime. +- Produced artifacts load lazily per outcome row (no object-store fan-out on first paint); + rows with no files degrade to message output. +- Three distinct zero states (never-run / per-section-empty / first-run-failed), not one + "no data" panel. New agents get plain-language guidance, not zeroed charts. +- **Never-run Overview = guidance, not a launcher.** The Overview reports work; it does not + run the agent or pick its template. A never-run agent gets a guidance state — a frame of + what the page tracks + a single pointer to the Playground + a preview of what will appear — + not a run action, checklist, or template gallery. It self-fills on the first trace; nothing + to dismiss. (Supersedes the earlier run-CTA and 3-step-checklist ideas.) +- This workspace names views/data only; layout, components, and visuals are design's + (Claude design / Figma). +- No commits during this planning session. + +## Open implementation questions + +1. **Feature-flag mechanism** — reuse an existing FE flag pattern, or a new one keyed to + workflow kind? *Recommend:* reuse the existing flag pattern already used for agent + surfaces; key the default-on by workflow kind. +2. **Outcomes feed depth (N)** — how many recent runs on first paint before "view all"? + *Recommend:* 10, bounded `LIMIT N`, matching the concept artifact. +3. **Aggregation freshness budget** — is live per-request aggregation acceptable for + Phase 1, and what staleness is tolerable before Slice 5 becomes required? *Recommend:* + ship Phase 1 live; set a first-paint latency threshold that triggers Slice 6. +4. **Needs-you resolution in place** — resolve pending interactions directly on Overview + (embed `ApprovalDock`) or link out to the session? *Recommend:* link out in Phase 1, + embed later if it proves worth it. +5. **Manual "Run now" on an *active* agent** — the never-run empty state points to the + Playground (Overview reports, it doesn't launch). Does the active health band still keep a + manual "Run now" (re-run a working agent on demand), or defer to the Playground/triggers + for consistency? *Recommend:* keep a manual re-run on the active band, but if it stays, + reuse the existing manual-trigger path — Overview only invokes it, never owns running. + Resolve before Slice 1 finalizes the band. +9. **Agent cost/token attribution — fix at ingest or on the FE?** (research.md §6, Slice 6 + item 1.) **Traced — the cost fix is one line.** The `unit.* → incremental.* → cumulative.*` + pipe already exists and already handles costs (`span_data_builders.py:171-179`), and + `record_usage` already stamps `gen_ai.usage.cost` on the agent root — the only gap is a + missing semconv row. Option (a): add `("gen_ai.usage.cost", "ag.metrics.unit.costs.total")` + to semconv; cost then flows to `costs.cumulative.total` (the default spec) with no + price-table or span-type work. Covers cost-reporting harnesses; token-only harnesses still + need price-table derivation. Watch the double-count risk if the harness batch is later + bridged, and note it's ingest-time only (not retroactive). Option (b): FE reads reported + per-run usage from the trace and aggregates client-side. *Recommend:* (a) — now known to be + cheap and the correct source of truth; (b) is a stopgap. **Tokens are a separate issue** — + their pipe is complete, so a `-` means `record_usage` isn't firing, not a mapping gap; + diagnose with a live root-span attribute dump. Ship Cost/Token views degraded until (a) + lands. Needs a backend owner; the mapping itself is a one-line PR. +6. ~~Context-window map ownership~~ — **RESOLVED by #5434.** No shared map to own: the + denominator is `contextWindowForModel` in `@agenta/entities/workflow`, sourced from the + model catalog on the harness-capabilities doc. Overview imports the selector. Slice 4 + depends on #5434 being merged (it is stacked on #5402). +7. ~~Composer vs. Overview measure~~ — **RESOLVED by #5434.** Occupancy is the settled + measure everywhere; the running sum was dropped. No divergence to reconcile. +8. ~~Overview data path for context usage~~ — **RESOLVED (traced).** (a) The + harness-capabilities doc is a **global, project-independent** atom + (`harnessCatalogQueryAtom` ← `GET /workflows/catalog/harnesses/`); Overview reads it with + `useAtomValue(harnessCapabilitiesAtomFamily(""))`, no `AgentChatSlice` coupling. (b) Per + run, `model` is on the **LLM child span** (`ag.meta.request.model` / + `ag.meta.response.model`) while the token total is on the **workflow root** span — pair the + root with its LLM child. (c) `harness` is **not on the trace**; it's agent config + (`agent.harness.kind`), stable per agent — read it from the agent like the composer does. + Call `contextWindowForModel(globalCapabilities, agentHarness, runModel)`; unknowns degrade + to raw token count. Implementation note captured in `plan.md` Slice 4. + +## Next action + +Context-usage feasibility (Q6–8) is fully traced and settled. The analytics endpoint is now +audited (research.md §5–6): it is the gateway for run-level charts/distributions, it is +root-only, and agent cost/token attribution is a backend gap (new Q9). Remaining open items +are the product/rollout calls (Q1–5) plus the backend-owned attribution decision (Q9). Share +the workspace and Q1–5 + Q9 for agreement. On sign-off, hand the view catalog (`design.md`) +to design for the Figma pass, and start Slice 0 on an implementation branch (separate from +this planning session). Slice 4 depends on #5434 (the context-usage primitive) being merged; +its Cost/Token data depends on Q9/Slice 6.