From 122e71d44691a2fe989c7822de23b0566e6680e3 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 21 Jul 2026 13:23:13 +0600 Subject: [PATCH 1/5] Add design documentation for agent Overview page redesign - Introduced README.md outlining the redesign goals and structure. - Created context.md detailing the current problems and scope of the Overview page. - Added design.md specifying views, empty states, and onboarding processes. - Developed plan.md for the phased implementation strategy. - Included research.md summarizing current data sources and their usage. - Established status.md to track the planning stage and decisions made. --- .../agent-overview-redesign/README.md | 42 ++++++ .../agent-overview-redesign/context.md | 68 ++++++++++ .../agent-overview-redesign/design.md | 109 ++++++++++++++++ .../projects/agent-overview-redesign/plan.md | 79 +++++++++++ .../agent-overview-redesign/research.md | 123 ++++++++++++++++++ .../agent-overview-redesign/status.md | 54 ++++++++ 6 files changed, 475 insertions(+) create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/README.md create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/context.md create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/design.md create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/plan.md create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/research.md create mode 100644 docs/design/agent-workflows/projects/agent-overview-redesign/status.md diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md new file mode 100644 index 0000000000..758d6eea9f --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -0,0 +1,42 @@ +# Agent Overview redesign + +**Status:** Planning (design workspace only — no implementation) +**Date:** 2026-07-21 + +Redesign the agent Overview page so it shows an agent's *work* instead of the +prompt-management views it inherited (deployments, variants, evaluations). The page is +organized around three questions — **does it need me?**, **what has it been doing?**, **is +it healthy?** — and handles the empty/no-data state and fresh-agent onboarding as +first-class states, not afterthoughts. This workspace names the views and data (each +grounded in a verified backend source) and slices the implementation; it does not prescribe +the visual design. + +## Decisions + +- Replace prompt-era Overview views with agent-native ones, gated by workflow kind. +- Organize around three questions; lead with work (outcomes + produced artifacts), not + config. +- Keep the charts; relabel Requests → Runs. +- Phase 1 composes existing data only (tracing, mounts, session interactions, triggers); + no new backend endpoints. +- Artifacts load lazily per row; file-less runs degrade to message output. +- Three distinct zero states; new agents get onboarding, not zeroed charts. +- Name views/data here; leave layout/visuals to design. + +## Deliverables + +- [context.md](context.md) — problem, scope, non-goals, product language, success + criteria. +- [research.md](research.md) — current wiring and the verified backend data sources, with + `file:line`. +- [design.md](design.md) — the view catalog, empty-state behavior, and onboarding flow. +- [plan.md](plan.md) — sliced implementation (Slice 0–5) with exit checks. +- [status.md](status.md) — living source of truth: locked decisions + open questions. + +## Intended outcome + +Opening an agent's Overview, a user immediately sees whether it needs them, what it has +produced (plain-language outcomes and downloadable artifacts), and whether it is healthy — +with drill-down to traces, tools, and token detail available but never forced. A +brand-new agent sees a short onboarding path to its first successful run and first trigger +instead of a wall of zeros. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md new file mode 100644 index 0000000000..b86be05940 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md @@ -0,0 +1,68 @@ +# Context — Agent Overview redesign + +## Problem + +The agent Overview page is inherited wholesale from the prompt-management era. Today it +renders `DeploymentOverview`, `VariantsOverview`, `ObservabilityOverview`, and +`LatestEvaluationRunsTable` +(`web/oss/src/pages/w/[workspace_id]/p/[project_id]/apps/[app_id]/overview/index.tsx`). + +For an agent — an autonomous worker that runs on a trigger, uses tools, produces outputs, +and sometimes needs a human — those views answer the wrong questions. Deployments (dev / +staging / prod), a prompt/variant list, and evaluation runs describe a prompt asset being +promoted through environments. They do not describe an agent's *work*. A person opening an +agent's Overview cannot currently see what it has produced, whether it needs them, or +whether it is healthy. + +There is also no considered empty state. A brand-new agent that has never run drops the +user onto the same prompt-era panels rendering zeros, with no path toward first value. + +## Scope + +- The agent Overview page: what it shows for an **active** agent that has run. +- The **empty / no-data** state for an agent that exists but has never produced activity. +- The **fresh-agent onboarding** path: from "just created" to first successful run and + first connected trigger. +- A named catalog of agent-workflow views/data the page can draw from, each grounded in a + real backend source (see `design.md` and `research.md`). + +## Out of scope for the first delivery + +- New backend endpoints or aggregation jobs. The first delivery composes what already + exists; anything needing new server-side aggregation is called out as Phase 2 in + `plan.md`, not built here. +- Cross-agent / workspace-level dashboards. This is the single-agent Overview only. +- Redesigning the playground, the trace/turn inspector, triggers management, or + connections management — Overview links out to those; it does not replace them. +- Prompt-management surfaces themselves. We are replacing which views the *agent* Overview + composes, not deleting the deployment/variant components (other workflow kinds still use + them via `RequireWorkflowKind`). +- Final visual/layout design. That is owned by design (Claude design / Figma). This + workspace names the views and data; it does not prescribe pixels. + +## Product language + +- **Run** — a single execution of the agent (one invocation / session). Replaces the + prompt-era "Request". +- **Outcome** — the human-readable result of a run: what the agent did, plus its state + (completed / waiting on a human / failed). +- **Artifact** — a file the agent produced during a run, stored in a session mount; + previewable and downloadable. +- **Trigger** — what started a run: a schedule, an inbound event (Slack / GitHub via + Composio), or a manual "Run now". +- **Needs-you item** — a human-in-the-loop interaction the agent is blocked on: an + approval, a question (elicitation), or a client-tool prompt. +- **Connection** — an external integration the agent authenticates to; may be healthy or + need reconnecting. + +## Success criteria + +1. Opening an active agent's Overview answers three questions above the fold: **does it + need me?**, **what has it been doing?**, **is it healthy?** +2. The page shows the agent's actual work — outcomes and produced artifacts — not its + configuration status. +3. A brand-new agent sees an onboarding experience with a clear first action, never a wall + of zeroed prompt-era panels. +4. Every view on the page maps to a named, verified backend data source (`design.md`). +5. Non-technical and technical users can both read the page: plain-language outcomes up + top, drill-down detail (traces, tokens, tool calls) available but not forced. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md new file mode 100644 index 0000000000..9de24423bd --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -0,0 +1,109 @@ +# Design — views, empty state, onboarding + +This names the **views/data** the Overview can show and how the three states behave. It +does not prescribe layout, components, or visuals — those are design's to decide (Claude +design / Figma). Every view cites its source from `research.md`. + +Concept reference (prior exploration artifact): +https://claude.ai/code/artifact/f2963882-117c-434d-8618-e70222c50683 + +## The three questions the page answers + +Order of priority, top to bottom: + +1. **Does it need me?** — needs-you items (blocked runs). +2. **What has it been doing?** — outcomes feed + produced artifacts. +3. **Is it healthy?** — status, success rate, cost/latency/token trends, tool reliability, + connection health. + +## View catalog (active agent) + +Each entry: **view** — one-line purpose — *source* (from `research.md`). + +### Status & health band +- **Agent status** — on / idle, is it currently running — *tracing (recent trace state)*. +- **Last run / next run** — when it last ran, when it runs next — *tracing (last trace) + + triggers (cron)*. +- **Success rate** — share of recent runs that completed — *tracing aggregate*. +- **Avg cost / run** — typical spend per run — *tracing aggregate (`gen_ai.usage.cost`)*. +- **Run now** — manual trigger — *triggers (manual)*. + +### Needs-you +- **Pending human items** — count + list of approvals / questions / client-tool prompts the + agent is blocked on, each linking to where it's answered — *session interactions, + `status = pending`*. + +### What your agent produced (outcomes feed) +Per recent run, one row that adapts to what the run produced: +- **Outcome summary** — plain-language "what it did" — *`ag.meta.final_result`*. +- **Run state** — completed / waiting on a human / failed — *trace status + interactions*. +- **Failure reason** — shown inline when failed — *`ag.exception.message`*. +- **Produced artifacts** — file chips: name, type (extension), size, preview, download — + *session mounts (`MountFile` + `MountCredentials`)*. +- **Message output** — text snippet, for agents that reply instead of writing files — + *`ag.data.outputs.completion`* (row degrades to this when no mount files). +- **Run meta** — trigger source · duration · cost · tools-used count — *tracing + + triggers*. + +### Metrics (keep the charts) +- **Runs** (was "Requests"), **Avg latency**, **Cost**, **Tokens** over time — + *`ObservabilityOverview` aggregates, relabelled*. + +### Reliability & efficiency (drill-down, not forced) +- **Most-used tools** — which tools it calls, counts, pass/fail flag — *`ag.tool.name` / + `ag.meta.tool.call.result`*. +- **Where it's failing** — failures grouped by cause — *`ag.exception.type`*. +- **Token & cache efficiency** — token breakdown and cache savings — *`ag.metrics.unit.tokens.*`*. + +### Connections & triggers +- **Connections** — external integrations and whether any need reconnecting (Reconnect + action) — *`TriggerConnectionStatus` READY / NEEDS_AUTH / NEEDS_INPUT*. +- **Triggers** — active schedules / event subscriptions and next scheduled run — + *`TriggerSubscription` + cron*. + +## Empty / no-data state (agent exists, never produced activity) + +Not one panel. Distinguish and handle: +- **New agent, never run** — onboarding (below), not zeroed charts. +- **Per-section empty** — a section with no data yet shows an inviting empty state naming + what will appear there and how to make it appear, not a blank box or a zero. +- **First run failed** — treat as activity: show the failure and the reason, plus a "try + again / open trace" path. Do not show it as "no data". + +Rules: +- Never render prompt-era panels (deployments / variants / evaluations) for an agent, empty + or not. +- Never show a wall of zero charts as the primary content for a new agent. +- Every empty section states the one action that fills it. + +## Fresh-agent onboarding (just created → first value) + +Replace the empty Overview body with a getting-started experience. The milestones map to +real capabilities and each is individually detectable, so the checklist self-completes: + +1. **Run it once** — a "Run now" that produces the first outcome. *Done when the first + trace exists.* +2. **Connect an integration** — if the agent's tools/triggers need one. *Done when a + `TriggerConnectionStatus` is READY.* +3. **Add a trigger** — a schedule or event subscription so it runs on its own. *Done when a + `TriggerSubscription` exists.* + +Behavior: +- Show progress (e.g. 1 of 3), mark steps done as the underlying data appears, and dismiss + the onboarding once the agent has real activity — after which the active-agent views take + over. +- Reuse the existing agent-home onboarding / composer patterns and `useCreateAgent` flow + rather than inventing a parallel onboarding. +- Keep it to the few steps that lead to first value; do not gate the page behind it (a user + can always ignore the checklist and use "Run now"). + +## Cross-cutting principles + +- Lead with work, not configuration. Config status (connections, triggers) is supporting, + not the headline. +- Plain language on top, drill-down underneath. Non-technical readers get outcomes; + technical readers can open traces / token detail / tool calls. +- Calm and uncrowded. Prioritize the three questions; everything else is secondary or + behind a click. +- Reuse Agenta's existing design system and the already-built agent surfaces + (`ApprovalDock`, `TurnInspector`, agent-home onboarding). diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md new file mode 100644 index 0000000000..803a441ef5 --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -0,0 +1,79 @@ +# Plan — sliced implementation + +## Scope now + +- Frontend-only composition of data sources that **already exist** (tracing, mounts, + session interactions, triggers). No new backend endpoints in Phase 1. +- Behind agent-workflow-kind gating (`RequireWorkflowKind`) so prompt-era views are + untouched for other kinds. +- Feature-flag the new agent Overview so it can ship dark and be switched on per + environment. +- Anything needing new server-side aggregation is Phase 2 (called out in Slice 4), not + built in Phase 1. + +Each slice leaves the tree working and testable. + +## Slice 0 — Pin the current page and gate + +1. Add an agent-kind branch in the Overview page entry so agent workflows render a new + `AgentOverview` container while all other kinds keep the current + Deployment/Variants/Observability/Evaluation composition. +2. Ship `AgentOverview` as a stub behind a feature flag (default off) that renders the + existing charts only, to prove the gate without regressing anyone. +- **Exit:** an agent workflow with the flag off shows today's page; with the flag on shows + the stub with charts. No other workflow kind changes. + +## Slice 1 — Status & health band + Runs charts + +1. Build the status/health band: agent status, last run, next run (cron), success rate, + avg cost, Run now — all from existing tracing aggregates + trigger schedule. +2. Relabel the reused charts (Requests → Runs) and mount them under the band. +- **Exit:** for an agent with runs, the band shows real last-run/next-run/success/cost and + the charts render; "Run now" triggers a run. + +## Slice 2 — Outcomes feed (with adaptive artifact/message rows) + +1. Fetch the N most recent runs (bounded `LIMIT N`, no fan-out) and render one outcome row + each: summary (`ag.meta.final_result`), state, failure reason, run meta. +2. Lazy-load produced artifacts per row on expand: list mount files, render file chips + (name / ext / size), wire preview + download through `MountCredentials`. +3. Degrade a row with no mount files to its message output + (`ag.data.outputs.completion`). +- **Exit:** recent runs show plain-language outcomes; a run that wrote files shows + previewable/downloadable chips; a text-only run shows its message; a failed run shows its + reason. No object-store LIST happens until a row is expanded. + +## Slice 3 — Needs-you, connections, triggers + +1. Needs-you: count + list of `pending` session interactions, each linking to where it's + answered (reuse `ApprovalDock` where possible). +2. Connections: list with health from `TriggerConnectionStatus`; Reconnect for NEEDS_AUTH. +3. Triggers: active schedules / subscriptions + next scheduled run. +- **Exit:** a blocked agent shows its pending items and they resolve from Overview; a + connection needing reauth shows Reconnect; schedules show next run. + +## Slice 4 — Empty state + onboarding + +1. Detect the three zero states (never-run / per-section-empty / first-run-failed) and + render the right one; never show zeroed prompt-era panels. +2. Build the getting-started checklist (Run it once / Connect / Add a trigger) with + self-completing steps driven by first-trace / READY-connection / subscription-exists + detection. Reuse agent-home onboarding patterns + `useCreateAgent`. +3. Dismiss onboarding once the agent has real activity; hand off to active-agent views. +- **Exit:** a brand-new agent sees onboarding with a working first action and steps that + tick off as data appears; a first failed run shows the failure, not "no data". + +## Slice 5 (Phase 2, optional) — Materialized aggregates + +Only if Phase 1 render latency proves the per-request aggregation too costly at scale. +1. Index mount file metadata in Postgres at write time (avoid object-store LIST for chips). +2. Materialize run aggregates (success rate, cost, tool pass/fail) at ingest. +3. Serve the band + reliability panels from the materialized store; tolerate labelled + staleness. +- **Exit:** Overview first paint no longer depends on live fan-out; numbers are within the + stated staleness budget. + +## Flag rollout + +Ship Slices 0–4 behind the flag, dogfood, then default on for the agent kind. Slice 5 is a +performance backstop, not a prerequisite. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md new file mode 100644 index 0000000000..d433c6702e --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md @@ -0,0 +1,123 @@ +# Research — what the code gives us today + +Everything below is verified against current code. `file:line` where a specific symbol is +cited. Anything not verified is listed as an open question, not a fact. + +## Current wiring (what Overview renders now) + +`web/oss/src/pages/w/[workspace_id]/p/[project_id]/apps/[app_id]/overview/index.tsx` +composes, via dynamic import: + +- `DeploymentOverview` — `web/oss/src/components/pages/overview/deployments/DeploymentOverview.tsx` + (dev / staging / prod environment cards). +- `VariantsOverview` — `web/oss/src/components/pages/overview/variants/VariantsOverview.tsx` + (prompt/variant list). +- `ObservabilityOverview` — `web/oss/src/components/pages/overview/observability/ObservabilityOverview.tsx` + (the Requests / Latency / Cost / Tokens charts). +- `LatestEvaluationRunsTable` — from `web/oss/src/components/EvaluationRunsTablePOC`. + +The header block resolves the current workflow from `currentWorkflowAtom` +(`@/oss/state/workflow`) and works for both apps and evaluators. The four content views are +prompt-asset-shaped: environments, variants, evaluations. Only `ObservabilityOverview` +(the charts) is arguably agent-relevant, and even it is labelled in prompt terms +("Requests"). + +## Data sources available for agent-native views + +### 1. Runs / outcomes — tracing + +Agent execution emits OTel-style traces. DTOs in `api/oss/src/core/tracing/dtos.py`: +`SimpleTrace(Link, Lifecycle)` with `SimpleTraceKind` = {ADHOC, EVAL, PLAY} and +`SimpleTraceChannel` = {OTLP, WEB, SDK, API}; `Invocation(SimpleTrace)`. Spans carry +semantic-convention attributes (`ag.*`, `gen_ai.*`) verified present in the SDK and API: + +- `ag.meta.final_result` — the run's result summary ("what it did"). **Verified present.** +- `ag.meta.agent_name`, `ag.meta.model_name`, `ag.meta.provider`. +- `ag.tool.name`, `ag.meta.tool.name`, `ag.meta.tool.call.{result,arguments,id}` — tool + calls, for "which tools it used" and pass/fail. +- `ag.data.outputs.completion` — text output (for message-output agents). +- `ag.data.multimedia.image.url` — image output. +- `ag.metrics.unit.tokens.{total,prompt,completion,reasoning,cache_read,cache_creation,cached}` + — token breakdown incl. cache savings. +- `ag.metrics.acc.duration.total` — run duration. **Verified present.** +- `ag.exception.{type,message,stacktrace,escaped}` — failure reason inline. +- `gen_ai.usage.{cost,input_tokens,output_tokens,total_tokens,cache_read.input_tokens}`. + +Per-run cost, duration, tools-used, tokens, outcome, and failure reason are all derivable +from a single trace. The existing charts already aggregate over traces. + +### 2. Produced artifacts — session mounts + +`api/oss/src/core/mounts/dtos.py` and `.../sessions/mounts/dtos.py`: + +- `Mount(Identifier, Slug, Header, Lifecycle)` — project-scoped, optional `session_id`; a + session mount is a mount viewed through a session (`SessionMount`). +- `MountFile { path: str, size: int = 0, is_folder: bool = False, mtime: Optional[int] }` + — `mtime` is object-store LastModified epoch ms + (`api/oss/src/core/mounts/dtos.py:58-65`). `MountFileList` enumerates them. +- `MountFileContent { path, content }` — preview a file. +- `MountCredentials` — short-lived, prefix-scoped signed creds; the master key never + leaves the API, scoped to `///*`, expires in minutes. This + is how the UI previews/downloads a file without proxying bytes through the API. + +So "files the agent produced" (name, type via extension, size, mtime, preview, download) +is fully supported. Not every agent writes files — message-output agents use §1's +`ag.data.outputs.completion` instead. The row must adapt (file chips vs. message snippet). + +### 3. Needs-you items — session interactions + +`api/oss/src/core/sessions/interactions/dtos.py:10-21`: + +- `SessionInteractionKind` = {`user_approval`, `user_input`, `client_tool`}. +- `SessionInteractionStatus` = {`pending`, `responded`, `resolved`, `cancelled`}. + `pending` = "awaiting a reaction" — this is the count that drives a "needs you" badge. + +`user_approval` = approve/deny gate; `user_input` = the agent asked a question; +`client_tool` = a tool the client must run. A pending interaction means the agent is +blocked on the human. + +### 4. Triggers and schedules + +`api/oss/src/core/triggers/dtos.py`: + +- `TriggerProviderKind` = {COMPOSIO} (`:41`); `TriggerAuthScheme` = {OAUTH, API_KEY} + (`:45`). +- `ConnectionStatus` values READY / NEEDS_AUTH / NEEDS_INPUT (`:117-119`); + `TriggerConnectionStatus(ConnectionStatus)` (`:187`). This drives "connection healthy vs. + needs reconnect". +- `TriggerSubscription` (`:265`) — a subscription (e.g. Composio webhook for Slack / + GitHub). Schedules are cron (5-field, UTC), validated via croniter and matched via + `croniter.match`. + +So "trigger source" (schedule / inbound event / manual), "next scheduled run" (from cron), +and "connection needs reconnect" (from `ConnectionStatus`) are all available. + +### 5. Aggregate metrics — existing charts + +`ObservabilityOverview` already computes run count, latency, cost, tokens over the trace +store. Reusable as-is; only the labels change (Requests → Runs). + +## FE surfaces that already consume this data (reuse, don't rebuild) + +From the platform map: `AgentChatSlice` (`AgentChatTransport`, `ClientTools`, +`ApprovalDock`, `TurnInspector`), `pages/agents`, agent-home +(templates/composer/onboarding, `useCreateAgent` ephemeral→commit). Data layer is Fern +`@agenta/api-client` via `@agenta/sdk`, with entities modules `workflow` / `gatewayTool` / +`gatewayTrigger`. Overview should link into these, and can reuse `ApprovalDock` / +`TurnInspector` for drill-down. + +## Seams where this can break (the plan pins each) + +1. **Aggregation cost.** Outcomes + artifacts + needs-you span traces, mounts, and + interactions across many sessions. Naïvely fanning out per row (object-store LIST per + run) is expensive. Cheap: already-computed chart aggregates and bounded `LIMIT N` + recent-run lookups. Expensive: windowed GROUP BYs and per-row object-store LISTs. +2. **Not every agent produces files.** The outcomes row must degrade to message output + when a run wrote no mount files, so the feed is never empty for text agents. +3. **Empty vs. new vs. failing-first-run** are three different zero states and must not + collapse into one "no data" panel. +4. **Workflow-kind gating.** Overview is shared across workflow kinds via + `RequireWorkflowKind`. Agent views must render only for the agent kind; deployment / + variant views stay for the kinds that use them. +5. **Staleness tolerance.** `mtime` / aggregates can lag. Acceptable for Overview if we + label freshness; not acceptable to block first paint on a slow fan-out. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md new file mode 100644 index 0000000000..b4d477611c --- /dev/null +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -0,0 +1,54 @@ +# Status + +**Last updated:** 2026-07-21 + +## Current stage + +Planning workspace drafted. Problem, research, view catalog, and a sliced plan are written +and grounded in current code. No implementation has started. This is a design-workspace +only; no code, no branch, no commit. + +## Locked decisions + +- The agent Overview replaces the prompt-era composition (deployments / variants / + evaluations) with agent-native views, gated by workflow kind — prompt-era views stay for + the kinds that use them. +- The page is organized around three questions, in priority order: **does it need me?**, + **what has it been doing?**, **is it healthy?** +- Lead with the agent's *work* (outcomes + produced artifacts), not its configuration + status. +- Keep the existing charts; relabel Requests → Runs. +- Phase 1 composes only data sources that already exist (tracing, mounts, session + interactions, triggers). No new backend endpoints in Phase 1. +- Produced artifacts load lazily per outcome row (no object-store fan-out on first paint); + rows with no files degrade to message output. +- Three distinct zero states (never-run / per-section-empty / first-run-failed), not one + "no data" panel. New agents get onboarding, not zeroed charts. +- Onboarding reuses agent-home onboarding patterns + `useCreateAgent`; steps self-complete + from real data and dismiss once the agent has activity. +- This workspace names views/data only; layout, components, and visuals are design's + (Claude design / Figma). +- No commits during this planning session. + +## Open implementation questions + +1. **Feature-flag mechanism** — reuse an existing FE flag pattern, or a new one keyed to + workflow kind? *Recommend:* reuse the existing flag pattern already used for agent + surfaces; key the default-on by workflow kind. +2. **Outcomes feed depth (N)** — how many recent runs on first paint before "view all"? + *Recommend:* 10, bounded `LIMIT N`, matching the concept artifact. +3. **Aggregation freshness budget** — is live per-request aggregation acceptable for + Phase 1, and what staleness is tolerable before Slice 5 becomes required? *Recommend:* + ship Phase 1 live; set a first-paint latency threshold that triggers Slice 5. +4. **Needs-you resolution in place** — resolve pending interactions directly on Overview + (embed `ApprovalDock`) or link out to the session? *Recommend:* link out in Phase 1, + embed later if it proves worth it. +5. **"Run now" ownership** — does Overview trigger a run itself or hand off to the existing + manual-run path? *Recommend:* reuse the existing manual-trigger path; Overview only + invokes it. + +## Next action + +Share the workspace and the five open questions for agreement. On sign-off, hand the view +catalog (`design.md`) to design for the Figma pass, and start Slice 0 on an implementation +branch (separate from this planning session). From 14fda068a69401da82e67b4670c928e3672f1c90 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 21 Jul 2026 15:20:54 +0600 Subject: [PATCH 2/5] Enhance agent Overview redesign documentation with resource usage metrics and context usage details --- .../agent-overview-redesign/README.md | 5 ++ .../agent-overview-redesign/context.md | 4 ++ .../agent-overview-redesign/design.md | 44 +++++++++++-- .../projects/agent-overview-redesign/plan.md | 24 ++++++- .../agent-overview-redesign/research.md | 64 +++++++++++++++++++ .../agent-overview-redesign/status.md | 24 ++++++- 6 files changed, 157 insertions(+), 8 deletions(-) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md index 758d6eea9f..f73abb5125 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -17,6 +17,11 @@ the visual design. - Organize around three questions; lead with work (outcomes + produced artifacts), not config. - Keep the charts; relabel Requests → Runs. +- Surface resource usage as a first-class group — context usage (how full the model's + window gets), token consumption, cache savings, cost per run — mapped to three personas + (owner, builder, budget owner). +- Reuse PR #5402's context-budget primitive (Arda) for context usage — occupancy measure, + shared `MODEL_CONTEXT_WINDOWS` map; don't fork it. - Phase 1 composes existing data only (tracing, mounts, session interactions, triggers); no new backend endpoints. - Artifacts load lazily per row; file-less runs degrade to message output. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md index b86be05940..75c7d73903 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md @@ -66,3 +66,7 @@ user onto the same prompt-era panels rendering zeros, with no path toward first 4. Every view on the page maps to a named, verified backend data source (`design.md`). 5. Non-technical and technical users can both read the page: plain-language outcomes up top, drill-down detail (traces, tokens, tool calls) available but not forced. +6. The page surfaces the agent's resource usage in terms each persona can act on — context + usage (how full the model's context window gets), token consumption, cache savings, and + cost per run — not just aggregate charts. Each of these ties to a named source in + `design.md` and maps to a persona in the "Who reads this" section. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md index 9de24423bd..ac045e9da7 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -49,11 +49,47 @@ Per recent run, one row that adapts to what the run produced: - **Runs** (was "Requests"), **Avg latency**, **Cost**, **Tokens** over time — *`ObservabilityOverview` aggregates, relabelled*. -### Reliability & efficiency (drill-down, not forced) -- **Most-used tools** — which tools it calls, counts, pass/fail flag — *`ag.tool.name` / - `ag.meta.tool.call.result`*. +### Reliability (drill-down, not forced) +- **Most-used tools** — which tools it calls, counts, pass/fail flag — *Overview: + `ag.tool.name` / `ag.meta.tool.call.result` from the trace store. Live composer + equivalent (same style as the context-budget indicator, no new fetch): tool message-parts + (`part.type` `tool-*`/`dynamic-tool`, name via `partToolName`, pass/fail via `part.state`); + see `research.md` §3b.* - **Where it's failing** — failures grouped by cause — *`ag.exception.type`*. -- **Token & cache efficiency** — token breakdown and cache savings — *`ag.metrics.unit.tokens.*`*. + +### Resource usage & cost (the "what is it consuming" group) +- **Context usage** — how full the model's context window gets per run (occupancy: run + tokens vs. the model's window), so a user can see runs approaching compaction/truncation — + *reuses PR #5402's primitive. Occupancy per run = `ag.metrics.unit.tokens.total` / + `gen_ai.usage.total_tokens` from the trace store, keyed by `ag.meta.model_name`, against + the window from `resolveModelContextWindow` / `MODEL_CONTEXT_WINDOWS` + (`web/oss/src/components/AgentChatSlice/assets/contextBudget.ts`). Window unknown → show + raw token count without the percentage.* Use the **occupancy** measure (current fullness / + compaction predictor), not the cumulative running sum. See `research.md` for the full + reconciliation and the litellm follow-up for the denominator. +- **Token consumption** — breakdown of prompt / completion / reasoning / cached tokens per + run and over time — *`ag.metrics.unit.tokens.{prompt,completion,reasoning,cache_read,cache_creation,cached}`*. +- **Cache savings** — how much prompt caching is saving (cache-read share, cost avoided) — + *`ag.metrics.unit.tokens.cache_read` / `gen_ai.usage.cache_read.input_tokens`*. +- **Cost** — cost per run, avg per run, and trend; optionally cost per tool — + *`gen_ai.usage.cost`*. +- **Model & provider** — which model/provider each run used (agents can switch models) — + *`ag.meta.model_name` / `ag.meta.provider`*. + +## Who reads this (persona → what they use) + +The catalog covers three readers without three pages — each view already maps to one of +them; the layout just orders by priority: + +- **Owner / operator** (often non-technical): *does it need me, is it working, what's it + costing?* → needs-you, outcomes feed, status band, success rate, cost. +- **Builder / developer**: *why did it fail, is it efficient, am I near the context limit?* + → context usage, token consumption, tool reliability, failures, trace drill-down. +- **Budget owner / stakeholder**: *what's the spend and is caching helping?* → cost trend, + cost per run, cache savings. + +Plain-language work stays on top for the operator; usage/efficiency detail sits lower and +behind clicks for the builder and budget reader. ### Connections & triggers - **Connections** — external integrations and whether any need reconnecting (Reconnect diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md index 803a441ef5..112b83710a 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -52,7 +52,25 @@ Each slice leaves the tree working and testable. - **Exit:** a blocked agent shows its pending items and they resolve from Overview; a connection needing reauth shows Reconnect; schedules show next run. -## Slice 4 — Empty state + onboarding +## Slice 4 — Reliability & resource-usage panels + +1. Reliability: most-used tools (counts + pass/fail from `ag.tool.name` / + `ag.meta.tool.call.result`) and failures grouped by cause (`ag.exception.type`). +2. Resource usage: token consumption breakdown, cache savings, cost per run/trend, and + model/provider per run — all from trace token/cost attributes. +3. Context usage: reuse PR #5402's primitive + (`AgentChatSlice/assets/contextBudget.ts` — `resolveModelContextWindow` / + `MODEL_CONTEXT_WINDOWS`). Occupancy per run = run total tokens (trace store) vs. the + model's window, keyed by `ag.meta.model_name`; use the occupancy measure (not the + running sum); degrade to a raw token count when the window is `null`. First, lift the + shared bits (the window map + resolver) out of `AgentChatSlice` to a shared location so + composer and Overview read one source; coordinate with #5402 rather than forking the map. +- **Exit:** a user can see which tools an agent leans on and how reliable they are, how full + the context window gets per run (occupancy), and where token/cost is going — with + unknown-window runs showing a token count rather than a broken percentage, and no + duplicated context-window map. + +## Slice 5 — Empty state + onboarding 1. Detect the three zero states (never-run / per-section-empty / first-run-failed) and render the right one; never show zeroed prompt-era panels. @@ -63,7 +81,7 @@ Each slice leaves the tree working and testable. - **Exit:** a brand-new agent sees onboarding with a working first action and steps that tick off as data appears; a first failed run shows the failure, not "no data". -## Slice 5 (Phase 2, optional) — Materialized aggregates +## Slice 6 (Phase 2, optional) — Materialized aggregates Only if Phase 1 render latency proves the per-request aggregation too costly at scale. 1. Index mount file metadata in Postgres at write time (avoid object-store LIST for chips). @@ -75,5 +93,5 @@ Only if Phase 1 render latency proves the per-request aggregation too costly at ## Flag rollout -Ship Slices 0–4 behind the flag, dogfood, then default on for the agent kind. Slice 5 is a +Ship Slices 0–5 behind the flag, dogfood, then default on for the agent kind. Slice 6 is a performance backstop, not a prerequisite. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md index d433c6702e..dc5e98742b 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md @@ -46,6 +46,45 @@ semantic-convention attributes (`ag.*`, `gen_ai.*`) verified present in the SDK Per-run cost, duration, tools-used, tokens, outcome, and failure reason are all derivable from a single trace. The existing charts already aggregate over traces. +**Context usage** (how full the model's context window gets) — prior art already exists: +**PR #5402** (`fe-feat/session-context-info`, Arda, open/draft) adds a "Context X / max +(Y%)" budget indicator to the agent chat composer. Read its helpers before building the +Overview version — reuse, don't reinvent: + +- `web/oss/src/components/AgentChatSlice/assets/contextBudget.ts` — `computeContextBudget`, + `resolveModelContextWindow`, and the `MODEL_CONTEXT_WINDOWS` map. +- `web/oss/src/components/AgentChatSlice/components/ContextBudgetIndicator.tsx` — the + presentational strip. + +What that PR establishes (and corrects in this research): + +- **The backend does NOT expose per-model context windows to the frontend.** The model + registry the FE reads stores only *cost*. The `context_window` field I found earlier in + `sdks/python/agenta/sdk/agents/model_catalog.py:74` is SDK-side curated data, not surfaced + through the API — so it is not a usable denominator for a FE view today. The denominator + instead comes from the hardcoded FE `MODEL_CONTEXT_WINDOWS` map (substring/longest-match on + the model id), returning `null` for unknown models → the view drops the "/ max (%)" and + shows raw tokens. Documented follow-up: source the window from litellm + `get_model_info().max_input_tokens` instead of the static map. +- **Two candidate measures**, shown side by side in v1 until one is picked on a live agent: + - **occupancy** — the *latest* turn's total tokens = how full the window is *now*; grows + with the conversation and **drops after a compaction**, so it's the compaction predictor. + This is the measure that means "context usage". + - **running sum** — Σ of every turn's tokens; cumulative, double-counts resent history, + never drops. A usage meter, not an occupancy meter. +- **Token source (FE):** per-turn usage is stamped on assistant messages + (`message.metadata.usage`, read via `getMessageUsage` in + `web/oss/src/components/AgentChatSlice/assets/trace.ts`) — no new fetch, works for live and + reloaded sessions. + +Implication for Overview: the *concept* and the *window map* are shared with the composer, +but the *data path differs*. The composer reads a single live session's `messages`. Overview +aggregates across many historical runs, so it reads per-run token totals from the trace +store (`ag.metrics.unit.tokens.total` / `gen_ai.usage.total_tokens`) and applies the same +`resolveModelContextWindow` map. Occupancy per run = that run's total tokens vs. the model's +window. This connects to the known token-overhead behavior (a bare turn can cost ~15K tokens +from advertised tool schemas); context usage makes that visible on Overview. + ### 2. Produced artifacts — session mounts `api/oss/src/core/mounts/dtos.py` and `.../sessions/mounts/dtos.py`: @@ -76,6 +115,31 @@ is fully supported. Not every agent writes files — message-output agents use `client_tool` = a tool the client must run. A pending interaction means the agent is blocked on the human. +### 3b. Tool usage — same live layer as the context budget (no new fetch) + +Tool usage is derivable from the same live `messages` array that PR #5402's +`ContextBudgetIndicator` reads — a "Tools used" readout can mirror the context-budget one +client-side, and the Overview "Most-used tools" panel has a clean live-session equivalent. +Tool calls are message *parts*: + +- A part is a tool call when `part.type.startsWith("tool-")` or `part.type === "dynamic-tool"` + (`web/oss/src/components/AgentChatSlice/AgentConversation.tsx:193-194`; also + `state/expandState.ts:39`, `components/ApprovalDock.tsx:26`). +- Tool name: `partToolName(part)` (`assets/toolDisplay.ts:79` — `dynamic-tool` carries it on + `toolName`, typed parts are `type.replace(/^tool-/, "")`); human label via + `resolveToolDisplay`. +- Outcome per call: `part.state` — `output-available` = success, `output-error` / + `output-denied` = failed (`SETTLED` set, `components/ToolActivity.tsx:35`); also + `input-available` (running), `approval-requested` / `approval-responded` (HITL). +- Each part also carries `toolCallId`, `input`, `output`, `errorText`. `ToolActivity.tsx` + already renders individual calls. + +So which-tools / call-count / pass-fail are all computable from the live session with no new +backend work — the composer parallel to context budget. **Overview** aggregates across +historical runs instead, from the trace store (`ag.tool.name` / `ag.meta.tool.call.result`), +per §1. Same concept, two data paths (live messages vs. trace store) — same split as context +usage. + ### 4. Triggers and schedules `api/oss/src/core/triggers/dtos.py`: diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md index b4d477611c..ad17fcdf55 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -18,6 +18,19 @@ only; no code, no branch, no commit. - Lead with the agent's *work* (outcomes + produced artifacts), not its configuration status. - Keep the existing charts; relabel Requests → Runs. +- Resource usage is a first-class group, not just charts: context usage (occupancy: run + tokens vs. the model's window), token consumption, cache savings, cost per run, and + model/provider. Context usage degrades to a raw token count when the window size is + unknown. +- Context usage **reuses PR #5402** (Arda, `fe-feat/session-context-info`), which already + builds the composer's context-budget indicator. Overview uses the **occupancy** measure + (current fullness / compaction predictor), not the cumulative running sum. The + context-window denominator comes from that PR's FE `MODEL_CONTEXT_WINDOWS` map (the backend + registry stores only cost, not window sizes); do not fork the map — lift it to a shared + location and coordinate with #5402. Follow-up (owned by #5402): source the window from + litellm `get_model_info().max_input_tokens`. +- The view catalog serves three personas (owner/operator, builder, budget owner) from one + page via priority ordering, not three separate pages. - Phase 1 composes only data sources that already exist (tracing, mounts, session interactions, triggers). No new backend endpoints in Phase 1. - Produced artifacts load lazily per outcome row (no object-store fan-out on first paint); @@ -39,13 +52,22 @@ only; no code, no branch, no commit. *Recommend:* 10, bounded `LIMIT N`, matching the concept artifact. 3. **Aggregation freshness budget** — is live per-request aggregation acceptable for Phase 1, and what staleness is tolerable before Slice 5 becomes required? *Recommend:* - ship Phase 1 live; set a first-paint latency threshold that triggers Slice 5. + ship Phase 1 live; set a first-paint latency threshold that triggers Slice 6. 4. **Needs-you resolution in place** — resolve pending interactions directly on Overview (embed `ApprovalDock`) or link out to the session? *Recommend:* link out in Phase 1, embed later if it proves worth it. 5. **"Run now" ownership** — does Overview trigger a run itself or hand off to the existing manual-run path? *Recommend:* reuse the existing manual-trigger path; Overview only invokes it. +6. **Context-window map ownership** — where does the shared `MODEL_CONTEXT_WINDOWS` / + `resolveModelContextWindow` live once both composer and Overview use it, and does that + lift land in #5402 or here? *Recommend:* lift it to a shared FE location as part of #5402 + (or a small precursor PR) so Overview imports rather than forks; sequence Slice 4 after + #5402 merges. Confirm with Arda. +7. **Composer vs. Overview measure** — #5402 shows occupancy + running sum side by side to + pick one live. Overview should not wait on that: it uses occupancy now. *Confirm* the two + surfaces can settle the measure independently (Overview = occupancy; composer picks its + own). ## Next action From e6ad3db237ea81ef74e7b39697a387afaa5ea112 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 21 Jul 2026 18:36:34 +0600 Subject: [PATCH 3/5] Refine context usage documentation and update status to reflect changes from PR #5434 --- .../agent-overview-redesign/README.md | 5 +- .../agent-overview-redesign/design.md | 17 +-- .../projects/agent-overview-redesign/plan.md | 24 +++-- .../agent-overview-redesign/research.md | 102 ++++++++++++------ .../agent-overview-redesign/status.md | 52 +++++---- 5 files changed, 128 insertions(+), 72 deletions(-) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md index f73abb5125..6d6d662c1d 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -20,8 +20,9 @@ the visual design. - Surface resource usage as a first-class group — context usage (how full the model's window gets), token consumption, cache savings, cost per run — mapped to three personas (owner, builder, budget owner). -- Reuse PR #5402's context-budget primitive (Arda) for context usage — occupancy measure, - shared `MODEL_CONTEXT_WINDOWS` map; don't fork it. +- Reuse the shipped context-budget primitive (PR #5402 + #5434) for context usage — + occupancy measure; denominator from `contextWindowForModel` (`@agenta/entities/workflow`, + sourced from the model catalog), not a hardcoded map. - Phase 1 composes existing data only (tracing, mounts, session interactions, triggers); no new backend endpoints. - Artifacts load lazily per row; file-less runs degrade to message output. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md index ac045e9da7..4eaf075e03 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -60,13 +60,16 @@ Per recent run, one row that adapts to what the run produced: ### Resource usage & cost (the "what is it consuming" group) - **Context usage** — how full the model's context window gets per run (occupancy: run tokens vs. the model's window), so a user can see runs approaching compaction/truncation — - *reuses PR #5402's primitive. Occupancy per run = `ag.metrics.unit.tokens.total` / - `gen_ai.usage.total_tokens` from the trace store, keyed by `ag.meta.model_name`, against - the window from `resolveModelContextWindow` / `MODEL_CONTEXT_WINDOWS` - (`web/oss/src/components/AgentChatSlice/assets/contextBudget.ts`). Window unknown → show - raw token count without the percentage.* Use the **occupancy** measure (current fullness / - compaction predictor), not the cumulative running sum. See `research.md` for the full - reconciliation and the litellm follow-up for the denominator. + *reuses the shipped primitive from PR #5402 + #5434. Occupancy per run = + `ag.metrics.unit.tokens.total` / `gen_ai.usage.total_tokens` from the trace store; the + denominator per run = `contextWindowForModel(capabilities, agentHarness, runModel)` + (`@agenta/entities/workflow`), sourced from the model catalog — not a hardcoded map. + Inputs (traced, `research.md` §1 / Q8): `capabilities` from the global harness-catalog + atom; `agentHarness` from agent config (`agent.harness.kind`), not the trace; `runModel` + from the run's LLM child span. Window `null` → show raw token count, no bar/percent.* Use the + **occupancy** measure (running sum was dropped in #5434). Match the composer's ambient + meter: fill bar + "Context N% used", amber `>= 75%`, red `>= 90%`. See `research.md` §1 for + the full reconciliation. - **Token consumption** — breakdown of prompt / completion / reasoning / cached tokens per run and over time — *`ag.metrics.unit.tokens.{prompt,completion,reasoning,cache_read,cache_creation,cached}`*. - **Cache savings** — how much prompt caching is saving (cache-read share, cost avoided) — diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md index 112b83710a..d6309bc050 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -58,17 +58,21 @@ Each slice leaves the tree working and testable. `ag.meta.tool.call.result`) and failures grouped by cause (`ag.exception.type`). 2. Resource usage: token consumption breakdown, cache savings, cost per run/trend, and model/provider per run — all from trace token/cost attributes. -3. Context usage: reuse PR #5402's primitive - (`AgentChatSlice/assets/contextBudget.ts` — `resolveModelContextWindow` / - `MODEL_CONTEXT_WINDOWS`). Occupancy per run = run total tokens (trace store) vs. the - model's window, keyed by `ag.meta.model_name`; use the occupancy measure (not the - running sum); degrade to a raw token count when the window is `null`. First, lift the - shared bits (the window map + resolver) out of `AgentChatSlice` to a shared location so - composer and Overview read one source; coordinate with #5402 rather than forking the map. +3. Context usage: reuse the shipped primitive (PR #5402 + #5434). Denominator per run = + `contextWindowForModel(capabilities, agentHarness, runModel)` — already exported from + `@agenta/entities/workflow`, sourced from the model catalog on the harness catalog (no map + to lift, no fork). Wire the three inputs from their real locations (traced, question #8): + `capabilities` from the global `harnessCapabilitiesAtomFamily("")`; `agentHarness` from the + agent config (`agent.harness.kind`) — NOT the trace; `runModel` from the run's LLM child + span (`ag.meta.request.model` / `ag.meta.response.model`), while the token total comes off + the workflow root span — so read root + LLM child together per run. Occupancy = latest + turn's total tokens vs. that window; degrade to a raw token count when the window is + `null`. Match the composer's ambient-meter styling (bar + "N% used", amber `>= 75%`, + red `>= 90%`). - **Exit:** a user can see which tools an agent leans on and how reliable they are, how full - the context window gets per run (occupancy), and where token/cost is going — with - unknown-window runs showing a token count rather than a broken percentage, and no - duplicated context-window map. + the context window gets per run (occupancy, resolved from the catalog via + `contextWindowForModel`), and where token/cost is going — with unknown-window runs showing + a token count rather than a broken percentage. ## Slice 5 — Empty state + onboarding diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md index dc5e98742b..aa874dd587 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md @@ -46,44 +46,80 @@ semantic-convention attributes (`ag.*`, `gen_ai.*`) verified present in the SDK Per-run cost, duration, tools-used, tokens, outcome, and failure reason are all derivable from a single trace. The existing charts already aggregate over traces. -**Context usage** (how full the model's context window gets) — prior art already exists: -**PR #5402** (`fe-feat/session-context-info`, Arda, open/draft) adds a "Context X / max -(Y%)" budget indicator to the agent chat composer. Read its helpers before building the -Overview version — reuse, don't reinvent: - -- `web/oss/src/components/AgentChatSlice/assets/contextBudget.ts` — `computeContextBudget`, - `resolveModelContextWindow`, and the `MODEL_CONTEXT_WINDOWS` map. -- `web/oss/src/components/AgentChatSlice/components/ContextBudgetIndicator.tsx` — the - presentational strip. - -What that PR establishes (and corrects in this research): - -- **The backend does NOT expose per-model context windows to the frontend.** The model - registry the FE reads stores only *cost*. The `context_window` field I found earlier in - `sdks/python/agenta/sdk/agents/model_catalog.py:74` is SDK-side curated data, not surfaced - through the API — so it is not a usable denominator for a FE view today. The denominator - instead comes from the hardcoded FE `MODEL_CONTEXT_WINDOWS` map (substring/longest-match on - the model id), returning `null` for unknown models → the view drops the "/ max (%)" and - shows raw tokens. Documented follow-up: source the window from litellm - `get_model_info().max_input_tokens` instead of the static map. -- **Two candidate measures**, shown side by side in v1 until one is picked on a live agent: - - **occupancy** — the *latest* turn's total tokens = how full the window is *now*; grows - with the conversation and **drops after a compaction**, so it's the compaction predictor. - This is the measure that means "context usage". - - **running sum** — Σ of every turn's tokens; cumulative, double-counts resent history, - never drops. A usage meter, not an occupancy meter. +**Context usage** (how full the model's context window gets) — prior art exists and is now +settled across a stacked pair. Reuse it; do not reinvent: + +- **PR #5402** (`fe-feat/session-context-info`, Arda) introduced the composer's + context-budget indicator. +- **PR #5434** (`feat/extend-arda's-session-context-work`, Ashraf, stacked on #5402) + supersedes the parts below and is the current source of truth. + +Files: +- `web/oss/src/components/AgentChatSlice/assets/contextBudget.ts` — `computeContextBudget` + (occupancy only). +- `web/oss/src/components/AgentChatSlice/components/ContextBudgetIndicator.tsx` — the ambient + meter (`maxTokens: number | null` prop). +- `contextWindowForModel` — exported from `@agenta/entities/workflow`. + +What #5434 establishes (and **corrects the earlier finding in this research**): + +- **The context window IS available to the frontend — from the model catalog, not a + hardcoded map.** The SDK's `model_catalog.py` ships each model's `context_window` on the + **harness catalog** the FE already fetches — `GET /workflows/catalog/harnesses/`, a + **global, project-independent** query atom (`harnessCatalogQueryAtom` in + `web/packages/agenta-entities/src/workflow/state/inspectMeta.ts`, disk-seeded to + localStorage, background-revalidated), exposed via `harnessCapabilitiesAtomFamily("")`. + (NOT the `/inspect` response — inspect carries no behavior-changing `meta`; the file's own + header comment says so.) #5434 deleted the hardcoded `MODEL_CONTEXT_WINDOWS` map + + substring resolver and replaced them with an exact-id lookup: + `contextWindowForModel(capabilities, harness, modelId)` → + `capabilities[harness]?.model_catalog?.find(e => e.id === modelId)?.context_window ?? null` + (`@agenta/entities/workflow`). No new endpoint, every model in the picker covered + automatically, exact match instead of a guess. (This overrides the prior note that "the + backend stores only cost"; the *registry* does, but the *harness-capabilities doc* carries + `context_window`. No litellm follow-up needed.) +- **`harness` is now a required input.** `useAgentModelKeyStatus` returns `harness` (from + `agent.harness.kind`, e.g. `pi_core` / `claude`); the lookup is per-harness + per-model. +- **`context_window: null` still happens** for some catalog entries (e.g. the Claude + `default` alias) → degrade to a raw token count, no bar/percent. The fix is a data change + in `model_catalog.py`, not frontend. +- **Occupancy is the locked measure.** #5434 dropped the `Σ` running-sum from both the data + layer and the UI (it double-counted resent history and never dropped). Occupancy = the + latest turn's total tokens = how full the window is now; drops after a compaction. +- **UI is an ambient meter**, not a raw dual readout: a slim fill bar + "Context N% used", + escalating neutral → amber (`>= 0.75`) → red ("Context almost full", `>= 0.9`); exact + counts (`407k / 1M tokens`) in the tooltip. - **Token source (FE):** per-turn usage is stamped on assistant messages (`message.metadata.usage`, read via `getMessageUsage` in `web/oss/src/components/AgentChatSlice/assets/trace.ts`) — no new fetch, works for live and reloaded sessions. -Implication for Overview: the *concept* and the *window map* are shared with the composer, -but the *data path differs*. The composer reads a single live session's `messages`. Overview -aggregates across many historical runs, so it reads per-run token totals from the trace -store (`ag.metrics.unit.tokens.total` / `gen_ai.usage.total_tokens`) and applies the same -`resolveModelContextWindow` map. Occupancy per run = that run's total tokens vs. the model's -window. This connects to the known token-overhead behavior (a bare turn can cost ~15K tokens -from advertised tool schemas); context usage makes that visible on Overview. +Implication for Overview: the *concept*, the *denominator selector* (`contextWindowForModel`), +and the *occupancy measure* are all shared with the composer — only the *data path differs*. +The composer reads a single live session's `messages`. Overview aggregates across many +historical runs, so it reads per-run token totals from the trace store and, for each run, +resolves the denominator with `contextWindowForModel(capabilities, harness, model)`. + +Two inputs to that call are **not co-located on a run** (confirmed for open question #8): + +- **`capabilities`** — the global harness-catalog atom above. Readable anywhere with + `useAtomValue(harnessCapabilitiesAtomFamily(""))`; no `AgentChatSlice` coupling. ✅ +- **`model`** — sits on the run's **LLM child spans** (`ag.meta.request.model` / + `ag.meta.response.model`, mapped in `api/oss/src/apis/fastapi/otlp/opentelemetry/semconv.py`), + while the token total (`gen_ai.usage.total_tokens`) is rolled onto the **workflow root** + span (`sdks/python/agenta/sdk/agents/tracing.py:record_usage`). So pairing (model, tokens) + per run means reading the root **and** its LLM child — a per-run detail read, not just the + root aggregate. A run can hold multiple LLM spans; occupancy = the latest turn's total. +- **`harness`** — **not emitted on any span.** It is an agent-config property + (`agent.harness.kind`, e.g. `pi_core` / `claude`), stable per agent. Take it from the + agent, exactly as the composer does via `useAgentModelKeyStatus`; do not expect it on the + trace. If the agent's harness changed since an old run, or the run's model is absent from + the current harness's catalog, `contextWindowForModel` returns `null` → the raw-token + degrade already designed for. + +Occupancy per run = that run's latest-turn total tokens vs. its model's window. This connects +to the known token-overhead behavior (a bare turn can cost ~15K tokens from advertised tool +schemas); context usage makes that visible on Overview. ### 2. Produced artifacts — session mounts diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md index ad17fcdf55..099c612c32 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -1,6 +1,6 @@ # Status -**Last updated:** 2026-07-21 +**Last updated:** 2026-07-21 (context-window sourcing updated per PR #5434) ## Current stage @@ -22,13 +22,16 @@ only; no code, no branch, no commit. tokens vs. the model's window), token consumption, cache savings, cost per run, and model/provider. Context usage degrades to a raw token count when the window size is unknown. -- Context usage **reuses PR #5402** (Arda, `fe-feat/session-context-info`), which already - builds the composer's context-budget indicator. Overview uses the **occupancy** measure - (current fullness / compaction predictor), not the cumulative running sum. The - context-window denominator comes from that PR's FE `MODEL_CONTEXT_WINDOWS` map (the backend - registry stores only cost, not window sizes); do not fork the map — lift it to a shared - location and coordinate with #5402. Follow-up (owned by #5402): source the window from - litellm `get_model_info().max_input_tokens`. +- Context usage **reuses the shipped primitive** across a stacked pair: PR #5402 (Arda, + `fe-feat/session-context-info`) + **PR #5434** (Ashraf, + `feat/extend-arda's-session-context-work`, stacked on #5402). #5434 is the source of truth. + Overview uses the **occupancy** measure only (running sum was dropped in #5434). The + context-window denominator comes from the **model catalog on the harness-capabilities doc**, + resolved by `contextWindowForModel(capabilities, harness, modelId)` exported from + `@agenta/entities/workflow` — no hardcoded map, no litellm follow-up. `context_window: null` + entries (e.g. Claude `default` alias) degrade to a raw token count; filling them is a + `model_catalog.py` data change, not FE. Match the composer's ambient-meter UI (bar + + "Context N% used", amber `>= 75%`, red `>= 90%`). - The view catalog serves three personas (owner/operator, builder, budget owner) from one page via priority ordering, not three separate pages. - Phase 1 composes only data sources that already exist (tracing, mounts, session @@ -59,18 +62,27 @@ only; no code, no branch, no commit. 5. **"Run now" ownership** — does Overview trigger a run itself or hand off to the existing manual-run path? *Recommend:* reuse the existing manual-trigger path; Overview only invokes it. -6. **Context-window map ownership** — where does the shared `MODEL_CONTEXT_WINDOWS` / - `resolveModelContextWindow` live once both composer and Overview use it, and does that - lift land in #5402 or here? *Recommend:* lift it to a shared FE location as part of #5402 - (or a small precursor PR) so Overview imports rather than forks; sequence Slice 4 after - #5402 merges. Confirm with Arda. -7. **Composer vs. Overview measure** — #5402 shows occupancy + running sum side by side to - pick one live. Overview should not wait on that: it uses occupancy now. *Confirm* the two - surfaces can settle the measure independently (Overview = occupancy; composer picks its - own). +6. ~~Context-window map ownership~~ — **RESOLVED by #5434.** No shared map to own: the + denominator is `contextWindowForModel` in `@agenta/entities/workflow`, sourced from the + model catalog on the harness-capabilities doc. Overview imports the selector. Slice 4 + depends on #5434 being merged (it is stacked on #5402). +7. ~~Composer vs. Overview measure~~ — **RESOLVED by #5434.** Occupancy is the settled + measure everywhere; the running sum was dropped. No divergence to reconcile. +8. ~~Overview data path for context usage~~ — **RESOLVED (traced).** (a) The + harness-capabilities doc is a **global, project-independent** atom + (`harnessCatalogQueryAtom` ← `GET /workflows/catalog/harnesses/`); Overview reads it with + `useAtomValue(harnessCapabilitiesAtomFamily(""))`, no `AgentChatSlice` coupling. (b) Per + run, `model` is on the **LLM child span** (`ag.meta.request.model` / + `ag.meta.response.model`) while the token total is on the **workflow root** span — pair the + root with its LLM child. (c) `harness` is **not on the trace**; it's agent config + (`agent.harness.kind`), stable per agent — read it from the agent like the composer does. + Call `contextWindowForModel(globalCapabilities, agentHarness, runModel)`; unknowns degrade + to raw token count. Implementation note captured in `plan.md` Slice 4. ## Next action -Share the workspace and the five open questions for agreement. On sign-off, hand the view -catalog (`design.md`) to design for the Figma pass, and start Slice 0 on an implementation -branch (separate from this planning session). +Context-usage feasibility (Q6–8) is fully traced and settled; the remaining open items are +the product/rollout calls (Q1–5). Share the workspace and Q1–5 for agreement. On sign-off, +hand the view catalog (`design.md`) to design for the Figma pass, and start Slice 0 on an +implementation branch (separate from this planning session). Slice 4 depends on #5434 (the +context-usage primitive) being merged. From b705a28d1d944beb8ce7189c70059b58a3db5466 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 21 Jul 2026 19:30:24 +0600 Subject: [PATCH 4/5] Update agent Overview redesign documentation with analytics gateway integration and cost/token attribution details --- .../agent-overview-redesign/README.md | 14 ++- .../agent-overview-redesign/context.md | 4 +- .../agent-overview-redesign/design.md | 43 +++++-- .../projects/agent-overview-redesign/plan.md | 84 +++++++++---- .../agent-overview-redesign/research.md | 113 ++++++++++++++++-- .../agent-overview-redesign/status.md | 53 ++++++-- 6 files changed, 255 insertions(+), 56 deletions(-) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md index 6d6d662c1d..2b556d5eb0 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -16,15 +16,21 @@ the visual design. - Replace prompt-era Overview views with agent-native ones, gated by workflow kind. - Organize around three questions; lead with work (outcomes + produced artifacts), not config. -- Keep the charts; relabel Requests → Runs. +- Keep the charts, but drive them from the analytics gateway (`POST /analytics/query`) with + explicit specs — the current dashboard uses ~5% of the endpoint. Relabel Requests → Runs; + upgrade Latency from avg-only to percentiles. The endpoint is root-only (run-level), so + child-span breakdowns (tools, per-cause failures) stay on per-run trace reads. See + `research.md` §5–6. - Surface resource usage as a first-class group — context usage (how full the model's window gets), token consumption, cache savings, cost per run — mapped to three personas - (owner, builder, budget owner). + (owner, builder, budget owner). Agent cost/token totals depend on a backend attribution fix + (`research.md` §6); until it lands these views degrade gracefully, never showing a false zero. - Reuse the shipped context-budget primitive (PR #5402 + #5434) for context usage — occupancy measure; denominator from `contextWindowForModel` (`@agenta/entities/workflow`, sourced from the model catalog), not a hardcoded map. -- Phase 1 composes existing data only (tracing, mounts, session interactions, triggers); - no new backend endpoints. +- Phase 1 composes existing data only (tracing, mounts, session interactions, triggers) and + reuses `/analytics/query`; no new backend endpoints (one ingest-side attribution fix is a + Phase 2 dependency for the Cost/Token views). - Artifacts load lazily per row; file-less runs degrade to message output. - Three distinct zero states; new agents get onboarding, not zeroed charts. - Name views/data here; leave layout/visuals to design. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md index 75c7d73903..6b72f96a2e 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md @@ -69,4 +69,6 @@ user onto the same prompt-era panels rendering zeros, with no path toward first 6. The page surfaces the agent's resource usage in terms each persona can act on — context usage (how full the model's context window gets), token consumption, cache savings, and cost per run — not just aggregate charts. Each of these ties to a named source in - `design.md` and maps to a persona in the "Who reads this" section. + `design.md` and maps to a persona in the "Who reads this" section. (Cost/token *totals* + for agents depend on a backend attribution fix — `research.md` §6, `plan.md` Slice 6; the + views ship degraded, never a false zero, until it lands.) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md index 4eaf075e03..d096c864b3 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -45,17 +45,29 @@ Per recent run, one row that adapts to what the run produced: - **Run meta** — trigger source · duration · cost · tools-used count — *tracing + triggers*. -### Metrics (keep the charts) -- **Runs** (was "Requests"), **Avg latency**, **Cost**, **Tokens** over time — - *`ObservabilityOverview` aggregates, relabelled*. +### Metrics (charts) — from the analytics gateway +- **Runs** (was "Requests"), **Latency**, **Cost**, **Tokens** over time — *`POST + /analytics/query` with explicit `specs` (research.md §5–6), not the 5% the current charts + use.* The endpoint aggregates **root spans only** (one row per run), which is exactly + right for run-level charts. +- **Latency is the free upgrade**: request percentiles + histogram, not just the avg the + current chart shows (one 38 s run skews the mean). Show p50/p90/p95, keep avg as a + secondary stat. +- **Cost & Tokens are blocked on attribution** (research.md §6): agent runs don't populate + `costs.cumulative.total` (cost is derived only on LLM-type child spans; `gen_ai.usage.cost` + is unmapped) and only populate `tokens.cumulative.total` via a best-effort bridge. This is + why the live dashboard shows `-`. Depends on the backend fix (plan.md Slice 4 note + + Slice 6); until then, degrade to the run-level token count the agent reports, not a zero. ### Reliability (drill-down, not forced) -- **Most-used tools** — which tools it calls, counts, pass/fail flag — *Overview: - `ag.tool.name` / `ag.meta.tool.call.result` from the trace store. Live composer - equivalent (same style as the context-budget indicator, no new fetch): tool message-parts - (`part.type` `tool-*`/`dynamic-tool`, name via `partToolName`, pass/fail via `part.state`); - see `research.md` §3b.* -- **Where it's failing** — failures grouped by cause — *`ag.exception.type`*. +- **Most-used tools** — which tools it calls, counts, pass/fail flag — *`ag.tool.name` / + `ag.meta.tool.call.result`, on **tool child spans** → **not** servable by the root-only + `/analytics/query`; read per-run via bounded trace reads, or add a backend roll-up + (research.md §3b/§6). Live composer equivalent (same style as the context-budget indicator, + no new fetch): tool message-parts (`part.type` `tool-*`/`dynamic-tool`, name via + `partToolName`, pass/fail via `part.state`).* +- **Where it's failing** — failures grouped by cause — *`ag.exception.type`; child-span data, + same root-only caveat as tools.* ### Resource usage & cost (the "what is it consuming" group) - **Context usage** — how full the model's context window gets per run (occupancy: run @@ -71,13 +83,20 @@ Per recent run, one row that adapts to what the run produced: meter: fill bar + "Context N% used", amber `>= 75%`, red `>= 90%`. See `research.md` §1 for the full reconciliation. - **Token consumption** — breakdown of prompt / completion / reasoning / cached tokens per - run and over time — *`ag.metrics.unit.tokens.{prompt,completion,reasoning,cache_read,cache_creation,cached}`*. + run and over time — *`ag.metrics.unit.tokens.{prompt,completion,reasoning,cache_read,cache_creation,cached}`. + Run-level totals + distributions (mean/percentiles/histogram) come from the analytics + gateway (research.md §6) once agent tokens are attributed to the root; per-token-type + breakdown that isn't on the root reads per-run from the trace.* - **Cache savings** — how much prompt caching is saving (cache-read share, cost avoided) — *`ag.metrics.unit.tokens.cache_read` / `gen_ai.usage.cache_read.input_tokens`*. - **Cost** — cost per run, avg per run, and trend; optionally cost per tool — - *`gen_ai.usage.cost`*. + *`gen_ai.usage.cost`. **Attribution gap (research.md §6):** this attribute is dropped at + ingest (no semconv mapping) and cost is derived only on LLM-type child spans, so agent + runs currently have no root cost — the live dashboard shows `-`. Needs the backend fix + before this view has data.* - **Model & provider** — which model/provider each run used (agents can switch models) — - *`ag.meta.model_name` / `ag.meta.provider`*. + *`ag.meta.model_name` / `ag.meta.provider`; on LLM child spans, so **not** the root-only + analytics endpoint — read per-run from the trace (same as context usage's `runModel`).* ## Who reads this (persona → what they use) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md index d6309bc050..b8e5980aa6 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -8,8 +8,13 @@ untouched for other kinds. - Feature-flag the new agent Overview so it can ship dark and be switched on per environment. -- Anything needing new server-side aggregation is Phase 2 (called out in Slice 4), not - built in Phase 1. +- Anything needing new server-side aggregation is Phase 2 (flagged where it bites in Slice 4, + built in Slice 6), not built in Phase 1. +- **One backend dependency is surfaced, not a new endpoint:** agent cost/token + **attribution** at ingest (research.md §6). The existing `/analytics/query` endpoint is + reused as-is (it is the "gateway" — Slice 1/4); but agent runs don't populate the root + cost/token paths it reads, so the Cost/Token views degrade until that ingest fix lands + (Slice 4 note + Slice 6). This does not change "no new endpoints in Phase 1." Each slice leaves the tree working and testable. @@ -27,9 +32,14 @@ Each slice leaves the tree working and testable. 1. Build the status/health band: agent status, last run, next run (cron), success rate, avg cost, Run now — all from existing tracing aggregates + trigger schedule. -2. Relabel the reused charts (Requests → Runs) and mount them under the band. +2. Drive the charts from the analytics gateway (`POST /analytics/query`) with **explicit + `specs`**, not the defaults (research.md §5–6). Relabel Requests → Runs. Add **latency + percentiles** (p50/p90/p95 + histogram) — a near-free upgrade over the current avg-only + chart, which one slow run skews. Cost/Tokens use the same endpoint but are gated on the + attribution fix (Slice 4 note); until then degrade to the reported token count, never a + zero. - **Exit:** for an agent with runs, the band shows real last-run/next-run/success/cost and - the charts render; "Run now" triggers a run. + the charts render (latency shows a distribution, not just avg); "Run now" triggers a run. ## Slice 2 — Outcomes feed (with adaptive artifact/message rows) @@ -54,10 +64,23 @@ Each slice leaves the tree working and testable. ## Slice 4 — Reliability & resource-usage panels -1. Reliability: most-used tools (counts + pass/fail from `ag.tool.name` / - `ag.meta.tool.call.result`) and failures grouped by cause (`ag.exception.type`). -2. Resource usage: token consumption breakdown, cache savings, cost per run/trend, and - model/provider per run — all from trace token/cost attributes. +Two data paths here — keep them separate (research.md §6). `/analytics/query` aggregates +**root spans only** (`WHERE parent_id IS NULL`), so it serves run-level *numeric +distributions* but cannot see child-span categoricals. + +1. Reliability (**child-span data → per-run trace reads, not the gateway**): most-used tools + (counts + pass/fail from `ag.tool.name` / `ag.meta.tool.call.result`, on tool spans) and + failures grouped by cause (`ag.exception.type`). Read from bounded `LIMIT N` recent-run + traces; a backend roll-up onto the root (Slice 6) would let the gateway serve these later. +2. Resource usage: + - **Run-level numeric distributions via the gateway** (`/analytics/query` specs): + token totals and cost per run — mean/percentiles/histogram/trend — **once agent + cost/token attribution is fixed** (research.md §6; cost is dropped at ingest and + derived only on LLM-type spans, tokens ride a best-effort bridge). Until then this view + degrades to the reported token count and hides cost, never showing a false zero. + - **Cache savings** and **per-token-type breakdown / model & provider per run** are + child-span or reported-usage data → read per-run from the trace (same path as context + usage's `runModel`), not the gateway. 3. Context usage: reuse the shipped primitive (PR #5402 + #5434). Denominator per run = `contextWindowForModel(capabilities, agentHarness, runModel)` — already exported from `@agenta/entities/workflow`, sourced from the model catalog on the harness catalog (no map @@ -69,10 +92,12 @@ Each slice leaves the tree working and testable. turn's total tokens vs. that window; degrade to a raw token count when the window is `null`. Match the composer's ambient-meter styling (bar + "N% used", amber `>= 75%`, red `>= 90%`). -- **Exit:** a user can see which tools an agent leans on and how reliable they are, how full - the context window gets per run (occupancy, resolved from the catalog via - `contextWindowForModel`), and where token/cost is going — with unknown-window runs showing - a token count rather than a broken percentage. +- **Exit:** a user can see which tools an agent leans on and how reliable they are (from + per-run traces), how full the context window gets per run (occupancy, resolved from the + catalog via `contextWindowForModel`), and where tokens are going — with unknown-window runs + showing a token count rather than a broken percentage. **Cost is shown only where + attributed** (LLM-app runs today; agent runs after the Slice 6 ingest fix), never a false + zero. ## Slice 5 — Empty state + onboarding @@ -85,17 +110,30 @@ Each slice leaves the tree working and testable. - **Exit:** a brand-new agent sees onboarding with a working first action and steps that tick off as data appears; a first failed run shows the failure, not "no data". -## Slice 6 (Phase 2, optional) — Materialized aggregates - -Only if Phase 1 render latency proves the per-request aggregation too costly at scale. -1. Index mount file metadata in Postgres at write time (avoid object-store LIST for chips). -2. Materialize run aggregates (success rate, cost, tool pass/fail) at ingest. -3. Serve the band + reliability panels from the materialized store; tolerate labelled - staleness. -- **Exit:** Overview first paint no longer depends on live fan-out; numbers are within the - stated staleness budget. +## Slice 6 (Phase 2) — Backend attribution + materialized aggregates + +Two independent backend items. The first is a **correctness dependency** (agent cost/token +data), not just a performance backstop; the rest are performance/scale, done only if Phase 1 +render latency proves the per-request aggregation too costly. + +1. **Agent cost/token attribution** (unblocks the Cost/Token views, research.md §6). Pick + one: (a) map `gen_ai.usage.cost` in semconv and treat agent/workflow root spans as + cost-bearing (or bridge the harness's separate OTLP batch so its LLM spans roll onto the + root); or (b) accept the FE reading the agent's reported per-run usage from the trace and + aggregating client-side. Either makes the root cost/token paths the gateway reads + non-empty for agents. +2. **Roll child-span categoricals onto the root** (or expose a child-focused analytics + aggregation) so most-used-tools / failures-by-cause / model-mix can move from per-run + trace reads to the gateway — removes the reliability panels' fan-out. +3. Index mount file metadata in Postgres at write time (avoid object-store LIST for chips). +4. Materialize run aggregates (success rate, cost, tool pass/fail) at ingest; serve the band + + reliability panels from the materialized store, tolerating labelled staleness. +- **Exit:** agent Cost/Token views show real numbers; reliability/first-paint no longer + depend on live per-run fan-out; numbers are within the stated staleness budget. ## Flag rollout -Ship Slices 0–5 behind the flag, dogfood, then default on for the agent kind. Slice 6 is a -performance backstop, not a prerequisite. +Ship Slices 0–5 behind the flag, dogfood, then default on for the agent kind. Slice 6 is +mostly a performance backstop, **except item 1 (agent cost/token attribution)**, which gates +the Cost/Token views specifically — those degrade gracefully until it lands, so it does not +block the flag, only those two views' data. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md index aa874dd587..d149d2d227 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/research.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/research.md @@ -172,9 +172,11 @@ Tool calls are message *parts*: So which-tools / call-count / pass-fail are all computable from the live session with no new backend work — the composer parallel to context budget. **Overview** aggregates across -historical runs instead, from the trace store (`ag.tool.name` / `ag.meta.tool.call.result`), -per §1. Same concept, two data paths (live messages vs. trace store) — same split as context -usage. +historical runs instead, from the trace store (`ag.tool.name` / `ag.meta.tool.call.result`). +Note (see §6): `ag.tool.name` lives on **tool child spans**, so this is **not** servable by +the root-only `/analytics/query` aggregation — Overview reads it via per-run trace reads +(bounded `LIMIT N`), or it needs a backend roll-up onto the root. Same concept as context +usage: two data paths (live messages vs. per-run trace reads), not the aggregate endpoint. ### 4. Triggers and schedules @@ -192,10 +194,107 @@ usage. So "trigger source" (schedule / inbound event / manual), "next scheduled run" (from cron), and "connection needs reconnect" (from `ConnectionStatus`) are all available. -### 5. Aggregate metrics — existing charts - -`ObservabilityOverview` already computes run count, latency, cost, tokens over the trace -store. Reusable as-is; only the labels change (Requests → Runs). +### 5. Aggregate metrics — the analytics endpoint (the gateway) + +`ObservabilityOverview` → `AnalyticsDashboard` → `useObservabilityDashboard` → +`fetchGenerationsDashboardData` → **`POST /analytics/query`** (Fern `querySpansAnalytics`, +backend `operation_id="query_analytics"`, +`api/oss/src/apis/fastapi/tracing/router.py:118`). This one endpoint is far more capable +than the four charts it currently feeds — see §6. The current wiring uses ~5% of it. +Reusable, but do **not** treat it as "relabel and ship": the charts today throw away most of +what the endpoint returns, and its default specs don't attribute agent cost/tokens (§6). + +### 6. `POST /analytics/query` — full capability, root-only scope, and the cost/token gap + +Verified against `api/oss/src/core/tracing/{service,dtos}.py`, +`api/oss/src/dbs/postgres/tracing/{dao,utils}.py`, +`web/oss/src/services/tracing/lib/helpers.ts`, and the ingest roll-up in +`api/oss/src/core/tracing/utils/trees.py`. This section supersedes the earlier "reusable +as-is, only labels change" framing. + +**Request surface.** The endpoint takes a `TracingQuery` (arbitrary `filtering` + +time-`windowing`/`interval`) and a list of **`MetricSpec`** (`type` + attribute `path`, plus +histogram opts `bins`/`vmin`/`vmax`/`edge`; `dtos.py:249`). The frontend currently sends +**no `specs`** (`web/packages/agenta-entities/src/trace/api/api.ts`), so the backend applies +`DEFAULT_ANALYTICS_SPECS` — only 6 paths (`service.py:90`): duration, errors, cost.total, +tokens.total, trace-type, span-type. Then `analyticsToGeneration` (`helpers.ts:106`) reads +only `sum`/`count` off 5 of them → the 4 charts. + +**What it computes per time bucket, per requested spec** (`dao.py:426-484`, builders in +`utils.py`): +- **Numeric** (`numeric/continuous`, `numeric/discrete`): `count`, `sum`, `mean`, `min`, + `max`, `range`, **percentiles** (p25/p50/p75/p90/p95/p99/…) with IQR/CQV/PSC spreads, and + **histograms** (bin counts + shares). +- **Categorical / binary / string** (`categorical/single|multiple`, `binary`, `string`): + **value-count frequencies** — each distinct value + its count + uniqueness. + +**Root-only scope — the load-bearing constraint.** The analytics base CTE hardcodes +`WHERE parent_id IS NULL` (`utils.py:1067`); the `focus` field does not lift this. **Every +metric is aggregated over root/workflow spans only — one row per run.** Consequences: +- **Servable directly** (one call, better specs): run count, success/failure split, and + **latency distributions** (p50/p90/p95/p99 + histogram — a real upgrade over today's + avg-only, which one 38 s run skews). Plus **run-level cost & token distributions** *once + attribution is fixed*, and any **categorical stamped on the root** (kind, channel, origin, + environment, and the run-level final exception if it lands on the root). +- **NOT servable by this endpoint**: **per-tool** usage/pass-fail (`ag.tool.name` lives on + tool child spans), **per-LLM model/provider** mix (`ag.meta.*.model` on LLM child spans), + and **failures grouped by the failing child's** `ag.exception.type`. These are child-span + data; the root-only aggregation can't see them. They need per-run trace reads (as §1/§3b + already describe) or a backend change (roll those categoricals onto the root, or expose a + child-focused aggregation). **This corrects §3b's implication that "Most-used tools" is a + trace-store aggregate the dashboard path already serves — it is not, via this endpoint.** + +**The cost/token attribution gap (why the live dashboard shows Cost/Tokens `-` for agents).** +The default specs read `costs.cumulative.total` / `tokens.cumulative.total` on the root, +populated by the ingest roll-up (`calculate_and_propagate_metrics`), which is LLM-app-shaped: +- **Cost** is *derived*, never read from the agent. `gen_ai.usage.cost` has **no semconv + mapping** (`api/oss/src/apis/fastapi/otlp/opentelemetry/semconv.py`) and is dropped at + ingest. `calculate_costs` (`trees.py:547`) computes cost only on spans whose type ∈ + `{embedding, query, completion, chat, rerank}` (`trees.py:538`) using a model-keyed price + table. An agent's root is a `workflow`/`agent` span → not cost-bearing → **cost is always + empty for agent runs.** +- **Tokens** ride a best-effort bridge: the SDK stamps `gen_ai.usage.total_tokens` on the + `/invoke` workflow span via `record_usage` (`sdks/python/agenta/sdk/agents/tracing.py:213`) + → `unit.tokens.total` → `tokens.incremental.total` → rolled to `tokens.cumulative.total` + (`cumulate_tokens`). But `record_usage` early-returns when `usage.total` is falsy, and the + harness's real LLM spans arrive in a **separate OTLP batch** that cannot roll onto the root + — so tokens land only if that bridge fires with a non-zero total. + +Implication: the Resource-usage cost/token numbers cannot come from the default dashboard +path for agents. Either (i) fix attribution at ingest, or (ii) read the agent's reported +usage per run from the trace and aggregate on the FE. Tracked as a backend dependency in +`plan.md` (Slice 4 note + Slice 6). + +**The ingest roll-up path, traced (the fix is smaller than "make spans cost-bearing").** +The `unit.* → incremental.* → cumulative.*` pipe already exists and already handles costs: +`span_data_builders.py:171-179` renames every metric key unconditionally — +`unit.costs. → costs.incremental.`, `unit.tokens. → tokens.incremental.`, +`acc.costs. → costs.cumulative.`, `acc.tokens. → tokens.cumulative.`. So the **cost pipe is +built**; it is only starved at the semconv step. `record_usage` already stamps +`gen_ai.usage.cost` on the `/invoke` root (`tracing.py:234`), but semconv has no row for it, +so `unit.costs.total` is never created and the promotion never fires. **Minimal fix: one +semconv row** — `("gen_ai.usage.cost", "ag.metrics.unit.costs.total")`. Then: +`gen_ai.usage.cost` → (semconv) `unit.costs.total` → (`span_data_builders:175`) +`costs.incremental.total` → (`cumulate_costs`, and `calculate_costs` skips the workflow root +so it isn't clobbered — root ∉ `TYPES_WITH_COSTS`) `costs.cumulative.total` → read by the +default analytics spec. Cost appears for agents. + +Caveats on that one-liner: +- **Covers cost-reporting harnesses only.** `record_usage` sets cost only when `usage.cost` + is present and early-returns when `usage.total` is falsy. A harness that reports + tokens-but-no-cost still needs the price-table derivation — the one-liner is necessary, + not complete. +- **Latent double-count.** It is safe *because* the harness's real LLM spans arrive in a + separate OTLP batch (absent from the root's tree). A future "bridge the harness batch" fix + that puts price-derived child costs in the same tree would make the root count both. The + two fixes collide — pick one. +- **Ingest-time only.** No retroactive effect; cost shows for runs ingested after the change. + +**Cost vs. tokens are different failures.** Cost is broken *at the mapping* (definitive, above). +Tokens have a *complete* pipe already (`gen_ai.usage.total_tokens` → `unit.tokens.total` → +`tokens.incremental.total` → `tokens.cumulative.total`). So if the dashboard shows tokens as +`-`, the cause is `record_usage` not firing for that agent (or the harness-batch separation), +**not** a semconv gap — confirming it needs a live root-span attribute dump (running stack). ## FE surfaces that already consume this data (reuse, don't rebuild) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md index 099c612c32..ab239d39f6 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -1,6 +1,8 @@ # Status -**Last updated:** 2026-07-21 (context-window sourcing updated per PR #5434) +**Last updated:** 2026-07-21 (analytics endpoint audited: gateway capability, root-only +scope, and the agent cost/token attribution gap — research.md §5–6; cost roll-up path traced +— the fix is a one-line semconv row, Q9) ## Current stage @@ -17,11 +19,24 @@ only; no code, no branch, no commit. **what has it been doing?**, **is it healthy?** - Lead with the agent's *work* (outcomes + produced artifacts), not its configuration status. -- Keep the existing charts; relabel Requests → Runs. +- Keep the charts, but drive them from the **analytics gateway** (`POST /analytics/query`) + with **explicit `specs`** — the current dashboard uses ~5% of the endpoint. Relabel + Requests → Runs and upgrade Latency from avg-only to **percentiles + histogram** (research.md + §5–6). +- **The analytics endpoint is the gateway for run-level aggregates, and it is root-only** + (`WHERE parent_id IS NULL`, one row per run). It serves numeric distributions + root-stamped + categoricals. It does **not** serve child-span breakdowns (per-tool usage/pass-fail, per-LLM + model mix, failures by the failing child's cause) — those stay on per-run trace reads until a + backend roll-up (Slice 6). +- **Agent cost/token attribution is a real backend gap** (why the live dashboard shows Cost + and Tokens as `-`): `gen_ai.usage.cost` is unmapped at ingest and cost is derived only on + LLM-type child spans; agent tokens ride a best-effort `record_usage` bridge. The + Cost/Token views **degrade gracefully** (reported token count, cost hidden — never a false + zero) until the Slice 6 fix lands. Full detail: research.md §6. - Resource usage is a first-class group, not just charts: context usage (occupancy: run tokens vs. the model's window), token consumption, cache savings, cost per run, and model/provider. Context usage degrades to a raw token count when the window size is - unknown. + unknown; cost/token totals depend on the attribution fix above. - Context usage **reuses the shipped primitive** across a stacked pair: PR #5402 (Arda, `fe-feat/session-context-info`) + **PR #5434** (Ashraf, `feat/extend-arda's-session-context-work`, stacked on #5402). #5434 is the source of truth. @@ -35,7 +50,10 @@ only; no code, no branch, no commit. - The view catalog serves three personas (owner/operator, builder, budget owner) from one page via priority ordering, not three separate pages. - Phase 1 composes only data sources that already exist (tracing, mounts, session - interactions, triggers). No new backend endpoints in Phase 1. + interactions, triggers) and reuses the existing `/analytics/query` endpoint. No new backend + *endpoints* in Phase 1 — but one backend *ingest fix* (agent cost/token attribution) is a + surfaced dependency for the Cost/Token views, tracked in Slice 6 with graceful degradation + in the meantime. - Produced artifacts load lazily per outcome row (no object-store fan-out on first paint); rows with no files degrade to message output. - Three distinct zero states (never-run / per-section-empty / first-run-failed), not one @@ -62,6 +80,20 @@ only; no code, no branch, no commit. 5. **"Run now" ownership** — does Overview trigger a run itself or hand off to the existing manual-run path? *Recommend:* reuse the existing manual-trigger path; Overview only invokes it. +9. **Agent cost/token attribution — fix at ingest or on the FE?** (research.md §6, Slice 6 + item 1.) **Traced — the cost fix is one line.** The `unit.* → incremental.* → cumulative.*` + pipe already exists and already handles costs (`span_data_builders.py:171-179`), and + `record_usage` already stamps `gen_ai.usage.cost` on the agent root — the only gap is a + missing semconv row. Option (a): add `("gen_ai.usage.cost", "ag.metrics.unit.costs.total")` + to semconv; cost then flows to `costs.cumulative.total` (the default spec) with no + price-table or span-type work. Covers cost-reporting harnesses; token-only harnesses still + need price-table derivation. Watch the double-count risk if the harness batch is later + bridged, and note it's ingest-time only (not retroactive). Option (b): FE reads reported + per-run usage from the trace and aggregates client-side. *Recommend:* (a) — now known to be + cheap and the correct source of truth; (b) is a stopgap. **Tokens are a separate issue** — + their pipe is complete, so a `-` means `record_usage` isn't firing, not a mapping gap; + diagnose with a live root-span attribute dump. Ship Cost/Token views degraded until (a) + lands. Needs a backend owner; the mapping itself is a one-line PR. 6. ~~Context-window map ownership~~ — **RESOLVED by #5434.** No shared map to own: the denominator is `contextWindowForModel` in `@agenta/entities/workflow`, sourced from the model catalog on the harness-capabilities doc. Overview imports the selector. Slice 4 @@ -81,8 +113,11 @@ only; no code, no branch, no commit. ## Next action -Context-usage feasibility (Q6–8) is fully traced and settled; the remaining open items are -the product/rollout calls (Q1–5). Share the workspace and Q1–5 for agreement. On sign-off, -hand the view catalog (`design.md`) to design for the Figma pass, and start Slice 0 on an -implementation branch (separate from this planning session). Slice 4 depends on #5434 (the -context-usage primitive) being merged. +Context-usage feasibility (Q6–8) is fully traced and settled. The analytics endpoint is now +audited (research.md §5–6): it is the gateway for run-level charts/distributions, it is +root-only, and agent cost/token attribution is a backend gap (new Q9). Remaining open items +are the product/rollout calls (Q1–5) plus the backend-owned attribution decision (Q9). Share +the workspace and Q1–5 + Q9 for agreement. On sign-off, hand the view catalog (`design.md`) +to design for the Figma pass, and start Slice 0 on an implementation branch (separate from +this planning session). Slice 4 depends on #5434 (the context-usage primitive) being merged; +its Cost/Token data depends on Q9/Slice 6. From 6b1359ec92fa1bb314560a99b52ab03a66d38027 Mon Sep 17 00:00:00 2001 From: ashrafchowdury Date: Tue, 21 Jul 2026 21:24:00 +0600 Subject: [PATCH 5/5] Refine agent Overview documentation to clarify never-run empty state guidance and onboarding experience --- .../agent-overview-redesign/README.md | 13 ++--- .../agent-overview-redesign/context.md | 8 ++-- .../agent-overview-redesign/design.md | 48 +++++++++++-------- .../projects/agent-overview-redesign/plan.md | 17 ++++--- .../agent-overview-redesign/status.md | 21 +++++--- 5 files changed, 62 insertions(+), 45 deletions(-) diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md index 2b556d5eb0..598718b622 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/README.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/README.md @@ -6,8 +6,8 @@ Redesign the agent Overview page so it shows an agent's *work* instead of the prompt-management views it inherited (deployments, variants, evaluations). The page is organized around three questions — **does it need me?**, **what has it been doing?**, **is -it healthy?** — and handles the empty/no-data state and fresh-agent onboarding as -first-class states, not afterthoughts. This workspace names the views and data (each +it healthy?** — and handles the empty/no-data state, including a never-run agent, as a +first-class state, not an afterthought. This workspace names the views and data (each grounded in a verified backend source) and slices the implementation; it does not prescribe the visual design. @@ -32,7 +32,8 @@ the visual design. reuses `/analytics/query`; no new backend endpoints (one ingest-side attribution fix is a Phase 2 dependency for the Cost/Token views). - Artifacts load lazily per row; file-less runs degrade to message output. -- Three distinct zero states; new agents get onboarding, not zeroed charts. +- Three distinct zero states; a never-run agent gets plain guidance to the Playground (the + Overview reports work, it doesn't run the agent), not zeroed charts. - Name views/data here; leave layout/visuals to design. ## Deliverables @@ -41,7 +42,7 @@ the visual design. criteria. - [research.md](research.md) — current wiring and the verified backend data sources, with `file:line`. -- [design.md](design.md) — the view catalog, empty-state behavior, and onboarding flow. +- [design.md](design.md) — the view catalog, empty-state behavior, and never-run guidance. - [plan.md](plan.md) — sliced implementation (Slice 0–5) with exit checks. - [status.md](status.md) — living source of truth: locked decisions + open questions. @@ -50,5 +51,5 @@ the visual design. Opening an agent's Overview, a user immediately sees whether it needs them, what it has produced (plain-language outcomes and downloadable artifacts), and whether it is healthy — with drill-down to traces, tools, and token detail available but never forced. A -brand-new agent sees a short onboarding path to its first successful run and first trigger -instead of a wall of zeros. +brand-new agent sees plain guidance to where it can be tried (the Playground) and a preview +of what will appear, instead of a wall of zeros. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md index 6b72f96a2e..b11c7d4820 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/context.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/context.md @@ -21,8 +21,8 @@ user onto the same prompt-era panels rendering zeros, with no path toward first - The agent Overview page: what it shows for an **active** agent that has run. - The **empty / no-data** state for an agent that exists but has never produced activity. -- The **fresh-agent onboarding** path: from "just created" to first successful run and - first connected trigger. +- The **never-run empty state**: from "just created" to guidance on where to try the agent + (the Playground) and what the page will show once it has run. - A named catalog of agent-workflow views/data the page can draw from, each grounded in a real backend source (see `design.md` and `research.md`). @@ -61,8 +61,8 @@ user onto the same prompt-era panels rendering zeros, with no path toward first need me?**, **what has it been doing?**, **is it healthy?** 2. The page shows the agent's actual work — outcomes and produced artifacts — not its configuration status. -3. A brand-new agent sees an onboarding experience with a clear first action, never a wall - of zeroed prompt-era panels. +3. A brand-new agent sees plain-language guidance to where it can be tried (the Playground) + and a preview of what will appear, never a wall of zeroed prompt-era panels. 4. Every view on the page maps to a named, verified backend data source (`design.md`). 5. Non-technical and technical users can both read the page: plain-language outcomes up top, drill-down detail (traces, tokens, tool calls) available but not forced. diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md index d096c864b3..ed7590731b 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/design.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/design.md @@ -122,7 +122,8 @@ behind clicks for the builder and budget reader. ## Empty / no-data state (agent exists, never produced activity) Not one panel. Distinguish and handle: -- **New agent, never run** — onboarding (below), not zeroed charts. +- **New agent, never run** — the guidance state (below): a frame of what the page tracks + + a pointer to the Playground, not zeroed charts, not a run action, not a template picker. - **Per-section empty** — a section with no data yet shows an inviting empty state naming what will appear there and how to make it appear, not a blank box or a zero. - **First run failed** — treat as activity: show the failure and the reason, plus a "try @@ -134,26 +135,31 @@ Rules: - Never show a wall of zero charts as the primary content for a new agent. - Every empty section states the one action that fills it. -## Fresh-agent onboarding (just created → first value) - -Replace the empty Overview body with a getting-started experience. The milestones map to -real capabilities and each is individually detectable, so the checklist self-completes: - -1. **Run it once** — a "Run now" that produces the first outcome. *Done when the first - trace exists.* -2. **Connect an integration** — if the agent's tools/triggers need one. *Done when a - `TriggerConnectionStatus` is READY.* -3. **Add a trigger** — a schedule or event subscription so it runs on its own. *Done when a - `TriggerSubscription` exists.* - -Behavior: -- Show progress (e.g. 1 of 3), mark steps done as the underlying data appears, and dismiss - the onboarding once the agent has real activity — after which the active-agent views take - over. -- Reuse the existing agent-home onboarding / composer patterns and `useCreateAgent` flow - rather than inventing a parallel onboarding. -- Keep it to the few steps that lead to first value; do not gate the page behind it (a user - can always ignore the checklist and use "Run now"). +## Fresh-agent empty state (never run) — guidance, not a launcher + +The Overview is a **reporting** surface — it shows an agent's work; it isn't where you build +the agent or where you go to operate it. So a never-run agent gets a **guidance** state — not +a run action, not a template gallery. (We dropped both the earlier 3-step checklist and the +inline "Run now": the checklist led with configuration, and a launcher/picker makes the +Overview pretend to be a surface it isn't. Whether the *active* health band keeps a manual +"Run now" is a separate open question — see `status.md` Q5.) Structure only — wording is +design's: + +- **Frame + why it's empty.** State what this page tracks and that there is nothing yet + because the agent hasn't run. Plain, non-technical language. +- **One direction: the Playground.** A single pointer to the existing Playground (the + try/run surface) — the only affordance on the screen. No inline run, no steps. +- **Expectation-setter.** A faint preview of the sections that will appear (outcomes / + health / usage), so the empty page previews the filled one. +- **Config stays out of the empty state.** Connections/triggers are not shown here — a + connection needing auth surfaces as a needs-you item, and a trigger is added from the + triggers surface — never as an onboarding step. +- **Self-filling, nothing to dismiss.** Once a run exists (from the Playground or anywhere) + the active-agent views render normally. Empty-vs-active keys on the same first-trace signal + the rest of the page uses. + +Reuse the existing Playground route rather than inventing a launcher on Overview. Never gate +the page behind this state. ## Cross-cutting principles diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md index b8e5980aa6..c47ff9473d 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/plan.md @@ -99,16 +99,19 @@ distributions* but cannot see child-span categoricals. attributed** (LLM-app runs today; agent runs after the Slice 6 ingest fix), never a false zero. -## Slice 5 — Empty state + onboarding +## Slice 5 — Empty states + never-run guidance 1. Detect the three zero states (never-run / per-section-empty / first-run-failed) and render the right one; never show zeroed prompt-era panels. -2. Build the getting-started checklist (Run it once / Connect / Add a trigger) with - self-completing steps driven by first-trace / READY-connection / subscription-exists - detection. Reuse agent-home onboarding patterns + `useCreateAgent`. -3. Dismiss onboarding once the agent has real activity; hand off to active-agent views. -- **Exit:** a brand-new agent sees onboarding with a working first action and steps that - tick off as data appears; a first failed run shows the failure, not "no data". +2. Never-run agent → the guidance state (structure): a frame of what the page tracks + why + it's empty, one pointer to the Playground (reuse the existing Playground route), and a + faint preview of the sections that will appear. No run action, no checklist, no template + gallery, nothing to dismiss. +3. Empty-vs-active keys on the first-trace signal (the same one the active views use); once a + run exists the active-agent views render. Keep connections/triggers out of the empty state. +- **Exit:** a brand-new agent sees guidance to the Playground plus a preview of what will + appear; once it has run, the same page shows its work; a first failed run shows the failure, + not "no data". ## Slice 6 (Phase 2) — Backend attribution + materialized aggregates diff --git a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md index ab239d39f6..1c361bb8ce 100644 --- a/docs/design/agent-workflows/projects/agent-overview-redesign/status.md +++ b/docs/design/agent-workflows/projects/agent-overview-redesign/status.md @@ -2,7 +2,8 @@ **Last updated:** 2026-07-21 (analytics endpoint audited: gateway capability, root-only scope, and the agent cost/token attribution gap — research.md §5–6; cost roll-up path traced -— the fix is a one-line semconv row, Q9) +— the fix is a one-line semconv row, Q9. Never-run empty state settled: guidance to the +Playground — not a run action, checklist, or template gallery.) ## Current stage @@ -57,9 +58,12 @@ only; no code, no branch, no commit. - Produced artifacts load lazily per outcome row (no object-store fan-out on first paint); rows with no files degrade to message output. - Three distinct zero states (never-run / per-section-empty / first-run-failed), not one - "no data" panel. New agents get onboarding, not zeroed charts. -- Onboarding reuses agent-home onboarding patterns + `useCreateAgent`; steps self-complete - from real data and dismiss once the agent has activity. + "no data" panel. New agents get plain-language guidance, not zeroed charts. +- **Never-run Overview = guidance, not a launcher.** The Overview reports work; it does not + run the agent or pick its template. A never-run agent gets a guidance state — a frame of + what the page tracks + a single pointer to the Playground + a preview of what will appear — + not a run action, checklist, or template gallery. It self-fills on the first trace; nothing + to dismiss. (Supersedes the earlier run-CTA and 3-step-checklist ideas.) - This workspace names views/data only; layout, components, and visuals are design's (Claude design / Figma). - No commits during this planning session. @@ -77,9 +81,12 @@ only; no code, no branch, no commit. 4. **Needs-you resolution in place** — resolve pending interactions directly on Overview (embed `ApprovalDock`) or link out to the session? *Recommend:* link out in Phase 1, embed later if it proves worth it. -5. **"Run now" ownership** — does Overview trigger a run itself or hand off to the existing - manual-run path? *Recommend:* reuse the existing manual-trigger path; Overview only - invokes it. +5. **Manual "Run now" on an *active* agent** — the never-run empty state points to the + Playground (Overview reports, it doesn't launch). Does the active health band still keep a + manual "Run now" (re-run a working agent on demand), or defer to the Playground/triggers + for consistency? *Recommend:* keep a manual re-run on the active band, but if it stays, + reuse the existing manual-trigger path — Overview only invokes it, never owns running. + Resolve before Slice 1 finalizes the band. 9. **Agent cost/token attribution — fix at ingest or on the FE?** (research.md §6, Slice 6 item 1.) **Traced — the cost fix is one line.** The `unit.* → incremental.* → cumulative.*` pipe already exists and already handles costs (`span_data_builders.py:171-179`), and