Proposal: outcome and duration on product analytics events (turns, tools, approvals, connectors, schedules)

[frannouter]

Summary

PRODUCT_ANALYTICS already gives a deployment a stream of product events (gadget_created, connection_created,
workpiece_created, ...). We run a deployment of Cloudflare OS and have been extending that stream for a while. This
is not a PR: the contributing guide asks for a discussion first, and the change is bigger than a dozen lines in total.
We would like to know whether the maintainers want any of it, and if so, in which shape. If the answer is no, that is
fine too.

The gap

As far as we can tell from packages/workshop-backend/src/analytics.ts, the events that exist today have two
limitations:

1. They fire when the action succeeds. gadget_created, workpiece_created, connection_created and the rest are
   emitted after the happy path. A deployment can count what people did, but not what they tried and could not do, so
   it cannot tell "nobody uses scheduled tasks" from "everybody who tries them hits an error".
2. They carry no outcome and no duration. Nothing says how a turn ended, how long it ran, how long an approval
   waited, or how long connecting a connector took. These are the numbers an operator needs to answer "is the agent
   getting slower?", "which tool fails most?" or "where do people abandon the connect flow?".

In PRODUCT_ANALYTICS, nothing covers the agent turn, the tools it calls, approvals, or the connect flow, which is
where most of the questions come from once real people use a deployment.

Why not the existing agent spans?

We know agent-tracing.ts already emits invoke_agent, chat, execute_tool and tool_approval spans with
outcomes and timings, and they answer "what happened in this turn?" well. We would still like the same facts in the
stream, for three reasons:

• Queryable with SQL. The stream lands in a table, so "failure rate per tool, per week" or "median approval wait by
  connector" is one query, not a trace search.
• Retention. The stream keeps history for as long as the deployment keeps the table, which is what trend
  questions need.
• It joins with the other events. A turn, an approval or a connect failure can be put next to workpiece_created
  or gadget_opened for the same workspace, to ask whether people who hit an error come back.

There is also a limit that workshop-backend/AGENTS.md documents: a span's trace ends at the next event the Durable
Object receives once the invocation that opened it has returned, so a turn keeps all its spans only while the request
that started it stays open. An event written to the stream does not depend on that.

What we propose

New event names in the existing style (object_verb, snake_case, English). Properties are of the same kind the current
events use (ids, counts, booleans, closed lists). The exact names are open (the existing events say gatekeeper_id and
vendor_id, we wrote connection_id and vendor; see the questions below). Every event is emitted through
recordAnalytics (or a batch helper for the tool calls, below), so it also gets its activity/v1 metric, except where noted.

Event	Fires when	Properties
agent_turn_finished	An agent turn ends, however it ends	chat_id, model_id, duration_ms, tool_calls (count), outcome (ok, error, cancelled, limit, no_model), error_code (only on error)
error_shown	The user sees an error in a chat	chat_id, code, retryable (boolean)
tool_called	The agent calls a tool	chat_id, tool_name, connector_vendor (for a connector's tool), duration_ms, outcome (ok, error, cancelled), error_class
approval_requested	An action asks for approval	connection_id, vendor, action_kind, chat_id, request_id, auto (a rule will approve it)
approval_resolved	It is resolved	connection_id, vendor, action_kind, request_id, outcome (approved, rejected, automatic), by (person, rule), wait_ms
connector_connect_started / connector_connected / connector_connect_failed / connector_disconnected	A person connects or disconnects a connector account	vendor, reconnect (boolean), duration_ms (on connected), stage and error_class (on failed; stage is a closed list: start, handoff, save, reconnect, abandoned)
schedule_created / _enabled / _disabled / _deleted	A scheduled task changes	hook_id, by (agent, gadget, person) on _created
schedule_run	A scheduled task runs	hook_id, outcome (ok, error, rejected), duration_ms, delay_ms (how late it started)
screen_viewed / ui_action	The user navigates, or uses a menu that has no backend call	route (a route template, never a URL), previous_route, action (closed list)

Notes on the ones that are not obvious:

• tool_called is the expensive one. A long turn makes dozens of calls, and one send per call would hold the turn
  up. We collect the calls of a turn and send them in one PRODUCT_ANALYTICS.send([...]) when the turn ends, each
  stamped with the moment its call finished. We also give it no activity/v1 metric: a turn with many calls would write
  a lot of Analytics Engine points per invocation; we expected a busy turn to approach the per-invocation write cap, so we did not
  measure it; please check the current limit in Cloudflare's Analytics Engine documentation.
• schedule_run and the UI events go to the stream only, for the same reason (a task that runs every minute, or a
  page view, would flood Analytics Engine).
• approval_requested and approval_resolved share request_id. The id is only unique inside its workspace, so
  the join is gadget_id + request_id.
• agent_turn_finished after an approval: a turn resumed after an approval is a new turn, so it is another event.
• screen_viewed and ui_action are the largest piece and the least necessary. The client batches them and a
  single backend RPC records them, against a closed list of routes and actions defined in workshop-shared. We would
  drop this part first if the maintainers want a smaller scope.

We would also keep the envelope out of the proposal. Our deployment adds a tenant, a hashed person and a version to
each record; that is specific to how we host many tenants and has no place upstream.

The rule: never content

The stream is for behavior, not for what people write. We hold ourselves to this and would propose the same for any of
the events above:

• Every property is an id, a boolean, a number, or a value from a closed list (a union type in TypeScript). No event has
  a free-text field.
• Errors travel as a code or a class name (an HTTP status, an error's name), never as a message.
• Names that a model or a person can choose (a tool name, a skill name, a connector id) are only recorded when they
  match a known list or a strict pattern; otherwise the event goes without the name. A tool the model made up is
  recorded with no tool_name.
• Tool arguments and results, prompts, responses, file names, email addresses and note texts never appear.
• Most closed lists are enforced by the types: a route, a UI action or the key of a setting is a union of literals in
  ProductAnalyticsInput, so a value outside its list does not compile. A few are typed string and kept closed where
  they are emitted (a setting's value, the family of a failure site).
• The @ is enforced by a test: after every test in the analytics suites, a check reads all the records that were sent
  and fails if any of them contains an @ (an email, or something a person typed). The check does not know each
  property's list or pattern, so a new free-text property without an @ would pass it: that one is for review to catch.

If useful, we can contribute that check as its own PR before any new event, so the rule is in place first.

Analytics must never break the product

All of the emissions go through recordAnalytics, which already wraps the send in a try/catch and sends in
ctx.waitUntil. The call sites that run in the path of the action itself (deleting a workspace or a gadget and its
connection, saving an agent step with its request cards, retrying a blueprint publish) have a test that makes
PRODUCT_ANALYTICS.send throw, or the whole analytics call, and checks that the action still completes. The rest rely
on that try/catch.

How it could be split into small PRs

The guide asks for PRs of about a dozen lines. Each event needs a member in ProductAnalyticsInput, a case in
toActivityMetric (the switch is exhaustive), and one emission, so a simple event is roughly that size. Some will be
a bit over, agent_turn_finished first (it has the most properties and needs the duration and tool count tracked in
the turn), and we would ask for guidance on those rather than squeeze them. We would send them
one at a time, each with its test, in this order, and stop wherever the maintainers say stop:

1. agent_turn_finished (likely somewhat over a dozen lines): the type member, the metric case, and one call in the finally of the agent turn. Smallest
   and the most useful on its own.
2. error_shown: one call where the error message is posted to the chat.
3. approval_requested and approval_resolved: two members, two cases, two calls. They go together because the
   request id only means something with both.
4. connector_connect_started, connector_connected, connector_connect_failed: one PR for the connect flow and one
   for connector_disconnected. The failed event is the one with a subtlety (one failure per attempt, and an expiry that
   must not double-count), so it gets its own PR and its own description.
5. schedule_created, schedule_enabled, schedule_disabled, schedule_deleted: one PR per event or one for all
   four, whichever the maintainers prefer, since they are four lines each.
6. schedule_run: one call where the hook firing callback finishes.
7. tool_called last, since it is the only one that needs a helper: a small batch function so a turn sends once. We
   expect that to be over a dozen lines and would ask for guidance before writing it.
8. screen_viewed and ui_action only if there is interest. This one touches the frontend, workshop-shared and an
   RPC, so it is a different conversation.

Each PR would stay out of workshop-backend beyond the emission itself, and would not rename or change any existing
event.

Questions for the maintainers

1. Is this direction of interest at all, or is product analytics meant to stay as it is?
2. Would you rather have outcome and duration_ms added to the existing success events as well? We left them alone
   to keep the diffs small, but we are happy to try.
3. Naming: we used outcome, duration_ms, by and wait_ms, in English like the rest of the upstream, and for the
   connector connection_id / vendor where the existing events say gatekeeper_id / vendor_id. Tell us if you
   prefer the existing names or others.
4. Is "one event per PR" the right size, or would you prefer to review the turn and tool events together?

Thanks for reading.

[renezander030]

approval_resolved and tool_called don't share a key, so once an approved action runs and fails, or succeeds and turns out wrong, the stream can't say whether a person or a rule let it through. I'd put request_id and by on the tool_called that executes the approved action, which makes failure rate per approval path one join instead of a trace search.

Does a turn resumed after an approval know the request_id it resumes from, or would that have to come from the span?