Skip to content

Export Workers user spans over OTLP - #7686

Draft
danlapid wants to merge 3 commits into
mainfrom
dlapid/user-tracing-otlp
Draft

danlapid wants to merge 3 commits into
mainfrom
dlapid/user-tracing-otlp

Conversation

@danlapid

Copy link
Copy Markdown
Collaborator

Workers user spans reach a tracing backend today only through Streaming Tail Workers: the Workers Observability tail worker rebuilds them in JavaScript from tail events. Spans built that way are cut off when a Durable Object hands off work, carry coarse I/O-clock timestamps, and are sampled only after they already exist.

This gives the runtime what it needs to export user spans itself, as OTLP, the way other Cloudflare services already do:

  • an OTLP encoder for spans, written in Rust, behind the existing span submitter interface;
  • a tracer hook that lets the embedder open each invocation's root span, so the invocation and all of its user spans form one exported trace;
  • precise timestamps for user spans, which no longer reach user code directly.

workerd can also export on its own: a new tracing.otlp setting sends each invocation's spans to an OTLP/HTTP collector such as a local Jaeger. That makes Workers traces visible in local development without a tail worker.

Nothing changes unless an embedder installs the hook or the setting is configured.

danlapid and others added 3 commits October 10, 2026 07:00
Workers user spans reach a tracing backend today only through
Streaming Tail Workers: the Workers Observability tail worker rebuilds
them in JavaScript from tail events. Spans built that way are cut off
when a Durable Object hands off work, carry coarse I/O-clock
timestamps, and are sampled only after they already exist.

This gives the runtime what it needs to export user spans itself, as
OTLP, the way other Cloudflare services already do:

- an OTLP encoder for spans, written in Rust, behind the existing
  span submitter interface;
- a tracer hook that lets the embedder open each invocation's root
  span, so the invocation and all of its user spans form one exported
  trace;
- precise timestamps for user spans, which no longer reach user code
  directly.

workerd can also export on its own: a new `tracing.otlp` setting sends
each invocation's spans to an OTLP/HTTP collector such as a local
Jaeger. That makes Workers traces visible in local development without
a tail worker.

Nothing changes unless an embedder installs the hook or the setting is
configured.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The filter is a stopgap copied from the Workers Observability tail
worker. Record why it is unsatisfying and that it can be removed once
every binding talks to its backend over JSRPC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some bindings (D1, AI, Images, Vectorize) reach their backend with an
internal fetch to a placeholder host. Those fetch spans were dropped
when they ended, but by then their id had already been handed to the
backend as its parent, so any spans the backend exported were orphaned
at the top of the trace.

Decide when the fetch starts instead: a fetch to one of these hosts
opens no user span, and the binding's own span (such as d1_run) becomes
the subrequest's parent. Matching by host is still a stopgap until
every binding talks to its backend over JSRPC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant