feat(rivetkit): actor tracing, invocation metrics, and log correlation with otel

.claude/reference/testing.md

@@ -47,6 +47,7 @@ For RivetKit runtime or parity bugs, use `rivetkit-typescript/packages/rivetkit`
 
 - Keep RivetKit test fixtures scoped to the engine-only runtime.
 - Prefer targeted integration tests under `rivetkit-typescript/packages/rivetkit/tests/` over shared multi-driver matrices.
+- A span and its parent can arrive in different OTLP export batches, so a trace test that waits for the child and then asserts its `parentSpanId` is racy. Wait on a predicate over the whole exported span list until both are present, then assert the relationship.
 
 ## Frontend testing
 

CLAUDE.md

@@ -392,6 +392,7 @@ Load these only when the task touches the topic.
 - **[SQLite VFS parity](docs-internal/engine/sqlite-vfs.md)** — native Rust VFS ↔ WASM TypeScript VFS 1:1 parity rule, v2 storage keys, chunk layout, delete/truncate strategy. Read before touching either VFS.
 - **[SQLite optimizations](docs-internal/engine/SQLITE_OPTIMIZATIONS.md)** — brief tracker for SQLite cold-read, VFS, storage, preload, and benchmark optimization ideas.
 - **[TLS trust roots](docs-internal/engine/tls-trust-roots.md)** — rustls native+webpki union rationale, which clients use which backend.
+- **[RivetKit telemetry](docs-internal/engine/rivetkit-telemetry.md)** — Core-owned invocation and SQLite spans, ray semantics, schedule trace origins in `_rivet_meta`, native OTLP export. Read before touching actor tracing, metrics, or log correlation.
 - **[Sleep sequence](docs-internal/engine/sleep-sequence.md)** — engine lifecycle authority, `keepAwake` vs `waitUntil` semantics, grace deadline shutdown-token abort, `can_arm_sleep_timer` vs `can_finalize_sleep` predicates. Read before touching sleep/destroy lifecycle.
 
 ### Agent procedural (`.claude/reference/`)

docs-internal/engine/rivetkit-telemetry.md

@@ -0,0 +1,176 @@
+# RivetKit telemetry
+
+Internal reference for actor traces, invocation metrics, and log correlation. Core owns telemetry; runtime adapters activate its context in the host language. See [NAPI bridge](napi-bridge.md) for binding conventions and [Core internals](rivetkit-core-internals.md) for dispatch and lifecycle wiring.
+
+## Ownership
+
+- `rivetkit-core::telemetry` owns invocation spans, completion, and operation spans. `ActorMetrics` owns metric labels and recording.
+- `registry/napi-runtime.ts` resolves the current invocation and passes the active JavaScript span to Core. NAPI bindings only translate types.
+- TypeScript and Rust clients send ray and W3C trace headers. An actor-owned client also opens an outbound call span in Core.
+- Core exports its spans through the Rust OpenTelemetry (OTel) SDK. Applications export their own spans through their own SDK. Shared trace IDs and parent span IDs connect the two pipelines.
+
+## Spans
+
+All Core telemetry spans use the `rivetkit::telemetry` tracing target. Log layers exclude that target; the export layer excludes unrelated diagnostic spans.
+
+| Span | Kind | Parent or link |
+| --- | --- | --- |
+| `{actor}/{action}` | `server` | Incoming parent, otherwise root |
+| `{actor}/{action}` for a scheduled fire | `internal` | New root linked to the schedule's origin |
+| `{actor}/onRequest` | `server` | Incoming parent, otherwise root |
+| `{actor}/queue.send` for an external send | `producer` | Incoming parent, otherwise root |
+| `{callee}/{action}` for an actor-owned client call | `client` | Active application span, otherwise invocation |
+| `rivet.sqlite.{operation}` | `internal` | Active application span, otherwise invocation |
+| `{actor}/queue.receive` | `consumer` | Active application span, invocation, or root; linked to the message's origin |
+
+### Attributes
+
+| Scope | Attributes |
+| --- | --- |
+| Actor identity | `rivet.actor.id`, `rivet.actor.name`, `rivet.actor.key` |
+| Correlation | `rivet.ray.id` |
+| Invocation | `rivet.invocation.type`, `otel.status_code`, `error.type` on failure |
+| Action or scheduled invocation | `rivet.action.name` |
+| Raw HTTP request | `http.request.method`, `http.response.status_code` |
+| Queue send or receipt | `rivet.queue.name` |
+| SQLite operation | `rivet.operation.system`, `rivet.operation.name` |
+
+Raw request spans use the fixed name `onRequest`, not the request path. A 5xx response marks the invocation as failed with the status as `error.type`. A handler error records its `group.code`; Core generates the HTTP response after that invocation ends.
+
+SQLite and outbound actor-call spans are marked as errors with `actor.operation_abandoned` if tracking ends without a recorded result. This indicates an unknown outcome, not a confirmed operation failure, and is not returned to callers.
+
+`ActorMetrics::label_action_name` and `label_queue_name` replace undeclared names with `_OTHER`. Use these bounded names for span names and metric labels as well as queue attributes. This follows the OTel fallback convention for unknown caller-supplied values and prevents unbounded metric series.
+
+## Metrics and logs
+
+- `rivetkit_actor_invocations_total`
+- `rivetkit_actor_invocation_duration_seconds`
+
+Both metrics use actor name, action name, invocation type, and result labels. Invocation types are `action`, `scheduled`, `request`, and `queue_send`. Request and queue-send invocations use `onRequest` and `queue.send` as their action labels.
+
+Duration uses `MICRO_BUCKETS` to distinguish invocations shorter than the default Prometheus histogram's first bucket of 5 ms.
+
+Invocation loggers bind actor ID, name, key, and ray ID. Sampled invocation context also supplies `trace_id` and `span_id`. Actor fields follow existing TypeScript log casing; trace fields follow OTel correlation conventions. A Pino child logger retains the context it was created with; do not retain an action logger for later unrelated work.
+
+## Context propagation
+
+Core accepts `x-rivet-ray-id`, `traceparent`, and `tracestate` on actions, raw requests, and queue sends.
+
+A ray correlates related work across multiple traces and remains available when spans are sampled out. Trace and span IDs describe the individual traces and their parent relationships.
+
+- Rays accept 1–128 characters from `[A-Za-z0-9_-]`. Missing or invalid rays become UUIDs.
+- Invalid W3C trace context starts a root span without rejecting the request.
+- An actor-owned client uses the calling invocation's ray. External clients read `rivet.ray.id` from active OTel baggage. The Rust client's configured `ClientConfig::ray_id` is a fallback.
+- Outbound parent precedence is the call span, the active application span, then the invocation span.
+- TypeScript action calls, `handle.fetch()`, and queue sends propagate context. Explicit raw-request headers win. Setting either `traceparent` or `tracestate` preserves the caller's whole pair rather than mixing contexts.
+- Per-call context replaces static client telemetry headers. The Rust client reads context through `tracing-opentelemetry`; without that layer, no active W3C context is available.
+
+Rust shares names and validation through `rivetkit-client-protocol::telemetry_headers`. TypeScript defines header names in `common/actor-router-consts.ts` and baggage/trace handling in `common/otel-context.ts`.
+
+### JavaScript context
+
+Core cannot read the JavaScript span stack. `runWithActorInvocationContext` activates the invocation in `AsyncLocalStorage` and the OTel context manager while the callback runs. Core-bound operations then resolve that invocation and bind a more specific application span when present.
+
+```text
+Incoming request
+  → Core invocation span
+    → JavaScript handler with invocation context active
+      → Application span
+        → SQLite operation or outbound actor-call span
+```
+
+- Retained database, client, and schedule handles resolve the active invocation only when it belongs to the same Core actor generation; otherwise they retain their creation context. Core uses pointer identity through `Arc::ptr_eq` for this check.
+- The JavaScript context lookup runs inside invocations. The native binding call is skipped when the active span is absent or already matches the invocation span.
+- A transaction binds its parent at `beginTransaction`. Statements and commit retain that parent.
+- Queue sends, receipts, and `waitUntil` use the same invocation resolution. KV operations have no telemetry to attach.
+- `@opentelemetry/api` is a required dependency for reading outbound context and activating inbound context. RivetKit does not register a global context manager; the application owns it.
+
+Caller trace context provides correlation, not identity or authorization. Operators can strip the three incoming headers when they do not trust caller-selected correlation values.
+
+## Invocation lifecycle
+
+`ActorInvocation` owns the span, timer, metric labels, and exactly-once completion. Action, schedule, raw-request, and queue-send dispatch create it before enqueueing work.
+
+1. The adapter runs the callback with invocation context active.
+2. Sending the reply records the outcome and invocation metrics, and adds a `reply sent` event.
+3. The span ends after the last tracked `waitUntil` task settles. Deferred database work and logs retain invocation context.
+
+`c.keepAwake` work is awaited within the callback. Rejected enqueue attempts count as `result=error`. Unfinished invocations record `actor.dropped_reply`; scheduled work also stores that identity in `_rivet_schedule_history` for `cronHistory()`.
+
+Span kind derives from invocation type. Completion and metrics remain on the same path when tracing is disabled or sampled out.
+
+## Persisted origins
+
+Schedules and queue messages capture their sending or defining invocation's ray and W3C context. An active application span is the origin when one exists; otherwise the invocation span is used.
+
+Origins currently use versioned BARE values in `_rivet_meta`:
+
+| Origin | Key |
+| --- | --- |
+| Schedule | `schedule_trace_context:{event_id}` |
+| Queue message | `queue_trace_context:{message_id}` |
+
+Write and delete origins in the same batch as their owning rows. Schedule deletion that misses its row leaves the origin untouched. Malformed origins are logged and ignored without preventing delivery or execution.
+
+This storage is pending migration into columns on `_rivet_schedule_events` and `_rivet_queue`. It is an exception to the bootstrap-only role described in Core internals, not a precedent for new runtime metadata.
+
+### Schedules
+
+- Creation and redefinition capture the current origin. Recurring re-registration refreshes it even when cadence stays unchanged.
+- Each fire starts a new trace linked to the defining span and retains the defining ray. This avoids keeping one trace open for the lifetime of a recurring schedule.
+- Schedules defined outside an invocation, or before origin storage existed, have no origin; each fire gets a fresh ray.
+- Older runtimes ignore origin metadata. Redefining a schedule on an older runtime can leave a stale origin until a newer runtime redefines it. Schedule behavior is unaffected.
+
+### Queues
+
+- External sends create `queue.send` invocations. The span ends when the send is acknowledged or the sender's completion wait ends.
+- `next`, `nextBatch`, and `waitForNames` create one `queue.receive` span per message in Core. It ends when the message is handed back and links to its stored origin.
+- Inside an invocation, the receipt uses the receiving invocation's ray. From `run`, the receipt is a root span with the sender's ray.
+- Receipt spans do not cover subsequent application processing.
+
+## Export configuration
+
+Set OTel variables on the actor runner process:
+
+```sh
+OTEL_SERVICE_NAME=internal-api
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://collector:4318/v1/traces
+OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
+```
+
+Core's `telemetry::export` module is behind `native-runtime`. Hosts attach `export::layer()` to their subscriber and call `export::flush_best_effort()` during shutdown.
+
+- An OTLP endpoint enables native export unless standard SDK-disable or exporter controls disable it. Sampling, resources, service name, and batching use standard OTel environment variables.
+- `configured_protocol` reads the traces-specific protocol variable before the general OTLP variable. Supported values are `grpc`, `http/protobuf`, and `http/json`; the default is `http/protobuf`. Invalid values produce an error. Selection is explicit because the exporter builders use feature-dependent defaults rather than reading these variables.
+- Both `OTEL_EXPORTER_OTLP_HEADERS` and `OTEL_EXPORTER_OTLP_TRACES_HEADERS` are supported.
+- Batches export in the background. Clean shutdown attempts a bounded flush; a hard kill can lose queued spans. Export failures do not fail actor work.
+- The batch queue is bounded by `OTEL_BSP_MAX_QUEUE_SIZE`, default 2,048. Overflow drops spans. Sampling reduces exports but does not remove construction or propagation work. Export CPU contention can affect latency; measurements belong in benchmark artifacts.
+
+### SDK diagnostics
+
+The NAPI log bridge forwards `opentelemetry_sdk=warn` events to Pino, including `BatchSpanProcessor.SpanDroppingStarted`.
+
+- Keep `internal-logs` enabled on `opentelemetry` and `opentelemetry_sdk`.
+- Each registry replaces the sink because the previous callback's Node worker may have exited.
+- Unreference the sink's `ThreadsafeFunction` so it does not keep the event loop alive.
+- Older addons may lack `setTelemetryLogSink`. Log a warning rather than preventing actor startup.
+
+### Configuration failures
+
+| Symptom | Check |
+| --- | --- |
+| Application spans appear, Core spans do not | Configuring `NodeSDK` in code does not configure Rust export. Set the runner's OTel environment variables. |
+| Application spans start separate traces | Register an OTel context manager. `NodeSDK.start()` and `NodeTracerProvider.register()` do this; setting a bare tracer provider alone does not. |
+| Application and Core spans appear under different services | Configure `service.name` on a manually constructed JavaScript provider, or ensure its resource reads the environment. |
+
+## Data policy
+
+Record actor identity, bounded action/queue names, invocation type, HTTP method/status, ray/trace/span IDs, operation names, and error identity. Do not record action arguments/results, connection parameters, SQL text/bindings, actor state, arbitrary headers, or raw error messages.
+
+## Limitations
+
+- No dedicated spans for WebSocket handlers, lifecycle hooks, connection callbacks, KV, or actor-state operations.
+- WebSocket action messages carry no caller ray or trace context. Inspector actions do not inherit caller context.
+- Actor creation rays are not delivered to the actor runtime.
+- No Wasm host span export; the adapter retains rays but does not propagate invocation trace context.
+- No category opt-in API, custom sampling API, dedicated Effect integration, or `rivetkit/unstable/otel`. Standard OTel application context already works for outbound calls.