Skip to content

fix(session-recorder): prevent unbounded error loop from corrupted TextDecoder in emit - #305

Open
NickSolante wants to merge 2 commits into
hyperdxio:mainfrom
NickSolante:fix/session-recorder-decoder-crash-loop
Open

fix(session-recorder): prevent unbounded error loop from corrupted TextDecoder in emit#305
NickSolante wants to merge 2 commits into
hyperdxio:mainfrom
NickSolante:fix/session-recorder-decoder-crash-loop

Conversation

@NickSolante

@NickSolante NickSolante commented Aug 11, 2026

Copy link
Copy Markdown

Fixes #303

Problem

On Safari, the session recorder's emit() could enter a permanent, unbounded error loop: WebKit's TextDecoder can permanently corrupt after high cumulative decode volume and then throw RangeError: Bad value on valid input (WebKit #286266, wasm-bindgen report). emit() reused one module-level decoder for the page lifetime with no error handling, so a single corrupted instance turned every rrweb mutation flush into an uncaught error (~40/sec, observed 64,606 errors over 26 minutes from one session) until page unload. Full telemetry in #303.

Changes

  • Single-chunk events (the overwhelmingly common case) skip the encode→decode round-trip entirely the JSON string is already in hand, so most events never touch TextDecoder. This also removes the cumulative decode volume that ages the decoder toward WebKit's corruption threshold.
  • Multi-chunk events use a fresh per-event TextDecoder with { stream: true }, which also fixes a latent all-browser bug: byte-slice boundaries that bisected a multi-byte UTF-8 character previously produced U+FFFD replacement characters at chunk seams, silently corrupting reassembled replay JSON.
  • A failed decode is retried once with a new decoder, restarting from chunk 0 so streaming state stays consistent.
  • Circuit breaker: emit() failures are caught (each dropped event is surfaced via console.error), and after 10 consecutive failures the recorder stops instead of erroring on every event forever. init() resets the breaker state so a later recording lifetime starts clean.
  • Chunks are all converted before any is sent, so a mid-loop failure can't export a partial, unreassemblable chunk set.

Tests

  • splitIntoChunks unit tests: single-chunk identity (no decode), lossless multi-chunk reassembly, a multi-byte character bisected at the chunk boundary (fails against the old non-streaming decode), and recovery via the fresh-decoder retry (first decoder instance always throws RangeError: Bad value).
  • emit() error-handling tests (mocked rrweb/exporter/tracer): failures don't throw into rrweb dispatch, the breaker trips on the 10th consecutive failure and stops recording, nothing is emitted after tripping, and recording works again after a re-init.

compile:tsc clean, jest 10/10 passing.

Notes

  • Includes a changeset (patch bump for @hyperdx/otel-web-session-recorder).
  • Pre-existing and untouched here: MutationRateLimiter's constructor starts a setInterval that is never cleared (it also survives deinit()); the emit tests use fake timers to work around it. Happy to fix in a follow-up.

@changeset-bot

changeset-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 245f970

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@hyperdx/otel-web-session-recorder Patch
@hyperdx/browser Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the external Opened by an external contributor label Aug 11, 2026
@greptile-apps

greptile-apps Bot commented Aug 11, 2026

Copy link
Copy Markdown

Greptile Summary

The PR prevents repeated session-recorder failures caused by corrupted TextDecoder instances and preserves UTF-8 data across chunk boundaries.

  • Bypasses encoding and decoding for single-chunk events.
  • Uses fresh streaming decoders with one retry for multi-chunk events.
  • Adds an emit-failure circuit breaker that stops recording after repeated failures and resets on initialization.
  • Converts complete chunk sets before export and adds focused regression tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
packages/session-recorder/src/index.ts Adds guarded event processing, complete-set conversion, diagnostic reporting, and recorder teardown after repeated emit failures.
packages/session-recorder/src/sessionrecording-utils.ts Adds a single-chunk fast path and lossless streaming UTF-8 chunk decoding with a fresh-decoder retry.
packages/session-recorder/tests/emit.test.ts Covers swallowed emit failures, breaker activation, post-breaker suppression, and successful reinitialization.
packages/session-recorder/tests/sessionrecording-utils.test.ts Covers chunk identity, lossless reassembly, decoder retry, and multibyte boundary handling.
.changeset/session-recorder-decoder-crash-loop.md Documents the recorder fix and declares the package patch release.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[rrweb emits event] --> B[Serialize and enforce size limit]
    B --> C{Fits one chunk?}
    C -->|Yes| D[Use serialized string directly]
    C -->|No| E[Encode to UTF-8 bytes]
    E --> F[Decode chunks with fresh streaming decoder]
    F -->|Decode fails| G[Retry from chunk zero with new decoder]
    D --> H[Convert all chunks to logs]
    G --> H
    F --> H
    H --> I[Export complete chunk set]
    B -->|Failure| J[Increment consecutive failures]
    G -->|Retry fails| J
    J --> K{Ten failures?}
    K -->|No| A
    K -->|Yes| L[Stop recorder]
Loading

Reviews (4): Last reviewed commit: "Merge branch 'main' into fix/session-rec..." | Re-trigger Greptile

Comment thread packages/session-recorder/src/index.ts
@NickSolante
NickSolante force-pushed the fix/session-recorder-decoder-crash-loop branch from cb52d8d to c43ff08 Compare August 11, 2026 01:44
…xtDecoder in emit

WebKit's TextDecoder can permanently corrupt after high cumulative decode
volume and then throw `RangeError: Bad value` on valid input
(https://bugs.webkit.org/show_bug.cgi?id=286266). Because emit() reused one
module-level decoder with no error handling, a single corrupted instance
turned every rrweb flush into an uncaught error (~40/sec) until page unload.

- Skip the encode/decode round-trip entirely for single-chunk events,
  with a length*3 fast path that avoids encoding on the hot path at all
- Use a fresh per-event TextDecoder for multi-chunk events, with
  { stream: true } so chunk boundaries no longer bisect multi-byte
  characters into U+FFFD
- Retry a failed decode once with a new decoder
- Stop the recorder after 10 consecutive emit failures instead of
  erroring on every event forever

Circuit-breaker hardening (from review):
- init() resets paused/consecutiveEmitFailures so a re-init after a
  breaker trip records again with the full failure tolerance
- Every emit failure is surfaced via diag.error, not just in debug
- All chunks are converted before any is sent, so a mid-loop failure
  can't export a partial (unreassemblable) chunk set
- The breaker's deinit() can't throw back into rrweb, deinit() clears
  inited before calling stop, and init() finishes teardown if the trip
  happened during rrweb's synchronous initial snapshot
- emit tests are now self-contained per it-block

Fixes hyperdxio#303
@NickSolante
NickSolante force-pushed the fix/session-recorder-decoder-crash-loop branch from c43ff08 to ac43184 Compare August 11, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external Opened by an external contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session recorder: unbounded RangeError: Bad value crash loop on Safari shared TextDecoder reused in emit() with no error handling

2 participants