chore(from-transcript): graduate the #779 spike into a runnable fromTranscript helper with a live e2e - #824
Open
drewdrewthis wants to merge 1 commit into
Open
Conversation
WalkthroughChangesTranscript replay and seeded scenario proofs
Sequence Diagram(s)sequenceDiagram
participant Proof
participant fromTranscript
participant Scenario
participant LiveAgent
participant Judge
Proof->>fromTranscript: Load transcript and fork options
fromTranscript->>Scenario: Create seeded scenario
Scenario->>LiveAgent: Run with seeded history
LiveAgent-->>Scenario: Return assistant response
Scenario->>Judge: Evaluate recall criteria
Judge-->>Proof: Return verdict
Possibly related issues
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (2)
javascript/examples/vitest/from-transcript/reader.test.ts (1)
1-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument how this test suite is run.
Add the exact Vitest command, required dependencies, no-network behavior, and expected coverage scope.
As per coding guidelines, TypeScript tests must document running instructions, coverage requirements, dependencies, and examples.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@javascript/examples/vitest/from-transcript/reader.test.ts` around lines 1 - 9, Document the execution contract for the transcript reader tests in the test file’s header: include the exact Vitest command, required dependencies, explicit no-network behavior, and the expected coverage scope. Keep the existing fixture and mutation-test context, and add a concise usage example if needed to satisfy the TypeScript test documentation guidelines.Source: Coding guidelines
javascript/examples/vitest/from-transcript/cc-transcript.ts (1)
31-51: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftComplete the JSDoc contract for the new exported API.
Document parameters/properties, return values, thrown errors, limitations, and a usage example.
javascript/examples/vitest/from-transcript/cc-transcript.ts#L31-L51: document raw shapes and supported content blocks.javascript/examples/vitest/from-transcript/cc-transcript.ts#L57-L124: document parser/traversal errors and fork semantics.javascript/examples/vitest/from-transcript/cc-transcript.ts#L130-L142: document normalized-turn invariants.javascript/examples/vitest/from-transcript/cc-transcript.ts#L169-L238: document unsupported content and normalization behavior.javascript/examples/vitest/from-transcript/cc-transcript.ts#L244-L255: document defaults and fidelity trade-offs.javascript/examples/vitest/from-transcript/cc-transcript.ts#L259-L316: add emission examples and error conditions.javascript/examples/vitest/from-transcript/from-transcript.ts#L23-L58: document option and result semantics.javascript/examples/vitest/from-transcript/from-transcript.ts#L134-L187: add a complete seeded-scenario example.As per coding guidelines, all public TypeScript APIs and interfaces require complete JSDoc and usage examples.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@javascript/examples/vitest/from-transcript/cc-transcript.ts` around lines 31 - 51, Complete the JSDoc for all public APIs across javascript/examples/vitest/from-transcript/cc-transcript.ts (31-51, 57-124, 130-142, 169-238, 244-255, 259-316) and javascript/examples/vitest/from-transcript/from-transcript.ts (23-58, 134-187): document raw shapes and supported AnthropicBlock variants, parameters, normalized-turn invariants, parser/traversal and emission errors, fork semantics, unsupported-content handling, defaults, fidelity trade-offs, option/result behavior, and complete usage examples including a seeded scenario. Ensure every exported interface, type, function, and related API has accurate JSDoc without changing runtime behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@javascript/examples/vitest/from-transcript/cc-transcript.ts`:
- Around line 191-208: Replace all `any` escape hatches with explicit
discriminated types and reusable content-part type guards. In
javascript/examples/vitest/from-transcript/cc-transcript.ts:191-208, define or
reuse an internal message-ID type for `_msgId`; in
javascript/examples/vitest/from-transcript/cc-transcript.ts:295-312, type
assistant content parts explicitly. Update `messageText()` in
javascript/examples/vitest/from-transcript/from-transcript.ts:84-95 and
tool-part filtering in :101-130 to use those predicates. Narrow emitted
roles/parts in javascript/examples/vitest/from-transcript/reader.test.ts:89-112,
declare the fixture as `RawNode[]` in :156-165, and reuse the typed tool-part
predicates in :175-186; ensure no TypeScript `any` remains.
- Around line 114-121: Update blockText and all call sites handling classified
human turns so non-text image/file blocks are not silently omitted. Convert
supported blocks into corresponding AI-SDK content parts, or explicitly reject
unsupported blocks with a clear error; preserve the existing text handling and
ensure image-only prompts cannot become empty user messages.
- Around line 57-69: Update parseSessionFile to reject malformed JSON lines
instead of filtering them out, reporting the offending line. Update linearize to
validate every parent reference, reject missing parents, duplicate UUIDs, and
cycles with errors identifying the relevant UUID, and never return a partial
transcript when validation fails.
In `@javascript/examples/vitest/from-transcript/fixtures/real-cc-session.jsonl`:
- Around line 3-8: Sanitize the live-session JSONL fixture by replacing internal
prompts, tool inventories, workspace paths, session identifiers, timestamps,
memory content, and reasoning signatures with synthetic values while preserving
each record’s JSONL shape and parentUuid/tool relationships. Update the affected
entries throughout the fixture, including the referenced ranges, and ensure no
real operational or sensitive transcript data remains.
In `@javascript/examples/vitest/from-transcript/from-transcript.ts`:
- Line 47: Update parseSessionFile and the stats construction so rawLines
reflects the actual number of nonblank input JSONL lines, counted before
malformed or UUID-less records are filtered; alternatively rename the statistic
to parsedNodes if it is intended to report retained records, and apply the same
correction at the referenced reporting and assertion sites.
In `@javascript/examples/vitest/from-transcript/model.ts`:
- Around line 13-31: Validate the normalized SC779_PROVIDER value before model
selection in e2eModel and the related E2E_MODEL_ID configuration. Accept only
the supported OpenAI and Gemini/Google values, rejecting unsupported or
whitespace-containing values with a clear error instead of falling back to
OpenAI; preserve the existing credential and model behavior for valid providers.
In `@javascript/examples/vitest/from-transcript/proof.e2e.test.ts`:
- Around line 67-74: Replace the explicit any types in printVerdict and the
additional affected code around the proof result handling with the existing
result contract, using unknown only where values are genuinely untyped. Preserve
the current field access and logging behavior while ensuring TypeScript
validates success, metCriteria, unmetCriteria, and reasoning through the proper
interface.
In `@javascript/examples/vitest/from-transcript/reader.test.ts`:
- Around line 175-189: Update the test “structured-emit dropMatching leaves NO
orphaned tool parts (provider-valid seed)” to assert both pairing directions:
retain the existing check that every tool-result ID has a matching tool-call ID,
and add a check that every collected tool-call ID has a corresponding
tool-result ID.
---
Nitpick comments:
In `@javascript/examples/vitest/from-transcript/cc-transcript.ts`:
- Around line 31-51: Complete the JSDoc for all public APIs across
javascript/examples/vitest/from-transcript/cc-transcript.ts (31-51, 57-124,
130-142, 169-238, 244-255, 259-316) and
javascript/examples/vitest/from-transcript/from-transcript.ts (23-58, 134-187):
document raw shapes and supported AnthropicBlock variants, parameters,
normalized-turn invariants, parser/traversal and emission errors, fork
semantics, unsupported-content handling, defaults, fidelity trade-offs,
option/result behavior, and complete usage examples including a seeded scenario.
Ensure every exported interface, type, function, and related API has accurate
JSDoc without changing runtime behavior.
In `@javascript/examples/vitest/from-transcript/reader.test.ts`:
- Around line 1-9: Document the execution contract for the transcript reader
tests in the test file’s header: include the exact Vitest command, required
dependencies, explicit no-network behavior, and the expected coverage scope.
Keep the existing fixture and mutation-test context, and add a concise usage
example if needed to satisfy the TypeScript test documentation guidelines.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e3de4f90-76b8-419b-bbeb-e0e2c15e887e
⛔ Files ignored due to path filters (1)
javascript/examples/vitest/from-transcript/evidence/e2e-run.logis excluded by!**/*.log
📒 Files selected for processing (7)
javascript/examples/vitest/from-transcript/README.mdjavascript/examples/vitest/from-transcript/cc-transcript.tsjavascript/examples/vitest/from-transcript/fixtures/real-cc-session.jsonljavascript/examples/vitest/from-transcript/from-transcript.tsjavascript/examples/vitest/from-transcript/model.tsjavascript/examples/vitest/from-transcript/proof.e2e.test.tsjavascript/examples/vitest/from-transcript/reader.test.ts
drewdrewthis
force-pushed
the
issue779/transcript-to-scenario-helper
branch
from
July 18, 2026 15:17
27e5cd5 to
a304fb0
Compare
Collaborator
Author
Review verdict: READYReviewed at: No blocking concerns remain at this SHA. All blocking findings the review surfaced were fixed in this HEAD (below); no unresolved review threads. Resolved by review (found blocking, fixed at
|
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@javascript/examples/vitest/from-transcript/from-transcript.ts`:
- Around line 28-31: Add JSDoc comments to the forkAt and includeThinking
properties in FromTranscriptOptions, describing each option’s purpose and
explicitly stating its default behavior. Preserve the existing documentation for
the tool I/O option and do not alter the interface types or runtime behavior.
- Line 136: Document the public fromTranscript function with JSDoc covering its
parameters, return type, and usage example. Explicitly describe errors for an
unfound forkAt UUID and an out-of-range index, using the existing function
behavior and option types without changing implementation logic.
In `@javascript/examples/vitest/from-transcript/README.md`:
- Around line 1-2: Add a contribution-guidelines section to the README for the
fromTranscript scenario, linking to the project’s existing main contribution
guidelines rather than duplicating them. Keep the current title and description
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9181cd6a-23d5-48c8-bfb3-c91ba80f7310
⛔ Files ignored due to path filters (1)
javascript/examples/vitest/from-transcript/evidence/e2e-run.logis excluded by!**/*.log
📒 Files selected for processing (7)
javascript/examples/vitest/from-transcript/README.mdjavascript/examples/vitest/from-transcript/cc-transcript.tsjavascript/examples/vitest/from-transcript/fixtures/real-cc-session.jsonljavascript/examples/vitest/from-transcript/from-transcript.tsjavascript/examples/vitest/from-transcript/model.tsjavascript/examples/vitest/from-transcript/proof.e2e.test.tsjavascript/examples/vitest/from-transcript/reader.test.ts
🚧 Files skipped from review as they are similar to previous changes (4)
- javascript/examples/vitest/from-transcript/proof.e2e.test.ts
- javascript/examples/vitest/from-transcript/model.ts
- javascript/examples/vitest/from-transcript/cc-transcript.ts
- javascript/examples/vitest/from-transcript/reader.test.ts
…ranscript helper with a live e2e Turns a real Claude Code session .jsonl into a runnable Scenario (Strategy B, JSONL-only): walk the parentUuid tree, normalize (real-user vs tool_result; reassemble fragmented assistant turns; Anthropic to AI-SDK), seed via the turn-free addMessage side-door (dodging the max_turns budget), then a live agent() plus judge() replay the fork. Proven end-to-end with a live run (run id in the PR body); 20 mutation-proven unit tests plus a 3-test live e2e gated behind SC779_LIVE. The committed fixture is a PII-scrubbed real CC session (structure + KUMQUAT77 flow intact; tool/attachment/injected bodies elided). Graduates draft #789. No src/ change (example-space helper only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PW4HxMfgLxY8dmGK4AimfG
drewdrewthis
force-pushed
the
issue779/transcript-to-scenario-helper
branch
from
July 18, 2026 15:27
a304fb0 to
64eed88
Compare
Collaborator
Author
CodeRabbit findings — disposition (HEAD
|
langwatch-agent
requested review from
rogeriochaves
and removed request for
rogeriochaves
July 30, 2026 06:35
langwatch-agent
added
the
hound-checked
Contributor
|
Automated low-risk assessment This PR was evaluated against the repository's Low-Risk Pull Requests procedure and does not qualify as low risk.
This PR requires a manual review before merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Graduates the #779 spike (draft #789) into a working, tested
fromTranscripthelper that turns a real Claude Code session.jsonlinto a runnable Scenario — proven end-to-end with a livescenario.run(...)(run id below), not a mock.What & why
When a real agent makes a real mistake, there was no path from "here is the session where it went wrong" to "here is a Scenario that reproduces it." This helper is that path for the Claude Code adapter (scope-locked to the native JSONL per the owner steer on #779; the from-traces Strategy C and full-sandbox Strategy D stay deferred in #779, out of scope here).
javascript/examples/vitest/from-transcript/:cc-transcript.ts— reader/adapter: walks theparentUuidtree leaf→root (sidechain-aware, cycle-guarded), distinguishes a real user turn from atool_resultcarried asrole:user, re-merges an assistant message fragmented acrossthinking/text/tool_uselines, recovers eachtool_result's tool name by pairing back to itstool_use, and emits AI-SDK v6ModelMessage[](withflattenToolsfor cross-model seeds).from-transcript.ts—fromTranscript(path, { forkAt })→ aSeededScenariowhoseseedStepbulk-seeds the captured history through the turn-freestate.addMessage()side-door, so a seed of any length costs zeromaxTurnsbudget (the [SPIKE] Create a runnable Scenario from a real agent transcript — Claude Code failure-replay as first use case #779 gotcha). SDK-free, so the reader stays unit-testable.reader.test.ts(20 unit tests, no API) ·proof.e2e.test.ts(live e2e, gated behindSC779_LIVE) ·model.ts(provider selector: OpenAI default;SC779_PROVIDER=geminifor the run below).Reuse-first: the existing run loop +
message()injection +ScenarioExecutionState.addMessageare used as-is — nosrc// published-SDK change, so there is no structural SDK surface to declare as drift. (The spike'sbuildScenarioFromTranscriptis renamed tofromTranscriptto match the name #779 uses.)How I can prove I was successful
Real transcript → runnable Scenario that ACTUALLY RAN (live models, not mocked). Full log committed at
javascript/examples/vitest/from-transcript/evidence/e2e-run.log.scenariobatch_3GgLkgfhAdLZMEmBI4kwGaeneKdfixtures/real-cc-session.jsonl(theKUMQUAT77memory-recall session;KUMQUAT77is a synthetic test token that grants nothing).parentUuidchain → 20 normalized turns (2 human / 2 injected / 7 tool / 9 assistant) → a 12-message seed; a real User-role turn ("What was the exact token…") sits at the fork; the liveagent()then took the next turn and ajudge()rendered a verdict.pnpm exec vitest: 3 passed (3) in ~36s. (The scripted path is seed → single liveagent()turn →judge()— no user-simulator loop; Proof B proves the judge discriminates, so this is a real verdict, not a rubber-stamp, but not a multi-turn interactive conversation.)SC779_PROVIDER=gemini); the committed test defaults to OpenAIgpt-5-mini(the spike-validated path), env-switchable viamodel.ts..jsonlend-to-end → runnable Scenario"KUMQUAT77", judge success=truescenario.run), run id capturedscenariobatch_3GgLkgfhAdLZMEmBI4kwGaeneKd+ LangWatch URL abovemax_turnsseed-budget gotcha handledmessage()×N +proceed()→"Reached maximum turns (4)"; the side-door seed under the same budget → success=truereader.test.ts20/20; mutation check — removing theparentUuidreverse turns 5 reader tests red, reverting → 20/20 greeneslint from-transcript/exit 0, 0 eslint-disables;tsc --noEmit0 errors in these filesNo UI surface. Backend-only CLI/SDK helper; its observable output is the terminal run above (committed as
evidence/e2e-run.log) + the LangWatch run. Nothing to screenshot.Human verification
Then point it at your own session:
fromTranscript("~/.claude/projects/<slug>/<session>.jsonl", { forkAt: { beforeLastAssistant: true } })and dropseedStepintoscenario.run({ script: [seed.seedStep, scenario.agent(), scenario.judge({ criteria })] }). Seefrom-transcript/README.md.CI note
CI's
Test (Examples)step is red for an environmental reason, not a defect in this PR. The shared OpenAI account is out of quota (insufficient_quota), so every live-OpenAI example fails right now — including pre-existingtests/voice/*andweather-agent, not just this PR; main's last green run predates the exhaustion. This PR's unit tests pass in CI (✓ from-transcript/reader.test.ts), and its live e2e is gated behindSC779_LIVE(skipped by default) so it does not add to the red — it is proven by the committed run receipt above.Scope / notes
MEMORY.md+ infra details; structure, tool names, thinking, and the KUMQUAT77 flow are preserved, all sensitive bodies elided.🤖 Generated with Claude Code
https://claude.ai/code/session_01PW4HxMfgLxY8dmGK4AimfG