fix(langsmith): bound test bodies and retry on fresh worker to stop CI hangs

[DABH]

fix(langsmith): bound test bodies and retry on fresh worker to stop CI hangs

What was happening

Integration-test jobs intermittently froze inside the contrib/langsmith suite with the ava
progress counter stuck (e.g. … 28 tests, 3 failures heartbeating every 30s) until the job's
20-30m timeout killed the leg. Reproduces on main; observed on run 31517630441 attempt 1
(job 93866798620, linux-arm / Node 20) and previously on main run 30591851926.

The stuck test was always signal-child: emits a closed SignalChildWorkflow marker nested under the workflow run. From the hung job's raw log, the failure chain is:

1. client.workflow.execute(SignalChildWorkflow, …) never settles.
2. ava's 120s inactivity watchdog fires and interrupts the test process; the SDK Runtime's
   shutdownSignals handler swallows the SIGTERM and shuts the Worker down gracefully instead of
   exiting, so Worker.runUntil rejects with
   PromiseCompletionTimeoutError: Promise did not resolve within 0ms after Worker completed shutdown.
3. The still-pending GetWorkflowExecutionHistory long-poll retry then surfaces an uncaught
   Error: Channel has been shut down when the connection closes.
4. The ava worker process never exits; the suite counter freezes and the job burns its full
   timeout. One stalled test therefore taxes every PR with a 20-30 minute hang.

Root cause

A first-workflow-task delivery stall between the ephemeral dev server and the worker, triggered
by CPU contention at worker startup:

• The workflow (parent — or in one observed variant, the child) sits RUNNING with its first
  workflow task PENDING_WORKFLOW_TASK_STATE_SCHEDULED, attempt: 1, forever. History ends at
  WorkflowTaskScheduled. Normal-queue first workflow tasks have no schedule-to-start timeout,
  so nothing on the server ever times out or redelivers — the workflow can never make progress.
• At stall time, DescribeTaskQueue on the workflow task queue reported pollers: [] in the
  never-polled variant (the server had not received a single workflow-queue poll from the worker
  45s after the worker entered RUNNING), while a manual PollWorkflowTaskQueue from a bare
  client instantly (2-5 ms) received the stuck task — reproduced on every probed stall. The
  task was dispatchable all along; the backlog was healthy.
• sdk-core TRACE for a stalled worker shows zero workflow-subsystem activity between RUNNING
  and shutdown (no poll RPC bursts, no bridge iterations), while healthy iterations open poll
  streams within milliseconds of RUNNING.

So the loss is on the worker/poll-delivery side of the pair, not in the test logic, not in the
LangSmith plugin (the workflow never even started executing), and not in a specific server
version. Ruled out empirically (300+ iteration loops per configuration, all still stalling):
CLI 1.8.2/server 1.31.2, CLI 1.7.3/1.31.2, CLI 1.6.0/server 1.30.0, matching.enableMigration=false,
matching.useNewMatcher=false, and single-partition task queues.

This suite is uniquely exposed because every case boots a fresh Worker on a fresh task queue on a
fresh per-file dev server and immediately starts a workflow — dozens of cold-start races per run,
on the slowest CI legs (linux-arm/x64 Node 20 under full job load).

The exact sdk-core mechanism (why the non-sticky WFT poller can fail to issue polls from worker
startup under CPU starvation) still deserves an upstream look; this PR fixes the CI hang at the
test-harness layer, which is also where the hang-amplification lives.

The fix

withTracingWorker (used by every case in the suite) now:

• runs the body via worker.runUntil(async () => …) so the worker is polling before the body
  starts any workflow;
• bounds each body attempt at 30s; on a stall it terminates workflows leaked by the aborted
  attempt, rolls the run collector back to its pre-attempt snapshot, logs a warning (surfaced in
  the archived test log for CI diagnosability), and retries on a fresh worker + fresh task
  queue, up to 3 attempts total.

A stall now recovers in place; a triple-stall (never observed) fails visibly in bounded time
(3 x 30s < ava's 120s inactivity cap, so the watchdog/SIGTERM wedge path can no longer engage).

Reproduction and validation

Standalone repro (not committed): loop the exact signal-child scenario (fresh worker + fresh task
queue per iteration, maxCachedWorkflows: 2, 1s sticky timeout) under 6x CPU load.

• Before the fix: the loop stalled fatally in 7/7 runs, within 1-90 iterations each
  (~1-5% of iterations under load); every stall matched the CI signature (first WFT
  SCHEDULED/attempt 1 forever; manual poll wins the task instantly).

• Mitigated logic, same loop, same load: 66/66 iterations passed with 3 real stalls absorbed
  by the retry (including one double-stall that recovered on the third attempt).

• Real suite with the fix, 25 full-suite runs (17 of them under 6x CPU load, one with
  REUSE_V8_CONTEXT=false for matrix parity; note the langsmith tests do not read that variable):
  25/25 runs, 46/46 tests each. Three of those runs absorbed a real stall — two show the
  +30s wall-clock signature (53.8s / 53.2s vs the ~23s norm) and one has the retry captured in
  its archived log:

  withTracingWorker: Test body did not settle within 30000ms on task queue
  langsmith-test-cf030962-… (attempt 1/3); assuming the first-workflow-task
  delivery stall; retrying on a fresh worker
  …
  # tests 46
  # pass 46
  # fail 0
  

• 24/24 loaded runs of the signal-child test file alone.

Changelog

Test-infrastructure only — no user-facing change to @temporalio/langsmith (nothing under
src/ outside __tests__/ is touched), so no CHANGELOG.md entry; please apply the
skip-changelog label.

Update: retry budget recalibrated after first CI run (second commit)

The first CI run of this PR (31544937973) exercised the designed behavior and exposed a
calibration gap on the linux-arm Node 24 leg (the macos-arm Node 24 failure on that run was an
unrelated packages/test integration-update flake — its langsmith suite passed):

• readonly-determinism stalled on 2/3 attempts, recovered on the third — retry working as
  designed.
• signal-child stalled on 3/3 back-to-back attempts and exhausted the budget, failing fast
  with HarnessStallError (6m55s leg — fail-fast held, no hang).

Attempt forensics from the archived log rule out "30s too tight for slow arm runners": each fresh
worker reached RUNNING within ~280ms of the previous attempt's shutdown, sibling tests on the leg
ran at normal speed, and every stalled attempt showed zero activity for its entire 30s window
(attempt boundaries 23:08:56.499 → 23:09:26.505 → 23:09:26.784 → 23:09:56.786 → 23:09:57.052 →
23:10:27.054; each exactly 30.0s of silence). The stalls are correlated in time — the degraded
state persists across consecutive fresh workers for a window on the order of 90s+.

Recalibration: keep the 30s per-attempt bound (an unmitigated stall never self-recovers — waiting
longer per attempt is pure waste; only a fresh worker recovers), and raise the cap to 6
attempts, sampling a ~3 minute window — double the worst observed sequence. This is safe
against ava's 120s inactivity watchdog because ava debounces that timer on any stateChange record
carrying a testFile, which includes worker stdout/stderr chunks (ava 5.3.1 lib/fork.js tags
all records; lib/api.js debounces on them): the harness's per-retry warning guarantees output
every ≤30s while attempts continue, so the watchdog only fires for a genuine silent wedge — its
backstop role, unchanged. A persistent degradation now fails loudly at ~3 minutes instead of
hanging.

Recalibration validation (6x CPU load): 8/8 full-suite runs green (one absorbed stall, captured
in log), 16/16 signal-child-file runs green (one absorbed stall), and 51 harness-shaped loop
iterations absorbing five real stalls — every one manual-poll-verified (task dispatched to a
fresh poller in 2-23ms at stall time) and every one recovered on the second attempt.

Notes / residual risk

• The retry re-invokes the case body; bodies are effectively idempotent here (collector state is
  rolled back and leaked workflows are terminated first). One test file (test-replay-parenting)
  counts raw createRun calls via a wrapper outside the collector; a stalled-then-retried run of
  that specific case could overcount and fail visibly — a bounded, diagnosable failure rather
  than a 30m hang, and only after an already-rare stall.
• The underlying first-WFT delivery stall is worth an sdk-core investigation (worker issues no
  normal-queue polls from startup under CPU starvation; evidence above). Filing that upstream is
  follow-up work; this PR removes its ability to freeze CI.