Skip to content

fix(utils): reject worker connections and calls on errors - #284

Open
darkdi wants to merge 2 commits into
apple:mainfrom
darkdi:fix/worker-error-rejection
Open

darkdi wants to merge 2 commits into
apple:mainfrom
darkdi:fix/worker-error-rejection

Conversation

@darkdi

@darkdi darkdi commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Worker startup errors leave connectWorker() pending, so the embedding connection cache cannot retry. Later uncaught errors also leave outstanding RPCs pending.

Reject the handshake and in-flight calls on worker errors, and release their callbacks. Keep future RPCs available: browser workers can survive uncaught errors.

Validation: TypeScript utils build, 127 utils tests, 144 viewer tests and Prettier pass. A real Chromium worker rejects startup/in-flight failures and then resumes RPC, existing-proxy and callback calls after a timer error.

AI-assisted change and validation.

A worker startup failure leaves the ready handshake pending, and later worker errors strand outstanding RPC calls. Propagate these errors, clear pending callbacks and reject further sends on the failed connection. Cover startup failures and in-flight crashes using real worker threads.

Signed-off-by: Dmitry Rantovov <rantovov5@gmail.com>
@darkdi

darkdi commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

The CI workflow is awaiting maintainer approval before any jobs can start. Could a maintainer approve this run? Local validation is in the PR description.

@Manishram-ai Manishram-ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built packages/utils (126 tests pass) and tried it in Chrome with a real Web Worker. A browser worker keeps running after an uncaught error, so one stray throw in a timer makes every later connection.call() reject, while the same page keeps working on main. The viewer would stay stuck until a reload, because connect() in packages/viewer/src/embedding/index.ts holds on to the connection once the handshake has resolved. With the two suggestions applied I get 127 passed and the Chrome page keeps working.

Comment thread packages/utils/src/worker_helper.ts Outdated
Comment on lines +93 to +105
const callbacks = new Map<string, (...args: any[]) => any>();
let failure: Error | undefined;

worker.onerror = (event: ErrorEvent) => {
failure = event.error instanceof Error ? event.error : new Error(event.message || "Worker failed");
reject(failure);
for (const call of pending.values()) call.reject(failure);
pending.clear();
callbacks.clear();
};

function send(msg: Record<string, any>, callArgs?: any[]): Promise<any> {
if (failure) return Promise.reject(failure);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A startup error never reaches send, because connection isn't handed out until the handshake finishes. So this check only runs after the handshake, when a browser worker is still alive. Dropping it keeps the new rejection for the handshake and for in-flight calls.

Suggested change
const callbacks = new Map<string, (...args: any[]) => any>();
let failure: Error | undefined;
worker.onerror = (event: ErrorEvent) => {
failure = event.error instanceof Error ? event.error : new Error(event.message || "Worker failed");
reject(failure);
for (const call of pending.values()) call.reject(failure);
pending.clear();
callbacks.clear();
};
function send(msg: Record<string, any>, callArgs?: any[]): Promise<any> {
if (failure) return Promise.reject(failure);
const callbacks = new Map<string, (...args: any[]) => any>();
worker.onerror = (event: ErrorEvent) => {
// An uncaught error doesn't stop a browser worker, so later calls can still go through.
const failure = event.error instanceof Error ? event.error : new Error(event.message || "Worker failed");
reject(failure);
for (const call of pending.values()) call.reject(failure);
pending.clear();
callbacks.clear();
};
function send(msg: Record<string, any>, callArgs?: any[]): Promise<any> {

Comment on lines +59 to +74
it("rejects pending and subsequent calls when the worker crashes", async () => {
const worker = startWorker(`
const runtime = createWorkerRuntime();
globalThis.onmessage = runtime.handler;
runtime.registerFunction("wait", () => new Promise(() => {}));
runtime.registerFunction("crash", () => new Promise(() => {
setImmediate(() => { throw new Error("runtime failed"); });
}));
`);
const connection = await connectWorker(worker);
await Promise.all([
expect(connection.call("wait")).rejects.toThrow("runtime failed"),
expect(connection.call("crash")).rejects.toThrow("runtime failed"),
]);
await expect(connection.call("wait")).rejects.toThrow("runtime failed");
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last assertion needs the failure check, and without it the call times out on the dead Node thread, so I took it out. The new test fires onerror by hand because a Node thread can't survive an uncaught error, and it fails on the current code.

Suggested change
it("rejects pending and subsequent calls when the worker crashes", async () => {
const worker = startWorker(`
const runtime = createWorkerRuntime();
globalThis.onmessage = runtime.handler;
runtime.registerFunction("wait", () => new Promise(() => {}));
runtime.registerFunction("crash", () => new Promise(() => {
setImmediate(() => { throw new Error("runtime failed"); });
}));
`);
const connection = await connectWorker(worker);
await Promise.all([
expect(connection.call("wait")).rejects.toThrow("runtime failed"),
expect(connection.call("crash")).rejects.toThrow("runtime failed"),
]);
await expect(connection.call("wait")).rejects.toThrow("runtime failed");
});
it("rejects pending calls when the worker crashes", async () => {
const worker = startWorker(`
const runtime = createWorkerRuntime();
globalThis.onmessage = runtime.handler;
runtime.registerFunction("wait", () => new Promise(() => {}));
runtime.registerFunction("crash", () => new Promise(() => {
setImmediate(() => { throw new Error("runtime failed"); });
}));
`);
const connection = await connectWorker(worker);
await Promise.all([
expect(connection.call("wait")).rejects.toThrow("runtime failed"),
expect(connection.call("crash")).rejects.toThrow("runtime failed"),
]);
});
it("still accepts calls after an error the worker survives", async () => {
const worker = startWorker(`
const runtime = createWorkerRuntime();
globalThis.onmessage = runtime.handler;
runtime.registerFunction("sum", (a, b) => a + b);
`);
const connection = await connectWorker(worker);
// A Node thread dies on an uncaught error but a browser worker keeps running, so fire the event by hand.
worker.onerror({ message: "stray error" });
await expect(connection.call("sum", 2, 3)).resolves.toBe(5);
});

Signed-off-by: Dmitry Rantovov <rantovov5@gmail.com>
@darkdi

darkdi commented Nov 10, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in 4f7ae93. Confirmed the recovery regression with a real Chromium worker before changing it; RPCs, existing proxies and callbacks now work after an uncaught timer error. Startup/in-flight rejection remains covered. All 127 utils and 144 viewer tests pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants