Skip to content

Replace Duktape 1.0.2 with QuickJS 2026-06-04 (ES2025) - #221

Closed
evgeny-boger wants to merge 18 commits into
masterfrom
quickjs-port
Closed

Replace Duktape 1.0.2 with QuickJS 2026-06-04 (ES2025)#221
evgeny-boger wants to merge 18 commits into
masterfrom
quickjs-port

Conversation

@evgeny-boger

@evgeny-boger evgeny-boger commented Aug 13, 2026

Copy link
Copy Markdown
Member

What

Swaps the ES5-era Duktape 1.0.2 engine for QuickJS 2026-06-04 (latest Bellard release), bringing ES2023/ES2024 and most of ES2025 to rule scripts — classes with private fields, async/await + working promise microtasks, Object.groupBy, iterator helpers, Set algebra, toSorted/findLast, arrow-function rule callbacks, BigInt, and more. See sample-es2025.js.

How

  • QuickJS is a pinned git submodule (third_party/quickjs → bellard/quickjs 3d5e064, byte-identical to the official 2026-06-04 tarball). Patches, if ever needed, live as commits on a fork and rebase cleanly against upstream.
  • internal/quickjsduk implements the go-duktape API surface wbrules uses, on QuickJS — wired in with a one-line go.mod replace. wbrules engine code is untouched except stack-trace-format parsing in escontext.go. cgo compiles the submodule sources through five one-line wrapper files: no new build system, and armhf/arm64 cross-compilation rides the existing CGO_ENABLED=1 CC=<cross-gcc> flow.
  • Duktape semantics preserved where wbrules depends on them: per-file realms, calling-context dispatch for native callbacks, per-realm require() cache with module.static, Duktape's rc-error strings, plus a pending-job pump so promises resolve (Duktape 1.x had none).

Validation

  • All 36 wbrules test suites pass against the production wbgo.so (single-process go test ./wbrules/, 49.9 s). Three test-data updates for engine-behavior differences, each documented in PORT-QUICKJS.md.
  • Deployed to a WB8 over stock 2.46.2: every production rule file loads with zero script errors; MQTT round-trips verified.
  • Measured back-to-back on the same device and ruleset:
metric Duktape 2.46.2 QuickJS (this PR)
RSS / PSS steady state 37.6 / 36.1 MB 36.7 / 35.3 MB
MQTT reaction latency, median (n=300) 6.98 ms 7.61 ms
ES5 compute bench (sample-bench.js) ~1300 ms ~310 ms (4.2×)

Memory at parity, reaction latency dominated by the MQTT/driver path either way, raw JS ~4× faster.

Notes for review

  • PORT-QUICKJS.md is the full engineering log: ported semantics, the three behavior changes, packaging notes (plugin/binary -trimpath pairing, -fuse-ld=bfd for arm64 on binutils ≥ 2.44), and benchmark methodology.
  • CI needs git submodule update --init and, for cross-builds, crossbuild-essential-* + foreign-arch libc6 (or the existing sbuild chroots).
  • Suggested follow-up: fork bellard/quickjs into the wirenboard org and point .gitmodules there.

🤖 Generated with Claude Code


Update: TypeScript support + wild-corpus hardening

TypeScript rules (6c9b649): .ts files in /etc/wb-rules/ transpile on load via a persistent typescript-go child (~1 ms warm) and run immediately; type checking happens in the background and never delays execution ("run first, check later"). Findings land in the rules log (TS check: file:line:col) and as retained JSON on /wbrules/ts-check/<file> for editor integration. Tracebacks map back to .ts source lines through the emitted source map. Ships tsgo (/usr/lib/wb-rules/tsgo) and types/wb-rules.d.ts (builtin API declarations, also used by the homeui editor). New flags: -tsgo, -ts-types, -js-timeout.

Wild-corpus validation (cbe69b3): 663 user scripts harvested from wiki/support/GitHub were loaded through the engine — final sweep: 508 load OK, 5 need user modules not present, 150 fail for script-side reasons (forum-paste fragments, missing /etc configs), 0 crashes/hangs. The corpus is not distributed; every bug it exposed is pinned by a minimal synthetic regression test:

  • engine shutdown races with in-flight timer/cron/shell callbacks (sync queue redesign — never closed, syncDone handshake)
  • trackMqtt() before any publish crashed the engine MQTT client (lazy start on subscribe)
  • runaway JS (while(true)) froze the engine forever → new watchdog via JS_SetInterruptHandler (-js-timeout, default 60 s); stock Duktape wb-rules freezes identically
  • 'use strict' scripts threw on every dev/PersistentStorage write (proxy set traps didn't return true)
  • exports/module.exports now in scope for plain rule files
  • require() of a missing module reports cannot find module "id" instead of rc -100
  • spurious stack overflow on valid code: QuickJS's C-stack anchor vs Go goroutine↔OS-thread migration (anchor+operation fused into single cgo calls)
  • one data race (-race now viable: checkptr-clean shim, race-built plugin pairing documented)

Validated on a WB8: 2.47.0~quickjs4 deployed, TS demo rules live, checker output verified end to end.

evgeny-boger and others added 3 commits August 13, 2026 18:05
QuickJS is pinned as a git submodule (third_party/quickjs, bellard/quickjs
@ 3d5e064 = the 2026-06-04 release, verified byte-identical to the official
tarball) and compiled by cgo through one-line wrapper files in
internal/quickjsduk — no extra build system, cross-compilation rides the
existing CGO cross-CC flow.

internal/quickjsduk reimplements the go-duktape API surface wbrules uses
(wired via a go.mod replace, so wbrules code is untouched): value-stack
semantics, per-file realms via JS_NewContext, heap stash, Go-func
trampolines with Duktape's rc-error strings, Duktape 1.x CommonJS
(per-realm cache, relative ids, cycle-safe), calling-realm dispatch for
native callbacks, a pending-job pump for promise reactions, and
JS_UpdateStackTop per API entry (Go migrates goroutines across OS threads).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
escontext.go: parse QuickJS stack-line format; take the error message from
the error value (Duktape embedded it in .stack); substitute the script path
into syntax-error tracebacks.

Engine-behavior test-data updates: rule locations attribute multi-line
defineRule calls to their first line; StorableObject bookkeeping fields are
now non-enumerable (spec-correct for-in walks the proxy prototype chain);
the email-emoji log shows proper UTF-8 instead of Duktape's CESU-8 leak.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Makefile: prefer bfd for arm/arm64 external linking (Go forces gold, gone
from binutils >= 2.44). debian: build with system golang-go. Samples:
sample-es2025.js (ES2024/25 feature showcase incl. arrow-function rules)
and sample-bench.js (ES5 benchmark rules used for the published numbers).
PORT-QUICKJS.md documents the port, test status (36/36 suites), and the
WB8 hardware validation: memory parity, latency parity, ~4x compute.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codacy-production

codacy-production Bot commented Aug 13, 2026

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 3 critical · 39 high · 53 medium · 5 minor

Alerts:
⚠ 100 issues (≤ 0 issues of at least minor severity)

Results:
100 new issues

Category Results
UnusedCode 39 medium
BestPractice 1 medium
11 high
ErrorProne 23 high
Security 2 minor
5 high
3 critical
13 medium
CodeStyle 3 minor

View in Codacy

🟢 Metrics 609 complexity · 17 duplication

Metric Results
Complexity 609
Duplication 17

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

evgeny-boger and others added 2 commits August 14, 2026 10:39
Scripts harvested from wiki.wirenboard.com, support.wirenboard.com and
GitHub were loaded through the engine (corpus_test.go, WB_RULES_CORPUS
env, shardable). The corpus is not distributed; every bug it found is
pinned by a minimal synthetic regression test instead.

- Shutdown races: never close syncQueue (a send on a closed channel
  panics even inside select); Stop handshakes via syncDone, late
  timer/cron/shell thunks are dropped deterministically. TestStopUnderLoad.
- trackMqtt() as a script's first action lazily starts the engine MQTT
  client, like Publish does. TestTrackMqttAsFirstAction.
- JS execution watchdog via JS_SetInterruptHandler: runaway rules
  (while(true)) error out instead of freezing the engine loop forever.
  SetExecutionTimeLimit + JsExecutionLimit option. TestRunawayScriptInterrupted.
- Strict-mode writes: dev/PersistentStorage/StorableObject proxy set
  traps return true ('use strict' scripts threw TypeError). TestStrictModeProxyWrites.
- exports/module.exports in scope for plain rule files. TestExportsInRuleFile.
- require() of a missing module throws 'cannot find module id', not a
  bare rc -100. TestMissingModuleErrorMessage.
- Spurious 'stack overflow': QuickJS's stack anchor and the deep
  operation must share one cgo call (goroutines migrate OS threads
  between calls) - qjd_eval/qjd_call/... wrappers.
- syncQueueActive is atomic (race found by -race); opaque-id pointer
  fabrication moved into C for checkptr; tests resolve ../modules from
  the source-file path, immune to cwd changes by chdir-ing fixtures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
.ts rule files transpile on load through a persistent 'tsgo --api
--async' child (~1ms warm, LSP-framed JSON-RPC) and run immediately;
type checking runs afterwards in the background (tsgo --noEmit --lib
esnext with types/wb-rules.d.ts) and never delays execution. Findings
are logged as '[rule warning] TS check: file:line:col' and published as
retained JSON on /wbrules/ts-check/<virtual path> for editor UIs; a
clean re-check clears the topic. Tracebacks map through the V3 source
map back to .ts lines. Declaration files (.d.ts) in watched dirs are
skipped. The tsgo child dies with the daemon (PDEATHSIG) and is shipped
in the deb (/usr/lib/wb-rules/tsgo) together with the declarations
(/usr/share/wb-rules/types/wb-rules.d.ts). New flags: -tsgo, -ts-types,
-js-timeout. Samples: demo-typescript.ts, demo-ts-typecheck.ts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
evgeny-boger and others added 13 commits August 14, 2026 13:24
Verified against the pinned QuickJS build: all probed ES2025 features
pass (iterator helpers, Set methods, Promise.try, RegExp.escape,
duplicate named groups, Float16Array); of the ES2026 candidates only
explicit resource management (using/DisposableStack) is missing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
All confirmed by independent verification against the code:

- PcallProp (every rule/timer/device callback) and JsonEncode/JsonDecode
  still called JS_Invoke/JS_JSONStringify/JS_ParseJSON as bare cgo calls,
  re-opening the stack-anchor/thread-migration window on the hottest
  dispatch path; now fused via qjd_invoke/qjd_json_* wrappers.
- The JS execution watchdog never fired inside promise jobs (execStart
  was zero during the pump): a spinning .then reaction hung the engine
  forever despite -js-timeout. Each pending job is now armed as an
  outermost entry. Regression: TestExecutionTimeLimitInPromiseJob.
- .ts files loaded as raw JavaScript when tsgo was unavailable; they now
  fail with a clear 'TypeScript rules need tsgo' error.
- handleStop closes syncDone before flipping the active flag - the old
  order left a window where MaybeCallSync ran thunks inline on foreign
  goroutines.
- ESEngine.Stop stops the engine loop before killing tsgo (a draining
  .ts load could respawn an orphan child); transpile no longer kills the
  warm tsgo child on syntax errors (only on transport errors); background
  checks are capped at 2 concurrent tsgo processes; a per-file generation
  guard stops a slow check for an old file revision from overwriting the
  newer verdict.
- DestroyHeap deletes the rtReg entry (runtimeState leak).
- Test robustness: corpus drainer survives post-test Verify timeouts,
  module-missing classification matches the new error text, watchdog
  tests fail cleanly instead of hanging the binary, TsTypesPath resolved
  from source location, corpus fixture closes its bolt DB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Downloaded the es2026 tag of tc39/ecma262 and probed the pinned QuickJS
build against the actual 7-proposal feature list: 6 of 7 supported
(getOrInsert, Iterator.concat, JSON.rawJSON, Uint8Array base64/hex,
Math.sumPrecise, Error.isError); only Array.fromAsync is missing.
using/DisposableStack, Atomics.pause and Temporal are ES2027 track, not
ES2026 (Atomics.pause is supported anyway).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
The shared Jenkins deb pipeline has no prebuilt tsgo binaries, so the
install step now packages without TypeScript support when the binary is
absent (the engine rejects .ts files with a clear error in that case)
instead of failing the build.

The Actions workflow runs vet + the shim and engine test suites on
every PR, building wbgo.so from wbgo-private with the same toolchain in
the same job (plugin/binary pairing). Requires a WBGO_PRIVATE_TOKEN
repo secret with read access to wirenboard/wbgo-private.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
… ts-check topic

The retained /wbrules/ts-check/<file> topic had structural lifecycle
problems (stale diags surviving daemon restarts, nothing cleared on file
delete/rename/disable) and no consumer besides the web editor. Replaced
by the established mqtt-rpc surface, both documented in
asyncapi.mqtt-rpc.yml:

- Editor.Check(path): synchronous type check of one rule file with the
  engine's own tsgo and installed declarations - the authoritative
  verdict, pulled by editors on open/save instead of pushed retained.
- Editor.GetTypes: returns the installed wb-rules.d.ts so editors can
  validate against the API of the engine actually running, eliminating
  vendored-declaration version skew.

The load-time background check and its 'TS check:' log warnings stay
unchanged - they are the only type feedback for rule files that never
pass through the editor (scp/vim/ansible deployments, the dominant
wild-corpus workflow).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Fixes the suite failure the previous commit shipped with (the rpc
fixture verifies the exact announcement list).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
…dings

Findings from the dedicated adversarial review of the RPC restructure:

- Editor.Check no longer runs a tsgo process synchronously on the
  serially-dispatched RPC goroutine (it head-of-line blocked every
  editor RPC for seconds per check). The engine already checks each .ts
  file on load/save; Check now reads that cached verdict and answers
  'pending' while a check is in flight - clients poll. Statuses:
  ready/pending/not-ts/unsupported; 'not-ts' also disambiguates
  disabled files (previously indistinguishable from a clean verdict).
- Diagnostics from other files (imports/references) now carry their
  file in the reply instead of being silently mis-anchored.
- asyncapi: add the channel definitions both new operations referenced
  (the earlier insertion silently failed; operations pointed at
  nonexistent channels), update the Check reply schema.
- TS_CHECK_* constants live in their own const block: the main block
  derives itemType from iota, which counts every preceding spec - the
  first attempt shifted SOURCE_ITEM_* by 4 and broke rule reloading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
- Watchdog: a Go callback re-entering JS from inside a promise job
  (require(), runRules()) went depth 0->1->0 and disarmed the execution
  limit for the rest of the job; nested returns also drained the pending
  job queue mid-job, breaking run-to-completion. The job pump now counts
  as an active entry. Regression: TestExecutionTimeLimitSurvivesNestedEntryInJob.
- Async rule errors: the JobErrorHandler documented as wired to the rule
  log was never wired - exceptions after the first await died on stderr
  only. Now per-heap (SetJobErrorHandler) and routed to the engine log
  as 'async rule error: ...'.
- tsgo child: 15s watchdog kills a wedged (alive but stuck) child so
  the engine loop cannot block forever in a .ts load; response ids are
  verified (desync = transport error -> respawn); on-demand checks get a
  60s timeout + PDEATHSIG; the reaper/ensureStarted ProcessState race is
  gone (liveness tracked via c.cmd under the mutex).
- Editor.Check: files that fail transpile now get a terminal 'ready'
  verdict carrying the syntax error instead of 'pending' forever; .d.ts
  answers 'not-ts'; same-file diag matching by path suffix, not basename.
- Transpiled TypeScript runs strict again: the stripped 'use strict'
  prologue is re-added inside the single-line wrapper (line numbers
  intact) - a typo'd assignment now throws instead of creating a global.
- Stop(): drain the sync queue while waiting for the stop ack - with a
  full buffer the main loop could block on its own CallSync before
  reaching handleStop, deadlocking shutdown.
- goThreadFinalize queues leftover stack values for the reap path
  instead of freeing under regMu (finalizer re-entry self-deadlock).
- Duktape.version reports 10002 (the 1.0.2 being emulated), not 20600.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AoTGMFABCCENSPaHThxAru
@evgeny-boger

Copy link
Copy Markdown
Member Author

Split per review request into two stacked PRs: #223 (QuickJS engine + wild-corpus hardening, base master) and #224 (TypeScript support, stacked on #223). The quickjs-port branch stays for history; all content is preserved in the split branches (squashed, with original SHAs referenced in the commit messages).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant