Adds a new `bench` mechanism by jasnell · Pull Request #7669 · cloudflare/workerd

jasnell · 2026-10-08T20:50:17Z

Similar to wd-test but for benchmarks...

Planning to use this as part of the TS streams and other API optimizations.

*.wd-bench file

using Workerd = import "/workerd/workerd.capnp";

const unitTests :Workerd.Config = (
  services = [(
    name = "text-encoder-bench",
    worker = (
      modules = [(name = "worker", esModule = embed "text-encoder-bench.js")],
    ),
  )],
);

*.js file

const TYPES = {
  ascii: 'a',
  'one-byte': '\xff',
  'two-byte': '\u011f',
};
const LENGTHS = [32, 256, 1024, 8192];

export default {
  bench(b) {
    const encoder = new TextEncoder();
    for (const [type, char] of Object.entries(TYPES)) {
      for (const len of LENGTHS) {
        const input = char.repeat(len);
        b.run(`encode ${type} ${len}`, () => b.blackBox(encoder.encode(input)));
      }
    }
    for (const [type, char] of Object.entries(TYPES)) {
      for (const len of LENGTHS) {
        const input = char.repeat(len);
        // Enough space for any UTF-8 encoding of the input.
        const buffer = new Uint8Array(len * 3);
        b.run(`encodeInto ${type} ${len}`, () =>
          b.blackBox(encoder.encodeInto(input, buffer))
        );
      }
    }
  },
};

$ just wd-bench //src/workerd/api/tests:text-encoder-bench@
bazel run //src/workerd/api/tests:text-encoder-bench@ --config=benchmark --
INFO: Invocation ID: f1ef914e-5810-4ad9-8dca-fbaa43ceddce
WARNING: Build options --@@+http+capnp-cpp//src/kj:debug_memory, --@@rules_rust+//rust/settings:extra_rustc_flag, --compilation_mode, and 3 more have changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //src/workerd/api/tests:text-encoder-bench@ (1 packages loaded, 7481 targets configured, 9 aspect applications).
INFO: Found 1 target...
Target //src/workerd/api/tests:text-encoder-bench@ up-to-date:
  bazel-bin/src/workerd/api/tests/text-encoder-bench@_wd_bench.sh
INFO: Elapsed time: 236.953s, Critical Path: 223.02s
INFO: 418 processes: 11545 action cache hit, 287 disk cache hit, 13 internal, 118 processwrapper-sandbox.
INFO: Build completed successfully, 418 total actions
INFO: Running command line: bazel-bin/src/workerd/api/tests/text-encoder-bench@_wd_bench.sh src/workerd/server/workerd bench src/workerd/api/tests/text-encoder-bench.wd-bench '--compat-date=2000-01-01'
workerd/server/server.c++:8410: debug: [ BENCH ] text-encoder-bench
workerd/server/server.c++:8440: debug: [ DONE ] text-encoder-bench (41.8968367s)
Results compare runs on this machine and build; they don't predict production latency.
workerd 2026-10-08, release build, V8 15.5.35.7, Intel(R) Xeon(R) w9-3575X, governor performance, compat date 2000-01-01

text-encoder-bench
  overhead per call, subtracted: 27.2 ns, 905 ns async; blackBox() 32.1 ns
  case                      median   95% CI             mean     cpu median  samples
  encode ascii 32           311 ns   309 ns - 314 ns    313 ns   311 ns      50 x 65536
  encode ascii 256          320 ns   317 ns - 324 ns    322 ns   320 ns      50 x 65536
  encode ascii 1024         373 ns   372 ns - 377 ns    391 ns   373 ns      50 x 65536   (high variance: try more samples or a longer minTime)
  encode ascii 8192         950 ns   948 ns - 954 ns    953 ns   950 ns      50 x 32768
  encode one-byte 32        321 ns   319 ns - 325 ns    324 ns   321 ns      50 x 65536
  encode one-byte 256       552 ns   549 ns - 557 ns    559 ns   552 ns      50 x 65536
  encode one-byte 1024      1.31 us  1.30 us - 1.33 us  1.32 us  1.31 us     50 x 16384
  encode one-byte 8192      8.48 us  8.40 us - 8.56 us  8.51 us  8.48 us     50 x 4096
  encode two-byte 32        377 ns   374 ns - 380 ns    380 ns   377 ns      50 x 65536
  encode two-byte 256       758 ns   744 ns - 773 ns    756 ns   758 ns      50 x 32768
  encode two-byte 1024      2.15 us  2.14 us - 2.16 us  2.14 us  2.15 us     50 x 16384
  encode two-byte 8192      15.1 us  14.9 us - 15.3 us  15.6 us  15.1 us     50 x 2048    (high variance: try more samples or a longer minTime)
  encodeInto ascii 32       203 ns   201 ns - 205 ns    204 ns   203 ns      50 x 131072
  encodeInto ascii 256      185 ns   184 ns - 187 ns    187 ns   185 ns      50 x 131072
  encodeInto ascii 1024     196 ns   195 ns - 198 ns    196 ns   196 ns      50 x 131072
  encodeInto ascii 8192     312 ns   311 ns - 315 ns    326 ns   312 ns      50 x 65536   (high variance: try more samples or a longer minTime)
  encodeInto one-byte 32    215 ns   214 ns - 216 ns    215 ns   215 ns      50 x 131072
  encodeInto one-byte 256   420 ns   416 ns - 423 ns    419 ns   420 ns      50 x 65536
  encodeInto one-byte 1024  1.08 us  1.08 us - 1.10 us  1.09 us  1.08 us     50 x 32768
  encodeInto one-byte 8192  7.41 us  7.34 us - 7.46 us  7.45 us  7.40 us     50 x 4096
  encodeInto two-byte 32    233 ns   232 ns - 234 ns    233 ns   233 ns      50 x 131072
  encodeInto two-byte 256   602 ns   598 ns - 606 ns    601 ns   602 ns      50 x 32768
  encodeInto two-byte 1024  1.89 us  1.87 us - 1.90 us  1.89 us  1.89 us     50 x 16384
  encodeInto two-byte 8192  13.5 us  13.4 us - 13.5 us  13.5 us  13.5 us     50 x 2048

Also sets us up for some more nodejs_compat work later but that's secondary.

guybedford

Built and ran this locally: histogram_test, bench-test@, bench-report-test@, the server-test bench case, text-encoder-bench@smoke and exported-handler-proxy-test@ all pass, and a real run of workerd bench produces sensible output. The harness design (C++ timing loop, calibrated batches, overhead measured the same way as user cases and subtracted) looks good.

My main question is the scope of the histogram crate. As I read it, bench reaches Histogram::new/record/percentile/percentile_ci/mean_ci/mean/stddev/skewness/kurtosis, which is a few hundred lines of hdr.rs/histogram.rs/stats.rs. The rest of the crate (cbor.rs export/import, qrde.rs, Welch/Mann-Whitney, the percentile/linear/log iterators, EWMA, record_corrected/record_delta/add/subtract, and the Node golden parity suite in lib-test.rs + testdata/) looks like it's there for a node:perf_hooks createHistogram() implementation rather than for the harness. Is that the intended scope of this PR? If not, could we drop those parts here (roughly 3k lines plus the fixtures) and bring them in with the perf_hooks PR that has a consumer for them?

A few smaller things inline.

jasnell · 2026-10-10T11:27:23Z

... Is that the intended scope of this PR? If not, could we drop those parts here

They are intended to also support the perf_hooks stuff and more nodejs_compat but there are pieces that are used here and the bench tool will be incrementally expanded to use more of those. Going to need to do this work anyway, planning to do it very soon so would rather keep it here.

The histogram statistics need erfc() and lgamma(); the std equivalents (float_erf, float_gamma) are unstable. libm has no dependencies.

Adds src/rust/histogram, which provides HDR histograms that behave like Node.js's native perf_hooks histograms. It will back the statistics of the `workerd bench` harness and node:perf_hooks createHistogram(). `hdr` ports the parts of HdrHistogram_c (0.12.0, as vendored by Node.js; public domain under CC0) that Node.js uses: bucket layout, recording, merging, the internal-counter reset, the value-equivalence helpers, percentiles, mean, standard deviation, and the recorded, percentile, linear, and logarithmic iterators. It follows the C code closely, including the round-half-up percentile target, the iterators' boundary steps, and the order of floating-point operations, so that results match Node.js exactly. The shift operations, and with them a nonzero normalizing index offset, are not ported because Node.js never uses them. `Histogram` adds the state Node.js keeps alongside the HDR data: the exceeds count, the reset count, the recordDelta() timestamp, and the EWMA, with Node.js's snapshot(), diff(), add(), subtract(), and reset() semantics.

Ports the analysis functions of Node.js's src/histogram.cc: skewness and excess kurtosis, a Student's t confidence interval for the mean, an exact binomial confidence interval for percentiles, Welch's t-test, the Mann-Whitney U test with tie correction, Cohen's d, Cliff's delta, and the two-sample Kolmogorov-Smirnov statistic, with the numerical helpers they rely on (the regularized incomplete beta function and the normal, Student's t, and binomial distributions). The port keeps Node.js's operation order, so results match to the last bit except where libm's lgamma() and erfc() differ from the platform libm by an ulp. The two-sample functions walk the counts arrays by index, as Node.js does, without checking that the layouts match. QRDE and the asymptotic incomplete beta approximations it uses are not ported yet.

Implements Node.js's histogram export format, the CBOR map behind histogram.export() and perf_hooks.importHistogram(). Export writes format version 2, byte-for-byte what Node.js writes. Import accepts versions 1 and 2 with Node.js's validation: duplicate keys, indexes that are not strictly increasing, a counts length that does not match the layout, counts that overflow, and a total count that does not match the counts are rejected, and unknown keys are skipped in version 2 data only. Unlike Node.js, import rejects a nonzero normalizing index offset. Node.js never writes one, and the HDR core does not port the index normalization it would require. The tests check exports against bytes produced by Node.js 26.10, which writes version 1; the only difference is the version value.

Ports histogram.qrde() from Node.js: Harrell-Davis quantiles at the requested probability boundaries, with the "none", "hdr", and "all" dequantization modes, the tail-trimming support search for histograms with many buckets, and the Edgeworth approximation for concentrated beta weights. The incomplete beta function gains the asymptotic paths QRDE needs; the exact path that the other statistics use is unchanged. Node.js computes QRDE on its thread pool and returns a promise. Here it is synchronous, since the computation is linear in the occupied buckets per boundary, and there is accordingly no equivalent of the snapshot cache.

Adds a test_compile_data attribute for files that a crate's unit tests read at compile time with include_str! or include_bytes!, such as test vectors. Like the *-test.rs files, they are inputs of the test target only.

Adds node-golden.json, the results of every histogram query that the crate implements, from Node.js 26.10, over 11 datasets (empty, tiny, constant, uniform, wide, bimodal, heavy-tailed, and out-of-range) in 7 layouts (figures 1 to 5, lowest 1, 1000, and 1024), plus the two-sample tests over 6 pairs. generate-node-golden.mjs produces it; the datasets use integer arithmetic only and are stored in the output. The tests require identical results where the crate performs the same operations as Node.js: counts, min and max, mean, standard deviation, percentiles, the percentile, linear, and logarithmic iterations, cdf(), percentile confidence intervals, and export bytes. Results that go through lgamma() or erfc() are compared to 1e-12 relative, since libm can differ from glibc by an ulp. Every export is also imported and checked again. Floats in the file are strings: serde_json's default float parsing is not correctly rounded, which showed up as one-ulp mismatches, while str::parse::<f64>() is. The file is excluded from prettier, which would reflow it into about a hundred thousand lines.

The benchmark harness records its samples into histograms and reports their statistics. The bridge exposes creating a histogram, recording into it, and a summary with the statistics the harness reports: count, min, max, mean, standard deviation, median, p75, p99, and confidence intervals on the median and the mean, plus skewness and kurtosis.

Set by the upcoming `workerd bench` subcommand. Code that exists only for benchmarking checks it, so that it fails closed anywhere else.

BenchParams carries the options for one bench() handler; BenchReport holds the results: per case, wall and CPU time per call with summary statistics and the raw samples. workerd bench --format=json will print the report with capnp's JSON codec, so field names are its JSON keys.

A Worker's `bench(b, env, ctx)` handler registers cases with `b.run(name, fn, options)`; the runtime then runs the timing loop, so user code never sees a high-resolution clock. For each case, the runner calls `fn` repeatedly to warm up, doubles the batch size until a batch takes at least minTime / samples (and 1 ms), then times `samples` batches, reading the monotonic and thread CPU clocks only between batches. Promises returned by `fn` are awaited before the next call. Batches are separated by a turn of the event loop. `setup` and `teardown` run untimed around each batch, and `b.blackBox(value)` returns its argument through a native call that V8 can't see into. The event is WorkerInterface::bench(params, groupBuilder), delivered like test(): only WorkerEntrypoint implements it, and the server's wrappers forward it. Unlike test(), it awaits the handler directly instead of through finishScheduled(), since the report needs the handler's result. Failures of the handler or of a case are recorded in the report; statistics come from the histogram crate. BenchController construction requires bench mode, which only `workerd bench` sets, so the event fails closed anywhere else. The handler isn't a member of ExportedHandler, which is unwrapped from the export for every event and reads each handler property. Reading a new property there would break exports that reject unknown properties, like miniflare's dev registry proxy, a WorkerEntrypoint that returns a Proxy from its constructor. Instead, as for `queue`, a separate BenchExportedHandler is unwrapped from the export's `self` only when a benchmark runs, through Worker::Api::getBenchTypeHandler(), whose default implementation throws. exported-handler-proxy-test checks that delivering an event reads no property outside the existing handler names. Compatibility: this adds `bench` to the ExportedHandler typings and the BenchController and BenchCaseOptions types to the generated typings, following the `test` precedent. Nothing in production dispatches the event or reads the property. Unlike `test`, `bench` isn't added to the WorkerEntrypoint or DurableObject classes: their methods are RPC methods, and an existing method named `bench` with another signature would stop type-checking. Such methods keep working over RPC.

Runs the bench handlers of the entrypoints that match the service and entrypoint patterns, as Server::test() runs test handlers, adding a group to a BenchReport for each. Progress is logged like test()'s, with [ BENCH ], [ DONE ], [ FAIL ], and [ SKIP ] lines. The startup that test() did inline moves to startLocalRun(), which both use. Its state lives in one heap object, destroyed after the run as the locals were. Only plain-object exports have bench handlers. In bench mode, validation leaves `bench` out of the methods it reports for WorkerEntrypoint and Workflow classes, whose methods are RPC methods, so an unrelated RPC method named `bench` isn't run as a benchmark. Production validation doesn't change.

`workerd bench <config> [filter]` runs the config's bench handlers and writes the results to stdout, or to --output, as a table or with --format=json as the BenchReport. It takes the same config files and filter syntax as `workerd test`, plus an optional /<case-pattern> suffix, and the same --compat-date, --all-autogates, and --no-verbose flags. --quick caps budgets for checking that benchmarks run. The report's environment records the workerd and V8 versions, build mode, CPU model, and frequency governor, with warnings for non-release builds and governors other than "performance".

Benchmarked code often makes a subrequest on every call, as in `env.SELF.fetch()`, and tail workers would add their cost to each one. No request gets tracers in bench mode. The bench server test now clears bench mode when it's done, so that it doesn't change later tests in the same binary.

@smoke

wd_bench(src = "x.wd-bench", data = [...]) takes a config in wd_test()'s format and, for each compat variant, defines a `bazel run` target that runs `workerd bench` (tagged workerd-benchmark, so CI's benchmark filter excludes it) and an @smoke test that runs each case briefly with --quick, so CI checks that benchmarks keep working. @benchmark.json writes the default variant's report, off by default like wd_cc_benchmark()'s @benchmark.csv. `just wd-bench <target> [args]` runs one in the benchmark configuration. wd_test.bzl gains wd_cli_test(), a public name for its test rule, which wd_bench() uses for the smoke tests. As for wd_test(), configs can declare a writable disk service named TEST_TMPDIR, e.g. for Durable Object storage. The smoke tests get Bazel's test temporary directory; the run targets and @benchmark.json create a new temporary directory and remove it afterwards, so no target runs against persistent storage unless its config names a path. text-encoder-bench ports bench-text-encoder.c++. Each case is a single call, where the C++ version measured a fetch whose handler looped 1000 times.

Before the first group runs, the runner times three internal cases the same way as user cases: an empty function, a function that returns a resolved promise, and `blackBox(undefined)`. Their medians are kept for the rest of the process. Each case's samples have the matching empty-case cost subtracted, chosen by whether its function returned a promise, never going below zero. A case whose median before subtraction is under twice the overhead is flagged atFloor, since it mostly measures the runner (or V8 optimized its work away). Its variance is meaningless then, so it isn't also flagged highVariance. Each group reports the overhead it subtracted, so raw values can be recovered, and the text report shows it under the group name.

Adds a section on JavaScript benchmarks to docs/benchmarking.md: the bench handler and BenchController, how cases are timed, wd_bench() and running benchmarks, reading the report, and why classes have no bench handlers.

Tail workers are skipped under `workerd bench`, since they add their cost to every request; --trace runs them, to measure that cost. Skipping them now happens when a Worker's channels are linked, rather than for each request: the tail workers are looked up, so config errors are still reported, and then dropped.

The bench server test asserted that an empty async case is flagged at the floor, which depends on timing and failed under load. bench-test checks the report's statistics with fixed samples instead: which overhead is subtracted, clamping at zero, and when cases are flagged at the floor or high-variance.

`just new-bench //path/to:name-bench` scaffolds a wd_bench(): a .wd-bench config, a module with an example bench() handler, and the BUILD rule, as `just new-test` does for wd_test().

Lists `just new-bench` and `just wd-bench`, wd_bench() under test types, and where the bench harness lives; adds `bench` to the server's subcommands.

The Node.js parity tests take about 10 seconds in an unoptimized build, since Rust tests run serially, so under load they exceeded a medium test's 15-second timeout. Optimized, they take under a second.

The recipe copies the snapshot from bazel-bin, whose outputs are read-only, and cp keeps their modes, so types/generated-snapshot/ experimental was left read-only. Git then couldn't replace the files in it, and a checkout or rebase that changed them failed with "unable to unlink old ...: Permission denied". The recipe already made the directory writable before removing it; it now does so after the copy too.

Benchmarks usually measure current behavior, but wd_bench()'s default variant copied wd_test()'s 2000-01-01, so `just wd-bench ...@` measured year-2000 semantics, and @all-compat-flags (2999-12-31) enables flags whose dates haven't arrived. `workerd bench --compat-date=latest` now overrides every worker's date with the newest one this build supports: today's date, or MAXIMUM_COMPATIBILITY_DATE if that's earlier, since the maximum is set a little into the future. Without --compat-date, workers that omit compatibilityDate get that date rather than a config error, through the new Server::setDefaultCompatibilityDate(); workers that set a date keep it. The report records the resolved override in Environment.compatDate and the default in the new Environment.defaultCompatDate. wd_bench()'s default and @all-autogates variants now pass --compat-date=latest unless compat_date is set. Since the date is only known at run time, it can't come from the macro itself.

jasnell requested review from a team as code owners October 8, 2026 20:50

jasnell requested a review from dario-piotrowicz October 8, 2026 20:50

jasnell force-pushed the jasnell/wd-bench branch from 908b943 to 6b7735c Compare October 8, 2026 20:54

This comment was marked as resolved.

Sign in to view

ask-bonk Bot reviewed Oct 8, 2026

View reviewed changes

npaun reviewed Oct 8, 2026

View reviewed changes

Comment thread build/wd_bench.bzl

npaun reviewed Oct 8, 2026

View reviewed changes

Comment thread build/wd_bench.bzl

jasnell force-pushed the jasnell/wd-bench branch 2 times, most recently from 712d651 to 36fb9cc Compare October 9, 2026 03:45

ask-bonk Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread build/wd_bench.bzl

Comment thread src/workerd/io/worker.c++

jasnell force-pushed the jasnell/wd-bench branch 4 times, most recently from 2d8cb27 to c9f80fc Compare October 9, 2026 20:28

jasnell requested review from danlapid, guybedford and npaun October 9, 2026 23:45

guybedford reviewed Oct 10, 2026

View reviewed changes

jasnell added 9 commits October 10, 2026 11:28

Add libm Rust dependency

32d3a83

The histogram statistics need erfc() and lgamma(); the std equivalents (float_erf, float_gamma) are unstable. libm has no dependencies.

Let wd_rust_crate tests embed data files

0877187

Adds a test_compile_data attribute for files that a crate's unit tests read at compile time with include_str! or include_bytes!, such as test vectors. Like the *-test.rs files, they are inputs of the test target only.

Add a process-wide bench mode

3ffd0bc

Set by the upcoming `workerd bench` subcommand. Code that exists only for benchmarking checks it, so that it fails closed anywhere else.

jasnell added 14 commits October 10, 2026 12:00

Document workerd bench and wd_bench()

0322d96

Adds a section on JavaScript benchmarks to docs/benchmarking.md: the bench handler and BenchController, how cases are timed, wd_bench() and running benchmarks, reading the report, and why classes have no bench handlers.

Add just new-bench

bc50f8e

`just new-bench //path/to:name-bench` scaffolds a wd_bench(): a .wd-bench config, a module with an example bench() handler, and the BUILD rule, as `just new-test` does for wd_test().

Mention workerd bench in AGENTS.md

f2effbe

Lists `just new-bench` and `just wd-bench`, wd_bench() under test types, and where the bench harness lives; adds `bench` to the server's subcommands.

Make the histogram crate's tests large

8ee7d01

The Node.js parity tests take about 10 seconds in an unoptimized build, since Rust tests run serially, so under load they exceeded a medium test's 15-second timeout. Optimized, they take under a second.

jasnell force-pushed the jasnell/wd-bench branch from c9f80fc to a031007 Compare October 10, 2026 12:00

ask-bonk Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread src/workerd/server/cli-main.c++

jasnell force-pushed the jasnell/wd-bench branch from a031007 to a4a6f8d Compare October 10, 2026 13:14

ask-bonk Bot reviewed Oct 10, 2026

View reviewed changes

Comment thread src/workerd/api/tests/bench-default-compat-date-test.js

jasnell requested a review from guybedford October 10, 2026 13:49

Navigation Menu

Repository navigation

Adds a new `bench` mechanism - #7669

Adds a new `bench` mechanism#7669
jasnell wants to merge 24 commits into
mainfrom
jasnell/wd-bench

jasnell commented Oct 8, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guybedford left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasnell commented Oct 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jasnell commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guybedford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasnell commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jasnell commented Oct 8, 2026 •

edited

Loading

jasnell commented Oct 10, 2026 •

edited

Loading