Windows QoS gate: Hoard vs the system heap (geomean floor 1.25x)

[emeryberger]

Now that Windows benchmark output is captured (#119 + alloc8#14), Windows can be performance-checked, not just crash-checked.

Design

The comparison allocators (mimalloc/jemalloc) aren't built on Windows, so the only other allocator present is the Windows system heap — which is exactly the heap Hoard replaces, and which the benchmarks already measure as their uninjected baseline. So the meaningful Windows question is "am I at least as fast as what I replace?", across the whole suite.

scripts/qos_windows.py parses the baseline and Hoard result files and reports the per-benchmark ratio plus the geometric mean. Each benchmark runs once per job, so a single ratio is noisy (larson especially, being false-sharing sensitive) — but the geomean over ten benchmarks is steady, so the gate keys on the geomean and merely reports the rest.

First measurement

benchmark               system          hoard     hoard/system
alloc-test          31,225,137     52,293,433            1.67x  faster
cache-scratch       46,300,155    101,570,792            2.19x  faster
cache-thrash       156,304,340    303,310,636            1.94x  faster
cfrac               19,148,783     26,899,972            1.40x  faster
larson               5,514,093      4,802,417            0.87x  SLOWER
malloc-large         7,075,639     10,622,477            1.50x  faster
mstress             25,243,059     37,930,224            1.50x  faster
rptest               5,806,144     11,145,388            1.92x  faster
sh6bench            10,604,375     22,021,096            2.08x  faster
sh8bench            20,336,963     20,255,113            1.00x
geomean                                                  1.55x

Hoard is faster on 8 of 10, ties sh8bench, and trails only single-run larson.

Report-only, for now

Gathering geomean variance across runs before committing a hard floor — I'm not going to repeat the macOS gate's false-positive (which failed a healthy merge because a runner was degraded for the whole job). It emits a ::warning:: if the geomean drops below a provisional 1.15x in the meantime.

Verified locally: pass/fail/warn paths all behave (exit 1 + message on regression, exit 0 on pass, ::warning:: on report-only).

🤖 Generated with Claude Code

[emeryberger]

Now gating (updated from report-only).

Calibrated across five runs of unchanged code: geomean(hoard/system) = 1.55, 1.54, 1.65, 1.59, 1.44. Individual benchmarks are noisy (larson 0.87–0.90, sh8bench 0.94–1.00, cache-thrash 1.66–1.94), but the geomean barely moves — which is the whole point of keying on it.

Floor is 1.25×: ~13% below the lowest observed (1.44), so it won't flake, while still tripping if Hoard ever falls toward parity with the heap it replaces. Note the 1.44 run vindicates the conservative choice — a snugger 1.40 floor would have nearly failed a perfectly healthy run.

Current run: PASS at 1.44×.