perf(ci): parallelize Python tests and shorten the macOS Rust job - #8628
Open
wjones127 wants to merge 10 commits into
Open
perf(ci): parallelize Python tests and shorten the macOS Rust job#8628wjones127 wants to merge 10 commits into
wjones127 wants to merge 10 commits into
Conversation
CI end-to-end time per commit is a median of 26.6 min and a p90 of 60.9 min, measured over the ~1000 PR-triggered runs in a two-day window. The Python workflow is the critical path on 74 of 132 commits and Rust on 51. Three of the largest contributors were single-threaded work on multi-core runners: - `Compatibility Tests` ran 292 tests single-process on a 4-core runner for 1283s, and was the last job to finish in 25 of 33 Python runs. The suite is subprocess-bound, and `nightly_run.yml` already runs part of it under xdist at `nproc * 4`. `COMPAT_TEMP_VENV` also disabled the persistent venv cache that `conftest.py` supports, so each run rebuilt ~18 venvs from PyPI -- leaving it unset lets xdist workers share one venv per version through the flock that `VenvFactory` already holds for exactly this purpose. - `make test` ran 1363 tests serially in 673s in five separate jobs. The Makefile grows a `PYTEST_WORKERS` knob, defaulted off so a local `make test` is unchanged, and the `run_tests` action passes `auto`. - `mac-build` used plain `cargo test`, one test binary at a time. macOS is the scarcest runner pool in CI (p90 queue 16.7 min against under 2 min for every other pool), so this job is now nextest and drops the bench compile-check that `build-no-lock` already performs on Linux. Doctests are unaffected: nextest does not run them, but `linux-arm` and `windows-build` still invoke `cargo test`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
A-python
`compat_versions` built its parameter list by iterating a set of version strings. Set iteration order for strings depends on PYTHONHASHSEED, which differs per process, so the collected test IDs came out in a different order every run -- 11 distinct orderings across 12 seeds locally. That was invisible while pytest ran in one process. Under pytest-xdist each worker is its own process with its own hash seed, so the workers disagree on collection order and xdist aborts the run with "Different tests were collected". Because `make test` collects (and then skips) the compat directory, this broke the whole unit suite, not just the compat job. Sorting by version fixes the ordering and also makes test IDs stable between runs, which `--lf` and shard-to-shard comparisons both rely on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
windows-build was the last job to finish the Rust workflow at 21.0m, after mac-build moved to nextest. It used plain `cargo test`, which runs one test binary at a time; `Run tests` alone was 10.2m of that. Switches to nextest and drops the benchmark compile-check, matching what mac-build now does. Doctests are unaffected -- nextest skips them, but linux-arm still runs `cargo test` with all features. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`test_table_roundtrip` called bare `os.fork()` and let the child fall through into the rest of the test and the rest of the pytest session. Under a single pytest process that merely duplicated some work. Under pytest-xdist the child also inherits the worker's execnet connection to the controller, so it reports a second result for the same test; the controller then hits `ValueError: list.remove(x): x not in list` in its scheduler and aborts the whole session. That killed the Linux x86 and ARM jobs, while macOS happened to win the race and Windows skips the test entirely. The child now does its post-fork reads and leaves through `os._exit`, and the parent waits on it. That also makes the test stronger: previously nothing the child asserted could fail the run, so the post-fork read was never actually verified. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
linux-arm was left on `cargo test` when mac-build and windows-build moved to nextest, to keep doctest coverage. It is now the longest job in the workflow at 20.5m, 13.7m of which is running tests one binary at a time. Run the tests under nextest and the doctests as a separate `cargo test --doc` step, so the bulk of the work parallelizes without losing doctests. The feature list is resolved once into the environment rather than recomputed per step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 908d03b.
The child of test_fork's os.fork() dies of a signal on macOS before it can read the dataset. That predates this branch -- the test never waited on the child, so a crash there was invisible -- and fixing it is out of scope here. Keep the assertion where it holds so the Linux path stays covered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lindera fixture unzips a dictionary into the checked-out models tree and removes it on teardown. Under xdist one worker's teardown deletes the dictionary while another is still reading it, which failed the Windows job with a missing dict.da. The tokenizer configs name that path relative to the repo, so the model cannot be relocated to a per-worker directory. Group these tests onto a single worker instead, via --dist loadgroup and an xdist_group mark applied to every test that uses the fixture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every Linux job ran `sudo apt update && sudo apt install protobuf-compiler libssl-dev` before doing any work. That step takes ~4.4 minutes on a healthy run -- as long as some of the jobs themselves -- because the update refreshes every mirror index on the image. It also hangs: four jobs across two runs today sat in apt until their own timeouts killed them at 30-45 minutes and reported the PR red with none of their real work run. One of them took the coverage chain's merge step down with it. libssl-dev and pkg-config are already on the runner images, on both x86_64 and arm64, so protobuf-compiler was the only package any of this actually installed. Install it with taiki-e/install-action, already used here for nextest and llvm-cov, which fetches a prebuilt binary in seconds. It also copies protoc's bundled include directory alongside the binary and exports PROTOC, which the protos need for their google/protobuf imports. The coverage-test job asked only for libssl-dev, so its step is dropped outright. Jobs needing packages that are genuinely absent (libhdf5-dev, the QEMU build deps) keep apt; the QEMU step gains its own `apt update`, which it had been getting from the step replaced here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wjones127
marked this pull request as ready for review
August 19, 2026 21:58
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Python workflow is CI's long pole — the critical path on 74 of 132 commits — and this takes it from 26.3 to 16.6 minutes end to end (−37%).
Compatibility TestsPython Linux 3.14 / 3.13 / 3.10 x86_64Python Linux 3.14 ARMwindowsPython macOS 3.14 ARMwindows-buildmac-build (stable)Baselines are medians over ~1000 production PR runs; "after" figures are from green jobs only.
make testgains aPYTEST_WORKERSknob, passed asautoby therun_testsaction. It defaults to empty, so a localmake testis unchanged.mac-build/windows-builduse nextest, and drop a bench compile-checkbuild-no-lockalready does. Doctests still run vialinux-arm.libssl-devandpkg-configare already on the images.macOS Rust is the weakest result — repeat runs range from −18% to no change.