Skip to content

perf(ci): parallelize Python tests and shorten the macOS Rust job #8628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wjones127 wants to merge 10 commits into
base: main
Choose a base branch
from
Open

perf(ci): parallelize Python tests and shorten the macOS Rust job #8628

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
82e76d0
perf(ci): parallelize Python tests and shorten the macOS Rust job
wjones127 Aug 18, 2026
30122d7
Merge branch 'main' into perf/ci-parallel-tests
wjones127 Aug 19, 2026
db01ac7
fix(python): make compat version parametrization deterministic
wjones127 Aug 19, 2026
ad36c06
perf(ci): run Windows Rust tests under nextest
wjones127 Aug 19, 2026
3e63f07
fix(python): exit the forked child in test_fork
wjones127 Aug 19, 2026
908d03b
perf(ci): run linux-arm Rust tests under nextest
wjones127 Aug 19, 2026
5bb30d4
Revert "perf(ci): run linux-arm Rust tests under nextest"
wjones127 Aug 19, 2026
3a2996a
test(python): scope the forked-child assertion to Linux
wjones127 Aug 19, 2026
a84e5c0
test(python): pin the lindera tests to one xdist worker
wjones127 Aug 19, 2026
df769e4
perf(ci): install protoc from a release binary instead of apt
wjones127 Aug 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/compat-pair.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,10 @@ jobs:
# Toolchain for the build-from-source provisioning path (refs without a wheel).
- uses: actions-rust-lang/setup-rust-toolchain@a0b538fa0b742a6aa35d6e2c169b4bd06d225a98 # v1
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
- name: Install build deps
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Install host deps
run: pip install pytest pytest-xdist pyarrow packaging maturin
- name: Resolve kinds
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/file_verification.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,10 @@ jobs:
with:
python-version: "3.11"

- name: Install Build Requirements
run: |
sudo apt update
sudo apt install -y protobuf-compiler
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc

- name: Validate AWS Credentials
run: |
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ jobs:
workspaces: |
lance
java/lance-jni -> ../target/rust-maven-plugin/lance-jni
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # cargo-llvm-cov
Expand All @@ -59,10 +59,10 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
# pin the toolchain version to avoid surprises
- uses: actions-rust-lang/setup-rust-toolchain@a0b538fa0b742a6aa35d6e2c169b4bd06d225a98 # v1
with:
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/nightly_run.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ jobs:
contents: read
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Run Jumbo String/Binary Tests
run: |
echo "Running jumbo tests for Lance 2.0 and 2.1..."
Expand Down Expand Up @@ -71,10 +71,10 @@ jobs:
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
with:
workspaces: python
- name: Install build deps
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Install host deps
run: pip install pytest pytest-xdist pyarrow packaging maturin
# Build HEAD once and feed it to the suite as the prebuilt reader, so the two writer
Expand Down
20 changes: 12 additions & 8 deletions .github/workflows/python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,10 @@ jobs:
ruff format --check --diff python
ruff check python
pyright
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Lint Rust
run: |
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | grep -v protoc | sort | uniq | paste -s -d "," -`
Expand Down Expand Up @@ -185,7 +185,7 @@ jobs:
compat:
needs: linux-wheel
timeout-minutes: 60
runs-on: ubuntu-24.04
runs-on: ubuntu-24.04-8x
name: Compatibility Tests
defaults:
run:
Expand All @@ -208,11 +208,15 @@ jobs:
- name: Install dependencies
run: |
pip install $(ls target/wheels/pylance-*.whl)[tests,ray]
# Nearly every test here shells out to an old pylance in its own venv, so the
# job spends most of its time waiting on subprocesses rather than on CPU.
# Oversubscribing cores follows what nightly_run.yml already does with this
# suite. Leaving COMPAT_TEMP_VENV unset is what makes that pay off: the venvs
# then land in the shared cache directory, where VenvFactory's flock lets all
# workers reuse one venv per version instead of each building its own.
- name: Run compatibility tests
run: |
make compattest
env:
COMPAT_TEMP_VENV: 1
make compattest PYTEST_WORKERS=$(( $(nproc) * 2 ))

@lance-gatekeeper lance-gatekeeper Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each xdist worker imports compat_decorator.py and independently queries PyPI and Fury while collecting tests. Those lru_caches are process-local; the added sort only normalizes order. If one request times out or returns a newer release, that worker builds a different parameter set and xdist aborts with Different tests collected, making this parallel compat job nondeterministically fail. Resolve the versions once before spawning workers and distribute the same immutable snapshot.

Reproducer run at df769e4

From python/, I used the current compat_test decorator and made gw1 see one extra stable release during collection:

/home/agent/tmp/xdist_version_repro/conftest.py:

import io
import json
import os
import subprocess
import urllib.request

def urlopen(*args, **kwargs):
    releases = {"0.40.0": {}, "4.1.0": {}}
    if os.environ["PYTEST_XDIST_WORKER"] == "gw1":
        releases["5.0.0"] = {}
    return io.BytesIO(json.dumps({"releases": releases}).encode())

def run(args, **kwargs):
    return subprocess.CompletedProcess(
        args, 0, stdout="Available versions: 5.1.0b1\n", stderr=""
    )

urllib.request.urlopen = urlopen
subprocess.run = run

/home/agent/tmp/xdist_version_repro/test_collection.py:

import sys

sys.path.insert(0, "python/tests/compat")
from compat_decorator import UpgradeDowngradeTest, compat_test

@compat_test()
class Sample(UpgradeDowngradeTest):
    def __init__(self, path):
        self.path = path
uv run pytest -n 2 --dist loadgroup -q /home/agent/tmp/xdist_version_repro/test_collection.py

Expected: both workers collect the same tests. Observed: exit 1 with Different tests were collected between gw0 and gw1; only gw1 had the two [5.0.0] cases.


linux-arm:
timeout-minutes: 45
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/run_tests/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ inputs:
required: false
description: "Install and preload the prebuilt memtest library"
default: "false"
pytest-workers:
required: false
description: "pytest-xdist worker count; 'auto' matches the runner's cores"
default: "auto"
runs:
using: "composite"
steps:
Expand Down Expand Up @@ -42,4 +46,4 @@ runs:
- name: Run python tests
shell: bash
working-directory: python
run: make test
run: make test PYTEST_WORKERS=${{ inputs.pytest-workers }}
8 changes: 4 additions & 4 deletions .github/workflows/rust-benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ jobs:
runs-on: warp-ubuntu-latest-arm64-8x
timeout-minutes: 120
steps:
- name: Apt-get
run: |
sudo apt update
sudo apt install -y protobuf-compiler
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Run linalg benchmarks
Expand Down
90 changes: 49 additions & 41 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@ jobs:
timeout-minutes: 30
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Check documentation
run: RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps

Expand All @@ -81,10 +81,10 @@ jobs:
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Get features
run: |
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | sort | uniq | paste -s -d "," -`
Expand Down Expand Up @@ -122,10 +122,10 @@ jobs:
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
with:
key: llvm-cov-ci
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # cargo-llvm-cov
with:
Expand Down Expand Up @@ -167,10 +167,6 @@ jobs:
# Temporary mitigation for https://github.com/rust-lang/rust/issues/159261.
rustup toolchain install nightly-2026-07-13 --component llvm-tools-preview
rustup default nightly-2026-07-13
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y libssl-dev
- name: Start DynamoDB and S3
run: docker compose -f docker-compose.yml up -d --wait
- name: Install cargo-llvm-cov
Expand Down Expand Up @@ -284,10 +280,10 @@ jobs:
rustup default stable
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
- name: Install dependencies
run: |
sudo apt -y -qq update
sudo apt install -y protobuf-compiler libssl-dev pkg-config
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Build tests
run: |
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | grep -v -e protoc | sort | uniq | paste -s -d "," -`
Expand Down Expand Up @@ -315,10 +311,10 @@ jobs:
- name: Remove Cargo.lock
run: rm -f Cargo.lock
- uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Build all
run: |
ALL_FEATURES=`cargo metadata --format-version=1 --no-deps | jq -r '.packages[] | .features | keys | .[]' | grep -v -e protoc | sort | uniq | paste -s -d "," -`
Expand Down Expand Up @@ -346,15 +342,21 @@ jobs:
run: |
rustup update ${{ matrix.toolchain }} && rustup default ${{ matrix.toolchain }}
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
- name: Install cargo-nextest
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # nextest
with:
tool: nextest
# macOS runner capacity is the scarcest in CI, so this job is kept as short
# as it can be. nextest runs each test in its own process and schedules
# across all cores, rather than one test binary at a time. Benchmarks are
# only compile-checked here, which build-no-lock already does on Linux;
# nothing about bench compilation is macOS-specific.
- name: Build tests
run: |
cargo test --profile ci --locked --features fp16kernels,cli,dynamodb,substrait --no-run
cargo nextest run --cargo-profile ci --locked --features fp16kernels,cli,dynamodb,substrait --no-run
- name: Run tests
run: |
cargo test --profile ci --features fp16kernels,cli,dynamodb,substrait
- name: Check benchmarks
run: |
cargo check --profile ci --benches --features fp16kernels,cli,dynamodb,substrait
cargo nextest run --cargo-profile ci --features fp16kernels,cli,dynamodb,substrait
windows-build:
runs-on: windows-latest-4x
defaults:
Expand All @@ -372,12 +374,17 @@ jobs:
7z x protoc.zip
Add-Content $env:GITHUB_PATH "C:\protoc\bin"
shell: powershell
- name: Install cargo-nextest
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # nextest
with:
tool: nextest
# Same reasoning as mac-build: nextest schedules across all cores instead
# of running one test binary at a time, and build-no-lock already
# compile-checks the benchmarks on Linux.
- name: Build tests
run: cargo test --profile ci --locked --no-run
run: cargo nextest run --cargo-profile ci --locked --no-run
- name: Run tests
run: cargo test --profile ci
- name: Check benchmarks
run: cargo check --profile ci --benches
run: cargo nextest run --cargo-profile ci

qemu-pre-haswell:
# Verifies that lance-linalg's runtime SIMD dispatch still works
Expand Down Expand Up @@ -410,13 +417,14 @@ jobs:
with:
path: ${{ runner.temp }}/lance-qemu/8.2.10/qemu-x86_64
key: qemu-user-8.2.10-x86_64-linux-user-${{ runner.os }}-${{ runner.arch }}-v1
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Install QEMU build dependencies
if: steps.qemu-cache.outputs.cache-hit != 'true'
run: |
sudo apt update
sudo apt install -y \
ninja-build \
pkg-config \
Expand Down Expand Up @@ -485,10 +493,10 @@ jobs:
with:
submodules: true
- uses: Swatinem/rust-cache@779680da715d629ac1d338a641029a2f4372abb5 # v2
- name: Install dependencies
run: |
sudo apt update
sudo apt install -y protobuf-compiler libssl-dev
- name: Install protoc
uses: taiki-e/install-action@66068bfca13dcb2ea07c3f613ca2836a37c755d5 # protoc
with:
tool: protoc
- name: Install ${{ matrix.msrv }}
run: |
rustup toolchain install ${{ matrix.msrv }}
Expand Down
11 changes: 11 additions & 0 deletions python/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,17 @@ ifeq ($(CI), true)
PYTEST_ARGS += --durations=30
endif

# Number of pytest-xdist workers, e.g. `auto` or an explicit count. Empty (the
# default) runs serially; CI sets it per runner size. Much of the suite is spent
# waiting on IO rather than saturating the Rust core's own thread pools, so
# sharding across processes wins back time a single pytest process leaves idle.
PYTEST_WORKERS ?=
ifneq ($(strip $(PYTEST_WORKERS)),)
# loadgroup rather than the default scheduler so that tests sharing state on
# disk can be pinned to a single worker with @pytest.mark.xdist_group.
PYTEST_ARGS += -n $(PYTEST_WORKERS) --dist loadgroup
endif

help: ## Show this help
@awk 'BEGIN {FS = ":.*##"; printf "\nUsage:\n make \033[36m<target>\033[0m\n"} /^[a-zA-Z_-]+:.*?##/ { printf " \033[36m%-20s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)

Expand Down
2 changes: 2 additions & 0 deletions python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ tests = [
"polars[pyarrow,pandas]",
"psutil",
"pytest",
"pytest-xdist",
"tqdm",
"datafusion>=54,<55",
]
Expand All @@ -80,6 +81,7 @@ tests = [
"polars[pyarrow,pandas]==1.34.0",
"psutil==7.1.0",
"pytest==8.4.2",
"pytest-xdist==3.8.0",
"tqdm==4.67.1",
"datafusion==54.0.0",
"opentelemetry-sdk==1.30.0",
Expand Down
10 changes: 8 additions & 2 deletions python/python/tests/compat/compat_decorator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,8 +233,14 @@ def check_write(self):
# Write data
pass
"""
version = set([min_version, *VERSIONS])
versions = [v for v in version if Version(v) >= Version(min_version)]
# Sorted rather than taken straight off the set: set iteration order for
# strings depends on PYTHONHASHSEED, which differs per process, so every
# pytest-xdist worker would otherwise collect these parameters in its own
# order and xdist rejects the run as an inconsistent collection.
versions = sorted(
(v for v in {min_version, *VERSIONS} if Version(v) >= Version(min_version)),
key=Version,
)

def decorator(cls):
# Extract existing parametrize marks from the class
Expand Down
Loading
Loading