[QEC] Hardware CI for the realtime decoding server (NVQLink lab: DGX Spark + GB200) - #787
Closed
cketcham2333 wants to merge 11 commits into
Closed
[QEC] Hardware CI for the realtime decoding server (NVQLink lab: DGX Spark + GB200)#787cketcham2333 wants to merge 11 commits into
cketcham2333 wants to merge 11 commits into
Conversation
Hardware-in-the-loop CI for the decoding server on machines with real
ConnectX NICs, an FPGA syndrome source, and CC >= 9.0 GPUs -- none of
which GitHub CI has (its A100/L4 runners make every device_graph /
CUDA-graph path structurally unreachable).
New: docker/decoding-server/hw_ci/run_hw_ci.sh
Host-side runner: takes a cudaqx commit, clones it, builds the dev
image locally (layer-cached, never pushed), fail-fasts if the image's
baked CUDA-Q mismatches the commit's .cudaq_version pin (--build-base
builds the base locally for pre-merge pin bumps), then runs 23 lanes
sequentially with a PASS/FAIL/SKIP(reason) summary (exit-77 = SKIP;
--strict makes skips fatal). Lanes: the realtime_decoding_demo
matrix (4 decoders x udp / two-process cpu_roce / FPGA cpu_roce host
dispatch / FPGA device_graph), the two-process cpu_roce ctests, the
hsb_fpga decoding-server script over cpu_roce and gpu_roce, the
CC>=9.0-only ctests (qldpc-graph, mixed-dispatch, FP8 ONNX), and the
gpu_roce QLDPC bridge as a below-the-server cross-check. The Ising
TRT bundle is rebuilt from the gated HF model on every run
(ising-prepare lane; HF_TOKEN passed only to that lane's exec).
Proprietary artifacts (nv-qldpc plugin, cudevice archive) come from a
read-only bind mount; missing pieces surface as named SKIPs. CUDA
arch is auto-detected (GB200=100, Spark GB10=121). Two supported
configurations: DGX Spark with the loopback cable (--no-fpga
--roce-pair DEV0,DEV1; the single cable is FPGA XOR loopback) and
GB200 with the FPGA cabled (--roce-pair rxe --fpga-device DEV).
New: docker/decoding-server/dev.Dockerfile
Build+run environment on ghcr.io/nvidia/cudaqx-dev: RDMA userspace,
DOCA 3.3.0 gpunetio-dev, Holoscan SDK, TensorRT, cuStabilizer, the
Ising exporter's Python env (CPU torch) + hf CLI, and a prebuilt
holoscan-sensor-bridge 2.6.0-EA2 -- the build_qec.sh test-time
installs moved into cached image layers. cudaq-realtime (HSB=ON)
and cudaqx itself are built per-commit by hw_ci/container_build.sh,
not baked in. Known gap documented in the README: the base's
Mellanox ibverbs-providers has no rxe provider, so SoftRoCE lanes
currently SKIP.
Demo (docs/sphinx/examples/qec/realtime_decoding_demo):
- run_realtime_decoding.sh: new --page-size option (default 384).
The fixed 384-byte geometry made the device_graph mode refuse to
run on 16K/64K-page kernels -- i.e. on stock GB200 -64k images,
the advertised platform; the fail-fast now computes and suggests a
compliant value instead of dead-ending.
- Fix playback invocation: --hololink was silently ignored by
hsb_fpga_syndrome_playback after the #754 rename (it only parses
--hsb-ip), so every FPGA run of the example exited with usage.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…tion
Host-side validation of the hardware CI runner on a DGX Spark (cu13.0
dev image), run in BOTH cable configurations -- ConnectX loopback pair
and FPGA -- with proprietary artifacts and an HF token staged, surfaced
issues in the runner, the dev image, the container build, and several
never-before-executed test paths (cpu_roce two-process, gpu_roce/
device_graph GPU dispatch, ising/trt). Final state: loopback config
13 passed / 0 failed / 9 skipped (named); FPGA config 14 passed /
7 skipped (named) / 1 known FAIL (see below).
Runner (docker/decoding-server/hw_ci/run_hw_ci.sh):
- Resolve ibdev -> netdev via host sysfs during pair setup. The old
`[ -n ] && [ -n ]` guard never tripped under set -e (short-circuited
&& failures are exempt), so setup drove `ip` with empty netdev names.
- Re-assert the RoCE pair addresses and wait for the IPv4-mapped RoCE
GIDs before every cpu_roce lane: NetworkManager-managed ports drop
statically added addresses on NM's DHCP retry timer, which killed
lanes minutes after a successful setup ("no IPv4-mapped RoCEv2 GID
found"). README documents the permanent nmcli fix.
- Make --repo authoritative on every run (remote set-url before fetch);
a stale origin otherwise pinned all runs to the first-ever --repo.
- ctest lanes run with --timeout 900 so a wedged test is bounded well
below ctest's 1500 s per-test default.
- --hf-token-prompt reads the Hugging Face token from the terminal
(hidden input, never written to disk) for runs from shared/public
accounts, and tokenless runs fall back to a pre-built Ising bundle
staged at <artifacts-dir>/ising-bundle when present -- the trt lanes
then still run, and ising-prepare SKIPs with a reason noting the HF
download/export path was not exercised.
Dev image (docker/decoding-server/dev.Dockerfile):
- Install mlnx-ofed-kernel-utils (userspace only, from the DOCA repo):
it provides ibdev2netdev, which four in-tree --setup-network helpers
shell out to. Modern mlnx-tools no longer ships the tool, and the
cudaqx-dev base image never had it.
- Install patchelf: libs/qec's add_target_libs_to_wheel patches the
rpath of staged external decoder plugins at configure time and fails
(with an empty error) when patchelf is absent.
- Switch torch to the cu130 CUDA build: the pinned Ising-Decoding
exporter is GPU-only (its local_run.sh preflights
torch.cuda.is_available()), so the previous CPU-torch choice failed
the ising-prepare lane before inference. cu130 matches the image's
toolkit and runs on GB200 (sm_100) and Spark GB10 (sm_121).
- Build holoscan-sensor-bridge with DOCA_SEND_BLUE_FLAME=0: in-tree mlx5
drivers (e.g. the Spark's -nvidia kernel, no OFED) reject BlueFlame UAR
allocation, killing every gpu_roce lane at transceiver start
("Failed to create UAR: DOCA Driver call failure"); NONCACHE doorbells
are functionally equivalent for the CI lanes. Verified with a minimal
doca_uar_create reproducer: BLUEFLAME fails and NONCACHE succeeds on
both rdma-core 2601 and 2510, so this is kernel-driver behavior, not
userspace version skew.
Runner: the container now runs with --ulimit memlock=-1:-1 (matching the
long-lived dev container; --privileged does not raise ulimits).
Container build (docker/decoding-server/hw_ci/container_build.sh):
- Hand cmake a writable copy of the nv-qldpc plugin: /artifacts is
mounted read-only and the external-decoder install patches the
plugin's rpath in place. The decoder-plugins symlink then serves the
patched copy.
Tests (first cpu_roce execution of the two-process suites):
- DecodingServerTwoProcess.TwoProcessHostDispatchDualDecoders: add the
udp-only skip guard its sibling tests already have. Shared-wire demux
cannot work on rendezvous transports: a multi-decoder server opens one
ring per decoder and blocks in each ring's connect(), while this
caller dials a single endpoint -- the test hung to the ctest timeout.
- surface_code-1-cqr-two-process-test.sh: named SKIP (exit 77) for
NUM_LOGICAL > 1 off udp (device-scoped endpoint args exist only for
udp), and TERM -> KILL escalation in server shutdown: decoding_server
ignores SIGTERM while blocked in a rendezvous accept(), which
otherwise wedges the script's `wait` until the ctest timeout. That
server-side SIGTERM gap is a real cpu_roce robustness bug worth a
separate fix in the bridge/server bring-up.
- Register SKIP_RETURN_CODE 77 and TIMEOUT 300 for both two-process app
tests.
Validated end-to-end on the Spark, both configurations:
- Loopback: all four decoders (pymatching, multi_error_lut,
nv-qldpc-decoder, trt_decoder) PASS over udp and cpu_roce with full
dispatch/residual-error criteria; the two-process ctest lanes PASS
with the multi-decoder cases skipping by name.
- FPGA: all four decoders PASS on the FPGA source (SIF playback,
BRAM-verified), device_graph GPU dispatch PASSes (first run on GB10),
and hsb-fpga-server PASSes on both cpu_roce and gpu_roce. One known
FAIL remains: extra/gpu-roce-qldpc-bridge trips its ILA-capture
verification (a pre-arm reading of 194/500 samples suggests stale ILA
counter state between lanes) while the product paths it cross-checks
PASS at the server level -- tracked separately.
- ising-prepare PASSes (gated download + on-GPU export, 8 s); the trt
lanes PASS both from a fresh bundle and from a staged tokenless
bundle; the qldpc-graph and mixed-dispatch ctests register and PASS
with the proprietary artifacts staged.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
bridge-lane pacing Three follow-ups from continued Spark validation: - Drop the holoscan-sensor-bridge DOCA_SEND_BLUE_FLAME=0 build tweak. The gpu_roce UAR failures that motivated it were environmental, not a platform property: pointed at the FPGA-cabled port, the stock BlueFlame build passes every FPGA lane end-to-end. - README: the Spark FPGA invocation must name the port actually cabled to the FPGA (roceP2p1s0f0 on the current lab Spark). A wrong-but-linked --fpga-device still passes the HSB control plane and BRAM verification but fails with "ILA: captured 0 of N expected samples". - Pass --spacing 100 to the gpu-roce-qldpc-bridge lane: at the playback tool's default 10 us inter-shot spacing the ILA verification deterministically undercounts (194/500 on the Spark, reproducible across doorbell modes, ports, and reboots); at 100 us -- the same pacing the demo lanes use -- it captures 500/500 with all 100 shots verified and corrections matched. - Clarify the runner's build-phase messages: "Container build complete" read as if the docker image had just finished building; the messages now distinguish the layer-cached image build from the per-commit source build (cudaq-realtime + cudaqx + demo) that runs inside the container. With these, the FPGA configuration certifies clean on the Spark: every lane PASSes or SKIPs with a named reason, zero failures. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…age, OFED-compat rdma_rxe fallback On hosts whose ib_core comes from DOCA/MLNX-OFED DKMS (GB200 #2, 6.17.0-1008-nvidia-64k + DOCA-OFED 25.10), --roce-pair rxe had two blockers: - the distro rdma_rxe.ko cannot load against the OFED ib_core ("disagrees about version of symbol ib_*"), and OFED dropped the rxe driver from its source tree. Stage a patched upstream v6.17 rxe driver in the image (hw_ci/rxe-ofed/: three compat deltas + an iser-style external-module makefile); in rxe mode the runner mounts /lib/modules + /usr/src read-only, builds it against the host's ofa_kernel tree inside the privileged container, and insmods it -- only when the distro modprobe fails. - Mellanox ibverbs-providers ships only the mlx5 provider, and its provider ABI (rdmav59) differs from Ubuntu's rdma-core. Build librxe from the same Mellanox rdma-core source release (DOCA SOURCES bundle, sha256-pinned; the rxe provider is shipped there but if(0)-disabled in CMakeLists.txt) into the image. Validated on GB200: module loads against DOCA-OFED 25.10; provider opens rxe devices in-container. Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
- dev.Dockerfile: do not apt-install cmake for the rxe provider
build. noble's cmake 3.28 shadows the base image's cmake 4.x at
/bin/cmake, and the in-container source build (cudaq-realtime)
requires >= 4.0; rdma-core needs only >= 3.18.1, which the base
cmake satisfies.
- run_hw_ci.sh: bind-mount /dev/infiniband instead of --device.
--device snapshots char devices at container creation, so the
uverbs node of the rxe device that setup_roce_pair creates AFTER
container start never appeared inside, and ibv_devinfo reported
"IB device 'hwci_rxe0' wasn't found". A bind mount is live; the
privileged container loses no device-cgroup allowance.
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
…p dead lane and bring-up note GB200 validation finished at dd9ae05: 21 passed / 0 failed across udp, cpu_roce-over-SoftRoCE, and FPGA (mlx5_4), with ai-decoder-fp8 the only (permanent) skip. Follow-ups from that bring-up: - --roce-pair defaults to rxe (SoftRoCE self-loop); the old omitted-means-skip behavior moves to an explicit --roce-pair none - --fpga-device defaults to mlx5_4 (the GB200 lab wiring) - drop --repo: the commit under test is always cloned from the repo this script lives in, which also makes local unpushed commits directly testable - drop the ai-decoder-fp8 lane: the ctest is never registered until CUDAQX_QEC_ENABLE_REALTIME_PIPELINE returns upstream, so the permanent named skip carries no signal; re-add the lane when the pipeline lands - delete the GB200 bring-up handoff note, per its own instructions now that the machine is validated Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
cketcham2333
marked this pull request as ready for review
August 11, 2026 23:44
cketcham2333
requested review from
bmhowe23,
melody-ren,
tlshannon and
vedika-saravanan
August 11, 2026 23:44
Collaborator
|
Thanks Chuck! Tried this out on GB200 and it works. |
…eir model
Two users running the CI on one machine collided: the container name
was derived from the commit SHA alone (docker rm -f would kill the
other's run), and the FPGA, SoftRoCE objects (hwci-dummy0/hwci_rxe0),
and RoCE pair addressing are host-global regardless of SHA.
- take a host-wide flock on /tmp/cudaqx-hw-ci.lock for every real run
(--list is exempt); a second invocation fails fast naming the active
run's user/pid/sha, or queues behind it with --lock-wait
- the kernel releases the flock when the holder dies, so crashes can
never leave a stale lock; for a live-but-wedged holder (including an
orphaned child that inherited the fd) --force-unlock kills whatever
fuser says is actually holding the lock, with a sudo hint when the
holder belongs to another user
- container name gains $USER (hwci-$USER-$SHORT_SHA) so a
--keep-container debugging container survives someone else later
testing the same SHA
- rename the trt lanes to trt_decoder(ising): the plugin is the generic
TensorRT decoder, but the engine under test is the exported Ising
model, and the summary table should say so
- README: one-run-at-a-time + recovery documented; lane list clarified
Signed-off-by: Chuck Ketcham <cketcham@nvidia.com>
bmhowe23
approved these changes
Aug 13, 2026
cketcham2333
enabled auto-merge (squash)
August 13, 2026 22:38
bmhowe23
requested changes
Aug 13, 2026
| @@ -0,0 +1,72 @@ | |||
| # SPDX-License-Identifier: GPL-2.0 | |||
Collaborator
There was a problem hiding this comment.
We cannot check GPL-code into this repo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A self-contained hardware CI runner (
docker/decoding-server/hw_ci/) thatexercises the realtime decoding server on real RDMA/FPGA hardware. Given a
commit, it clones it, builds a layer-cached dev image
(
dev.Dockerfileon top of the pin-matchedcudaqx-devbase), compilescudaq-realtime + cudaqx + the demo binaries inside a privileged container,
and runs the test lanes. Every lane ends in the summary table as PASS,
FAIL, or a named SKIP, so lost coverage is always visible.
Lane coverage
udp, cpu_roce (two-process), and the FPGA syndrome source
hsb-fpga-serveron both wires (cpu_roce + gpu_roce), thegpu-roce-qldpc-bridge cross-check, and the GB200-class GPU ctests
unreachable in normal CI (qldpc-graph, mixed-dispatch)
bundle on-GPU each run (
--hf-token-prompt/HF_TOKEN/ staged-bundlefallback for tokenless machines)
SoftRoCE on DOCA-OFED hosts (GB200)
Two gaps blocked
--roce-pair rxe(the default) on hosts whoseib_corecomes from DOCA/MLNX-OFED DKMS, and both are handled automatically:
rdma_rxe.kocannot bind OFED'sib_core(symbol CRCmismatch) and OFED dropped the rxe driver, so the image stages a
3-line-patched upstream v6.17 rxe source (
hw_ci/rxe-ofed/); whenmodprobefails, the runner builds it against the host'sofa_kerneltree (mounted RO) and insmods it from the privileged container
ibverbs-providersships no rxe userspace provider and itsprovider ABI differs from Ubuntu's, so the image builds
librxefromthe matching Mellanox rdma-core source release (sha256-pinned)