Distributed NCCL tests fail: getBackend no longer returns a ProcessGroupNCCL

[shoumikhin]

What happens

Every multi-device TensorRT test fails as soon as it tries to use NCCL. The error is the same one 20 times:

RuntimeError: [Error thrown at core/runtime/TRTEngine.cpp:916]
Expected nccl_pg != nullptr to be true but got false

That check is Backend is not ProcessGroupNCCL, and it appears 74 times in a single job log. Result on a recent run: 20 failed, 71 passed in tests/py/dynamo/distributed/.

Which tests

• distributed/test_native_nccl.py — 19 of the 20
• distributed/test_export_save_load.py::test_export_save_load_round_trip — the remaining one, failing with the same NCCL error rather than an export error

Where it comes from

core/runtime/TRTEngine.cpp asks the process group for its NCCL backend and then casts it:

auto backend = pg->getBackend(c10d::ProcessGroup::BackendType::NCCL);
TORCHTRT_CHECK(backend != nullptr, "ProcessGroup '" << this->group_name << "' has no NCCL backend");
auto* nccl_pg = dynamic_cast<c10d::ProcessGroupNCCL*>(backend.get());
TORCHTRT_CHECK(nccl_pg != nullptr, "Backend is not ProcessGroupNCCL");   // line 916, this is what fails

The first check passes, so getBackend does return an object. The second fails, so whatever it returns is not a ProcessGroupNCCL. The cast yields null and the check trips.

Possibly relevant: the tests build their group with dist.new_group(ranks=[0]), a single-rank subgroup. That is the kind of group most likely to be backed by a wrapper rather than the concrete NCCL class.

It used to pass

Same test file, same Torch-TensorRT code path, two different runs:

	Torch	Result
#4464's own merge run, 2026-08-28	2.15.0.dev20260826/27+cu130	0 nccl_pg errors, 91 passed
a run today, 2026-09-02	2.15.0.dev20260902+cu132	74 nccl_pg errors, 20 failed

So this is not a change in core/runtime/TRTEngine.cpp. Nothing in that file moved between those runs.

What I could not determine

Two things changed between those runs, not one: the PyTorch nightly date and the CUDA channel (cu130 → cu132). I have no run that holds one fixed while moving the other, so I cannot say which is responsible.

This is harder to narrow than it sounds, because the distributed suite is lanes=("nightly",) in tests/ci/suites.py. It does not run on main pushes at all — main's latest commit has zero distributed rows. It only runs on a pull request carrying the ci: nightly label, which is why this surfaces on some PRs and is invisible on others.

I also ruled out one plausible explanation: an ABI mismatch between the Torch that libtorchtrt was built against and the one it runs with. A torch-2.14.0.dev20260728 string does appear in the job log, but it is a Bazel URL for an unrelated config, not the installed Torch.

Suggested next step

The missing fact is what type getBackend now returns. Logging that on cast failure would name it instead of leaving us to guess, and it is a small change to the error path:

TORCHTRT_CHECK(
    nccl_pg != nullptr,
    "Backend for ProcessGroup '" << this->group_name << "' is not ProcessGroupNCCL, got "
        << c10::demangle(typeid(*backend).name()));

With that in one CI run, the fix becomes obvious: either accept the new type, reach through the wrapper, or ask the group differently.

I deliberately have not changed the cast itself. Loosening it would quiet the check while possibly handing TensorRT a communicator that is not NCCL, which is worse than a failing test.

Reproducing

# needs a multi-GPU runner; the suite is nightly-lane only
cd tests/py/dynamo
python -m pytest -ra distributed/test_native_nccl.py

On a PR, add the ci: nightly label, otherwise the suite is skipped entirely.