[CUDA] Port ShardedMoE to the new MoE GEMM backend

[tianleiwu]

Description

The MoE GEMM refactor (#28467) moved the CUDA MoE backend to contrib_ops/cuda/llm/moe_gemm/ and deleted contrib_ops/cuda/moe/ft_moe/moe_kernel.h together with ort_fastertransformer::CutlassMoeFCRunner. moe.cc/moe_base.h were ported to the new onnxruntime::llm::kernels::cutlass_kernels API, but sharded_moe.{h,cc} were not, so ShardedMoE still included the deleted header and any build with onnxruntime_USE_NCCL=ON failed to compile.

This PR ports ShardedMoE to the new MoE runner, shares the kernel invocation with the non-sharded MoE op, and fixes two correctness bugs in that shared path that were exposed while validating expert parallelism end to end.

Related PR: #31139 tried to remove the op to avoid build error.

Summary of Changes

Port ShardedMoE to the new MoE runner

File	Change
contrib_ops/cuda/moe/moe_base.h	Adds a protected MoEBase::RunMoe<T>() that holds the whole kernel invocation (routing, weight packing, workspace, runMoe) and takes an MOEParallelismConfig, a GEMM profiler and the output pointer. MoE and ShardedMoE now share one implementation.
contrib_ops/cuda/moe/moe.cc	Moves the kernel body out of MoE<T>::ComputeInternal into MoEBase::RunMoe<T>; ComputeInternal only validates inputs and calls it with a default (non-parallel) config. Adds explicit instantiations for float/MLFloat16/BFloat16.
contrib_ops/cuda/collective/sharded_moe.h	Drops the deleted ft_moe/moe_kernel.h include; keeps only the shard attributes plus a GEMM profiler and its mutex.
contrib_ops/cuda/collective/sharded_moe.cc	Rewritten on top of RunMoe<T>. Builds the MOEParallelismConfig from the parsed parallelism type (EP: ep_size = world_size, ep_rank = local_experts_start_index / local_num_experts; TP: tp_size = world_size, tp_rank = nccl_->Rank()), validates the sharding attributes, and combines the per-rank results with a single FuncAllReduce (ncclAllReduce SUM) over a temporary partial-output tensor. The old ncclBroadcast/get_total_rows_info combine and the ncclSend/ncclRecv path are gone.

The all-reduce combine is correct because the new runner already produces a partial result on each rank: finalizeMoeRoutingKernel skips experts outside [start_expert, start_expert + num_experts_per_node) and writes zeros for tokens with no local expert, and the FC2 bias is only added for locally routed experts.

Per-rank shape validation

File	Change
contrib_ops/cpu/moe/moe_helper.h	Weights and every tensor indexed by the weights' expert dimension (fc1/fc2/fc3 weights, scales, zero points and fc1_experts_bias) are now validated against local_num_experts instead of the global num_experts. fc2_experts_bias stays global because it is applied after the ranks are combined. Also enforces num_experts % local_num_experts == 0 when expert parallelism is inferred.

Gated-activation fixes in the shared MoE path

Both bugs affect the existing non-sharded MoE op too, and are the reason expert parallelism did not match the unsharded reference:

• Activation mapping. When a separate fc3_experts_weights input is present, FC1 and FC3 are packed into a single [E, 2 * inter_size, hidden_size] buffer, but the activation was still passed as the non-gated Silu/Gelu. The kernel then used inter_size as the per-expert stride of a buffer with 2 * inter_size rows per expert and read another expert's weights. Silu is now mapped to Swiglu and Gelu to Geglu in that case.
• FC1 bias stride. The gated GEMM indexes the FC1 bias with the same 2 * inter_size per-expert stride (bias_offset = expert * inter_size * gated_size_mul), so an [E, inter_size] bias aliased across experts. The bias is now padded into [E, 2 * inter_size] with the gate half holding the bias and the linear half zero.

Test

File	Change
test/python/transformers/sharded_moe/test_sharded_moe.py	THRESHOLD_EP relaxed from 1e-6 to 3e-2 (same as THRESHOLD_TP). Expert parallelism is no longer bit-exact: the sharded and unsharded sessions run the same CUTLASS kernels with a different number of experts per rank, so they may select a different GEMM tactic and accumulate in a different order.

Testing

Built with NCCL enabled (--use_cuda --enable_nccl --nccl_home ..., SM80) and validated on a 4×A100 node:

• mpirun -n 4 python test_sharded_moe.py — OK, 8/8 parity checks (EP and TP for hidden_size ∈ {128, 1024} × inter_size ∈ {512, 2048}, 64 experts, 1024 rows). Max observed diff is 1–2 fp16 ulp (0.000977 … 0.03125).
• A per-configuration sweep isolated the gated-activation bugs: fc3 + fc1_bias had a max diff of 3.735 before the fix and 0.000977 after; all other input combinations were unaffected.
• Non-sharded regressions on the same build: test_moe_cuda.py 48 passed / 15 skipped, test_qmoe_cuda.py 109 passed / 13 skipped.
• lintrunner -a clean.

Note for anyone reproducing the MPI test: ORT is not built with USE_MPI, so NcclContext bootstraps over TCP and each rank needs LOCAL_RANK, LOCAL_WORLD_SIZE, RANK0_IP=127.0.0.1, NCCL_IB_DISABLE=1, NCCL_SOCKET_IFNAME=lo.

Motivation and Context

• Fixes the onnxruntime_USE_NCCL=ON build break introduced by QMoE CUDA EP — FP4/FP8/WFP4AFP8 Quantized Mixture-of-Experts + MoE GEMM Refactor #28467.
• MoEParallelType::EPAndTP still returns "Expert and Tensor Parallelism is not supported yet", as before.
• FuncCustomAllReduce is intentionally not used: its CUDA IPC handle exchange in ipc_utils.cc is #ifdef USE_MPI, which no ORT CMake configuration defines, so it fails at runtime in every shipped build.