feat: enable SM103 (B300 / Blackwell Ultra) across the attention SM allowlists - #207
Open
filipemartinsubrobotics wants to merge 1 commit into
Open
Conversation
…llowlists SM103 is absent from every SM allowlist on the build and attention paths, so engine construction on a B300 fails during ONNX parsing with "no variants support SM103", even though the kernels it needs are the ones SM100 already uses. Add 103 (and __CUDA_ARCH__ == 1030) alongside the existing 100/101 entries in six gates: the supported_sms lists of the CuTe DSL FMHA variants, the two context-attention runner predicates, the XQA JIT SM allowlist, and the two XQA arch branches that select the shared-memory budget and the Q/K/V staging path. SM103 is grouped with SM100/101/110 rather than given its own branch: it shares their 227 KB shared-memory budget and the same tiled staging path. Verified on a DGX B300 (CUDA 13.0, TensorRT 10.13.2) with NVIDIA#205 also applied: the sm_103 CuTe DSL archive builds 95 variants, the plugin registry loads 111 creators, an INT4-AWQ ONNX graph parses with 1965 layers and 0 errors, and the engine builds to an 875.4 MB plan that deserializes cleanly. Numerical accuracy and throughput on SM103 are not measured here. Fixes NVIDIA#206 Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #206.
SM103 (B300 / Blackwell Ultra) is absent from every SM allowlist on the build and
attention paths, so Edge-LLM cannot build an engine on that part at all — even
though the kernels it needs are the ones SM100 already uses. With the six
allowlist entries in this PR the part works: an INT4-AWQ engine builds and
generates correct output, as does FP16.
Symptom
On a B300, engine construction fails during ONNX parsing with no variant of the
attention kernels available:
The failure surfaces as a plugin creation error during parse, which points away
from the cause.
Change
Six gates, across five files. Every changed line adds
103(or__CUDA_ARCH__ == 1030) alongside the100/101entries already present; the only two added linesthat do not are clang-format rewrapping the two lists that grew past the column
limit. In
build_cutedsl.pythe same one-line edit repeats across the FMHA variantdeclarations and the two
int4_fp16_gemmgenerators (73 lines in total).kernelSrcs/build_cutedsl.pysupported_smson the CuTe DSL FMHA variants and on theint4_fp16_gemmGEMM/GEMV generatorscpp/kernels/contextAttentionKernels/cuteDslFMHARunner.cppisSupportedBlackwellFmhacpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.cppisFMHAV2SMcpp/kernels/decodeAttentionKernels/decoderXQAJitCompiler.cppkALLOWED_SM_VERSIONSkernelSrcs/xqa/mha.cukernelSrcs/xqa/utils.cuhkMAX_SMEM_SIZEarch branchSM103 is grouped with SM100/101/110 throughout rather than given a branch of its
own: it shares their 227 KB shared-memory budget and the same tiled staging path,
so in each of these gates it belongs in the existing set.
decoderXQAJitCompiler.cppis deliberately touched on one line only. Building anengine on this part also needs the NVRTC include-path fix in #205, which changes
the same file; that is a separate defect and is not duplicated here.
Verification
Built and measured on a DGX B300 (SM103), CUDA 13.0, TensorRT 10.13.2, with #205
also applied, using an INT4-AWQ (W4A16, group 128) export of a 2.44 B Cosmos
reasoner:
CuTe DSL archive builds for
sm_103(compile_gpu_arch: sm_103a, CUDA 13.0.48):95 variants across
int4_fp16_gemm;fmha.Plugin registry loads 111 creators (before:
no variants support SM103).The XQA decode kernel JIT-compiles for the part:
Compiled XQA kernel for SM103, dtype=1, kv_dtype=1, head_dim=128, q_heads_per_kv=2, tokens_per_page=128 ... with NVRTC (105944 bytes, 114 ms).ONNX parse: 1965 layers, 0 errors; standalone engine build: 875.4 MB plan.
Through
llm_build+llm_inference(the same tools a Jetson deployment uses),from an Edge-LLM ONNX export of that checkpoint: the engine builds, and
INT4-AWQ inference produces correct output, 2/2 requests, no enqueue errors:
An FP16 control through the checkpoint builder on the same part is also correct
(405 tok/s, batch 1), so both precisions exercise the attention path these gates
enable — CuTe DSL FMHA for context, XQA for decode — and both are numerically
sound on SM103, not merely running.
A separate problem, not caused by this PR
Worth recording because I hit it while verifying, and it is not SM103-specific
as far as I can tell. Driving the same checkpoint through
experimental/builder/cli.py+experimental.server.LLMinstead of the ONNX pathyields degenerate output (repeated tokens, unrelated fragments) on the same GPU,
while FP16 through that same builder is fine. What I checked:
--int4-gemm-plugin-version1 and 2),so it is not one kernel.
--externalize-weights embedding),so it is not the runtime checkpoint binding.
repack_modelopt_awq(weight, 2)reproduces the ONNXweightinitializer exactly(2,097,152/2,097,152 bytes),
weight_scale.Tmatches to FP16 rounding, andpre_quant_scalematches exactly.So identical INT4 weights produce correct results through the ONNX importer and
wrong ones through the programmatic builder graph. I have not isolated it further
and I only have SM103 to test on, so I am noting it here rather than filing a
diagnosis I cannot support; happy to open a separate issue if that is useful.
One build-configuration note for anyone reproducing this:
ENABLE_CUTE_DSLdefaults to
fmha, so an INT4 model needs-DENABLE_CUTE_DSL="fmha;int4_fp16_gemm"explicitly. Without it the INT4 plugincompiles with
CUTE_DSL_INT4_FP16_GEMM_ENABLEDundefined andenqueuereturns-1from the#elsebranch with no diagnostic, surfacing only asFailed to enqueue status -1. That is existing behaviour, not something this PRchanges, but it cost me a while to find and a one-line log there would help.
Two things I noticed but did not change. The
gdn(4 variants),ssd(10) andnvfp4_moe(7) groups inbuild_cutedsl.pystill list100without103; noneof them is on this model's path, so I left them rather than widen the patch to
kernels I cannot exercise.
And
kALLOWED_SM_VERSIONSindecoderXQAJitCompiler.cpphas no110entry, althoughisFMHAV2SMlists110and both XQA arch branches handle
__CUDA_ARCH__ == 1100. That looks like thesame class of omission this PR fixes for SM103, but I have no SM110 part to test
on, so I have left it alone rather than guess.
Found during the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026,
while bringing up an INT4-AWQ Cosmos3-Edge reasoner on a DGX B300 for an
offline-first search-and-rescue robotics entry (Team UBR Stack). The production
target is a Jetson Orin Nano; the B300 is a bench machine used for evaluation.