Skip to content

feat: enable SM103 (B300 / Blackwell Ultra) across the attention SM allowlists - #207

Open
filipemartinsubrobotics wants to merge 1 commit into
NVIDIA:mainfrom
filipemartinsubrobotics:feat/sm103-blackwell-ultra
Open

feat: enable SM103 (B300 / Blackwell Ultra) across the attention SM allowlists#207
filipemartinsubrobotics wants to merge 1 commit into
NVIDIA:mainfrom
filipemartinsubrobotics:feat/sm103-blackwell-ultra

Conversation

@filipemartinsubrobotics

@filipemartinsubrobotics filipemartinsubrobotics commented Sep 11, 2026

Copy link
Copy Markdown

Fixes #206.

SM103 (B300 / Blackwell Ultra) is absent from every SM allowlist on the build and
attention paths, so Edge-LLM cannot build an engine on that part at all — even
though the kernels it needs are the ones SM100 already uses. With the six
allowlist entries in this PR the part works: an INT4-AWQ engine builds and
generates correct output, as does FP16.

Symptom

On a B300, engine construction fails during ONNX parsing with no variant of the
attention kernels available:

[E] no variants support SM103

The failure surfaces as a plugin creation error during parse, which points away
from the cause.

Change

Six gates, across five files. Every changed line adds 103 (or __CUDA_ARCH__ == 1030) alongside the 100/101 entries already present; the only two added lines
that do not are clang-format rewrapping the two lists that grew past the column
limit. In build_cutedsl.py the same one-line edit repeats across the FMHA variant
declarations and the two int4_fp16_gemm generators (73 lines in total).

File Gate
kernelSrcs/build_cutedsl.py supported_sms on the CuTe DSL FMHA variants and on the int4_fp16_gemm GEMM/GEMV generators
cpp/kernels/contextAttentionKernels/cuteDslFMHARunner.cpp isSupportedBlackwellFmha
cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.cpp isFMHAV2SM
cpp/kernels/decodeAttentionKernels/decoderXQAJitCompiler.cpp kALLOWED_SM_VERSIONS
kernelSrcs/xqa/mha.cu Q/K/V shared-memory staging arch branch
kernelSrcs/xqa/utils.cuh kMAX_SMEM_SIZE arch branch

SM103 is grouped with SM100/101/110 throughout rather than given a branch of its
own: it shares their 227 KB shared-memory budget and the same tiled staging path,
so in each of these gates it belongs in the existing set.

decoderXQAJitCompiler.cpp is deliberately touched on one line only. Building an
engine on this part also needs the NVRTC include-path fix in #205, which changes
the same file; that is a separate defect and is not duplicated here.

Verification

Built and measured on a DGX B300 (SM103), CUDA 13.0, TensorRT 10.13.2, with #205
also applied, using an INT4-AWQ (W4A16, group 128) export of a 2.44 B Cosmos
reasoner:

  • CuTe DSL archive builds for sm_103 (compile_gpu_arch: sm_103a, CUDA 13.0.48):
    95 variants across int4_fp16_gemm;fmha.

  • Plugin registry loads 111 creators (before: no variants support SM103).

  • The XQA decode kernel JIT-compiles for the part:
    Compiled XQA kernel for SM103, dtype=1, kv_dtype=1, head_dim=128, q_heads_per_kv=2, tokens_per_page=128 ... with NVRTC (105944 bytes, 114 ms).

  • ONNX parse: 1965 layers, 0 errors; standalone engine build: 875.4 MB plan.

  • Through llm_build + llm_inference (the same tools a Jetson deployment uses),
    from an Edge-LLM ONNX export of that checkpoint: the engine builds, and
    INT4-AWQ inference produces correct output, 2/2 requests, no enqueue errors:

    "The rescue team should immediately assess the immediate hazards and provide
    cover for the victim while ensuring the responder is clear of any further
    dangers before providing aid."
    finish_reason: end-of-sequence

An FP16 control through the checkpoint builder on the same part is also correct
(405 tok/s, batch 1), so both precisions exercise the attention path these gates
enable — CuTe DSL FMHA for context, XQA for decode — and both are numerically
sound on SM103, not merely running.

A separate problem, not caused by this PR

Worth recording because I hit it while verifying, and it is not SM103-specific
as far as I can tell. Driving the same checkpoint through
experimental/builder/cli.py + experimental.server.LLM instead of the ONNX path
yields degenerate output (repeated tokens, unrelated fragments) on the same GPU,
while FP16 through that same builder is fine. What I checked:

  • Both INT4 plugin versions fail the same way (--int4-gemm-plugin-version 1 and 2),
    so it is not one kernel.
  • It fails with weights externalized and baked (--externalize-weights embedding),
    so it is not the runtime checkpoint binding.
  • The builder's own assembly is byte-identical to the working ONNX:
    repack_modelopt_awq(weight, 2) reproduces the ONNX weight initializer exactly
    (2,097,152/2,097,152 bytes), weight_scale.T matches to FP16 rounding, and
    pre_quant_scale matches exactly.

So identical INT4 weights produce correct results through the ONNX importer and
wrong ones through the programmatic builder graph. I have not isolated it further
and I only have SM103 to test on, so I am noting it here rather than filing a
diagnosis I cannot support; happy to open a separate issue if that is useful.

One build-configuration note for anyone reproducing this: ENABLE_CUTE_DSL
defaults to fmha, so an INT4 model needs
-DENABLE_CUTE_DSL="fmha;int4_fp16_gemm" explicitly. Without it the INT4 plugin
compiles with CUTE_DSL_INT4_FP16_GEMM_ENABLED undefined and enqueue returns
-1 from the #else branch with no diagnostic, surfacing only as
Failed to enqueue status -1. That is existing behaviour, not something this PR
changes, but it cost me a while to find and a one-line log there would help.

Two things I noticed but did not change. The gdn (4 variants), ssd (10) and
nvfp4_moe (7) groups in build_cutedsl.py still list 100 without 103; none
of them is on this model's path, so I left them rather than widen the patch to
kernels I cannot exercise.

And kALLOWED_SM_VERSIONS in
decoderXQAJitCompiler.cpp has no 110 entry, although isFMHAV2SM lists 110
and both XQA arch branches handle __CUDA_ARCH__ == 1100. That looks like the
same class of omission this PR fixes for SM103, but I have no SM110 part to test
on, so I have left it alone rather than guess.


Found during the NVIDIA / OpenHackathons / Oracle Open Models Codefest 2026,
while bringing up an INT4-AWQ Cosmos3-Edge reasoner on a DGX B300 for an
offline-first search-and-rescue robotics entry (Team UBR Stack). The production
target is a Jetson Orin Nano; the B300 is a bench machine used for evaluation.

…llowlists

SM103 is absent from every SM allowlist on the build and attention paths, so
engine construction on a B300 fails during ONNX parsing with "no variants
support SM103", even though the kernels it needs are the ones SM100 already
uses.

Add 103 (and __CUDA_ARCH__ == 1030) alongside the existing 100/101 entries in
six gates: the supported_sms lists of the CuTe DSL FMHA variants, the two
context-attention runner predicates, the XQA JIT SM allowlist, and the two
XQA arch branches that select the shared-memory budget and the Q/K/V staging
path. SM103 is grouped with SM100/101/110 rather than given its own branch: it
shares their 227 KB shared-memory budget and the same tiled staging path.

Verified on a DGX B300 (CUDA 13.0, TensorRT 10.13.2) with NVIDIA#205 also applied:
the sm_103 CuTe DSL archive builds 95 variants, the plugin registry loads 111
creators, an INT4-AWQ ONNX graph parses with 1965 layers and 0 errors, and the
engine builds to an 875.4 MB plan that deserializes cleanly. Numerical accuracy
and throughput on SM103 are not measured here.

Fixes NVIDIA#206

Signed-off-by: Filipe Martins <293984334+filipemartinsubrobotics@users.noreply.github.com>
@filipemartinsubrobotics
filipemartinsubrobotics requested a review from a team September 11, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SM103 (B300 / Blackwell Ultra) is missing from five SM allowlists across the build and the attention path

1 participant