Qwen3.8-27B NVFP4 base engine build fails: Myelin removeEmptyProducersFromSubgraph not implemented for PLUGIN_V3 (Jetson Thor, TRT 10.13.3.9 and 10.14.1.48)

[ruiqi14312]

Environment

• Jetson AGX Thor (sm_110), L4T R38.4 (JetPack 7.1-class), CUDA 13.0
• TensorRT-Edge-LLM 0.10.0 (bb29145)
• TensorRT tested: 10.13.3.9-1+cuda13.0 (JetPack repo) AND 10.14.1.48-1+cuda13.0 (sbsa CUDA network repo) — same failure on both

Repro

Checkpoint: NVFP4 Qwen3.8-27B (ModelOpt NVFP4, group_size 16, MTP extra tensors), e.g. https://huggingface.co/lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL

1. tensorrt-edgellm-export CKPT OUT --mtp → succeeds (after locally patching two exporter issues: (a) system scipy 1.8 vs numpy 2.x — from numpy import Inf ImportError when the venv uses --system-site-packages; (b) GDN fusion concatenating single-element weight_scale_2/input_scale buffers into a 4-element vector, which TRT's ONNX parser rejects: importTRT_FP4DynamicQuantize: Scale input must be a scalar — fixed by keeping numel==1 buffers unconcatenated in fuse_gdn_input_projections).
2. llm_build --onnxDir OUT/llm --engineDir ENG --maxBatchSize 1 --maxInputLen 4096 --maxKVCacheCapacity 32768 --maxVerifyTreeSize 4 --specBase →

[ERROR] IBuilder::buildSerializedNetwork: Error Code 1: Internal Error
(Not implemented for node type PLUGIN_V3. In removeEmptyProducersFromSubgraph
at optimizer/myelin/rewrite/removeEmptyTensors.cpp:257)   // TRT 10.13
... removeEmptyTensors.cpp:306                              // TRT 10.14

3. The MTP draft engine (--specDraft on OUT/mtp_draft) builds fine on both TRT versions. The draft ONNX contains only trt_edgellm::AttentionPlugin x1; the base ONNX additionally has trt_edgellm::gated_delta_net x48 and trt_edgellm::causal_conv1d x48, so the GDN plugins appear to be the PLUGIN_V3 nodes Myelin cannot handle.
4. The experimental direct (ONNX-less) builder fails identically at the same Myelin pass, so this is not ONNX-path specific.

Questions

• Which TRT version was Qwen3.8-27B Day-0 support validated against on Thor? Does it require JetPack 7.2 (CUDA 13.2 / TRT 10.16)?
• Is there a builder flag/env to route GDN plugin nodes around the failing Myelin rewrite pass?

[ruiqi14312]

Update: TRT 10.16.1.11-1+cuda13.2 (from the CUDA network repo, ubuntu2404/sbsa) fixes the build failure. The same ONNX and the same llm_build flags get past the Myelin pass on 10.16 and the engine compiles successfully (base + MTP draft, both build and run). So the PLUGIN_V3 removeEmptyProducersFromSubgraph limitation is resolved somewhere between 10.14.1 and 10.16.1.

Two notes from the investigation:

1. Build-time host memory: compiling the 27B NVFP4 base engine peaked above ~88 GB RSS (OOM-killed at 122 GB RAM with another ~20 GB in use). It only succeeded after stopping other GPU processes and adding swap. Worth documenting as a build requirement for the dense 27B on Thor.
2. Separately, the third-party checkpoint I started with (lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL) turned out to produce garbage text in BOTH TensorRT Edge-LLM and a reference HF transformers load — i.e. its quantization is corrupt, independent of this issue. A different community NVFP4 checkpoint of the same model family (hwkranger/Qwen3.8-27B-heretic-ara-NVFP4) exports, builds, and generates coherently with MTP acceptance ~2.75.

Remaining question: with the official MTP config (3 draft tokens / 4 verify positions), we measure ~18 tok/s decode on Qwen3.8-27B NVFP4 (acceptance 2.75, base verification 102 ms/iter, clocks locked via jetson_clocks, 120W mode) vs the published 37.6 tok/s for Qwen3.5-27B NVFP4 MTP (acceptance 2.84). Is the 2x gap expected between Qwen3.5-27B and Qwen3.8-27B (layer count / GDN config), or is there a known runtime tuning (CUDA graphs, tactic sources) we are missing?

[fffiore]

Encountered the same issue. The cause may be setting 0 as the static/max dimension in the optimization profile for dynamic inputs (e.g. spec_verify_phase_marker) connected around PLUGIN_V3 nodes. TRT 10.x's Myelin empty tensor elimination pass crashes when it encounters PLUGIN_V3.

Fixing the input profile shape range to min=0, max=1 (or ensuring max dim > 0) instead of a fixed 0-dim tensor resolves this crash and builds successfully.

In cpp/builder/llmBuilder.cpp, the spec_verify_phase_marker optimization profile was set to a static shape of 0 in the context/generation profile:

// Buggy code in setupLinearAttentionSpecVerifyProfiles:
result &= setOptimizationProfile(
    &contextProfile, binding_names::kSpecVerifyPhaseMarker, createDims({0}), createDims({0}), createDims({0}));
result &= setOptimizationProfile(
    &generationProfile, binding_names::kSpecVerifyPhaseMarker, createDims({0}), createDims({1}), createDims({1}));

[ruiqi14312]

Confirmed — that is the exact code path. \llmBuilder.cpp:1538\ sets the context profile for \kSpecVerifyPhaseMarker\ to {0},{0},{0}, and our base ONNX (48 \ rt_edgellm::gated_delta_net\ + 48 \causal_conv1d\ PLUGIN_V3 nodes) crashed Myelin on TRT 10.13.3.9 and 10.14.1.48, while the MTP draft ONNX (no GDN plugins) built fine on both. Your min=0/max>0 explanation is consistent with everything we observed.

A second data point for the thread: upgrading system TensorRT to 10.16.1.11-1+cuda13.2 (CUDA network repo, ubuntu2404/sbsa) also resolves it — same ONNX, same flags, build succeeds with the 0-dim profile, so 10.16's Myelin tolerates this case. If we ever need to build on 10.13/10.14 again we will apply your profile fix.

Still open from my original report: the runtime performance question — ~18 tok/s decode on Qwen3.8-27B NVFP4 MTP (acceptance 2.75, base verify 102 ms/iter, jetson_clocks, 120W) vs the published 37.6 tok/s for Qwen3.5-27B. Is that gap expected for Qwen3.8 (layer count / GDN config), or is there runtime tuning we are missing (CUDA graphs on the decode loop, tactic sources)?

[nvluxiaoz]

Thanks. I am glad that updating to 10.16 resolves it. I am doing some measurement of this issue.