[CUDA] MatMulBlockQuantizedFp8Weight: fold the W8A8 activation QDQ into the decode GEMV

[tianleiwu]

Description

When MatMulBlockQuantizedFp8Weight is given an activation scale (the W8A8 path), it ran QuantizeDequantizeActivationFp8Kernel over the activation A into a scratch buffer, then read that buffer back in the GEMV. At decode shapes (m <= 8) A is tiny, so that round trip is almost entirely launch overhead and a pointless trip to memory.

This applies the same quantize-dequantize in registers, on the activation values the GEMV has already loaded. Fp8ActQdq16 mirrors the standalone kernel element for element, so the result is bit-identical.

Scope is limited to the m <= 8 GEMV fast path. The cuBLAS path is unchanged and still uses the standalone kernel, so the [M, K] scratch buffer is now only allocated when that path is actually taken.

Opt out with ORT_DISABLE_FUSED_FP8_ACT_QDQ=1.

Verification

Bit-identical to the previous two-step path, checked over 26112 elements across six shapes covering m in {1, 2, 4}, block_size in {16, 128, 2048}, and both the MMA and scalar GEMV kernels.

Motivation and Context

Measured on Qwen3.6-35B-A3B MTP (N=3) on H200: the standalone QDQ kernel disappears from the trace entirely, removing 40 launches/step and 54.3 us/step.

In isolation the win is largest at small N (-9.7% at N=512) and shrinks as N grows, because the fused kernel redoes the QDQ once per A-load rather than once per element. That trade is favorable exactly in the decode regime this fast path serves.

Builds on the decode GEMV added in #31155.

[Copilot]

Pull request overview

This PR optimizes the CUDA contrib op MatMulBlockQuantizedFp8Weight decode GEMV fast path (small M) by fusing the optional W8A8 activation quantize/dequantize (QDQ) into the GEMV kernels, avoiding a separate QDQ kernel launch and an [M, K] scratch-buffer round trip.

Changes:

• Add an optional act_scale parameter to the FP8 decode GEMV launcher and kernels, enabling in-register W8A8 activation QDQ during GEMV.
• Gate allocation/use of the activation scratch buffer so it’s only created for the cuBLAS GEMM path (or when fused QDQ is explicitly disabled via ORT_DISABLE_FUSED_FP8_ACT_QDQ=1).
• Add an environment-variable kill switch to disable the fused activation QDQ.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.h	Extends GEMV launcher API with optional act_scale and documents fused activation QDQ behavior.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cu	Implements the in-register activation QDQ helper and wires act_scale through scalar + MMA GEMV kernels.
onnxruntime/contrib_ops/cuda/math/matmul_block_scaled_fp8.cc	Adds kill-switch env var and avoids allocating activation scratch when fused QDQ is enabled for decode GEMV.