Use matrix kernels for global-scale gather_qqmm - #4481
Conversation
|
The tests are failing for CUDA backend, not sure of the reason, we can probably disable the test for CUDA for now and create an issue for it so we can look into it later. |
|
It seems that the CI failure is just a numeric precision issue, I think we can just allow for a bigger error in CUDA. |
|
I think it is happening, because we don't have proper global scaling for cuda.. In metal gather_qqmm always uses gather_qmm (because for metal we don't have quantized x quantized instructions). Because we plan to switch to |
50b106a to
5fc7dd0
Compare
|
Sounds good to me! |
5fc7dd0 to
8d19ced
Compare
This wires up the new gather_qmm kernels to optimize gather_qqmm with global scale.
Performance
Using the NVIDIA Model Optimizer on Qwen/Qwen3.6-35B-A3B with nvfp4_mlp_only p2048/g128