[CUDA/CPU/WebGPU] Add state_window to LinearAttention and CausalConvWithState - #31157
Draft
tianleiwu wants to merge 2 commits into
Draft
[CUDA/CPU/WebGPU] Add state_window to LinearAttention and CausalConvWithState#31157tianleiwu wants to merge 2 commits into
tianleiwu wants to merge 2 commits into
Conversation
…ithState Adds an optional `state_window` attribute (default 0 = legacy behavior) to the LinearAttention and CausalConvWithState contrib operators. When state_window = W > 0 the state tensors gain a leading window axis: CausalConvWithState: (W, batch_size, channels, kernel_size - 1) LinearAttention: (W, batch_size, num_kv_heads, head_size_k, head_size_v) Slot W-1 holds the state after the last token; earlier slots hold the carry state after each of the preceding tokens. This lets a multi-token (MTP / speculative decode) step roll back to any accepted prefix without re-running the recurrence. W = 0 keeps the existing 4D/3D shapes for backward compatibility. Implemented for CPU, CUDA and WebGPU, with the shared window index math factored into causal_conv_with_state_helper.h and linear_attention_helper.h. Also documents that GQA's FlashDecoding fast-decode path is valid for sequence_length >= 1 and adds multi-token decode coverage to test_gqa.py. TODO before opening the PR: regenerate docs/ContribOperators.md (build ORT python, then `python tools/python/gen_contrib_doc.py`).
tianleiwu
force-pushed
the
tlwu/20260730/state_window_ops
branch
from
July 30, 2026 21:12
5b6d5db to
264c080
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an optional
state_windowattribute (default0= current behavior) to theLinearAttentionandCausalConvWithStatecontrib operators, implemented for CPU, CUDA andWebGPU.
When
state_window = W > 0, the recurrent state tensors gain a leading window axis:state_window = 0state_window = WCausalConvWithState(batch_size, channels, kernel_size - 1)(W, batch_size, channels, kernel_size - 1)LinearAttention(batch_size, num_kv_heads, head_size_k, head_size_v)(W, batch_size, num_kv_heads, head_size_k, head_size_v)Slot
W-1holds the state after the last token of the step; slotjholds the carry state afterthe
j-th token.W = 0keeps the existing 3D/4D shapes, so this is backward compatible and noexisting model changes behavior.
Motivation and Context
These two ops carry a recurrent state, which makes them the blocker for multi-token
(speculative / MTP) decoding. A full-attention model can verify
N+1draft tokens in one forwardand then simply discard the rejected KV-cache tail. A linear-attention model cannot: after a
verify step the state reflects all
N+1tokens, so if only the firstkare accepted thereis no way to recover the state at position
k— the only correct fallback is to re-run therecurrence over the accepted prefix.
With
state_window, the op writes the intermediate per-token carry states into the window as itsweeps, so the rejection path is a slot select instead of a recomputation. This is what allows
num_speculative_tokens = 3(verify batchM = 4,recurrent_state_window = 4) to run onQwen3.6-35B-A3B, a hybrid model with 30 linear-attention layers out of 40.
This PR is enablement, so it has no standalone kernel speedup to report. For context, the
end-to-end payoff it unlocks, measured on 1x H200 SXM (SM90, CUDA 13.0) with
Qwen3.6-35B-A3B-NVFP4 + MTP
N=3, CUDA graph on:Also included: a comment documenting that GQA's FlashDecoding fast-decode path is valid for
sequence_length >= 1(its causal masking and split-KV reduction match regular FlashAttention),plus multi-token decode coverage in
test_gqa.py.Tests
onnxruntime/test/contrib_ops/causal_conv_with_state_op_test.cc— new windowed cases.onnxruntime/test/contrib_ops/linear_attention_op_test.cc— new windowed cases.onnxruntime/test/python/transformers/test_gqa.py— multi-token (MTP-shaped) decode.Windowed and non-windowed runs are compared against the same reference, so
state_window = Wmust reproduce slot-by-slot what
W = 0produces when stepped one token at a time.Note
docs/ContribOperators.mdstill needs to be regenerated for the two schema changes(
python tools/python/gen_contrib_doc.pyafter a build). Keeping this as a draft until thatis in. It also touches
group_query_attention.cc/bert_defs.cc/test_gqa.py, whichoverlap #29904 — whichever lands second will need a rebase.