Skip to content

[CUDA/CPU/WebGPU] Add state_window to LinearAttention and CausalConvWithState - #31157

Draft
tianleiwu wants to merge 2 commits into
mainfrom
tlwu/20260730/state_window_ops
Draft

[CUDA/CPU/WebGPU] Add state_window to LinearAttention and CausalConvWithState#31157
tianleiwu wants to merge 2 commits into
mainfrom
tlwu/20260730/state_window_ops

Conversation

@tianleiwu

Copy link
Copy Markdown
Contributor

Description

Adds an optional state_window attribute (default 0 = current behavior) to the
LinearAttention and CausalConvWithState contrib operators, implemented for CPU, CUDA and
WebGPU.

When state_window = W > 0, the recurrent state tensors gain a leading window axis:

op state shape, state_window = 0 state shape, state_window = W
CausalConvWithState (batch_size, channels, kernel_size - 1) (W, batch_size, channels, kernel_size - 1)
LinearAttention (batch_size, num_kv_heads, head_size_k, head_size_v) (W, batch_size, num_kv_heads, head_size_k, head_size_v)

Slot W-1 holds the state after the last token of the step; slot j holds the carry state after
the j-th token. W = 0 keeps the existing 3D/4D shapes, so this is backward compatible and no
existing model changes behavior.

Motivation and Context

These two ops carry a recurrent state, which makes them the blocker for multi-token
(speculative / MTP) decoding. A full-attention model can verify N+1 draft tokens in one forward
and then simply discard the rejected KV-cache tail. A linear-attention model cannot: after a
verify step the state reflects all N+1 tokens, so if only the first k are accepted there
is no way to recover the state at position k — the only correct fallback is to re-run the
recurrence over the accepted prefix.

With state_window, the op writes the intermediate per-token carry states into the window as it
sweeps, so the rejection path is a slot select instead of a recomputation. This is what allows
num_speculative_tokens = 3 (verify batch M = 4, recurrent_state_window = 4) to run on
Qwen3.6-35B-A3B, a hybrid model with 30 linear-attention layers out of 40.

This PR is enablement, so it has no standalone kernel speedup to report. For context, the
end-to-end payoff it unlocks, measured on 1x H200 SXM (SM90, CUDA 13.0) with
Qwen3.6-35B-A3B-NVFP4 + MTP N=3, CUDA graph on:

context ms/step decode tok/s tokens accepted / step
256 8.441 325.0 2.744
16384 9.567 283.9 2.716

Also included: a comment documenting that GQA's FlashDecoding fast-decode path is valid for
sequence_length >= 1 (its causal masking and split-KV reduction match regular FlashAttention),
plus multi-token decode coverage in test_gqa.py.

Tests

  • onnxruntime/test/contrib_ops/causal_conv_with_state_op_test.cc — new windowed cases.
  • onnxruntime/test/contrib_ops/linear_attention_op_test.cc — new windowed cases.
  • onnxruntime/test/python/transformers/test_gqa.py — multi-token (MTP-shaped) decode.

Windowed and non-windowed runs are compared against the same reference, so state_window = W
must reproduce slot-by-slot what W = 0 produces when stepped one token at a time.

Note

docs/ContribOperators.md still needs to be regenerated for the two schema changes
(python tools/python/gen_contrib_doc.py after a build). Keeping this as a draft until that
is in. It also touches group_query_attention.cc / bert_defs.cc / test_gqa.py, which
overlap #29904 — whichever lands second will need a rebase.

…ithState

Adds an optional `state_window` attribute (default 0 = legacy behavior) to the
LinearAttention and CausalConvWithState contrib operators. When state_window = W > 0
the state tensors gain a leading window axis:

  CausalConvWithState: (W, batch_size, channels, kernel_size - 1)
  LinearAttention:     (W, batch_size, num_kv_heads, head_size_k, head_size_v)

Slot W-1 holds the state after the last token; earlier slots hold the carry state
after each of the preceding tokens. This lets a multi-token (MTP / speculative
decode) step roll back to any accepted prefix without re-running the recurrence.
W = 0 keeps the existing 4D/3D shapes for backward compatibility.

Implemented for CPU, CUDA and WebGPU, with the shared window index math factored
into causal_conv_with_state_helper.h and linear_attention_helper.h.

Also documents that GQA's FlashDecoding fast-decode path is valid for
sequence_length >= 1 and adds multi-token decode coverage to test_gqa.py.

TODO before opening the PR: regenerate docs/ContribOperators.md
(build ORT python, then `python tools/python/gen_contrib_doc.py`).
@tianleiwu
tianleiwu force-pushed the tlwu/20260730/state_window_ops branch from 5b6d5db to 264c080 Compare July 30, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant