Fix vmap axes for input_positions in dot_product_attention_with_rope - #5537
Open
TheSaiEaranti wants to merge 1 commit into
Open
Fix vmap axes for input_positions in dot_product_attention_with_rope#5537TheSaiEaranti wants to merge 1 commit into
TheSaiEaranti wants to merge 1 commit into
Conversation
Passing input_positions crashed with "vmap got inconsistent sizes for array axes to be mapped": the heads-axis vmap used in_axes=(-2, -1), mapping the positions' last axis, which is seq_len rather than a heads axis, and out_axes was a 2-tuple for a function returning a single array. Positions are shared by every head, so broadcast them with in_axes=(-2, None). The single vmap also covers the input_positions=None case, replacing the branch. Add a regression test: explicit arange positions match the default, a uniform shift preserves the output (RoPE encodes relative position), and non-uniform positions change it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes the
input_positionsargument ofdot_product_attention_with_rope(added in #5481), which crashes on every call that passes it:The heads-axis vmap used
in_axes=(-2, -1), mapping the positions' last axis — which isseq_len, not a heads axis — so any call withnum_heads != seq_lenraises, andout_axes=(-2, -1)was a 2-tuple for a function returning a single array. Both docstrings (dot_product_attention_with_ropeandMultiHeadAttention.__call__) documentinput_positionsof shape(..., seq_len,), and there was no test coverage for it.Positions are shared by every head, so the fix broadcasts them:
jax.vmap(rope, in_axes=(-2, None), out_axes=-2). Sincein_axes=Nonealso handlesinput_positions=None, the single vmap replaces the previous branch.Tests
test_dot_product_attention_with_rope_input_positionspins three properties:arange(seq_len)positions produce the same output as the default (they are the default),arange + 3leaves every pairwise difference unchanged, which distinguishes real broadcasting from positions being ignored per-head,arange * 2) change the output, proving positions take effect.The test fails on
main(with the vmap error above) and passes with the fix. Fulltests/nnx/nn/attention_test.py: 127 passed.ruff checkclean.Checklist