[Pipodi]

The mode is opt-in (APS5_SPANNED_WAVE=1) and off by default, so nothing changes unless it is set. The Ghost of Yōtei device loss reported earlier is found and fixed in a015284 (see The Yōtei device loss).

What

On a 32-lane host (NVIDIA) a wave64 compute program currently runs two guest lanes per invocation: every instruction is emitted twice, once per 32-lane half (laneCount == 2). For large programs this doubles the module and roughly doubles the driver's compile time. Ghost of Yōtei's loading compute programs reach 1.1–1.8M SPIR-V words and take 25–100 s each to compile cold.

This adds a spanned wave mode for compute programs whose workgroup is exactly one wave of 64 invocations. The program runs at one guest lane per invocation, the wave spans the workgroup's two 32-lane subgroups, and anything that crosses the two halves goes through workgroup memory:

• Lane numbering: lane = SubgroupId * 32 + SubgroupLocalInvocationId. The module declares a 64×1×1 workgroup, and the guest's LocalInvocationId/GlobalInvocationId are derived from the lane, the same way the two-lane form already does.
• Ballots (ctx.Ballot, EmitSubgroupBallot): each lane ORs its bit into a two-dword wave_exchange slot between workgroup barriers, so both halves see the 64-bit mask. FindLSB works over both words.
• Shuffles that stay inside a 32-lane half use a plain subgroup shuffle, with no barrier: DPP moves, v_permlane16/x16, the 32-lane permute, and ds_swizzle/ds_bpermute through EmitDsMaskedLaneRead. This goes through the new ctx.ShuffleInHalf and EmitSubgroupShuffleInHalf.
• Shuffles to an arbitrary lane (v_readlane, read-first-lane) go through workgroup memory: store, barrier, load.
• The LDS lock loop stays per subgroup. The loop that emulates 64-bit LDS atomics runs in divergent control flow, where a barrier would hang. So its elect and its "any lane still pending" ballot both use the host subgroup (EmitHostSubgroupBallot), and the lock is still exclusive with two electors.
• Half-wave reduction shortcut: EmitHalfWaveReduction is skipped; the generic v_readlane path handles it.
• Image tables: a spanned wave counts as one subgroup for tableIndexNonUniform, since its read-first-lane keys are wave-wide.
• BDA faults: a spanned program keeps every invocation running after a BDA fault and reads zeros, as programs with wave LDS barriers already do. Stopping one invocation would leave the other 63 waiting at the next exchange barrier.
• Switch: the driver reads APS5_SPANNED_WAVE once and sets SpirvTarget::spannedWaves, only on a device with 32-lane compute subgroup control and computeFullSubgroups. The field is in the cache key and the request serialization (version 32).
• Full subgroups: the lane number needs two full 32-lane subgroups, so the artifact carries fullSubgroups and the driver creates spanned pipelines with REQUIRE_FULL_SUBGROUPS and a required size of 32.
• Fallback: if the emitter refuses the spanned form (SpannedWaveUnsupported, a shuffle wider than 32 bits), the program is emitted again in the two-lane form. Other emit errors are not caught.

The two-lane form only ever branches on wave-uniform values, because both halves share one invocation. So the barriers added here are always reached in uniform control flow.

Measured

Offline (agc_compute_pipeline_replay, RTX 4090, driver 617.42, fresh driver cache per run, modules materialized from dumped Yōtei requests):

program (present variant)	two-lane	spanned
0x8000289300	76.3 s (1.07M words)	43.9 s (0.66M)
0x800022d100	49.6 s (1.19M)	24.5 s (0.71M)
0x80001fd400	13.7 s	8.4 s
0x8000305900 (654 shuffles, 556 ballots)	50.2 s	51.1 s

These spanned numbers were measured with a subgroup-ballot variant that is replaced here; see open item 1.

In game (Ghost of Yōtei, cold driver cache for every spanned module, earlier version of the mode): about 40 minutes with no errors or device loss. The giant programs' cold builds dropped from 14/183/68 s to 4/36 s (0x8000284e00), from 19/24/94 s to 7/10 s (0x8000289300) and from 25/25/59 s to 8/11 s (0x800022d100). The boot reached its slow loading phase 13.6 minutes earlier than the two-lane run.

Other hardware and titles, from the comments here: @ThatsLeo ran Stray (PPSA02101) on Intel Arc / ANV with the full suite and the game, and @fdfawef324 replayed 92 Demon's Souls (PPSA01342) requests (−27.7% SPIR-V words with the mode on). Neither saw a fault.

The Yōtei device loss

With the mode on, Ghost of Yōtei lost the device (NVIDIA TDR) in every run around frame 817. It looked tied to 0x80001e5f00's second variant, but that program has no LDS instructions. The program behind it is 0x80003af600. Its new variant was built right before every loss, and a loop guard caught a loop in it running past a million exits.

How it was found

• An offline harness ran the real 0x80003af600 request with its captured SRT regions and synthetic buffers. With random buffers over 64×36 groups, spanned lost the device with the same device-fault signature, while two-lane passed. Every other case gave bit-identical outputs in both modes.
• Narrowed to one workgroup, (7,4). It still hung with the loop guard at 4096, so it was a barrier deadlock, not a long loop.
• Ending the program early with s_endpgm at block starts put it after the first pass of an inner s_ff1/v_readfirstlane waterfall.
• In the spanned SPIR-V, 108 of 111 OpControlBarrier sit only inside loops. The other 3 sit inside per-invocation conditionals, lane active and LDS address in bounds. They are the spanned ballot of LockedLdsUpdate, the spin lock that emulates the program's ds_or_b64.

When a whole host subgroup has no active lane at that ds_or_b64, it never reaches the ballot's three barriers. The other subgroup then pairs them with unrelated barriers. The loop either leaves with a lane still pending, so an OR is lost, or it hangs. a015284 ballots the lock loop within its host subgroup.

• Regression test: LdsMaskWaterfall.cpp (6006d45, 641e53b) runs an LDS bitmask waterfall and an exec-masked ds_or_b64, two-lane and then spanned, setting the mode on the request itself. Without the fix (rebuilt on this branch), spanned fails with exec 0xffffffff00000000: lane 0 slot word 1 is 1, expected 65537. After it, both modes pass.
• Offline replay: the same 64×36 dispatch completes, with the same output hash as two-lane.
• In game (Yōtei on a base with this fix, spanned): a cold run went past frame 842 at 1320 s, and a warm run past 811 s, with no device fault. Both later ran out of VRAM after the difficulty screen, on a base without fix(agc): drop resource-cache entries that hold released sampled textures #6492 and our sampled-budget change.

The TechnicalDebt entry records the general limit: any other cross-lane exchange emitted inside per-lane control flow would hang a spanned wave the same way, and the emitter does not reject one. It also records how a program is checked today (spirv-dis, the constructs enclosing each OpControlBarrier, the condition of each enclosing selection). Only this program's SPIR-V has been checked. #6670 (@MarcZX100) fixes the same lock's initialization, a separate race.

Tested

Windows 11 Pro 26200, RTX 4090, driver 617.42. This branch at 5fb23f7 on main 65d56a2:

• Full suite: 1318 of 1319 pass. agc_driver_pixel_lds fails with a vkWaitForFences timeout in its wave32 pass, and fails the same way on main 65d56a2 without this PR. misaligned_writer_ahead is not built: main's GpuIndirectArgumentsMisalignedWriterAhead.cpp needs 65536-byte alignment, which MinGW's object format does not allow.
• lds_mask_waterfall: both modes pass. With the lock-loop ballot reverted to EmitSubgroupBallot, the spanned pass fails with spanned masked ds_or_b64, exec 0xffffffff00000000: lane 0 slot word 1 is 1, expected 65537.
• Cross-lane and LDS tests with APS5_SPANNED_WAVE=1 (the driver logs the mode as on): all 12 pass. They are lds_mask_waterfall, lds_atomics, lds_atomics64, lds_integer_atomics, dpp8, dpp8_pk_fmac, dpp16_row_share_xmask, ds_swizzle, permlane16, read_first_lane, look_back_scan and wave_uniform.
• Full suite with APS5_SPANNED_WAVE=1 (earlier base): 66 GPU tests run 357 spanned shaders, and all of them pass. These cover DPP, ds_swizzle, permlane16, read-first-lane, look-back scan, wave-uniform logic, LDS/GDS atomics, append/consume and scalar control flow. agc_shader_memory fails because it asserts the structure of the two-lane form.

Open items

1. Ballot cost. A ballot here takes three workgroup barriers and 64 LDS atomics. A cheaper version (subgroup ballot plus one word per subgroup, two barriers) made Yōtei lose the device within a minute. I haven't found out why yet; it may be another barrier-placement problem (recorded in TechnicalDebt).
2. Barrier check. The emitter does not reject a spanned exchange inside per-lane control flow; there is no uniformity analysis to build that on. Checked by hand for 0x80003af600 only.
3. Tests. A new test should compare the two-lane and spanned results of the same programs.
4. Pathological compiles. One Yōtei program (0x80001e5f00, 942 ballots) took 816 s to compile in the earlier spanned version. With this version its second variant compiles in 33–37 s.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt (none)
• No notes, investigation .md files or images added to the repository
• New third-party code is a submodule built from source (none)
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: PPSA26344 (faster cold boot; not yet measured from boot to spinner)
• GPUs tested: NVIDIA GeForce RTX 4090, driver 617.42, Windows 11 Pro 26200
• AI-assisted: yes

🤖 Generated with Claude Code

[ThatsLeo]

Intel data for this, since the numbers above are NVIDIA only. Fedora 44, Intel Arc (Meteor Lake iGPU, Mesa 26.2.3 ANV, 32-lane host subgroup), Stray (PPSA02101), head 9abe1c53d merged onto main 66f89476.

APS5_SPANNED_WAVE=1 works on ANV. The full suite with it set: 1108 of 1109 pass. The one failure is agc_shader_memory, which checks the two-lane shape of the SPIR-V it builds, so it fails by construction. None of Stray's 50 compute programs falls back.

It helps much more on ANV than the NVIDIA table suggests. Stray's post-processing programs 0x2010cf0000, 0x2010dd0000 and 0x2010de0000 (8x8x1, wave64, 6,060 to 12,576 code words) are what freeze it after the calibration screen on Intel: the queue sits in vkCreateComputePipelines. Created alone (Mesa cache off), the first one goes from more than 10 minutes and more than 13 GB of RSS, when I stopped it, to 19.4 s and 678 MB:

0x2010cf0000	SPIR-V words	ANV pipeline creation	peak RSS
two lanes	624,132	> 10 min (stopped)	> 13 GB
spanned	334,137	19.4 s	678 MB

The required 32-lane subgroup is what ANV can't handle. The second program, spanned, is 530,653 words and still didn't finish in 20 minutes. perf puts nearly all the time in Mesa's register allocator spilling one value per iteration (ra_add_node_interference, brw_reg_alloc::spill_reg): at SIMD32 each lane has a quarter of the registers it has at SIMD8. The same module created with a required size of 16 takes 5.6 s (478 MB), and with no required size (ANV picks) 2.2 s (240 MB).

In spanned form Stray's two largest programs use no subgroup operation at all, and nothing else in the mode needs the subgroup to be 32 wide except the lane number (SubgroupId * 32 + SubgroupLocalInvocationId) and the in-half shuffles. So I tried, locally, a variant where those two don't depend on the width either:

• the lane is LocalInvocationIndex (the spanned slot index already was LocalInvocationIndex & 63);
• EmitSubgroupShuffleInHalf goes through wave_exchange like the arbitrary shuffles, reading lane (lane & 32) | (target & 31);
• the artifact then reports host subgroup size 1, so the driver requires no subgroup size for it.

With that, every spanned Stray program has no OpGroupNonUniform* left (the three that aren't spanned keep theirs and their 32-lane requirement). The suite passes the same way on Arc and on lavapipe (the same agc_shader_memory failure). In game, Stray then compiles all three programs (the third is 1,012,298 words spanned, 1,925,789 in two lanes), gets past its calibration and main menu, and plays its opening cinematic with correct colours. The branch is ThatsLeo:spanned-any-subgroup (one commit on 9abe1c53d, switch APS5_SPANNED_ANY_SUBGROUP=1) if you want to take it into this PR. It is a prototype: the emitter reports the choice back through a thread-local, and the gating is an environment variable, and the in-half shuffles pay two workgroup barriers each, so it may be worth using only where the driver can choose a narrower width, or only for programs above some size.

AI-assisted: yes.

[fdfawef324]

Demon's Souls (PPSA01342), CPU-side only: 92 captured requests through agc_shader_replay, main 0ae1deb against 0ae1deb + 9abe1c5. Windows Server 2025, Ryzen 9 5900X, GCC 15.2 Release, SPIRV-Tools off, no shader cache, 3 interleaved rounds (median seconds per request).

	main	PR, off	PR, APS5_SPANNED_WAVE=1
--prepare pass	89	89	89
words total/median/p95	47.24M/349k/1.39M	same	34.14M/269k/940k (−27.7%)
seconds total/median	84.4/0.39	84.2/0.46	79.8/0.42
recompile pass	71	71	71
words total/median/p95	12.05M/39.7k/609k	same	7.34M/29.2k/351k (−39.1%)
seconds total/median	96.0/0.44	94.5/0.45	88.5/0.46

All three columns fail the same requests: 3 stencil-export replay refusals, and in recompile also 17 SRT descriptor-source evaluations and 1 invalid runtime image descriptor. The 17 SRT failures are registration-time preparation dumps that carry no dispatch user data, replayed as if they were dispatch captures (#6134 names them apart). They are not stops in the title.

Only the 78 compute programs with 64-invocation workgroups change. All of them get smaller (median ratio 0.70 for prepare, 0.64 for recompile), and none fall back. The 3 refused graphics requests go through the new fallback and are emitted twice (open item 3).

The seconds are within noise: main and switch-off differ by up to 3%.

Not tested: pipeline creation (no GPU, so the RTX 3060 was not involved), spirv-val, and running the game.

AI-assisted: yes

[adriadam10]

Thanks for the fix for the lock-loop deadlock. Reviewed on the diff; things to address before this can merge:

• LdsMaskWaterfall.cpp: the test never turns the mode on. APS5_SPANNED_WAVE is read from the environment in Recompiler.cpp:527 and the test registration doesn't set it, so in ctest/CI both loops run the two-lane form and the test passes without the fix. Make it exercise both modes explicitly (see the next point for how) and show it fails on 0843282d1/3cf9a9bbe without eccf0ce2a.
• Recompiler.cpp:527: spannedWaves changes the emitted SPIR-V but is not in CacheKey.hpp (~l.263) or RequestSerializer.cpp (~l.804), so the disk cache can serve a module built in one mode to the other, and replays can't reproduce it. Carry it as a SpirvTarget field set by the driver and include it in both (your open item 3), instead of a process-wide static getenv.
• SpirvEmitter.cpp:273: the spanned numbering SubgroupId * 32 + SubgroupLocalInvocationId assumes two full 32-lane subgroups. Without REQUIRE_FULL_SUBGROUPS (your open item 2) that isn't guaranteed (e.g. varying subgroup sizes on ANV), so either set the flag or only enable the mode when it's guaranteed.
• Recompiler.cpp:535: the fallback catches every std::exception; limit it to the spanned-specific failures so real emit errors still surface.
• Open item 1 (cheaper ballot loses the device for an unknown reason) points at a barrier-placement problem not yet understood; the debt line says any other cross-lane exchange inside per-lane control flow hangs. Please document how spanned programs are checked for that (the SPIR-V scan you did for 0x80003af600) or make the emitter reject a barrier outside uniform control flow.

[Pipodi]

Thanks for the review. Rebased on main 65d56a2 and addressed in 7a6cc71, 641e53b and 5fb23f7 (fix commit is now a015284):

• Test runs both modes. LdsMaskWaterfall now sets request.target.spannedWaves itself, two-lane then spanned, and checks result.fullSubgroups so a run that silently falls back to two-lane fails. The spanned pass is skipped (with a message) on a device without full 32-lane compute subgroups. Without a015284 (the lock-loop fix) the spanned pass fails: spanned masked ds_or_b64, exec 0xffffffff00000000: lane 0 slot word 1 is 1, expected 65537. With it, both passes succeed.
• Switch in the target. APS5_SPANNED_WAVE is now read once by the driver, which sets SpirvTarget::spannedWaves. The recompiler no longer calls getenv. The field is in CacheKey and in the request serializer (version 32, round-trip test added), and the disk-cache test checks that it changes the key.
• Full subgroups. The driver enables computeFullSubgroups where the device has it and turns the mode on only then, together with subgroupSizeControl for 32-lane compute. The artifact gets a fullSubgroups flag (disk cache and pipeline identity included), and spanned pipelines are created with REQUIRE_FULL_SUBGROUPS and a required size of 32. Spanned modules declare 64×1×1, so the X-multiple rule holds. Compute pipeline captures accept the flag only with a required subgroup size.
• Fallback. It now catches only SpannedWaveUnsupported, which the emitter throws for a shuffle wider than 32 bits, the one spanned-specific refusal. Every other emit error surfaces as before.
• Barriers outside uniform control flow. I did not add an emitter check: the emitter has no uniformity analysis to build it on. The TechnicalDebt entry now says how a spanned program is checked today (spirv-dis, list the constructs enclosing each OpControlBarrier, read each enclosing selection's condition), that it was done only for 0x80003af600, and that the emitter does not reject such a barrier. Open item 1 (the cheaper ballot that also lost the device) is recorded there as possibly another barrier-placement problem. If you'd rather have this enforced before merge, I can add a check on the finished module that refuses spanned emission when a barrier sits under a selection whose condition is not derived from scalar registers.

Tested on an RTX 4090: full suite 1318 of 1319 pass. agc_driver_pixel_lds times out in vkWaitForFences in its wave32 pass, and fails the same way on main 65d56a2 without this PR. The 12 cross-lane and LDS tests also pass with APS5_SPANNED_WAVE=1. Body updated.

[adriadam10]

Re-checked the new head against each point of the previous review:

• LdsMaskWaterfall now sets request.target.spannedWaves itself, runs two-lane then spanned and checks result.fullSubgroups.
• spannedWaves is a SpirvTarget field set by the driver, part of the cache key and of the request serialization (v32), with tests for both.
• fullSubgroups is carried in the artifact, the disk cache and the pipeline identity, and spanned pipelines are created with REQUIRE_FULL_SUBGROUPS.
• The two-lane fallback catches only SpannedWaveUnsupported.
• TechnicalDebt records how barriers are checked and that the emitter has no uniformity check.

Built and ran the Agc/AgcDriver tests on the merged tree: all pass except agc_driver_tessellation (RADV segfault in pipeline-library linking, unrelated to this change). Note the spanned pass of lds_mask_waterfall skips on RADV and lavapipe, so it only runs on 32-lane NVIDIA/ANV; it was covered by your RTX 4090 run. Merging once the full Build is green.