Recompiler: NVIDIA hangs (device lost) when read_bda_span lanes split between in-place loads and the fallback; branch and commit ready

On NVIDIA (RTX 3060, Windows Server 2025, driver 32.0.15.9186), a compute dispatch that reads guest memory through read_bda_span (the shared DontInline BDA dword read from 5623c0e) never finishes. This happens when its lanes split between the span's in-place dword loads and its per-dword byte fallback. Windows resets the GPU after about 3 s and the recorder's fence wait returns VK_ERROR_DEVICE_LOST.

On main this fails 10 tests on that machine: agc_driver_flat_loads, _x2_x3, _x4, agc_driver_global_loads, _x2_x3, _x4, agc_driver_flat_apertures_1, _2, _4 and agc_driver_pixel_interlock. It does not reproduce on lavapipe, so Linux CI passes. Any title shader that takes this path on NVIDIA can hit the same hang.

A ready fix, one commit, +5/−2 in core/shader/recompiler/SpirvBackend/src/SpirvBdaRead.cpp, with no code comments, TechnicalDebt changes or third-party code:

Branch: fdfawef324:fix/bda-span-read-split
Commit: the single commit at the branch tip, kept rebased onto main (first posted as 870b8bdf)

What

In DefineBdaSpanReadFunction, the in-place loads and the fallback were the two arms of one selection (EmitValueIfElse(state, wide, ...)). They are now two selections, joined after both: the loads under wide (mapped && aligned), then the fallback under !wide.

Every lane takes the path it took before.
Loads, lookups, fault records, Volatile and invocation stops are unchanged.
DontInline stays, so the compile-time win of 5693c77/5623c0e9 is kept.
The SPIR-V passes spirv-val --target-env vulkan1.2 before and after, so the old module was valid as well.

How it was narrowed down

All on the RTX 3060 with agc_driver_flat_loads:

Variant	Result
`main`	hang, ~3 s then device lost
`APS5_BDA_BYTE_READS=1` (no span function)	pass, ~1 ms
span function without `DontInline`	pass
fallback arm only (in-place arm removed)	pass
in-place loads replaced by byte loads	hang
in-place arm without loads	no hang
every mapped lane in place, only unmapped lanes fall back	pass
the fix	pass
the fix, with about half the lanes forced into the fallback	pass

The hang needs lanes in the in-place loads and lanes in the fallback's nested calls (read_bda_dword_bytes → get_bda_pointer) at the same time, inside the non-inlined function. These were each ruled out:

the 32-bit load and its alignment
the wrapped dword * 4 - firstBytes address of unused dwords
the stop variant's early returns
the probe call itself
the OBS and Steam implicit layers

A second problem it exposed

The device loss was hidden behind another error. The lost batch keeps its kept objects, so its lease on the test's guest block is never dropped. The test's teardown (GuestBlock::~GuestBlock → Mutation().Remove) then waits the 60 s APS5_PIN_WAIT_MS and throws still pinned by GPU work or an import while main unwinds, so the process aborts with 0xc0000409 and the device-lost message never prints. That is a separate change and is not in this commit.

Tested

Windows Server 2025, WinLibs GCC 15.2.0 r7, Release, RTX 3060 (driver 32.0.15.9186):

Full ctest: 765/765 pass, 5 skipped as before. The 10 tests above fail on main on this machine.
agc_driver_pixel_interlock: passes 5 of 5 repeats.
Serial times of agc_driver_flat_apertures_0–_5: 0.9–8.4 s.
Dreaming Sarah (PPSA02929): title screen at 60 FPS with the same log. The shader disk cache is invalidated by the new source version as expected.

Written with AI assistance.

A second NVIDIA data point. Windows 11 (10.0.26200), NVIDIA GeForce RTX 4070 Laptop GPU, driver 616.92, WinLibs GCC 15.2.0 r7, Release. I ran with VK_LOADER_DRIVERS_DISABLE=*dzn*, because the Dozen ICD on this machine crashes in its own threads (#3255). The verbose log shows the RTX 4070 as the selected device.

main 4be74d4f: all 355 agc_* / video_out tests pass, including the ten listed here (agc_driver_flat_loads, _x2_x3, _x4, agc_driver_global_loads and its variants, agc_driver_flat_apertures_1/2/4, agc_driver_pixel_interlock). So the hang does not reproduce on this GPU and driver. Your RTX 3060's 32.0.15.9186 is driver 591.86, so this may depend on the driver version as well as the GPU.
With 870b8bdf merged:
- those tests pass 3/3 (--repeat until-fail:3);
- ctest -R "^agc_|video_out|^shader|recompiler" passes 356/356.
So the split causes no regression here.

Follow-up, since Linux CI builds with ANYPS5_ENABLE_SPIRV_TOOLS=ON: I reran with that option on the same RTX 3060 (Windows, driver 32.0.15.9186), on main 1690db9.

ON, on main: the same 10 tests still end in device loss, so spirv-opt does not avoid the hang. Every other test passes (761/761).
ON, with the branch: all 15 BDA tests pass, the ten above plus agc_driver_bda_device, agc_driver_bda_span_reads and flat_apertures_0, _3 and _5. So the optimizer doesn't merge the two selections back into the hanging form. Run serially, agc_driver_flat_apertures_0 to _5 take 3 to 7 s each.

The branch is rebased onto main 1690db9. Its full ctest, merged with my other open branches, passes 771/771 apart from guest_pthread_rwlock and guest_pthread_start, which timed out only while a second build was running and pass when rerun alone.

Current main872 control comparison (2026-10-14 UTC)

Unmodified main872 and restored #4451 head365850f33, same Windows11/WinLibs15.2.0r7 Release/SPIRV-Tools ON checkout/build directory: the five existing controls (BDA device/span, wave32 exec-lane/subgroup, standard coverage) pass five repetitions on RTX5080/610.62 with GPL on and off, without skips. No span device-loss/timeout remains in this selected comparison; no full-suite or cause-bisect claim.

AMD1002:13c0/32.0.21036.18 reproduces span and both wave32 failures with GPL off on main and restored candidate; all three pass on. Span leaves byte32768 at0xcdcdcdcd instead of0x523f7ebd. This is a different vendor/signature from the reported NVIDIA hang. #4768 maintains the exact matrix and scope; the new dithered failure remains separately qualified. Shader cache disabled, Vulkan required, serial30s bounds; passes run5, failures once. AI-assisted: Codex.

Earlier RTX5080 residual comparison (2026-10-13)

Latest main075 recheck: #4451 head 0bcf45842, Windows 11/RTX 5080/610.62/WinLibs 15.2.0 r7/Release, shader cache off. Full NVIDIA: 1054 passed / 0 failed / 8 skipped of 1062, maximum 24.0818 s. No full-suite NVIDIA failure remains on this pin. BDA device/span/interlock each pass five serial repetitions with matched main and restored candidate, without control skips. All 15 changed core production/header/fixture files are replaced by exact main 075 (CMake registrations retained) for the matched run, then restored byte-exactly/full rebuilt. This is a matched production/fixture comparison, not a whole-main suite or a cause bisect. The older e514/2cc failure signatures below retain their original scopes. No attribution to the old proposed span split or driver-specific fix; no title/LTO/driver-cache-coldness claim. AI-assisted: Codex.

Additional main2cc comparison: #4451 rebase3d3ad6a92, Windows11/RTX5080/610.62/WinLibs15.2r7/Release, g++.exe/empty launcher, shader cache off. Full1039-test run:1029 pass,2 fail,8 skip, max24.406s. The residual pair is again BDA span reads/pixel interlock. Replacing all15 changed core production/header/fixture files with exact2cc main (CMake registrations retained), then byte-exact restoration/full rebuild, gives the same serial results in both controls: BDA device passes five repeats, span/interlock fail on their first execution. This time span reports vkWaitForFences recorder: Vulkan result -4, interlock vkDeviceWaitIdle: Vulkan result -4; the earlier e514 span timeout below retains its original scope. Original BDA/interlock fixtures, same directory/GPU/tools, GPL on with flag absent, Vulkan required, sequential repeat-until-fail5/timeout30. No root-cause, span-split, old-driver recurrence or full-green claim. AI-assisted: Codex.

Windows11 Enterprise26200, RTX5080/driver610.62, WinLibs GCC15.2r7 (c++.exe), Release/bundled dependencies/SPIRV-ToolsOFF. During the canonical #4474 rebase, base e51428bf23a531a0e575222f2833be90ce103096, the full1028-test suite has three failures: agc_driver_bda_device, agc_driver_bda_span_reads and agc_driver_pixel_interlock. The device fixture then passes×5 in isolated controls. The span/interlock pair fails on its first execution in each controlled run:

Production	BDA device	Span reads /pixel interlock
All three changed production files replaced with exact e514 main	5/5 pass	both fail
Exact restoration of rebase /full rebuild	5/5 pass	both fail

The BDA/interlock fixture sources are unchanged in this comparison. This is not a whole untouched-main checkout. The span run reports a Vulkan fence timeout/result2; interlock reports vkDeviceWaitIdle: Vulkan result -4. Vulkan log selection confirms RTX5080. Main/rebased comparison uses the same build directory, tools, GPU, shader-cache-disabled environment and sequential --parallel1 --repeat until-fail:5 --timeout30 campaign. Failing fixtures stop on their first run. No byte-read/GPL/LTO switch or driver-cache-clearing control was added to this campaign. The initial parallel-only BDA-device failure is not explained.

This records current reproducible residuals and limits attribution to #4474; it does not validate the proposed span split, identify lane divergence as the cause, or establish that the old closed driver-specific problem returned. Historical aperture/byte-read observations below retain their own artifacts and scope. No duplicate issue. AI-assisted: Codex.

RTX 5080 aperture observations; byte-read control remains failing (2026-10-11)

Additional data for distinguishing these failures on Windows 11, RTX 5080, NVIDIA driver 610.62, WinLibs GCC 15.2.0 r7, Release, ANYPS5_ENABLE_SPIRV_TOOLS=OFF, shader cache disabled. GPU selection is verified in the logs. The shader/AGC-driver sources exactly match pinned main ccf9ff9; this build also carries the four-file libc unlocked-I/O adaptation 2e12a9f4. I have not yet rerun on a completely unmodified-main libc or applied/tested branch #3289, so this is an observation, not confirmation of that fix or its root cause.

Full ctest -j12 (806 tests) reports 795 passes, 5 documented display/no-Vulkan skips and 6 failures. Aperture parts 0, 2, 3, 4, 5 fail; part 1 passes. In that parallel run, parts 2/5 printed vkDeviceWaitIdle: Vulkan result -4; other aperture failures end 0xC0000409.

A serial rerun reproduced all five failing aperture parts, about 69.94–70.37 s each, ending 0xC0000409 with still pinned by GPU work or an import. This corroborates the slow pinned-allocation teardown symptom described here. It does not establish why GPU work stopped progressing.

The suggested diagnostic control APS5_BDA_BYTE_READS=1 was then run serially for parts 0 and 2 with ANYPS5_REQUIRE_VULKAN=1, RTX 5080 selected and cache disabled. Both still timed out at an explicit 30 s harness limit, with recorded batch still running on the GPU after 5 s. No pass is claimed for the byte-read control. That differs from the control result on the RTX 3060 in this issue and leaves the relationship unresolved.

The sixth full-suite failure, agc_driver_f32_compare_constant, also fails in a serial run (0.402 s), but prints no comparison-mismatch diagnostic. I am tracking it separately and have not attributed it to this hang.

These are synthetic regression-test observations. Our PPSA99902 dump currently stops before guest entry at missing localeconv; it has not reached these GPU paths with the fresh conversion. No new issue or PR opened for these observations. AI-assisted investigation: Codex.

Newer pinned main90c observation (2026-10-11, `1642b5c`)

Additional full-runtime check on pinned main 90c152e plus four libc locale/test/debt files only, with no graphics source changes relative to that pin. Windows11/RTX5080/NVIDIA610.62 (queried), Release, Vulkan/MSAA required, shader cache off, SPIRV-Tools off; ctest -j4 --timeout30. Full build/122 libraries pass; 809/819 tests pass,5skip,5fail. Apertures0/2/3/5 time out at the global30s harness cutoff; aperture4 fails at10.48s. All passing tests<30s, max19.68s. The earlier sixth failure, agc_driver_f32_compare_constant, now passes in 6.933s on this newer pin. No single upstream change or root cause is isolated by that comparison.

The preserved older f609 integration with locale carry e22af77 separately passes823/828,5skip,0fail, including the five aperture groups; the newer f32_compare_constant test is absent there. Different base/runtime, so this does not establish a fix for this issue. Earlier serial/byte-read controls above retain their original ccf9 scope; not rerun as new90c controls. #3245/#3249 contain the libc validation details. AI-assisted tests/reporting: Codex; no new PR.

Newer-main Windows RTX5080 observations (2026-10-11, a73)

On main a73a1af9 plus focused min/max59c6d5e, full build/all122 configured libraries succeed. Windows/WinLibs15.2 r7/Release, RTX5080 explicitly selected, Vulkan/MSAA required, no shader cache, SPIRV-Tools disabled, full823-test ctest -j4 --timeout30: flat_apertures 0,2,3,4,5 each hit the30s harness cutoff. A newer-main math+locale integration full run has the same five cutoffs. agc_f32_compare passes in both runs. Separate Windows failures are reported on #3411 and #3371; full suite totals are810 pass/5 skip/8 fail and811/5/7 respectively.

Additional GPU observation: agc_driver_depth_fast_clear_submit fails in both full runs with vkWaitForFences graphics: Vulkan result2, followed by teardown recorder result-4. It passes a focused serial run0.344180s, but fails a serial control with only Math.cpp reverted to exact a73 source/native libc rebuilt9.79195s; all other test executables/libraries unchanged. This is variable and does not show a math-caused regression or establish the same root cause as the aperture timeouts. No timeout was increased, GPU fallback used or correctness bypass added. Logs are retained externally. AI-assisted validation/report: Codex.

Final prerequisite-integration run (2026-10-11, 5cd5ea)

Fork5cd5ea, maina73 with math/locale/validated kernel prerequisites, same Windows/RTX5080/no-cache controls: full824-test run 812 pass,5 skip,7 fail. Flat_apertures0/2/3/5 hit30s cutoffs; flat_apertures4 instead fails10.7034s after GPU batch-still-running5s/10s messages and vkDeviceWaitIdle: Vulkan result-4. Depth_fast_clear_submit fails23.3099s with fence result2/teardown-4. windows_exceptions again fails; Ampr and agc_f32_compare pass. Failure modes vary; no new cause/fix attribution. The separately controlled title now enters guest then refuses color mode0x1a (#2970); that loader/color milestone does not validate aperture behavior. AI-assisted report: Codex.

Newer-main full run and serial aperture recheck (2026-10-11,74981fa+3034273)

Fork3034273 adds only the separate D64 color proposal to main74981fa, now including upstream's intervening BDA span/host-import changes. Windows11/Winlibs GCC15.2r7/Release/RTX5080/no shader cache/Vulkan required: completed ctest --parallel2 --timeout30, 833 tests:822 pass,5 skip,6 fail,196.45s. Flat_apertures0/2/3/4/5 all time out; part1 passes. Windows unwind remains the other failure. Depth_fast_clear_submit,Ampr and the D64 CPU/GPU groups pass this full run.

Serial recheck --parallel1 --timeout30: flat_apertures0 times out again; depth_fast_clear_submit passes. This confirms that this aperture symptom still reproduces without concurrent test processes on this stack; it does not identify the stalled shader/driver cause or prove the upstream BDA changes ineffective generally. No unmodified-main full control on this base or root-cause bisect performed.

An earlier unbounded serial suite hung at flat_apertures4; its log is retained and that process stopped after checking its identity. The explicit-timeout completed run is the result above. All three relevant D64 groups each pass5repeats afterward, so no claim that every GPU operation is affected. AI-assisted validation/report: Codex.

Full newer-main integration observation (2026-10-11,0f863+d8fd7c7)

Windows11/Winlibs GCC15.2r7/Release/RTX5080/Vulkan required/no shader cache, forkd8fd7c7: completed ctest --parallel2 --timeout30, 835:824pass,5skip,6fail,196.62s. Flat_apertures0/2/3/5 time out;4fails after GPU batch-wait messages/device-lost teardown;1passes. windows_exceptions remains the other failure. Depth_fast_clear_submit and Ampr pass this run. The current narrow stack retains upstream BDA/coherence changes and omits our older tracker/MSAA changes; no fix/bisect attribution.

Separately, controlled title startup advances past D64 to #3036,5/5; that does not validate the aperture path. Math/CRT/three CPU-GPU color groups each5passes after the full suite. Original title/baseline preserved, fork-only/no PR. AI-assisted validation/report: Codex.

Another driver 591.86 data point, with the fix tested and its compile time measured.

RTX 4060 Laptop (Ada), driver 591.86, Windows 11, WinLibs GCC 15.2.0 r7, Release, SPIRV-Tools on, main a11c8f7:

main: agc_driver_flat_loads*, agc_driver_global_loads*, agc_driver_flat_apertures_2, agc_driver_flat_apertures_4 and agc_driver_pixel_interlock fail as described. The GPU goes idle while the batch never completes, the device is lost after about 20 s, and then the 60 s lease wait aborts.
With b54fe62 cherry-picked, these and agc_driver_flat_apertures_0–_5, agc_driver_bda_device and agc_driver_bda_span_reads pass (15 of 15).
Pipeline creation time, from a synthetic compute program with N global_load_dwordx4 read sites (one read_bda_span call each). The module is made unique per run so the driver cache misses, and the results are checked against the CPU:

	256 reads	512 reads
`main`	1.9 s	7.0–7.6 s
`b54fe62`	1.9 s	7.6–7.9 s
`read_bda_span` without `DontInline`	298 s	not run

So the split keeps the compile time 5623c0e bought.

I arrived at a different shape independently: the in-place dword loads go into their own DontInline function that makes no calls, and read_bda_span calls it. It also passes, and also costs nothing in compile time (1.95 s and 7.2 s). That fits your finding that the trigger is the in-place loads and the nested-call fallback in one selection. Yours is smaller, so I'm not opening another PR for it.

It also matters for #2440 (branchless buffer loads under robustBufferAccess, closed by the intake bot). On current main, that change makes agc_driver_flat_apertures_1 hang here on 591.86 as well, in a row whose lanes only touch LDS. With b54fe62 added, every agc_driver_flat_apertures_* part passes.

Independent Linux cross-validation of the ready fix in commit 3389172454cf063d1ba5f40008e22818ffeec0b8.

Setup: Linux kernel 6.14.0-37-generic, GCC 13.3 Release with SPIRV-Tools, RTX 5060 (Blackwell), NVIDIA 590.48.01. Baseline 2a5eca11cd1bf583db39ce67034e8453842ac244. No title files used.

The baseline flat-load fixture reproduces the pinned-allocation abort. APS5_BDA_BYTE_READS=1 passes. With the upstream +5/-2 SpirvBdaRead.cpp change, all nine originally failing span-read/pixel-interlock fixtures pass. The affected flat/global/aperture/interlock subset passes three serial repeats per successful fixture: 21 passed, 1 failed, no skips. AMD RADV cross-check passes all 22 affected fixtures.

The remaining NVIDIA failure is wave32_subgroup: a phase-labelled copy of the fixture passes its ordinary 32-wide compute target and fails the synthetic target forced to subgroupSize=64. That appears separate from the span-read hang; no root cause or fix is claimed for it.

Method limitation: I rebuilt only the upstream source file into a copied shader archive and isolated renderer library, using the baseline compile/link commands. The checkout and baseline binaries were not changed. A broad 967-test runtime cross-check found the nine GPU failures resolved, the wave32 failure remaining, plus two loader-test failures caused by the global patched/unpatched library override. Both loader tests pass without the override. This is independent runtime cross-validation, not a clean full build of the author's branch or a claim that its full suite passes.

AI-assisted verification with Codex

Thanks to the original author for the fix. Nothing here is a game FPS result.

fix(shader): split a BDA span read's in-place loads from its fallback

fix(shader): load a BDA span in place from its own function

fix(shader): split a BDA span read's in-place loads from its fallback

fix(shader): load a BDA span in place from its own function

Upstream sync 8D: shader recompiler slice (13 commits) (#470)

Navigation Menu

Repository navigation

Recompiler: NVIDIA hangs (device lost) when read_bda_span lanes split between in-place loads and the fallback; branch and commit ready #3291

What

How it was narrowed down

A second problem it exposed

Tested

Current main872 control comparison (2026-10-14 UTC)

Earlier RTX5080 residual comparison (2026-10-13)

RTX 5080 aperture observations; byte-read control remains failing (2026-10-11)

Newer pinned main90c observation (2026-10-11, `1642b5c`)

Newer-main Windows RTX5080 observations (2026-10-11, a73)

Final prerequisite-integration run (2026-10-11, 5cd5ea)

Newer-main full run and serial aperture recheck (2026-10-11,74981fa+3034273)

Full newer-main integration observation (2026-10-11,0f863+d8fd7c7)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Recompiler: NVIDIA hangs (device lost) when read_bda_span lanes split between in-place loads and the fallback; branch and commit ready #3291

Description

What

How it was narrowed down

A second problem it exposed

Tested

Activity

reyani-dev commented on Oct 11, 2026

fdfawef324 commented on Oct 11, 2026

Amoenus commented on Oct 11, 2026

Current main872 control comparison (2026-10-14 UTC)

Earlier RTX5080 residual comparison (2026-10-13)

RTX 5080 aperture observations; byte-read control remains failing (2026-10-11)

Newer pinned main90c observation (2026-10-11, 1642b5c)

Newer-main Windows RTX5080 observations (2026-10-11, a73)

Final prerequisite-integration run (2026-10-11, 5cd5ea)

Newer-main full run and serial aperture recheck (2026-10-11,74981fa+3034273)

Full newer-main integration observation (2026-10-11,0f863+d8fd7c7)

karanvora2599 commented on Oct 11, 2026

MojoisDev commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Newer pinned main90c observation (2026-10-11, `1642b5c`)