Repository navigation
fix(agc): order wave LDS accesses with memory barriers on MoltenVK - #10858
maximilianfv wants to merge 1 commit into
Conversation
SPIRV-Cross widens a barrier's execution scope to its memory scope, so the subgroup-scope OpControlBarrier with workgroup memory semantics that orders one wave's LDS accesses becomes a threadgroup_barrier. Emitted inside a branch only some waves take, those barriers paired with other waves' barriers, and agc_driver_lds_tree_reduction and agc_driver_lds_subword_store_branch read LDS before it was written. MoltenVK now takes the memory-barrier path AMD's proprietary driver already uses. Co-authored-by: Himansh Raj <67262056+iamthehimansh@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MarcZX10
left a comment
There was a problem hiding this comment.
Comment-only review of exact head 5138b1b1d25664561055b3b5b5810d9780dd93cb.
The failure mechanism is plausible and the change makes the two MoltenVK regression tests pass in the reported M1 Pro setup. I am not approving yet because the new VK_DRIVER_ID_MOLTENVK policy applies the fence-only path to every MoltenVK device, while the evidence is one M1 Pro and the debt entry itself says the path relies on 32-lane SIMD-group lockstep that Metal does not guarantee. Apple's guidance says SIMD-group size varies by GPU and advises querying it at runtime; it also distinguishes simdgroup_barrier (which waits for all threads in the SIMD group) from a memory fence. See Apple's SIMD-group and synchronization guidance.
Could you clarify whether SubgroupControlBarrierUsable is reached only when MoltenVK's reported Vulkan subgroup size matches the assumed guest wave size, and what evidence supports the fence-only lowering on other MoltenVK GPUs? If the current evidence only establishes M1 Pro, please consider limiting the policy to a supported subgroup/device condition or documenting the exact tested support boundary. I have not independently run the Metal/Vulkan tests; the three-run results and generated shader observations are author-reported.
AI-assisted review: yes.
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
Thanks. Answers from the code at this head ( When the fence-only path is reached. Only when
The host subgroup size is the device's reported Evidence. M1 Pro only (MoltenVK 1.4.2, macOS 27). I have no other Apple GPU and no Intel or AMD Mac to test on. The debt entry states this boundary and the lockstep assumption. What Metal gets. In the shaders dumped from the M1 Pro run, every new barrier is |
|
A Windows build and run of this head: Windows 11 Home 10.0.26200, WinLibs GCC 15.2.0 r7 (MinGW-w64 UCRT), Release.
AI-assisted: yes (Claude Code) |
jotitadev
left a comment
There was a problem hiding this comment.
A second Apple GPU (M2 Max) on a main-based tree, with both MoltenVK builds. The other runs here are Windows, where the change is a no-op.
Machine: Apple M2 Max, macOS 27.2, x86_64 Release build under Rosetta 2.
Tree: main b986c11a34 + the open macOS PRs needed to build and open a device (#10366, #7631, #10659, #10660, #11249, #11358, #10669, #10552, #7804, #10676, #11447, #11437, #11554, #11705) + this PR at 5138b1b1d2 (clean merge). Control: the same tree without this PR. ANYPS5_NO_SHADER_CACHE=1, -j1 --timeout 30.
Drivers: MoltenVK 1.4.2 (LunarG SDK 1.4.363.0) and MoltenVK main + SPIRV-Cross #2704.
| Without | With (3 runs) | |
|---|---|---|
agc_driver_lds_tree_reduction |
fails on both drivers: "round 0: thread 32 (wave 1) read 904, expected 32896" | passes 3/3 on both drivers |
agc_driver_lds_subword_store_branch |
passes, 6/6 on each driver | passes 3/3 on both drivers |
agc_subgroup_barrier |
passes | passes |
- The tree-reduction failure is the same value you saw on the M1 Pro (904), so that one reproduces on a second Apple GPU and on MoltenVK
main. - The subword store branch test did not fail here without the change (1 + 5 repeated runs per driver). On this machine it does not show the garbage reads, so it is weaker evidence for the fence-only path than the tree reduction.
- Full
^agcsuite, 1037 tests, MoltenVK 1.4.2: 41 failures without, 40 with.lds_tree_reductionis the only difference, so nothing new fails. - Validation layer (
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, SDK 1.4.363.0, loader output shows it inserted) on the three tests: noVUID-on either side.
GPUs tested: Apple M2 Max (MoltenVK 1.4.2 and MoltenVK main + #2704)
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
What
Under MoltenVK,
agc_driver_lds_tree_reductionandagc_driver_lds_subword_store_branchread LDS before another wave had written it. This makesSubgroupControlBarrierUsable(SubgroupBarrier.hpp) return false forVK_DRIVER_ID_MOLTENVKas well, so MoltenVK takes the subgroup-scopeOpMemoryBarrierpath that AMD's proprietary driver already uses (#8134).WorkgroupScopeFenceUsabledoes not change: MoltenVK keeps workgroup-scope fences, and the subword store test passes with them.Cause, from a MoltenVK shader dump: the recompiler orders one guest wave's LDS accesses with
OpControlBarrier(Subgroup execution scope, Workgroup memory scope) inside the branch afters_cbranch_execz. SPIRV-Cross'sCompilerMSL::emit_barriertakes the wider of the execution and memory scopes and emitsthreadgroup_barrier(mem_flags::mem_threadgroup). The tree-reduction dump has 22threadgroup_barriers for the guest's 5s_barriers. Waves that skip the branch pair their next barrier with another wave's extra one and read LDS before it is written: the tree reduction read 904, and the subword store read changing garbage. With this change the subgroup-scopeOpMemoryBarrierbecomes a simdgroup-scopeatomic_thread_fencewith no execution barrier. That relies on a 32-lane simdgroup running in lockstep, which Metal does not guarantee; the new debt entry "Wave LDS ordering on MoltenVK" indocs/dev/debt/shader.mdrecords this.The
agc_subgroup_barrierdriver table (tests/SubgroupBarrier.cpp) used oneusableflag for both functions. MoltenVK now differs between the two (false, true), so the flag is split intocontrolBarrierandworkgroupFence. Every other row keeps the old value in both fields.On Linux and Windows the generated code does not change: the driver ID is never MoltenVK there.
Prior art and credit:
#ifdef __APPLE__) in both files; this one is unconditional, because the driver ID alone already selects MoltenVK. It had code comments, which are dropped under the no-comments rule. It flipped the singleusableflag, butmainlater addedWorkgroupScopeFenceUsable, which shares the table (fix(shader): widen LDS atomic fences to device scope on AMD's proprietary driver #8489), so this version splits the flag instead. Credited with aCo-authored-by:trailer.SubgroupBarrier.hppor the subgroup barrier test. Neither does mugurc's macOS series.SubgroupBarrier.hpp: fix(agc): keep a DCC target's refresh proof while its keys are pending driver stores #10159 carries only theWorkgroupScopeFenceUsableaddition already onmainfrom fix(shader): widen LDS atomic fences to device scope on AMD's proprietary driver #8489, on an older base, and does not change MoltenVK's path. fix(agc): create tessellation pipelines without the pipeline cache on MoltenVK #10046 does not touch the header; itsVulkanDevice.cppdiff only shows theSubgroupControlBarrierUsablecall as context.Tested
Apple M1 Pro, macOS 27.0.1, x86_64 build under Rosetta 2, Vulkan through MoltenVK 1.4.2 from the LunarG Vulkan SDK 1.4.363.
maindoes not build on macOS by itself, so I tested on mugurc's macOS series (ci/macos-series-2, 27e9689) merged withmainat 522e32e. The tree also carried other open macOS fixes (#10419 build parts, #10540, #10562, #10641, #10642, #10650, #10651, #10652) and one of my unopened test fixes (agc_driver_graphics). Only this change was toggled between the runs. I tested before rebasing onto currentmain; the rebased diff is the same.Both sides:
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ctest --test-dir build -R '^(agc_subgroup_barrier|agc_driver_lds_.*)$' -E 'audio_in|voice' -V(16 tests), three runs each.agc_driver_lds_subword_store_branchandagc_driver_lds_tree_reductionfailed.was not found but was requested by env var VK_INSTANCE_LAYERSline, and both have 0Validation Errorlines.Not run on Linux or Windows: the generated code does not change there, and
agc_subgroup_barrieris a table check that does not depend on the host. Not tested with any title.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code