Skip to content

fix(shader): run tessellation evaluation stages one lane per invocation on devices without subgroups there - #13807

Merged
adriadam10 merged 2 commits into
boykopovar:mainfrom
frankea:fix/tes-single-lane-without-subgroups
Nov 1, 2026
Merged

adriadam10 merged 2 commits into
boykopovar:mainfrom
frankea:fix/tes-single-lane-without-subgroups

Conversation

@frankea

@frankea frankea commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

What

MoltenVK and lavapipe leave the tessellation evaluation stage out of subgroupSupportedStages. MoltenVK 1.4.2 on an Apple M5 Max reports TESSELLATION_CONTROL | FRAGMENT | COMPUTE (vulkaninfo; MVKDevice.mm adds only the control stage to fragment and compute), and lavapipe reports FRAGMENT | COMPUTE | TASK | MESH (lvp_device.c). The recompiler lays every tessellation evaluation program out for the device's subgroup width, so a domain program with lane operations declares GroupNonUniform and ValidateShaders refuses the draw: AGC graphics: subgroup capability 61 is unsupported for shader stage 4. #4470 handled the same refusal for the vertex stage.

Title: Gran Turismo 7 (PPSA01317) on an Apple M5 Max through MoltenVK stops with this error at the Used Car dealership's car showcase, about 870 s into a run, on its own domain programs. The new Tessellation.cpp draw reproduces that refusal on MoltenVK without this change and passes with it (see Tested); I have no GT7 run with this change that has reached the dealership yet.

Tested

Windows 11 on an RTX 4090 (driver 616.92, Vulkan 1.4.341), winlibs GCC 15.2.0, Release with BUILD_TESTING=ON. The 4090 has subgroups in the tessellation evaluation stage, so its draws keep the subgroup path; the one-lane path runs there only in verifySingleLaneDomainWave, which compiles for explicit targets.

  • 33 ctest entries pass 3 times in a row on this branch (53e4e8e796, and again at 32a6bdb6d3) and on main 833be4a: request_serializer, the 21 entries of agc_shader_memory_tests, agc_driver_tessellation, agc_driver_tessellation_depth_range_remap, agc_tessellation_pipeline_cache, agc_tessellation_factor_ring, agc_tessellation_lowering, agc_depth_range_remap, agc_driver_pixel_barycentric_emulation, agc_driver_mesh_barycentric_emulation, agc_driver_rect_list_corner, agc_driver_provoking_vertex and agc_driver_vertex_elected_atomic_max (skipped: the 4090 has vertex subgroups). They pass once more under VK_LAYER_KHRONOS_validation with ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1 (ctest -V) on both, with no VUID- lines and no was not found but was requested by env var VK_INSTANCE_LAYERS; the loader log for agc_driver_tessellation in that environment shows the layer inserted.

Apple M5 Max, macOS 27.0.1, MoltenVK 1.4.2 (Vulkan SDK 1.4.363.0), x86-64 under Rosetta 2:

  • request_serializer and the 21 agc_shader_memory_tests entries pass 3 times in a row on this branch, built natively. Two local build aids that are not part of this pull request were needed for that: the duplicate write_watch test registration removed as in test(libc): register the macOS write watch fault tests once #13220, and the GuestMemory.cpp part of build(agc): compile Lifecycle, GuestMemory and GuestBufferMemory on macOS #11447.
  • main does not build the AGC driver on macOS by itself, so the GPU check runs on my macOS tree (main 102b7a5 with the open macOS host pull requests and my open GT7 pull requests) at 48539c856b, which carries the same change (serialization version 52 there), with this pull request's Tessellation.cpp change applied. The new draw passes, and agc_driver_tessellation then stops at a later check, did not take its blue channel from the hull's relative patch index across instances, which fails the same way on that tree with neither the change nor the new draw. With only the HostSubgroupSize line removed, the new draw stops with AGC graphics: subgroup capability 61 is unsupported for shader stage 4, the error GT7 hits. Under VK_LAYER_KHRONOS_validation (inserted, per VK_LOADER_DEBUG=layer) the run with the change has no VUID- lines.

New checks:

  • verifySingleLaneDomainWave (tests/ShaderMemory.cpp): domain programs with lane operations and with a v_readlane_b32 waterfall loop declare no subgroup capability, built-in or instruction without tessellation evaluation subgroups, whatever the device's subgroup width, and still use subgroup operations with them; turning off vertex subgroups leaves a domain program on subgroups and turning off tessellation evaluation subgroups leaves a vertex program on them; a read of lane 31 throws; the flag survives serialization and changes the cache key. With the HostSubgroupSize line removed it fails with a domain program without tessellation evaluation subgroups declared a subgroup capability, built-in or instruction.
  • RequestSerializerTests.cpp: both values round-trip at version 51 as its last byte, a version 51 request without it is rejected as truncated, a version 49 request reads true and is written back as version 51 with the flag set, and a version 50 request is rejected in either layout (as is NVNQQTIAAAA= in ShaderMemory.cpp's unsupported-version list); both rejection checks fail with the version 50 check removed. The existing layout checks move by the one trailing byte.
  • DomainFirstLaneCount (tests/execution/Tessellation.cpp): DomainCode with its depth taken from v_cvt_f32_u32 of v_readfirstlane_b32 of v_mbcnt (0 at any wave width), drawn through RequirePatches. It runs on subgroups on the 4090 and one lane per invocation on MoltenVK; lavapipe in CI takes the one-lane path too, which I could not run here.
  • tools/check_conventions.py --base origin/main passes, and the branch merges cleanly with main 828ac0b, whose 19 new commits touch none of its files. tools/pr_overlap.py --branch fix/tes-single-lane-without-subgroups against main de83e8f finds only the 12 version 50 pull requests above; 51 pull requests that conflict with main themselves share files with this branch and could not be compared.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: PPSA01317 Gran Turismo 7 (PPSA01317) #4697 (Gran Turismo 7), domain programs with lane operations at the Used Car dealership on Apple GPUs through MoltenVK
  • GPUs tested: NVIDIA RTX 4090 (Windows 11, driver 616.92, Vulkan 1.4.341); Apple M5 Max (macOS 27.0.1, MoltenVK 1.4.2)
  • AI-assisted: yes (Claude Code)

…on on devices without subgroups there

MoltenVK and lavapipe leave the tessellation evaluation stage out of
subgroupSupportedStages, so the driver's shader validation refuses every
domain program that declares GroupNonUniform ("subgroup capability 61 is
unsupported for shader stage 4"). Gran Turismo 7 stops there on macOS at
the Used Car dealership.

The device target now records whether the tessellation evaluation stage
has subgroups. When it does not, tessellation evaluation programs outside
the mesh path get a host subgroup width of 1 and the one-lane rules that
vertex programs already use without vertex subgroups: the lane id is 0,
a ballot is the invocation's own bit, a shuffle returns the invocation's
value and an elect is true. Reads of other lanes still throw. Devices
with tessellation evaluation subgroups emit the same SPIR-V as before.
The request serialization moves to version 51.
@github-actions github-actions Bot added unblocks title Names a title ID on the Unblocks line shader core/shader docs agc libSceAgc* labels Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 7410 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7087, L7478
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L942, L951, L1010
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L363, L393, L527, L554-556, L574, L601, L630, L641, L771
PR 10547 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7067, L7087, L7092, L7249, L7411, L7478-7480
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L942, L951, L1010
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L328, L363, L372, L393, L401, L527, L538, L554-556, L574, L601, L612, L630-631, L641, L759, L771, L798, L813, L854, L862
PR 10846 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7067, L7087, L7092, L7249, L7411, L7418, L7430, L7478-7480
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L942, L951, L1010
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L328, L363, L372, L393, L401, L527, L538, L554-556, L574, L601, L612, L630, L636, L641, L690, L759, L771, L785, L798, L813, L854
PR 11351 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7067, L7087, L7092, L7249, L7411, L7478-7480
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L942, L951, L1010
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L328, L363, L372, L393, L401, L527, L538, L554-557, L601, L612, L630, L641, L759, L771, L798, L813-814, L854, L861-862
PR 11483 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7067, L7087, L7092, L7249, L7411, L7478-7480
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L942, L951, L1010
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L328, L363, L372, L393, L401, L527, L538, L554-556, L574, L601, L612, L630, L641, L759, L771, L798, L813-814, L854, L861-862
PR 11551 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L6820, L6941, L7062, L7087, L7247-7249, L7478
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp L901, L951
core/shader/recompiler/tests/RequestSerializerTests.cpp L319, L363, L393, L527, L554-556, L574, L601, L630, L641, L771, L853-854

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1395 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 2434 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4040 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4472 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5021 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 5371 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
docs/dev/debt/shader.md
PR 5609 core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 5925 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/shader/recompiler/Recompiler.hpp
PR 6188 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/shader/recompiler/Recompiler.hpp
PR 6235 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 6280 docs/dev/debt/shader.md
PR 6412 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
docs/dev/debt/shader.md
PR 6488 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 6804 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6858 core/shader/recompiler/Recompiler.cpp
PR 6924 core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 6970 core/shader/recompiler/Recompiler.hpp
PR 6977 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7223 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 7275 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7369 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7392 docs/dev/debt/shader.md
PR 7449 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7464 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7622 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7642 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/libs/prx/libSceAgcDriver/tests/execution/Tessellation.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 7684 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
docs/dev/debt/shader.md
PR 7689 core/shader/recompiler/Recompiler.hpp
PR 7694 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
PR 7922 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
docs/dev/debt/shader.md
PR 8004 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 8014 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 8939 core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 9176 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9183 docs/dev/debt/shader.md
PR 9287 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9289 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9356 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
PR 9387 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 9432 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9445 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9489 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9633 core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 9748 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 9749 core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 9790 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 9852 core/shader/recompiler/Recompiler.hpp
PR 9898 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9969 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 10069 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 10193 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/libs/prx/libSceAgcDriver/tests/execution/Tessellation.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 10233 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
docs/dev/debt/shader.md
PR 10351 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 10359 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
PR 10442 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 10652 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
docs/dev/debt/shader.md
PR 10720 core/shader/recompiler/Recompiler.hpp
PR 10757 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 10867 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
docs/dev/debt/shader.md
PR 10940 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 10941 core/shader/recompiler/Recompiler.cpp
docs/dev/debt/shader.md
PR 11435 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 11436 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 11554 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 11920 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12120 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12324 core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
PR 12536 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12662 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 12828 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 12855 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12904 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12980 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
core/shader/recompiler/tests/RequestSerializerTests.cpp
docs/dev/debt/shader.md
PR 13148 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 13159 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 13863 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 13879 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 13920 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 13928 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/libs/prx/libSceAgcDriver/tests/execution/Tessellation.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 13929 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
core/libs/prx/libSceAgcDriver/tests/execution/Tessellation.cpp
core/shader/recompiler/CacheKey.hpp
core/shader/recompiler/Recompiler.cpp
core/shader/recompiler/Recompiler.hpp
PR 14212 docs/dev/debt/shader.md

@oneandonlydean

Copy link
Copy Markdown
Collaborator

@frankea Reviewed at 53e4e8e796. On devices that report tessellation evaluation subgroups (NVIDIA, AMD) the generated SPIR-V is unchanged, and the extra cache-key word only rolls keys the way SourceVersion already does on every recompiler change. Two points:

  1. The new debt line says "a read of another lane throws", but SpirvAluEmitterMath.cpp only checks the v_readlane selector range when the selector is an immediate. A runtime selector passes ReadLaneEliminator (it skips non-immediates) and reaches EmitSubgroupShuffle, whose one-lane branch returns the invocation's own value whatever the selector is. So a domain program that reads lane 31 through a scalar register gets its own value instead of an error. The vertex one-lane path has the same gap, but until now these tessellation evaluation programs were refused on lavapipe and MoltenVK. Could the one-lane path throw for a selector it can't prove addresses the invocation's own lane (keeping the waterfall case that passes today), with a test that reads a lane-varying value through a runtime selector? If you'd rather leave that for a follow-up, please make the debt line say that only constant-lane reads throw.
  2. Version 50 is skipped, but Deserialize now accepts every version up to 51 and reads the new field only for 51, so a version-50 blob is read in the 49 layout. Rejecting 50, with a small negative test, would keep a stray blob from being misread.

Merges with #13341 and #13774 are clean in both orders. A run of the tessellation draw on a device without tessellation evaluation subgroups (the Linux lavapipe lane) would confirm it; it was skipped on this head.

…cked lane selectors

Version 50 is skipped, so a request with that number is refused instead
of being read in the version 49 layout.

The one-lane tessellation evaluation debt entry now states that only a
v_readlane_b32 of a constant lane throws: a lane selector held in a
register is not checked and reads the invocation's own value, as on the
vertex path.
@frankea

frankea commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

@oneandonlydean Thanks for the review. I pushed 32a6bdb6d3 on top of 53e4e8e796.

  1. I've left the runtime-selector check for a follow-up and made the debt line say what is checked: a v_readlane_b32 of another lane named by a constant throws, while a lane selector held in a register is not checked and reads the invocation's own value, as on the vertex path. Making the one-lane path throw unless it can prove the selector is the invocation's own lane also changes the vertex path from fix(recompiler): run vertex programs as one-lane waves without vertex subgroups #4470, so I'd rather do it in its own pull request, for both stages, keeping the s_ff1_i32_b64 waterfall and with the runtime-selector test you describe.
  2. Deserialize now rejects version 50 with unsupported recompile request serialization version. RequestSerializerTests checks a version 50 request in both the 49 and the 51 layout, and NVNQQTIAAAA= joins the unsupported-version list in ShaderMemory.cpp; both checks fail with the rejection removed.

The same 33 focused tests pass 3 times on the RTX 4090 at 32a6bdb6d3, and once more under VK_LAYER_KHRONOS_validation with no VUID- lines (layer inserted per the loader log). On the M5 Max, request_serializer and the 21 agc_shader_memory_tests entries pass 3 times at 32a6bdb6d3. I have no Linux lavapipe machine here. The device without tessellation evaluation subgroups I could run is MoltenVK on the M5 Max, where the new Tessellation.cpp draw passes with the change and stops on subgroup capability 61 is unsupported for shader stage 4 without it, so the lavapipe lane of a CI run would be the confirmation. I updated the description to match.

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 53e4e8e merged onto main 828ac0b · tested 2026-10-31 08:24 UTC · 25 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,328 passed · 5 failed (0 new, 2 also fail on main, 3 unstable on main) · 41 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,374 of 3,048 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (870 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_graphics_maps_fallback 33.7 s, ⚠️ agc_driver_bda_device 33.6 s, ⚠️ agc_driver_graphics 31.8 s
Intermittent validation messages (1, seen once in the parallel run, not reproduced when the test was re-run alone 2 times; not counted against this PR)
agc_driver_depth_clear_hle: VUID-vkCmdCopyBuffer-commonparent (6x)
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_depth_clear_hle: failed in 1 of the last 10 main runs
agc_driver_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 8 of the last 10 main runs
Failing on main too (2, not counted against this PR)
guest_mmap_executable
guest_socket_options_multicast
Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)
agc_driver_job_manager_compute_capacity: Failed in the parallel run; also passed only on retry in 158 other runs in the last 7 days
Validation IDs already present on main (3 IDs in 387 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (387 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, and 379 more
VUID-vkCmdCopyImage-srcImage-01548 (12 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 4 more
VUID-vkCmdCopyImage-srcImage-01551 (12 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 4 more
Skipped (41)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_labels: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is eithe
Reproduce
git fetch origin 53e4e8e7961030f0cee59e992b0b0cc1421f5cec 828ac0b2ae74aac181ba128fb59b08a1534f2041
git checkout 828ac0b2ae74aac181ba128fb59b08a1534f2041 && git merge --no-ff 53e4e8e7961030f0cee59e992b0b0cc1421f5cec
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@TechLevelZero

TechLevelZero commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ NVIDIA GPU test: passed

PR head 32a6bdb merged onto main 828ac0b · tested 2026-10-31 10:00 UTC · 26 min

GPU NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,328 passed · 5 failed (0 new, 2 also fail on main, 3 unstable on main) · 41 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 2,374 of 3,048 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (870 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_graphics_maps_fallback 33.7 s, ⚠️ agc_driver_bda_device 30.7 s
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 8 of the last 10 main runs
Failing on main too (2, not counted against this PR)
guest_mmap_executable
guest_socket_options_multicast
Validation IDs already present on main (3 IDs in 387 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (387 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, and 379 more
VUID-vkCmdCopyImage-srcImage-01548 (12 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 4 more
VUID-vkCmdCopyImage-srcImage-01551 (12 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 4 more
Skipped (41)
agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_labels: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is eithe
Reproduce
git fetch origin 32a6bdb6d3d69d084bc24fd915c67027f0efb216 828ac0b2ae74aac181ba128fb59b08a1534f2041
git checkout 828ac0b2ae74aac181ba128fb59b08a1534f2041 && git merge --no-ff 32a6bdb6d3d69d084bc24fd915c67027f0efb216
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@JJenkx

JJenkx commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Windows GPU run of head 32a6bdb6d3, merged onto main e8c25dd3ba (clean merge, local only).

Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, Radeon RX 6700 XT with AMD proprietary driver 26.10.1 (Vulkan 1.4.315), WinLibs GCC 15.2.0 UCRT r7, Ninja, Release, BUILD_TESTING=ON. Khronos validation layer 1.4.363 requested for every test (VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation).

  • All 23 CTest entries of agc_driver_tessellation_tests, agc_shader_memory_tests, which builds the test files this PR changes: pass, 3 of 3 runs each (--repeat until-fail:3), no failures, no skips.
  • They create Vulkan instances on the RX 6700 XT: the validation layer was inserted 9 times across the runs and reported no VUIDs.

AI-assisted: yes (Claude Code).

@jotitadev jotitadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The lavapipe run the description says it couldn't get, of head 32a6bdb6d3. Lavapipe has no subgroups in the tessellation evaluation stage, so it takes the new one-lane path. That path works, and the new draw catches the refusal without the change.

Setup: Ubuntu 26.04, GCC 15.2, Release. Lavapipe from Mesa 26.0.3; its subgroupSupportedStages here is FRAGMENT | COMPUTE | TASK | MESH. An RTX 3070 with NVIDIA 595.99.02 serves as a control, since it has subgroups in that stage. The head is merged --no-ff onto main 809c7fa2df with no conflicts. ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1, ctest -j1.

  • Green, lavapipe: the 24 entries of agc_driver_tessellation_tests, agc_shader_memory_tests and request_serializer_tests pass 3/3 with VVL 1.4.341, with no VUID- lines. agc_driver_tessellation runs all the way to Tessellation tests passed, with no skip of the new draw. The only skips are the two unmapped hull-read checks that llvmpipe always skips.
  • Green, 3070: the same 24 pass with VVL 1.3.275 and with no layer.
  • Red, lavapipe: main plus only this PR's Tessellation.cpp fails agc_driver_tessellation with AGC graphics: subgroup capability 61 is unsupported for shader stage 4. That's the GT7 error. On the 3070, the same red passes, as expected.
  • ^agc on lavapipe (minus dcc/flip/lds_atomics64): 1132/1132 with the PR.

GPUs tested: Mesa 26.0.3 lavapipe, Ubuntu 26.04 (one-lane path); NVIDIA GeForce RTX 3070, 595.99.02, Ubuntu 26.04 (subgroup path)

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the diff against the one-lane vertex path from #4470, the cache key and the serialization (version 50 skipped and rejected, version 51 flag read as true for older requests), the new tests (the domain-wave check and the DomainFirstLaneCount draw fail without the HostSubgroupSize line) and the debt entry after the selector wording fix. Lavapipe, NVIDIA and AMD runs by @jotitadev, @TechLevelZero and @JJenkx cover the one-lane and subgroup paths; merged.

frankea added a commit to frankea/AnyPS5 that referenced this pull request Nov 1, 2026
Tessellation evaluation programs without subgroups now take the one-lane
path from boykopovar#13807, so verifySingleLaneDomainWave checks that a
v_readlane_b32 at the lane v_readfirstlane_b32 takes from v5 throws there
and builds on a 32-lane tessellation evaluation subgroup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* docs shader core/shader unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants