Repository navigation
fix(shader): run tessellation evaluation stages one lane per invocation on devices without subgroups there - #13807
Conversation
…on on devices without subgroups there
MoltenVK and lavapipe leave the tessellation evaluation stage out of
subgroupSupportedStages, so the driver's shader validation refuses every
domain program that declares GroupNonUniform ("subgroup capability 61 is
unsupported for shader stage 4"). Gran Turismo 7 stops there on macOS at
the Used Car dealership.
The device target now records whether the tessellation evaluation stage
has subgroups. When it does not, tessellation evaluation programs outside
the mesh path get a host subgroup width of 1 and the one-lane rules that
vertex programs already use without vertex subgroups: the lane id is 0,
a ballot is the invocation's own bit, a shuffle returns the invocation's
value and an elect is true. Reads of other lanes still throw. Devices
with tessellation evaluation subgroups emit the same SPIR-V as before.
The request serialization moves to version 51.
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
@frankea Reviewed at
Merges with #13341 and #13774 are clean in both orders. A run of the tessellation draw on a device without tessellation evaluation subgroups (the Linux lavapipe lane) would confirm it; it was skipped on this head. |
…cked lane selectors Version 50 is skipped, so a request with that number is refused instead of being read in the version 49 layout. The one-lane tessellation evaluation debt entry now states that only a v_readlane_b32 of a constant lane throws: a lane selector held in a register is not checked and reads the invocation's own value, as on the vertex path.
|
@oneandonlydean Thanks for the review. I pushed
The same 33 focused tests pass 3 times on the RTX 4090 at |
✅ NVIDIA GPU test: passedPR head
Intermittent validation messages (1, seen once in the parallel run, not reproduced when the test was re-run alone 2 times; not counted against this PR)Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (2, not counted against this PR)Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)Validation IDs already present on main (3 IDs in 387 tests)Skipped (41)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
✅ NVIDIA GPU test: passedPR head
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (2, not counted against this PR)Validation IDs already present on main (3 IDs in 387 tests)Skipped (41)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
|
Windows GPU run of head Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, Radeon RX 6700 XT with AMD proprietary driver 26.10.1 (Vulkan 1.4.315), WinLibs GCC 15.2.0 UCRT r7, Ninja, Release,
AI-assisted: yes (Claude Code). |
jotitadev
left a comment
There was a problem hiding this comment.
The lavapipe run the description says it couldn't get, of head 32a6bdb6d3. Lavapipe has no subgroups in the tessellation evaluation stage, so it takes the new one-lane path. That path works, and the new draw catches the refusal without the change.
Setup: Ubuntu 26.04, GCC 15.2, Release. Lavapipe from Mesa 26.0.3; its subgroupSupportedStages here is FRAGMENT | COMPUTE | TASK | MESH. An RTX 3070 with NVIDIA 595.99.02 serves as a control, since it has subgroups in that stage. The head is merged --no-ff onto main 809c7fa2df with no conflicts. ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1, ctest -j1.
- Green, lavapipe: the 24 entries of
agc_driver_tessellation_tests,agc_shader_memory_testsandrequest_serializer_testspass 3/3 with VVL 1.4.341, with noVUID-lines.agc_driver_tessellationruns all the way toTessellation tests passed, with no skip of the new draw. The only skips are the two unmapped hull-read checks that llvmpipe always skips. - Green, 3070: the same 24 pass with VVL 1.3.275 and with no layer.
- Red, lavapipe:
mainplus only this PR'sTessellation.cppfailsagc_driver_tessellationwithAGC graphics: subgroup capability 61 is unsupported for shader stage 4. That's the GT7 error. On the 3070, the same red passes, as expected. ^agcon lavapipe (minus dcc/flip/lds_atomics64): 1132/1132 with the PR.
GPUs tested: Mesa 26.0.3 lavapipe, Ubuntu 26.04 (one-lane path); NVIDIA GeForce RTX 3070, 595.99.02, Ubuntu 26.04 (subgroup path)
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
|
Checked the diff against the one-lane vertex path from #4470, the cache key and the serialization (version 50 skipped and rejected, version 51 flag read as |
Tessellation evaluation programs without subgroups now take the one-lane path from boykopovar#13807, so verifySingleLaneDomainWave checks that a v_readlane_b32 at the lane v_readfirstlane_b32 takes from v5 throws there and builds on a 32-lane tessellation evaluation subgroup.
What
MoltenVK and lavapipe leave the tessellation evaluation stage out of
subgroupSupportedStages. MoltenVK 1.4.2 on an Apple M5 Max reportsTESSELLATION_CONTROL | FRAGMENT | COMPUTE(vulkaninfo; MVKDevice.mm adds only the control stage to fragment and compute), and lavapipe reportsFRAGMENT | COMPUTE | TASK | MESH(lvp_device.c). The recompiler lays every tessellation evaluation program out for the device's subgroup width, so a domain program with lane operations declaresGroupNonUniformandValidateShadersrefuses the draw:AGC graphics: subgroup capability 61 is unsupported for shader stage 4. #4470 handled the same refusal for the vertex stage.SpirvTarget::tessellationEvaluationSubgroups(defaulttrue) records whether the device has subgroups in that stage;VulkanDevice::buildTargetsets it fromsubgroupSupportedStages, next tovertexSubgroups. When it isfalse,HostSubgroupSizegives tessellation evaluation programs a host subgroup width of 1, so they take the one-lane rules fix(recompiler): run vertex programs as one-lane waves without vertex subgroups #4470 added for vertex programs: the lane id is 0, a ballot is the invocation's own bit, a shuffle returns the invocation's own value and an elect is true. Av_readlane_b32of another lane named by a constant throws; a lane selector held in a register is not checked and reads the invocation's own value, as on the vertex path (recorded in the debt entry; a check for it, on both paths, is left for a follow-up). Devices with subgroups in the stage emit the same SPIR-V as before.graphics->meshset (tessellation followed by a guest geometry program: the domain program feeds the generated capture geometry stage and the geometry half runs on the mesh path) keeps the device width. That path needs geometry and mesh shaders, which MoltenVK does not have, and it is not exercised here.RecompileCacheKey, and the request serialization moves to version 51 with the flag as its last byte; older requests read it astrue, and version 50 is rejected. I skipped 50 because 12 open pull requests already write it (fix(shader): share wave64 VGPRs past the private ones between the two halves #7410, feat(agc): blend into 11_11_10 float color targets in the pixel shader #9020, feat(agc): draw alpha-tested packed-target merges that write depth as a depth pass and an EQUAL merge pass #9335, feat(agc): blend into 10_11_11 unorm color targets in the pixel shader #9748, feat(agc): source indirect instance user data from BaseInstance #10359, fix(agc): prefer host interpolation on RADV to avoid the quad seam #10547, fix(agc): span table-reading wave64 programs by default on Intel drivers that share handles across fused threads #10846, perf(shader): keep native f16 conversion NaN results when a startup probe matches RDNA #11351, fix(shader): declare f16 and f64 DenormPreserve together when their modes are shared #11483, feat(agc): test the depth bounds in the pixel program when the device lacks the depthBounds feature #11551, feat(agc): apply a T#'s MIN_LOD clamp in the shader without VK_EXT_image_view_min_lod #12662, fix(agc): bind a T# minimum LOD through an S# that holds the LOD at 0 #12980), and they are the only open pull requests whose changes conflict with this one (the version lines and the serialization tests). The shader disk cache format is unchanged: its key already holds theRecompileCacheKeywords and the host subgroup width, and no encoded field changes.docs/dev/debt/shader.mdfor the one-lane tessellation evaluation waves.Title: Gran Turismo 7 (PPSA01317) on an Apple M5 Max through MoltenVK stops with this error at the Used Car dealership's car showcase, about 870 s into a run, on its own domain programs. The new
Tessellation.cppdraw reproduces that refusal on MoltenVK without this change and passes with it (see Tested); I have no GT7 run with this change that has reached the dealership yet.Tested
Windows 11 on an RTX 4090 (driver 616.92, Vulkan 1.4.341), winlibs GCC 15.2.0, Release with
BUILD_TESTING=ON. The 4090 has subgroups in the tessellation evaluation stage, so its draws keep the subgroup path; the one-lane path runs there only inverifySingleLaneDomainWave, which compiles for explicit targets.53e4e8e796, and again at32a6bdb6d3) and onmain833be4a:request_serializer, the 21 entries ofagc_shader_memory_tests,agc_driver_tessellation,agc_driver_tessellation_depth_range_remap,agc_tessellation_pipeline_cache,agc_tessellation_factor_ring,agc_tessellation_lowering,agc_depth_range_remap,agc_driver_pixel_barycentric_emulation,agc_driver_mesh_barycentric_emulation,agc_driver_rect_list_corner,agc_driver_provoking_vertexandagc_driver_vertex_elected_atomic_max(skipped: the 4090 has vertex subgroups). They pass once more underVK_LAYER_KHRONOS_validationwithANYPS5_NO_SHADER_CACHE=1andANYPS5_REQUIRE_VULKAN=1(ctest -V) on both, with noVUID-lines and nowas not found but was requested by env var VK_INSTANCE_LAYERS; the loader log foragc_driver_tessellationin that environment shows the layer inserted.Apple M5 Max, macOS 27.0.1, MoltenVK 1.4.2 (Vulkan SDK 1.4.363.0), x86-64 under Rosetta 2:
request_serializerand the 21agc_shader_memory_testsentries pass 3 times in a row on this branch, built natively. Two local build aids that are not part of this pull request were needed for that: the duplicatewrite_watchtest registration removed as in test(libc): register the macOS write watch fault tests once #13220, and theGuestMemory.cpppart of build(agc): compile Lifecycle, GuestMemory and GuestBufferMemory on macOS #11447.maindoes not build the AGC driver on macOS by itself, so the GPU check runs on my macOS tree (main102b7a5 with the open macOS host pull requests and my open GT7 pull requests) at48539c856b, which carries the same change (serialization version 52 there), with this pull request'sTessellation.cppchange applied. The new draw passes, andagc_driver_tessellationthen stops at a later check,did not take its blue channel from the hull's relative patch index across instances, which fails the same way on that tree with neither the change nor the new draw. With only theHostSubgroupSizeline removed, the new draw stops withAGC graphics: subgroup capability 61 is unsupported for shader stage 4, the error GT7 hits. UnderVK_LAYER_KHRONOS_validation(inserted, perVK_LOADER_DEBUG=layer) the run with the change has noVUID-lines.New checks:
verifySingleLaneDomainWave(tests/ShaderMemory.cpp): domain programs with lane operations and with av_readlane_b32waterfall loop declare no subgroup capability, built-in or instruction without tessellation evaluation subgroups, whatever the device's subgroup width, and still use subgroup operations with them; turning off vertex subgroups leaves a domain program on subgroups and turning off tessellation evaluation subgroups leaves a vertex program on them; a read of lane 31 throws; the flag survives serialization and changes the cache key. With theHostSubgroupSizeline removed it fails witha domain program without tessellation evaluation subgroups declared a subgroup capability, built-in or instruction.RequestSerializerTests.cpp: both values round-trip at version 51 as its last byte, a version 51 request without it is rejected as truncated, a version 49 request readstrueand is written back as version 51 with the flag set, and a version 50 request is rejected in either layout (as isNVNQQTIAAAA=inShaderMemory.cpp's unsupported-version list); both rejection checks fail with the version 50 check removed. The existing layout checks move by the one trailing byte.DomainFirstLaneCount(tests/execution/Tessellation.cpp):DomainCodewith its depth taken fromv_cvt_f32_u32ofv_readfirstlane_b32ofv_mbcnt(0 at any wave width), drawn throughRequirePatches. It runs on subgroups on the 4090 and one lane per invocation on MoltenVK; lavapipe in CI takes the one-lane path too, which I could not run here.tools/check_conventions.py --base origin/mainpasses, and the branch merges cleanly withmain828ac0b, whose 19 new commits touch none of its files.tools/pr_overlap.py --branch fix/tes-single-lane-without-subgroupsagainstmainde83e8f finds only the 12 version 50 pull requests above; 51 pull requests that conflict withmainthemselves share files with this branch and could not be compared.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)