feat(agc): emulate PA_SC_AA_MASK sample masks by oneandonlydean · Pull Request #6015 · boykopovar/AnyPS5

oneandonlydean · 2026-10-14T19:20:10Z

What

Emulates PA_SC_AA_MASK_X0Y0_X1Y0 / PA_SC_AA_MASK_X0Y1_X1Y1 (one 16-bit sample mask per pixel of the 2x2 quad) instead of refusing the draw with AGC graphics: sample masks are unsupported.

A mask equal for the four pixels is the Vulkan pSampleMask.
A mask that differs between the pixels is applied in the pixel shader: a pixel whose mask covers all or none of its samples is demoted to a helper invocation (VK_EXT_shader_demote_to_helper_invocation, so derivatives of the covered pixels stay defined); a partial mask is ANDed into gl_SampleMask.
Without demote, OpKill is used only for pixel programs that read no neighbouring lanes; implicit-LOD sampling, LOD queries, whole-quad mode and cross-lane reads (ReadFirstLane, Ballot, PermuteU32) are refused.
Refused explicitly: sample masks with conservative rasterization, partial uniform masks with centroid inputs, and early-Z pixel programs that write memory under depth, stencil or depth-bounds tests.
The masks are part of the pipeline, draw and shader cache keys; the recompile request moves to version 28 (quad pixel mask) and 29 (quad sample masks).
Technical debt entry added in docs/dev/debt/libSceAgcDriver.md.

Carries @MichonGoddijn231849's #448 and @fdfawef324's revision (codex/samplemask-no-demote-review), authorship kept; supersedes #448.

Rebased onto current main: next to main's dithered alpha-to-coverage flag (request version 25) and runtimeDescriptorArray validation parameter.

Tested

Linux (CachyOS), Release, BUILD_TESTING=ON, ANYPS5_ENABLE_SPIRV_TOOLS=ON: all agc_ tests (576) and the rest of ctest pass; request_serializer needed its main-side replays moved to version 29 (last commit). New tests: agc_shader_quad_mask_no_demote, agc_driver_quad_pixel_mask, agc_driver_sample_mask_draw, agc_driver_sample_mask_depth_stencil, agc_driver_sample_mask_multisample.
GTA V (PPSA04263): on main Story Mode stops with AGC graphics: sample masks are unsupported. With this PR it gets past that, through the Story Mode loading screen and onto the prologue title card (Ludendorff, North Yankton). The next stop is AGC graphics: CB resolve into a target without a resident image is not implemented, a separate topic.

Checklist

github-actions · 2026-10-14T19:26:10Z

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on `main`)	Same new files
#448		`core/libs/prx/libSceAgcDriver/tests/execution/QuadPixelMask.cpp`
#5200	`core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawRegisterKey.cpp` L147 `core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp` L2973, L3088-3089, L3091, L3216-3219, L3221-3222 `core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp` L342, L395, L767, L800 `core/shader/recompiler/Recompiler.hpp` L156 `core/shader/recompiler/tests/RequestSerializerTests.cpp` L205-206, L243-244, L267-268, L390-391, L405
#5304	`core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawDecode.cpp` L128
#5500	`core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawRegisterKey.cpp` L147 `core/libs/prx/libSceAgcDriver/Graphics/include/State.hpp` L144 `core/libs/prx/libSceAgcDriver/Graphics/src/State.cpp` L1039 `core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp` L2973, L3091, L3216, L3221 `core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp` L342, L395, L767, L800
#6177	`core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp` L284 `core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp` L2973, L3091, L3216-3217, L3221-3222 `core/shader/recompiler/ControlFlow/src/RequestSerializer.cpp` L767, L800 `core/shader/recompiler/tests/RequestSerializerTests.cpp` L205-206, L243-244, L267-269, L390-392, L405, L407
#6188	`core/libs/prx/libSceAgcDriver/Execution/src/Driver/Draw/DrawLookup.cpp` L75

fdfawef324 · 2026-10-14T23:23:21Z

GPUs tested: NVIDIA GeForce RTX 3060 12 GB, driver 591.86, Windows Server 2025 (10.0.26100), Ryzen 9 5900X.

Build: MinGW GCC 15.2, Release -O3, BUILD_TESTING=ON; libSceAgcDriver plus the test executables below, exit 0, no warnings in files this PR touches.

Commits: PR head 9519b53c6, and the same six commits rebased locally onto main b9d9ec605 (clean, no conflicts, not pushed). Each run 3x with ctest -j1:

New/changed (8): request_serializer, agc_shader_memory, agc_shader_quad_mask_no_demote, agc_driver_graphics, agc_driver_quad_pixel_mask, agc_driver_sample_mask_draw, agc_driver_sample_mask_depth_stencil, agc_driver_sample_mask_multisample: 8/8 passed in all 6 runs, none skipped.
Existing (36): other agc_driver_graphics_tests/agc_shader_memory_tests modes, agc_driver_image_sample_*, agc_driver_sample_multisample_target, agc_driver_*alpha_to_coverage*, agc_driver_prepared_graphics*, agc_multisample_guest_access: 36/36 passed in all 6 runs.

Result lines: quad pixel mask tests passed (demote to helper invocation), 4x sample mask tests passed, quad sample mask depth and stencil tests passed, sample mask draws through the driver passed.

Not tested: the kill path without demote on a GPU (this device exposes demote), GTA V, AMD/Intel GPUs, the full ctest suite.

AI-assisted: yes

With one sample, PA_SC_AA_MASK leaves one coverage bit per pixel of the 2x2 quad, so a draw can cover only some pixels of every quad (GTA V renders a target one quarter of the pixels at a time this way). The pixel stage info carries that 4-bit mask; a fragment shader whose mask is not full starts by killing the invocation unless its gl_FragCoord quad position is covered, and runs without early fragment tests. The mask is part of the recompile cache key and the serialized request (version 24). Sample masks of multisampled draws stay refused.

With PA_SC_AA_MASK covering only some pixels of a quad, the uncovered pixels run on the hardware as helper lanes, so derivatives of the covered pixels stay defined. OpKill ends those invocations; OpDemoteToHelperInvocation keeps them as helpers. The device enables VK_EXT_shader_demote_to_helper_invocation when available and passes the DemoteToHelperInvocation capability and its SPIR-V extension to the recompiler target and the graphics shader validation. A demoted pixel also reads as a helper, so its initial EXEC bit is clear as on the hardware. Without demote, OpKill stays for pixel programs that read no neighbouring lanes; implicit-LOD sampling, LOD queries and cross-lane reads are refused. The quad pixel mask execution test now also samples a mipmapped texture with implicit LOD under the mask.

…ask without demote A program that enables whole-quad mode (s_wqm_b64) and then reads a ballot or readfirstlane sees the helper lanes, which OpKill removes.

Decode the four PA_SC_AA_MASK fields at the rasterized sample count. Use the native Vulkan mask when all four pixels agree and apply varying masks in the fragment shader, preserving helpers and combining coverage with alpha-to-coverage and shader mask exports. Add the normalized masks to pipeline, draw and shader keys and the versioned replay request. Refuse unsupported conservative-raster, centroid and early depth/stencil memory-write cases. Synthetic readbacks cover driver submission, depth/stencil suppression, fragment stores and one-sample and four-sample coverage.

ReadFirstLane, Ballot and PermuteU32 can consume uncovered helper lanes after a fragment explicitly enables EXEC. Classify those operations as neighbour reads so a partial quad mask refuses their kill fallback when demote is unavailable. ReadFirstLane also stays protected when a constant whole-quad instruction folds away. The CPU regression covers explicit EXEC, constant WQM, a comparison ballot and ds_permute. Unmasked programs still compile, supported masked programs retain demote and their subgroup operation, and constant colour keeps the kill fallback. The recovered source fails the readfirstlane and ballot checks; the corrected source passes both the focused check and shader-memory suite. Co-Authored-By: Codex <noreply@openai.com>

…n the serializer replays Requests replayed from versions 22 to 24 now re-serialize at version 29 with the default quad pixel mask (0xf) and quad sample masks (0xffff) after the dither and front-face flags, plus the empty linked call fallback list.

oneandonlydean · 2026-10-15T00:03:16Z

Rebased onto main (debt entry next to #5631's). A pixel program #5631 skips is replaced by the null pixel program, which applies the quad mask at entry like any other, so the two combine. Thanks @fdfawef324 for the Windows runs.

oneandonlydean mentioned this pull request Oct 14, 2026

feat(agc): single-sample sample masks as a per-quad-pixel discard #448

Closed

9 tasks done

oneandonlydean force-pushed the carry/448-sample-masks branch from c7c1e51 to f1e4dec Compare October 14, 2026 19:25

github-actions Bot mentioned this pull request Oct 14, 2026

fix(agc): skip unused pixel stages with retained export formats #5631

Merged

11 tasks done

github-actions Bot added build shader core/shader docs agc libSceAgc* labels Oct 14, 2026

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

oneandonlydean force-pushed the carry/448-sample-masks branch from f1e4dec to 9519b53 Compare October 14, 2026 22:04

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

MichonGoddijn231849 and others added 6 commits October 15, 2026 12:59

fix(agc): count whole-quad mode as a neighbour-lane read for a quad m…

90783f2

…ask without demote A program that enables whole-quad mode (s_wqm_b64) and then reads a ballot or readfirstlane sees the helper lanes, which OpKill removes.

oneandonlydean force-pushed the carry/448-sample-masks branch from 9519b53 to 5fd9c80 Compare October 15, 2026 00:03

github-actions Bot mentioned this pull request Oct 15, 2026

fix(shader): propagate color view slice to pixel ancillary input #5500

Closed

11 tasks done

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 15, 2026

This was referenced Oct 15, 2026

perf(shader): rely on robustBufferAccess2 for direct buffer bounds #6177

Closed

perf(agc): compile uncached draws ahead of queue 0 #6188

Open

SandstromPL mentioned this pull request Oct 15, 2026

fix(agc): rasterize into one-sample targets as one sample when MSAA_ENABLE is clear #6196

Merged

12 tasks done

SandstromPL mentioned this pull request Oct 15, 2026

fix(agc): rasterize a multisampled depth target without MSAA_ENABLE at the pixel center #6014

Merged

11 tasks done

oneandonlydean merged commit 760ffa6 into boykopovar:main Oct 15, 2026
8 checks passed

karanvora2599 mentioned this pull request Oct 15, 2026

perf(shader): choose the wave64 compute layout per program #4339

Closed

11 tasks done

cookie42073-cmd mentioned this pull request Oct 15, 2026

fix(shader): budget pixel lane proof for NHL 26 replay #6180

Merged

12 tasks done

MiguelAdolfsen mentioned this pull request Oct 15, 2026

perf(agc): refresh only the layers a use of an array color target covers #6202

Closed

11 tasks done

RINNECODER mentioned this pull request Oct 15, 2026

Upstream sync 11D: shader recompiler (13 commits) RINNECODER/AnyPS5#511

Merged

LeandroLP23 mentioned this pull request Oct 15, 2026

fix(shader): kill demoted quad-mask pixels when the pixel program returns #6255

Merged

11 tasks done

SoRadGaming mentioned this pull request Oct 15, 2026

perf(agc): widen small epoch collects to their 64 KiB block on Windows #6273

Merged

11 tasks done

LeandroLP23 mentioned this pull request Nov 1, 2026

AGC: agc_driver_quad_pixel_mask samples the wrong mip level under a partial quad mask on Windows (RTX 5060 Ti and Radeon 860M) #14237

Open

Navigation Menu

Repository navigation

feat(agc): emulate PA_SC_AA_MASK sample masks - #6015

feat(agc): emulate PA_SC_AA_MASK sample masks#6015
oneandonlydean merged 6 commits into
boykopovar:mainfrom
oneandonlydean:carry/448-sample-masks

oneandonlydean commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

fdfawef324 commented Oct 14, 2026

Uh oh!

oneandonlydean commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

oneandonlydean commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tested

Checklist

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fdfawef324 commented Oct 14, 2026

Uh oh!

oneandonlydean commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

oneandonlydean commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading