feat(agc): rebuild the hull's relative patch index when it is used as data by Celegans12 · Pull Request #4201 · boykopovar/AnyPS5

Celegans12 · 2026-10-12T01:12:20Z

What

Stacked on #4172; please review the last commit only.

Tessellation lowering refuses hull and domain programs whose relative patch index (hull v1[7:0], domain v7) reaches anything but a ring or LDS address, because one Vulkan patch per hull patch slot has no such index of its own.

Hull and domain programs that use the index as data now get (PrimitiveID mod PRIM_GRP_SIZE) mod NUM_PATCHES (a domain's gl_PrimitiveID is its hull patch's, so both stages compute the same number): hull threadgroups take NUM_PATCHES patches out of each GE_CNTL.PRIM_GRP_SIZE primitive group.
The host's PrimitiveID restarts with each instance, so this is only right when every instance starts a new threadgroup, i.e. with GE_CNTL.BREAK_WAVE_AT_EOI (bit 18) set. With it clear the instances share threadgroups and the index counts on through them, so such programs are still refused. fix(shader): number an instanced draw's hull patches across its instances #4450 numbers the patches across instances for that case.
PRIM_GRP_SIZE (bits 8:0) and BREAK_WAVE_AT_EOI join the tessellation configuration (draw, validation and pipeline keys, the hull and domain cache keys, recompile request version 24). A missing GE_CNTL throws. Local programs still refuse the index: there it is a relative vertex slot whose value depends on how the hardware lays out vertices in a threadgroup.

The model is not measured on the console; debt/shader.md says so. PAL says a group size that is not a multiple of the patch count is not allowed, yet PPSA26344's terrain draws run with PRIM_GRP_SIZE 64 and NUM_PATCHES 63 (GE_CNTL 0x8040, BREAK_WAVE_AT_EOI clear), so on their own they are refused here and need #4450.

Tested

Based on main f265cd5 plus #4172. Windows 11 (MinGW-w64 GCC 15.2.0) and Linux: configure -DBUILD_TESTING=ON, targeted test builds, ctest. Measured on #4222's head, which contains this PR:

Windows NVIDIA: agc_driver_tessellation, agc_shader_memory, agc_driver_graphics, agc_tessellation_factor_ring and agc_driver_bda_device pass (5/5)
Linux (upstream's job in WSL, GCC 13.3, llvmpipe), fix(shader): record BDA faults in hull programs instead of refusing them #4222's head merged with current main: the same 5 tests pass (5/5)
Full suite on fix(shader): record BDA faults in hull programs instead of refusing them #4222's head (which contains this commit): Windows NVIDIA 1110/1111, the failure is log_macros, which fails on main too; Linux (upstream's job in WSL) merged with main 5a81f60: 1089/1091, agc_driver_bc_block_copy and agc_driver_depth_compare_copy fail on that main as well

agc_driver_tessellation gains RequireRelativePatches: the hull writes float(rel + 1) as a patch constant, checked at NUM_PATCHES/group 3/3, 2/4, 3/4 (patches 6 and 7, which rule out 0, id mod 3 and id mod 4) and 3/2 with BREAK_WAVE_AT_EOI set; a missing GE_CNTL and a clear BREAK_WAVE_AT_EOI are refused. A domain pair (DomainRelativeData) runs the same cases. agc_driver_graphics decodes GE_CNTL 0x8040 and 0x5fe3f, and agc_shader_memory covers version 24 and the hull/domain keys. They fail on #4172's head with the old refusal.

Not tested: console behaviour.

Checklist

Celegans12 · 2026-10-12T02:09:37Z

I force-pushed this after a Yōtei run showed the first version was too strict. The terrain draw runs with PRIM_GRP_SIZE 64 and NUM_PATCHES 63, even though its hull header says 63/63, so the old "must be a multiple" refusal stopped the game. The index is now (PrimitiveID mod PRIM_GRP_SIZE) mod NUM_PATCHES, which is the same as before whenever the size is a multiple, and only a missing GE_CNTL is refused. The description and the TechnicalDebt line say this now, and RequireRelativePatches gained 3/4 and 3/2 cases that tell the formulas apart.

Checks after the change

Linux (upstream's job in WSL), fix(shader): record BDA faults in hull programs instead of refusing them #4222's head on top of this, merged with main ceb342b: 924/925 pass; agc_driver_depth_compare_copy also fails on main.
Windows NVIDIA full ctest: 903/907, the same four failures as without this change.
The game's hull request now gets past this step when replayed.

Celegans12 · 2026-10-12T03:45:28Z

One more push here: Yōtei's next terrain domain program (0x8068fbd900) uses the relative patch index as data too, so domain programs now get the same (PrimitiveID mod PRIM_GRP_SIZE) mod NUM_PATCHES. A domain's gl_PrimitiveID is its hull patch's, so both stages agree on the number, and PRIM_GRP_SIZE now keys the domain cache key as well. Local programs still refuse it, since their index is a vertex slot that depends on how the hardware packs a threadgroup. RequireRelativePatches gained a domain pair over the same four configurations. Linux (upstream's job, #4222's head on top, merged with main d146ec1): 928/929, with agc_driver_depth_compare_copy failing on main too.

Pipodi · 2026-10-12T11:56:43Z

Data point from Ghost of Yōtei (PPSA26344) on the instancing model here: on that title, the instances of a tessellated draw share hull threadgroups. Restarting the relative patch index with each instance gives the wrong index there.

What the title programs

GE_CNTL = 0x8040: PRIM_GRP_SIZE 64, BREAK_WAVE_AT_EOI (bit 18) = 0.
VGT_LS_HS_CONFIG = 0x1043f: NUM_PATCHES 63, 4 input and 4 output control points.
The terrain draws are DRAW_INDIRECT. Their GPU-written records hold 4 vertices and 2 to 3,157 instances, so each instance is a single patch.

Why the per-instance restart is wrong here

With BREAK_WAVE_AT_EOI clear, a threadgroup runs on into the next instance. Patch k of instance i is then linear patch i * patchesPerInstance + k, and the relative index is ((i * P + k) mod PRIM_GRP_SIZE) mod NUM_PATCHES. Here P = 1, so the hardware gives instance i the index (i mod 64) mod 63. The host's PrimitiveID restarts at 0 with each instance, so (PrimitiveID mod 64) mod 63 is 0 for every instance.

There is also a timing problem for indirect draws. The hull is compiled when the packet is decoded, before the records are read, so it can't see the record's instance count.

What we have locally

We have this working in #4450, a draft stacked on #4222 (only its top commit is new):

Instance numbering. The local stage passes its draw-relative instance (InstanceIndex - BaseInstance) to the hull through one extra interface slot. When an instanced draw's instances share threadgroups, groupedPatch numbers the patches instance * P + PrimitiveID and then takes the same two remainders.
Restart cases. BREAK_WAVE_AT_EOI is decoded into the tessellation configuration (request v20, draw and pipeline keys). When it is set, or when P is a multiple of the group, the per-instance PrimitiveID is kept.
Indirect draws. The hull and domain are recompiled for each indirect record's instance count, and P is in their cache keys.
Device feature. The device enables shaderDrawParameters. Without it, an instanced hull that uses its relative index as data is refused.
Domain. A domain that uses its relative index as data in such a draw is also refused for now, because passing the instance on to the domain is not implemented.
Tests. An execution test draws two instances of three patches with first instance 1. The correct numbering, a per-instance restart and the absolute instance each give a different color, and the test reads them back. The full suite passes apart from failures that also happen on fix(shader): record BDA faults in hull programs instead of refusing them #4222's base.
In the game. With this change the hull compiles for every record (2 to 3,157 instances) and the title gets past these draws to its post-load popover. Under feat(agc): rebuild the hull's relative patch index when it is used as data #4201's model the same draws would render without an error, but with every instance's relative index at 0.

Would you rather fold this into #4201, or keep #4450 as a separate PR on top of #4222? Either works for us. If you fold it in, I'll close #4450.

A smaller point from reading the stack: the hull's patch-constant bound compares address.constant with one patch's control points (outputs * controlStride). The domain's PatchInput check uses patchesPerGroup * outputs * controlStride. It makes no difference for Yōtei, but the two bounds probably should agree.

(AI-assisted: written with Claude.)

blawesom · 2026-10-12T12:56:37Z

Rebase handoff — the chain is complete: rebase/4114 → rebase/4163 → rebase/4172 → this, all on current main. Authorship unchanged; this link applied cleanly (no conflicts).

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV):

builds; tools/check_conventions.py --base origin/main passes
full suite: 949/949 passed

To pick up the whole chain:

git fetch https://github.com/blawesom/AnyPS5 rebase/4114 rebase/4163 rebase/4172 rebase/4201

Heads: 4114=d0ea5f4d1, 4163=652f2763d, 4172=65ad9582b, 4201=53a35dc8c.

Celegans12 · 2026-10-13T01:44:35Z

You're right, and thanks for checking it against the title's registers. I had assumed every instance starts its own hull threadgroups, and with GE_CNTL 0x8040 that assumption doesn't hold, so on Yōtei #4201 alone would draw the terrain with every instance at relative patch 0 and never say so.

I'd keep #4450 as its own PR on top of the stack. It's your work and your test, and it carries the in-game result. To make #4201 honest on its own, it now decodes GE_CNTL.BREAK_WAVE_AT_EOI (bit 18) into the tessellation configuration and refuses a hull or domain that uses its relative index as data when the bit is clear. It only rebuilds the index when every instance is sure to start a new threadgroup. I used your field name, breakAtInstance, and the same draw, validation and pipeline keys. #4201 also puts it in the hull and domain cache keys, since there it decides whether a program compiles at all. The downside is that #4201 alone now refuses Yōtei's terrain draws until #4450 is in. Your PR then replaces that refusal with the instance numbering and can drop its own decode. I can't refuse only the instanced draws in #4201, because draw entries are reused across instance counts, which your PR handles.

The whole chain had to be rebased anyway. main now writes request version 19 (#4031 landed this morning), so the stack moved to 20/21/22 and breakAtInstance sits in #4201's version 22. Your commit then needs no request version of its own. I've rebased it onto the new stack locally to check that the two fit together. The rebase drops your decode and serialization lines, swaps the cache-key bit for your TessellationInstancePatches, and replaces the refusal with your numbering. I'm happy to push that to a branch on my fork if it saves you the conflicts; it's still your commit and your PR. On that local rebase, agc_driver_tessellation, agc_driver_graphics and agc_shader_memory pass. agc_shader_memory needed one line, because your instanced copy now inherits breakAtInstance = true from the hull fixture. The terrain hull with 0x8040 recompiles there too (388,903 words).

On the smaller point, I agree. #4163 now checks the hull's patch-constant writes against the same bound as the domain's reads: above the control points of all NUM_PATCHES patches, not just one patch's. The descending-record test hull therefore draws at 128 patches per group and is refused at 129. At 129 its lowest record would land inside the last patch's control points. Replayed offline with the title's own GE_CNTL 0x8040, the terrain hull (0x80002fc700) now stops at the new refusal. With the bit set in the request, it passes the tighter bound and recompiles completely (388,903 SPIR-V words).

Details

GE_CNTL decode: PRIM_GRP_SIZE bits 8:0, BREAK_WAVE_AT_EOI bit 18. agc_driver_graphics checks 0x8040 → 64 primitives, bit clear; 0x5fe3f → 63, bit set.
New refusals in agc_driver_tessellation: HullRelativeData and DomainRelativeData with the bit clear ("GE_CNTL.BREAK_WAVE_AT_EOI is clear, so the instances of a draw can share hull threadgroups"). RequireRelativePatches runs its four configurations with the bit set. agc_shader_memory checks that the bit survives serialization and changes only the hull and domain cache keys.
With the refusal line removed, agc_driver_tessellation fails with "a tessellation program was accepted although GE_CNTL.BREAK_WAVE_AT_EOI is clear". With breakAtInstance taken out of the cache key, agc_shader_memory fails on the key check.
Windows NVIDIA on fix(shader): record BDA faults in hull programs instead of refusing them #4222's head 3f165b6: 1020/1021 pass (log_macros fails here on every branch). Linux (upstream's job in WSL), fix(shader): record BDA faults in hull programs instead of refusing them #4222's head merged with main d1fd937: 1003/1004 pass; agc_driver_depth_compare_copy also fails on main.

Celegans12 · 2026-10-14T01:56:40Z

Quick note: I restacked this on #4172's new head, which now has the block-condition fix from the review there. While rebasing I noticed main took request version 23 for the pixel layer export flag, so this PR's tessellation fields now come in at version 24 (agc_shader_memory checks the new prefix). Nothing else changed in this commit. The tessellation, shader memory, graphics, factor ring and BDA device tests pass on #4222's head on Windows and on Linux.

A wave64 hull program narrows EXEC with s_and_saveexec_b64 and restores it later, so its exec words reach the lane-bit test as ANDs with the host ballot, as loop or join phis, and through separate ballots for the two halves. Tessellation lowering refused them as "uses a lane mask other than its own lane's bit". The own-lane bit of a mask pair is now rebuilt from the words' operations (ballots, bitwise ops, selects, phis, constants); both halves must have the same shape and ballot the same predicate. Dead hull cycles such as a saved-exec loop are removed before the remaining-ballot check, so a mask whose other lanes are read (s_bcnt1_i32_b64 exec) is still refused. The words are replaced through identities and block conditions are resolved, so a branch on a compare copied to VCC keeps its condition. Local and domain programs keep their exec words for the half-wave matcher. Block conditions fix and HullCompareBranch test from EagleBearerYS.

… data Hull programs whose relative patch index (v1 bits 7:0) reached anything but a tessellation ring or LDS address threw, because the lowering runs every Vulkan patch as relative patch 0. The quad terrain hull 0x80002fc700 in Ghost of Yotei is one, and so is the domain 0x8068fbd900 (v7). Such hulls and domains now take (PrimitiveID mod GE_CNTL.PRIM_GRP_SIZE) mod VGT_LS_HS_CONFIG.NUM_PATCHES, as if hull threadgroups took NUM_PATCHES patches at a time out of each primitive group; both stages see the same PrimitiveID for a patch. This holds only when every instance starts a new threadgroup, so it needs GE_CNTL.BREAK_WAVE_AT_EOI (bit 18); with the bit clear the instances share threadgroups and such programs still throw. PRIM_GRP_SIZE and BREAK_WAVE_AT_EOI join the tessellation configuration (draw keys, hull and domain cache keys, request version 24). A missing GE_CNTL throws, and so do local programs, whose relative vertex index depends on vertex reuse that Vulkan does not expose.

Celegans12 · 2026-10-14T03:07:00Z

Rebased the chain again after #5234 landed. The only overlap was in the tests and the debt list: #5234's integer-partitioning test and this PR's relative-patch test both sit in Tessellation.cpp, and both debt entries are kept. Nothing in the code changed, and request version 24 is still free on main. New heads: #4172 866ed445b9, this one 2eda9f893c, #4222 c1e4a0ba8b. On #4222's head the tessellation, shader memory, graphics, factor ring and BDA device tests pass on Windows (NVIDIA) and on Linux merged with main f265cd5.

github-actions · 2026-10-14T05:28:15Z

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on `main`)	Same exports	Same new files
#4451	`core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp` L1867-1868, L1874

…nces With GE_CNTL.BREAK_WAVE_AT_EOI clear, the instances of a tessellated draw share hull threadgroups: the relative patch index counts on through the instances, while the host's PrimitiveID restarts with each one. The local stage now passes its draw-relative instance (InstanceIndex - BaseInstance) to the hull, which numbers its patches instance * patches-per-instance + PrimitiveID before taking (mod PRIM_GRP_SIZE) mod NUM_PATCHES. Instances that start a threadgroup anyway (BREAK_WAVE_AT_EOI, or patches per instance a multiple of the group) keep the per-instance PrimitiveID. This replaces boykopovar#4201's refusal of a hull or domain that uses its relative index as data while BREAK_WAVE_AT_EOI is clear, and builds on its decode of the bit (breakAtInstance, request serialization and draw keys). - Hull and domain cache keys hold the patches per instance instead of the bit; an indirect draw recompiles its hull and domain for each record's instance count. - The device enables shaderDrawParameters when available. Without it, a hull using its relative index as data in such a draw is refused, as is a domain doing so (passing the instance on to the domain is not implemented). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… PRIM_GRP_SIZE) mod NUM_PATCHES)

GE_CNTL.BREAK_WAVE_AT_EOI (bit 18) joins the tessellation configuration as breakAtInstance (draw keys, hull and domain cache keys) and hull or domain programs that use their relative patch index as data throw when it is clear, as boykopovar#4201 does since its 2026-10-13 rebase. On this branch the field is request version 21 (17-19 are the tessellation chain, 20 the 10_11_11 blend); upstream boykopovar#4201 writes 22.

…nces With GE_CNTL.BREAK_WAVE_AT_EOI clear, the instances of a tessellated draw share hull threadgroups: the relative patch index counts on through the instances, while the host's PrimitiveID restarts with each one. The local stage now passes its draw-relative instance (InstanceIndex - BaseInstance) to the hull, which numbers its patches instance * patches-per-instance + PrimitiveID before taking (mod PRIM_GRP_SIZE) mod NUM_PATCHES. Instances that start a threadgroup anyway (BREAK_WAVE_AT_EOI, or patches per instance a multiple of the group) keep the per-instance PrimitiveID. - This replaces boykopovar#4201's refusal of shared threadgroups; boykopovar#4201 already decodes GE_CNTL.BREAK_WAVE_AT_EOI into the tessellation configuration. - Hull and domain cache keys hold the patches per instance; an indirect draw recompiles its hull and domain for each record's instance count. - The device enables shaderDrawParameters when available. Without it, a hull using its relative index as data in such a draw is refused, as is a domain doing so (passing the instance on to the domain is not implemented). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added shader core/shader docs agc libSceAgc* labels Oct 12, 2026

Celegans12 mentioned this pull request Oct 12, 2026

fix(shader): record BDA faults in hull programs instead of refusing them #4222

Merged

11 tasks done

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from 5610817 to f1f83f2 Compare October 12, 2026 02:06

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from f1f83f2 to e81cb5e Compare October 12, 2026 02:56

Celegans12 mentioned this pull request Oct 12, 2026

fix(agc): lower hull exec masks restored through phis to own lane bits #4172

Merged

11 tasks done

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from e81cb5e to 4c1e461 Compare October 12, 2026 03:45

github-actions Bot mentioned this pull request Oct 12, 2026

fix(agc): export the stencil reference from pixel shaders with VK_EXT_shader_stencil_export #4355

Merged

9 tasks done

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 12, 2026

Pipodi mentioned this pull request Oct 12, 2026

fix(shader): number an instanced draw's hull patches across its instances #4450

Merged

9 tasks done

This was referenced Oct 12, 2026

fix(agc): bound tessellation patch records by the draw's patch count #4114

Merged

feat(agc): read hull patch constants in tessellation domain programs #4163

Merged

gjonn mentioned this pull request Oct 12, 2026

feat(shader): read hull inputs at a control-point index computed from data #4727

Merged

10 tasks done

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from 4c1e461 to 6c19457 Compare October 13, 2026 01:44

This was referenced Oct 13, 2026

feat(agc): emulate default eight-sample dithered alpha coverage #4451

Merged

fix(shader): place hull input components by their offset in the control-point slot #4723

Merged

github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 13, 2026

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 13, 2026

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from 1ad64e4 to cecca8a Compare October 13, 2026 12:17

github-actions Bot mentioned this pull request Oct 13, 2026

fix(shader): read the ancillary render target index from Layer when the draw exports it #5097

Merged

10 tasks done

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 13, 2026

EagleBearerYS mentioned this pull request Oct 13, 2026

fix(shader): lower hulls with one input or one output control point #5457

Merged

11 tasks done

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from cecca8a to f823bf9 Compare October 14, 2026 01:56

github-actions Bot mentioned this pull request Oct 14, 2026

feat(shader): link callees of calls whose target is loaded from a table #4706

Merged

10 tasks done

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

Celegans12 added 2 commits October 14, 2026 04:57

Celegans12 force-pushed the feat/agc-tessellation-relative-patch-data branch from f823bf9 to 2eda9f8 Compare October 14, 2026 03:06

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026

oneandonlydean self-assigned this Oct 14, 2026

nikowolf123 mentioned this pull request Oct 14, 2026

feat(agc): blend into color targets with a reversed component order #5200

Merged

11 tasks done

oneandonlydean merged commit 5ee94cf into boykopovar:main Oct 14, 2026
8 of 9 checks passed

This was referenced Oct 15, 2026

Upstream sync 11D: shader recompiler (13 commits) RINNECODER/AnyPS5#511

Merged

Upstream sync 11E: AGC driver onto Metal (29 commits) RINNECODER/AnyPS5#512

Merged

TheBlenDed pushed a commit to TheBlenDed/AnyPS5 that referenced this pull request Oct 25, 2026

local: take boykopovar#4201's general relative patch formula ((id mod…

127a126

… PRIM_GRP_SIZE) mod NUM_PATCHES)

TheBlenDed pushed a commit to TheBlenDed/AnyPS5 that referenced this pull request Oct 25, 2026

local: take boykopovar#4201's domain relative patch index (4c1e461)

f56fce9

Navigation Menu

Repository navigation

feat(agc): rebuild the hull's relative patch index when it is used as data - #4201

feat(agc): rebuild the hull's relative patch index when it is used as data#4201
oneandonlydean merged 2 commits into
boykopovar:mainfrom
Celegans12:feat/agc-tessellation-relative-patch-data

Celegans12 commented Oct 12, 2026 •

edited

Loading

Uh oh!

Celegans12 commented Oct 12, 2026

Uh oh!

Celegans12 commented Oct 12, 2026

Uh oh!

Pipodi commented Oct 12, 2026

Uh oh!

blawesom commented Oct 12, 2026

Uh oh!

Celegans12 commented Oct 13, 2026

Uh oh!

Celegans12 commented Oct 14, 2026

Uh oh!

Celegans12 commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Celegans12 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tested

Checklist

Uh oh!

Celegans12 commented Oct 12, 2026

Uh oh!

Celegans12 commented Oct 12, 2026

Uh oh!

Pipodi commented Oct 12, 2026

Uh oh!

blawesom commented Oct 12, 2026

Uh oh!

Celegans12 commented Oct 13, 2026

Uh oh!

Celegans12 commented Oct 14, 2026

Uh oh!

Celegans12 commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Celegans12 commented Oct 12, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading