[milescoviello]

What

Every shader source keeps up to 256 materialized results of Recompile(request, capture) in its result memo (64 sets of 4), with no limit on their bytes. A result's bindings carry about twenty vectors each, sized by the descriptor counts, so with large image tables one result can hold tens of KiB. In Ghost of Tsushima Director's Cut (PPSA02225), whose opening runs into the 20 GiB memory cap on my host, the memos held 470 to 575 MiB after 5 minutes and kept growing, at a hit rate of about 13 %.

This bounds the bytes all memos hold together.

How

• Each memo entry records the bytes its result keeps: the result's own vectors and each binding's vectors (bit vectors as bits). The module isn't counted, because the prepared module cache shares it.
• Both of a source's tables count: the result memo and perf(shader): reuse a memoized result by patching its copied words #10423's masked memo. A masked entry also counts its patch recipe. A result held by both tables counts once in each, so the count errs high.
• The recency stamps come from one clock for all sources instead of one per source, so entries of different sources compare.
• After an insert pushes the held bytes over the budget, the least recently used entries of both tables of all sources are dropped until three quarters of the budget is left. Memos register in a list of sources when created, and the trim reads it under its own lock. Trims serialize on one blocking lock, and a trim reads the held bytes again after each pass until they are within the budget, so results that other threads insert while it runs are trimmed too.
• The budget is 256 MiB unless APS5_RESULT_MEMO_MB sets another; 0 sets none. APS5_RESULT_MEMO_MB is neutral for the disk cache key, since it doesn't change any result.
• APS5_PROFILE_DRAW's [recompile] result memo line now also reports trimmed results and the results and MiB held.
• SetResultMemoBudget and ResultMemoBytes let the new tests set a budget and read the bytes held. SetResultMemoTrimHook sets a one-shot function that the trim calls right after it reads the held bytes, so a test can insert from other threads at exactly that point.
• perf(shader): reuse an evicted memo result's bindings for the next miss #9049 changes the same miss path: it takes the set's least recently used entry out to reuse its bindings. The two merge without conflicts, but whichever lands second should subtract the taken entry's bytes where perf(shader): reuse an evicted memo result's bindings for the next miss #9049 empties the slot, or the held bytes drift upward.

Tested

• Current head 8868e09624, rebased onto main 442c43a. perf(shader): reuse a memoized result by patching its copied words #10423 landed first, so this head counts and trims its masked memo as well (one commit now). Linux (Ubuntu 26.04.1, GCC 15.2, Release, RTX 4070 Ti, 615.71.09):
  • The 37 ^agc_shader_ tests pass 3 of 3, including the three budget tests and perf(shader): reuse a memoized result by patching its copied words #10423's agc_shader_table_snapshot_reuse/_unmasked.
  • The two exact budget tests run with APS5_NO_MASKED_RESULT_MEMO=1. The new agc_shader_result_memo_budget_masked runs with the masked memo on. It checks that a first result counts its masked entry too, and that a trim to one byte empties both tables, so the next result is a full one, not a patched one.
  • Mutant: a trim that skips the masked table fails it with a trim to one byte left 620 bytes.
  • Khronos validation 1.4.363 over the same set: 34 of 34 on main 442c43a and 37 of 37 on this head, no layer missing, no VUID- messages on either.
  • Windows 11 VM (GT 1030, driver 582.78, WinLibs GCC 15.2.0 r7): 41 of 41 pass 3 of 3 on main and 44 of 44 pass 3 of 3 on this head.
• Persona 5 Royal (PPSA05110) on this head (thuanquang, Windows 11, RX 6700 XT, WinLibs GCC 15.2.0 r7, Release, main 73ad397 plus feat(libSceImeBackend): declare 22 sceImeBackend exports #10887, fix(libSceGameUpdate): answer no update instead of throwing on request creation #10614, fix(libSceVideodec2): mark returned pictures as not discarded #10913 and fix(agc): keep at most 4 GiB of the host heap from the host import budget #7844 with this PR merged). Three runs, each 780 s into the first gameplay scene, with private bytes sampled from outside the process:
  • No budget (APS5_RESULT_MEMO_MB=0, main's behaviour): 19,408 / 19,416 MiB at 500 / 700 s, peak 18.99 GiB.
  • This PR's default, 256 MiB: 18,652 / 18,680 MiB, peak 18.25 GiB.
  • Memos off (APS5_NO_RESULT_MEMO=1): 18,512 / 18,514 MiB, peak 17.98 GiB.
  • So the memos hold about 900 MiB there without a budget and about 150 MiB with the default one, and the budget saves about 0.74 GiB of private memory. The runs shared the GPU with other work, so frame rates aren't compared.
• Earlier head dbec127257, rebased onto main 5e6ae6e (include list conflict in Recompiler.hpp, both kept). Linux (Ubuntu 26.04.1, GCC 15.2, Release, RTX 4070 Ti, 615.71.09): the 44 tests matching ^agc_shader_result_memo_budget|^agc_shader_memory$|^agc_shader_|^request_serializer$|shader_disk_cache|^agc_driver_graphics$|image_table pass 3 of 3. Khronos layer 1.4.363 on that set: main 42 of 42, this head 44 of 44, no VUID on either side. Windows 11 VM (GT 1030, 582.66): main 6 of 6 and this head 8 of 8 of the memo, shader memory, serializer and disk cache tests, 3 of 3.

Ubuntu 26.04.1, GCC 15.2, NVIDIA RTX 4070 Ti (615.71.09).

Rebased onto main 0a0ed72 as 59700b2200. git range-diff shows both commits unchanged. The tests below marked "this head" ran on it.

Ghost of Tsushima Director's Cut (PPSA02225). Main can't take this title past its first minutes, so these runs use @EagleBearerYS's test/ghost-9742-9717 stack with #8115, #8121, #8213, #8608 and #9318, and this change applied with no budget unless APS5_RESULT_MEMO_MB set one. Each run is 300 s warm, journal replay off, 20 GiB cap. Results and MiB are the memo line at the end of each run; hits are its last 10 s.

run	budget	results held	MiB held	hits	RssAnon at 290 s
P	none	11,407	473	12.7 %	5,103 MB
U2	none	11,516	550	12.7 %	5,240 MB
P2	none	11,513	489	12.7 %	4,756 MB
B	256 MiB	11,339	235	13.1 %	4,364 MB
N	memo off (APS5_NO_RESULT_MEMO=1)				4,621 MB

The budget halves the memo's bytes without changing its hit rate. The number of results held barely changes, so the trimmed least recently used results are the large ones. RssAnon varies by a few hundred MB between identical runs here, so the memo's own count is the measure.

The Last of Us Part I (PPSA03396), from @Pacuka (comment): CachyOS, RX 6650 XT, 32 GB. Without this change, the process's anonymous memory grew about 1 GiB every 10 s, until the host ran out of RAM and swap and a dispatch failed with std::bad_alloc after 1.5 to 5 minutes. An allocation tracker put the never-freed bytes under materializePreparedMemoized: 962 MiB of DescriptorBindingBuilder::Populate vector copies and 410 MiB under materializeResult after 2 minutes. APS5_NO_RESULT_MEMO=1 stopped the growth. With this PR's two commits, anonymous memory stayed between 2 and 4 GiB through a 7-minute run with no failure.

Forspoken (PPSA03984), on main 895f47f with this change (an earlier head with the same code), 300 s, 16 GiB cap. Hits are over the whole run:

run	budget	hits	trimmed	results held	MiB held
cold	none (APS5_RESULT_MEMO_MB=0)	10.2 %	0	44,387	223
warm	256 MiB	11.0 %	11,819	43,883	221

Here the memo stays near the budget, and trimming doesn't cost hits.

Two later warm runs of that build, one with the budget off, lost the device at 161 s (VK_ERROR_DEVICE_LOST in vkWaitForFences). With the budget off this change only counts bytes, and the eviction order within a set is the same as main's. In a second set of runs past that point, 4 of this build (1 cold and 3 warm, budget off) and 5 of main at the same base (1 cold and 4 warm) all ran without a device loss. I read it as a fault of that GPU session and mention it in case anyone sees it again.

Tests

• New agc_shader_result_memo_budget (tests/ShaderMemory.cpp, CPU only): a pixel program that reads its user data, compiled through Recompile(request, capture) with four user data values. With a budget of three results, a fourth is trimmed. The memo stays within the budget after every insert, the two most recently used results still hit, and the oldest misses. Clearing the budget lets the memo grow past it again.
• New agc_shader_result_memo_budget_threads (CPU only), for the race @MarcZX10 described: with three results in a budget of three, the test inserts a fourth, and the trim hook starts two threads that each insert one more. The hook waits until both insertions are counted, so they land after the running trim has read the held bytes. The test then checks that the memo ends within its budget.
  • With the earlier try_to_lock trim (the threads skip their own trims, and the running trim works from the count it read before them), it fails 10 of 10: "results inserted during a trim left the memo over its budget".
  • With this head's trim it passes 20 of 20, together with agc_shader_result_memo_budget, on 59700b2200 and before the rebase on 2fb7fb6e57.
• The 42 tests matching ^agc_shader_result_memo_budget|^agc_shader_memory$|^agc_shader_|^request_serializer$|shader_disk_cache|^agc_driver_graphics$|image_table pass 3 of 3 on this head (41 before the rebase, on 2fb7fb6e57).
• Khronos validation layer 1.4.363, main 0a0ed72 against this head over ^agc_driver_graphics$|image_table|^agc_driver_prepared (34 tests each): both pass, and both report only main's VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (x14, which fix(agc): copy a linked pipeline's color sample counts only when it has color attachments #7985 fixes). Nothing is new in this branch.
• tools/env_reference.py --check finds the new row. Its one remaining error, APS5_NO_WGI, is undocumented on main too.
• With the trim disabled (an early return in trimResultMemo), agc_shader_result_memo_budget fails 3 of 3: "the memo holds more than its budget".
• Windows 11 VM (GeForce GT 1030, NVIDIA R580, WinLibs GCC, Release, ANYPS5_REQUIRE_VULKAN=1): the 7 tests matching ^agc_shader_result_memo_budget$|^agc_shader_memory$|^request_serializer$|shader_disk_cache pass 3 of 3 on the first head of this branch. On this head, the 8 tests matching ^agc_shader_result_memo_budget|^agc_shader_memory$|^request_serializer$|shader_disk_cache, agc_shader_result_memo_budget_threads among them, pass 3 of 3. The 6 of them that exist on main pass there too, at main 0a0ed72.

Checklist

• Based on current main; no other open PR implements the same functions (perf(shader): reuse an evicted memo result's bindings for the next miss #9049 changes the same miss path in materializePreparedMemoized to reuse an evicted result's bindings; see Overlap)
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: NVIDIA RTX 4070 Ti (Linux); the new test runs on the CPU; AMD Radeon RX 6650 XT, RADV, CachyOS (@Pacuka)
• AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

[MarcZX10]

The global budget can remain exceeded after concurrent insertions. trimResultMemo uses try_to_lock; when another trim owns the mutex, this call returns without trimming or recording pending work. If the holder sampled heldBytes before the other insertion, it may trim only to 75% of the old budget and exit; absent a later insertion, the new bytes remain over budget indefinitely. Could the losing caller retry/schedule a trim, or could trimming serialize through a blocking lock? A focused concurrent test should arrange an insertion after the active trim reads heldBytes and confirm the budget converges without requiring another compile.

[MarcZX10]

The global result memo budget can remain exceeded after concurrent insertions. trimResultMemo uses try_to_lock; when another trim owns the mutex, this call returns without trimming or recording pending work. If the holder sampled heldBytes before the concurrent insertion, it may trim only to 75% of the earlier value and exit, leaving the new bytes over budget indefinitely if no later insertion occurs. Please serialize/retry/schedule the trim and add a focused concurrent regression test. I posted the detailed interleaving in the PR discussion. Existing reported single-threaded tests and GPU measurements are useful but do not exercise this case; I could not rerun locally because the build failed on an unrelated object with / out of space.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6959	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp L438
PR 6970	core/shader/recompiler/Recompiler.hpp L16
PR 9897	core/libs/prx/libSceAgcDriver/CMakeLists.txt L2559

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 1368	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1395	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1896	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 2434	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5021	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/Recompiler.cpp
PR 5371	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
PR 5609	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp
PR 5925	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6146	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6187	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6188	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/Recompiler.hpp
PR 6192	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6209	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6235	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.hpp
PR 6280	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/ShaderDiskCache.cpp
PR 6300	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6412	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6481	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6488	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 6642	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6808	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6924	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 7048	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7201	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp
PR 7275	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7410	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp
PR 7449	docs/dev/ENVIRONMENT.md
PR 7457	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 7484	core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp docs/dev/ENVIRONMENT.md
PR 7604	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 7642	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 7684	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/ShaderDiskCache.cpp
PR 7689	core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp
PR 8004	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp docs/dev/ENVIRONMENT.md
PR 8014	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 8246	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 8484	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 8592	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9012	docs/dev/ENVIRONMENT.md
PR 9176	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 9248	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 9287	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 9289	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 9345	docs/dev/ENVIRONMENT.md
PR 9356	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 9387	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 9402	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9508	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 9676	docs/dev/ENVIRONMENT.md
PR 9700	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9748	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp
PR 9852	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 10026	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10049	core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp
PR 10193	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp core/shader/recompiler/ShaderDiskCache.cpp docs/dev/ENVIRONMENT.md
PR 10233	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp
PR 10442	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 10547	core/libs/prx/libSceAgcDriver/tests/ShaderMemory.cpp core/shader/recompiler/Recompiler.cpp core/shader/recompiler/Recompiler.hpp docs/dev/ENVIRONMENT.md
PR 11157	core/libs/prx/libSceAgcDriver/CMakeLists.txt

[MarcZX10]

Re-reviewed exact head 95955a6 against current main 9d6adeb. The concurrency issue from my earlier review is addressed: trimming now serializes on a blocking mutex, reloads the global held-byte count until it is within budget, and includes a separate eight-thread regression test registered with CTest. I also checked that the new test has its own registration and is included in the Windows runtime-path properties.

I reran the convention check against current main, the 172-script registration check, and git diff --check; all pass. I could not run the C++ budget tests locally because this workspace has no configured AnyPS5 build or test executable. The PR reports the focused CPU tests, including the concurrent budget case, passing on Linux and Windows; hosted product build jobs are skipped by repository policy.

The description calls out the remaining integration point with open #9049: if that reuse change lands after this PR, its slot removal must also update the byte/entry counters. The current branch is mergeable and the same-function overlap scan found no duplicate implementation. With that integration detail recorded and the earlier concurrency finding resolved, I found no blocking issue on this head.

[milescoviello]

@MarcZX10 thanks for the re-review. One correction to what the approved head shows: its eight-thread test passes with the old try_to_lock trim too. I ran it 20 times against the old trim and it never failed, because the window is too narrow for a small pixel program. So it checks the new trim, but it isn't the regression test you asked for.

A version that arranges the race is on fix/result-memo-budget-race-test (2fb7fb6e57). I haven't pushed it here, so the head you approved stays unchanged.

• It adds a one-shot hook, SetResultMemoTrimHook, which the trim calls right after it reads heldBytes. Setting it takes the trim mutex.
• With three results in a budget of three, the test inserts a fourth, and the hook starts two threads that each insert one more. It waits until both insertions are counted.
• With the old trim, those threads skip their trims, and the running trim works from the count it read before them, so the memo ends one result over: 10 of 10 runs fail with "results inserted during a trim left the memo over its budget". The new trim passes 20 of 20, and the 41 focused tests pass 3 of 3.

Would you rather have that version on this PR, or keep the head as approved? I'll push it here only if you want it.

AI-assisted: yes (Claude Code)

[MarcZX10]

The author’s follow-up establishes that the current eight-thread test passes with both the new and old trim implementation, so it does not detect the reported race. The proposed deterministic race test on fix/result-memo-budget-race-test is described as 10/10 failures with the old trim and 20/20 passes with the new one, which directly covers the missing regression case. Please add that test (or an equivalent committed red/green test) to this PR and rerun the focused 41-test suite. The current head is still clean against main and policy checks pass, but I’m withdrawing my earlier approval until the committed test distinguishes the fix from the old behavior.

AI-assisted review.

[milescoviello]

@MarcZX10 thanks. The deterministic test is on this PR now: the head is 2fb7fb6e57, with the same parent as the approved 95955a6b45. Against that head it adds only SetResultMemoTrimHook and the rewritten agc_shader_result_memo_budget_threads.

On this head:

• agc_shader_result_memo_budget_threads and agc_shader_result_memo_budget pass 20 of 20.
• With the old try_to_lock trim, the threads test fails 10 of 10: "results inserted during a trim left the memo over its budget".
• The 41 focused tests pass 3 of 3.

I updated How and Tested in the description to match: the blocking trim, the hook and the new test. A Windows VM run of this head comes next. The VM is busy with another PR's run for about an hour.

AI-assisted: yes (Claude Code)

[MarcZX10]

Re-reviewed exact head 2fb7fb6e57db6751f3815732022a8f2ce08dcaac. The prior race concern is addressed in the committed test: the trim hook starts two inserts after the active trim sampled heldBytes, waits for their accounting, then the test asserts the completed trim is within budget. The description reports the old implementation failing 10/10 and this implementation passing 20/20, plus the focused 41-test Linux suite and Windows focused tests. The implementation holds a blocking trim mutex and reloads the byte count after each eviction pass, matching the tested race.

I also ran the exact-head convention check and git diff --check; both pass. check_main_sync.py reports a clean merge, but this head is 2 commits ahead and 138 behind current upstream/main (bd4906d7), with both branches changing Recompiler.cpp, Recompiler.hpp, ShaderDiskCache.cpp, the AGC CMake registrations and ENVIRONMENT.md. The intervening main changes include recompiler fixes and new tests in those shared files. Please rebase this branch onto current main, resolve the shared-file changes, and rerun the focused result-memo tests and the matched Vulkan validation checks on the resulting main/head pair. Until those checks cover the current base, I can’t approve this stale snapshot. This is a current-base and validation request; the previously reported race itself is resolved.

[milescoviello]

@MarcZX10 rebased onto main 0a0ed72. The head is 59700b2200, and git range-diff shows both commits unchanged. The rebase had no conflicts in the shared files. On the new main/head pair:

• agc_shader_result_memo_budget and _threads pass 20 of 20.
• The 42 focused tests (41 before; main added one in the set) pass 3 of 3.
• Khronos validation layer 1.4.363, main 0a0ed72 against this head over ^agc_driver_graphics$|image_table|^agc_driver_prepared: 34 of 34 on both. Both report only main's pAttachments-00605 (x14), and nothing is new in the head.
• Windows 11 VM (GT 1030): the 8 memo, shader-memory, serializer and disk-cache tests pass 3 of 3 on the head, and the 6 that exist on main pass there.

The description's Tested section now names this pair.

AI-assisted: yes (Claude Code)

[EagleBearerYS]

@milescoviello a note on order with #10000 (ours, the sampler and sampled-pair limits): both PRs add test names to the same set_tests_properties(agc_shader_bindless ...) line in core/libs/prx/libSceAgcDriver/CMakeLists.txt, so they conflict on that one line. #10000 goes second: when this PR merges first, #10000 takes the conflict and keeps both PRs' test names. Nothing needed on your side.

[milescoviello]

@EagleBearerYS thanks, agreed. If #10000 lands first instead, I'll keep both PRs' test names on that line here.

AI-assisted: yes (Claude Code)

[Pacuka]

Data point from The Last of Us Part I (PPSA03396), CachyOS, RX 6650 XT, 32 GB RAM. Without this PR the process's anonymous memory grew about 1 GiB every 10 s until the host ran out (RAM + 16 GB swap) and a dispatch failed with std::bad_alloc after 1.5-5 minutes. An allocation tracker (LD_PRELOAD, allocations ≥ 256 KiB by call stack) showed the never-freed bytes under materializePreparedMemoized: DescriptorBindingBuilder::Populate vector copies held 962 MiB (1248 allocations of ~0.8 MiB, none freed) and materializeResult 410 MiB after 2 minutes. APS5_NO_RESULT_MEMO=1 stopped the growth. With 64817c5 and 59700b2 on top of my branch, anonymous memory stayed between 2 and 4 GiB for a 7-minute run with no failure.

AI-assisted: yes (Claude Code)

[milescoviello]

@Pacuka thank you. A second title that runs out of memory without this, with an allocation tracker pointing at the same materializePreparedMemoized path, is the strongest evidence this PR has. I added the run to Tested and the card to the GPU list.

AI-assisted: yes (Claude Code)

[milescoviello]

@MarcZX10 rebased onto current main 5e6ae6e as dbec127257, still the same two commits. The only conflict was the include list of Recompiler.hpp: main added <ranges> where this branch adds <functional>, and both stay. The journal and disk cache format versions are main's (46).

On that main/head pair:

• Linux (Ubuntu 26.04.1, GCC 15.2, Release, RTX 4070 Ti, NVIDIA 615.71.09): the 44 tests matching ^agc_shader_result_memo_budget|^agc_shader_memory$|^agc_shader_|^request_serializer$|shader_disk_cache|^agc_driver_graphics$|image_table pass 3 of 3, both result memo budget tests among them.
• Khronos validation layer (LunarG SDK 1.4.363, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1), same set: main 5e6ae6e passes its 42 of 42, this head 44 of 44. Neither log reports the layer missing, and neither has a VUID.
• Windows 11 VM (GeForce GT 1030, 582.66, WinLibs GCC 15.2.0 r7, Release): the tests matching ^agc_shader_result_memo_budget|^agc_shader_memory$|^request_serializer$|shader_disk_cache pass 3 of 3: main 6 of 6, this head 8 of 8.

Also a cross-title look at how much the memos hold, from jemalloc heap profiles on main (300 s each, Linux). Live heap reached through materializeResult/materializePreparedMemoized, which is what this budget caps at 256 MiB by default: Super Meat Boy 3D (PPSA31162) 717 MiB, MXGP 26 (PPSA30839) 548 MiB, Ghost of Tsushima (PPSA02225) about 376 MiB. Some of those results are also kept by the draw caches, so the budget frees less than the difference. An A/B with this head on the first two titles is running, and I'll post its RSS numbers.

AI-assisted: yes (Claude Code)

[milescoviello]

The A/B with this head (dbec127257) against main 5e6ae6e, as promised. Linux, RTX 4070 Ti (615.71.09), Release libs, 12 GB cap, 300 s with pad input and APS5_PROFILE_DRAW=1, main/PR interleaved. Each side's first run fills its own shader cache version (the recompiler sources differ), so the second runs are the warm pair. MB of RssAnon, sampled every 0.5 s:

title, run	main peak	this PR peak	main at the end	this PR at the end	PR memo trims in the last 10 s
Super Meat Boy 3D, cold	3,079	2,089 (stopped at 51 s, below)	2,994	1,951
Super Meat Boy 3D, warm	2,559	2,578	2,492	2,527	18,372
MXGP 26, cold	3,613	3,754	2,610	2,419	12,860
MXGP 26, warm	3,401	3,230	2,387	1,636	12,757

• Super Meat Boy 3D shows no difference: its memos sit near the 256 MiB budget, so the trims free little the draw caches don't also hold.
• MXGP ends the warm run 751 MB (31 %) lower, with the peak 171 MB (5 %) lower; the cold pair ends 191 MB lower. That is one pair per state on a title whose memory varies with the scene, so I'd read it as a direction, not a number. The case this PR was written for, a title that grows without bound, is still Pacuka's The Last of Us run in the description.
• The cold Super Meat Boy 3D run with this head stopped at 51 s on storage image access to depth/stencil surface 0x30917d0000 is not implemented (compute shader 0x1b40210000). The warm run with the same head played the full 300 s, as did every main run. Nothing in this PR touches image classification, so I take it for a timing-dependent path of the cold run. I'll run that title again cold on both sides to make sure.

AI-assisted: yes (Claude Code)

[milescoviello]

Follow-up on the cold Super Meat Boy 3D stop: four more cold runs (no AnyPS5 shader cache, KEEP_CACHE=0), interleaved, 300 s each. main 5e6ae6e stopped once at 49 s with the same storage image access to depth/stencil surface ... is not implemented from compute shader 0x1b40210000, and ran the full time once. This head ran the full time both times. So the stop is a timing-dependent path that main hits too, not this PR.

AI-assisted: yes (Claude Code)

[meh7an]

@milescoviello the overlap check pairs this PR with #10423 (masked result memo) and #10757 (recycled results). #10423 adds a second per-source table, maskedMemo, whose entries own full RecompileResults, and #10757 keeps up to 8 evicted results per thread for reuse. Neither is in ResultMemoBytes(), so with all three merged the budget would not bound them.

Proposal: this PR goes first. Once it merges, I rebase #10423 and #10757 onto it:

• masked-table entries count toward ResultMemoBytes() (a result held by both tables counts once) and are trimmed with the exact table;
• the recycle pool counts too, and is emptied first when a trim runs;
• agc_shader_result_memo_budget gains masked hits and recycled results.

#10423 and #10757 then declare Depends on: #9989. Does that order work for you?

[milescoviello]

@meh7an yes, that order works for me: this PR first, then #10423 and #10757 rebased onto it with Depends on: #9989. Counting the masked table and the recycle pool in ResultMemoBytes() keeps the budget meaning "bytes the memos hold", and emptying the recycle pool first on a trim fits how this PR trims: it evicts the least recently used entries until the memos hold three quarters of the budget, and a recycled result can never be a hit, so it should go before any memo entry. A result held by both tables counting once fits too.

This head (dbec127257) is rebased onto main after MarcZX10's last request and is waiting on their re-review. If anything changes ResultMemoBytes() here before it merges, I'll say so on your two PRs.

AI-assisted: yes (Claude Code)

[thuanquang]

@milescoviello here is a data point for the rebase onto main with #10423.

• Title: Persona 5 Royal (PPSA05110, own dump).
• Machine: Windows 11 26200, RX 6700 XT, WinLibs GCC 15.2.0 r7, Release.
• Build: main d2b3ee3bcc plus feat(libSceImeBackend): declare 22 sceImeBackend exports #10887, fix(libSceGameUpdate): answer no update instead of throwing on request creation #10614, fix(libSceVideodec2): mark returned pictures as not discarded #10913 and fix(agc): keep at most 4 GiB of the host heap from the host import budget #7844, which the title needs to load and run.
• Runs: two runs follow the same scripted route into the first gameplay scene and then stand still from about 200 s. The HUD in the frame dumps confirms both reached gameplay. The only difference between them is APS5_NO_RESULT_MEMO=1. Private bytes are sampled from outside the process.

	private at 300 / 500 / 700 s (MiB)	peak private
result memos on	19,303 / 19,310 / 19,321	18.87 GiB
APS5_NO_RESULT_MEMO=1	18,399 / 18,396 / 18,411	17.98 GiB

In this scene the memos hold about 0.9 GiB, flat from 300 to 700 s. This includes #10423's masked table, which main now has. The two runs shared the GPU with other work, so their frame rates are not comparable. Once this PR is rebased and counts maskedMemo, I can run the same pair with it.

AI-assisted: yes (Claude Code); results above were run on this machine.

[milescoviello]

@thuanquang thank you. The flat 0.9 GiB from 300 to 700 s is a useful baseline for what the memos hold in that scene.

#10423 landed before this PR, so I've rebased onto main 442c43a and folded the masked memo into the budget:

• Each maskedMemo entry counts its result plus its patch recipe.
• The trim walks both tables of every source under one LRU clock.
• A new agc_shader_result_memo_budget_masked test (masked memo on) checks that the masked entries are counted and that a trim empties that table too. The two existing budget tests now run with APS5_NO_MASKED_RESULT_MEMO=1, so their byte counts stay exact.

I'm running the shader tests on it now. I'll push it here once they pass and post when it's up. Then the same pair (default budget vs APS5_NO_RESULT_MEMO=1) would show whether 256 MiB holds Persona 5's scene.

AI-assisted: yes (Claude Code)

[milescoviello]

@thuanquang @MarcZX10 rebased onto current main 442c43a and pushed as 8868e09: one commit, with #10423's masked memo counted and trimmed under the same budget.

• Linux (Ubuntu 26.04.1, RTX 4070 Ti, GCC 15.2, Release): the 37 ^agc_shader_ tests pass 3 of 3. They include the three budget tests and perf(shader): reuse a memoized result by patching its copied words #10423's agc_shader_table_snapshot_reuse/_unmasked.
• Mutant: a trim that skips the masked table fails the new agc_shader_result_memo_budget_masked with a trim to one byte left 620 bytes.

The matched Khronos validation comparison (main 442c43a vs this head over the same 37) and my Windows VM run are going now. I'll put both in the description when they finish.

AI-assisted: yes (Claude Code)

[thuanquang]

@milescoviello here is the Persona 5 Royal pair on 8868e09624, as promised.

• Build: the same machine and route as before. The base is main 73ad397dba plus feat(libSceImeBackend): declare 22 sceImeBackend exports #10887, fix(libSceGameUpdate): answer no update instead of throwing on request creation #10614, fix(libSceVideodec2): mark returned pictures as not discarded #10913 and fix(agc): keep at most 4 GiB of the host heap from the host import budget #7844, with this PR merged in cleanly.
• Runs: three runs from that one build, each 780 s into the first gameplay scene, then standing still. The frame dumps' HUD confirms that every run reached gameplay, at 242 to 307 s after New Game. The snapshots at 500 and 700 s are all in gameplay.
• Measurement: private bytes are sampled from outside the process.

result memos	private at 500 / 700 s (MiB)	peak private
no budget (APS5_RESULT_MEMO_MB=0, main's behaviour)	19,408 / 19,416	18.99 GiB
this PR's default, 256 MiB	18,652 / 18,680	18.25 GiB
off (APS5_NO_RESULT_MEMO=1)	18,512 / 18,514	18.08 GiB

In this scene the memos hold about 900 MiB without a budget and about 150 MiB with the default one, so the budget saves about 0.74 GiB of private memory. All three runs shared the GPU with other work, so I don't compare their frame rates.

AI-assisted: yes (Claude Code); results above were run on this machine.

[milescoviello]

@thuanquang thank you, that's the measurement this PR needed: about 900 MiB down to about 150 MiB, 0.74 GiB of private memory saved in Persona 5's first gameplay scene, with the memos-off run as the floor. I've added it to Tested.

AI-assisted: yes (Claude Code)

Dismissing a stale review: the requested changes were addressed after it (new head) and the merge queue blocks on it. Please request changes again if anything is still open.

[adriadam10]

Checked the byte accounting (result, bindings, masked entry and patch recipe), the lock order (source then registry; the trim takes its own mutex and then each source's, outside the insert lock), the shared LRU clock, and that the three budget tests are registered, including the Windows PATH list. APS5_RESULT_MEMO_MB is neutral for the disk cache key and documented. The earlier concurrency request is covered by agc_shader_result_memo_budget_threads, and the head is rebased onto current main with the masked memo counted. Memory numbers from Persona 5 Royal and The Last of Us Part I back the change; merged.

[milescoviello]

Thanks for the review and the merge, @adriadam10.

AI-assisted: yes (Claude Code)