[milescoviello]

What

Ghost of Tsushima (PPSA02225 02.024.000) is killed at the 20 GB memory cap about 150 s in, in every run with a warm shader cache: at 121, 159, 152 and 155 s in four runs. Each time, anonymous memory climbs from about 4 GB by 2.5 to 5 GB within 8 to 10 s, and the run is killed.

A preloaded allocation hook that prints a backtrace each time one thread's allocations pass another 512 MiB shows where it goes. Eleven threads of the prebuild pool run Driver::dispatch's resolve-ahead task, which calls VulkanDevice::PrebuildComputePipeline and then vkCreateComputePipelines. Inside that call, NVIDIA's compiler (libnvidia-gpucomp.so) allocates 0.5 to 1 GB per thread. Ghost of Tsushima's compute modules reach 800,000 SPIR-V words (3.2 MB), so the pool, with half the hardware threads (12 here), runs eleven of these compiles at once. The pipelines never reach the driver's disk cache, because the run is killed first, so every run dies at the same place.

With APS5_PREBUILD_THREADS=2 the same run goes the full 308 s, with anon peaking at 7.4 GB. So it is the number of large compiles in flight that kills it, not the compiles themselves.

How

CompileBudget (Execution/include/CompileBudget.hpp) bounds the SPIR-V bytes being compiled at once. A build that would exceed the limit waits until earlier ones finish. A module larger than the whole limit builds alone: while one waits, builds that would fit wait behind it, so it runs as soon as the builds already in flight finish, and smaller builds can't starve it.

VulkanDevice takes a CompileBudget::Hold sized to the module's SPIR-V around both background compute builds:

• PrebuildComputePipeline, the dispatch-ahead build above. It checks HasComputePipeline again after the wait, because a dispatch may have built the pipeline meanwhile.
• ReplayComputePipeline, the pipeline journal's compute replay. It has the same shape, a pool of half the hardware threads building recorded modules, and a journal holds the same giant modules. With a warm driver cache these builds are mostly cache hits, but with a cold one the replay meets the same compiles.

A dispatch that needs its pipeline now never waits on the budget. If a prebuild is still queued, the dispatch builds the pipeline itself.

A compute prebuild waits for the budget inside a worker of the prebuild pool, which the graphics library prebuilds share, so a burst of large compute modules could otherwise hold every worker (review). SubmitComputePrebuild puts compute prebuilds in the same pool marked as budgeted, and the pool starts one only while fewer than half of its workers are running them (at least one). A worker takes the first queued task it may start. So library prebuilds queued behind waiting compute prebuilds run on the other workers, compute prebuilds still start in the order they were queued, and the pool's thread count still bounds all prebuilds together. ENVIRONMENT.md's APS5_PREBUILD_THREADS row says so.

The limit is APS5_BACKGROUND_COMPILE_KIB, 8192 KiB by default. 0 builds them one at a time, and ENVIRONMENT.md documents it. The default comes from measurements, after review (data):

• Module sizes: in five titles' dumped requests, compute modules have medians of 239 to 671 KiB and means of 0.65 to 1.5 MiB. Twelve modules of mean size at once come to 7.8 MiB (Crimson Desert) to 17.6 MiB (Forspoken).
• Ghost of Tsushima with a fresh driver cache, one run per setting. Without a limit, peak anon was 9.3 GB, and queue 0 spent 75.6 s compiling pipelines itself. At 8 MiB, peak anon was 6.7 GB, queue 0 spent 37.6 s, and about 430 pipelines were built in 240 s against about 300. Fewer contending compiles finish sooner. 16 MiB was killed at the cap at 217 s.

#9626 (MarcZX10) bounds registration-time shader preparation by process memory, defaulting to half of physical RAM. It covers a different stage: the recompiler's preparation, not the driver's pipeline compile. The two don't touch the same code.

Tested

• This head a937addfb8 (XairRrR's review):
  • ReplayComputePipeline checks again under the pipeline mutex after the budget hold, as PrebuildComputePipeline does.
  • agc_driver_compute_prebuild_share runs with APS5_PREBUILD_THREADS=4 from its CTest entry. It passes 1 of 1 with APS5_PREBUILD_THREADS=1 exported in the shell.
  • The 12 tests matching ^agc_driver_compute_prebuild_share$|^agc_compile_budget$|^agc_driver_registration_prepare_stack$|^agc_driver_shutdown_threads pass 3 of 3.
  • Validation layer (Khronos 1.4.363, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 ctest -V) over the same regex: the merge base f759e165d9 passes 10 of 10, and this head 12 of 12. Neither log has a missing-layer line or a VUID- line. These tests don't print the devices they open, so I can't say how many the layer checked. XairRrR's run also shows no VUID- lines.

Linux (Ubuntu 26.04.1, GCC 15.2), RTX 4070 Ti with NVIDIA 615.71.09.

• Ghost of Tsushima, 20 GB cap, warm caches, the same input script. It doesn't run on plain main (it stops at 37 s on a native multisample depth import, fix(agc): accept a colour alias over a native multisample depth plane when HTILE proves its clear #9742), so these runs use a local stack: main d13b3d7c55 with feat(agc): sample the stencil plane of a native multisampled D32S8 target #9717, fix(agc): accept a colour alias over a native multisample depth plane when HTILE proves its clear #9742, fix(agc): bound the draw cache by bytes #9129, fix(agc): build a partially committed sampled 3D view from its committed slices #10064 and fix(agc): keep a registered shader's register state as a sorted list #10727 merged (5b7d732a35). With this change cherry-picked on top (89eaf64f77):

Run	Result	Peak anon
stack, default prebuild pool (4 runs)	killed at 121, 159, 152, 155 s	6.6, 9.5, 9.2, 9.4 GB at the last sample before the kill
stack, APS5_PREBUILD_THREADS=2	ran the full 308 s	7.4 GB
stack + this change, its first 4 MiB default	ran the full 428 s	7.3 GB

• Tests at this head (feb568b46d, on main f759e165d9): the new agc_compile_budget covers the default and configured limits, eight 40-byte holds under a 100-byte limit never holding more than 80 bytes at once, a zero limit running holds one at a time, a 500-byte hold under a 100-byte limit running alone, and a waiting 500-byte hold keeping out a 40-byte hold that would fit. That last case fails with the old wait condition. It passes 3 of 3 (--repeat until-fail:3). At 74fb986752, which differs from this head only in linking libc into the new test target, agc_pipeline_journal and agc_pipeline_journal_long_paths also pass 3 of 3. agc_driver_pipeline_journal fails at both this head and main f759e165d9, 3 of 3 each, built the same way, with the same message (AGC graphics: the oversized record was not counted as a write failure, in OversizedRecordKeepsTheJournal). So that failure predates this change. In both logs, the device opened for that case enumerates only llvmpipe, because the NVIDIA driver drops out after the earlier instances. The journal then turns itself off for a foreign device. test(agc): keep the Vulkan drivers loaded across the pipeline journal test #11721 (jotitadev) keeps the Vulkan drivers loaded across this test for that reason. Every case before it passes, including the replays that go through ReplayComputePipeline. The one after it, HeldModuleTests, doesn't run on either side.
• Pool share at 759ec763c9: the new agc_driver_compute_prebuild_share submits one blocking compute prebuild per prebuild worker (12 here). It checks that exactly half start, that a library prebuild queued behind them runs while they wait, and that all of them finish once released. It passes 3 of 3. With SubmitComputePrebuild submitting unbudgeted tasks, as compute prebuilds were submitted before this commit, it fails: 8 of 12 blocked compute prebuilds started, not 6. agc_compile_budget, agc_driver_registration_prepare_stack and the agc_driver_shutdown_threads* and agc_driver_prepared_shaders* tests, 34 in all, pass 3 of 3 at c5193e673f. That commit has the same code; the test was then changed to release its blocked tasks when a check fails, so a failure ends in 10 s instead of hanging. Ghost of Tsushima with this commit (the setup of the 8 MiB run above) runs to the 300 s timeout with an anon peak of 6.3 GB. That compares with 6.7 GB before this commit and 9.3 GB without a limit. About 40 to 46 % of its graphics libraries are built at the draw with every setting, no limit included (counts).
• Windows 11 VM at 759ec763c9: agc_driver_compute_prebuild_share, agc_compile_budget, agc_driver_registration_prepare_stack and the agc_driver_shutdown_threads* tests (12) pass 3 of 3. The merge base f759e165d9 passes the 10 of them it has, 3 of 3.
• Windows 11 VM (WinLibs GCC 15.2, Release, GT 1030 with NVIDIA R580): agc_compile_budget and agc_driver_pipeline_journal pass 3 of 3 at feb568b46d. agc_driver_pipeline_journal also passes 3 of 3 at main f759e165d9. The first Windows build failed to link the new test (__gxx_personality_v0), because it didn't link libc the way agc_handoff_spin_tests does. beaf0279aa fixed that.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none (fix(agc): build a replayed compute pipeline once when a dispatch races it #12657 also changes ReplayComputePipeline; agreed on fix(agc): build a replayed compute pipeline once when a dispatch races it #12657: this lands first and fix(agc): build a replayed compute pipeline once when a dispatch races it #12657 rebases after it, taking the hold before it registers the build)
• Fixes: none
• Unblocks: PPSA02225 Ghost of Tsushima Director's Cut (PPSA02225) — bring-up progress #4461 (Ghost of Tsushima, past its memory spike at about 150 s)
• GPUs tested: NVIDIA RTX 4070 Ti (Linux, 615.71.09)
• AI-assisted: yes

🤖 Generated with Claude Code

[oneandonlydean]

@milescoviello thanks, the budget is a clean fix for the memory spike and I found no deadlock: the prebuild runs on the pool holding no lock, and a dispatch never waits on the budget. Two things before this goes in:

1. Oversized modules can be delayed. Hold has no queue, so while smaller builds keep entering (they fit), held may not reach 0 and a module larger than the limit keeps waiting. "A module larger than the whole limit builds alone, so nothing waits forever" only holds once the smaller builds stop. Please make a waiting oversized hold keep new holds out (for example, a waiting-oversize flag or a FIFO ticket), and add a test case for it.
2. The default needs data beyond one title. 4 MiB is sized from the 3.2 MB module of one title on one driver, and "most titles have modules of tens of KiB, so parallelism is unchanged" is not backed by numbers. Please add either the SPIR-V size distribution of compute modules from one or two other titles, or a cold-cache comparison (time to finish the prebuild and the number of dispatches that had to build their own pipeline) on a title with many 0.5 to 2 MB modules, before and after.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 6867	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp L1574-1576, L1688, L1694-1695

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 1368	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1395	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 1896	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp docs/dev/ENVIRONMENT.md
PR 2434	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4040	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4449	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 4472	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 5021	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5371	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5925	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6185	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 6186	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 6187	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6188	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6192	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6209	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6280	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6300	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6412	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6481	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6501	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 6804	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6808	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6905	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6924	docs/dev/ENVIRONMENT.md
PR 6960	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6977	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7048	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7085	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7275	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7369	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7449	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 7457	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 7464	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7514	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 7604	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 7642	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 7682	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 7689	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7922	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 8004	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 8014	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 8140	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 8246	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 8484	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 8592	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9012	docs/dev/ENVIRONMENT.md
PR 9020	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9335	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9345	docs/dev/ENVIRONMENT.md
PR 9356	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 9387	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 9432	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9489	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9700	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9712	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 9748	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9884	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9891	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9893	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9895	core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 9897	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9898	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 9899	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 10026	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 10069	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 10193	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/include/Driver/Shaders/ShaderRegistry.hpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 10233	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10442	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 10472	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 10492	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10497	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10547	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 10652	core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 10757	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 10894	docs/dev/ENVIRONMENT.md
PR 10920	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 11351	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 11442	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 11444	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 11554	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 11948	docs/dev/ENVIRONMENT.md
PR 12070	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 12120	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp docs/dev/ENVIRONMENT.md
PR 12324	core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md
PR 12544	docs/dev/ENVIRONMENT.md
PR 12569	docs/dev/ENVIRONMENT.md
PR 12828	core/libs/prx/libSceAgcDriver/CMakeLists.txt core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 12848	docs/dev/ENVIRONMENT.md
PR 12862	core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 12863	docs/dev/ENVIRONMENT.md
PR 12895	docs/dev/ENVIRONMENT.md
PR 12970	docs/dev/ENVIRONMENT.md

[milescoviello]

@oneandonlydean thanks, both are right.

1. f07182a387 makes a waiting larger-than-limit hold keep new holds out: holds that would fit wait while one is queued to run alone, and it enters as soon as the builds already in flight finish. agc_compile_budget has a new case for this. A 40-byte hold that fits must not enter while a 500-byte hold waits, and the 500-byte hold must run first. With the old wait condition it fails on that check, and with the fix it passes 10 of 10 standalone. The CMake target and the Windows VM are next.
2. You're right that "tens of KiB" was a guess, and the first numbers already contradict it: Crimson Desert's compute modules are 160 to 470 KB. I'm measuring the compute-module SPIR-V sizes from the dumped request sets of Crimson Desert, MXGP 26, Super Meat Boy 3D and Forspoken. I'll choose the default from that and check it on Ghost of Tsushima. I'll post the distribution here.

AI-assisted: yes (Claude Code)

[milescoviello]

@oneandonlydean here is the data for point 2.

Compute-module SPIR-V sizes from the dumped shader requests (agc_shader_replay --prepare, the size each prepared compute module has, which is what a prebuild hands the driver):

Title	Compute modules	Median	p90	Max	Mean	12 at the mean	≥ 1 MiB	> 4 MiB
Crimson Desert (PPSA25286 01.000.568)	191	239 KiB	854 KiB	18.1 MiB	667 KiB	7.8 MiB	15	8
MXGP 26 (PPSA30839 01.003.000)	304	671 KiB	2.5 MiB	10.6 MiB	1.1 MiB	13.2 MiB	100	15
Super Meat Boy 3D (PPSA31162 02.000.001)	46	618 KiB	3.6 MiB	9.1 MiB	1.2 MiB	14.3 MiB	14	4
Forspoken (PPSA03984 01.000.001)	63	512 KiB	3.9 MiB	8.7 MiB	1.5 MiB	17.6 MiB	22	6
Ghost of Tsushima (PPSA02225 02.024.000)	89	580 KiB	2.2 MiB	4.8 MiB	886 KiB	10.4 MiB	23	3

So the description's "tens of KiB" was wrong, and I've removed it. With the pool's 12 threads busy on average-sized modules, these titles would have 8 to 18 MiB in flight. A 4 MiB limit would cut that parallelism in every one of them.

Before and after on Ghost of Tsushima. MXGP can't serve as the title here: in my no-network sandbox it stops at 9 s on sceHttp2SetResolveRetry (#11617) and then at its DNS lookup. So these runs use Ghost of Tsushima, on the same local stack as the description, with this PR's three commits. Each run is 300 s at the 20 GB cap, with a warm recompiler cache, a fresh empty driver cache (so every prebuild is a real compile) and a 0–240 s trace. Builds are counted from the trace's Vulkan.ComputePipeline slices by thread: the prebuild pool ("shader prepare" threads) and queue 0's worker, which builds a pipeline itself when no prebuild has done it yet. One run per setting:

APS5_BACKGROUND_COMPILE_KIB	Result	Peak anon	Prebuild pool: builds, compile time	Queue 0 worker: builds, compile time
1048576 (no effective limit, as main)	full 306 s	9.3 GB	217, 423 s	68, 75.6 s
16384	killed at 217 s, at the cap	9.1 GB	(no trace, killed first)
8192	full 309 s	6.7 GB	278, 341 s	107, 37.6 s
4096	full 308 s	6.3 GB	237, 233 s	95, 40.3 s

With the limit, fewer large compiles contend for the CPU, so each finishes sooner. In the same 240 s window the 8 MiB run built about 430 pipelines in all (pool, queue 0 and the other queue workers), against about 300 without a limit. Queue 0 spent half as long compiling (37.6 s against 75.6 s), although it built more pipelines itself, and peak anon was 2.6 GB lower. The prebuild doesn't finish within 240 s at any setting, because Ghost of Tsushima keeps compiling through its opening load. So I can't give a time-to-finish, only these totals. These are single runs on one title. The run without a limit survived this time, at 9.3 GB with the 11.2 GB of guest memory against a 20 GB cap, so how far each run gets near the cap depends partly on chance.

Default: feb568b46d sets it to 8 MiB. That is above Crimson Desert's 7.8 MiB at the mean, so Crimson Desert isn't held back on average. It lets two or three of Ghost of Tsushima's 2 to 3 MB modules compile at once. For MXGP 26, Super Meat Boy 3D and Forspoken, 12 average modules at once (13 to 18 MiB) would be held to 8 MiB. Their modules are of the size where the runs above show less contention, but I haven't measured those titles themselves. I'll update the description with these tables.

AI-assisted: yes (Claude Code)

[milescoviello]

@oneandonlydean the description now has the size and Ghost of Tsushima tables, the 8 MiB default and the starvation case. At head feb568b46d, agc_compile_budget passes 3 of 3 on Linux (RTX 4070 Ti) and on the Windows 11 VM (GT 1030), and agc_driver_pipeline_journal passes 3 of 3 on Windows.

AI-assisted: yes (Claude Code)

[oneandonlydean]

@milescoviello thanks for the size tables and the before/after run. The waiting-oversized fix and its test look right to me.

One new point. A compute prebuild that waits on the budget does its waiting inside a worker of the prebuild pool (SubmitLibraryPrebuild → RegistrationPreparePool::Prebuild()). The vertex and pixel library prebuilds (PrebuildStageLibrary) use the same pool and aren't budgeted. During a burst of large compute modules, every worker in that pool (hardware_concurrency / 2, so 12 on your machine) can be blocked waiting on the budget. Graphics library prebuilds queued behind them then don't run, and those draws build their libraries themselves. Your table may already show the compute side of this: queue 0 built 107 pipelines itself at 8 MiB against 68 without a limit.

Could you do one of these?

• From the same cold-cache run, report how many graphics pipeline libraries were built at the draw rather than by a prebuild, before and after.
• Or keep budget waiters off the shared workers, for example by waiting for the budget before the task takes a pool slot, or by giving compute prebuilds their own queue.

[milescoviello]

@oneandonlydean right, the wait sat inside a shared prebuild worker. 759ec763c9 takes your second option.

Compute prebuilds now go through SubmitComputePrebuild, into the same pool but marked as budgeted. The pool starts one only while fewer than half of its workers are running them (at least one), and a worker takes the first queued task it may start. Library prebuilds queued behind waiting compute prebuilds therefore run on the other half, and compute prebuilds still start in queue order. The pool size still bounds all prebuilds together, so graphics and compute compiles can't add up past what they could before.

The new agc_driver_compute_prebuild_share submits one blocking compute prebuild per prebuild worker (12 here). It checks that exactly 6 start and that a library prebuild queued behind them runs while they wait, then releases them and checks that all finish. It passes 3 of 3. With compute prebuilds submitted unbudgeted, as before this commit, it fails at its first check: 8 of 12 blocked compute prebuilds started, not 6. The 34 budget, pool, shutdown and prepared-shader tests pass 3 of 3 as well. On the Windows 11 VM (GT 1030) the new test, agc_compile_budget, agc_driver_registration_prepare_stack and the agc_driver_shutdown_threads* tests (12 in all) pass 3 of 3 at 759ec763c9. The description is updated.

I haven't rerun Ghost of Tsushima on this yet. The machine doesn't have room for a 20 GB run right now, so the description says so too.

AI-assisted: yes (Claude Code)

[milescoviello]

@oneandonlydean here's the Ghost of Tsushima data for your first option, plus a run of the new head. Setup as before: combo/got28 (5b7d732a35) with this PR's commits cherry-picked, fresh driver cache each run, 20 GB cap, the same input script, trace window 0 to 240 s. One run per setting. Built at the draw means on the draw committer or a queue worker; prebuilt means on the prebuild pool.

Setting	Graphics libraries built at the draw	Graphics libraries prebuilt	Share at the draw	Compute pipelines built at dispatch	Anon peak	Run
no limit	110	149	42 %	82	9,302 MB	to the 300 s timeout
4 MiB	154	178	46 %	135	6,324 MB	to the timeout
8 MiB, before 759ec763c9	178	270	40 %	158	6,669 MB	to the timeout
8 MiB, at 759ec763c9 (pool share)	135	165	45 %	96	6,323 MB	to the timeout

About 40 to 46 % of the graphics libraries were built at the draw with every setting, no limit included. So in this title the libraries built at the draw aren't the ones the budget waits held up, and the pool share doesn't move that fraction outside run-to-run variation. It does keep the compile memory down as before: anon peak 6.3 GB against 9.3 GB without a limit. The share is a guard for the burst you described; these runs don't show the burst.

AI-assisted: yes (Claude Code)

[XairRrR]

Reviewed — approved

The budget and the pool share merge cleanly, build, and pass the focused tests on an RTX 4070 Ti SUPER with the validation layer, with no VUID- lines on either side. Mutating each of the two fixes makes its new test fail. The points below are non-blocking.

What I ran

• PR head: 759ec763c984ae6ffacd84e5fb55dfbbf0c52075
• main: b433c40f2f46aa2e59e19806010a839ac8572d64
• Merged tree: main + head with --no-ff, no conflicts
• Machine: Arch Linux 7.2.11-zen, GCC 16.2.1, Intel Core i9-14900KF (32 hardware threads, so the prebuild pool has 16 workers and up to 8 budgeted), NVIDIA GeForce RTX 4070 Ti SUPER (driver 615.78.08), Release, Ninja, in a clean worktree

git diff --check origin/main...pr-12147
python3 tools/check_conventions.py --base origin/main
cmake --build build -j10 --target libs agc_compile_budget_tests agc_driver_compute_prebuild_share_tests \
  agc_driver_pipeline_journal_tests agc_driver_prepared_shaders_tests agc_driver_registration_prepare_shutdown_tests \
  agc_driver_registration_prepare_stack_tests agc_driver_shutdown_threads_tests agc_driver_graphics_tests
R='^(agc_compile_budget|agc_driver_compute_prebuild_share|agc_driver_pipeline_journal|agc_pipeline_journal|agc_pipeline_journal_long_paths|agc_driver_registration_prepare_shutdown|agc_driver_registration_prepare_stack|agc_driver_shutdown_threads.*|agc_driver_prepared_shaders.*)$'
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 \
  VK_LOADER_DEBUG=error,warn,layer ctest --test-dir build -R "$R" -j4 -V --output-on-failure   # merged tree, then main
ctest --test-dir build -R '^(agc_compile_budget|agc_driver_compute_prebuild_share)$' --repeat until-fail:10 --output-on-failure
for n in 1 2 3; do APS5_PREBUILD_THREADS=$n build/tests/agc_driver_compute_prebuild_share_tests; done

Results

Check	Result
Merge with main	clean
git diff --check	clean
check_conventions.py	All conventions checks passed.
Build of the targets above	OK, no compiler warnings
Focused tests, merged tree, validation layer	38 of 38 pass
Same tests on main, validation layer	36 of 36 pass (the two new tests don't exist there)
VUID- lines	0 on the merged tree, 0 on main
was not found but was requested by env var VK_INSTANCE_LAYERS	absent from both logs; the loader inserts VK_LAYER_KHRONOS_validation in both
agc_compile_budget, agc_driver_compute_prebuild_share repeated 10 times	10 of 10 pass
agc_driver_compute_prebuild_share with APS5_PREBUILD_THREADS=2 and 3	pass
agc_driver_compute_prebuild_share with APS5_PREBUILD_THREADS=1	fails (see Non-blocking 2)
Longest test	agc_driver_pipeline_journal 10.5 s; every test is under 30 s

Merged tree, ctest excerpt

986: compute prebuild share tests passed
 1/38 Test  #986: agc_driver_compute_prebuild_share ...............................   Passed    0.01 sec
707: compile budget tests passed
 7/38 Test  #707: agc_compile_budget ..............................................   Passed    0.28 sec
...
100% tests passed out of 38

The device for the journal tests is NVIDIA GeForce RTX 4070 Ti SUPER in every case. The description says agc_driver_pipeline_journal fails in OversizedRecordKeepsTheJournal on both sides because the NVIDIA driver drops out after the earlier instances. On this host it passes on both main and the merged tree.

APS5_PREBUILD_THREADS=1

compute prebuild share: a library prebuild queued behind 1 blocked compute prebuilds did not run while they waited
threads=1 rc=1
compute prebuild share tests passed
threads=2 rc=0
compute prebuild share tests passed
threads=3 rc=0

Negative control

1. All production files from main (the Execution directory and ENVIRONMENT.md; the new CompileBudget.hpp left in place): agc_driver_compute_prebuild_share_tests doesn't build (SubmitComputePrebuild is not a member of AgcDriver::DriverDetail).

2. Mutations, built and run on the merged tree:

   • In CompileBudget::Hold, a hold that fits waits only on held + bytes <= limit, dropping the waitingAlone == 0 term from f07182a387. agc_compile_budget then fails with a 40-byte hold that fits entered while a 500-byte hold was waiting to run alone. This confirms with a test the fix for the first point in the earlier review comment.
   • SubmitComputePrebuild submits with budgeted = false. agc_driver_compute_prebuild_share then fails after 10 s with a library prebuild queued behind 16 blocked compute prebuilds did not run while they waited.

   After that I restored the merged files.

Findings

Blocking

None.

Non-blocking

1. Journal replay doesn't check again after waiting for the budget. ReplayComputePipeline checks computePipelines and computePipelinesBuilding, then takes the hold, then compiles without checking again. PrebuildComputePipeline does check again after the hold. Before this change the gap between the check and the compile was short. Now it lasts as long as the budget wait, which in the scenario the PR targets is several large compiles. A dispatch that needs the same pipeline during that wait builds it itself. The replay then compiles it a second time, and try_emplace throws the result away. That duplicate compile is the expensive, memory-heavy work the budget exists to limit. #12657 fixes the underlying race by registering the replay in computePipelinesBuilding. By the order agreed there, it will take this PR's hold before it registers the build, so its check needs to run after the hold to close the window. Suggested fix, here or as part of the #12657 rebase: take the hold before the contains check, or repeat the check under computePipelinesMutex after the hold and return false on a hit, as PrebuildComputePipeline does.
2. The pool-share test depends on the host and fails with APS5_PREBUILD_THREADS=1. PrebuildThreads() copies the pool's sizing, so the test runs with 16 workers here, 12 on the author's machine, and 1 worker if a developer exports APS5_PREBUILD_THREADS=1. With one worker, budgetedLimit is max(1, 0) = 1, the whole pool. The library prebuild then can't run beside the blocked compute prebuild, and the test fails with the message above. That matches the documented "at least one", so the code isn't wrong. Suggested fix: have the CTest entry set APS5_PREBUILD_THREADS to a fixed value (for example 4), so the result doesn't depend on the host or the developer's environment. Or skip with return code 77 when the pool has fewer than two workers.
3. Tested has no validation-layer comparison. CONTRIBUTING.md asks AGC driver changes to run the focused tests on main and on the change with VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1, and to compare the VUID- lines. The description reports plain ctest runs only. The change makes no new Vulkan calls, and my run above shows 0 VUID- lines on both sides on NVIDIA. Please add your own comparison to Tested all the same, or cite this one.

Not verified

• No title run. I didn't launch Ghost of Tsushima or any other title, so the memory figures and the 8 MiB default in the description are taken as reported.
• Windows, other GPU vendors and lavapipe.
• The interaction with #12657. I read its diff but didn't build or run it on top of this branch.
• The full test suite, which I didn't run. I ran only the 38 budget, pool, shutdown, prepared-shader and journal tests listed above.

[milescoviello]

@XairRrR thanks. All three are in a937addfb8:

1. ReplayComputePipeline now checks computePipelines and computePipelinesBuilding again under the mutex after it takes the hold, and returns false on a hit, as PrebuildComputePipeline does. fix(agc): build a replayed compute pipeline once when a dispatch races it #12657's rebase can take the hold before it registers the build, as agreed.
2. The CTest entry for agc_driver_compute_prebuild_share sets APS5_PREBUILD_THREADS=4. With APS5_PREBUILD_THREADS=1 exported in the shell, it still passes.
3. Tested now has the validation-layer comparison: the merge base passes 10 of 10 and this head 12 of 12, with no VUID- lines on either side. These tests don't print the devices they open, so I've cited your run too.

The 12 focused tests pass 3 of 3.

AI-assisted: yes (Claude Code)

[XairRrR]

Re-checked — approved

a937addfb8 adds the check after the budget wait in ReplayComputePipeline and pins APS5_PREBUILD_THREADS=4 for the pool-share test, and the description now has the validation-layer comparison; all three earlier points are resolved, and the merged tree builds and passes the focused tests with no VUID- lines.

What changed since my last review

• New commit a937addfb8 (5 lines, on top of 759ec763c9, which I reviewed before):
  • VulkanDevice.cpp L2333-L2337: after ReplayComputePipeline takes the CompileBudget::Hold, it checks computePipelines and computePipelinesBuilding again under computePipelinesMutex and returns false on a hit.
  • CMakeLists.txt L3286: agc_driver_compute_prebuild_share gets APS5_PREBUILD_THREADS=4 appended to its environment.
• The author replied to the three points, and Tested now has a validation-layer run on the merge base and on the head.

What I ran

• PR head: a937addfb8b0229ec05ce1048db5f4985015a5e9
• main: 52c09d567d2fa453494dd720fcf53a98149468e4
• Merged tree: main + head with --no-ff, no conflicts
• Machine: Arch Linux 7.2.11-zen, GCC 16.2.1, Intel Core i9-14900KF, NVIDIA GeForce RTX 4070 Ti SUPER (driver 615.78.08), Release, Ninja, in a clean worktree

git diff --check origin/main...pr-12147
python3 tools/check_conventions.py --base origin/main
cmake --build build -j10 --target libs agc_compile_budget_tests agc_driver_compute_prebuild_share_tests \
  agc_driver_pipeline_journal_tests agc_driver_prepared_shaders_tests agc_driver_registration_prepare_shutdown_tests \
  agc_driver_registration_prepare_stack_tests agc_driver_shutdown_threads_tests
R='^(agc_compile_budget|agc_driver_compute_prebuild_share|agc_driver_pipeline_journal|agc_pipeline_journal|agc_pipeline_journal_long_paths|agc_driver_registration_prepare_shutdown|agc_driver_registration_prepare_stack|agc_driver_shutdown_threads.*|agc_driver_prepared_shaders.*)$'
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1 \
  VK_LOADER_DEBUG=error,warn,layer ctest --test-dir build -R "$R" -j4 -V --output-on-failure   # merged tree, then main
ctest --test-dir build --show-only=json-v1 -R '^agc_driver_compute_prebuild_share$'
APS5_PREBUILD_THREADS=1 ctest --test-dir build -R '^(agc_compile_budget|agc_driver_compute_prebuild_share)$' --repeat until-fail:10 --output-on-failure

Previous findings

1. Journal replay doesn't check again after the budget wait — Resolved. The second check at L2333-L2337 matches the one in PrebuildComputePipeline, takes the same mutex, and tests both maps, so a dispatch that built or started building the pipeline during the wait now makes the replay return false instead of compiling a duplicate. Returning false there is consistent with the existing try_emplace(...).second result for a pipeline that already exists.
2. The pool-share test depends on the host and fails with APS5_PREBUILD_THREADS=1 — Resolved. CTest lists the test's environment as ANYPS5_NO_SHADER_CACHE=1;APS5_PREBUILD_THREADS=4;APS5_NO_PIPELINE_JOURNAL=1. With APS5_PREBUILD_THREADS=1 exported, agc_compile_budget and agc_driver_compute_prebuild_share pass 10 of 10 under ctest. Running the test binary directly with APS5_PREBUILD_THREADS=1 still fails with the same message as before, which is expected now that the CTest entry carries the setting.
3. Tested has no validation-layer comparison — Resolved. The description now reports the comparison on the merge base and the head (author-reported). My own run on this head agrees, see below.

Results

Check	Result
Merge with main	clean
git diff --check	clean
check_conventions.py	All conventions checks passed.
Build of the targets above, merged tree	OK, 0 compiler warnings
Focused tests, merged tree, validation layer	38 of 38 pass
Same tests on main, validation layer	36 of 36 pass (agc_compile_budget and agc_driver_compute_prebuild_share don't exist there)
VUID- lines	0 on the merged tree, 0 on main
was not found but was requested by env var VK_INSTANCE_LAYERS	absent from both logs; both name VK_LAYER_KHRONOS_validation 384 times
Longest test	8.3 s on the merged tree, 11.4 s on main; every test is under 30 s

Merged tree, ctest excerpt

 1/38 Test  #986: agc_driver_compute_prebuild_share ...............................   Passed    0.00 sec
 7/38 Test  #707: agc_compile_budget ..............................................   Passed    0.28 sec
...
100% tests passed out of 38
Total Test time (real) =   9.61 sec

APS5_PREBUILD_THREADS=1

# under ctest, 10 repeats
100% tests passed out of 2

# test binary run directly
compute prebuild share: a library prebuild queued behind 1 blocked compute prebuilds did not run while they waited
rc=1

New findings

Blocking

None.

Non-blocking

1. No test covers the new check after the wait. I removed the four lines at L2333-L2337 on the merged tree and rebuilt: agc_driver_pipeline_journal, agc_pipeline_journal and agc_pipeline_journal_long_paths still pass 3 of 3. That is expected, since hitting the window needs a dispatch to build the same pipeline while the replay waits on the budget, and the change is small and mirrors PrebuildComputePipeline. If #12657's rebase adds a test for its computePipelinesBuilding registration, a case where the replay's hold is blocked while the same key gets built would cover this check too.

Not verified

• No title run; the Ghost of Tsushima figures in the description are taken as reported.
• Windows, other GPU vendors and lavapipe.
• The interaction with #12657 on top of this branch.
• The full test suite. I ran only the 38 budget, pool, shutdown, prepared-shader and journal tests listed above.

[TechLevelZero]

✅ NVIDIA GPU test: passed

PR head a937add merged onto main 52c09d5 · tested 2026-10-29 11:12 UTC · 28 min

GPU	NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests	✅ 2,277 passed · 5 failed (0 new, 5 also fail on main) · 39 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection	2,321 of 2,873 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation	✅ no new validation IDs vs main (895 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ agc_driver_bda_device 53.8 s, ⚠️ agc_driver_pipeline_journal 34.4 s, ⚠️ agc_driver_lazy_resolved_abi_switch_set 31.5 s

Failing on main too (5, not counted against this PR)

agc_compute_pipeline_inspection
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast

Validated but not compared (2, not in the main baseline's validation pass)

agc_compile_budget
agc_driver_compute_prebuild_share

Validation IDs already present on main (4 IDs in 435 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (435 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_array_layer_target, agc_driver_array_layer_target_cpu, agc_driver_bda_anchor_reads, and 427 more
VUID-vkCmdCopyImage-srcImage-01548 (11 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_storage_dispatch, and 3 more
VUID-vkCmdCopyImage-srcImage-01551 (11 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_storage_dispatch, and 3 more
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target

Skipped (39)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_labels: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SD

Reproduce

git fetch origin a937addfb8b0229ec05ce1048db5f4985015a5e9 52c09d567d2fa453494dd720fcf53a98149468e4
git checkout 52c09d567d2fa453494dd720fcf53a98149468e4 && git merge --no-ff a937addfb8b0229ec05ce1048db5f4985015a5e9
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}