perf(agc): link pipelines with link-time optimization on half the hardware threads by Tamalero · Pull Request #12323 · boykopovar/AnyPS5

Tamalero · 2026-10-28T09:24:11Z

What

Background link-time optimization of graphics pipelines (PipelineLibrary.cpp) ran on one worker thread. Each job is a vkCreateGraphicsPipelines with VK_PIPELINE_CREATE_LINK_TIME_OPTIMIZATION_BIT_EXT, a full compile of the linked pipeline, and the jobs ran one after another. Until a pipeline's job finishes, draws use the fast-linked pipeline, which can run slower on the GPU. A title that registers many pipelines at once keeps one host thread in the driver's compiler while the others are idle.

Fix: the library store keeps a pool of workers. Unset, APS5_PIPELINE_LTO_THREADS uses half the hardware threads, at least two, the same rule as the registration prebuild pool (#10388). A worker starts when a job is queued and the idle workers do not outnumber the jobs already waiting, so a title with few jobs still uses one thread, and a worker that was woken but has not taken its job yet does not count as free. WaitForOptimizedPipelines, ClearPipelineLibraries and StopPipelineLibraryWorker drain and join every worker. The pipeline cache was already taken through PipelineCacheSynchronization::LockForPipelineUse, a shared lock, so the workers compile in parallel.

The queue-failure tests hold the only worker to check queue growth, so they now run with APS5_PIPELINE_LTO_THREADS=1. The new agc_pipeline_library_workers test runs with 2: a job held on one worker must not stop a second job from finishing, three jobs must run exactly once each, the pool must not exceed its limit, and both workers must stop. agc_pipeline_library_workers_burst queues two jobs at once behind one idle worker and requires both to run on two workers. An APS5_PIPELINE_LTO_THREADS that is not a whole number from 1 through the hardware thread count fails at the first background link; agc_pipeline_library_workers_invalid_{negative,text,zero,suffix,above_hardware} cover it.

Tested

Linux 7.3 (CachyOS), GCC 16.2.1, Release, BUILD_TESTING=ON, i9-14900K (32 hardware threads, so 16 workers unset). Head e231257 on main 4bc9e77 for the RADV runs and the line below; the test lines after it were run on 2e6cd4b on main bb04a4f unless they say otherwise.

On e231257: all 32 agc_pipeline tests pass (ctest -R '^agc_pipeline' -j8), and tools/check_conventions.py --base 4bc9e779aa passes.
agc_pipeline_library_workers passes. Negative check (on the earlier head 07369ec): with PipelineLibrary.cpp from main, it fails with "a job waited behind a held job with a second worker allowed".
agc_pipeline_library_workers_burst passes 200 of 200. Negative check (07369ec): with the earlier spawn condition (waitingWorkers == 0) it fails 50 of 50.
agc_pipeline_library_workers_invalid_* pass. With the earlier strtoul parser all five fail (07369ec).
All 32 agc_pipeline tests pass on this head.
agc_pipeline_queue_failure_{wait,clear,stop} pass with one worker.
ctest -R '^agc' -j8 on 07369ec (main 2960e9a): 1112/1113 passed. agc_driver_depth_clear_hle failed once under the parallel run; alone it passes 4 of 4 on that head and 3 of 3 on main 2960e9a. Not rerun on 2e6cd4b.
tools/check_conventions.py --base origin/main passes.
Performance, RADV (AMD Radeon Pro V620, Mesa 25.2.8): the change runs there, but I measured no gain. High on Life (PPSA15388) queues about 400 link jobs per run, about 375 of them in one burst at a level load with a cold cache. On main one worker leaves up to 141–145 jobs queued; with this PR 12 workers keep the queue at 10–13 and finish the burst about 4 s sooner after it starts. Load freeze (12.5/12.9 s against 13.2/12.8 s), frame time and GPU time after it, and warm gameplay frame time do not differ beyond run-to-run spread. Cost: 11 more threads, 30–145 MiB more peak RSS. NVIDIA's proprietary driver does not run the background link at all (fix(agc): skip the background pipeline relink on NVIDIA's proprietary driver #3434), so the 4090 runs further down do not exercise this change. Details, table and digests: perf(agc): link pipelines with link-time optimization on half the hardware threads #12323 (comment).
High on Life (PPSA15388), main 2960e9a against 07369ec, spawn view after Continue, the same save each run, no input, APS5_TRACE_SECONDS=300:20, warm caches. The pool never starts in this title on this driver (fix(agc): skip the background pipeline relink on NVIDIA's proprietary driver #3434; 137 threads in every run on both builds), so these runs do not exercise the change. In the fast scene state: p50 51.3 / 53.1 / 51.5 / 50.5 ms on main, 51.8 / 51.8 ms with this PR. Observed only on the PR side, not yet separable from chance: of 6 PR launches, 2 crashed 3 s in before the first frame (the start-up race in High on Life: about 1 launch in 5 crashes at startup when two threads release the same Wwise asset request #10905, which also occurs on main), and 2 of the 4 completed runs were in the title's slower scene state (120.3 and 170.2 ms, the same signature as slow main runs in perf: compute dispatches miss the dispatch cache when their buffer addresses or layouts change every frame (PPSA01748 menu, PPSA15388) #12640). main: 4 launches, 0 crashes, 0 slow. Four runs per build cannot compare these rates; that needs more interleaved runs on one base. All runs, traces and digests: perf(agc): link pipelines with link-time optimization on half the hardware threads #12323 (comment).
Related, not changed here: the pipeline journal. In the same Ratchet runs, the black screen before the menu took 26–40 minutes with one thread in the NVIDIA compiler, then about 1 minute once the journal held the title's pipelines (1,403 pipelines and 1,178 modules, 966 MiB), with 1 worker or 8 requested (the setting has no effect on this driver, see above). One run reached the 1 GiB APS5_PIPELINE_JOURNAL_MAX_MIB cap (pipelines.journal reached its size cap at 966 s, 734 pipelines loaded at start). The rewrite then keeps the newest records, so a large title may lose pipelines it needs at startup. I did not show that the cap caused the long startups, so the default stays.

Summary lines, High on Life on RADV (`main` 4bc9e77 + counter patch = eea81e370cbf, this PR e231257 + counter patch = 22abb199377a): cold runs 1–4 (main, PR, PR, main), then warm runs 1–4 (main, PR, PR, main)

Summary lines, High on Life (main 2960e9a runs 1, 4, 7, 8; this PR at 07369ec runs 2, 9, then 5 and 6 in the slower scene state)

Checklist

🤖 Generated with Claude Code

Tested and written by Claude (AI assistant) on a contributor's machine.

github-actions · 2026-10-28T09:24:25Z

This pull request is labeled perf, so it needs the summary line of a capture on main and one with the change, from the same title, scene and machine, as the Performance report form asks. Please paste both aps5-summary.jsonl lines, attach the two traces and add the output of tools/trace_digest.py.

MarcZX10

I found a configuration edge case that can make the new worker pool try to create an effectively unbounded number of threads. std::strtoul("-1", nullptr, 10) returns ULONG_MAX on this Linux host, and malformed text returns 0, which the current max(1, ...) silently turns into one worker. With APS5_PIPELINE_LTO_THREADS=-1, the limit becomes 18,446,744,073,709,551,615 here; each queued job can start another worker while none is idle, until thread creation fails or exhausts resources. Please validate the whole decimal string and overflow, and cap/reject values to a documented reasonable maximum (for example, hardware concurrency).

Separately, this perf PR still lacks the required same-title, same-scene, same-machine main/change aps5-summary.jsonl lines, attached traces, and trace_digest.py output. The PR description says those will be added later; the repository performance-report rule still applies before approval.

Tamalero · 2026-10-28T09:40:42Z

@MarcZX10 thanks, you're right. Fixed in 93d15cc:

APS5_PIPELINE_LTO_THREADS is parsed with std::from_chars over the whole string, and the value must be from 1 through the hardware thread count. Anything else (-1, abc, 0, 8x, a value above the thread count, or an overflow) throws APS5_PIPELINE_LTO_THREADS must be a whole number from 1 through N, the hardware thread count at the first background link, before any worker starts. Unset still uses half the hardware threads, at least two. docs/dev/ENVIRONMENT.md states the range.
New tests agc_pipeline_library_workers_invalid_{negative,text,zero,suffix,above_hardware} run the test binary with each value and require that error and no worker. With the previous parser all five fail; with this commit they pass, along with the existing worker and queue-failure tests (9/9).

On the performance numbers: agreed, they're required. I'll add the main and PR summary lines from the same machine and scene, both traces and the trace_digest.py output, from High on Life (PPSA15388). Both builds are ready; the capture needs a short play session to stand at a fixed spot.

Tested and written by Claude (AI assistant) on a contributor's machine.

MarcZX10

Re-reviewed exact head 93d15cca50f7f3336076c6cd433791e871bd998f. The earlier worker-count blocker is resolved: std::from_chars consumes the full string, rejects parse/overflow/zero/out-of-range values before creating a worker, and the new invalid-setting cases cover the relevant inputs. The remaining blocker is the required performance report. This PR is labeled perf, and the current description still says the same-scene main/change captures will be added later; it has no two aps5-summary.jsonl lines, attached traces, or tools/trace_digest.py output for the High on Life scene. Please add that paired evidence on the exact current head and matching main before approval. I have not approved; hosted Windows is pending and the general Linux product job is skipped.

github-actions · 2026-10-28T09:47:10Z

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on `main`)	Same exports	Same new files
PR 11914	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` L13-14

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 1896	`docs/dev/ENVIRONMENT.md`
PR 4472	`docs/dev/ENVIRONMENT.md`
PR 5925	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp`
PR 6186	`docs/dev/ENVIRONMENT.md`
PR 6924	`docs/dev/ENVIRONMENT.md`
PR 7449	`docs/dev/ENVIRONMENT.md`
PR 7457	`docs/dev/ENVIRONMENT.md`
PR 7514	`docs/dev/ENVIRONMENT.md`
PR 7604	`docs/dev/ENVIRONMENT.md`
PR 7622	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp`
PR 7642	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` `docs/dev/ENVIRONMENT.md`
PR 8004	`docs/dev/ENVIRONMENT.md`
PR 8246	`docs/dev/ENVIRONMENT.md`
PR 8484	`docs/dev/ENVIRONMENT.md`
PR 9012	`docs/dev/ENVIRONMENT.md`
PR 9289	`docs/dev/ENVIRONMENT.md`
PR 9345	`docs/dev/ENVIRONMENT.md`
PR 9356	`docs/dev/ENVIRONMENT.md`
PR 9387	`docs/dev/ENVIRONMENT.md`
PR 9508	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` `docs/dev/ENVIRONMENT.md`
PR 9676	`docs/dev/ENVIRONMENT.md`
PR 9712	`docs/dev/ENVIRONMENT.md`
PR 9852	`docs/dev/ENVIRONMENT.md`
PR 9875	`docs/dev/ENVIRONMENT.md`
PR 10193	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` `docs/dev/ENVIRONMENT.md`
PR 10442	`docs/dev/ENVIRONMENT.md`
PR 10678	`docs/dev/ENVIRONMENT.md`
PR 10757	`docs/dev/ENVIRONMENT.md`
PR 10894	`docs/dev/ENVIRONMENT.md`
PR 11436	`docs/dev/ENVIRONMENT.md`
PR 12324	`docs/dev/ENVIRONMENT.md`
PR 12746	`docs/dev/ENVIRONMENT.md`
PR 12849	`docs/dev/ENVIRONMENT.md`
PR 13486	`docs/dev/ENVIRONMENT.md`
PR 13793	`docs/dev/ENVIRONMENT.md`
PR 13809	`docs/dev/ENVIRONMENT.md`
PR 13920	`docs/dev/ENVIRONMENT.md`
PR 13928	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` `docs/dev/ENVIRONMENT.md`
PR 13929	`core/libs/prx/libSceAgcDriver/Graphics/src/PipelineLibrary.cpp` `docs/dev/ENVIRONMENT.md`
PR 14051	`docs/dev/ENVIRONMENT.md`
PR 14262	`docs/dev/ENVIRONMENT.md`
PR 14299	`docs/dev/ENVIRONMENT.md`
PR 14943	`docs/dev/ENVIRONMENT.md`
PR 15059	`docs/dev/ENVIRONMENT.md`
PR 15136	`docs/dev/ENVIRONMENT.md`
PR 15169	`docs/dev/ENVIRONMENT.md`
PR 15529	`docs/dev/ENVIRONMENT.md`
PR 15548	`docs/dev/ENVIRONMENT.md`
PR 15570	`docs/dev/ENVIRONMENT.md`

oneandonlydean · 2026-10-28T09:58:37Z

@Tamalero thanks, the pool and the shutdown path look right to me. Two small things, neither blocking:

In enqueue, waitingWorkers == 0 treats a worker that was notified but has not woken yet as idle. If several jobs arrive in a burst while one worker is idle, the burst queues behind that one worker even though the pool is below its limit. Spawning when store.waitingWorkers <= store.jobs.size() (checked before the push) would keep adding workers in that case.
ENVIRONMENT.md: could you say that values below 1 mean 1? APS5_PIPELINE_JOURNAL_THREADS uses 0 to disable, so 0 here could be read the same way.

For the gate we'd like the frame-time and shader-compile stutter comparison you mentioned (main vs this PR, same scene), ideally including one run with 8 or fewer hardware threads and the peak memory during a registration burst.

adriadam10

Read the pool, the shutdown path and the new tests; they look right, and agc_pipeline_library_workers needs the second worker to pass. What is left:

The PR is labeled perf and the body still has no same-scene captures. Please add the two aps5-summary.jsonl lines (main and this PR, same title, scene and machine), attach both traces and paste the tools/trace_digest.py output (AGENTS.md; the labels check asks for the same). The Ratchet & Clank numbers are one run each on an older main, so they do not replace it.
Dean's two points, small: in enqueue (PipelineLibrary.cpp:109), waitingWorkers == 0 counts a notified-but-not-yet-woken worker as idle, so a burst queues behind it; spawn while store.waitingWorkers <= store.jobs.size() (checked before the push). And say in docs/dev/ENVIRONMENT.md:372 that 0 is rejected (unlike APS5_PIPELINE_JOURNAL_THREADS, where 0 disables).

Happy to look again once the report is attached.

Tamalero · 2026-10-28T22:13:52Z

@oneandonlydean @adriadam10 thanks, both points are in e7af1f2.

Burst behind a woken worker: enqueue now starts a worker while store.waitingWorkers <= store.jobs.size(), checked before the push, so a worker that was notified but has not taken its job yet no longer counts as free. A new test, agc_pipeline_library_workers_burst (2 workers), reproduces the case: one job finishes, so one worker is idle, then two jobs are queued at once and the first one is held. The second must finish while the first is held. With the old condition it fails in 50 of 50 runs ("a burst queued behind the idle worker it woke with a second worker allowed"). With this commit it passes 200 of 200, and all 26 agc_pipeline tests pass.
ENVIRONMENT.md: 0 is rejected like any other value outside 1 through the hardware thread count. The row now says so and notes that 0 does not disable the link, unlike APS5_PIPELINE_JOURNAL_THREADS. Dean, values below 1 are rejected rather than treated as 1, following MarcZX10's earlier review.

The performance report for this head is in the next comment.

Tested and written by Claude (AI assistant) on a contributor's machine.

Tamalero · 2026-10-28T22:13:59Z

Performance report for head e7af1f2 against main 316615d (its base), same machine, title and scene.

Setup: High on Life (PPSA15388, 01.120.000). Each run is Continue from the same save, followed by no input until the window closes: keyboard and mouse ignored (APS5_NO_PAD_INPUT=1), and the two menu presses injected through APS5_PAD_INPUT_FILE, so every run looks at the spawn view in Blim City with the same camera. APS5_TRACE_SECONDS=300:20, APS5_TRACE_EVENTS=2097152 (the default 262,144 filled after about 6 s at this scene), shader cache, pipeline journal and NVIDIA cache warm. Order main, PR, PR, main. i9-14900K (32 threads, so 16 LTO workers unset), RTX 4090 with NVIDIA 615.78.08, Linux 7.3 (CachyOS), stock GPU clocks.

Run	Presents in 20 s	p50	p90	p99	max	GPU busy / present
`main` 1	123	165.3	180.3	214.3	217.4 ms	44.1 ms
PR 1	103	179.5	253.0	285.8	299.5 ms	52.2 ms
PR 2	108	173.4	239.0	305.6	311.5 ms	45.8 ms
`main` 2	111	175.3	235.5	295.3	300.6 ms	47.7 ms

Result: no measurable difference in this title. The two main runs differ from each other by 10 ms at p50, as much as either PR run differs from them.

Why nothing changes here: High on Life never has two background LTO jobs queued at once, so the pool never starts a second worker. The process thread count, sampled every second from /proc, follows the same sequence on both builds in every run: 131, then 135 when the level loads, then 136 and 137 when the trace window opens and closes (the trace's own threads). A second LTO worker would add a thread on the PR build. Both queue workers are about 99 % busy in this scene, mostly in Driver.Packet.memory_wait, which this PR does not touch.

Other points from the review:

Peak memory: VmHWM by the end of the trace window was 14,048 / 14,067 / 14,223 / 14,163 MiB (main, PR, PR, main), so no difference during startup registration either.
Shader-compile stutter: in the same scene the first camera pan over a new area stalls for 2–3 s on both builds. Resident memory grows by 1.3 GiB in 3 s and the thread count does not change, so this is asset streaming, not pipeline linking. A repeat of the same pan does not stall.
8 or fewer hardware threads: not run, because the pool never grows past one worker in this title, so a smaller limit has nothing to cap. I can run it if you still want the number.
Where it should matter: a title that registers many pipelines at once. The Ratchet & Clank: Rift Apart (PPSA01474) numbers in the PR body show that case, but they are not a clean comparison (one run each, an older main with local patches, and the title does not run on main yet).

Also seen, not related to this PR: the spawn view loads in one of two states, about 19.5 FPS or about 6 FPS, on both builds. Four earlier runs in the fast state (default event buffer, so their traces dropped events) gave the same picture: p50 50.3 / 55.8 / 51.3 / 50.3 ms (main, PR, PR, main), 137 threads on both builds. I'll report the slow state separately.

The four traces are attached: aps5-trace-316615d042c0-main-1.json.gz, aps5-trace-e7af1f2af21e-pr-1.json.gz, aps5-trace-e7af1f2af21e-pr-2.json.gz, aps5-trace-316615d042c0-main-2.json.gz.

trace_digest.py: `main` 316615d, run 1 (aps5-trace-316615d042c0-main-1.json.gz)

seconds 300:20 from present 4471: 123 presents, frame p50 165.3 / p90 180.3 / p99 214.3 / max 217.4 ms, GPU busy 44.1 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit 316615d, dropped: no

{"commit":"316615d042c0","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":4471,"ms":20041.345},"presents":123,"frame_ms":{"p50":165.255,"p90":180.32,"p99":214.335,"max":217.424},"gpu_busy_ms_per_present":44.064,"worker_busy_pct":{"queue worker 0x0":98.992,"queue worker 0x20":98.707},"top_phases_ms_per_frame":{"Driver.Packet":321.507,"Driver.Packet.memory_wait":153.798,"Graphics.Draw":83.29,"Graphics.Draw.draw_and_resource_release":77.918,"Driver.Packet.dispatch":48.864,"Driver.Packet.draw":39.678,"Driver.Draw":39.49,"Driver.Dispatch":37.035},"trace_dropped":false,"trace":"aps5-trace-316615d042c0-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 4590: 217.4 ms at 19306 ms: Driver.Packet (queue worker 0x20) 215.0, Driver.Packet.memory_wait (queue worker 0x20) 210.1, Driver.Packet (queue worker 0x0) 194.7, Recorder.Submit (draw committer) 106.6
present 4536: 214.3 ms at 10631 ms: Driver.Packet (queue worker 0x20) 211.7, Driver.Packet (queue worker 0x0) 192.3, Driver.Packet.memory_wait (queue worker 0x20) 152.3, Driver.Packet.dispatch (queue worker 0x0) 113.7
present 4553: 213.7 ms at 13375 ms: Driver.Packet (queue worker 0x20) 211.2, Driver.Packet.memory_wait (queue worker 0x20) 205.9, Driver.Packet (queue worker 0x0) 190.8, Driver.Packet.draw (queue worker 0x0) 72.2
present 4511: 212.8 ms at 6448 ms: Driver.Packet (queue worker 0x20) 209.8, Driver.Packet.memory_wait (queue worker 0x20) 204.9, Driver.Packet (queue worker 0x0) 187.6, Graphics.Draw (draw committer) 88.4
present 4533: 205.3 ms at 10118 ms: Driver.Packet (queue worker 0x20) 202.6, Driver.Packet.memory_wait (queue worker 0x20) 197.6, Driver.Packet (queue worker 0x0) 182.4, Driver.Packet.draw (queue worker 0x0) 75.8

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	134.7 (83 %)	5.3 (3 %)	23.0	104.9	Driver.Packet 139.7, Driver.EndSubmission 0.2
draw committer	88.5 (54 %)	0.6 (0 %)	73.7	26.4	Graphics.Draw 72.4, Recorder.Submit 16.1
queue worker 0x20	9.2 (6 %)	153.3 (94 %)	0.4	1.7	Driver.Packet 160.2, Driver.EndSubmission 0.3
presenter	1.1 (1 %)	14.9 (9 %)	146.9	0.1	VideoOut.Flip 16.0, Driver.Present 0.0
gpu	44.0 (27 %)	0.0 (0 %)	118.9	0.0	batch 44.0

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 299.9, Driver.Packet.memory_wait 153.8, Graphics.Draw 72.4, Graphics.Draw.draw_and_resource_release 68.7, Driver.Packet.dispatch 47.1, batch 44.0, Driver.Dispatch 35.0, Driver.Packet.draw 28.9, Driver.Draw 28.7, Resources.Complete 26.0

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 133.2 ms at 19385 ms, present 4590
Driver.Packet.memory_wait (queue worker 0x20) 121.8 ms at 13461 ms, present 4553
Driver.Packet.memory_wait (queue worker 0x20) 119.2 ms at 10198 ms, present 4533
Driver.Packet.memory_wait (queue worker 0x20) 112.8 ms at 6543 ms, present 4511
Driver.Packet.memory_wait (queue worker 0x20) 111.1 ms at 742 ms, present 4475
Driver.Packet.memory_wait (queue worker 0x20) 105.1 ms at 4060 ms, present 4496
Driver.Packet.memory_wait (queue worker 0x20) 99.9 ms at 10377 ms, present 4534
Driver.Packet.memory_wait (queue worker 0x20) 99.1 ms at 19937 ms, present 4593
Driver.Packet.memory_wait (queue worker 0x20) 98.4 ms at 7918 ms, present 4519
Driver.Packet.memory_wait (queue worker 0x20) 97.9 ms at 11369 ms, present 4540

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 4668: 13.5 ms at 10770 ms, present 4536: GPU no ranges; recorded by Graphics.Draw (draw committer) 44.0, Graphics.Draw.draw_and_resource_release (draw committer) 43.9, Resources.Prepare (draw committer) 2.7, Resources.Complete (draw committer) 1.4
batch 3608: 12.6 ms at 8307 ms, present 4521: GPU no ranges; recorded by Graphics.Draw (draw committer) 33.1, Graphics.Draw.draw_and_resource_release (draw committer) 33.0, Resources.Prepare (draw committer) 2.8, Resources.Complete (draw committer) 1.4
batch 2538: 11.0 ms at 5767 ms, present 4506: GPU no ranges; recorded by Graphics.Draw (draw committer) 50.2, Graphics.Draw.draw_and_resource_release (draw committer) 50.2, Resources.Prepare (draw committer) 2.7, Resources.Complete (draw committer) 1.4
batch 413: 10.6 ms at 942 ms, present 4476: GPU no ranges; recorded by Graphics.Draw (draw committer) 45.1, Graphics.Draw.draw_and_resource_release (draw committer) 45.0, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4
batch 2397: 9.8 ms at 5427 ms, present 4504: GPU no ranges; recorded by Graphics.Draw (draw committer) 48.2, Graphics.Draw.draw_and_resource_release (draw committer) 48.2, Resources.Prepare (draw committer) 2.9, Resources.Complete (draw committer) 1.4

trace_digest.py: this PR e7af1f2, run 1 (aps5-trace-e7af1f2af21e-pr-1.json.gz)

seconds 300:20 from present 5535: 103 presents, frame p50 179.5 / p90 253.0 / p99 285.8 / max 299.5 ms, GPU busy 52.2 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit e7af1f2, dropped: no

{"commit":"e7af1f2af21e","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":5535,"ms":20088.451},"presents":103,"frame_ms":{"p50":179.532,"p90":253.049,"p99":285.779,"max":299.506},"gpu_busy_ms_per_present":52.224,"worker_busy_pct":{"queue worker 0x0":99.123,"queue worker 0x20":98.627},"top_phases_ms_per_frame":{"Driver.Packet":381.044,"Driver.Packet.memory_wait":174.404,"Graphics.Draw":87.734,"Graphics.Draw.draw_and_resource_release":81.838,"Driver.Packet.dispatch":63.687,"Driver.Dispatch":47.068,"Driver.Packet.draw":46.686,"Driver.Draw":46.481},"trace_dropped":false,"trace":"aps5-trace-e7af1f2af21e-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 5586: 299.5 ms at 10556 ms: Driver.Packet (queue worker 0x20) 296.1, Driver.Packet (queue worker 0x0) 275.7, Driver.Packet.memory_wait (queue worker 0x20) 207.2, Graphics.Draw (draw committer) 90.8
present 5622: 285.8 ms at 17107 ms: Driver.Packet (queue worker 0x20) 282.6, Driver.Packet.memory_wait (queue worker 0x20) 273.4, Driver.Packet (queue worker 0x0) 260.5, Driver.Packet.dispatch (queue worker 0x0) 124.3
present 5595: 269.6 ms at 12301 ms: Driver.Packet (queue worker 0x0) 243.4, Driver.Packet (queue worker 0x20) 221.5, Driver.Packet.memory_wait (queue worker 0x20) 190.1, Graphics.Draw (draw committer) 88.3
present 5569: 266.6 ms at 6962 ms: Driver.Packet (queue worker 0x20) 262.7, Driver.Packet (queue worker 0x0) 241.4, Driver.Packet.memory_wait (queue worker 0x20) 238.7, Graphics.Draw (draw committer) 99.6
present 5584: 258.6 ms at 10064 ms: Driver.Packet (queue worker 0x20) 255.9, Driver.Packet (queue worker 0x0) 233.1, Driver.Packet.memory_wait (queue worker 0x20) 190.2, Graphics.Draw (draw committer) 85.5

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	160.2 (82 %)	9.7 (5 %)	25.2	115.3	Driver.Packet 169.5, Driver.EndSubmission 0.3
draw committer	96.4 (49 %)	6.8 (3 %)	91.8	29.0	Graphics.Draw 75.7, Recorder.Submit 20.5
queue worker 0x20	21.4 (11 %)	173.2 (89 %)	0.4	1.6	Driver.Packet 187.9, Driver.EndSubmission 4.2
presenter	0.7 (0 %)	22.2 (11 %)	172.1	0.1	VideoOut.Flip 22.8, Driver.Present 0.1
gpu	52.1 (27 %)	0.0 (0 %)	142.9	0.0	batch 52.1

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 357.4, Driver.Packet.memory_wait 174.4, Graphics.Draw 75.7, Graphics.Draw.draw_and_resource_release 71.7, Driver.Packet.dispatch 61.6, batch 52.1, Driver.Dispatch 44.7, Recorder.Submit 41.8, Resources.Complete 38.9, Vulkan.Dispatch 38.6

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 142.3 ms at 10706 ms, present 5586
Driver.Packet.memory_wait (queue worker 0x20) 138.4 ms at 3899 ms, present 5554
Driver.Packet.memory_wait (queue worker 0x20) 124.4 ms at 11867 ms, present 5592
Driver.Packet.memory_wait (queue worker 0x20) 123.7 ms at 7097 ms, present 5569
Driver.Packet.memory_wait (queue worker 0x20) 122.6 ms at 3230 ms, present 5551
Driver.Packet.memory_wait (queue worker 0x20) 122.3 ms at 11206 ms, present 5589
Driver.Packet.memory_wait (queue worker 0x20) 121.8 ms at 12049 ms, present 5593
Driver.Packet.memory_wait (queue worker 0x20) 120.1 ms at 13238 ms, present 5599
Driver.Packet.memory_wait (queue worker 0x20) 107.5 ms at 11675 ms, present 5591
Driver.Packet.memory_wait (queue worker 0x20) 107.1 ms at 9385 ms, present 5580

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 3820: 15.5 ms at 10284 ms, present 5584: GPU no ranges; recorded by Graphics.Draw (draw committer) 57.7, Graphics.Draw.draw_and_resource_release (draw committer) 57.7, Resources.Prepare (draw committer) 2.5, Resources.Complete (draw committer) 1.4
batch 1980: 12.2 ms at 5279 ms, present 5560: GPU no ranges; recorded by Graphics.Draw (draw committer) 42.7, Graphics.Draw.draw_and_resource_release (draw committer) 42.7, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4
batch 3216: 11.9 ms at 8727 ms, present 5576: GPU no ranges; recorded by Graphics.Draw (draw committer) 60.8, Graphics.Draw.draw_and_resource_release (draw committer) 60.7, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.3
batch 3062: 11.5 ms at 8304 ms, present 5574: GPU no ranges; recorded by Graphics.Draw (draw committer) 15.0, Graphics.Draw.draw_and_resource_release (draw committer) 15.0, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4
batch 141: 11.5 ms at 402 ms, present 5536: GPU no ranges; recorded by Graphics.Draw (draw committer) 55.3, Graphics.Draw.draw_and_resource_release (draw committer) 55.3, Resources.Prepare (draw committer) 2.4, Resources.Complete (draw committer) 1.4

trace_digest.py: this PR e7af1f2, run 2 (aps5-trace-e7af1f2af21e-pr-2.json.gz)

seconds 300:20 from present 4139: 108 presents, frame p50 173.4 / p90 239.0 / p99 305.6 / max 311.5 ms, GPU busy 45.8 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit e7af1f2, dropped: no

{"commit":"e7af1f2af21e","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":4139,"ms":20071.876},"presents":108,"frame_ms":{"p50":173.375,"p90":239.019,"p99":305.604,"max":311.533},"gpu_busy_ms_per_present":45.77,"worker_busy_pct":{"queue worker 0x0":99.109,"queue worker 0x20":98.82},"top_phases_ms_per_frame":{"Driver.Packet":367.197,"Driver.Packet.memory_wait":180.7,"Graphics.Draw":80.364,"Graphics.Draw.draw_and_resource_release":74.772,"Driver.Packet.dispatch":59.456,"Driver.Packet.draw":52.114,"Driver.Dispatch":52.102,"Driver.Draw":51.912},"trace_dropped":false,"trace":"aps5-trace-e7af1f2af21e-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 4220: 311.5 ms at 14582 ms: Driver.Packet (queue worker 0x20) 308.9, Driver.Packet.memory_wait (queue worker 0x20) 303.3, Driver.Packet (queue worker 0x0) 285.6, Recorder.Submit (draw committer) 112.8
present 4139: 305.6 ms at 0 ms: Driver.Packet (queue worker 0x20) 302.5, Driver.Packet.memory_wait (queue worker 0x20) 295.2, Driver.Packet (queue worker 0x0) 281.3, Recorder.Submit (draw committer) 112.0
present 4212: 261.0 ms at 13156 ms: Driver.Packet (queue worker 0x20) 258.6, Driver.Packet.memory_wait (queue worker 0x20) 251.0, Driver.Packet (queue worker 0x0) 235.9, Graphics.Draw (draw committer) 89.0
present 4195: 258.3 ms at 10083 ms: Driver.Packet (queue worker 0x20) 255.9, Driver.Packet.memory_wait (queue worker 0x20) 247.9, Driver.Packet (queue worker 0x0) 233.3, Driver.Packet.draw (queue worker 0x0) 92.5
present 4222: 255.9 ms at 15091 ms: Driver.Packet (queue worker 0x20) 252.8, Driver.Packet.memory_wait (queue worker 0x20) 238.5, Driver.Packet (queue worker 0x0) 231.3, Driver.Packet.draw (queue worker 0x0) 90.7

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	156.8 (84 %)	4.5 (2 %)	24.5	113.4	Driver.Packet 161.0, Driver.EndSubmission 0.2
draw committer	92.4 (50 %)	0.2 (0 %)	93.3	28.1	Graphics.Draw 68.7, Recorder.Submit 16.9
queue worker 0x20	6.2 (3 %)	179.2 (96 %)	0.4	1.7	Driver.Packet 183.1, Driver.EndSubmission 0.3
presenter	0.7 (0 %)	20.5 (11 %)	164.6	0.1	VideoOut.Flip 20.7, Driver.Present 0.5
gpu	45.7 (25 %)	0.0 (0 %)	140.1	0.0	batch 45.7

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 344.1, Driver.Packet.memory_wait 180.7, Graphics.Draw 68.7, Graphics.Draw.draw_and_resource_release 64.8, Driver.Packet.dispatch 57.4, Driver.Dispatch 49.7, Resources.Complete 45.9, batch 45.7, Vulkan.Dispatch 45.6, Vulkan.Dispatch.shader_resources 45.1

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 164.7 ms at 10869 ms, present 4199
Driver.Packet.memory_wait (queue worker 0x20) 157.9 ms at 2653 ms, present 4153
Driver.Packet.memory_wait (queue worker 0x20) 154.9 ms at 19645 ms, present 4245
Driver.Packet.memory_wait (queue worker 0x20) 150.7 ms at 10083 ms, present 4195
Driver.Packet.memory_wait (queue worker 0x20) 148.6 ms at 14582 ms, present 4220
Driver.Packet.memory_wait (queue worker 0x20) 147.4 ms at 17835 ms, present 4236
Driver.Packet.memory_wait (queue worker 0x20) 143.6 ms at 7097 ms, present 4178
Driver.Packet.memory_wait (queue worker 0x20) 142.6 ms at 16554 ms, present 4229
Driver.Packet.memory_wait (queue worker 0x20) 142.3 ms at 14746 ms, present 4220
Driver.Packet.memory_wait (queue worker 0x20) 142.0 ms at 156 ms, present 4139

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 6239: 10.7 ms at 15322 ms, present 4222: GPU no ranges; recorded by Graphics.Draw (draw committer) 59.6, Graphics.Draw.draw_and_resource_release (draw committer) 59.5, Resources.Prepare (draw committer) 2.7, Resources.Complete (draw committer) 1.5
batch 4221: 10.1 ms at 10320 ms, present 4195: GPU no ranges; recorded by Graphics.Draw (draw committer) 55.3, Graphics.Draw.draw_and_resource_release (draw committer) 55.2, Resources.Prepare (draw committer) 2.8, Resources.Complete (draw committer) 1.3
batch 6833: 10.1 ms at 16873 ms, present 4230: GPU no ranges; recorded by Graphics.Draw (draw committer) 57.6, Graphics.Draw.draw_and_resource_release (draw committer) 57.6, Resources.Prepare (draw committer) 2.4, Resources.Complete (draw committer) 1.3
batch 5261: 9.9 ms at 12794 ms, present 4209: GPU no ranges; recorded by Graphics.Draw (draw committer) 43.2, Graphics.Draw.draw_and_resource_release (draw committer) 43.2, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4
batch 6608: 9.8 ms at 16276 ms, present 4227: GPU no ranges; recorded by Graphics.Draw (draw committer) 39.1, Graphics.Draw.draw_and_resource_release (draw committer) 39.0, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4

trace_digest.py: `main` 316615d, run 2 (aps5-trace-316615d042c0-main-2.json.gz)

seconds 300:20 from present 6970: 111 presents, frame p50 175.3 / p90 235.5 / p99 295.3 / max 300.6 ms, GPU busy 47.7 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit 316615d, dropped: no

{"commit":"316615d042c0","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":6970,"ms":20099.169},"presents":111,"frame_ms":{"p50":175.314,"p90":235.543,"p99":295.301,"max":300.607},"gpu_busy_ms_per_present":47.712,"worker_busy_pct":{"queue worker 0x0":99.034,"queue worker 0x20":98.434},"top_phases_ms_per_frame":{"Driver.Packet":356.843,"Driver.Packet.memory_wait":169.089,"Graphics.Draw":86.573,"Graphics.Draw.draw_and_resource_release":80.167,"Driver.Packet.dispatch":72.197,"Driver.Dispatch":57.105,"Resources.Complete":52.17,"Vulkan.Dispatch":50.516},"trace_dropped":false,"trace":"aps5-trace-316615d042c0-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 7066: 300.6 ms at 17056 ms: Driver.Packet (queue worker 0x0) 275.1, Driver.Packet (queue worker 0x20) 220.8, Driver.Packet.memory_wait (queue worker 0x20) 214.5, Driver.Packet.draw (queue worker 0x0) 85.9
present 7051: 295.3 ms at 14073 ms: Driver.Packet (queue worker 0x20) 292.2, Driver.Packet.memory_wait (queue worker 0x20) 278.4, Driver.Packet (queue worker 0x0) 270.6, Graphics.Draw (draw committer) 159.4
present 7057: 280.9 ms at 15322 ms: Driver.Packet (queue worker 0x20) 278.1, Driver.Packet.memory_wait (queue worker 0x20) 273.7, Driver.Packet (queue worker 0x0) 255.3, Recorder.Submit (draw committer) 143.2
present 7079: 265.9 ms at 19650 ms: Driver.Packet (queue worker 0x20) 259.1, Driver.Packet (queue worker 0x0) 238.9, Driver.Packet.memory_wait (queue worker 0x20) 178.5, Graphics.Draw (draw committer) 161.4
present 7056: 253.1 ms at 15069 ms: Driver.Packet (queue worker 0x20) 250.3, Driver.Packet.memory_wait (queue worker 0x20) 240.7, Driver.Packet (queue worker 0x0) 226.2, Graphics.Draw (draw committer) 148.3

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	151.1 (83 %)	5.0 (3 %)	24.9	115.0	Driver.Packet 155.8, Driver.EndSubmission 0.3
draw committer	91.6 (51 %)	0.3 (0 %)	89.2	28.6	Graphics.Draw 74.9, Recorder.Submit 16.6
queue worker 0x20	11.8 (7 %)	168.9 (93 %)	0.4	1.7	Driver.Packet 177.6, Driver.EndSubmission 0.3
presenter	0.7 (0 %)	23.6 (13 %)	156.7	0.1	VideoOut.Flip 24.3, Driver.Present 0.0
title	0.0 (0 %)	0.0 (0 %)	181.1	0.0	Shader.ResolveGraphicsAbi 0.0
title	0.0 (0 %)	0.0 (0 %)	181.1	0.0	Shader.ResolveGraphicsAbi 0.0, Shader.ResolveGraphicsStagesAbi 0.0
title	0.0 (0 %)	0.0 (0 %)	181.1	0.0	Shader.ResolveGraphicsStagesAbi 0.0
title	0.0 (0 %)	0.0 (0 %)	181.1	0.0	Shader.ResolveGraphicsStagesAbi 0.0
title	0.0 (0 %)	0.0 (0 %)	181.1	0.0	Shader.ResolveGraphicsAbi 0.0
gpu	47.8 (26 %)	0.0 (0 %)	133.2	0.0	batch 47.8

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 333.4, Driver.Packet.memory_wait 169.1, Graphics.Draw 74.9, Graphics.Draw.draw_and_resource_release 70.2, Driver.Packet.dispatch 70.2, Driver.Dispatch 54.8, Resources.Complete 51.7, Vulkan.Dispatch 48.9, Vulkan.Dispatch.shader_resources 48.4, batch 47.8

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 199.6 ms at 14162 ms, present 7051
Driver.Packet.memory_wait (queue worker 0x20) 181.5 ms at 15416 ms, present 7057
Driver.Packet.memory_wait (queue worker 0x20) 164.0 ms at 16887 ms, present 7065
Driver.Packet.memory_wait (queue worker 0x20) 155.4 ms at 5296 ms, present 6999
Driver.Packet.memory_wait (queue worker 0x20) 155.2 ms at 11252 ms, present 7034
Driver.Packet.memory_wait (queue worker 0x20) 152.9 ms at 15163 ms, present 7056
Driver.Packet.memory_wait (queue worker 0x20) 150.6 ms at 18040 ms, present 7070
Driver.Packet.memory_wait (queue worker 0x20) 145.1 ms at 12020 ms, present 7038
Driver.Packet.memory_wait (queue worker 0x20) 144.1 ms at 19391 ms, present 7077
Driver.Packet.memory_wait (queue worker 0x20) 141.1 ms at 4214 ms, present 6993

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 4307: 11.4 ms at 10091 ms, present 7027: GPU no ranges; recorded by Graphics.Draw (draw committer) 55.2, Graphics.Draw.draw_and_resource_release (draw committer) 55.2, Resources.Prepare (draw committer) 2.5, Resources.Complete (draw committer) 1.4
batch 6796: 11.3 ms at 16092 ms, present 7060: GPU no ranges; recorded by Graphics.Draw (draw committer) 19.6, Graphics.Draw.draw_and_resource_release (draw committer) 19.6, Resources.Prepare (draw committer) 2.5, Resources.Complete (draw committer) 1.4
batch 2079: 11.2 ms at 4993 ms, present 6997: GPU no ranges; recorded by Graphics.Draw (draw committer) 54.2, Graphics.Draw.draw_and_resource_release (draw committer) 54.1, Resources.Prepare (draw committer) 2.6, Resources.Complete (draw committer) 1.4
batch 2971: 11.1 ms at 7077 ms, present 7009: GPU no ranges; recorded by Graphics.Draw (draw committer) 39.3, Graphics.Draw.draw_and_resource_release (draw committer) 39.3, Resources.Prepare (draw committer) 3.2, Resources.Complete (draw committer) 1.8
batch 1930: 10.2 ms at 4616 ms, present 6995: GPU no ranges; recorded by Graphics.Draw (draw committer) 44.4, Graphics.Draw.draw_and_resource_release (draw committer) 44.4, Resources.Prepare (draw committer) 2.5, Resources.Complete (draw committer) 1.4

Tested and written by Claude (AI assistant) on a contributor's machine.

Tamalero · 2026-10-28T22:17:43Z

Trace data:

aps5-trace-316615d042c0-main-2.json.gz
aps5-trace-e7af1f2af21e-pr-2.json.gz
aps5-trace-e7af1f2af21e-pr-1.json.gz
aps5-trace-316615d042c0-main-1.json.gz

Tamalero · 2026-10-29T01:01:49Z

A second title with the same result: LEGO Batman: Legacy of the Dark Knight (PPSA16833, 01.010.000) never starts a second LTO worker either.

Builds: main e13027a, and e13027a with this PR's three commits on top. They differ only in libSceAgcDriver and libSceVideoOut. On this PR's base 316615d the title stops before its menu: first on GTA V (PPSA04264) stops when the save loads: compute shader reads or writes a converted float image with 16-bit data #11425 (fix fix(shader): load and store 16-bit data of converted packed images #11580, open), then on sceHttp2GetMemoryPoolStats (fixed on main by fix(libSceHttp2): report a live context's pool size from sceHttp2GetMemoryPoolStats #11616, merged after 316615d), then on sceHttp2SetResolveRetry (fix(libSceHttp2): retry a failed lookup the number of times sceHttp2SetResolveRetry sets #11617, open). So I used e13027a, where it plays without errors.
Route, same on both: main menu, Continue, the snow map loading, then a Fidelity/Performance switch (main: Fidelity→Performance; PR: Performance→Fidelity→Performance).
Result: the process thread count, sampled every second, stays at 108–109 on both builds from the main menu through the map load and every mode switch. A second worker would add a thread on the PR build.

So, as in High on Life, the background LTO queue never holds two jobs at once in this title, and the PR changes nothing here. One run each, RTX 4090, NVIDIA 615.78.08, Linux 7.3.

Tested and written by Claude (AI assistant) on a contributor's machine.

MarcZX10

Re-checked — changes requested

The new worker and burst regressions are useful, but the branch and performance measurements still use older main snapshots. The PR is 740 commits behind current main 52c09d567d2fa453494dd720fcf53a98149468e4; the author’s pipeline tests are reported on main 316615d04, and the separate title data uses e13027a1f. The repository checklist requires a current-main base, and this perf change needs its same-scene baseline/change evidence on the refreshed head.

What I ran

Reviewed exact head e7af1f2af21e73e997eea9f5f3f6a8292661eed1 against current main.
Ran git merge-tree --write-tree and git diff --check.
Read the worker/burst tests, negative controls, and reported title data.

Results

Check	Result
Current-main merge preview	Clean; current `main` is not in the branch history
`git diff --check`	Passed
Focused AGC tests	Author reports the pipeline tests pass, including 200/200 burst runs, on older bases
Performance evidence	The report is tied to older main snapshots; title results say no second LTO worker starts on that route
Current-main run	Not reported

Please refresh onto current main, rerun the focused worker tests, and update the required same-scene performance comparison on that head. I did not run the GPU tests locally.

Tamalero · 2026-10-29T07:29:39Z

@MarcZX10 refreshed: the branch is rebased onto current main 2960e9a (no conflicts), head 07369ec, and everything below is on that head and that main.

Tests on 07369ec: all 32 agc_pipeline tests pass, agc_pipeline_library_workers_burst passes 200 of 200 repeats, and tools/check_conventions.py --base origin/main passes. ctest -R '^agc' -j8: 1112/1113; agc_driver_depth_clear_hle failed once under the parallel run, and alone it passes 4 of 4 on this head and 3 of 3 on main 2960e9a.

Same-scene comparison: High on Life (PPSA15388), spawn view after Continue, the same save restored before every run, no input (APS5_NO_PAD_INPUT=1, the two menu presses injected through APS5_PAD_INPUT_FILE), APS5_TRACE_SECONDS=300:20, APS5_TRACE_EVENTS=2097152, warm caches after one discarded warm-up run. Every run that started is listed:

Run	Build	p50	p90	GPU busy / present	Threads
1	`main` `2960e9a`	51.3	56.5 ms	28.2 ms	137
2	PR `07369ec`	51.8	66.4 ms	28.5 ms	137
3	PR	crashed 3 s after launch (jump to address 0), before the first frame
4	`main`	53.1	57.5 ms	29.1 ms	137
5	PR	120.3	155.2 ms	33.9 ms	137
6	PR	170.2	187.5 ms	35.6 ms	137
7	`main`	51.5	56.0 ms	28.8 ms	137
8	`main`	50.5	54.3 ms	28.2 ms	137
9	PR	51.8	55.7 ms	28.2 ms	137
10	PR	crashed 3 s after launch, as run 3

"Threads" is the highest process thread count between 60 and 340 s, sampled every second from /proc.

Reading:

In the fast state the builds match: 50.5 to 53.1 ms on main against 51.8 and 51.8 ms with the PR.
The pool never starts a second LTO worker in this title: 137 threads in every run on both builds. A second worker would add a thread.
Runs 5 and 6 are the title's slower scene state. On this main none of the four runs landed in it, but main 316615d and 319621b did in several runs yesterday (perf: compute dispatches miss the dispatch cache when their buffer addresses or layouts change every frame (PPSA01748 menu, PPSA15388) #12640). Run 6 has the same signature as a slow main run from my earlier report: about 107 dispatches per frame, about 326 µs of shader_resources per dispatch, 38 ms of Resources.Complete and 166 ms of memory_wait per frame, against 121, 430 µs, 53 ms and 175 ms on main 316615d. Fast runs on both builds have 81 to 84 dispatches and about 3 µs per dispatch. Runs 3 and 10 are the start-up race of High on Life: about 1 launch in 5 crashes at startup when two threads release the same Wwise asset request #10905, which also stops main 1 run in 5. Here 2 of the 4 completed PR runs and 0 of the 4 main runs were slow, which happens by chance in about 1 split in 5 (2 slow runs out of 8), so I can't rule out an effect from runs alone. But with no second worker the PR runs the same code path as main.

So, as before: no measurable change in this title, because it never queues two LTO jobs at once. The evidence for a gain is still the Ratchet & Clank data in the body, which can't be repeated on main until #12322 and the tail rule from #11700 land.

Summary lines and traces: all eight completed runs' summary lines are in the PR body. Traces attached: aps5-trace-2960e9a6e7e1-main-1/-2.json.gz (runs 1 and 4), aps5-trace-07369ec4046c-pr-1/-2.json.gz (runs 2 and 9), aps5-trace-07369ec4046c-pr-slow-1/-2.json.gz (runs 5 and 6).

trace_digest.py of run 1 (main)

seconds 300:20 from present 6244: 378 presents, frame p50 51.3 / p90 56.5 / p99 83.3 / max 114.8 ms, GPU busy 28.2 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit 2960e9a, dropped: no

{"commit":"2960e9a6e7e1","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":6244,"ms":20024.03},"presents":378,"frame_ms":{"p50":51.307,"p90":56.527,"p99":83.273,"max":114.85},"gpu_busy_ms_per_present":28.155,"worker_busy_pct":{"queue worker 0x0":97.354,"queue worker 0x20":95.562},"top_phases_ms_per_frame":{"Driver.Packet":101.532,"Driver.Packet.memory_wait":43.924,"Graphics.Draw":31.32,"Driver.Packet.draw":30.905,"Driver.Draw":30.707,"Graphics.Draw.draw_and_resource_release":25.561,"Driver.Draw.shader_compile_and_link":21.531,"Shader.DrawStage":18.209},"trace_dropped":false,"trace":"aps5-trace-2960e9a6e7e1-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 6475: 114.9 ms at 12191 ms: Driver.Packet (queue worker 0x0) 89.9, Graphics.Draw (draw committer) 82.2, Graphics.Draw.draw_and_resource_release (draw committer) 78.4, Driver.Packet.dispatch (queue worker 0x0) 67.6
present 6478: 90.9 ms at 12419 ms: Driver.Packet (queue worker 0x0) 65.6, Driver.Packet (queue worker 0x20) 56.2, Graphics.Draw (draw committer) 55.6, Graphics.Draw.draw_and_resource_release (draw committer) 51.3
present 6578: 89.9 ms at 17674 ms: Driver.Packet (queue worker 0x20) 86.9, Driver.Packet.memory_wait (queue worker 0x20) 79.3, Driver.Packet (queue worker 0x0) 67.2, Driver.Packet.draw (queue worker 0x0) 57.8
present 6465: 83.3 ms at 11637 ms: Driver.Packet (queue worker 0x20) 80.6, Driver.Packet.memory_wait (queue worker 0x20) 76.5, Driver.Packet (queue worker 0x0) 60.7, Driver.Packet.draw (queue worker 0x0) 52.8
present 6354: 80.5 ms at 5720 ms: Driver.Packet (queue worker 0x20) 77.7, Driver.Packet.memory_wait (queue worker 0x20) 72.1, Driver.Packet (queue worker 0x0) 57.2, Driver.Packet.draw (queue worker 0x0) 46.9

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	26.4 (50 %)	2.2 (4 %)	24.4	113.0	Driver.Packet 28.2, Driver.EndSubmission 0.3
draw committer	20.2 (38 %)	0.2 (0 %)	32.6	28.0	Graphics.Draw 19.9, Recorder.Submit 0.2
queue worker 0x20	7.7 (15 %)	44.8 (85 %)	0.4	1.8	Driver.Packet 50.0, Driver.EndSubmission 0.2
presenter	0.3 (1 %)	0.4 (1 %)	52.3	0.1	VideoOut.Flip 0.7, Driver.Present 0.0
gpu	28.2 (53 %)	0.0 (0 %)	24.8	0.0	batch 28.2

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 78.2, Driver.Packet.memory_wait 43.9, batch 28.2, Graphics.Draw 19.9, Driver.Packet.draw 19.0, Driver.Draw 18.8, Graphics.Draw.draw_and_resource_release 15.8, Driver.Packet.dispatch 12.0, Driver.Draw.shader_compile_and_link 8.8, Driver.Dispatch 4.1

Longest waits

Driver.WorkerWait (queue worker 0x20) 65.8 ms at 12240 ms, present 6475
Driver.Packet.memory_wait (queue worker 0x20) 60.0 ms at 17700 ms, present 6578
Driver.Packet.memory_wait (queue worker 0x20) 51.6 ms at 5744 ms, present 6354
Driver.Packet.memory_wait (queue worker 0x20) 50.5 ms at 14757 ms, present 6522
Driver.Packet.memory_wait (queue worker 0x20) 50.0 ms at 4287 ms, present 6327
Driver.Packet.memory_wait (queue worker 0x20) 49.3 ms at 13261 ms, present 6493
Driver.Packet.memory_wait (queue worker 0x20) 49.2 ms at 19100 ms, present 6604
Driver.Packet.memory_wait (queue worker 0x20) 48.5 ms at 10107 ms, present 6436
Driver.Packet.memory_wait (queue worker 0x20) 47.5 ms at 2889 ms, present 6300
Driver.Packet.memory_wait (queue worker 0x20) 47.5 ms at 8563 ms, present 6407

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 19945: 6.9 ms at 14868 ms, present 6524: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.5, Driver.Packet.dispatch (queue worker 0x0) 0.5, Driver.Dispatch (queue worker 0x0) 0.5, Vulkan.Dispatch (queue worker 0x0) 0.1
batch 2043: 6.7 ms at 1494 ms, present 6273: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.6, Driver.Packet.dispatch (queue worker 0x0) 0.5, Driver.Dispatch (queue worker 0x0) 0.5, Vulkan.Dispatch (queue worker 0x0) 0.2
batch 10596: 6.7 ms at 7820 ms, present 6393: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.3, Driver.Packet.dispatch (queue worker 0x0) 0.3, Driver.Dispatch (queue worker 0x0) 0.3, Vulkan.Dispatch (queue worker 0x0) 0.1
batch 8238: 6.7 ms at 6069 ms, present 6360: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.2, Driver.Packet.dispatch (queue worker 0x0) 0.2, Driver.Dispatch (queue worker 0x0) 0.2, Vulkan.Dispatch (queue worker 0x0) 0.1
batch 10669: 6.6 ms at 7872 ms, present 6394: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.7, Driver.Packet.dispatch (queue worker 0x0) 0.7, Driver.Dispatch (queue worker 0x0) 0.7, Vulkan.Dispatch (queue worker 0x0) 0.1

trace_digest.py of run 2 (PR)

seconds 300:20 from present 7151: 352 presents, frame p50 51.8 / p90 66.4 / p99 142.7 / max 173.8 ms, GPU busy 28.5 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit 07369ec, dropped: no

{"commit":"07369ec4046c","dirty":false,"env":{"APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"2097152","APS5_TRACE_SECONDS":"300:20"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 300:20","first_present":7151,"ms":20010.445},"presents":352,"frame_ms":{"p50":51.826,"p90":66.432,"p99":142.661,"max":173.823},"gpu_busy_ms_per_present":28.476,"worker_busy_pct":{"queue worker 0x0":97.52,"queue worker 0x20":95.867},"top_phases_ms_per_frame":{"Driver.Packet":109.264,"Driver.Packet.memory_wait":47.051,"Graphics.Draw":34.249,"Driver.Packet.draw":32.348,"Driver.Draw":32.151,"Graphics.Draw.draw_and_resource_release":28.072,"Driver.Draw.shader_compile_and_link":21.732,"Shader.DrawStage":18.354},"trace_dropped":false,"trace":"aps5-trace-07369ec4046c-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 7451: 173.8 ms at 15984 ms: Driver.Packet (queue worker 0x20) 171.7, Driver.Packet.memory_wait (queue worker 0x20) 164.8, Driver.Packet (queue worker 0x0) 149.8, Graphics.Draw (draw committer) 141.0
present 7487: 161.2 ms at 18798 ms: Driver.Packet (queue worker 0x20) 156.9, Driver.Packet (queue worker 0x0) 137.0, Graphics.Draw (draw committer) 129.1, Graphics.Draw.draw_and_resource_release (draw committer) 123.9
present 7465: 145.4 ms at 17077 ms: Driver.Packet (queue worker 0x20) 142.7, Driver.Packet.memory_wait (queue worker 0x20) 135.6, Driver.Packet (queue worker 0x0) 120.5, Graphics.Draw (draw committer) 112.0
present 7491: 142.7 ms at 19168 ms: Driver.Packet (queue worker 0x20) 140.5, Driver.Packet.memory_wait (queue worker 0x20) 132.4, Driver.Packet (queue worker 0x0) 116.0, Graphics.Draw (draw committer) 111.1
present 7486: 141.4 ms at 18656 ms: Driver.Packet (queue worker 0x20) 139.0, Driver.Packet.memory_wait (queue worker 0x20) 130.8, Driver.Packet (queue worker 0x0) 117.0, Graphics.Draw (draw committer) 108.7

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	30.2 (53 %)	2.2 (4 %)	24.5	114.0	Driver.Packet 32.0, Driver.EndSubmission 0.3
draw committer	23.3 (41 %)	0.2 (0 %)	33.4	28.2	Graphics.Draw 22.8, Recorder.Submit 0.4
queue worker 0x20	8.0 (14 %)	48.4 (85 %)	0.4	1.7	Driver.Packet 53.9, Driver.EndSubmission 0.3
presenter	0.3 (0 %)	0.4 (1 %)	56.2	0.1	VideoOut.Flip 0.7, Driver.Present 0.0
title	0.0 (0 %)	0.0 (0 %)	56.8	0.0	Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveAbi 0.0
gpu	28.5 (50 %)	0.0 (0 %)	28.4	0.0	batch 28.5

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 85.9, Driver.Packet.memory_wait 47.0, batch 28.5, Graphics.Draw 22.8, Driver.Packet.draw 20.4, Driver.Draw 20.2, Graphics.Draw.draw_and_resource_release 18.3, Driver.Packet.dispatch 13.5, Driver.Draw.shader_compile_and_link 8.9, Driver.Dispatch 5.4

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 90.4 ms at 16064 ms, present 7451
Driver.Packet.memory_wait (queue worker 0x20) 87.7 ms at 18867 ms, present 7487
Driver.Packet.memory_wait (queue worker 0x20) 85.3 ms at 19222 ms, present 7491
Driver.Packet.memory_wait (queue worker 0x20) 84.4 ms at 16890 ms, present 7463
Driver.Packet.memory_wait (queue worker 0x20) 84.3 ms at 16707 ms, present 7461
Driver.Packet.memory_wait (queue worker 0x20) 83.0 ms at 17682 ms, present 7472
Driver.Packet.memory_wait (queue worker 0x20) 82.1 ms at 18712 ms, present 7486
Driver.Packet.memory_wait (queue worker 0x20) 72.4 ms at 17144 ms, present 7465
Driver.Packet.memory_wait (queue worker 0x20) 71.8 ms at 18023 ms, present 7477
Driver.Packet.memory_wait (queue worker 0x20) 69.1 ms at 18206 ms, present 7479

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 17894: 7.2 ms at 13215 ms, present 7399: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 1.0, Driver.Packet.dispatch (queue worker 0x0) 1.0, Driver.Dispatch (queue worker 0x0) 0.9, Vulkan.Dispatch (queue worker 0x0) 0.5
batch 5482: 6.9 ms at 3979 ms, present 7227: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.2, Driver.Packet.dispatch (queue worker 0x0) 0.2, Driver.Dispatch (queue worker 0x0) 0.1, Vulkan.Dispatch (queue worker 0x0) 0.0
batch 8660: 6.9 ms at 6332 ms, present 7271: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.4, Driver.Packet.dispatch (queue worker 0x0) 0.4, Driver.Dispatch (queue worker 0x0) 0.4, Vulkan.Dispatch (queue worker 0x0) 0.1
batch 2577: 6.9 ms at 1879 ms, present 7187: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.4, Driver.Packet.dispatch (queue worker 0x0) 0.4, Driver.Dispatch (queue worker 0x0) 0.3, Vulkan.Dispatch (queue worker 0x0) 0.1
batch 8083: 6.9 ms at 5913 ms, present 7263: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 0.3, Driver.Packet.dispatch (queue worker 0x0) 0.3, Driver.Dispatch (queue worker 0x0) 0.3, Vulkan.Dispatch (queue worker 0x0) 0.2

Tested and written by Claude (AI assistant) on a contributor's machine.

FILES:
aps5-trace-07369ec4046c-pr-2.json.gz
aps5-trace-07369ec4046c-pr-slow-2.json.gz
aps5-trace-07369ec4046c-pr-slow-1.json.gz
aps5-trace-2960e9a6e7e1-main-2.json.gz
aps5-trace-07369ec4046c-pr-1.json.gz
aps5-trace-2960e9a6e7e1-main-1.json.gz

MarcZX10

Re-check — changes requested

The new High on Life report is more informative, but it still does not establish a performance benefit or clear the observed risk:

In the fast state, main and the PR are effectively identical, and the title never starts a second LTO worker, so this run does not exercise the intended extra-worker path.
Two of four completed PR runs entered the slow state, versus zero of four main runs. The profile resembles a previously observed slow main run, so these samples do not prove causation, but they also do not rule out a regression.
Two of four PR runs crashed before the first frame, versus zero of four main runs in this pair. The report notes a startup race also occurs on main; the present sample is too small to compare its rate reliably.

Please keep the startup-crash and slow-state observations visible, and provide enough interleaved same-base runs to establish whether this change affects either rate. For the claimed gain, use a workload where the second LTO worker actually starts; the report says the Ratchet & Clank comparison cannot yet be repeated on main until #12322 and #11700 land. If that is the only qualifying workload, keep the performance claim pending those prerequisites and report the exact composed baseline/change pair then.

The branch is based on 2960e9a; current upstream main is aaccd0b. Please refresh the branch and rerun the focused worker tests on the resulting exact head before merge. The author-reported 1112/1113 AGC run, 200/200 burst repeats, and traces are not independently rerun by me.

Tamalero · 2026-10-30T02:42:51Z

@MarcZX10 refreshed: rebased onto current main bb04a4f with no conflicts (git range-diff shows the three commits unchanged), head 2e6cd4b, submodules at main's pins.

On 2e6cd4b (Linux 7.3, GCC 16.2.1, Release, BUILD_TESTING=ON):

The 16 host-only pipeline tests pass: agc_pipeline_library_workers, agc_pipeline_library_workers_burst, agc_pipeline_library_workers_invalid_{negative,text,zero,suffix,above_hardware}, agc_pipeline_link_optimization, agc_pipeline_queue_failure_{wait,clear,stop} and agc_pipeline_cache_allocation_{startup,now,busy,destructor,worker}.
agc_pipeline_library_workers_burst passes 200 of 200 (ctest --repeat until-fail:200).
git diff --check and tools/check_conventions.py --base origin/main pass.
The 16 agc_pipeline tests that open a Vulkan device (agc_driver_graphics_tests, agc_driver_recorder_tests) also pass, so all 32 pass on this head.

On the rest, agreed:

The performance claim stays pending. Ratchet & Clank: Rift Apart (PPSA01474) is the only title here where the second worker starts, and it cannot run on plain main yet. Besides the tail rule from libSceAjm.native: Ratchet & Clank: Rift Apart hangs at the title screen because a run after the ATRAC9 gapless total decodes the last superframe's padding as a new segment #11700, my stack for it carries the open PRs fix(shader): read a chased pointer's high word at its own offset #9749, fix(agc): bind a written TYPE 0 or 1 static image slot as a null storage image #10906 and feat(shader): raise the sampler and sampled-pair limits past the 32-bit sampler masks #10000 and local AJM, AudioOut2 and AMM workarounds. The pair will be that composed stack with and without this PR, listed exactly. (Corrected: an earlier version of this comment said it waits only on libSceAjm.native: Ratchet & Clank: Rift Apart hangs at the title screen because a run after the ATRAC9 gapless total decodes the last superframe's padding as a new segment #11700.)
The High on Life start-up crashes (2 of 4 PR runs, 0 of 4 main runs) and slow-state runs (2 of 4 against 0 of 4) stay visible. Four runs per build cannot separate those rates; that needs more interleaved runs on one base.

Tested and written by Claude (AI assistant) on a contributor's machine.

TechLevelZero · 2026-10-30T03:14:36Z

✅ NVIDIA GPU test: passed

PR head 2e6cd4b merged onto main 9f748f4 · tested 2026-10-30 22:20 UTC · 21 min


GPU	NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, `-DANYPS5_ENABLE_SPIRV_TOOLS=ON`)
Tests	✅ 2,400 passed · 5 failed (0 new, 2 also fail on main, 3 unstable on main) · 42 skipped (`ANYPS5_REQUIRE_VULKAN=1`)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with `VK_LAYER_KHRONOS_validation` and `ANYPS5_NO_SHADER_CACHE=1`; main ran the same way
Test selection	2,447 of 3,021 tests run: the ones this change can reach, chosen by `tools/select_tests.py` from main's build graph
Vulkan validation	✅ no new validation IDs vs main (917 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ `agc_driver_graphics_maps_fallback` 30.9 s

Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)

agc_driver_submit_progress: failed in 5 of the last 9 main runs
agc_submit_progress: failed in 5 of the last 9 main runs
agc_submit_progress_no_timeline: failed in 7 of the last 9 main runs

Failing on main too (2, not counted against this PR)

guest_mmap_executable
guest_socket_options_multicast

Validated but not compared (7, not in the main baseline's validation pass)

agc_pipeline_library_workers
agc_pipeline_library_workers_burst
agc_pipeline_library_workers_invalid_above_hardware
agc_pipeline_library_workers_invalid_negative
agc_pipeline_library_workers_invalid_suffix
agc_pipeline_library_workers_invalid_text
agc_pipeline_library_workers_invalid_zero

Validation IDs already present on main (4 IDs in 428 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (428 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_alias_copy_label, agc_driver_array_layer_target, agc_driver_array_layer_target_cpu, agc_driver_bda_anchor_reads, and 420 more
VUID-vkCmdCopyImage-srcImage-01548 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-vkCmdCopyImage-srcImage-01551 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target

Skipped (42)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or plat

Reproduce

git fetch origin 2e6cd4b241446652dc2ae2167b22bad2e95e6a93 9f748f4d8e36ce3714f3b95f3386316e110a9b22
git checkout 9f748f4d8e36ce3714f3b95f3386316e110a9b22 && git merge --no-ff 2e6cd4b241446652dc2ae2167b22bad2e95e6a93
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}

MarcZX10

Re-checked — new head, changes requested

The refreshed report is candid about the unproven performance claim, but the exact-head qualification still uses an older main snapshot and supplies no matched performance capture.

What I ran

Reviewed head 2e6cd4b241446652dc2ae2167b22bad2e95e6a93 against current main 5ecef095b61e7a406b279c97896f3a8a6fbd6353; merge preview tree is 7cfa228aa4e9f80ce389075ab390d1ce073c967c.

git merge-tree --write-tree refs/remotes/upstream/main refs/remotes/upstream/pr-12323
git diff --check refs/remotes/upstream/main...refs/remotes/upstream/pr-12323
git diff --name-only bb04a4f1ea0175e9e4f149f0359a764d62ff3296..refs/remotes/upstream/main -- core/libs/prx/libSceAgcDriver/CMakeLists.txt docs/dev/ENVIRONMENT.md

Results

Check	Result
Merge preview	Clean
Diff check	Passed; no output
Hosted conventions and Windows	Passed
Linux/product and specialized hosted jobs	Skipped
Pipeline tests	Author reports 32/32 on head `2e6cd4b`, including 200/200 burst repeats; not independently run
Performance evidence	PR description explicitly says the claim is pending; the High on Life route never starts a second worker

Findings

Blocking

This PR is labeled perf, but the required same-title, same-scene, same-machine main/change capture pair is not present. The report says the only workload that starts a second worker awaits #11700 and the stated composed stack; the High on Life runs do not exercise the change.
The author’s current report uses main bb04a4f. Current main 5ecef09 has since changed the shared AGC CMake file and environment documentation. The current merge preview is clean, but no focused comparison is reported on this latest base.

Not verified

I did not run a local build, pipeline tests, GPU tests, or performance captures. The test and trace results above are author-reported.

To get this to a good state

Please refresh and rerun the focused test comparison against current main. Keep the performance claim pending until the qualifying workload can be compared on a matched baseline/change stack; then add the required aps5-summary.jsonl lines, traces, and tools/trace_digest.py output. I’ll re-check the new head and rerun these source checks.

Tamalero · 2026-10-30T06:39:39Z

@MarcZX10 a correction that changes this PR's evidence: none of my game measurements exercised it. Since #3434 (merged 2026-10-11), PipelineLinkTimeOptimizationUsable returns false on VK_DRIVER_ID_NVIDIA_PROPRIETARY, so on my RTX 4090 the background link-time optimization never queues a job, and the worker pool is never started. Every base I measured on already had #3434.

How it showed up: I ran the Ratchet & Clank: Rift Apart (PPSA01474) pair on the composed stack. Base: main bb04a4f + #9749 (babbb0e) + #10906 (06cb325) + #10000 (7f4e97c) + local AJM gapless and AudioOut2 workarounds. Change: the same stack + this PR's three commits. Same route each run (title → New Game → opening → the crowd scene), Performance mode, warm caches:

Run	Build	Average FPS	Peak threads
1	base	1.75	103
2	PR	2.42	103
4	base	1.96	103
5	PR	2.19	103

Run 3 (PR) was closed early by hand and is not counted. The peak thread count never moved, so I added a local log line (not pushed) at enqueue in PipelineLibrary.cpp and ran the PR build over the same route: zero link jobs queued in 659 s. The FPS differences above are run-to-run variation; the two builds execute the same code on this driver.

This also voids the Ratchet numbers in the body ("menu about 7–9 FPS against 4.5…"): those came from a local worker-count switch on a base that already had #3434, so the link workers cannot have caused them. I have removed them and the performance claim from the body.

What is left is the change itself for drivers where the link runs (RADV and other non-NVIDIA drivers). I'll measure it on an AMD GPU (RADV) next, look for a title that queues more than one link job at once, and post the main/change pair with summary lines, traces and digests if I find one. Until then there is no performance evidence for this PR, and I'm not asking for it to be merged.

Tested and written by Claude (AI assistant) on a contributor's machine.

MarcZX10 · 2026-10-30T06:44:46Z

Re-checked — performance evidence pending

Thanks for correcting the measurements: the title captures did not queue an LTO job, so they do not measure this change; the perf checklist still needs matched main and change captures from a run that exercises LTO, with both summary lines, traces, and trace_digest.py output (repository PR template at the current base). I’ll reassess the evidence when it is posted.

adriadam10

The pool, the spawn condition, the shutdown path, the range check and the tests look right to me (the new tests need a second worker to pass). What is left is the perf evidence, and it blocks the merge:

The PR body says none of the game captures exercised this change: since #3434 PipelineLinkTimeOptimizationUsable is false on NVIDIA's proprietary driver, so no LTO job was ever queued in those runs. A perf PR needs the matched aps5-summary.jsonl lines for main and the change, both traces and the tools/trace_digest.py output, from a run that does queue LTO jobs. Please capture it on a driver where the link runs (RADV or another non-NVIDIA driver), same title, scene and machine, and show that jobs were queued (a count at enqueue, or PipelineLibraryWorkers() above 1).
Because the default changes from one worker to half the hardware threads on every other driver, include peak RSS before/after on that run: each concurrent LTO compile holds its own compiler memory.
If you cannot run a driver where it applies, say so in the PR and we will decide from the tests alone, but the perf label check will stay red until the report is there.

Nothing else to change in the code.

Tamalero · 2026-10-30T18:11:07Z

@adriadam10 thanks. Ratchet & Clank, the title I measured on NVIDIA, can't give the RADV pair: on my only AMD card (Radeon Pro V620, RADV) it is OOM-killed about 4 minutes in, on main-based builds with and without this PR, because RADV cannot compile one of its compute shaders (#13538; the same happens on Mesa 25.2.8 and 26.2.6). So there is no matched Ratchet capture on AMD to compare.

I'll rebase this PR on current main and try High on Life (PPSA15388) on the V620 instead: main and the change on the same scene, with a count at enqueue to show link jobs are queued, PipelineLibraryWorkers(), the summary lines, traces and trace_digest.py output, and peak RSS for both. If High on Life doesn't queue LTO jobs there either, I'll say so here.

Tested and written by Claude (AI assistant) on a contributor's machine.

…dware threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…he hardware thread count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…rker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Tamalero · 2026-10-30T22:08:30Z

@MarcZX10 @adriadam10 the RADV captures. Summary: on RADV this title does queue link jobs, and the pool runs and removes the backlog, but I measured no effect on load time or frame time. I can't show a performance gain from this PR on the one title I can run on AMD.

Setup. High on Life (PPSA15388) on a Radeon Pro V620, RADV Mesa 25.2.8, Linux 7.0, in a VM with 24 hardware threads, so the PR starts at most 12 workers. 3840x2160. Every run restores the same save, presses Cross at the title and at Continue, and gets no other input. Builds:

main 4bc9e77 and this PR rebased on it, now pushed as e231257.
Both carry the same local patch, which is not part of the PR. It prints link jobs enqueued and completed, busy workers, the largest queue and workers started. The summary lines therefore show eea81e370cbf (main + patch) and 22abb199377a (this PR + patch).

Link jobs are queued on this driver. Every run queues about 400 jobs. With a warm cache the queue never holds more than 1–2. With an empty cache about 375 of them arrive in one burst while the level loads, about 190–200 s after launch.

Cold runs (new, empty shader cache each run). Order main, PR, PR, main, done twice: untraced, then traced from 135 s to 180 s after the first present.

	`main`	this PR
workers started	1, 1, 1, 1	12, 12, 12, 12
largest queue	142, 144, 145, 141	10, 13, 10, 11
burst start → all jobs done (2 s sampling)	about 15–19 s	about 10–14 s
all jobs done, seconds after launch	~208–210	~204–210
load freeze (longest frame, traced runs)	12.5 s, 12.9 s	13.2 s, 12.8 s
after the freeze, traced runs: frame p50 / p90	222.2 / 342.5, 217.7 / 343.6 ms	212.0 / 353.1, 208.3 / 350.0 ms
after the freeze: GPU busy per present	87.6, 88.9 ms	87.9, 86.3 ms
peak RSS (VmHWM), untraced / traced	11,303, 11,303 / 11,970, 11,962 MiB	11,380, 11,336 / 12,085, 12,106 MiB
peak threads	135 / 137	146 / 148

Warm runs, traced in gameplay from 220 s to 240 s after the first present, order main, PR, PR, main. Frame p50 199.7 and 219.2 ms on main, 214.7 and 199.3 ms with the PR. Driver.Draw.shader_compile_and_link per frame: 66.8 and 69.9 ms against 64.5 and 66.0 ms.

Why there is no visible effect here:

With the PR, completions keep pace with enqueues, so the burst is limited by how fast the title creates pipelines, not by the link.
Draws use the fast-linked pipelines meanwhile, and the frame resumes before main has drained its backlog.
The load freeze is spent in Vulkan.GraphicsPipelinePrebuild on the shader prepare threads (15.4 s inclusive in cold run 1), which is the registration prebuild, not the link.
In gameplay the frame is bound by Driver.Packet.memory_wait and the GPU.

Cost: 11 more threads, and peak RSS 30–145 MiB higher in these runs.

Not done:

Dean's run at 8 or fewer hardware threads.
Any second title on AMD.

If a change without a measured gain isn't wanted, I'm fine with closing it.

Traces (8) and the warm-run digests are attached in the next comment. The cold-run digests are below.

trace_digest.py, cold run 1 (main)

seconds 135:45 from present 954: 372 presents, frame p50 21.6 / p90 248.3 / p99 635.7 / max 12522.8 ms, GPU busy 25.6 ms per present
AMD Radeon Pro V620 (RADV NAVI21), radv Mesa 25.2.8-0ubuntu0.24.04.4, Linux 7.0.0-38-generic, 3840x2160, commit eea81e370cbf, dropped: no

{"commit":"eea81e370cbf","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_NULL_INVALID_IMAGES":"1","APS5_OCCLUSION_ALL_VISIBLE":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_ASSUME_UNLISTED_CHUNKS_INSTALLED":"1","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"135:45","APS5_UNNORMALIZED_ANY_LEVEL":"1"},"gpu":"AMD Radeon Pro V620 (RADV NAVI21)","driver":"radv Mesa 25.2.8-0ubuntu0.24.04.4","os":"Linux 7.0.0-38-generic","cpu":"AMD EPYC 7452 32-Core Processor","cpu_threads":24,"memory_mib":76385,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 135:45","first_present":954,"ms":45219.578},"presents":372,"frame_ms":{"p50":21.636,"p90":248.329,"p99":635.698,"max":12522.847},"gpu_busy_ms_per_present":25.621,"worker_busy_pct":{"queue worker 0x0":85.654,"queue worker 0x20":80.4},"top_phases_ms_per_frame":{"Driver.Packet":196.236,"Driver.Packet.memory_wait":84.122,"Vulkan.GraphicsPipeline":62.169,"Vulkan.GraphicsPipelineLibrary":61.718,"Graphics.Draw":56.986,"Driver.Packet.draw":49.936,"Driver.Draw":49.728,"Graphics.Draw.draw_and_resource_release":48.829},"trace_dropped":false,"trace":"aps5-trace-eea81e370cbf-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 1232: 12522.8 ms at 8158 ms: Vulkan.GraphicsPipelinePrebuild (shader prepare) 15427.7, Vulkan.GraphicsPipeline (shader prepare) 15415.6, Vulkan.GraphicsPipelineLibrary (shader prepare) 15342.8, Driver.Packet (queue worker 0x0) 11729.1
present 1233: 2118.7 ms at 20681 ms: Driver.Packet (queue worker 0x20) 2105.1, Driver.Packet.memory_wait (queue worker 0x20) 2075.8, Driver.Packet (queue worker 0x0) 2045.3, Driver.Packet.draw (queue worker 0x0) 1657.6
present 1234: 652.1 ms at 22800 ms: Driver.Packet (queue worker 0x20) 629.9, Driver.Packet (queue worker 0x0) 623.0, Driver.Packet.memory_wait (queue worker 0x20) 319.2, Driver.Packet.dispatch (queue worker 0x20) 300.2
present 1165: 635.7 ms at 6399 ms: Driver.WorkerWait (queue worker 0x20) 635.1, Driver.WorkerWait (queue worker 0x0) 570.4, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 570.4, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x20) 569.8
present 968: 604.0 ms at 1016 ms: Driver.WorkerWait (queue worker 0x20) 603.0, Driver.WorkerWait (queue worker 0x0) 545.9, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 545.9, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x20) 544.9

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
shader prepare	0.7 (1 %)	0.0 (0 %)	120.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	120.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	120.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	120.9	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.6 (1 %)	0.0 (0 %)	120.9	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.7 (1 %)	0.0 (0 %)	120.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.5 (0 %)	0.0 (0 %)	121.0	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.6 (1 %)	0.0 (0 %)	120.9	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.5 (0 %)	0.0 (0 %)	121.0	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (0 %)	0.0 (0 %)	121.1	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.6 (1 %)	0.0 (0 %)	120.9	0.0	Shader.PrepareArtifact 0.6
shader prepare	1.1 (1 %)	0.0 (0 %)	120.5	0.0	Shader.PrepareArtifact 1.1
shader prepare	3.3 (3 %)	0.0 (0 %)	118.2	0.0	Vulkan.GraphicsPipelinePrebuild 3.2, Vulkan.ComputePipeline 0.1
shader prepare	3.6 (3 %)	0.0 (0 %)	117.9	0.0	Vulkan.GraphicsPipelinePrebuild 3.5, Vulkan.ComputePipeline 0.2
shader prepare	4.2 (3 %)	0.0 (0 %)	117.3	0.0	Vulkan.GraphicsPipelinePrebuild 3.9, Vulkan.ComputePipeline 0.3
shader prepare	3.7 (3 %)	0.0 (0 %)	117.9	0.0	Vulkan.GraphicsPipelinePrebuild 3.6, Vulkan.ComputePipeline 0.1
shader prepare	3.5 (3 %)	0.0 (0 %)	118.0	0.0	Vulkan.GraphicsPipelinePrebuild 3.5
shader prepare	3.8 (3 %)	0.0 (0 %)	117.8	0.0	Vulkan.GraphicsPipelinePrebuild 3.7, Vulkan.ComputePipeline 0.1
shader prepare	3.9 (3 %)	0.0 (0 %)	117.7	0.0	Vulkan.GraphicsPipelinePrebuild 3.9
shader prepare	3.6 (3 %)	0.0 (0 %)	118.0	0.0	Vulkan.GraphicsPipelinePrebuild 3.6
shader prepare	3.5 (3 %)	0.0 (0 %)	118.1	0.0	Vulkan.GraphicsPipelinePrebuild 3.3, Vulkan.ComputePipeline 0.1
shader prepare	3.7 (3 %)	0.0 (0 %)	117.8	0.0	Vulkan.GraphicsPipelinePrebuild 3.4, Vulkan.ComputePipeline 0.3
shader prepare	3.9 (3 %)	0.0 (0 %)	117.6	0.0	Vulkan.GraphicsPipelinePrebuild 3.9
shader prepare	3.9 (3 %)	0.0 (0 %)	117.6	0.0	Vulkan.GraphicsPipelinePrebuild 3.8, Vulkan.ComputePipeline 0.1
queue worker 0x0	93.6 (77 %)	18.2 (15 %)	9.8	31.4	Driver.Packet 98.3, Driver.EndSubmission 0.3
draw committer	62.1 (51 %)	3.0 (2 %)	56.5	6.2	Graphics.Draw 56.4, Recorder.Submit 5.2
queue worker 0x20	11.1 (9 %)	110.2 (91 %)	0.3	0.7	Driver.Packet 92.2, Driver.EndSubmission 5.3
presenter	5.0 (4 %)	3.3 (3 %)	113.3	0.1	VideoOut.Flip 8.2, Driver.Present 0.0
title	2.7 (2 %)	0.0 (0 %)	118.8	0.0	Shader.ResolveAbi 2.6, Shader.ResolveGraphicsStagesAbi 0.1
title	3.2 (3 %)	0.0 (0 %)	118.3	0.0	Shader.ResolveAbi 3.0, Shader.ResolveGraphicsStagesAbi 0.2
title	0.6 (0 %)	1.0 (1 %)	120.0	0.0	Shader.Register 0.5, Shader.ResolveGraphicsStagesAbi 0.0
title	0.2 (0 %)	0.0 (0 %)	121.3	0.0	Shader.Register 0.2
title	2.6 (2 %)	0.0 (0 %)	119.0	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.1
title	2.8 (2 %)	0.0 (0 %)	118.8	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.2
title	2.5 (2 %)	0.0 (0 %)	119.0	0.0	Shader.ResolveAbi 2.4, Shader.ResolveGraphicsStagesAbi 0.2
title	2.7 (2 %)	0.0 (0 %)	118.9	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.1
title	2.2 (2 %)	0.0 (0 %)	119.4	0.0	Shader.ResolveAbi 2.1, Shader.ResolveGraphicsStagesAbi 0.0
title	0.1 (0 %)	0.0 (0 %)	121.5	0.0	Shader.Register 0.1
title	0.5 (0 %)	0.0 (0 %)	121.1	0.0	Shader.Register 0.5
title	0.0 (0 %)	0.0 (0 %)	121.5	0.0	Shader.Register 0.0
title	0.2 (0 %)	0.0 (0 %)	121.4	0.0	Shader.Register 0.2
title	0.2 (0 %)	0.0 (0 %)	121.4	0.0	Shader.Register 0.2
title	0.1 (0 %)	0.0 (0 %)	121.4	0.0	Shader.Register 0.1
gpu	26.1 (21 %)	0.0 (0 %)	95.5	0.0	batch 26.1

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 190.5, Driver.Packet.memory_wait 84.1, Vulkan.GraphicsPipeline 62.2, Vulkan.GraphicsPipelineLibrary 61.7, Graphics.Draw 56.4, Driver.Packet.draw 49.6, Driver.Draw 49.3, Graphics.Draw.draw_and_resource_release 47.1, Vulkan.GraphicsPipelinePrebuild 43.4, Driver.WorkerWait 36.9

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 5466.8 ms at 12959 ms, present 1232
Driver.Packet.memory_wait (queue worker 0x20) 2166.8 ms at 9820 ms, present 1232
Driver.WorkerWait (queue worker 0x20) 1657.3 ms at 8163 ms, present 1232
Driver.Packet.memory_wait (queue worker 0x20) 1281.2 ms at 19375 ms, present 1232
Driver.Packet.memory_wait (queue worker 0x20) 1158.2 ms at 20934 ms, present 1233
Driver.Packet.memory_wait (queue worker 0x20) 762.8 ms at 12194 ms, present 1232
Driver.Packet.memory_wait (queue worker 0x20) 690.7 ms at 22096 ms, present 1233
Driver.WorkerWait (queue worker 0x0) 644.8 ms at 8163 ms, present 1232
Driver.WorkerWait (queue worker 0x20) 616.5 ms at 6387 ms, present 1164
Driver.WorkerWait (queue worker 0x20) 585.4 ms at 1021 ms, present 968

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 1389: 49.2 ms at 19030 ms, present 1232: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 94.6, Driver.Packet.dispatch (queue worker 0x0) 94.6, Driver.Dispatch (queue worker 0x0) 94.6, Vulkan.ComputePipeline (queue worker 0x0) 63.7
batch 2024: 17.1 ms at 23801 ms, present 1235: GPU no ranges; recorded by Graphics.Draw (draw committer) 7.9, Graphics.Draw.draw_and_resource_release (draw committer) 7.2, Recorder.Submit (draw committer) 0.4, Resources.Complete (draw committer) 0.2
batch 1244: 15.4 ms at 13965 ms, present 1232: GPU no ranges; recorded by Graphics.Draw (draw committer) 144.2, Graphics.Draw.draw_and_resource_release (draw committer) 140.4, Vulkan.GraphicsPipeline (draw committer) 129.1, Vulkan.GraphicsPipeline.create (draw committer) 127.5
batch 1692: 15.3 ms at 22107 ms, present 1233: GPU no ranges; recorded by Graphics.Draw (draw committer) 5.8, Graphics.Draw.draw_and_resource_release (draw committer) 5.2, Recorder.Submit (draw committer) 0.2, Graphics.Draw.validate (draw committer) 0.0
batch 1390: 14.9 ms at 19112 ms, present 1232: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 184.1, Driver.Packet.dispatch (queue worker 0x0) 112.3, Driver.Dispatch (queue worker 0x0) 112.1, Driver.Packet.draw (queue worker 0x0) 67.5

trace_digest.py, cold run 2 (this PR)

seconds 135:45 from present 954: 354 presents, frame p50 20.4 / p90 242.6 / p99 698.5 / max 13185.8 ms, GPU busy 26.0 ms per present
AMD Radeon Pro V620 (RADV NAVI21), radv Mesa 25.2.8-0ubuntu0.24.04.4, Linux 7.0.0-38-generic, 3840x2160, commit 22abb199377a, dropped: no

{"commit":"22abb199377a","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_NULL_INVALID_IMAGES":"1","APS5_OCCLUSION_ALL_VISIBLE":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_ASSUME_UNLISTED_CHUNKS_INSTALLED":"1","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"135:45","APS5_UNNORMALIZED_ANY_LEVEL":"1"},"gpu":"AMD Radeon Pro V620 (RADV NAVI21)","driver":"radv Mesa 25.2.8-0ubuntu0.24.04.4","os":"Linux 7.0.0-38-generic","cpu":"AMD EPYC 7452 32-Core Processor","cpu_threads":24,"memory_mib":76385,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 135:45","first_present":954,"ms":45068.103},"presents":354,"frame_ms":{"p50":20.433,"p90":242.618,"p99":698.495,"max":13185.836},"gpu_busy_ms_per_present":25.972,"worker_busy_pct":{"queue worker 0x0":85.494,"queue worker 0x20":81.825},"top_phases_ms_per_frame":{"Driver.Packet":206.795,"Driver.Packet.memory_wait":89.873,"Vulkan.GraphicsPipeline":65.615,"Vulkan.GraphicsPipelineLibrary":65.119,"Graphics.Draw":60.813,"Driver.Packet.draw":53.004,"Driver.Draw":52.802,"Graphics.Draw.draw_and_resource_release":52.488},"trace_dropped":false,"trace":"aps5-trace-22abb199377a-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 1217: 13185.8 ms at 7637 ms: Vulkan.GraphicsPipelinePrebuild (shader prepare) 15404.3, Vulkan.GraphicsPipeline (shader prepare) 15392.6, Vulkan.GraphicsPipelineLibrary (shader prepare) 15316.7, Driver.Packet (queue worker 0x0) 12212.6
present 1218: 2238.0 ms at 20823 ms: Driver.Packet (queue worker 0x20) 2228.8, Driver.Packet.memory_wait (queue worker 0x20) 2188.0, Driver.Packet (queue worker 0x0) 2168.9, Driver.Packet.draw (queue worker 0x0) 1698.0
present 957: 1348.0 ms at 61 ms: Driver.WorkerWait (queue worker 0x20) 1347.9, Driver.WorkerWait (queue worker 0x0) 1288.2, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 1288.2, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x20) 1288.2
present 1219: 698.5 ms at 23061 ms: Driver.Packet (queue worker 0x20) 680.5, Driver.Packet (queue worker 0x0) 672.3, Driver.Packet.dispatch (queue worker 0x20) 349.8, Driver.Dispatch (queue worker 0x20) 342.8
present 1222: 576.9 ms at 24633 ms: Driver.Packet (queue worker 0x20) 551.8, Driver.Packet (queue worker 0x0) 534.0, Driver.Packet.memory_wait (queue worker 0x20) 504.3, Graphics.Draw (draw committer) 302.0

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
shader prepare	0.7 (1 %)	0.0 (0 %)	126.6	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.8 (1 %)	0.0 (0 %)	126.5	0.0	Shader.PrepareArtifact 0.8
shader prepare	1.1 (1 %)	0.0 (0 %)	126.2	0.0	Shader.PrepareArtifact 1.1
shader prepare	0.7 (1 %)	0.0 (0 %)	126.6	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.6 (1 %)	0.0 (0 %)	126.7	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.7 (1 %)	0.0 (0 %)	126.6	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	126.6	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.6 (1 %)	0.0 (0 %)	126.7	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.6 (0 %)	0.0 (0 %)	126.7	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.7 (1 %)	0.0 (0 %)	126.6	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.8 (1 %)	0.0 (0 %)	126.5	0.0	Shader.PrepareArtifact 0.8
shader prepare	0.7 (1 %)	0.0 (0 %)	126.7	0.0	Shader.PrepareArtifact 0.7
shader prepare	4.0 (3 %)	0.0 (0 %)	123.3	0.0	Vulkan.GraphicsPipelinePrebuild 4.0
shader prepare	3.8 (3 %)	0.0 (0 %)	123.5	0.0	Vulkan.GraphicsPipelinePrebuild 3.8
shader prepare	3.9 (3 %)	0.0 (0 %)	123.4	0.0	Vulkan.GraphicsPipelinePrebuild 3.8, Vulkan.ComputePipeline 0.1
shader prepare	4.0 (3 %)	0.0 (0 %)	123.3	0.0	Vulkan.GraphicsPipelinePrebuild 4.0
shader prepare	3.9 (3 %)	0.0 (0 %)	123.5	0.0	Vulkan.GraphicsPipelinePrebuild 3.5, Vulkan.ComputePipeline 0.3
shader prepare	4.2 (3 %)	0.0 (0 %)	123.1	0.0	Vulkan.GraphicsPipelinePrebuild 3.7, Vulkan.ComputePipeline 0.6
shader prepare	4.0 (3 %)	0.0 (0 %)	123.3	0.0	Vulkan.GraphicsPipelinePrebuild 3.9, Vulkan.ComputePipeline 0.1
shader prepare	3.9 (3 %)	0.0 (0 %)	123.4	0.0	Vulkan.GraphicsPipelinePrebuild 3.9
shader prepare	3.7 (3 %)	0.0 (0 %)	123.6	0.0	Vulkan.GraphicsPipelinePrebuild 3.5, Vulkan.ComputePipeline 0.2
shader prepare	4.1 (3 %)	0.0 (0 %)	123.2	0.0	Vulkan.GraphicsPipelinePrebuild 3.9, Vulkan.ComputePipeline 0.2
shader prepare	3.9 (3 %)	0.0 (0 %)	123.4	0.0	Vulkan.GraphicsPipelinePrebuild 3.9
shader prepare	3.8 (3 %)	0.0 (0 %)	123.5	0.0	Vulkan.GraphicsPipelinePrebuild 3.7, Vulkan.ComputePipeline 0.2
queue worker 0x0	98.2 (77 %)	20.4 (16 %)	8.7	31.2	Driver.Packet 103.1, Driver.EndSubmission 0.4
draw committer	66.3 (52 %)	2.3 (2 %)	58.7	6.2	Graphics.Draw 60.2, Recorder.Submit 5.4
queue worker 0x20	11.9 (9 %)	115.1 (90 %)	0.3	0.7	Driver.Packet 98.1, Driver.EndSubmission 5.8
presenter	4.7 (4 %)	4.1 (3 %)	118.6	0.1	VideoOut.Flip 8.7, Driver.Present 0.1
title	3.1 (2 %)	0.0 (0 %)	124.2	0.0	Shader.ResolveAbi 2.9, Shader.ResolveGraphicsStagesAbi 0.2
title	3.1 (2 %)	0.0 (0 %)	124.2	0.0	Shader.ResolveAbi 2.9, Shader.ResolveGraphicsStagesAbi 0.1
title	0.6 (0 %)	1.7 (1 %)	125.1	0.0	Shader.Register 0.6, Shader.ResolveGraphicsStagesAbi 0.0
title	0.3 (0 %)	0.0 (0 %)	127.0	0.0	Shader.Register 0.3
title	2.9 (2 %)	0.0 (0 %)	124.4	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.4
title	2.7 (2 %)	0.0 (0 %)	124.6	0.0	Shader.ResolveAbi 2.6, Shader.ResolveGraphicsStagesAbi 0.1
title	2.6 (2 %)	0.0 (0 %)	124.7	0.0	Shader.ResolveAbi 2.4, Shader.ResolveGraphicsStagesAbi 0.2
title	2.8 (2 %)	0.0 (0 %)	124.5	0.0	Shader.ResolveAbi 2.6, Shader.ResolveGraphicsStagesAbi 0.2
title	2.6 (2 %)	0.0 (0 %)	124.7	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.1
title	0.1 (0 %)	0.0 (0 %)	127.2	0.0	Shader.Register 0.1
title	0.2 (0 %)	0.0 (0 %)	127.1	0.0	Shader.Register 0.2
title	0.4 (0 %)	0.0 (0 %)	126.9	0.0	Shader.Register 0.4
title	0.2 (0 %)	0.0 (0 %)	127.1	0.0	Shader.Register 0.2
title	0.1 (0 %)	0.0 (0 %)	127.2	0.0	Shader.Register 0.1
title	0.1 (0 %)	0.0 (0 %)	127.3	0.0	Shader.Register 0.1
gpu	26.7 (21 %)	0.0 (0 %)	100.7	0.0	batch 26.7

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 201.3, Driver.Packet.memory_wait 89.9, Vulkan.GraphicsPipeline 65.6, Vulkan.GraphicsPipelineLibrary 65.1, Graphics.Draw 60.2, Driver.Packet.draw 52.6, Driver.Draw 52.4, Graphics.Draw.draw_and_resource_release 50.8, Vulkan.GraphicsPipelinePrebuild 45.6, Driver.WorkerWait 38.1

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 5808.0 ms at 12623 ms, present 1217
Driver.Packet.memory_wait (queue worker 0x20) 2234.2 ms at 9263 ms, present 1217
Driver.WorkerWait (queue worker 0x20) 1624.3 ms at 7638 ms, present 1217
Driver.WorkerWait (queue worker 0x20) 1330.9 ms at 57 ms, present 956
Driver.WorkerWait (queue worker 0x0) 1287.7 ms at 57 ms, present 956
Driver.Packet.memory_wait (queue worker 0x20) 1270.3 ms at 19534 ms, present 1217
Driver.Packet.memory_wait (queue worker 0x20) 1156.2 ms at 21166 ms, present 1218
Driver.Packet.memory_wait (queue worker 0x20) 921.5 ms at 11699 ms, present 1217
Driver.WorkerWait (queue worker 0x0) 833.6 ms at 7638 ms, present 1217
Driver.Packet.memory_wait (queue worker 0x20) 712.5 ms at 22335 ms, present 1218

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 1391: 49.0 ms at 19095 ms, present 1217: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 94.8, Driver.Packet.dispatch (queue worker 0x0) 94.7, Driver.Dispatch (queue worker 0x0) 94.7, Vulkan.ComputePipeline (queue worker 0x0) 62.8
batch 1235: 16.9 ms at 13255 ms, present 1217: GPU no ranges; recorded by Graphics.Draw (draw committer) 551.8, Graphics.Draw.draw_and_resource_release (draw committer) 544.1, Vulkan.GraphicsPipeline (draw committer) 522.3, Vulkan.GraphicsPipeline.create (draw committer) 518.6
batch 2103: 16.8 ms at 24112 ms, present 1220: GPU no ranges; recorded by Graphics.Draw (draw committer) 5.7, Graphics.Draw.draw_and_resource_release (draw committer) 5.0, Recorder.Submit (draw committer) 0.3, Graphics.Draw.validate (draw committer) 0.1
batch 1764: 14.8 ms at 22332 ms, present 1218: GPU no ranges; recorded by Graphics.Draw (draw committer) 6.3, Graphics.Draw.draw_and_resource_release (draw committer) 5.6, Recorder.Submit (draw committer) 0.2, Graphics.Draw.validate (draw committer) 0.1
batch 1392: 14.6 ms at 19177 ms, present 1217: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 218.1, Driver.Packet.dispatch (queue worker 0x0) 110.5, Driver.Dispatch (queue worker 0x0) 110.3, Driver.Packet.draw (queue worker 0x0) 100.7

trace_digest.py, cold run 3 (this PR)

seconds 135:45 from present 860: 481 presents, frame p50 17.3 / p90 200.9 / p99 564.7 / max 12768.7 ms, GPU busy 18.5 ms per present
AMD Radeon Pro V620 (RADV NAVI21), radv Mesa 25.2.8-0ubuntu0.24.04.4, Linux 7.0.0-38-generic, 3840x2160, commit 22abb199377a, dropped: no

{"commit":"22abb199377a","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_NULL_INVALID_IMAGES":"1","APS5_OCCLUSION_ALL_VISIBLE":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_ASSUME_UNLISTED_CHUNKS_INSTALLED":"1","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"135:45","APS5_UNNORMALIZED_ANY_LEVEL":"1"},"gpu":"AMD Radeon Pro V620 (RADV NAVI21)","driver":"radv Mesa 25.2.8-0ubuntu0.24.04.4","os":"Linux 7.0.0-38-generic","cpu":"AMD EPYC 7452 32-Core Processor","cpu_threads":24,"memory_mib":76385,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 135:45","first_present":860,"ms":45214.353},"presents":481,"frame_ms":{"p50":17.258,"p90":200.865,"p99":564.703,"max":12768.685},"gpu_busy_ms_per_present":18.451,"worker_busy_pct":{"queue worker 0x0":82.066,"queue worker 0x20":74.775},"top_phases_ms_per_frame":{"Driver.Packet":142.649,"Driver.Packet.memory_wait":60.307,"Vulkan.GraphicsPipeline":48.262,"Vulkan.GraphicsPipelineLibrary":47.906,"Graphics.Draw":42.85,"Graphics.Draw.draw_and_resource_release":37.036,"Driver.Packet.draw":36.676,"Driver.Draw":36.53},"trace_dropped":false,"trace":"aps5-trace-22abb199377a-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 1258: 12768.7 ms at 11178 ms: Vulkan.GraphicsPipelinePrebuild (shader prepare) 15422.3, Vulkan.GraphicsPipeline (shader prepare) 15411.4, Vulkan.GraphicsPipelineLibrary (shader prepare) 15340.1, Driver.Packet (queue worker 0x0) 11987.4
present 1259: 2204.4 ms at 23946 ms: Driver.Packet (queue worker 0x20) 2193.2, Driver.Packet.memory_wait (queue worker 0x20) 2166.4, Driver.Packet (queue worker 0x0) 2130.3, Driver.Packet.draw (queue worker 0x0) 1709.4
present 997: 1592.2 ms at 3646 ms: Driver.WorkerWait (queue worker 0x20) 1584.0, Driver.WorkerWait (queue worker 0x0) 1403.2, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 1403.2, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x20) 1395.1
present 1260: 697.8 ms at 26151 ms: Driver.Packet (queue worker 0x20) 678.9, Driver.Packet (queue worker 0x0) 669.0, Driver.Packet.dispatch (queue worker 0x20) 363.4, Driver.Dispatch (queue worker 0x20) 355.4
present 1261: 564.7 ms at 26848 ms: Driver.Packet (queue worker 0x20) 554.7, Driver.Packet (queue worker 0x0) 509.9, Driver.Packet.memory_wait (queue worker 0x20) 497.8, Graphics.Draw (draw committer) 317.9

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
title	0.0 (0 %)	0.0 (0 %)	94.0	0.0
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.7 (1 %)	0.0 (0 %)	93.3	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (0 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.6 (1 %)	0.0 (0 %)	93.4	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.6 (1 %)	0.0 (0 %)	93.4	0.0	Shader.PrepareArtifact 0.6
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	0.5 (1 %)	0.0 (0 %)	93.5	0.0	Shader.PrepareArtifact 0.5
shader prepare	3.1 (3 %)	0.0 (0 %)	90.9	0.0	Vulkan.GraphicsPipelinePrebuild 3.0, Vulkan.ComputePipeline 0.1
shader prepare	2.7 (3 %)	0.0 (0 %)	91.3	0.0	Vulkan.GraphicsPipelinePrebuild 2.6, Vulkan.ComputePipeline 0.1
shader prepare	2.8 (3 %)	0.0 (0 %)	91.2	0.0	Vulkan.GraphicsPipelinePrebuild 2.8
shader prepare	3.0 (3 %)	0.0 (0 %)	91.0	0.0	Vulkan.GraphicsPipelinePrebuild 2.7, Vulkan.ComputePipeline 0.3
shader prepare	2.8 (3 %)	0.0 (0 %)	91.2	0.0	Vulkan.GraphicsPipelinePrebuild 2.8
shader prepare	2.8 (3 %)	0.0 (0 %)	91.2	0.0	Vulkan.GraphicsPipelinePrebuild 2.6, Vulkan.ComputePipeline 0.1
shader prepare	2.8 (3 %)	0.0 (0 %)	91.2	0.0	Vulkan.GraphicsPipelinePrebuild 2.8
shader prepare	2.8 (3 %)	0.0 (0 %)	91.2	0.0	Vulkan.GraphicsPipelinePrebuild 2.8
shader prepare	3.2 (3 %)	0.0 (0 %)	90.8	0.0	Vulkan.GraphicsPipelinePrebuild 3.1, Vulkan.ComputePipeline 0.1
shader prepare	3.0 (3 %)	0.0 (0 %)	91.0	0.0	Vulkan.GraphicsPipelinePrebuild 2.8, Vulkan.ComputePipeline 0.2
shader prepare	2.7 (3 %)	0.0 (0 %)	91.3	0.0	Vulkan.GraphicsPipelinePrebuild 2.7
shader prepare	2.9 (3 %)	0.0 (0 %)	91.1	0.0	Vulkan.GraphicsPipelinePrebuild 2.8, Vulkan.ComputePipeline 0.1
queue worker 0x0	69.9 (74 %)	16.7 (18 %)	7.5	21.7	Driver.Packet 73.1, Driver.EndSubmission 0.3
draw committer	46.5 (49 %)	2.1 (2 %)	45.4	4.2	Graphics.Draw 42.5, Recorder.Submit 3.5
queue worker 0x20	8.4 (9 %)	85.4 (91 %)	0.2	0.5	Driver.Packet 65.6, Driver.EndSubmission 4.5
presenter	4.5 (5 %)	4.6 (5 %)	84.9	0.1	VideoOut.Flip 9.1, Driver.Present 0.0
title	2.6 (3 %)	0.0 (0 %)	91.4	0.0	Shader.ResolveAbi 2.4, Shader.ResolveGraphicsStagesAbi 0.2
title	2.2 (2 %)	0.0 (0 %)	91.8	0.0	Shader.ResolveAbi 2.0, Shader.ResolveGraphicsStagesAbi 0.2
title	0.2 (0 %)	0.6 (1 %)	93.2	0.0	Shader.Register 0.2, Shader.ResolveGraphicsStagesAbi 0.0
title	0.1 (0 %)	0.0 (0 %)	93.9	0.0	Shader.Register 0.1
title	2.1 (2 %)	0.0 (0 %)	91.9	0.0	Shader.ResolveAbi 1.9, Shader.ResolveGraphicsStagesAbi 0.2
title	2.1 (2 %)	0.0 (0 %)	91.9	0.0	Shader.ResolveAbi 1.7, Shader.ResolveGraphicsStagesAbi 0.3
title	2.1 (2 %)	0.0 (0 %)	91.9	0.0	Shader.ResolveAbi 2.0, Shader.ResolveGraphicsStagesAbi 0.1
title	2.2 (2 %)	0.0 (0 %)	91.8	0.0	Shader.ResolveAbi 1.8, Shader.Register 0.2
title	2.0 (2 %)	0.0 (0 %)	92.0	0.0	Shader.ResolveAbi 1.9, Shader.ResolveGraphicsStagesAbi 0.1
title	0.1 (0 %)	0.0 (0 %)	93.9	0.0	Shader.Register 0.1
title	0.0 (0 %)	0.0 (0 %)	94.0	0.0	Shader.Register 0.0
title	0.1 (0 %)	0.0 (0 %)	93.9	0.0	Shader.Register 0.1
title	0.3 (0 %)	0.0 (0 %)	93.7	0.0	Shader.Register 0.3
title	0.0 (0 %)	0.0 (0 %)	94.0	0.0	Shader.Register 0.0
title	0.1 (0 %)	0.0 (0 %)	93.9	0.0	Shader.Register 0.1
gpu	18.9 (20 %)	0.0 (0 %)	75.1	0.0	batch 18.9

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 138.7, Driver.Packet.memory_wait 60.3, Vulkan.GraphicsPipeline 48.3, Vulkan.GraphicsPipelineLibrary 47.9, Graphics.Draw 42.5, Driver.WorkerWait 36.8, Driver.Packet.draw 36.5, Driver.Draw 36.3, Graphics.Draw.draw_and_resource_release 35.8, Vulkan.GraphicsPipelinePrebuild 33.5

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 5667.0 ms at 15987 ms, present 1258
Driver.Packet.memory_wait (queue worker 0x20) 2175.0 ms at 12754 ms, present 1258
Driver.WorkerWait (queue worker 0x20) 1576.1 ms at 11178 ms, present 1258
Driver.WorkerWait (queue worker 0x20) 1555.4 ms at 3650 ms, present 997
Driver.WorkerWait (queue worker 0x0) 1387.0 ms at 3650 ms, present 997
Driver.Packet.memory_wait (queue worker 0x20) 1248.6 ms at 22674 ms, present 1258
Driver.Packet.memory_wait (queue worker 0x20) 1210.9 ms at 24236 ms, present 1259
Driver.Packet.memory_wait (queue worker 0x20) 891.2 ms at 15094 ms, present 1258
Driver.Packet.memory_wait (queue worker 0x20) 688.3 ms at 25450 ms, present 1259
Driver.WorkerWait (queue worker 0x0) 631.2 ms at 11178 ms, present 1258

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 1876: 49.0 ms at 22267 ms, present 1258: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 105.1, Driver.Packet.dispatch (queue worker 0x0) 105.0, Driver.Dispatch (queue worker 0x0) 105.0, Vulkan.ComputePipeline (queue worker 0x0) 67.5
batch 1726: 15.2 ms at 16902 ms, present 1258: GPU no ranges; recorded by Graphics.Draw (draw committer) 825.6, Graphics.Draw.draw_and_resource_release (draw committer) 812.6, Vulkan.GraphicsPipeline (draw committer) 749.9, Vulkan.GraphicsPipeline.create (draw committer) 743.1
batch 1877: 14.9 ms at 22348 ms, present 1258: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 184.9, Driver.Packet.dispatch (queue worker 0x0) 109.7, Driver.Dispatch (queue worker 0x0) 109.5, Driver.Packet.draw (queue worker 0x0) 72.9
batch 2583: 13.9 ms at 27390 ms, present 1261: GPU no ranges; recorded by Graphics.Draw (draw committer) 0.9, Graphics.Draw.draw_and_resource_release (draw committer) 0.8, Recorder.Submit (draw committer) 0.2, Resources.Prepare (draw committer) 0.1
batch 3066: 13.8 ms at 28813 ms, present 1265: GPU no ranges; recorded by Graphics.Draw (draw committer) 1.5, Graphics.Draw.draw_and_resource_release (draw committer) 1.3, Recorder.Submit (draw committer) 0.2

trace_digest.py, cold run 4 (main)

seconds 135:45 from present 896: 370 presents, frame p50 21.7 / p90 237.1 / p99 701.2 / max 12946.4 ms, GPU busy 25.2 ms per present
AMD Radeon Pro V620 (RADV NAVI21), radv Mesa 25.2.8-0ubuntu0.24.04.4, Linux 7.0.0-38-generic, 3840x2160, commit eea81e370cbf, dropped: no

{"commit":"eea81e370cbf","dirty":false,"env":{"APS5_LOG_TIMESTAMPS":"1","APS5_NO_PAD_INPUT":"1","APS5_NULL_INVALID_IMAGES":"1","APS5_OCCLUSION_ALL_VISIBLE":"1","APS5_PAD_INPUT_FILE":"<path>","APS5_PLAYGO_ASSUME_UNLISTED_CHUNKS_INSTALLED":"1","APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"135:45","APS5_UNNORMALIZED_ANY_LEVEL":"1"},"gpu":"AMD Radeon Pro V620 (RADV NAVI21)","driver":"radv Mesa 25.2.8-0ubuntu0.24.04.4","os":"Linux 7.0.0-38-generic","cpu":"AMD EPYC 7452 32-Core Processor","cpu_threads":24,"memory_mib":76385,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 135:45","first_present":896,"ms":45336.393},"presents":370,"frame_ms":{"p50":21.729,"p90":237.099,"p99":701.207,"max":12946.383},"gpu_busy_ms_per_present":25.238,"worker_busy_pct":{"queue worker 0x0":86.019,"queue worker 0x20":79.851},"top_phases_ms_per_frame":{"Driver.Packet":197.376,"Driver.Packet.memory_wait":84.883,"Vulkan.GraphicsPipeline":61.729,"Vulkan.GraphicsPipelineLibrary":61.283,"Graphics.Draw":58.355,"Driver.Packet.draw":52.026,"Driver.Draw":51.82,"Graphics.Draw.draw_and_resource_release":50.29},"trace_dropped":false,"trace":"aps5-trace-eea81e370cbf-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 1176: 12946.4 ms at 8337 ms: Vulkan.GraphicsPipelinePrebuild (shader prepare) 15196.2, Vulkan.GraphicsPipeline (shader prepare) 15182.5, Vulkan.GraphicsPipelineLibrary (shader prepare) 15108.5, Driver.Packet (queue worker 0x0) 11907.7
present 1177: 2197.1 ms at 21283 ms: Driver.Packet (queue worker 0x20) 2184.9, Driver.Packet.memory_wait (queue worker 0x20) 2156.4, Driver.Packet (queue worker 0x0) 2119.2, Driver.Packet.draw (queue worker 0x0) 1695.5
present 938: 881.7 ms at 1549 ms: Driver.WorkerWait (queue worker 0x20) 874.3, Driver.WorkerWait (queue worker 0x0) 684.1, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x0) 684.0, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x20) 676.5
present 1178: 701.2 ms at 23480 ms: Driver.Packet (queue worker 0x20) 678.7, Driver.Packet (queue worker 0x0) 670.4, Driver.Packet.memory_wait (queue worker 0x20) 367.4, Driver.Packet.dispatch (queue worker 0x20) 305.7
present 1179: 606.2 ms at 24181 ms: Driver.Packet (queue worker 0x20) 595.5, Driver.Packet (queue worker 0x0) 547.5, Driver.Packet.memory_wait (queue worker 0x20) 519.2, Graphics.Draw (draw committer) 275.6

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
title	0.0 (0 %)	0.0 (0 %)	122.5	0.0
shader prepare	0.9 (1 %)	0.0 (0 %)	121.7	0.0	Shader.PrepareArtifact 0.9
shader prepare	0.7 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.8 (1 %)	0.0 (0 %)	121.7	0.0	Shader.PrepareArtifact 0.8
shader prepare	0.8 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.8
shader prepare	0.7 (1 %)	0.0 (0 %)	121.9	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.8 (1 %)	0.0 (0 %)	121.7	0.0	Shader.PrepareArtifact 0.8
shader prepare	0.7 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.8 (1 %)	0.0 (0 %)	121.7	0.0	Shader.PrepareArtifact 0.8
shader prepare	0.7 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	0.7 (1 %)	0.0 (0 %)	121.8	0.0	Shader.PrepareArtifact 0.7
shader prepare	1.1 (1 %)	0.0 (0 %)	121.5	0.0	Shader.PrepareArtifact 1.1
shader prepare	3.7 (3 %)	0.0 (0 %)	118.8	0.0	Vulkan.GraphicsPipelinePrebuild 3.2, Vulkan.ComputePipeline 0.5
shader prepare	3.5 (3 %)	0.0 (0 %)	119.0	0.0	Vulkan.GraphicsPipelinePrebuild 3.5
shader prepare	3.9 (3 %)	0.0 (0 %)	118.6	0.0	Vulkan.GraphicsPipelinePrebuild 3.9
shader prepare	4.1 (3 %)	0.0 (0 %)	118.4	0.0	Vulkan.GraphicsPipelinePrebuild 3.9, Vulkan.ComputePipeline 0.2
shader prepare	3.6 (3 %)	0.0 (0 %)	118.9	0.0	Vulkan.GraphicsPipelinePrebuild 3.6
shader prepare	3.4 (3 %)	0.0 (0 %)	119.2	0.0	Vulkan.GraphicsPipelinePrebuild 3.3, Vulkan.ComputePipeline 0.1
shader prepare	4.0 (3 %)	0.0 (0 %)	118.6	0.0	Vulkan.GraphicsPipelinePrebuild 3.8, Vulkan.ComputePipeline 0.2
shader prepare	3.6 (3 %)	0.0 (0 %)	118.9	0.0	Vulkan.GraphicsPipelinePrebuild 3.5, Vulkan.ComputePipeline 0.2
shader prepare	3.4 (3 %)	0.0 (0 %)	119.1	0.0	Vulkan.GraphicsPipelinePrebuild 3.4
shader prepare	3.5 (3 %)	0.0 (0 %)	119.0	0.0	Vulkan.GraphicsPipelinePrebuild 3.5
shader prepare	4.0 (3 %)	0.0 (0 %)	118.5	0.0	Vulkan.GraphicsPipelinePrebuild 3.7, Vulkan.ComputePipeline 0.2
shader prepare	3.4 (3 %)	0.0 (0 %)	119.1	0.0	Vulkan.GraphicsPipelinePrebuild 3.4
queue worker 0x0	95.3 (78 %)	18.3 (15 %)	9.0	30.4	Driver.Packet 99.3, Driver.EndSubmission 0.7
draw committer	64.2 (52 %)	2.2 (2 %)	56.2	5.8	Graphics.Draw 57.9, Recorder.Submit 4.9
queue worker 0x20	11.0 (9 %)	111.2 (91 %)	0.3	0.7	Driver.Packet 92.5, Driver.EndSubmission 5.1
presenter	4.3 (4 %)	5.1 (4 %)	113.2	0.1	VideoOut.Flip 9.3, Driver.Present 0.0
title	3.6 (3 %)	0.0 (0 %)	118.9	0.0	Shader.ResolveAbi 3.2, Shader.ResolveGraphicsStagesAbi 0.4
title	2.9 (2 %)	0.0 (0 %)	119.6	0.0	Shader.ResolveAbi 2.8, Shader.ResolveGraphicsStagesAbi 0.1
title	0.4 (0 %)	1.0 (1 %)	121.2	0.0	Shader.Register 0.4, Shader.ResolveGraphicsStagesAbi 0.0
title	0.2 (0 %)	0.0 (0 %)	122.3	0.0	Shader.Register 0.2
title	2.6 (2 %)	0.0 (0 %)	119.9	0.0	Shader.ResolveAbi 2.4, Shader.ResolveGraphicsStagesAbi 0.2
title	2.9 (2 %)	0.0 (0 %)	119.6	0.0	Shader.ResolveAbi 2.8, Shader.ResolveGraphicsStagesAbi 0.1
title	2.7 (2 %)	0.0 (0 %)	119.8	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.2
title	2.8 (2 %)	0.0 (0 %)	119.7	0.0	Shader.ResolveAbi 2.5, Shader.ResolveGraphicsStagesAbi 0.3
title	2.4 (2 %)	0.0 (0 %)	120.1	0.0	Shader.ResolveAbi 2.3, Shader.ResolveGraphicsStagesAbi 0.1
title	0.7 (1 %)	0.0 (0 %)	121.9	0.0	Shader.Register 0.7
title	0.2 (0 %)	0.0 (0 %)	122.3	0.0	Shader.Register 0.2
title	0.2 (0 %)	0.0 (0 %)	122.4	0.0	Shader.Register 0.2
title	0.2 (0 %)	0.0 (0 %)	122.3	0.0	Shader.Register 0.2
title	0.2 (0 %)	0.0 (0 %)	122.3	0.0	Shader.Register 0.2
title	0.1 (0 %)	0.0 (0 %)	122.4	0.0	Shader.Register 0.1
gpu	25.7 (21 %)	0.0 (0 %)	96.8	0.0	batch 25.7

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 191.9, Driver.Packet.memory_wait 84.9, Vulkan.GraphicsPipeline 61.7, Vulkan.GraphicsPipelineLibrary 61.3, Graphics.Draw 57.9, Driver.Packet.draw 51.8, Driver.Draw 51.5, Graphics.Draw.draw_and_resource_release 48.6, Vulkan.GraphicsPipelinePrebuild 42.8, Driver.WorkerWait 38.0

Longest waits

Driver.Packet.memory_wait (queue worker 0x20) 5639.1 ms at 13219 ms, present 1176
Driver.Packet.memory_wait (queue worker 0x20) 2030.0 ms at 10081 ms, present 1176
Driver.WorkerWait (queue worker 0x20) 1739.1 ms at 8342 ms, present 1176
Driver.Packet.memory_wait (queue worker 0x20) 1299.8 ms at 19959 ms, present 1176
Driver.Packet.memory_wait (queue worker 0x20) 1166.5 ms at 21586 ms, present 1177
Driver.Packet.memory_wait (queue worker 0x20) 976.8 ms at 12240 ms, present 1176
Driver.WorkerWait (queue worker 0x20) 851.8 ms at 1554 ms, present 938
Driver.WorkerWait (queue worker 0x0) 754.6 ms at 8337 ms, present 1176
Driver.Packet.memory_wait (queue worker 0x20) 707.3 ms at 22760 ms, present 1177
Driver.WorkerWait (queue worker 0x0) 673.0 ms at 1553 ms, present 938

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 1528: 49.5 ms at 19534 ms, present 1176: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 107.0, Driver.Packet.dispatch (queue worker 0x0) 107.0, Driver.Dispatch (queue worker 0x0) 107.0, Vulkan.ComputePipeline (queue worker 0x0) 88.6
batch 1376: 21.9 ms at 13885 ms, present 1176: GPU no ranges; recorded by Graphics.Draw (draw committer) 312.6, Graphics.Draw.draw_and_resource_release (draw committer) 306.8, Vulkan.GraphicsPipeline (draw committer) 273.6, Vulkan.GraphicsPipeline.create (draw committer) 270.8
batch 2221: 17.1 ms at 24611 ms, present 1179: GPU no ranges; recorded by Graphics.Draw (draw committer) 7.0, Graphics.Draw.draw_and_resource_release (draw committer) 6.3, Resources.Complete (draw committer) 0.5, Resources.Prepare (draw committer) 0.4
batch 1650: 15.6 ms at 20028 ms, present 1176: GPU no ranges; recorded by Graphics.Draw (draw committer) 49.3, Graphics.Draw.draw_and_resource_release (draw committer) 47.9, Vulkan.GraphicsPipeline (draw committer) 38.0, Vulkan.GraphicsPipeline.create (draw committer) 37.3
batch 1871: 15.3 ms at 22775 ms, present 1177: GPU no ranges; recorded by Graphics.Draw (draw committer) 5.7, Graphics.Draw.draw_and_resource_release (draw committer) 5.1, Recorder.Submit (draw committer) 0.3, Resources.Complete (draw committer) 0.3

Tested and written by Claude (AI assistant) on a contributor's machine.

TechLevelZero · 2026-10-31T00:04:45Z

✅ NVIDIA GPU test: passed

PR head e231257 merged onto main e8efeed · tested 2026-10-31 00:04 UTC · 24 min


GPU	NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, `-DANYPS5_ENABLE_SPIRV_TOOLS=ON`)
Tests	✅ 2,414 passed · 5 failed (0 new, 2 also fail on main, 3 unstable on main) · 42 skipped (`ANYPS5_REQUIRE_VULKAN=1`)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with `VK_LAYER_KHRONOS_validation` and `ANYPS5_NO_SHADER_CACHE=1`; main ran the same way
Test selection	2,461 of 3,041 tests run: the ones this change can reach, chosen by `tools/select_tests.py` from main's build graph
Vulkan validation	✅ no new validation IDs vs main (915 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ `agc_driver_bda_device` 57.5 s, ⚠️ `agc_driver_graphics_maps_fallback` 34.2 s

Intermittent validation messages (1, seen once in the parallel run, not reproduced when the test was re-run alone 2 times; not counted against this PR)

agc_driver_depth_clear_hle: VUID-vkCmdCopyBuffer-commonparent (6x)

Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)

agc_driver_depth_clear_hle: failed in 1 of the last 10 main runs
agc_submit_progress: failed in 6 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 8 of the last 10 main runs

Failing on main too (2, not counted against this PR)

guest_mmap_executable
guest_socket_options_multicast

Validated but not compared (7, not in the main baseline's validation pass)

agc_pipeline_library_workers
agc_pipeline_library_workers_burst
agc_pipeline_library_workers_invalid_above_hardware
agc_pipeline_library_workers_invalid_negative
agc_pipeline_library_workers_invalid_suffix
agc_pipeline_library_workers_invalid_text
agc_pipeline_library_workers_invalid_zero

Validation IDs already present on main (4 IDs in 420 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (420 tests): agc_compute_pipeline_replay, agc_driver, agc_driver_address_passes_draw_transitions, agc_driver_address_passes_targets_split, agc_driver_array_layer_target_cpu, agc_driver_bda_span_reads_draw_pipeline, agc_driver_bitfield_mask, agc_driver_buffer_atomic_csub, and 412 more
VUID-vkCmdCopyImage-srcImage-01548 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-vkCmdCopyImage-srcImage-01551 (13 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_cpu_write_after_memo, agc_driver_depth_plane_cpu_write_after_page_unwatch, agc_driver_depth_plane_cpu_write_after_partial_unwatch, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_partial_store_unimported, agc_driver_depth_plane_sampling, and 5 more
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target

Skipped (42)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_msaa_dcc_compared_import_2: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_4: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_msaa_dcc_compared_import_8: skipped, the device reports host imports that keep write protection, so the write-watch probe compares none of them
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop_main: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_fresh: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or plat

Reproduce

git fetch origin e231257e214cee585d8b587294e1a58c1cf58070 e8efeed58af48d0682ab2e0eab72cfaf996e4886
git checkout e8efeed58af48d0682ab2e0eab72cfaf996e4886 && git merge --no-ff e231257e214cee585d8b587294e1a58c1cf58070
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}

Tamalero · 2026-11-01T00:18:47Z

@MarcZX10 @adriadam10 @oneandonlydean the NVIDIA captures that exercise LTO. Summary: with link-time optimization forced on, the pool runs on NVIDIA too, and it keeps the link queue empty, but I measured no gain in frame time. Together with the RADV runs above, I have no game result that shows a performance benefit from this PR.

Setup. Ratchet & Clank: Rift Apart (PPSA01474) on an RTX 4090, NVIDIA 615.78.08, Linux, 32 hardware threads, 3840x2160, Performance mode. The route was driven by hand each time: title → New Game → opening → the first crowd scene, about 9–12 minutes (run 2: 24). NVIDIA's proprietary driver gets no background LTO since #3434, so both builds carry a local one-line switch that keeps it on (APS5_FORCE_PIPELINE_LTO, not part of this PR), plus the local [ltodiag] counters. This title does not reach gameplay on plain main, so both builds are the same composed stack: main bb04a4f + #9749 + #10906 + #10000 and local probes. The builds differ only in this PR's three commits, patch-identical to the pushed head e231257. The summary lines show 435e5fd4f798 (stack + switch + counters) and dc88856933ae (the same + this PR). There was one warm-up run, then base, PR, PR, base, with the same caches throughout.

The pool runs. Every run queued about 1,000 link jobs. On the base build's single worker the queue reached 30 and 4 jobs in the measured runs (242 in the warm-up, the only run with mostly cold caches). With the PR, 3 to 5 workers started and the queue never held more than 1 job.

Run	Build	Frame time, 120–480 s	Trace window 240–360 s: p50 / p90	GPU per present (window)
warm-up	base	–	520 / 2,768 ms	363 ms
1	base	726 ms	523 / 4,468 ms	412 ms
2	PR	177 ms	184 / 207 ms	88 ms
3	PR	657 ms	340 / 1,724 ms	230 ms
4	base	664 ms	285 / 986 ms	177 ms
5	base, LTO off	608 ms	287 / 1,212 ms	245 ms

Runs 3 and 4 differ by about 1%. Run 1 is a little slower, as the first run after a warm-up usually is.
Run 2 is an outlier I can't explain: about 4× faster with about 5× less GPU time per frame, steady for 24 minutes. Run 3, with the same build, did not repeat it. I don't count it for the PR.
Run 5, the same base build with LTO left off (NVIDIA's normal path), was about 8% faster than the LTO runs. On this title and GPU, the background relink costs more than it saves. That agrees with keeping fix(agc): skip the background pipeline relink on NVIDIA's proprietary driver #3434's default.
No device loss in any run.

Not done: a pair with cold caches, the only case here where a single worker falls far behind (242 queued).

If the gate needs a measured frame-time gain, I don't have one on either driver, and I'm fine with closing this PR.

Summary lines, digests and traces

The trace-window summary line of each run, in the order of the table (warm-up, 1–5):

{"commit":"435e5fd4f798","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":898,"ms":132619.301},"presents":83,"frame_ms":{"p50":519.620,"p90":2767.669,"p99":21039.814,"max":21039.814},"gpu_busy_ms_per_present":363.339,"worker_busy_pct":{"queue worker 0x0":99.076,"queue worker 0x20":52.278,"queue worker 0x28":0.007},"top_phases_ms_per_frame":{"Driver.Packet":2396.239,"Graphics.Draw":1192.566,"Graphics.Draw.draw_and_resource_release":1181.180,"Driver.Packet.draw":1084.771,"Driver.Draw":1084.420,"Driver.Draw.draw_enqueued":1027.746,"Resources.Complete":990.859,"Driver.Packet.memory_wait":786.729},"trace_dropped":false,"trace":"aps5-trace-435e5fd4f798-2.json.gz"}
{"commit":"435e5fd4f798","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":817,"ms":126786.334},"presents":82,"frame_ms":{"p50":523.088,"p90":4468.201,"p99":13476.628,"max":13476.628},"gpu_busy_ms_per_present":411.894,"worker_busy_pct":{"queue worker 0x0":99.216,"queue worker 0x20":62.541,"queue worker 0x28":0.006},"top_phases_ms_per_frame":{"Driver.Packet":2477.756,"Graphics.Draw":1146.185,"Graphics.Draw.draw_and_resource_release":1135.917,"Driver.Packet.draw":1045.585,"Driver.Draw":1045.167,"Driver.Draw.draw_enqueued":972.988,"Driver.Packet.memory_wait":927.102,"Resources.Complete":888.362},"trace_dropped":false,"trace":"aps5-trace-435e5fd4f798-2.json.gz"}
{"commit":"dc88856933ae","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":1442,"ms":120142.230},"presents":658,"frame_ms":{"p50":184.412,"p90":207.259,"p99":234.628,"max":959.252},"gpu_busy_ms_per_present":87.504,"worker_busy_pct":{"queue worker 0x0":98.427,"queue worker 0x20":61.614,"queue worker 0x28":0.045},"top_phases_ms_per_frame":{"Driver.Packet":291.602,"Driver.Packet.memory_wait":110.481,"Graphics.Draw":95.230,"Graphics.Draw.draw_and_resource_release":92.607,"Driver.Packet.dispatch":58.559,"Driver.Dispatch":57.699,"Recorder.Sync":54.899,"Driver.Packet.draw":48.632},"trace_dropped":false,"trace":"aps5-trace-dc88856933ae-2.json.gz"}
{"commit":"dc88856933ae","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":889,"ms":125132.970},"presents":101,"frame_ms":{"p50":339.894,"p90":1723.525,"p99":14360.233,"max":18166.432},"gpu_busy_ms_per_present":230.376,"worker_busy_pct":{"queue worker 0x0":96.079,"queue worker 0x20":53.504,"queue worker 0x28":0.008},"top_phases_ms_per_frame":{"Driver.Packet":1798.950,"Graphics.Draw":934.490,"Graphics.Draw.draw_and_resource_release":926.527,"Driver.Packet.draw":831.358,"Driver.Draw":831.015,"Driver.Draw.draw_enqueued":776.729,"Resources.Complete":759.587,"Driver.Packet.memory_wait":590.328},"trace_dropped":false,"trace":"aps5-trace-dc88856933ae-2.json.gz"}
{"commit":"435e5fd4f798","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":851,"ms":123505.033},"presents":123,"frame_ms":{"p50":285.158,"p90":986.153,"p99":13834.537,"max":18254.046},"gpu_busy_ms_per_present":176.786,"worker_busy_pct":{"queue worker 0x0":96.142,"queue worker 0x20":56.513,"queue worker 0x28":0.009},"top_phases_ms_per_frame":{"Driver.Packet":1494.274,"Graphics.Draw":744.104,"Graphics.Draw.draw_and_resource_release":737.315,"Driver.Packet.draw":700.893,"Driver.Draw":700.552,"Driver.Draw.draw_enqueued":647.229,"Resources.Complete":599.806,"Driver.Packet.memory_wait":510.021},"trace_dropped":false,"trace":"aps5-trace-435e5fd4f798-2.json.gz"}
{"commit":"435e5fd4f798","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":925,"ms":124088.785},"presents":114,"frame_ms":{"p50":287.116,"p90":1211.866,"p99":12828.497,"max":14952.710},"gpu_busy_ms_per_present":244.802,"worker_busy_pct":{"queue worker 0x0":98.175,"queue worker 0x28":0.008,"queue worker 0x20":51.116},"top_phases_ms_per_frame":{"Driver.Packet":1599.475,"Graphics.Draw":852.474,"Graphics.Draw.draw_and_resource_release":844.619,"Driver.Packet.draw":757.846,"Driver.Draw":757.516,"Driver.Draw.draw_enqueued":705.977,"Resources.Complete":696.308,"Driver.Packet.memory_wait":514.858},"trace_dropped":false,"trace":"aps5-trace-435e5fd4f798-2.json.gz"}

Traces and trace_digest.py output for every run except run 2 are attached. Run 2's trace is 35 MB, over the attachment limit, so its digest is below; the trace is available on request.

Run 2 digest

**seconds 240:120** from present 1442: 658 presents, frame p50 184.4 / p90 207.3 / p99 234.6 / max 959.3 ms, GPU busy 87.5 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 615.78.08, Linux 7.3.0-4-cachyos, 3840x2160, commit dc88856933ae, dropped: no

```json
{"commit":"dc88856933ae","dirty":false,"env":{"APS5_AJM_GAPLESS_END_TAIL":"1","APS5_AUDIOOUT2_COPY_AT_SET":"1","APS5_FORCE_PIPELINE_LTO":"1","APS5_PIPELINE_JOURNAL_ENTRIES":"16384","APS5_PIPELINE_JOURNAL_MAX_MIB":"4096","APS5_SKIP_REFUSED_DISPATCHES":"1","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_SECONDS":"240:120","APS5_TRACE_SUMMARY_EVERY":"60"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-4-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA01474","title":"Ratchet & Clank: Rift Apart","content_version":"01.005.003","resolution":"3840x2160","window":{"spec":"seconds 240:120","first_present":1442,"ms":120142.23},"presents":658,"frame_ms":{"p50":184.412,"p90":207.259,"p99":234.628,"max":959.252},"gpu_busy_ms_per_present":87.504,"worker_busy_pct":{"queue worker 0x0":98.427,"queue worker 0x20":61.614,"queue worker 0x28":0.045},"top_phases_ms_per_frame":{"Driver.Packet":291.602,"Driver.Packet.memory_wait":110.481,"Graphics.Draw":95.23,"Graphics.Draw.draw_and_resource_release":92.607,"Driver.Packet.dispatch":58.559,"Driver.Dispatch":57.699,"Recorder.Sync":54.899,"Driver.Packet.draw":48.632},"trace_dropped":false,"trace":"aps5-trace-dc88856933ae-2.json.gz"}

Slowest frames (largest slices inside each, inclusive)

present 1692: 959.3 ms at 44955 ms: Driver.WorkerWait (queue worker 0x28) 959.2, Driver.Packet (queue worker 0x0) 934.3, Driver.Packet (queue worker 0x20) 795.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 794.6
present 1695: 264.7 ms at 46329 ms: Driver.WorkerWait (queue worker 0x28) 264.6, Driver.Packet (queue worker 0x0) 242.3, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 191.6, Driver.Packet (queue worker 0x20) 187.5
present 1690: 258.9 ms at 44547 ms: Driver.WorkerWait (queue worker 0x28) 258.9, Driver.Packet (queue worker 0x0) 230.0, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 190.2, Driver.Packet (queue worker 0x20) 188.1
present 1688: 257.7 ms at 44041 ms: Driver.WorkerWait (queue worker 0x28) 257.6, Driver.Packet (queue worker 0x0) 231.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 174.4, Driver.Packet (queue worker 0x20) 172.7
present 1689: 249.1 ms at 44298 ms: Driver.WorkerWait (queue worker 0x28) 249.0, Driver.Packet (queue worker 0x0) 220.3, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 176.7, Driver.Packet (queue worker 0x20) 175.5

Threads (ms per frame)

thread	busy	wait	untraced	in slices < 20 us	largest
queue worker 0x0	149.9 (82 %)	6.8 (4 %)	25.9	157.7	Driver.Packet 154.5, Driver.EndSubmission 0.1
draw committer	77.7 (43 %)	0.3 (0 %)	104.6	38.2	Graphics.Draw 77.2, Recorder.Submit 0.3
queue worker 0x20	1.2 (1 %)	181.1 (99 %)	0.3	1.1	Driver.Packet 111.9, Driver.EndSubmission 0.4
queue worker 0x28	0.1 (0 %)	182.5 (100 %)	0.0	0.1	Driver.EndSubmission 0.1, Driver.Packet 0.0
title	0.0 (0 %)	0.0 (0 %)	182.6	0.9	Shader.ResolveGraphicsStagesAbi 0.0, Shader.ResolveGraphicsAbi 0.0
presenter	0.3 (0 %)	2.3 (1 %)	180.0	0.1	VideoOut.Flip 2.6
gpu	87.5 (48 %)	0.0 (0 %)	95.1	0.0	batch 87.5

Largest slices (ms per frame, inclusive, all threads): Driver.Packet 266.4, Driver.WorkerWait 254.7, Driver.WorkerWait.dequeue_order 135.3, Driver.WorkerWait.idle_and_completion_poll 119.3, Driver.Packet.memory_wait 110.5, batch 87.5, Graphics.Draw 77.2, Graphics.Draw.draw_and_resource_release 75.7, Recorder.Sync 54.9, Driver.Packet.dispatch 54.5

Longest waits

Driver.WorkerWait (queue worker 0x28) 959.2 ms at 44955 ms, present 1691
Driver.Packet.memory_wait (queue worker 0x20) 793.1 ms at 44959 ms, present 1692
Driver.WorkerWait (queue worker 0x28) 264.6 ms at 46329 ms, present 1694
Driver.WorkerWait (queue worker 0x28) 258.9 ms at 44547 ms, present 1689
Driver.WorkerWait (queue worker 0x28) 257.6 ms at 44040 ms, present 1687
Driver.WorkerWait (queue worker 0x28) 249.0 ms at 44298 ms, present 1688
Driver.WorkerWait (queue worker 0x28) 237.4 ms at 46594 ms, present 1695
Driver.WorkerWait (queue worker 0x28) 234.6 ms at 63576 ms, present 1787
Driver.WorkerWait (queue worker 0x28) 233.1 ms at 46096 ms, present 1693
Driver.WorkerWait (queue worker 0x28) 232.5 ms at 99647 ms, present 1987

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

batch 171152: 52.4 ms at 63585 ms, present 1788: GPU no ranges; recorded by Recorder.Sync (draw committer) 57.8
batch 148026: 51.8 ms at 23978 ms, present 1577: GPU no ranges; recorded by Recorder.Sync (draw committer) 57.5, Recorder.Reap (draw committer) 0.1
batch 152076: 51.8 ms at 30664 ms, present 1614: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 127.4, Driver.Packet.memory_wait (queue worker 0x20) 127.1, Driver.WorkerWait (queue worker 0x20) 72.8, Driver.WorkerWait.dequeue_order (queue worker 0x20) 61.0
batch 185167: 51.7 ms at 86785 ms, present 1916: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 127.8, Driver.Packet.memory_wait (queue worker 0x20) 127.4, Driver.WorkerWait (queue worker 0x20) 74.8, Driver.WorkerWait.dequeue_order (queue worker 0x20) 62.5
batch 141922: 51.7 ms at 14028 ms, present 1522: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 114.7, Driver.Packet.memory_wait (queue worker 0x20) 114.7, Driver.Dispatch (queue worker 0x20) 0.0, Driver.Packet.flush_between_packets (queue worker 0x20) 0.0


</details>

</details>

*Tested and written by Claude (AI assistant) on a contributor's machine.*

[nvidia-ratchet-5-lto-off-base.json.gz](https://github.com/user-attachments/files/34228901/nvidia-ratchet-5-lto-off-base.json.gz)
[nvidia-ratchet-1-base.json.gz](https://github.com/user-attachments/files/34228902/nvidia-ratchet-1-base.json.gz)
[nvidia-ratchet-4-base.json.gz](https://github.com/user-attachments/files/34228904/nvidia-ratchet-4-base.json.gz)
[nvidia-ratchet-0-warmup-base.json.gz](https://github.com/user-attachments/files/34228905/nvidia-ratchet-0-warmup-base.json.gz)
[nvidia-ratchet-3-pr.json.gz](https://github.com/user-attachments/files/34228906/nvidia-ratchet-3-pr.json.gz)

[nvidia-ratchet-0-warmup-base.digest.txt](https://github.com/user-attachments/files/34229460/nvidia-ratchet-0-warmup-base.digest.txt)
[nvidia-ratchet-1-base.digest.txt](https://github.com/user-attachments/files/34229461/nvidia-ratchet-1-base.digest.txt)
[nvidia-ratchet-3-pr.digest.txt](https://github.com/user-attachments/files/34229462/nvidia-ratchet-3-pr.digest.txt)
[nvidia-ratchet-4-base.digest.txt](https://github.com/user-attachments/files/34229463/nvidia-ratchet-4-base.digest.txt)
[nvidia-ratchet-5-lto-off-base.digest.txt](https://github.com/user-attachments/files/34229464/nvidia-ratchet-5-lto-off-base.digest.txt)

CMakeLists.txt: main moved the AGC test registrations into tests/cmake/ fragments; the PR's agc_pipeline_library_workers registrations and the APS5_PIPELINE_LTO_THREADS=1 on agc_pipeline_queue_failure now live in agc_pipeline_library_workers_tests.cmake and agc_pipeline_queue_failure_tests.cmake (agc_test_split resolve).

TechLevelZero · 2026-11-04T02:29:38Z

✅ NVIDIA GPU test: passed

PR head 7a2d632 merged onto main 1c8c0ef · tested 2026-11-04 02:29 UTC · 23 min


GPU	NVIDIA RTX 2000 Ada Generation, driver 595.99.02, Vulkan 1.4.329
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, `-DANYPS5_ENABLE_SPIRV_TOOLS=ON`)
Tests	✅ 2,791 passed · 6 failed (0 new, 3 also fail on main, 3 unstable on main) · 50 skipped (`ANYPS5_REQUIRE_VULKAN=1`)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with `VK_LAYER_KHRONOS_validation` and `ANYPS5_NO_SHADER_CACHE=1`; main ran the same way
Test selection	2,847 of 3,842 tests run: the ones this change can reach, chosen by `tools/select_tests.py` from main's build graph
Vulkan validation	✅ no new validation IDs vs main (958 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ `agc_driver_pipeline_journal` 120.1 s

Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)

agc_driver_pipeline_journal: failed in 4 of the last 10 main runs
agc_submit_progress: failed in 5 of the last 10 main runs
agc_submit_progress_no_timeline: failed in 4 of the last 10 main runs

Failing on main too (3, not counted against this PR)

agc_driver_image_table
guest_mmap_executable
guest_socket_options_multicast

Validated but not compared (7, not in the main baseline's validation pass)

agc_pipeline_library_workers
agc_pipeline_library_workers_burst
agc_pipeline_library_workers_invalid_above_hardware
agc_pipeline_library_workers_invalid_negative
agc_pipeline_library_workers_invalid_suffix
agc_pipeline_library_workers_invalid_text
agc_pipeline_library_workers_invalid_zero

Validation IDs already present on main (8 IDs in 531 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (531 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 523 more
VUID-vkCmdCopyImage-srcImage-01548 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (7 tests): agc_driver_depth_clear_hle, agc_driver_depth_plane_buffer_memory, agc_driver_depth_plane_partial_store, agc_driver_depth_plane_sampling, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume

Skipped (50)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_sample_order_grid_reference: skipped, the device does not keep depth drawn at custom sample locations at those locations, even after a transition that names them: the composite of uniform renders holds the guest's depth in 0 of 3005 sloped samples, so no path can be compared with it bit for bit
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_stencil_ref_export: skipped, the device lacks VK_EXT_shader_stencil_export
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_controls: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_cpuflip_in_order: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_dcc: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_deferred: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_eop: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
vide

Reproduce

git fetch origin 7a2d632b5d092a00f4c3795b61f864f387463f5d 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 7a2d632b5d092a00f4c3795b61f864f387463f5d
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 32 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 4 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}

TechLevelZero · 2026-11-04T03:39:40Z

✅ AMD GPU test: passed

PR head 7a2d632 merged onto main 1c8c0ef · tested 2026-11-04 03:39 UTC · 19 min


GPU	AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, `-DANYPS5_ENABLE_SPIRV_TOOLS=ON`)
Tests	✅ 2,778 passed · 8 failed (0 new, 7 also fail on main, 1 unstable on main) · 61 skipped (`ANYPS5_REQUIRE_VULKAN=1`)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with `VK_LAYER_KHRONOS_validation` and `ANYPS5_NO_SHADER_CACHE=1`; main ran the same way
Test selection	2,847 of 3,842 tests run: the ones this change can reach, chosen by `tools/select_tests.py` from main's build graph
Vulkan validation	✅ no new validation IDs vs main (958 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ `agc_driver_pipeline_journal` 120.1 s, ⚠️ `public_headers` 52.5 s, ⚠️ `agc_shader_sampler_limits` 38.1 s, ⚠️ `agc_driver_image_sample_unnormalized` 35.8 s, ⚠️ `agc_driver_lazy_resolved_abi` 31.6 s

Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)

agc_driver_pipeline_journal: failed in 8 of the last 10 main runs

Failing on main too (7, not counted against this PR)

agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
guest_socket_options_multicast

Validated but not compared (7, not in the main baseline's validation pass)

agc_pipeline_library_workers
agc_pipeline_library_workers_burst
agc_pipeline_library_workers_invalid_above_hardware
agc_pipeline_library_workers_invalid_negative
agc_pipeline_library_workers_invalid_suffix
agc_pipeline_library_workers_invalid_text
agc_pipeline_library_workers_invalid_zero

Validation IDs already present on main (8 IDs in 531 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (531 tests): agc_driver, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 523 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume

Skipped (61)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_storage_pending_key_refresh: skipped, refreshes over a pending key store are not testable here
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_net_ctl: skipped, the host has no IPv4 address
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan d

Reproduce

git fetch origin 7a2d632b5d092a00f4c3795b61f864f387463f5d 1c8c0efc8c41314402e6966da5bef6ea3d639d10
git checkout 1c8c0efc8c41314402e6966da5bef6ea3d639d10 && git merge --no-ff 7a2d632b5d092a00f4c3795b61f864f387463f5d
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}

Tamalero · 2026-11-04T23:23:41Z

@oneandonlydean thanks for merging main in. The CI bots show NVIDIA and AMD passing with nothing new. Before I spend more time on this one: do you want it to go ahead?

My own measurements show no frame-time gain on either driver. On RADV (High on Life) the gain is not measurable. On NVIDIA with LTO forced on (Ratchet), there is none, and leaving LTO off was about 8% faster. Both are in the comments above. adriadam10's and MarcZX10's change requests are still open on that basis.

If a bounded worker pool is still useful, for start-up latency or for reasons I haven't measured, I'm happy to address the reviews and add whatever capture you want. Otherwise I'd close it.

Tested and written by Claude (AI assistant) on a contributor's machine.

oneandonlydean · 2026-11-04T23:41:34Z

Hi Tamalero, thank you for measuring this so carefully. Your own numbers show no frame-time gain (none on RADV or NVIDIA, and LTO off was about 8% faster on NVIDIA), and start-up latency is the only case left unmeasured, so we are closing this rather than carrying it. If you measure a start-up gain later, please reopen with those numbers and we will take another look.

github-actions Bot added perf Performance change perf numbers missing perf pull request without a before and after summary line labels Oct 28, 2026

MarcZX10 suggested changes Oct 28, 2026

View reviewed changes

adriadam10 self-assigned this Oct 28, 2026

adriadam10 requested changes Oct 28, 2026

View reviewed changes

github-actions Bot added build docs agc libSceAgc* labels Oct 28, 2026

adriadam10 approved these changes Oct 28, 2026

View reviewed changes

github-actions Bot removed the perf numbers missing perf pull request without a before and after summary line label Oct 28, 2026

MarcZX10 suggested changes Oct 29, 2026

View reviewed changes

Tamalero force-pushed the perf/pipeline-lto-workers branch from e7af1f2 to 07369ec Compare October 29, 2026 07:29

MarcZX10 suggested changes Oct 29, 2026

View reviewed changes

Tamalero force-pushed the perf/pipeline-lto-workers branch from 07369ec to 2e6cd4b Compare October 30, 2026 02:42

MarcZX10 suggested changes Oct 30, 2026

View reviewed changes

adriadam10 requested changes Oct 30, 2026

View reviewed changes

Tamalero and others added 3 commits October 30, 2026 11:11

perf(agc): link pipelines with link-time optimization on half the har…

1831b61

…dware threads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(agc): reject an APS5_PIPELINE_LTO_THREADS that is not 1 through t…

c151053

…he hardware thread count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

perf(agc): start a pipeline worker for a burst behind a woken idle wo…

e231257

…rker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

oneandonlydean self-assigned this Oct 30, 2026

Tamalero force-pushed the perf/pipeline-lto-workers branch from 2e6cd4b to e231257 Compare October 30, 2026 22:08

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 31, 2026

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Nov 3, 2026

oneandonlydean closed this Nov 4, 2026

Conversation

Tamalero commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tested

Checklist

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Uh oh!

Tamalero commented Oct 28, 2026

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oneandonlydean commented Oct 28, 2026

Uh oh!

adriadam10 left a comment

Choose a reason for hiding this comment

Uh oh!

Tamalero commented Oct 28, 2026

Uh oh!

Tamalero commented Oct 28, 2026

Uh oh!

Tamalero commented Oct 28, 2026

Uh oh!

Tamalero commented Oct 29, 2026

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Re-checked — changes requested

What I ran

Results

Uh oh!

Tamalero commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Re-check — changes requested

Uh oh!

Tamalero commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TechLevelZero commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ NVIDIA GPU test: passed

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Re-checked — new head, changes requested

What I ran

Results

Findings

Blocking

Not verified

To get this to a good state

Uh oh!

Tamalero commented Oct 30, 2026

Uh oh!

MarcZX10 commented Oct 30, 2026

Re-checked — performance evidence pending

Uh oh!

adriadam10 left a comment

Choose a reason for hiding this comment

Uh oh!

Tamalero commented Oct 30, 2026

Uh oh!

Tamalero commented Oct 30, 2026

Uh oh!

TechLevelZero commented Oct 31, 2026

✅ NVIDIA GPU test: passed

Uh oh!

Tamalero commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tamalero commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Tamalero commented Oct 29, 2026 •

edited

Loading

Tamalero commented Oct 30, 2026 •

edited

Loading

TechLevelZero commented Oct 30, 2026 •

edited

Loading

Tamalero commented Nov 1, 2026 •

edited

Loading