Skip to content

perf(agc): resolve DISPATCH_INDIRECT thread dimensions on the GPU - #8066

Merged
adriadam10 merged 2 commits into
boykopovar:mainfrom
HQasper:perf/gpu-thread-dimensions
Oct 20, 2026
Merged

adriadam10 merged 2 commits into
boykopovar:mainfrom
HQasper:perf/gpu-thread-dimensions

Conversation

@HQasper

@HQasper HQasper commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What

A DISPATCH_INDIRECT whose initiator sets USE_THREAD_DIMENSIONS (bit 5) takes thread counts from its arguments, not group counts. indirectArgumentsPath sent every such dispatch to the CPU (IndirectThreadDimensions). The read has to wait for earlier GPU work that may store the arguments: since #6786 it finishes the leased work first (FinishLeaseWork), under the GPU mutex. GTA V's async compute queues issue these dispatches all the time. In one run (title, loading and 200 s of Story) the driver read 28839 of them on the CPU, and those reads took 71 s in total across the compute queues.

The argument kernel (DispatchArguments.comp) now does the conversion on the GPU. Given the group size per axis (COMPUTE_NUM_THREAD_X/Y/Z), it writes ceil(threads / size) as the group counts. It also writes the thread counts in words 3 to 5 of its output, and recordDispatch copies them into the program's ShaderData thread limit (dispatchThreadLimit) before the dispatch. The dispatch then stays on the GPU path like a group-count indirect dispatch.

Other changes:

  • The program. Driver::dispatch compiles these dispatches as partial-group programs with a placeholder limit (~0u on each axis). It sets the push constant size of the request to 0, so the ShaderData words land in a storage buffer the GPU can store into.
  • The template. The data words of the ShaderData buffer are forgotten after the store (forgetShaderDataWords), so the next refresh writes them all again.
  • Data buffers. Every data buffer now has TRANSFER_DST, not only the refreshable ones.
  • Moves. A dispatch move that would replace the ShaderData buffer is refused for these dispatches, since the overlay's copy of the words cannot take the store.
  • The CPU fallback. decideIndirect's CPU fallback (pending image results, pending label or copied writer, no host import, device-memory LDS) converts the counts to groups itself and stores the limit with vkCmdUpdateBuffer.
  • The switch. APS5_NO_GPU_THREAD_DIMENSIONS=1 restores the CPU read.

Tested

Windows 11 Pro 26300, NVIDIA GeForce RTX 5090 (617.42), MinGW-w64 Release build of this branch on main 7a015c0.

  • New test agc_driver_indirect_thread_dimensions. It submits four shapes on a compute queue: 100x1x1 threads in groups of 64x1x1, 5x3x2 in 4x2x2, 64x2x1 in 32x1x1, and 0x7x1 in 8x1x1. Each runs once with the arguments stored by a GPU producer in the same submission and once with arguments the CPU wrote. A counter checks that exactly the requested threads ran. The test also takes decideIndirect's CPU fallback, behind a GPU producer and a pending label over the arguments (cpuReason 2). Without the limit store the first shape runs 128 threads instead of 100.
  • agc_driver_indirect_thread_dimensions_cpu runs the same shapes with APS5_NO_GPU_THREAD_DIMENSIONS=1, with arguments the CPU wrote.
  • Full ctest -j 24: 1683 of 1685 pass. agc_driver_pixel_lds and agc_driver_pass_input_hazard fail on plain main 7a015c0 too on this RTX 5090.
  • With ANYPS5_GPU=AMD the indirect and dispatch-move tests (agc_driver_indirect_thread_dimensions, its _cpu variant, indirect_arguments_*, gpu_indirect_arguments_*, indirect_dispatch_limit, dispatch_move*, lds_beyond_device_limit, oversized_dispatch, pipelined_dispatch*) also pass on the AMD Radeon(TM) Graphics iGPU of a Ryzen 9 9950X3D (64-lane subgroups).
  • Vulkan validation on the RTX 5090 (Vulkan SDK 1.4.363.0): ctest --test-dir build -R agc_driver_indirect -V with ANYPS5_REQUIRE_VULKAN=1, ANYPS5_NO_SHADER_CACHE=1 and VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, on main 7a015c0 and on this branch (84a6fb0). Both logs show the loader inserting VK_LAYER_KHRONOS_validation, and neither says it was not found. main passes its 4 tests (agc_driver_indirect_arguments_pending_producer, agc_driver_indirect_arguments_written_ahead, agc_driver_indirect_dispatch_limit, agc_driver_indirect_patched_stage); this branch passes those and the two new ones. Both logs hold the same four VUID- lines, all from the existing agc_driver_indirect_arguments_pending_producer (2x VUID-vkCmdWaitEvents-srcStageMask-01158, 2x VUID-vkSetEvent-event-09543). The two new tests log none.

GTA V (PPSA04264) needs #6488 to reach Story and #7603 (merged since) not to hang there, so both builds carry them on this branch's base. In the summary lines, 7b18631f34aa is main 7a015c0 + #6488 (a48c5a7) + #7603 (e5b4bd7), and 72a70f6d87ef is the same build + this branch (84a6fb0). Both are Release builds on one Windows 11 Pro 26300 machine with a Ryzen 9 9950X3D and the RTX 5090 (617.42). The scene is the Story prologue, resumed from the same save before every run, at 3840x2160 with default settings and replayed pad input. Only APS5_TRACE_FRAMES=7500:300 is set besides the pad input file. Caches are warm, and both game folders hold the same shader and pipeline cache files. Base and change alternate in one session, three pairs:

Pair Build p50 p90 p99 Mean frame GPU busy per present
1 base 83.4 ms 84.7 ms 87.3 ms 83.5 ms 58.8 ms
1 change 81.1 ms 83.0 ms 85.6 ms 81.4 ms 59.0 ms
2 base 84.6 ms 85.9 ms 87.9 ms 84.8 ms 58.6 ms
2 change 81.3 ms 82.7 ms 85.4 ms 81.5 ms 59.1 ms
3 base 84.6 ms 85.9 ms 88.2 ms 84.7 ms 58.8 ms
3 change 82.0 ms 83.6 ms 85.6 ms 82.2 ms 59.5 ms

The median frame time drops by 2.7 %, 3.9 % and 3.1 % (84.2 to 81.5 ms on average, 3.3 %), and every change run is faster than every base run. GPU busy time per present stays about the same (58.8 against 59.2 ms on average). In the digests of pair 2 below, the draw committer's wait falls from 29.1 to 2.8 ms per frame, and GpuMutex.wait.draw (27.8 ms per frame in the base) leaves the largest slices.

A reviewer's run on a Linux laptop (RTX 5060 Laptop GPU, NVIDIA 615.71.09, GTA V PPSA04263 01.010.002, frames 3600:700, main d14a186 against the same + this branch, A B A B) found no difference (p50 -0.3 %). Whether that window had CPU thread-dimension reads was not checked, so the gain above holds for the RTX 5090 machine and scene only.

In separate runs of the same scene with APS5_PROFILE_DRAW=1, the [indirect] line reads cpu-side 30391 (thread dimensions 30391, ...); CPU argument reads took 66.6 s on the base and cpu-side 0 (thread dimensions 0, ...); CPU argument reads took 0.0 s with the change.

Every trace reports dropped events: the event buffer of queue worker 0x0 fills after about 8 s of the 25 s window and that of the draw committer after about 23 s, so the digests' per-thread tables cover only that part. totals_full is false everywhere, so the summary numbers are complete.

An earlier version of this description compared the change with APS5_NO_GPU_THREAD_DIMENSIONS=1 in one build on an older main with #6786 (6 %). Its summary lines all carried one commit and it also set APS5_TRACE_DIR, so this comparison of two builds replaces it.

Summary lines, in run order (base and change of pair 1, then pairs 2 and 3):

{"commit":"7b18631f34aa","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":25044.981},"presents":300,"frame_ms":{"p50":83.396,"p90":84.716,"p99":87.254,"max":87.593},"gpu_busy_ms_per_present":58.784,"worker_busy_pct":{"queue worker 0x0":3260.007,"queue worker 0x38":45.801,"queue worker 0x20":87.867,"queue worker 0x30":54.591,"queue worker 0x28":52.606},"top_phases_ms_per_frame":{"Driver.EndSubmission":2641.579,"Driver.Packet":279.223,"Driver.Packet.memory_wait":157.690,"Driver.Packet.draw":63.406,"Driver.Draw":63.078,"Driver.Packet.dispatch":50.642,"Driver.Draw.draw_enqueued":38.213,"Graphics.Draw":33.843},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.429,"events_lost":515260,"totals_full":false},"draw committer":{"events_full_after_s":24.440,"events_lost":6518,"totals_full":false}},"trace":"aps5-trace-7b18631f34aa-6976.json.gz"}
{"commit":"72a70f6d87ef","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":24412.405},"presents":300,"frame_ms":{"p50":81.126,"p90":83.000,"p99":85.552,"max":86.734},"gpu_busy_ms_per_present":59.033,"worker_busy_pct":{"queue worker 0x0":3474.282,"queue worker 0x38":44.650,"queue worker 0x28":59.382,"queue worker 0x20":84.807,"queue worker 0x30":48.179},"top_phases_ms_per_frame":{"Driver.EndSubmission":2748.638,"Driver.Packet":269.574,"Driver.Packet.memory_wait":152.638,"Driver.Packet.dispatch":54.212,"Graphics.Draw":53.091,"Driver.Dispatch":52.066,"Graphics.Draw.draw_and_resource_release":47.456,"Driver.Packet.draw":36.631},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.047,"events_lost":522055,"totals_full":false},"draw committer":{"events_full_after_s":23.899,"events_lost":5534,"totals_full":false}},"trace":"aps5-trace-72a70f6d87ef-33088.json.gz"}
{"commit":"7b18631f34aa","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":25427.010},"presents":300,"frame_ms":{"p50":84.639,"p90":85.939,"p99":87.886,"max":90.503},"gpu_busy_ms_per_present":58.649,"worker_busy_pct":{"queue worker 0x0":3259.212,"queue worker 0x38":51.112,"queue worker 0x30":41.957,"queue worker 0x20":81.680,"queue worker 0x28":63.741},"top_phases_ms_per_frame":{"Driver.EndSubmission":2681.263,"Driver.Packet":281.364,"Driver.Packet.memory_wait":158.059,"Driver.Packet.draw":64.121,"Driver.Draw":63.798,"Driver.Packet.dispatch":51.390,"Driver.Draw.draw_enqueued":38.590,"Graphics.Draw":34.733},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.396,"events_lost":535277,"totals_full":false},"draw committer":{"events_full_after_s":23.631,"events_lost":19898,"totals_full":false}},"trace":"aps5-trace-7b18631f34aa-17656.json.gz"}
{"commit":"72a70f6d87ef","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":24436.965},"presents":300,"frame_ms":{"p50":81.301,"p90":82.676,"p99":85.448,"max":85.534},"gpu_busy_ms_per_present":59.143,"worker_busy_pct":{"queue worker 0x0":3478.718,"queue worker 0x38":48.398,"queue worker 0x30":50.894,"queue worker 0x20":88.089,"queue worker 0x28":54.915},"top_phases_ms_per_frame":{"Driver.EndSubmission":2755.037,"Driver.Packet":274.103,"Driver.Packet.memory_wait":157.061,"Driver.Packet.dispatch":54.144,"Graphics.Draw":53.161,"Driver.Dispatch":51.988,"Graphics.Draw.draw_and_resource_release":47.690,"Driver.Packet.draw":36.649},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.190,"events_lost":524642,"totals_full":false},"draw committer":{"events_full_after_s":23.435,"events_lost":11125,"totals_full":false}},"trace":"aps5-trace-72a70f6d87ef-27876.json.gz"}
{"commit":"7b18631f34aa","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":25421.039},"presents":300,"frame_ms":{"p50":84.617,"p90":85.908,"p99":88.180,"max":89.627},"gpu_busy_ms_per_present":58.837,"worker_busy_pct":{"queue worker 0x0":3256.001,"queue worker 0x28":57.131,"queue worker 0x30":45.872,"queue worker 0x20":87.092,"queue worker 0x38":47.211},"top_phases_ms_per_frame":{"Driver.EndSubmission":2677.855,"Driver.Packet":280.366,"Driver.Packet.memory_wait":157.081,"Driver.Packet.draw":64.243,"Driver.Draw":63.919,"Driver.Packet.dispatch":51.337,"Driver.Draw.draw_enqueued":38.604,"Graphics.Draw":34.714},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.362,"events_lost":540270,"totals_full":false},"draw committer":{"events_full_after_s":23.514,"events_lost":21130,"totals_full":false}},"trace":"aps5-trace-7b18631f34aa-7248.json.gz"}
{"commit":"72a70f6d87ef","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":24647.342},"presents":300,"frame_ms":{"p50":82.014,"p90":83.568,"p99":85.617,"max":86.009},"gpu_busy_ms_per_present":59.503,"worker_busy_pct":{"queue worker 0x0":3479.772,"queue worker 0x38":46.340,"queue worker 0x28":59.045,"queue worker 0x20":86.177,"queue worker 0x30":46.840},"top_phases_ms_per_frame":{"Driver.EndSubmission":2779.638,"Driver.Packet":273.241,"Driver.Packet.memory_wait":155.311,"Driver.Packet.dispatch":54.427,"Graphics.Draw":53.995,"Driver.Dispatch":52.188,"Graphics.Draw.draw_and_resource_release":48.147,"Driver.Packet.draw":37.013},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.114,"events_lost":532240,"totals_full":false},"draw committer":{"events_full_after_s":22.900,"events_lost":20038,"totals_full":false}},"trace":"aps5-trace-72a70f6d87ef-22380.json.gz"}

The traces are attached.
aps5-trace-7b18631f34aa-6976.json.gz
aps5-trace-7b18631f34aa-7248.json.gz
aps5-trace-7b18631f34aa-17656.json.gz
aps5-trace-72a70f6d87ef-22380.json.gz
aps5-trace-72a70f6d87ef-27876.json.gz
aps5-trace-72a70f6d87ef-33088.json.gz

Digests of pair 2, from python tools/trace_digest.py <trace>:

base, aps5-trace-7b18631f34aa-17656.json.gz

frames 7500:300 from present 7500: 300 presents, frame p50 84.6 / p90 85.9 / p99 87.9 / max 90.5 ms, GPU busy 58.6 ms per present
NVIDIA GeForce RTX 5090, NVIDIA 617.42, Windows 10.0.26300, 3840x2160, commit 7b18631f34aa, dropped: yes

{"commit":"7b18631f34aa","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":25427.01},"presents":300,"frame_ms":{"p50":84.639,"p90":85.939,"p99":87.886,"max":90.503},"gpu_busy_ms_per_present":58.649,"worker_busy_pct":{"queue worker 0x0":3259.212,"queue worker 0x38":51.112,"queue worker 0x30":41.957,"queue worker 0x20":81.68,"queue worker 0x28":63.741},"top_phases_ms_per_frame":{"Driver.EndSubmission":2681.263,"Driver.Packet":281.364,"Driver.Packet.memory_wait":158.059,"Driver.Packet.draw":64.121,"Driver.Draw":63.798,"Driver.Packet.dispatch":51.39,"Driver.Draw.draw_enqueued":38.59,"Graphics.Draw":34.733},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.396,"events_lost":535277,"totals_full":false},"draw committer":{"events_full_after_s":23.631,"events_lost":19898,"totals_full":false}},"trace":"aps5-trace-7b18631f34aa-17656.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 7585: 90.5 ms at 7217 ms: Driver.EndSubmission (queue worker 0x0) 2842.5, Driver.WorkerWait (queue worker 0x30) 90.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x30) 90.5, Driver.WorkerWait (queue worker 0x28) 90.5
  • present 7580: 88.5 ms at 6783 ms: Driver.EndSubmission (queue worker 0x0) 2833.7, Driver.WorkerWait (queue worker 0x38) 88.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x38) 88.5, Driver.Packet (queue worker 0x30) 88.4
  • present 7587: 87.9 ms at 7393 ms: Driver.EndSubmission (queue worker 0x0) 2824.6, Driver.WorkerWait (queue worker 0x38) 87.9, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x38) 87.9, Driver.WorkerWait (queue worker 0x30) 87.9
  • present 7720: 87.9 ms at 18656 ms: Driver.Packet (queue worker 0x20) 87.2, Driver.Packet.memory_wait (queue worker 0x20) 87.2, Driver.Packet (queue worker 0x30) 87.1, Driver.Packet.memory_wait (queue worker 0x30) 87.1
  • present 7582: 87.7 ms at 6958 ms: Driver.EndSubmission (queue worker 0x0) 2762.0, Driver.WorkerWait (queue worker 0x38) 87.7, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x38) 87.7, Driver.Packet (queue worker 0x28) 87.4

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
queue worker 0x0 27.5 (32 %) 0.4 (0 %) 56.8 26.9 Driver.EndSubmission 27.7, Driver.Packet 0.2
draw committer 23.5 (28 %) 29.1 (34 %) 32.1 23.3 Graphics.Draw 23.4, Recorder.Submit 0.7
presenter 0.8 (1 %) 0.0 (0 %) 83.9 0.1 VideoOut.Flip 0.8, Driver.Present 0.0
queue worker 0x38 0.8 (1 %) 83.9 (99 %) 0.1 0.1 Driver.Packet 43.3, Driver.EndSubmission 0.0
queue worker 0x30 0.8 (1 %) 83.1 (98 %) 0.9 0.1 Driver.Packet 35.5, Driver.EndSubmission 0.0
queue worker 0x20 26.1 (31 %) 58.2 (69 %) 0.4 5.4 Driver.Packet 69.0, Driver.EndSubmission 0.0
queue worker 0x28 11.6 (14 %) 72.3 (85 %) 0.8 2.3 Driver.Packet 53.9, Driver.EndSubmission 0.0
gpu 58.6 (69 %) 0.0 (0 %) 26.1 0.0 batch 58.6

Largest slices (ms per frame, inclusive, all threads): Driver.EndSubmission 880.8, Driver.Packet 223.8, Driver.Packet.memory_wait 157.2, Driver.WorkerWait 135.0, Driver.WorkerWait.idle_and_completion_poll 135.0, batch 58.6, Driver.Packet.dispatch 46.9, GpuMutex.wait.draw 27.8, Graphics.Draw 23.4, Graphics.Draw.draw_and_resource_release 19.0

Longest waits

  • Driver.WorkerWait (queue worker 0x30) 971.1 ms at 15650 ms, present 7684
  • Driver.WorkerWait (queue worker 0x30) 969.0 ms at 13286 ms, present 7656
  • Driver.WorkerWait (queue worker 0x30) 803.1 ms at 4712 ms, present 7555
  • Driver.WorkerWait (queue worker 0x30) 802.8 ms at 17854 ms, present 7710
  • Driver.WorkerWait (queue worker 0x30) 713.5 ms at 18792 ms, present 7721
  • Driver.WorkerWait (queue worker 0x38) 712.0 ms at 23784 ms, present 7780
  • Driver.WorkerWait (queue worker 0x30) 631.3 ms at 11172 ms, present 7631
  • Driver.WorkerWait (queue worker 0x30) 627.2 ms at 12358 ms, present 7645
  • Driver.WorkerWait (queue worker 0x30) 560.7 ms at 7007 ms, present 7582
  • Driver.WorkerWait (queue worker 0x38) 551.1 ms at 471 ms, present 7505

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 8771: 17.4 ms at 6962 ms, present 7582: GPU no ranges; recorded by GpuMutex.wait.draw (draw committer) 20.4, Graphics.Draw (draw committer) 7.0, Graphics.Draw.draw_and_resource_release (draw committer) 5.4, Recorder.Submit (draw committer) 0.2
  • batch 9093: 17.2 ms at 7222 ms, present 7585: GPU no ranges; recorded by GpuMutex.wait.draw (draw committer) 20.4, Graphics.Draw (draw committer) 6.8, Graphics.Draw.draw_and_resource_release (draw committer) 5.4, Recorder.Submit (draw committer) 0.2
  • batch 8557: 17.2 ms at 6787 ms, present 7580: GPU no ranges; recorded by GpuMutex.wait.draw (draw committer) 20.4, Graphics.Draw (draw committer) 6.5, Graphics.Draw.draw_and_resource_release (draw committer) 5.0, Recorder.Submit (draw committer) 0.2
  • batch 22226: 17.1 ms at 17640 ms, present 7708: GPU no ranges; recorded by GpuMutex.wait.draw (draw committer) 20.5, Graphics.Draw (draw committer) 6.4, Graphics.Draw.draw_and_resource_release (draw committer) 5.0, Recorder.Submit (draw committer) 0.2
  • batch 17949: 17.0 ms at 14258 ms, present 7668: GPU no ranges; recorded by GpuMutex.wait.draw (draw committer) 20.4, Graphics.Draw (draw committer) 6.4, Graphics.Draw.draw_and_resource_release (draw committer) 5.0, Graphics.Draw.validate (draw committer) 0.2
change, aps5-trace-72a70f6d87ef-27876.json.gz

frames 7500:300 from present 7500: 300 presents, frame p50 81.3 / p90 82.7 / p99 85.4 / max 85.5 ms, GPU busy 59.1 ms per present
NVIDIA GeForce RTX 5090, NVIDIA 617.42, Windows 10.0.26300, 3840x2160, commit 72a70f6d87ef, dropped: yes

{"commit":"72a70f6d87ef","dirty":false,"env":{"APS5_PAD_INPUT_FILE":"<path>","APS5_TRACE_FRAMES":"7500:300"},"gpu":"NVIDIA GeForce RTX 5090","driver":"NVIDIA 617.42","os":"Windows 10.0.26300","resolution":"3840x2160","window":{"spec":"frames 7500:300","first_present":7500,"ms":24436.965},"presents":300,"frame_ms":{"p50":81.301,"p90":82.676,"p99":85.448,"max":85.534},"gpu_busy_ms_per_present":59.143,"worker_busy_pct":{"queue worker 0x0":3478.718,"queue worker 0x38":48.398,"queue worker 0x30":50.894,"queue worker 0x20":88.089,"queue worker 0x28":54.915},"top_phases_ms_per_frame":{"Driver.EndSubmission":2755.037,"Driver.Packet":274.103,"Driver.Packet.memory_wait":157.061,"Driver.Packet.dispatch":54.144,"Graphics.Draw":53.161,"Driver.Dispatch":51.988,"Graphics.Draw.draw_and_resource_release":47.69,"Driver.Packet.draw":36.649},"trace_dropped":true,"trace_dropped_threads":{"queue worker 0x0":{"events_full_after_s":8.19,"events_lost":524642,"totals_full":false},"draw committer":{"events_full_after_s":23.435,"events_lost":11125,"totals_full":false}},"trace":"aps5-trace-72a70f6d87ef-27876.json.gz"}

Slowest frames (largest slices inside each, inclusive)

  • present 7599: 85.5 ms at 8059 ms: Driver.EndSubmission (queue worker 0x0) 2913.7, Driver.WorkerWait (queue worker 0x28) 85.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 85.5, Driver.Packet (queue worker 0x20) 85.1
  • present 7598: 85.5 ms at 7973 ms: Driver.EndSubmission (queue worker 0x0) 2896.0, Driver.WorkerWait (queue worker 0x28) 85.5, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 85.5, Driver.Packet (queue worker 0x20) 85.4
  • present 7600: 85.4 ms at 8144 ms: Driver.WorkerWait (queue worker 0x28) 85.4, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x28) 85.4, Driver.Packet (queue worker 0x20) 84.9, Driver.Packet (queue worker 0x38) 84.4
  • present 7604: 85.4 ms at 8481 ms: Driver.Packet (queue worker 0x20) 84.5, Driver.Packet.memory_wait (queue worker 0x20) 84.5, Driver.Packet (queue worker 0x38) 84.4, Driver.Packet.memory_wait (queue worker 0x38) 84.4
  • present 7606: 85.0 ms at 8650 ms: Driver.WorkerWait (queue worker 0x38) 85.0, Driver.WorkerWait.idle_and_completion_poll (queue worker 0x38) 85.0, Driver.Packet (queue worker 0x20) 84.6, Driver.Packet (queue worker 0x28) 84.1

Threads (ms per frame)

thread busy wait untraced in slices < 20 us largest
queue worker 0x0 24.6 (30 %) 2.7 (3 %) 54.2 27.5 Driver.EndSubmission 27.1, Driver.Packet 0.2
draw committer 42.1 (52 %) 2.8 (3 %) 36.6 24.0 Graphics.Draw 42.0, Recorder.Submit 0.8
presenter 0.8 (1 %) 0.0 (0 %) 80.6 0.1 VideoOut.Flip 0.8, Driver.Present 0.0
queue worker 0x38 0.6 (1 %) 78.5 (96 %) 2.3 0.1 Driver.Packet 39.4, Driver.EndSubmission 0.0
queue worker 0x30 1.9 (2 %) 79.2 (97 %) 0.3 0.3 Driver.Packet 41.4, Driver.EndSubmission 0.0
queue worker 0x20 27.7 (34 %) 53.2 (65 %) 0.5 6.3 Driver.Packet 71.4, Driver.EndSubmission 0.0
queue worker 0x28 6.7 (8 %) 73.9 (91 %) 0.9 1.4 Driver.Packet 44.6, Driver.EndSubmission 0.0
gpu 59.1 (73 %) 0.0 (0 %) 22.3 0.0 batch 59.1

Largest slices (ms per frame, inclusive, all threads): Driver.EndSubmission 921.9, Driver.Packet 218.4, Driver.Packet.memory_wait 155.7, Driver.WorkerWait 124.9, Driver.WorkerWait.idle_and_completion_poll 124.9, batch 59.1, Driver.Packet.dispatch 45.7, Driver.Dispatch 43.8, Graphics.Draw 42.0, Graphics.Draw.draw_and_resource_release 37.4

Longest waits

  • Driver.WorkerWait (queue worker 0x30) 774.7 ms at 12451 ms, present 7652
  • Driver.WorkerWait (queue worker 0x28) 693.5 ms at 23176 ms, present 7784
  • Driver.WorkerWait (queue worker 0x38) 605.7 ms at 14730 ms, present 7680
  • Driver.WorkerWait (queue worker 0x38) 604.8 ms at 5175 ms, present 7563
  • Driver.WorkerWait (queue worker 0x38) 528.4 ms at 18792 ms, present 7730
  • Driver.WorkerWait (queue worker 0x28) 526.7 ms at 20577 ms, present 7752
  • Driver.WorkerWait (queue worker 0x28) 458.5 ms at 7856 ms, present 7596
  • Driver.WorkerWait (queue worker 0x30) 449.8 ms at 23501 ms, present 7788
  • Driver.WorkerWait (queue worker 0x28) 446.4 ms at 9759 ms, present 7619
  • Driver.WorkerWait (queue worker 0x38) 446.1 ms at 2890 ms, present 7535

Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)

  • batch 9892: 17.7 ms at 8401 ms, present 7603: GPU no ranges; recorded by Graphics.Draw (draw committer) 6.8, Graphics.Draw.draw_and_resource_release (draw committer) 5.2, GpuMutex.wait.draw (draw committer) 1.5, Recorder.Submit (draw committer) 0.2
  • batch 9410: 17.6 ms at 7977 ms, present 7598: GPU no ranges; recorded by Graphics.Draw (draw committer) 7.0, Graphics.Draw.draw_and_resource_release (draw committer) 5.5, GpuMutex.wait.draw (draw committer) 2.3, Recorder.Submit (draw committer) 0.2
  • batch 20052: 17.5 ms at 17047 ms, present 7709: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 2.6, Driver.Packet.dispatch (queue worker 0x20) 2.6, Driver.Dispatch (queue worker 0x20) 2.2, GpuMutex.wait.dispatch (queue worker 0x20) 1.7
  • batch 14782: 17.3 ms at 12574 ms, present 7654: GPU no ranges; recorded by Graphics.Draw (draw committer) 7.0, Graphics.Draw.draw_and_resource_release (draw committer) 5.5, GpuMutex.wait.draw (draw committer) 2.0, Recorder.Submit (draw committer) 0.3
  • batch 9507: 17.3 ms at 8063 ms, present 7599: GPU no ranges; recorded by Driver.Packet (queue worker 0x20) 2.5, Driver.Packet.dispatch (queue worker 0x20) 2.5, Driver.Dispatch (queue worker 0x20) 2.0, GpuMutex.wait.dispatch (queue worker 0x20) 1.6

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 5090 (driver 617.42) and AMD Radeon(TM) Graphics iGPU of a Ryzen 9 9950X3D (AMD driver 2.0.353), Windows 11 Pro 26300
  • AI-assisted: yes

An indirect dispatch with USE_THREAD_DIMENSIONS read its thread counts on the
CPU, which waits for the recorded work that stores them. The argument kernel now
turns them into group counts on the GPU and copies the counts into the
ShaderData thread limit, so the dispatch stays on the GPU path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added perf Performance change perf numbers missing perf pull request without a before and after summary line labels Oct 19, 2026
@github-actions

Copy link
Copy Markdown

This pull request is labeled perf, so it needs the summary line of a capture on main and one with the change, from the same title, scene and machine, as the Performance report form asks. Please paste both aps5-summary.jsonl lines, attach the two traces and add the output of tools/trace_digest.py.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 6186 docs/dev/ENVIRONMENT.md L234
PR 6804 core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp L277, L355
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp L584
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp L4610-4611, L4983
PR 7233 core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp L355
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp L4983
PR 8278 core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp L355

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1368 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1395 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 2434 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4040 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4072 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 4472 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5021 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5058 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/IndirectDispatch.cpp
PR 5371 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5431 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
docs/dev/ENVIRONMENT.md
PR 5441 core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5499 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 5609 core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 5925 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6044 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6146 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6187 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6188 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6192 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6194 core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6202 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6209 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6223 core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6280 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6290 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6292 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/IndirectDispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6300 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6412 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6481 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6501 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6539 core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 6615 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/IndirectDispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6636 core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
docs/dev/ENVIRONMENT.md
PR 6682 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
PR 6683 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6747 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6788 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6801 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6808 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6868 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6879 docs/dev/ENVIRONMENT.md
PR 6905 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6924 docs/dev/ENVIRONMENT.md
PR 6946 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
docs/dev/ENVIRONMENT.md
PR 6947 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
docs/dev/ENVIRONMENT.md
PR 6960 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 6970 core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 6977 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7002 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 7048 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7085 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7201 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7371 core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7395 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7446 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp
PR 7449 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7464 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
docs/dev/ENVIRONMENT.md
PR 7484 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
docs/dev/ENVIRONMENT.md
PR 7505 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7514 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7604 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7642 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/include/VulkanDevice.hpp
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Dispatch/Dispatch.cpp
core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
core/libs/prx/libSceAgcDriver/Graphics/include/Recorder.hpp
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/Recorder.cpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
docs/dev/ENVIRONMENT.md
PR 7644 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7677 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7689 core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 7800 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Graphics/include/ShaderResources.hpp
core/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cpp
PR 7888 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
PR 7922 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
docs/dev/ENVIRONMENT.md
PR 8004 core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp
docs/dev/ENVIRONMENT.md
PR 8077 docs/dev/ENVIRONMENT.md
PR 8206 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 8295 core/libs/prx/libSceAgcDriver/CMakeLists.txt

The agc_driver_indirect_thread_dimensions_cpu registration sat at the end of
the address-passes test block, where other open changes add their tests too.
It now has its own BUILD_TESTING block after the execution test name check,
with the same properties.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@HQasper

HQasper commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

The CTest registration moved to its own BUILD_TESTING block, so this no longer conflicts with #7705 or any other open pull request in libSceAgcDriver/CMakeLists.txt; the configured test list is identical before and after. #6804, #7233 and #7371 each append a parameter (borderColorTable, htileClear) to VulkanDevice::dispatch, and #6804 also to DispatchIndirect, where this change appends threadGroup. Those declaration lines conflict, but the features are unrelated: whichever lands second keeps both parameters, and a wrong order does not compile. #1364 (open, conflicts with main) moves small ShaderData blocks into a shared upload page at bufferOffset; if it lands first, the thread-limit copy here must add that offset. No other open pull request changes thread-dimension indirect dispatches. Depends on: none.

@wachi35676

Copy link
Copy Markdown
Contributor

Windows runs on a different NVIDIA card and an RDNA2 iGPU, plus one extra case the test doesn't cover: two thread-dimension dispatches of the same program in one submission.

Setup: Windows 11 26300, MinGW GCC 15.2 Release, main eb75006 + this PR (84a6fb0, merges cleanly). GPUs: RTX 5070 Ti (NVIDIA 617.42) and Ryzen 5 9600X Radeon iGPU (ANYPS5_GPU=AMD, 32.0.21042.62), both with ANYPS5_REQUIRE_VULKAN=1.

  • agc_driver_indirect_thread_dimensions and _cpu passed 3 of 3 on each GPU.
  • Back-to-back case. I added a local probe to IndirectThreadDimensions.cpp (diff below, not for merge). One submission holds two DISPATCH_INDIRECT packets with USE_THREAD_DIMENSIONS for the same counter program, with the same user data and groups of 64x1x1, each reading its own argument words. So both dispatches share one template and one ShaderData buffer, and only the limit store differs between them. Expected counts are the sums: 100+40, 40+130, and, with GPU-produced arguments, 100+40 and 1+190. A limit from the second store that leaked into the first dispatch, or a dropped limit, would give 80, 192 or similar. Each pair ran 3 rounds per test, under both agc_driver_indirect_thread_dimensions and _cpu. All pairs passed on both GPUs, 3 of 3 runs. So the trailing COMPUTE->ALL_COMMANDS barrier of the first dispatch orders its ShaderData reads before the second dispatch's vkCmdCopyBuffer/vkCmdUpdateBuffer into the same buffer, as the code intends.
  • Negative control: not possible as-is. The new test calls DispatchIndirect(..., shape.group), which doesn't compile against main's sources. _cpu (APS5_NO_GPU_THREAD_DIMENSIONS=1, main's path) passes, as expected.
  • ctest -R "agc|shader|spirv|recomp" (825 tests): on NVIDIA 3 failed (msaa_dcc_clear_4, pass_input_hazard, sgpr_pattern_fill_keys); on AMD 5 failed (msaa_dcc_clear_4, sgpr_pattern_fill_keys, layered_volume_vertex_no_gpl, draw_cache_reuse_off, stencil_replay_no_gpl). The same set fails on main on this machine (sgpr_pattern_fill_keys flakes under load). No new failures.
  • check_conventions.py --base origin/main passes.

No GTA V or other title runs here, so the frame-time claim is not re-measured.

probe
diff --git a/core/libs/prx/libSceAgcDriver/tests/execution/IndirectThreadDimensions.cpp b/core/libs/prx/libSceAgcDriver/tests/execution/IndirectThreadDimensions.cpp
index 37573087c..7e9e6fa98 100644
--- a/core/libs/prx/libSceAgcDriver/tests/execution/IndirectThreadDimensions.cpp
+++ b/core/libs/prx/libSceAgcDriver/tests/execution/IndirectThreadDimensions.cpp
@@ -172,6 +172,33 @@ void RunCpuFallback(Slot& slot, const Shape& shape) {
     Check(slot, shape, std::string(shape.name) + ", read on the CPU behind a pending argument label");
 }
 
+void RunPair(Slot& slot, Slot& second, std::array<std::uint32_t, 3> first, std::array<std::uint32_t, 3> next, bool producedOnGpu) {
+    Reset(slot);
+    Reset(second);
+    second.counterDescriptor = slot.counterDescriptor;
+    std::vector<std::uint32_t> words;
+    if (producedOnGpu) {
+        Program(words, AddressWriterCode.data(), {1u, 1u, 1u}, {Low(slot.arguments.data()), High(slot.arguments.data()), first[0], first[1], first[2]});
+        words.insert(words.end(), {0xc0031500u, 1, 1, 1, GroupCounts});
+        Program(words, AddressWriterCode.data(), {1u, 1u, 1u}, {Low(second.arguments.data()), High(second.arguments.data()), next[0], next[1], next[2]});
+        words.insert(words.end(), {0xc0031500u, 1, 1, 1, GroupCounts});
+    } else {
+        for (std::size_t axis = 0; axis < 3; ++axis) slot.arguments[axis] = first[axis];
+        for (std::size_t axis = 0; axis < 3; ++axis) second.arguments[axis] = next[axis];
+    }
+    Program(words, CounterCode.data(), {64u, 1u, 1u}, {Low(slot.counterDescriptor.data()), High(slot.counterDescriptor.data())});
+    words.insert(words.end(), {0xc0021102u, 1u, Low(slot.arguments.data()), High(slot.arguments.data())});
+    words.insert(words.end(), {0xc0011600u, 0u, ThreadDimensions});
+    words.insert(words.end(), {0xc0021102u, 1u, Low(second.arguments.data()), High(second.arguments.data())});
+    words.insert(words.end(), {0xc0011600u, 0u, ThreadDimensions});
+    words.insert(words.end(), {0xc0033700u, 0x00100200u, Low(&slot.done), High(&slot.done), 1u});
+    Packet packet{words.data(), static_cast<std::uint32_t>(words.size()), 0, {}};
+    Require(sceAgcDriverSubmitAcb(Compute, &packet) == 0, "compute submit failed");
+    AgcDriverWaitIdle_nid_postfix();
+    const Shape total{"pair", {first[0] * first[1] * first[2] + next[0] * next[1] * next[2], 1u, 1u}, {64u, 1u, 1u}};
+    Check(slot, total, std::string("two thread-dimension dispatches of one program in one submission, ") + std::to_string(first[0]) + " then " + std::to_string(next[0]) + (producedOnGpu ? " threads, arguments stored by the GPU" : " threads, arguments stored by the CPU"));
+}
+
 }
 
 int main() {
@@ -193,6 +220,14 @@ int main() {
             RunSubmitted(*slot, shape, false);
             std::cout << shape.name << ": passed\n";
         }
+        auto* second = new (block + 4096) Slot{};
+        for (int round = 0; round < 3; ++round) {
+            RunPair(*slot, *second, {100u, 1u, 1u}, {40u, 1u, 1u}, false);
+            RunPair(*slot, *second, {40u, 1u, 1u}, {130u, 1u, 1u}, false);
+            if (gpuThreadDimensions) RunPair(*slot, *second, {100u, 1u, 1u}, {40u, 1u, 1u}, true);
+            if (gpuThreadDimensions) RunPair(*slot, *second, {1u, 1u, 1u}, {190u, 1u, 1u}, true);
+        }
+        std::cout << "pair probe: passed" << std::endl;
         if (gpuThreadDimensions) {
             for (const auto& shape : Shapes) {
                 RunCpuFallback(*slot, shape);

AI-assisted: yes (Claude Code); results above were run on this machine.

@CPritch

CPritch commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Ran this on GTA V (PPSA04263 01.010.002), Linux 7.2.8-zen1-2-zen, NVIDIA GeForce RTX 5060 Laptop GPU, NVIDIA 615.71.09, Graphics Mode Performance, the Story prologue after Resume Story, with no input.
main d14a186 against main d14a186 + #8066 at 84a6fb0, same machine and session, two runs each in the order A B A B, window frames 3600:700, caches warm, APS5_TRACE_EVENTS=4194304 (BUILD.md's remedy for full trace buffers), no other APS5_TRACE_* or APS5_PROFILE_* switch, log not redirected. The harness navigates the menus from frame dumps in boot and load to present 3420 (APS5_DUMP_FRAMES=114, every 30 presents), which stop before the window.
Frame p50: main 81.1, 80.8 ms, with #8066 80.8, 80.7 ms (-0.2 ms, -0.3 %). GPU busy per present: 64.7 and 64.7 ms.
Whether GTA V's DISPATCH_INDIRECT dispatches with thread dimensions took the new GPU path in these runs was not checked.
The summary lines' Driver.EndSubmission (about 2,979 and 2,946 ms per frame) and worker_busy_pct for queue worker 0x0 (about 3,701 and 3,679 %) are not costs: on the pipelined path the Driver.EndSubmission timer stays open through the rewind tail, so nested tails are counted many times over (#8092; #8108 has the fix).
Not tested: other scenes, Fidelity mode, other GPUs.
Written with AI assistance (Claude Code); measured on the machine above.

Summary lines (main, then the PR):
{"commit":"d14a186f4615","dirty":false,"env":{"APS5_DUMP_FRAMES":"114","APS5_DUMP_FRAMES_EVERY":"30","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_FRAMES":"3600:700"},"gpu":"NVIDIA GeForce RTX 5060 Laptop GPU","driver":"NVIDIA 615.71.09","os":"Linux 7.2.8-zen1-2-zen","cpu":"AMD Ryzen AI 9 465 w/ Radeon 880M","cpu_threads":20,"memory_mib":31391,"title_id":"PPSA04263","title":"Grand Theft Auto V","content_version":"01.010.002","resolution":"3840x2160","window":{"spec":"frames 3600:700","first_present":3600,"ms":57589.746},"presents":700,"frame_ms":{"p50":81.071,"p90":89.216,"p99":112.759,"max":114.703},"gpu_busy_ms_per_present":64.591,"worker_busy_pct":{"queue worker 0x0":3701.0},"top_phases_ms_per_frame":{"Driver.EndSubmission":2979.061,"Driver.Packet":63.479,"Graphics.Draw":59.664,"Graphics.Draw.draw_and_resource_release":56.017,"Driver.Packet.draw":50.234,"Driver.Draw":50.019,"VideoOut.Flip":25.232,"Driver.Present":25.211},"trace_dropped":false,"trace":"aps5-trace-d14a186f4615-2448189.json.gz"}
{"commit":"0ed4594f2de0","dirty":false,"env":{"APS5_DUMP_FRAMES":"114","APS5_DUMP_FRAMES_EVERY":"30","APS5_NO_PAD_INPUT":"1","APS5_PAD_INPUT_FILE":"","APS5_TRACE_EVENTS":"4194304","APS5_TRACE_FRAMES":"3600:700"},"gpu":"NVIDIA GeForce RTX 5060 Laptop GPU","driver":"NVIDIA 615.71.09","os":"Linux 7.2.8-zen1-2-zen","cpu":"AMD Ryzen AI 9 465 w/ Radeon 880M","cpu_threads":20,"memory_mib":31391,"title_id":"PPSA04263","title":"Grand Theft Auto V","content_version":"01.010.002","resolution":"3840x2160","window":{"spec":"frames 3600:700","first_present":3600,"ms":57273.579},"presents":700,"frame_ms":{"p50":80.753,"p90":89.564,"p99":105.694,"max":112.216},"gpu_busy_ms_per_present":64.804,"worker_busy_pct":{"queue worker 0x0":3679.401},"top_phases_ms_per_frame":{"Driver.EndSubmission":2945.644,"Driver.Packet":62.559,"Graphics.Draw":59.691,"Graphics.Draw.draw_and_resource_release":56.117,"Driver.Packet.draw":49.536,"Driver.Draw":49.322,"VideoOut.Flip":25.823,"Driver.Present":25.812},"trace_dropped":false,"trace":"aps5-trace-0ed4594f2de0-2456265.json.gz"}

@github-actions github-actions Bot removed the perf numbers missing perf pull request without a before and after summary line label Oct 20, 2026
@HQasper

HQasper commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@wachi35676 Thanks for running it on two more GPUs, and for the back-to-back probe. Two dispatches of one program in one submission share the template and the ShaderData buffer and differ only in the limit store, so that is the ordering I most wanted checked, and the sums (140, 170, 191) rule out both a leaked and a dropped limit.

On the negative control: as noted under Tested, without the limit store the first shape runs 128 threads instead of 100, so the test does fail when the store is missing.

@CPritch Thanks for the second machine. -0.3 % is within run-to-run noise, so in this window on this laptop the change makes no measurable difference. The gain in the description comes from removing the CPU reads of thread-dimension arguments, each of which first finishes leased GPU work under the GPU mutex. It can only show where those reads are frequent in the window and the frame waits on them. Your summaries show far less queue-worker work than mine (Driver.Packet about 63 against 280 ms per frame, and no Driver.Packet.memory_wait or Driver.Packet.dispatch among the top phases), and the laptop is closer to GPU-bound (64.7 of 81 ms GPU busy). So either this part of the prologue issues few of these dispatches, or their reads are off the critical path here. APS5_PROFILE_DRAW=1 prints an [indirect] line every 10 s with the number of CPU thread-dimension reads and the time they took, which would tell which.

I've added your result to the description, so the speed-up isn't read as general.

@adriadam10 adriadam10 self-assigned this Oct 20, 2026
@adriadam10
adriadam10 merged commit 73b6a4c into boykopovar:main Oct 20, 2026
14 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the argument kernel's ceil(threads / size) and the thread-limit store, the CPU fallback in decideIndirect, the dispatch-move refusal for the ShaderData buffer, and the new test (it pins the exact thread count per shape, and the _cpu variant covers the switch). The back-to-back and AMD/NVIDIA runs from @wachi35676 and your Windows numbers cover the GPU side; merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs perf Performance change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants