Repository navigation
perf(recompiler): back compute scratch with a device buffer slot pool - #12764
Merged
oneandonlydean merged 8 commits intoNov 2, 2026
Merged
Conversation
github-actions
Bot
added
perf
|
This pull request is labeled |
Collaborator
|
@Pipodi Please keep private-aperture routing when FLAT pointer provenance is unknown. No src_private_base/limit read does not prove a pointer loaded from memory, supplied in SGPRs, or made from a literal cannot name scratch; the new default drops previously supported accesses to BDA. Eliminate the route only with a conservative address proof, or retain the safe default for this experiment. Please retain the original memory-fed Private/Mixed FlatApertures kernels without injecting the unused MovPrivateBase at lines 150/181. Add execution coverage of private pointers from memory/literals/computation and callee cases, with load/store/atomic and both wave sizes; structural absence of a constant or array is not proof that routing can be removed. In the new shader debt entry, use PPSA26344 instead of the game name and label the compile-time numbers as downstream-tree results. Also correct the cache-key sentence: the quoted APS5_FLAT_PRIVATE_ALWAYS name is automatically discovered into Generated::RecompilerSwitches and is not neutral, so normal disk keys already include its value. FormatVersion stays 49; changed source hashes invalidate older entries. @adriadam10 Keeping you in the loop as the assigned maintainer. Two smaller points from a second review:
|
Pipodi
marked this pull request as draft
October 29, 2026 09:00
Contributor
Author
|
@oneandonlydean thanks for the careful review. You're right on the main point, and I'm converting this to a draft. Provenance. Not reading Plan. The cost isn't the routing itself but the shape of the private branch: dynamically indexed scratch arrays at every FLAT site. In an offline replay of one 1.52M-word compute program (downstream tree, PPSA26344, RTX 4090), backing scratch with a storage buffer instead of function-local arrays brought the compile from 141-145 s to about 100 s, and NVIDIA's front end from 36-37 % to 1.4 % of it, the same as removing the branch. So the next version keeps private-aperture routing for every compute FLAT access and changes only how the private branch reaches scratch. No provenance assumption, no switch. Coverage I'll add: private pointers loaded from memory, from a literal and computed, through load, store and atomic, in wave32 and wave64, plus a callee case. Smaller points, all of them taken:
I'll mark it ready again once the storage-buffer version and the tests are in. |
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
Pipodi
force-pushed
the
perf/flat-private-aperture-1028
branch
from
October 29, 2026 11:28
0573779 to
659d07b
Compare
Pipodi
changed the title
Pipodi
marked this pull request as ready for review
October 29, 2026 11:28
github-actions
Bot
removed
the
perf numbers missing
Pipodi
force-pushed
the
perf/flat-private-aperture-1028
branch
from
October 29, 2026 11:38
659d07b to
83d6baf
Compare
Contributor
Author
|
Reworked as promised, and rebased onto current
One thing I can't fully explain is that the buffer needs |
Contributor
✅ NVIDIA GPU test: passedPR head
Intermittent validation messages (1, seen once in the parallel run, not reproduced when the test was re-run alone 2 times; not counted against this PR)Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (5, not counted against this PR)Passed on retry (2, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)Validated but not compared (1, not in the main baseline's validation pass)Validation IDs already present on main (4 IDs in 408 tests)Skipped (39)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
Collaborator
|
@Pipodi thanks for restoring conservative FLAT routing and trying a different optimization. One initialization failure path needs fixing: VulkanDevice.cpp:5356-5359 records vkCmdFillBuffer and immediately sets the cached pool's zeroed flag. Recorder::Submit can subsequently discard that unsubmitted batch (Recorder.cpp:3658-3666, including a vkEndCommandBuffer failure), while scratchPools still owns the pool. A later dispatch then skips its header clear, so scratch acquisition can read uninitialized ownership words and spin. Please make initialization follow successful ordered submission, with rollback/reinitialization after discard; simply clearing on every dispatch could overwrite another in-flight use. Add a fault-injection regression that discards the first initializing batch and retries the same shader/pool, checking initialization is actually executed before acquisition. Please account for separate recorded batches using the same cached pool before the initializing batch submits. This is source-derived; no GPU hang was induced here. |
This was referenced Oct 30, 2026
12 tasks done
Pipodi
force-pushed
the
perf/flat-private-aperture-1028
branch
from
October 30, 2026 07:23
83d6baf to
30930bc
Compare
Contributor
Author
|
@oneandonlydean thanks, you were right: a discarded batch left the cached pool marked zeroed with uninitialized lock words. Fixed in 30930bc, and rebased onto current
After the rebase: full ctest 3107/3107, and the FLAT aperture, scratch pool, LDS and guarded memory load tests with |
Contributor
|
An AMD RDNA 4 run on Windows of Windows 11 Pro 10.0.22631, AMD Radeon RX 9070 XT (
What I think happens, from the code: in To check, I swapped lines 3671 and 3672 locally (the throw now comes after AI-assisted: yes (Claude Code) |
Contributor
✅ NVIDIA GPU test: passedPR head
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (2, not counted against this PR)Validated but not compared (1, not in the main baseline's validation pass)Validation IDs already present on main (4 IDs in 392 tests)Skipped (41)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
12 tasks done
Contributor
✅ AMD GPU test: passedPR head
Unstable on main (4, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (9, not counted against this PR)Validation IDs already present on main (5 IDs in 484 tests)Skipped (55)ReproduceAutomated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
Contributor
|
@Pipodi confirmed on the AMD card where it failed: Same setup as my first run: Windows 11 Pro 10.0.22631, AMD Radeon RX 9070 XT (
This adds the Windows amdvlk side to the RADV run above. Thanks for the quick swap 🙂 AI-assisted: yes (Claude Code) |
Contributor
✅ NVIDIA GPU test: passedPR head
Unstable on main (4, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (4, not counted against this PR)Validation IDs already present on main (5 IDs in 484 tests)Skipped (46)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
12 tasks done
NVIDIA's front end spends tens of seconds per large compute program on the dynamically indexed Function arrays that held the per-lane scratch. A compute program that uses scratch (scratch_* or a FLAT access that the aperture split routes to the private aperture) now keeps it in a storage buffer instead, so no Function array is declared or indexed. FLAT routing is unchanged. The driver allocates one buffer per compute pipeline on its first dispatch, capped by APS5_SCRATCH_POOL_MIB (default 64), holding a lock word and an invocation counter per slot followed by the slots, and binds it in descriptor set 2 (the layout of the device-memory LDS buffer). The shader takes the slot count from the array length. Invocation 0 of each workgroup claims a slot with a compare-exchange scan that starts at its linear workgroup id and spins until it wins one; a barrier publishes the slot. Every invocation decrements the slot's counter on every compute exit and the last one frees the lock, so divergent exits and concurrent dispatches of one pipeline share the buffer safely. The emitter fails the compile if a compute return lacks the release. Programs without scratch keep their SPIR-V and pipeline layout. The buffer is decorated Coherent: without it NVIDIA lost earlier byte stores of a sequence of byte read-modify-writes to one dword. Lane-private read-modify-writes no longer use atomics. Tests: a new execution test covers the private aperture reached through a table load, a literal and a computed address (store, atomic, and both ways through scratch_*), a callee reached through s_swappc_b64, isolation of 64 workgroups over one or two slots, divergent wave exits with a second dispatch reusing the slots, and the module shape (no Function array, no binding in a program without scratch). The guarded memory load fixture skips the slot acquisition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The slot-lock header of a compute scratch pool was cleared by a fill recorded into the first dispatch batch that used the pool, and a flag set at record time said it was done. A batch that Recorder::Submit discarded unsubmitted (for example a vkEndCommandBuffer failure) left the pool cached with the flag set, so later dispatches skipped the clear and slot acquisition read uninitialized ownership words and could spin; a second dispatch recorded against the same pool before the first batch submitted trusted the same flag. The header is now zeroed by a one-off command batch that is submitted and waited on when the pool is created, before the dispatch's own batch opens, so discards and recording order no longer matter. The zeroed flag and the in-batch fill are gone. A new test discards the first batch that uses a pool (two dispatches against it) on top of dirtied pool memory and checks that the header reads zero and the retried dispatch completes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each compute program that uses scratch kept its own pool of up to APS5_SCRATCH_POOL_MIB (64 MiB) until device teardown, so the pools of many such programs added up without bound. All pools together are now held under APS5_SCRATCH_POOL_TOTAL_MIB (default 256 MiB). A new or enlarged pool first drops the least recently used pools until it fits; the newest pool is always kept. A dropped pool stays alive until the batches that recorded it complete, since the recorder keeps a reference, and the program gets a new pool with a zeroed header when it dispatches again. The test dispatches two scratch programs over a 1 MiB budget and checks that the second evicts the first, and that the first then runs correctly in a recreated pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On an RTX 4090 (driver 617.42) a scratch pool buffer of 2049 MiB returned wrong scratch values in the scratch dispatch bench, while 2047 MiB was correct, though maxStorageBufferRange is larger. One pool is now held under 2 GiB, under APS5_SCRATCH_POOL_MIB and under APS5_SCRATCH_POOL_TOTAL_MIB, so a large APS5_SCRATCH_POOL_MIB no longer creates a pool over the total budget or over 2 GiB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A workgroup looking for a scratch pool slot now loads the lock and tries the compare-exchange only when it reads free, so waiting workgroups stop issuing atomics on locks their holders use. In the scratch dispatch bench on an RTX 4090 with about 9 slots for 32768 workgroups (APS5_SCRATCH_POOL_MIB=1) the dispatch goes from 1377 ms to 1308 ms; with the default 64 MiB pool it is unchanged (389 to 395 ms). The debt notes the measured runtime cost of the pool against Function arrays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pipodi
force-pushed
the
perf/flat-private-aperture-1028
branch
from
November 2, 2026 03:47
ee700f7 to
28a30ed
Compare
Contributor
Author
|
Thanks, all four were fair. I measured each one; two of them changed the PR. 1. Runtime of a scratch-heavy program. New bench
So the PR costs about 2.4× at run time on a scratch-heavy kernel. 2. Which pool programs dispatch in the window. I added a log line at pool creation and ran warm pairs with 3. Pool memory. Fixed:
4. Slot claim spin. With slots below resident groups the waiting groups serialize. At 1 MiB (~9 slots) a 4096-group dispatch takes 172 ms against 49 ms at 64 MiB (~580 slots), so the cost is the slot count, not the atomics. I changed the claim to read the lock word first and try the Commits added, after a rebase onto Given point 1 and 2, I understand if you prefer this off by default or not merged until a title needs the compile-time saving: it saves about 40 s of compile per giant program and costs ~2.4× dispatch time on scratch programs, which this title does not dispatch before frame 530. |
# Conflicts: # core/libs/prx/libSceAgcDriver/Execution/src/VulkanDevice.cpp # docs/dev/debt/libSceAgcDriver.md # docs/dev/debt/shader.md
github-actions
Bot
added
needs rebase
Contributor
✅ NVIDIA GPU test: passedPR head
Unstable on main (3, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (3, not counted against this PR)Validation IDs already present on main (5 IDs in 483 tests)Skipped (45)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
SpirvEmitter.cpp: main now calls Finalize() before filling info and checks spanned-wave barriers on the words; keep that and run this branch's VerifyScratchPoolReturns(state, words) on the same words before returning.
Contributor
✅ NVIDIA GPU test: passedPR head
Unstable on main (4, these also failed in recent main runs on this machine; not counted against this PR)Failing on main too (3, not counted against this PR)Validation IDs already present on main (4 IDs in 510 tests)Skipped (44)ReproduceAutomated build and test on a contributor's NVIDIA machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs. |
12 tasks done
11 tasks done
This was referenced Nov 3, 2026
Raghav314
added a commit
to Raghav314/AnyPS5
that referenced
this pull request
Nov 3, 2026
The compute scratch pool from boykopovar#12764 sizes itself with OpArrayLength on a storage buffer at descriptor set 2. A GPU-mode stage maps its buffers onto VK_EXT_descriptor_heap through pushed addresses, which cannot carry an array length (VUID-VkPipelineShaderStageCreateInfo-pNext-11378), so every GPU-mode compute stage with scratch was refused by the heap and recompiled in CPU mode. GPU-mode compute stages keep scratch in function storage, as they did before the merge; the scratch pool test also runs in heap mode.
Raghav314
added a commit
to Raghav314/AnyPS5
that referenced
this pull request
Nov 3, 2026
GPU-mode compute stages now use the boykopovar#12764 scratch pool instead of function-storage scratch. The pool binds at descriptor set 5 through a pushed address after the workgroup memory address, and because a pushed address carries no array length, the stage reads its pool size in dwords from push data instead of OpArrayLength. The scratch pool test runs every case in heap mode and requires the descriptor heap pipeline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every compute FLAT load, store and atomic in a program with scratch tests its address against the shared and the private aperture (c35f29b). The private path reaches the per-lane scratch, held in Function
scratch_dwordsarrays indexed dynamically at every FLAT site. On NVIDIA that indexing is what makes large compute programs slow to compile: the SPIR-V front end (nvoglv64) spends about a third of the compile on it.This PR keeps the FLAT routing exactly as on
main, for every access, and changes only where scratch lives. A compute program that uses scratch (scratch_*, or a FLAT access routed to the private aperture) now keeps it in a storage buffer, so no Function array is declared or indexed:APS5_SCRATCH_POOL_MIB(default 64, never below one slot), bymaxStorageBufferRangeand by 2 GiB: on the RTX 4090 a larger buffer returned wrong values (2049 MiB fails, 2047 MiB passes). All pools together stay underAPS5_SCRATCH_POOL_TOTAL_MIB(default 256); a new pool evicts the least recently used ones first. The shader takes the slot count from the array length.tests/GuardedMemoryLoads.cppchanges too: that fixture runs the emitted code in its interpreter, which now skips the slot claim and offsets scratch indexes by the 2-dword slot header. It also gains checks for the new layout.The buffer is decorated
Coherent. Without it, on an RTX 4090, an unaligned 32-bit FLAT store to scratch (emitted as four byte read-modify-writes) loses the first byte of each pair sharing a dword:flat_apertures_2,flat_apertures_5andflat_aperture_sourcesfail, always with the same value. It is not lane overlap (4× lane spacing gives the same result), not spirv-opt (same with it off) andAliaseddoesn't help. The SPIR-V is a plain load, mask and store in program order, so this looks like a driver ordering issue, but I haven't read the GPU assembly to prove it. The debt entry indocs/dev/debt/shader.mdsays so.On the review of the first version: the provenance rule,
formsPrivateAperture,APS5_FLAT_PRIVATE_ALWAYSand the edit toFlatApertures.cppare gone.main'sFlatApertures.cppis unchanged and passes. Callees reached throughs_swappc_b64are linked into the caller's graph (GraphBuilderLinkCalls), so they are translated with it and use the same buffer; the new test covers that case.On the second review (discarded batch): the slot headers were zeroed inside the first dispatch's batch and the pool was marked zeroed before submit, so a discarded batch left a cached pool marked zeroed with garbage headers. Now the headers are zeroed once when the pool is created, in a batch submitted and waited on under the GPU lock; the
zeroedflag and the in-batch fill are gone, and the pool lookup runs before the recorder opens the dispatch's commands. The debt entry "Compute scratch pool header clear" indocs/dev/debt/libSceAgcDriver.mdcovers the blocking submit.On the AMD report: the test seam
Recorder::FailNextSubmitsForTestthrew beforevkEndCommandBuffer, so the discarded batch went back to the spare list while still recording, and the nextvkBeginCommandBufferon it failed on RDNA 4 (VUID-vkBeginCommandBuffer-commandBuffer-00049, result -13). The injected failure now comes after the end and stands for a failedvkQueueSubmit.The PR has six commits, meant to be read in order (squash on merge is fine):
a46f050af8perf(recompiler): back compute scratch with a device buffer slot pool — the change described above.f369273365fix(agc): zero the scratch pool header when the pool is created — the fix for the discarded-batch review.d9668fdbeefix(agc): end the command buffer before the injected submit failure — the fix for the AMD report; it touches only the test seam.cc61c3ae31perf(agc): hold all compute scratch pools under one memory budget —APS5_SCRATCH_POOL_TOTAL_MIBwith LRU eviction (review point 3).ce38162aa1fix(agc): hold one compute scratch pool under 2 GiB and the total budget.28a30ed35dperf(shader): read a scratch pool slot lock before trying to take it (review point 4).Why the dynamic indexing is the cost (offline replay, cold driver cache, one compile thread, PPSA26344 program 0x1994ed, 1.52M SPIR-V words)
mainWorkgroup memory can't hold it: 56 dwords per lane × 512 lanes is 114 KB.
The runtime side (third review) has a cost. A local bench, not in this PR: one compute program with 448 scratch dwords per lane, median of repeated dispatches, RTX 4090, driver 617.42. It is listed in the debt file as "Compute scratch runtime cost".
Dispatch time of a scratch-heavy program, `main` against this PR
main(Function arrays)CoherentdroppedAbout 2.4× slower at the default size;
Coherentis not the cost. With fewer slots than resident groups the groups wait for a slot: reading the lock before the compare-exchange gains about 5 % at 9 slots and nothing at 580.Tested
Windows 11, MinGW GCC 15.2, RTX 4090 (driver 617.42), i9-13900K.
agc_driver_scratch_pool(tests/execution/ScratchPool.cpp), wave32 and wave64:scratch_*;s_swappc_b64;ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation: 45/45 pass (FLAT apertures, LDS, scratch, guarded memory loads), noVUID-. Rerun of the 16 FLAT aperture and scratch pool tests withVK_LOADER_DEBUG=layerto confirm the layer loads: 16/16, noVUID-.agc_driver_scratch_poolfails (isolation: "round 0 lane 0 dword 0 read 0x00002200, expected 0x00000000"; divergent exit: "round 0 lane 160 dword 0 read 0x00003180, expected 0x00000280").RunDiscardedInitializationinagc_driver_scratch_pool(test seamRecorder::FailNextSubmitsForTest): discards the first dispatch's batch, then dispatches again on the cached pool. Fails before the header fix ("header word 0 reads 0xffffffff"), passes after.mainf7f03da: 3107/3107. The focused run above, repeated on the rebased branch: 43/43 FLAT aperture, scratch pool, LDS and guarded memory load tests, noVUID-. The rebase conflicted only wheremain'smeshWindowBufferfield (fix(shader): keep mesh register windows out of local memory on NVIDIA #12233) sits next toscratchPoolSlotDwords(kept both, same order in the disk cache writer and reader) and in the debt file.mainf2d874c: full ctest 3286/3286; focused run with the same environment 56/56 (FLAT apertures, scratch pool, LDS, guarded memory loads, recorder),agc_driver_scratch_poolincluded, noVUID-. The rebase conflicted only wheremain's compute pipeline binary key (0c572f1) is built: the key's first word now also sets bit 1 when the scratch pool set is bound, so keys of pipelines without scratch are unchanged.mainbe24202: full ctest 3283/3283. Focused run with the same environment (flat|scratch|lds|guarded|recorder): 64/64 on the PR, 63/63 onmain, noVUID-in either.mainaaccd0b and the first commit on top of it.PPSA26344 (Ghost of Yōtei 01.008.000), 3840x2160, fresh save, shader cache warmed by a previous run of the same build, frames 250-280 (loading).
mainaaccd0b and this PR alternate, three runs each. The PR build compiled about 13 compute programs with the slot pool before frame 350 (26 shader cache entries bind descriptor set 2; none onmain); Those 13 are compiled, not dispatched: a local build that logs every pool creation, run warm withAPS5_PROFILE_GPU=1for 1500 s, created no pool up to frame 517 (main) and 530 (PR). So no scratch program dispatches in the window, and the trace pair below can only show compile-side effects. GPU time over frames 250-280 was 2.08 s onmainand 2.20 s on the PR (noise; 50.9 s against 52.5 s over the whole run).main:{"commit":"aaccd0ba39b7","dirty":false,"env":{"APS5_GPU_CHECKPOINTS":"1","APS5_HOST_IMPORT_MIB":"16384","APS5_NO_PIPELINE_JOURNAL":"1","APS5_NO_RESULT_MEMO":"1","APS5_NO_SNAPSHOT_CHECK":"1APS5_SPANNED_WAVE=1","APS5_PIPELINE_DISPATCH":"1","APS5_PREBUILD_THREADS":"2","APS5_PREPARED_SPIRV_MIB":"6144","APS5_PROFILE_DRAW":"1","APS5_RESOURCE_CACHE_ENTRIES":"1024","APS5_STAGING_POOL_MIB":"2048","APS5_TRACE_FRAMES":"250:30"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","cpu":"13th Gen Intel(R) Core(TM) i9-13900K","cpu_threads":32,"memory_mib":32476,"title_id":"PPSA26344","title":"Ghost of Yōtei","content_version":"01.008.000","resolution":"3840x2160","window":{"spec":"frames 250:30","first_present":250,"ms":16996.075},"presents":30,"frame_ms":{"p50":308.640,"p90":1276.579,"p99":1890.406,"max":1890.406},"gpu_busy_ms_per_present":111.751,"worker_busy_pct":{"queue worker 0x0":75.642,"queue worker 0x52":1.078,"queue worker 0x53":0.000,"queue worker 0x20":27.602,"queue worker 0x30":62.036,"queue worker 0x38":59.136,"queue worker 0x48":24.100},"top_phases_ms_per_frame":{"Driver.Packet":1410.773,"Driver.Packet.memory_wait":757.111,"Driver.Packet.dispatch":422.516,"Driver.Dispatch":407.334,"GpuMutex.wait.other":160.332,"Vulkan.Dispatch":144.609,"Vulkan.Dispatch.shader_resources":130.486,"Driver.Packet.prepare_memory":110.346},"trace_dropped":false,"trace":"aps5-trace-aaccd0ba39b7-2756.json.gz"}This PR:
{"commit":"659d07bbc8c0","dirty":false,"env":{"APS5_GPU_CHECKPOINTS":"1","APS5_HOST_IMPORT_MIB":"16384","APS5_NO_PIPELINE_JOURNAL":"1","APS5_NO_RESULT_MEMO":"1","APS5_NO_SNAPSHOT_CHECK":"1APS5_SPANNED_WAVE=1","APS5_PIPELINE_DISPATCH":"1","APS5_PREBUILD_THREADS":"2","APS5_PREPARED_SPIRV_MIB":"6144","APS5_PROFILE_DRAW":"1","APS5_RESOURCE_CACHE_ENTRIES":"1024","APS5_STAGING_POOL_MIB":"2048","APS5_TRACE_FRAMES":"250:30"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","cpu":"13th Gen Intel(R) Core(TM) i9-13900K","cpu_threads":32,"memory_mib":32476,"title_id":"PPSA26344","title":"Ghost of Yōtei","content_version":"01.008.000","resolution":"3840x2160","window":{"spec":"frames 250:30","first_present":250,"ms":13989.698},"presents":30,"frame_ms":{"p50":263.902,"p90":1098.961,"p99":1768.352,"max":1768.352},"gpu_busy_ms_per_present":102.834,"worker_busy_pct":{"queue worker 0x0":83.139,"queue worker 0x52":0.717,"queue worker 0x53":0.000,"queue worker 0x20":26.429,"queue worker 0x30":69.186,"queue worker 0x38":66.501,"queue worker 0x48":26.299},"top_phases_ms_per_frame":{"Driver.Packet":1266.947,"Driver.Packet.memory_wait":702.545,"Driver.Packet.dispatch":377.420,"Driver.Dispatch":362.764,"Vulkan.Dispatch":132.113,"Vulkan.Dispatch.shader_resources":119.352,"GpuMutex.wait.other":110.325,"Driver.Packet.prepare_memory":87.509},"trace_dropped":false,"trace":"aps5-trace-659d07bbc8c0-15012.json.gz"}main(window, frame, GPU per present)The spread between runs of the same build (14.5-17.0 s) is larger than any difference between the two; no frame cost from the device-memory scratch shows in this scene.
APS5_NO_SNAPSHOT_CHECKreads1APS5_SPANNED_WAVE=1because of a missing comma in my launcher, soAPS5_SPANNED_WAVEis off in all six runs, as onmainby default.trace_digest, `main` (last pair)
frames 250:30 from present 250: 30 presents, frame p50 308.6 / p90 1276.6 / p99 1890.4 / max 1890.4 ms, GPU busy 111.8 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 617.42, Windows 10.0.26200, 3840x2160, commit aaccd0b, dropped: no
{"commit":"aaccd0ba39b7","dirty":false,"env":{"APS5_GPU_CHECKPOINTS":"1","APS5_HOST_IMPORT_MIB":"16384","APS5_NO_PIPELINE_JOURNAL":"1","APS5_NO_RESULT_MEMO":"1","APS5_NO_SNAPSHOT_CHECK":"1APS5_SPANNED_WAVE=1","APS5_PIPELINE_DISPATCH":"1","APS5_PREBUILD_THREADS":"2","APS5_PREPARED_SPIRV_MIB":"6144","APS5_PROFILE_DRAW":"1","APS5_RESOURCE_CACHE_ENTRIES":"1024","APS5_STAGING_POOL_MIB":"2048","APS5_TRACE_FRAMES":"250:30"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","cpu":"13th Gen Intel(R) Core(TM) i9-13900K","cpu_threads":32,"memory_mib":32476,"title_id":"PPSA26344","title":"Ghost of Y\u014dtei","content_version":"01.008.000","resolution":"3840x2160","window":{"spec":"frames 250:30","first_present":250,"ms":16996.075},"presents":30,"frame_ms":{"p50":308.64,"p90":1276.579,"p99":1890.406,"max":1890.406},"gpu_busy_ms_per_present":111.751,"worker_busy_pct":{"queue worker 0x0":75.642,"queue worker 0x52":1.078,"queue worker 0x53":0.0,"queue worker 0x20":27.602,"queue worker 0x30":62.036,"queue worker 0x38":59.136,"queue worker 0x48":24.1},"top_phases_ms_per_frame":{"Driver.Packet":1410.773,"Driver.Packet.memory_wait":757.111,"Driver.Packet.dispatch":422.516,"Driver.Dispatch":407.334,"GpuMutex.wait.other":160.332,"Vulkan.Dispatch":144.609,"Vulkan.Dispatch.shader_resources":130.486,"Driver.Packet.prepare_memory":110.346},"trace_dropped":false,"trace":"aps5-trace-aaccd0ba39b7-2756.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.WorkerWait 1550.5, Driver.WorkerWait.idle_and_completion_poll 1550.5, Driver.Packet 1407.2, Driver.Packet.memory_wait 757.1, Driver.Packet.dispatch 421.3, Driver.Dispatch 405.4, GpuMutex.wait.other 160.2, Vulkan.Dispatch 143.7
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
trace_digest, this PR (last pair)
frames 250:30 from present 250: 30 presents, frame p50 263.9 / p90 1099.0 / p99 1768.4 / max 1768.4 ms, GPU busy 102.8 ms per present
NVIDIA GeForce RTX 4090, NVIDIA 617.42, Windows 10.0.26200, 3840x2160, commit 659d07b, dropped: no
{"commit":"659d07bbc8c0","dirty":false,"env":{"APS5_GPU_CHECKPOINTS":"1","APS5_HOST_IMPORT_MIB":"16384","APS5_NO_PIPELINE_JOURNAL":"1","APS5_NO_RESULT_MEMO":"1","APS5_NO_SNAPSHOT_CHECK":"1APS5_SPANNED_WAVE=1","APS5_PIPELINE_DISPATCH":"1","APS5_PREBUILD_THREADS":"2","APS5_PREPARED_SPIRV_MIB":"6144","APS5_PROFILE_DRAW":"1","APS5_RESOURCE_CACHE_ENTRIES":"1024","APS5_STAGING_POOL_MIB":"2048","APS5_TRACE_FRAMES":"250:30"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","cpu":"13th Gen Intel(R) Core(TM) i9-13900K","cpu_threads":32,"memory_mib":32476,"title_id":"PPSA26344","title":"Ghost of Y\u014dtei","content_version":"01.008.000","resolution":"3840x2160","window":{"spec":"frames 250:30","first_present":250,"ms":13989.698},"presents":30,"frame_ms":{"p50":263.902,"p90":1098.961,"p99":1768.352,"max":1768.352},"gpu_busy_ms_per_present":102.834,"worker_busy_pct":{"queue worker 0x0":83.139,"queue worker 0x52":0.717,"queue worker 0x53":0.0,"queue worker 0x20":26.429,"queue worker 0x30":69.186,"queue worker 0x38":66.501,"queue worker 0x48":26.299},"top_phases_ms_per_frame":{"Driver.Packet":1266.947,"Driver.Packet.memory_wait":702.545,"Driver.Packet.dispatch":377.42,"Driver.Dispatch":362.764,"Vulkan.Dispatch":132.113,"Vulkan.Dispatch.shader_resources":119.352,"GpuMutex.wait.other":110.325,"Driver.Packet.prepare_memory":87.509},"trace_dropped":false,"trace":"aps5-trace-659d07bbc8c0-15012.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 1263.4, Driver.WorkerWait 1211.0, Driver.WorkerWait.idle_and_completion_poll 1211.0, Driver.Packet.memory_wait 702.5, Driver.Packet.dispatch 376.2, Driver.Dispatch 360.8, Vulkan.Dispatch 131.1, Vulkan.Dispatch.shader_resources 118.4
Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
Downstream only (Celegans12's yotei/best plus local changes, not main): PPSA26344 all cold to frame 1200
Same downstream build and environment in each row; only the scratch handling differs. Cold shader cache, NVIDIA driver cache moved aside, fresh save.
main)run570 also lacks one other local change (#6867), so its frame times are not a pure A/B; the compile times of the four programs are. Offline, from captures of the same programs (one compile thread, cold driver cache): 1.69M and 1.57M words compile in 100 s each with this PR, against 135-137 s for 1.45-1.52M words with the arrays.
Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)🤖 Generated with Claude Code