Repository navigation
perf: Code Violet (PPSA26528) title screen at 1.8 FPS on Windows: both queue workers busy comparing imported guest memory (3.6 TB per 300 s) and copying 4 GiB buffers per dispatch #6257
Description
Title and scene
PPSA26528, title screen (logo and START prompt, before pressing Start), warm caches after a restart; the main menu after pressing Start runs about three times slower still
What is slow or what changed
Frames take 0.55 s median (p90 0.71 s) at the title screen and 1.9 to 2.5 s at the main menu, on an RTX PRO 6000 Blackwell with driver 597.16 on Windows 11, at the title's 3840x2160. The GPU is busy for 155 ms of each 550 ms frame; both queue workers are at 100%. The large slices are Driver.Packet.dispatch (485 ms per frame, of which Resources.Complete 360 ms and Vulkan.Dispatch.shader_resources 351 ms) and Driver.Packet.memory_wait (324 ms per frame, WAIT_REG_MEM on the title's fences). About 1,300 compute dispatches run per frame.
A separate run with APS5_PROFILE_DRAW=1 and APS5_TRACE_SUMMARY_EVERY=30 (not the capture attached) names the two costs inside the resource phase:
- The [guestmem] line reports compare 4955904 calls 3659087 MiB 201.8 s over a 300 s run: two thirds of the worker time compares imported guest memory byte for byte, the Windows path that the technical debt entry on host-pointer imports describes. The title's resources include 16320x16320 virtual-texture pools viewed in five formats.
- [resources] guest upload of 4 regions, 4095.6 MiB (largest 4095.0 MiB, copied 4095.0 MiB, addressable 0) took 79 ms: a buffer the shaders declare as about 4 GiB is copied into a device buffer per dispatch that uses it, 508 times in the run at 80 to 440 ms each (66 s in total); the [buffers] line counts 20 TB copied out of imports and back on the GPU over the run.
- The dispatch cache (8192 entries) hits about 5% because the keys change every frame; 233,643 evictions in the run.
Summary lines
{"commit":"dae8651f631a","dirty":false,"env":{"APS5_TRACE_SECONDS":"60:30"},"gpu":"NVIDIA RTX PRO 6000 Blackwell Workstation Edition","driver":"NVIDIA 597.16","os":"Windows 10.0.22631","resolution":"3840x2160","window":{"spec":"seconds 60:30","first_present":44,"ms":30959.256},"presents":51,"frame_ms":{"p50":551.749,"p90":709.403,"p99":1533.122,"max":1533.122},"gpu_busy_ms_per_present":155.447,"worker_busy_pct":{"queue worker 0x0":99.911,"queue worker 0x20":97.964},"top_phases_ms_per_frame":{"Driver.Packet":1198.958,"Driver.Packet.dispatch":485.397,"Driver.Dispatch":484.533,"Resources.Complete":360.324,"Vulkan.Dispatch":357.057,"Vulkan.Dispatch.shader_resources":351.476,"Driver.Packet.memory_wait":323.558,"Driver.Packet.draw":267.333},"trace_dropped":false,"trace":"aps5-trace-dae8651f631a-6360.json.gz"}Traces
aps5-trace-dae8651f631a-6360.json.gz
Digest
50 frames over 29914.2 ms (no summary line found)
Slowest frames (largest slices inside each, inclusive)
- present 90: 1533.1 ms at 26248 ms: Driver.Packet (queue worker 0x20) 1531.1, Driver.Packet (queue worker 0x0) 1528.3, Driver.Packet.draw (queue worker 0x0) 1054.4, Driver.Draw (queue worker 0x0) 1054.3
- present 92: 733.4 ms at 28451 ms: Driver.Packet (queue worker 0x0) 728.9, Driver.Packet (queue worker 0x20) 556.7, Driver.Packet.dispatch (queue worker 0x20) 313.1, Driver.Dispatch (queue worker 0x20) 312.5
- present 93: 730.1 ms at 29184 ms: Driver.Packet (queue worker 0x20) 728.0, Driver.Packet (queue worker 0x0) 725.5, Driver.Packet.dispatch (queue worker 0x20) 345.3, Driver.Dispatch (queue worker 0x20) 344.7
- present 49: 723.9 ms at 2953 ms: Driver.Packet (queue worker 0x20) 721.8, Driver.Packet (queue worker 0x0) 719.1, Driver.Packet.dispatch (queue worker 0x20) 360.4, Driver.Dispatch (queue worker 0x20) 359.7
- present 52: 709.4 ms at 4973 ms: Driver.Packet (queue worker 0x20) 707.5, Driver.Packet (queue worker 0x0) 704.5, Driver.Packet.draw (queue worker 0x0) 348.9, Driver.Draw (queue worker 0x0) 348.8
Threads (ms per frame)
| thread | busy | wait | untraced | in slices < 20 us | largest |
|---|---|---|---|---|---|
| shader prepare | 0.0 (0 %) | 0.0 (0 %) | 598.3 | 0.0 | Shader.PrepareArtifact 0.0 |
| shader prepare | 0.0 (0 %) | 0.0 (0 %) | 598.3 | 0.0 | Shader.PrepareArtifact 0.0 |
| shader prepare | 0.0 (0 %) | 0.0 (0 %) | 598.3 | 0.0 | Vulkan.GraphicsPipelinePrebuild 0.0 |
| queue worker 0x0 | 463.8 (78 %) | 150.5 (25 %) | 0.0 | 24.9 | Driver.Packet 614.3, Driver.Completion 0.0 |
| queue worker 0x20 | 372.0 (62 %) | 243.9 (41 %) | 0.0 | 9.6 | Driver.Packet 602.7, Driver.EndSubmission 2.2 |
| presenter | 2.5 (0 %) | 0.0 (0 %) | 595.8 | 0.1 | Driver.Present 2.5 |
| gpu | 158.6 (27 %) | 0.0 (0 %) | 439.7 | 0.0 | batch 158.6 |
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 1217.0, Driver.Packet.dispatch 494.6, Driver.Dispatch 493.4, Resources.Complete 366.7, Vulkan.Dispatch 363.1, Vulkan.Dispatch.shader_resources 357.4, Driver.Packet.memory_wait 329.8, Driver.Packet.draw 272.5, Driver.Draw 272.4, Driver.Draw.draw_and_resource_release 168.7
Longest waits
- Driver.Packet.memory_wait (queue worker 0x20) 229.1 ms at 12309 ms, present 64
- Driver.Packet.memory_wait (queue worker 0x20) 228.0 ms at 12877 ms, present 65
- Driver.Packet.memory_wait (queue worker 0x20) 226.8 ms at 20996 ms, present 80
- Driver.Packet.memory_wait (queue worker 0x20) 224.7 ms at 5528 ms, present 52
- Driver.Packet.memory_wait (queue worker 0x20) 224.0 ms at 2806 ms, present 48
- Driver.Packet.memory_wait (queue worker 0x20) 221.2 ms at 1527 ms, present 46
- Driver.Packet.memory_wait (queue worker 0x20) 220.5 ms at 13441 ms, present 66
- Driver.Packet.memory_wait (queue worker 0x20) 218.8 ms at 25568 ms, present 88
- Driver.Packet.memory_wait (queue worker 0x20) 216.5 ms at 6741 ms, present 54
- Driver.Packet.memory_wait (queue worker 0x20) 216.4 ms at 4197 ms, present 50
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
- batch 6980: 29.6 ms at 28084 ms, present 91: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 55.9, Driver.Packet.draw (queue worker 0x0) 55.7, Driver.Draw (queue worker 0x0) 55.7, Driver.Draw.draw_and_resource_release (queue worker 0x0) 31.5
- batch 1784: 29.5 ms at 7489 ms, present 56: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 32.3, Driver.Packet (queue worker 0x0) 0.8, Driver.Packet.draw (queue worker 0x0) 0.8, Driver.Draw (queue worker 0x0) 0.8
- batch 2955: 29.5 ms at 11904 ms, present 64: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 31.9, Driver.Packet (queue worker 0x0) 0.6, Driver.Packet.draw (queue worker 0x0) 0.6, Driver.Draw (queue worker 0x0) 0.6
- batch 2401: 29.4 ms at 9949 ms, present 60: GPU no ranges; recorded by Driver.Packet (queue worker 0x0) 33.4, Driver.Packet.draw (queue worker 0x0) 33.2, Driver.Draw (queue worker 0x0) 33.2, Driver.Draw.gpu_mutex_wait (queue worker 0x0) 20.9
- batch 749: 29.4 ms at 2962 ms, present 49: GPU no ranges; recorded by Recorder.Sync (queue worker 0x0) 31.9, Driver.Packet (queue worker 0x0) 1.0, Driver.Packet.draw (queue worker 0x0) 1.0, Driver.Draw (queue worker 0x0) 1.0
Worker stacks
not sampled
Capture
- The caches were warm (the scene ran once and the title was restarted)
- No other APS5_TRACE_* or APS5_PROFILE_* switch was set and the log was not redirected
- Captures compared here come from the same machine, scene and settings
Activity
MarcZX10 commentedon Oct 16, 2026
I traced the reported upload through
ShaderResources::addGuestBuffer()andGuestBufferMemory::UploadFinish(). The descriptor's fullGetSize()is kept unless the device'smaxStorageBufferRangeforces a smaller bound. If no covering host import is available,copyRegion()reads and copies the region's full live span, so clipping the roughly 4 GiB V# would change valid shader addressability.The open import work (#6457, #6479 and #6730) addresses aligned import grouping, budget tracking and repeated refusal costs, but the profile excerpt does not show why this particular range fell back to a copied buffer. Could you capture the same dispatch on current
mainwithAPS5_TRACE_BIGBUF=1andAPS5_TRACE_HOST_IMPORT=1, and share the matching[bigbuf]line plus the import decision/refusal for that range? The descriptor base can be masked; the useful fields are its size/stride,hostImportAlignment, whether a covering import exists, and the refusal reason or import-budget state. That should distinguish an importability issue from a genuinely full-range resource before proposing a safe change.erm14254 commentedon Oct 16, 2026
Tried this on current
main(51acb79) with onlyAPS5_TRACE_BIGBUF=1andAPS5_TRACE_HOST_IMPORT=1set. The title needs the four bring-up pull requests listed in #6256 to get past its earlier stops, so they were rebased on top; none of them touches the import path. Windows 11 (10.0.22631), RTX PRO 6000 Blackwell Workstation Edition (NVIDIA 597.16), WinLibs GCC 15.2.0 r7, Release.It no longer reaches the title screen. About nine minutes in, before the dispatches that copy the 4 GiB range, it aborts in the function traced:
That is
Require((words[1] & 0x40000000u) == 0, ...)inShaderResources::addGuestBuffer. The check already exists at d92a8da, where the same title and pull requests reach the title screen, so something merged since then sends this shader's buffer through it.No
[bigbuf] uploadline was written before the abort (only six writebacks of one 1.3 MiB range), so I can't give you the import decision for the 4 GiB range on currentmainyet. I'll redo the capture once the title reaches its title screen onmainagain.AI-assisted: yes
erm14254 commentedon Oct 16, 2026
Latest plain
mainas well, without the bring-up pull requests (0615cd2, only the same two switches set): the title stops a few seconds after start, on a black screen, at the NVIDIA 597.16 shader-compiler crash listed in #6256:Up to that point the trace shows 6140 host imports, all accepted (the largest 256 MiB each), no refusal, no budget message and no
[bigbuf]line, so the 4 GiB range is never requested before the stop. The dispatch you asked about needs the bring-up pull requests, and with them currentmainaborts as described above.AI-assisted: yes
MarcZX10 commentedon Oct 17, 2026
Thanks for the retest. This narrows the current-main result: the plain-main run stopped in the NVIDIA shader compiler after 6,140 accepted host imports, before the reported 4 GiB dispatch; the bring-up stack stopped earlier on the reserved-bit check. Neither run reaches the import decision for that buffer, so there is still no evidence of a host-import refusal or safe basis for clipping its descriptor range. I will leave the performance path unmodified until a run reaches that dispatch and captures the matching import decision.
erm14254 commentedon Oct 24, 2026
Captured on plain
main2b16484, no other pull requests, with onlyAPS5_TRACE_BIGBUF=1andAPS5_TRACE_HOST_IMPORT=1. Windows 11 (10.0.22631), RTX PRO 6000 Blackwell Workstation Edition, NVIDIA 597.30 (the driver update removed the compiler crash from my last comment), WinLibs GCC 15.2.0 r7, Release, cold shader cache.The title ran to 357 presents, past the point where it shows its title screen, and there is no
[bigbuf]line in the whole log: no region of 1 MiB or more was copied. The import trace has 4,392 host imports, all accepted, none refused and no budget message; the largest is 688.8 MiB (0x3103980000+0x2b0d0000), and the imports are now made when the memory is mapped (by map). So on currentmainthe range is covered by imports and the 4 GiB copy from the report no longer happens.I did not measure frame times here (cold shader cache); the import comparisons from the report are not covered by this run.
AI-assisted: yes