Skip to content

perf(agc): prebuild registration libraries on half the hardware threads by default - #10388

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
Tamalero:perf/prebuild-pool-threads
Oct 27, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
Tamalero:perf/prebuild-pool-threads

Conversation

@Tamalero

Copy link
Copy Markdown
Contributor

What

On a device with graphics pipeline libraries, the registration-form vertex and pixel libraries are linked on the library prebuild pool. Since 361c7e2 that pool has one thread unless APS5_PREBUILD_THREADS says otherwise. On NVIDIA it is the startup bottleneck. With a warm shader cache, High on Life (PPSA15388) spends about 50 s with a single busy thread before its first present. perf record -g on that thread shows VulkanDevice::PrebuildGraphicsLibraries → Graphics::Pipeline::Prebuild → LinkPipelineFromLibraries → the NVIDIA compiler, with 96% of its samples in the compiler.

Change: the prebuild pool now defaults to the size of the registration preparation pool: half the hardware threads, at least two. APS5_PREBUILD_THREADS still sets the count, and APS5_PREBUILD_THREADS=1 gives the old behaviour. RegistrationPrepareStack counts the new default, and the docs/dev/ENVIRONMENT.md row describes it. agc_driver_shutdown_threads_queued_prebuilds now runs with APS5_PREBUILD_THREADS=1. It blocks one prebuild and checks that close drops the tasks queued behind it, and with more workers those tasks start right away on the other threads.

Tested

Hosts:

  • Linux 7.3 (CachyOS), i9-14900K (32 threads), RTX 4090 with NVIDIA 615.78.08, GCC 16.2, Release.
  • AMD leg: Ubuntu 24.04, EPYC 7452 (24 vCPUs), Radeon Pro V620 with RADV (Mesa 25.2.8).

Method: warm shader cache, no input. Time is measured from launch to the title's first present.

Title GPU Build Threads First present
PPSA15388 RTX 4090 main e13027a 1 76 s, 64 s
PPSA15388 RTX 4090 this PR 2a3cc34 16 (default) 11 s, 9 s
PPSA15388 RTX 4090 main 74f6fa8 / the same change on it 1 / 16 61 s, 62 s, 60 s, 71 s, 62 s / 10 s, 11 s, 10 s, 10 s, 9 s, 9 s, 9 s
PPSA15388 RTX 4090 main c188e01 1 / APS5_PREBUILD_THREADS=16 61 s, 60 s / 10 s, 9 s
PPSA01474 RTX 4090 main c188e01 1 / APS5_PREBUILD_THREADS=16 21 s, 13 s / 7 s, 5 s
PPSA01474 V620 (RADV) 74700a0 APS5_PREBUILD_THREADS=1 / =12 25 s, 24 s / 25 s, 25 s, 25 s
  • Four runs (three on main, one with this change) crashed after their first present, in a guest-side race of this title that predates both builds. Their first-present times are included.
  • On NVIDIA, startup to the first present is about 6× faster for PPSA15388 and 2–3× faster for PPSA01474. While prebuilding, 15–17 threads are busy instead of one.
  • On RADV the device also takes the pipeline-library path (pipelineLibrary=1), but the start time doesn't change. Linking isn't the bottleneck there, so the change neither helps nor hurts.
  • Gameplay: I checked whether the extra prebuild threads compete with the title during play. The test was PPSA15388 on c188e01, the same walking route twice, with APS5_TRACE_SUMMARY_EVERY=10. Over the walking intervals:
Threads Intervals / frames Frame p50 p90 p99 Worst
1 8 / 588 116 ms 248 ms 538 ms 950 ms
16 6 / 451 109 ms 229 ms 490 ms 876 ms

No regression.

Summary lines: PPSA15388 title screen, APS5_TRACE_SECONDS=10:10, same machine. These are the runs whose traces are attached. Once the title is up, frame times and GPU time are unchanged:

{"commit":"e13027a1f399","dirty":false,"env":{"APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_SECONDS":"10:10"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-3-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 10:10","first_present":362,"ms":10010.008},"presents":507,"frame_ms":{"p50":16.684,"p90":16.881,"p99":61.894,"max":789.127},"gpu_busy_ms_per_present":7.541,"worker_busy_pct":{"queue worker 0x0":69.483,"queue worker 0x20":58.327},"top_phases_ms_per_frame":{"Driver.Packet":25.107,"VideoOut.Flip":13.152,"VideoOut.Flip.vblank_wait":11.167,"Driver.Packet.memory_wait":10.184,"Driver.Packet.dispatch":6.601,"Driver.Dispatch":4.337,"Driver.Packet.pipeline_drain":4.113,"Graphics.Draw":3.794},"trace_dropped":false,"trace":"aps5-trace-e13027a1f399-2.json.gz"}
{"commit":"2a3cc34582af","dirty":false,"env":{"APS5_PLAYGO_UNLISTED_CHUNKS":"installed","APS5_TRACE_SECONDS":"10:10"},"gpu":"NVIDIA GeForce RTX 4090","driver":"NVIDIA 615.78.08","os":"Linux 7.3.0-3-cachyos","cpu":"Intel(R) Core(TM) i9-14900K","cpu_threads":32,"memory_mib":64108,"title_id":"PPSA15388","title":"High on Life","content_version":"01.120.000","resolution":"3840x2160","window":{"spec":"seconds 10:10","first_present":350,"ms":10011.932},"presents":500,"frame_ms":{"p50":16.683,"p90":18.011,"p99":69.776,"max":852.701},"gpu_busy_ms_per_present":7.070,"worker_busy_pct":{"queue worker 0x0":72.595,"queue worker 0x20":61.289},"top_phases_ms_per_frame":{"Driver.Packet":26.666,"VideoOut.Flip":12.675,"Driver.Packet.memory_wait":10.893,"VideoOut.Flip.vblank_wait":10.636,"Driver.Packet.pipeline_drain":5.999,"Driver.Packet.dispatch":5.616,"Driver.Dispatch":4.671,"Graphics.Draw":3.941},"trace_dropped":false,"trace":"aps5-trace-2a3cc34582af-2.json.gz"}

AGC tests:

  • On this PR's head and main e13027a, the 39 tests matching prebuild|registration_prepare|prepared_|shutdown_threads pass on both, with the same validation errors: vkCreateDevice-ppEnabledExtensionNames-01387 ×108 and VkPipelineColorBlendStateCreateInfo-pAttachments-00605 ×14.
  • Before the rebase, all 686 agc_driver tests were run with ctest -V -j8 on main 74f6fa8 and this change, with VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation ANYPS5_NO_SHADER_CACHE=1 ANYPS5_REQUIRE_VULKAN=1, on the RTX 4090.
  • Neither log reports the layer as not found.
  • Both logs have the same validation errors with the same counts, so this PR adds no new VUID-: vkCreateDevice-ppEnabledExtensionNames-01387 ×1506, VkWriteDescriptorSet-descriptorType-00337 ×18, VkPipelineColorBlendStateCreateInfo-pAttachments-00605 ×18, VkImageMemoryBarrier-image-03320 ×6.
  • Failures:
  • Fixed in this PR: agc_driver_shutdown_threads_queued_prebuilds failed with the new default ("closing ran the queued library prebuilds"). With one blocked worker, the other workers ran the queued prebuilds. Its environment now pins APS5_PREBUILD_THREADS=1. All nine agc_driver_shutdown_threads* tests pass.
  • Negative check: with ShaderRegistry.cpp from main, agc_driver_registration_prepare_stack fails with "the preparation and prebuild pools count 17 live threads of the 32 they started". With this PR it passes.
  • tools/check_conventions.py passes. tools/env_reference.py --check reports only APS5_NO_WGI, which is undocumented on main too.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 4090, 615.78.08, Linux 7.3 (all agc_driver tests and titles); AMD Radeon Pro V620, RADV Mesa 25.2.8, Ubuntu 24.04 (title start only)
  • AI-assisted: yes

🤖 Generated with Claude Code

Tested and written by Claude (AI assistant) on a contributor's machine.

…ds by default

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the perf Performance change label Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1368 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1395 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 1896 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 2434 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 4449 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5021 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5371 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 5925 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6187 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6188 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 6192 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6209 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6280 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6300 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6412 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6481 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6808 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 6924 docs/dev/ENVIRONMENT.md
PR 6977 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 7048 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7201 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7275 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 7449 docs/dev/ENVIRONMENT.md
PR 7457 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7514 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7604 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 7642 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
docs/dev/ENVIRONMENT.md
PR 7682 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 8004 docs/dev/ENVIRONMENT.md
PR 8246 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 8484 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 8542 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 8592 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 8963 docs/dev/ENVIRONMENT.md
PR 9012 docs/dev/ENVIRONMENT.md
PR 9129 docs/dev/ENVIRONMENT.md
PR 9176 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9248 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9287 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9289 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9345 docs/dev/ENVIRONMENT.md
PR 9356 docs/dev/ENVIRONMENT.md
PR 9387 docs/dev/ENVIRONMENT.md
PR 9431 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9432 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9508 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
docs/dev/ENVIRONMENT.md
PR 9672 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9673 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9674 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9675 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9676 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
docs/dev/ENVIRONMENT.md
PR 9748 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 9842 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9852 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 9891 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 9897 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10026 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10043 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10193 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
docs/dev/ENVIRONMENT.md
PR 10233 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 10435 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
PR 10442 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10757 core/libs/prx/libSceAgcDriver/CMakeLists.txt
docs/dev/ENVIRONMENT.md
PR 10773 core/libs/prx/libSceAgcDriver/CMakeLists.txt
PR 11207 core/libs/prx/libSceAgcDriver/CMakeLists.txt
core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp
docs/dev/ENVIRONMENT.md

@Tamalero

Copy link
Copy Markdown
Contributor Author

@babyhuey

Copy link
Copy Markdown
Contributor

Windows run of this head 2a3cc34582a, merged onto main 710cf9c0352 (clean merge, local only), with the selection CI's skipped Windows product job uses: Windows 11, Ryzen 7 5800X, RTX 3080, WinLibs GCC 15.2, Release, BUILD_TESTING=ON. The tree was configured fresh after the merge, then select_tests.py select / ninja -f select.ninja selected / --target libs / select_tests.py ctest --parallel 4 --timeout 120, with check_skips.py --job windows.

AI-assisted: yes (Claude Code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agc libSceAgc* build docs perf Performance change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants