Repository navigation
perf(agc): retain in-use storage textures up to budget ceiling - #7833
authurtabo-byte wants to merge 2 commits into
Conversation
github-actions
Bot
added
perf
|
This pull request is labeled |
github-actions
Bot
removed
the
perf numbers missing
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
Performance report — merge-base
Baseline {"commit":"eaae7e069d18","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_TEXTURE_CACHE_MIB":"1536","APS5_THREAD_PRIORITY":"high","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":1779.360},"presents":5,"frame_ms":{"p50":354.954,"p90":430.090,"p99":430.090,"max":430.090},"gpu_busy_ms_per_present":226.521,"worker_busy_pct":{"queue worker 0x0":97.772,"queue worker 0x20":35.397},"top_phases_ms_per_frame":{"Driver.Packet":472.712,"Driver.Packet.dispatch":173.356,"Driver.Dispatch":170.154,"Driver.Packet.device_lock":140.208,"Graphics.Draw":107.192,"Graphics.Draw.draw_and_resource_release":89.089,"Shader.ResolveAbi":70.415,"Vulkan.Dispatch":70.352},"trace_dropped":false,"trace":"aps5-trace-eaae7e069d18-12676.json.gz"}Change {"commit":"2630f37d0a71","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_TEXTURE_CACHE_MIB":"1536","APS5_THREAD_PRIORITY":"high","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":1761.653},"presents":5,"frame_ms":{"p50":339.948,"p90":417.823,"p99":417.823,"max":417.823},"gpu_busy_ms_per_present":230.117,"worker_busy_pct":{"queue worker 0x0":96.074,"queue worker 0x20":32.057},"top_phases_ms_per_frame":{"Driver.Packet":450.126,"Driver.Packet.dispatch":178.879,"Driver.Dispatch":175.999,"Driver.Packet.device_lock":155.997,"Vulkan.Dispatch":87.076,"Graphics.Draw":83.828,"Vulkan.Dispatch.shader_resources":79.853,"Graphics.Draw.draw_and_resource_release":75.252},"trace_dropped":false,"trace":"aps5-trace-2630f37d0a71-3232.json.gz"}
{"commit":"eaae7e069d18","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_TEXTURE_CACHE_MIB":"1536","APS5_THREAD_PRIORITY":"high","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":1779.36},"presents":5,"frame_ms":{"p50":354.954,"p90":430.09,"p99":430.09,"max":430.09},"gpu_busy_ms_per_present":226.521,"worker_busy_pct":{"queue worker 0x0":97.772,"queue worker 0x20":35.397},"top_phases_ms_per_frame":{"Driver.Packet":472.712,"Driver.Packet.dispatch":173.356,"Driver.Dispatch":170.154,"Driver.Packet.device_lock":140.208,"Graphics.Draw":107.192,"Graphics.Draw.draw_and_resource_release":89.089,"Shader.ResolveAbi":70.415,"Vulkan.Dispatch":70.352},"trace_dropped":false,"trace":"aps5-trace-eaae7e069d18-12676.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 466.7, Driver.WorkerWait 228.2, Driver.WorkerWait.dequeue_order 228.2, batch 220.3, Driver.Packet.dispatch 172.9, Driver.Dispatch 168.7, Driver.Packet.device_lock 139.2, Graphics.Draw 106.9, Graphics.Draw.draw_and_resource_release 88.8, Shader.ResolveAbi 70.4 Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
{"commit":"2630f37d0a71","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_TEXTURE_CACHE_MIB":"1536","APS5_THREAD_PRIORITY":"high","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":1761.653},"presents":5,"frame_ms":{"p50":339.948,"p90":417.823,"p99":417.823,"max":417.823},"gpu_busy_ms_per_present":230.117,"worker_busy_pct":{"queue worker 0x0":96.074,"queue worker 0x20":32.057},"top_phases_ms_per_frame":{"Driver.Packet":450.126,"Driver.Packet.dispatch":178.879,"Driver.Dispatch":175.999,"Driver.Packet.device_lock":155.997,"Vulkan.Dispatch":87.076,"Graphics.Draw":83.828,"Vulkan.Dispatch.shader_resources":79.853,"Graphics.Draw.draw_and_resource_release":75.252},"trace_dropped":false,"trace":"aps5-trace-2630f37d0a71-3232.json.gz"}Slowest frames (largest slices inside each, inclusive)
Threads (ms per frame)
Largest slices (ms per frame, inclusive, all threads): Driver.Packet 444.4, Driver.WorkerWait 230.7, Driver.WorkerWait.dequeue_order 230.7, batch 227.6, Driver.Packet.dispatch 178.4, Driver.Dispatch 174.4, Driver.Packet.device_lock 155.1, Vulkan.Dispatch 86.3, Graphics.Draw 83.5, Vulkan.Dispatch.shader_resources 78.8 Longest waits
Largest GPU batches (GPU ranges inside, and the CPU slices that recorded each, inclusive)
Both traces (decompressed) are attached here: https://gist.github.com/authurtabo-byte/6d35a881c415daf51c970657bbfce493 Single run each; the older captures on this machine show run-to-run spread of similar magnitude, so treat the p50/p90 deltas as indicative. Overlap review of |
fogydes
left a comment
fogydes
left a comment
There was a problem hiding this comment.
Verified on Windows 11 with MinGW-w64 GCC 15.2.0 (Release):
- Architecture & Performance: Introducing
StorageLimits {budget, ceiling}incore/libs/prx/libSceAgcDriver/Graphics/src/ShaderResources.cppmirrors the retention strategy ofadmitTexture()for sampled textures. Allowing in-flight textures (back->texture.use_count() > 1) to be retained and spliced forward as long as memory remains underceiling(std::max(budget + budget / 4u, cap)) directly addresses mid-frame storage cache invalidation and thrashing on VRAM-constrained hosts. - Conventions: No code comments added;
tools/check_conventions.pypasses cleanly.
Suggestions:
- Consider documenting this elevated ceiling and in-flight retention behaviour in
docs/dev/debt/libSceAgcDriver.mdunder### Functional->Texture cache budgets. - Consider adding an automated test fixture in
core/libs/prx/libSceAgcDriver/tests/execution/StorageCacheBudget.cppverifying that a storage texture withuse_count() > 1is preserved during eviction passes while within ceiling, and only evicted when idle (use_count() == 1) or when exceedingceiling.
adriadam10
left a comment
adriadam10
left a comment
There was a problem hiding this comment.
Thanks for the capture. A few things before this can go in:
- The headline numbers (p50 583 ms → 362 ms, 300,000 → 52,650 storage creations) compare
mainwithoutAPS5_TEXTURE_CACHE_MIBagainst the PR; your later same-settings capture (APS5_TEXTURE_CACHE_MIB=1536, same scene) shows 355 ms → 340 ms, which is within noise. Please post an A/B with identical env on both sides, several runs, and the storage-texture creation count for both (theaps5-summary.jsonllines don't carry it). - Memory: this keeps in-flight storage textures alive past the budget up to
ceiling. Please state the bound in the description (ceiling vs. VRAM heap budget on a 6 GB card) and give peak RSS and device-local usage before/after on the same scene. - Overlap: #6194 (grow the storage cache budget under eviction pressure) targets the same thrashing in
cachedStorageTexture/storageBudget, and #6202, #6292, #6615, #6718 touch the same functions. Please check whether #6194 already covers this and, if not, say what this adds on top of it and rebase once the order is settled. - No test pins the new retention behaviour; a unit test with a held
shared_ptrentry that survives eviction under the ceiling and is evicted over it would show the branch at ShaderResources.cpp:1119-1131 works.
|
Review follow-up (addresses the changes requested):
Given the A/B result above, the frame-time motivation is not reproducible on this scene; the change trades a small p50 regression for bounded residency. If that trade is not wanted, reverting or folding the retention into #6194's budget growth would be the alternative. |
|
Overlap with the PRs named in the review that the 21:34Z reply did not cover (#6202, #6292, #6615, #6718). Method:
Hunks are not yet applied on top of each other, so whether they conflict textually is not tested. Rebase order is left to the maintainers. |
|
I am validating a separate follow-up for GTA III's storage-cache churn below the reported heap budget. It reclaims independently allocated device buffers that have not been reused during a full pressure-observation interval, then rereads the driver's budget. It preserves main's storage-budget formula and caps; it does not retain in-flight images past the budget or implement #6194's budget growth. The #4472 author confirmed this case is separate and offered to rebase around it: #4472 (comment). Your The local branch passes the same 28 focused Vulkan-validation cases as its baseline. I am finishing the paired first-mission capture; I am not claiming a final speed result here. Prepared with AI assistance. |
|
Agreed on your order: land the cold-buffer release first, then this retention change rebases on top of it, preserving your reclaim-and-reread step in One note from our side: with Please link the follow-up PR here when you open it so the rebase lands against the right base. |
adriadam10
left a comment
adriadam10
left a comment
There was a problem hiding this comment.
Re-checked against current main; the head is unchanged and the earlier points still apply:
- No measured gain: with identical env on both sides p90 is unchanged and p50 is ~7.6% slower (315.9 → 340.0 ms), while RSS and device-local usage go up. Please show a title/GPU where the change wins under the same
APS5_TEXTURE_CACHE_MIBon both sides (storage creations down, frame time better), or close this. - Overlap: #6194, #6202, #6292, #6615, #6718 and the cold-buffer-release follow-up touch
cachedStorageTexture()/storageBudget(). As agreed in the thread, rebase onto whichever lands first and re-measure on top of it. - Test:
StorageCacheRetention.cpppins onlyKeepStorageOnBudgetEvict. The loop atShaderResources.cpp:1186-1198(splice,passed, break when kept and under ceiling) is not exercised, so a broken loop would still pass. Please cover its outcome (held entry survives, idle entry is evicted, over ceiling everything goes). - Minor:
overatShaderResources.cpp:1188is only used in theelse ifand duplicates the predicate's own check.
|
The agreed follow-up is now #8672, at Prepared with AI assistance. |
ea0fcec to
79146e6
Compare
|
@adriadam10 Thanks for the follow-up review. All 4 points have been addressed in the latest push:
|
|
AMD/Linux data point on another title with a large Setup
In this menu scene, dispatch shader resources cost about the same with and without the PR. So the cost here does not seem to come from storage textures evicted and re-created at the budget. There is no regression either: no GPU events and no new errors. With 13–14 frames per window the frame times are noisy, so I would not read the p50 difference as a slowdown. AI-assisted: yes (Claude Code); results above were run on this machine. |
|
A data point from 007 First Light (PPSA11386, content 01.018.000) on Linux 7.2.10, RTX 3080 10 GB, driver 615.78.08, Ryzen 9 7950X3D, 64 GB, Release builds, 3840x2160, warm shader cache, identical environment on both sides (
Storage-budget collapses with less headroom. From early gameplay, before the first prompt scene at about present 1,600, the storage image cache budget can fall to its 512 MiB floor and keep returning to it (P 11, 13 and 21 times, S 17, in the collapsed runs below). Those runs reached presents 1,596-1,689 in their 1,500 s, the others 2,101-2,227. With the heap budget the texture budgets see lowered by 500 MiB and by 250 MiB:
S's one collapse came with device memory full: 9,906 of 10,240 MiB in use (all processes), 305 device allocations that failed with Cost in the runs that did not collapse (both series above, plus 3 runs per side at default headroom stopped at present 1,580):
Early gameplay at default headroom (present 1,420, shortly after gameplay start, to the stop at 1,580, about 160 frames, 3 runs per side), per frame:
Storage image creations are not lower with S in this stretch; sampled texture creations are (I have not traced why). Storage creations inside the prompt scene were not compared. Frame time p50 from each run's gameplay start to the stop (138-185 presents per run; presents-weighted mean of the ~60 s summary windows' p50): P 1,499 / 1,367 / 1,317 ms, S 1,255 / 1,239 / 1,476 ms. So in these runs on this title and card, fewer reached the storage-budget floor with S (1 of 5 against 3 of 5, five runs per side), at about +420-440 MiB of device memory and more out-of-memory recoveries; the run where S did collapse was the memory-full kind. Early-gameplay FPS: mean 0.51 on both sides; frame p50 overlaps (P 1,317-1,499, S 1,239-1,476 ms). AI-assisted: yes (Claude Code) |
|
@authurtabo-byte I opened draft #10194 for the opt-in allocation breakdown requested in #10041. The overlap audit flags the storage-budget calculation: your My proposal is to keep your existing merge order and rebase the accounting work after yours lands, preserving your ceiling/retention policy and reporting the resulting budget snapshot. Your PR does not need to wait for #10194. Does that order work for you? If the accounting lands first, I can help resolve the reporting hook here. The accounting branch builds independently and changes no cache policy. Its test results and overlap details are in #10194. AI-assisted: yes. |
adriadam10
left a comment
adriadam10
left a comment
There was a problem hiding this comment.
Re-checked the new head (79146e64f0). The test point is resolved: the loop is now EvictStorageOnBudget and the new cases pin held-survives, idle-evicted, over-ceiling-evicts-all and unaccounted-survives; over is gone. Remaining:
- Gain not reproduced outside your machine. The default-budget win (p50 583 → 362 ms) is from one RTX 3050 6 GB laptop; with the same
APS5_TEXTURE_CACHE_MIBon both sides the A/B was 7.6% slower at p50. Independent runs: Code Violet on RX 6650 XT shows no difference; 007 First Light on RTX 3080 10 GB shows the same FPS, storage creations not lower, and +~420 MiB device memory with 9-16 recoveredVK_ERROR_OUT_OF_DEVICE_MEMORYallocations per run against 1-5 without the PR (fewer budget collapses, 1/5 vs 3/5). Please show a second machine where frame time or creation count improves, or scope the retention so it does not hold extra memory on cards where it doesn't help (docs/rules memory bound: it must pay for itself). - Ordering: as agreed, #8672 lands first (#6194 and #6202 also touch
cachedStorageTexture()/storageBudget()); rebase on top of it, keep its reclaim-and-reread step, and re-measure, since the ceiling depends on the budget those set. #10194 also touchesstorageLimits.
|
A correction to my comment of 2026-10-22 (#7833 (comment)), the "Peak RSS (VmHWM)" line under Cost in the runs that did not collapse. Those figures were each run's highest VmRSS sample, not its VmHWM: the harness reads both from the title's
VmHWM is above the highest sample by 12 to 1,277 MiB per run (S 167-1,277, P 12-525; a 10 s sample can miss the peak). The VmHWM means have S 15 MiB below P; the posted figures had S 601 MiB below P. No other number in that comment came from this column, and its summary did not use it. Both columns count resident memory only; zram swap was allowed for the title in all 12 runs, on both sides. AI-assisted: yes (Claude Code) |
What
Aligns storage texture cache eviction in
cachedStorageTexture()with the retention and ceiling pattern already established for sampled textures inadmitTexture():StorageLimits {budget, ceiling}caching device budget and ceiling (std::max(budget + budget / 4u, cap)).back->texture.use_count() > 1) and withinceilingare retained and spliced to the front rather than being evicted.use_count() == 1) or if total memory exceedsceiling.EvictStorageOnBudget(...)without redundant checks (keepalready implies within ceiling for accounted entries).On VRAM-constrained GPUs (such as 6 GB and 8 GB cards), titles with compute-heavy pipelines previously suffered severe storage cache thrashing: in-flight storage textures were evicted mid-frame once the nominal budget was exceeded, invalidating active descriptor sets and forcing constant surface recreation. Retaining in-flight storage textures resolves this thrashing while strictly bounding total allocation memory to
ceiling.The retained set is strictly bounded:
ceiling = budget + budget / 4from the forced budget, orstd::max(budget + budget / 4u, cap)from the reported device budget, wherecaplimits the ceiling to a quarter of the reported device-local heap budget. On a 6 GB card with ~4.8–5.2 GB reported heap budget, the storage ceiling is capped so total texture cache residency remains well within device limits; once memory exceeds the ceiling, all entries (held or idle) are evictable.Overlap & Coordination
mainyet; this branch is cleanly rebased on currentupstream/main. If perf(agc): reclaim idle device buffers under storage cache pressure #8672 merges first, this branch will rebase on top preserving the cold-buffer reclaim step incachedStorageTexture().Tested
Tested on Windows 10.0.26200, NVIDIA GeForce RTX 3050 6GB Laptop GPU, driver 617.42, MinGW GCC 15.2.0 (Release).
Platform 8 (PPSA18288), title sequence / gameplay, 3840x2160.
Identical-Environment A/B (Default Budget Under Eviction Pressure)
Testing with
APS5_TEXTURE_CACHE_MIB=1536on both sides masked eviction pressure because 1536 MiB exceeds the game's working set (~600–800 MiB), preventing eviction on bothmainand the candidate.Under default dynamic budget (unsetting
APS5_TEXTURE_CACHE_MIBon both sides, identical env:{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_PROFILE_DRAW":"1","APS5_TRACE_FRAMES":"5:5"}):main(eaae7e069d18)2630f37d0a71)Summary lines from
aps5-summary.jsonl:{"commit":"eaae7e069d18","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_PROFILE_DRAW":"1","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":4911.514},"presents":5,"frame_ms":{"p50":583.175,"p90":2577.531,"p99":2577.531,"max":2577.531},"gpu_busy_ms_per_present":353.385,"worker_busy_pct":{"queue worker 0x0":98.999,"queue worker 0x20":50.487},"top_phases_ms_per_frame":{"Vulkan.GraphicsPipeline":3259.426,"Vulkan.GraphicsPipelineLibrary":3229.993,"Vulkan.GraphicsPipelinePrebuild":2921.679,"Driver.Packet":1465.998,"Driver.Packet.dispatch":547.798,"Driver.Dispatch":539.674,"Shader.PrepareArtifact":520.19,"Graphics.Draw":517.02},"trace_dropped":false,"trace":"aps5-trace-eaae7e069d18-20576.json.gz"} {"commit":"2630f37d0a71","dirty":false,"env":{"APS5_HANDOFF_SPIN_US":"150","APS5_PIPELINE_DISPATCH":"1","APS5_PIPELINE_DMA":"1","APS5_PREBUILD_THREADS":"8","APS5_PROFILE_DRAW":"1","APS5_TRACE_FRAMES":"5:5"},"gpu":"NVIDIA GeForce RTX 3050 6GB Laptop GPU","driver":"NVIDIA 617.42","os":"Windows 10.0.26200","resolution":"3840x2160","window":{"spec":"frames 5:5","first_present":5,"ms":1895.614},"presents":5,"frame_ms":{"p50":361.846,"p90":449.147,"p99":449.147,"max":449.147},"gpu_busy_ms_per_present":253.697,"worker_busy_pct":{"queue worker 0x0":96.505,"queue worker 0x20":30.926},"top_phases_ms_per_frame":{"Driver.Packet":481.779,"Driver.Packet.dispatch":167.223,"Driver.Dispatch":164.551,"Driver.Packet.device_lock":159.063,"Graphics.Draw":98.701,"Graphics.Draw.draw_and_resource_release":84.356,"Vulkan.Dispatch":78.423,"Vulkan.Dispatch.shader_resources":72.688},"trace_dropped":false,"trace":"aps5-trace-eaae7e069d18-9108.json.gz"}Unit Tests
agc_storage_retention_tests(ctest -R agc_storage_retention) exercises bothKeepStorageOnBudgetEvictand the complete eviction loop inEvictStorageOnBudget:testHeldEntrySurvivesUnderCeiling: held entry survives under and at ceiling.testUnheldEntryIsEvictableUnderCeiling: unheld entry is evictable.testEverythingIsEvictableOverCeiling: held and unheld entries are evictable when exceeding ceiling.testUnaccountedEntryAlwaysSurvives: unaccounted entries survive.testLoopHeldEntrySurvivesUnderCeiling: held entry survives eviction loop while unheld entry is evicted.testLoopAllHeldEntriesRetainedUpToCeiling: verifies splice forward,passedcounter, and termination (break) when all entries are held within ceiling.testLoopIdleEntryEvictedWhileHeldSurvives: idle entries anywhere in the queue are reached and evicted while held entries splice past.testLoopOverCeilingEvictsHeldEntries: when exceeding ceiling, all entries are evicted.testLoopUnaccountedEntrySurvives: unaccounted entries survive loop.python tools/check_conventions.py --base upstream/main: All conventions checks passed.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)