Repository navigation
perf(agc): import adjacent host ranges as aligned blocks and stop re-trying budget refusals - #6457
Merged
oneandonlydean merged 22 commits intoOct 19, 2026
Conversation
Every host import now names the path that asked for it: - map: the map-time import of a GPU mapping; - build: an address-based build's walk and its region imports; - access: a lookup through HostImportFor; - join: a join at bind; - segment: a storage image write-back segment; - expand: a DB expand's store. The APS5_TRACE_HOST_IMPORT ok line, the import failure line and the budget refusal report end with the requester, so a traced run splits its imports by path. HostImportCounters returns the imports made, refused and failed per requester, with the live count and bytes. agc_driver_graphics checks that each requester's imports, refusals and failures are counted under it.
A run without APS5_TRACE_HOST_IMPORT had no count of the imports it made or refused. When the driver stops, one line now gives the live imports and the imports made, refused and failed, in total and per requester: [gpu] host imports at shutdown: N live (M MiB); made ... (... MiB), refused ... (... MiB), failed ...; by requester (made/refused/failed): map a/b/c, build ..., access ..., join ..., segment ..., expand ... It prints only when an import was made, refused or failed. agc_driver_graphics checks the line's text.
The storage write-back overhang test read both surfaces straight from guest memory after FlushAllPending and WaitIdle. A write-back through a host import is only recorded there; a reader of guest memory calls FlushGpuWrites, which waits for the batch that holds it. The test passed because the split surface took the CPU write-back, whose SubmitAndWait also drained the first surface's recorded store. With both surfaces imported, neither store had landed when the test read them.
…blocks An import covered one registered range unless adjacent ranges filled a whole aligned 8 MiB window. A title that maps its GPU heap in 64 KiB pieces got one import per piece from every requester: Village held about 27,000 of them at its first scene, enough to fill the budget. The run of an address is now the largest aligned power-of-two block, from twice the import alignment up to the window, that holds the address and that continuing readable ranges fill exactly. A block over the budget steps down to the largest block that fits what is left of it, counting what the requester may evict, and nothing is tried when not even the address's own range fits, so a full budget costs one refusal per request as before. A block that would replace an import a build holds, or that fails (a refused Windows alias, vkAllocateMemory), steps down to the next smaller block. A failed block is remembered until one of its ranges changes, instead of pinning the span to single ranges for the session. An address-based build no longer asks for the ranges inside the block it just imported. A storage image over 64 adjacent pieces now finds its block import at write-back instead of importing every piece as a segment. The split write-back test keeps its two ranges apart with a 64 KiB window.
An address-based build asks for every registered GPU range whenever it rebuilds its address space, and at a full budget it was refused each one again: a title held at its default budget counted 304,394 refusals over about 12,000 ranges at its title screen. A request that may not evict (the build's walk, an import when a range is mapped) now remembers a refused run with the live bytes and the budget at the refusal. Until live imports drop below those bytes, the budget changes or one of the run's ranges changes, a request for that range is answered from memory, still counted as refused and also as remembered, and a remembered block is skipped, so a block stepped down for the budget keeps its import across registry changes. A request that may evict always tries again, and a new import forgets the refusals of the runs it covers. The shutdown report and the budget line print how many refusals were remembered.
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
11 tasks done
…ite-back test Its registration and its command-line option followed the split write-back test's, where another open pull request adds its own. They now follow the copied-direct ordering and CMASK tests. No test changes.
EagleBearerYS
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 15, 2026
Expand named a DB expand's store, but nothing passes it: main has no DB expand, and the open DB expand change stores its planes only where host imports already hold them, through HostImportOverlaps and the storage write-back, without making or retiring an import. The requester could never count, so the shutdown line always ended with expand 0/0/0. agc_driver_graphics checks the requester names and the shutdown report without it.
11 tasks done
This was referenced Oct 16, 2026
This was referenced Oct 16, 2026
main 52798f7 declares ExistingHostImportFor next to HostImportFor (boykopovar#6892). Keep it beside this branch's HostImportFor, which takes a requester.
12 tasks done
Main keeps an import settle's objects per device in ImportSettle, next to where this branch adds ImportRefusal. Keep both. Main's fallback for a refused host-write alias returns after this branch's failure records.
oneandonlydean
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 19, 2026
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp # docs/dev/debt/libSceAgcDriver.md
oneandonlydean
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 19, 2026
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp # docs/dev/debt/libSceAgcDriver.md
github-actions
Bot
added
needs rebase
# Conflicts: # docs/dev/debt/libSceAgcDriver.md
github-actions
Bot
added
needs rebase
# Conflicts: # docs/dev/debt/libSceAgcDriver.md
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
oneandonlydean
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 19, 2026
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
oneandonlydean
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 19, 2026
# Conflicts: # core/libs/prx/libSceAgcDriver/Graphics/src/GuestBufferMemory.cpp
oneandonlydean
pushed a commit
that referenced
this pull request
Oct 19, 2026
…-import-blocks #6457's branch now carries main's per-device import settles, which this branch's GuestBufferMemory.cpp conflicted with.
This was referenced Oct 19, 2026
oneandonlydean
pushed a commit
that referenced
this pull request
Oct 19, 2026
…mber-evicting-refusals #6457's branch now carries main's per-device import settles. Its ImportSettle sits where this branch adds EvictingRefusal; keep both.
12 tasks done
12 tasks done
wachi35676
added a commit
to wachi35676/AnyPS5
that referenced
this pull request
Oct 20, 2026
Since host imports group adjacent ranges into aligned blocks (boykopovar#6457), an import can span 64 KiB blocks an earlier unwatched import left behind, and resident multisampled DCC targets with their metadata. ImportWatched with staysWatched took the whole import out of the write watch when any part of it was not watched before. It now walks and compares each run of watched blocks of the import on its own, and unwatches only the blocks that were not watched or whose walk failed. A resident DCC target took the generation of its metadata from a walk of the metadata's pages alone. CPU stores to the rest of the metadata's 64 KiB block made before that walk were stamped later by the import's walk, so the block read as written, and the import window marked the metadata page as possibly stored over: the next draw refused the target. The generations now come from a walk of the whole blocks. Fixes boykopovar#8103 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wachi35676
added a commit
to wachi35676/AnyPS5
that referenced
this pull request
Oct 20, 2026
Since host imports group adjacent ranges into aligned blocks (boykopovar#6457), an import can span both 64 KiB blocks that an earlier import left unwatched and the metadata of resident multisampled DCC targets. ImportWatched with staysWatched took the whole import out of the write watch when any part of it was not watched before. It now walks and compares each run of watched blocks of the import on its own, and unwatches only the blocks that were not watched or whose walk failed. A resident DCC target took the generation of its metadata from a walk of the metadata's pages alone. CPU stores to the rest of the metadata's 64 KiB block made before that walk were stamped later by the import's walk, so the block read as written, and the import window marked the metadata page as possibly stored over: the next draw refused the target. The generations now come from a walk of the whole blocks. agc_write_tracking and agc_driver_msaa_dcc_clear_* gain regression tests for both. Fixes boykopovar#8103 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 tasks done
12 tasks done
12 tasks done
EagleBearerYS
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 21, 2026
…atch On Windows a host import under 1 MiB leaves the write watch when it is made, and comes back only when its deferred settle read completes. retireImport drops the pending settle and never restores the watch, so an import retired before it settled left its range outside the write watch for good. Once no import fit the budget, an address-based build could neither import such a range nor heap mirror it (CollectWrites returned 0), and copied it. Since aligned block imports (boykopovar#6457), this happens to every 64 KiB piece a title maps one at a time: the piece is imported by the map, and when its neighbour is registered its run grows into a larger aligned block, so refreshImports retires the piece's import ("the range was mapped again", though the range is unchanged). Resident Evil Village (PPSA01556) on main 6696def aborted in its intro at 0:43 to 6:37 with "an address-based build copies 185 MiB of 2960 registered ranges, past 64 MiB (APS5_ADDRESS_COPY_MAX_MIB); the largest: 0x20c0c20000+ 0x10000 (0.1 MiB committed, outside the write-watched arena) ...". An import trace showed each copied page imported by the map, retired when its run grew, and never imported or watched again. refreshImports and the budget eviction now note the ranges of each unwatched import they retire that are still registered as the same range, and rewatch them once the import is gone: as a settled import does, except where another unwatched import or a live storage texture still covers them. Rewatch stamps the range as written, so the next build reads it in full. Ranges the guest mapped again or released keep the existing path. Tested: agc_driver_graphics gains retiredImportWatchTests (Windows): a 64 KiB piece imported on its own, then its neighbour registered and the budget cut below one piece. The piece must be back in the write watch and both pieces heap mirrored. Without the fix the piece stays unwatched and is copied.
EagleBearerYS
added a commit
to EagleBearerYS/AnyPS5
that referenced
this pull request
Oct 21, 2026
…atch On Windows a host import under 1 MiB leaves the write watch when it is made, and comes back only when its deferred settle read completes. retireImport drops the pending settle and never restores the watch, so an import retired before it settled left its range outside the write watch for good. Once no import fit the budget, an address-based build could neither import such a range nor heap mirror it (CollectWrites returned 0), and copied it. Since aligned block imports (boykopovar#6457), this happens to every 64 KiB piece a title maps one at a time: the piece is imported by the map, and when its neighbour is registered its run grows into a larger aligned block, so refreshImports retires the piece's import ("the range was mapped again", though the range is unchanged). Resident Evil Village (PPSA01556) on main 6696def aborted in its intro at 0:43 to 6:37 with "an address-based build copies 185 MiB of 2960 registered ranges, past 64 MiB (APS5_ADDRESS_COPY_MAX_MIB); the largest: 0x20c0c20000+ 0x10000 (0.1 MiB committed, outside the write-watched arena) ...". An import trace showed each copied page imported by the map, retired when its run grew, and never imported or watched again. refreshImports and the budget eviction now note the ranges of each unwatched import they retire that are still registered as the same range, and rewatch them once the import is gone: as a settled import does, except where another unwatched import or a live storage texture still covers them. Rewatch stamps the range as written, so the next build reads it in full. Ranges the guest mapped again or released keep the existing path. Tested: agc_driver_graphics gains retiredImportWatchTests (Windows): a 64 KiB piece imported on its own, then its neighbour registered and the budget cut below one piece. The piece must be back in the write watch and both pieces heap mirrored. Without the fix the piece stays unwatched and is copied. Co-authored-by: keravyn-ops <342686842+keravyn-ops@users.noreply.github.com>
11 tasks done
13 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A title that maps its GPU heap in 64 KiB pieces got one host import per piece. Resident Evil Village holds about 27,600 such ranges by its first scene and fills the import budget. At a full budget, every rebuild of the address-based build's address space asked again for every range the budget had refused.
APS5_TRACE_HOST_IMPORTlines;[gpu] host imports at shutdown: ….FlushGpuWrites, as readers of guest memory must.The worst case. A window that fills, or empties, one piece at a time with builds in between pays for each block upgrade: every doubling imports the block again, the old half included.
Overlap with #6499 (parl0v, the #6060 follow-up): it keeps an import whose head or tail a guest protect or unmap cuts off, re-attached to the surviving range with a new
VkBufferon the same memory.ImportWatched,replaceRangeor what a split keeps.GuestBufferMemory.cppin three places: the replaced-bytes sum inimportAllocation, the fields set on a new import, and the start ofrefreshImports. Whichever merges second keeps both sides, with the replaced-bytes sum counting perf(agc): keep a host import across a split of its guest range #6499'spinnedBytes. Either order works.With #7339 and #6678 (both merged): this branch's merge of
main3de84fa (399cc69) takes #7339's guard over a host import while it is built, with this PR's signature and requester counts. On the failure path that retries as smaller blocks, the failed attempt's alias is unmapped before the retry and handed to the guard as null, so it is unmapped once. #6678's settle structures and calls sit beside this PR's refusals, as its description lists. #7339's copy lifetime check mapped its source and destination back to back, which this PR imports as one aligned block, so the source's protection change retired both; 1b304c8 maps them with an unmapped block between them, so the check still covers two separate imports. The merge ofmain66e71b6 (0b84427) takes #5390'sHostImportOffsetwording inHostImportFor's declaration with this PR'srequesterparameter; this PR's own code computes no buffer offset asaddress - base, so #5390'sleadreaches it throughHostImportOffset.Related: #4260 (its first cause: a buffer crossing adjacent ranges of a window that is not full is now imported in place when one filled aligned block holds it; a buffer crossing a block or window edge is still copied. #4367 fixed its second cause, the Windows alias).
Tested
Resident Evil Village (PPSA01556), Windows 11, RTX 3070 Laptop GPU (driver 617.42), on main 6ea62f9 with #6366, #5703 and #5815 (open) and local commits for its sample-count stops, at main's default budget:
[gpu] host import budget: 5840 MiB live of 5840 MiB (APS5_HOST_IMPORT_MIB)).[gpu] host import budget: 5375 MiB live of 5840 MiB (APS5_HOST_IMPORT_MIB); 21138 imports (71872 MiB) refused so far, last 0x200200000+0x30000000. The last is the 768 MiB heap, retried again and again.Linux, lavapipe (CI's job mirror):
^agc598/598, and the new tests 10 of 10 at each commit.AcquireRegisteredper rebuild: 57.8 → 43.2 ms with the budget at 60 % of the registry, and 50.0 → 19.7 ms at 100 %;importRun: +111 ns per call.Windows (GitHub's runner, MinGW, no Vulkan device), on this branch before its rebase onto
main(0f85236, the same code; run): it builds, and 591 of 592^agctests pass (460 skip without Vulkan). The one failure,agc_driver_pm4_replay_tool(0xc0000139), and libkernel'sguest_kqueuealso fail onmainon that runner.Windows 11, RTX 3070 Laptop GPU (driver 617.42), at this head: the new and changed tests (
agc_storage_writeback_blocks,agc_driver_graphics,agc_driver_storage_write_back_overhang) pass 5 of 5. With the import code before the aligned blocks,agc_storage_writeback_blocksfails with "a 4 MiB image over 64 adjacent 64 KiB pieces: the write-back made 64 imports, expected 1", andagc_driver_graphicswith the host import report line (no remembered count).Merged with
main66e71b6 (0b84427, after #5390), on GitHub's runners (run): the host import, settle, lifetime and sparse tests pass 3 runs in a row on Linux, as do the^agctests (709, 0 failed), and the libc and libkernel suites pass; on Windows only the two replay tools named below fail. The earlier merge withmain3de84fa (1b304c8) ran the same way (run): the host import, settle and lifetime tests pass 3 runs in a row on Linux, as do the^agctests (709, 0 failed), and the libc and libkernel suites pass. On Windows the only failures areagc_driver_pm4_replay_toolandagc_shader_replay_tool, which fail the same way on that runner for an unrelated branch (run). Before the test commit, the merge withmain3b887b5 failedagc_driver_guest_memory_lifetime_copyon Linux with "source retirement unnecessarily retired the copy destination" (run); with it, that test and the host import tests passed 10 runs in a row (run).Checklist
main0029e62, merged withmain66e71b6. The tests above ran on 598dd15; the rebase changed one line of the debt file, next to main's new entry. No other open PR implements the same functions. Overlaps (git merge-treewith each):refreshImports' pruning ofstate.failed: merged, fix(agc): retain failed import windows across unrelated mappings #6442'ssameRangefor failures passes the budget toimportRun, as this PR's loop does.agc_storage_write_back, this PR'sagc_storage_writeback_blocks); if both land, both tests run on the merge.importRunfor every requester: one adjacent declaration.importRun's walk to the ends of the run per lease snapshot (walkRunBounds,importRunBounds), and hassameRangeandimportAllocation's whole-run lookup compare an import with the run's whole span. This PR imports an aligned block inside that run, or the containing range alone, and skips remembered refusals there, so an import can cover only part of its run. They conflict inimportRunandsameRangeinGuestBufferMemory.cpp. Whichever merges second should have this PR'simportRuntakefirst,lastandindexfromimportRunBounds, and keepsameRangeand the whole-run lookup comparing with the blockimportRunreturns for the current budget, since an aligned block import doesn't span the whole run. That merge hasn't been built yet.HostImportsfields and map time's call inGuestBufferMemory; keep both sides.NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file (none added).mdfiles or images added to the repository (attach them to this PR)