[fisherjoey]

What

ImportWatched keeps a copy of every non-zero page of the range it imports, so that afterwards it can tell a page the write watch reported from a page that still holds its old bytes (#3874, #5698). The copy is as big as the range. A first import of a 0.5 to 3 GiB guest heap holds that much host memory while the import runs (#7980).

With this change the window keeps a 64-bit digest of each 4 KiB page (HashPage, four xxHash64 lanes merged the way xxHash64 merges them) and WindowCopy::Holds compares digests rather than bytes. That is 8 bytes per page, 1/512 of the range: a 3 GiB import holds 6 MiB rather than 3 GiB.

Fixes #7980

• Each page gets the same decision as before. The one exception is a 64-bit collision, where a changed page would be taken as unchanged and stay unreported. The chance of that is about 2^-64 per reported page.
• Guest memory is read the same number of times as before: once before the import, and once after it for each page the watch reported.
• Zero pages no longer need their own path, so ZeroPage and WindowCopy::Zero are removed.

Tested

• New test CheckImportWindowFootprint in agc_write_tracking (glibc only, since it reads mallinfo2). It imports a 64 MiB watched range, all non-zero, and requires the heap growth seen inside the import to stay at or below 1 MiB. On main it fails with an import window over 64 MiB held 64 MiB of host memory during the import. With this change it passes. It then re-imports the range, writes the same byte to one page and a new byte to another, and checks that the first page reads as unchanged and the second as written.
• The existing import window tests (changed, unchanged, zero pages, other thread, store across imports) pass.
• agc_write_tracking, agc_import_watch and the other 50 tests run by agc_write_tracking_tests and agc_driver_recorder_tests (52 in all) on this branch, based on main dbac9fa, Linux, Release, GCC 16. On lavapipe (Mesa llvmpipe): 47 pass and 5 are skipped for missing capabilities. On an NVIDIA GeForce RTX 5070 Ti (driver 595.91.07): all 52 pass.
• Demon's Souls PPSA01341 v1.07, Linux, RTX 5070 Ti, glibc. These runs were not on current main. The base was main 93e54ce plus fix(agc): keep texture images inside their extent's mip chain #6280, a cherry-pick of fix(agc): reuse a recycled descriptor set only for a table of its own size #6567, a local texture-snapshot reuse patch, a local stencil-export workaround, and a local job-worker idle patch run with APS5_JOB_IDLE_SLEEP=1, which loads faster and makes the load burst sharper. I compared the same set with and without this commit and sampled host memory every 5 to 10 s:
  • Without it: the level load showed host jumps of +3 to +19 GB within 5 to 10 s across the earlier sets. One run printed a 493 MiB ImportWatched allocation. Another had a 2.9 GiB anonymous block appear in the 10 s before the host OOM killer stopped it.
  • With it: the largest 5 s jump during the level load was +1.09 GB. The level loaded, and the run used 11.3 GB once the load finished. A second, measuring run also loaded the level and finished the 900 s cleanly, peaking at 14.5 GB (anon plus swap). No frame rate claim.
  • On a current-main base (main 6762694 plus fix(agc): keep texture images inside their extent's mip chain #6280, fix(agc): land GPU writes over guest memory before a host write into it #6926, fix(agc): keep at most 4 GiB of the host heap from the host import budget #7844 and the stencil workaround, without the job-idle patch), with only this driver swapped in: the load started, no crash, full 900 s. The largest 10 s host jump during the load was +1.86 GB, against +1.69 and +2.08 GB without the change. At main's normal load speed Demon's Souls does not show the burst, so this shows the change is safe there but not a measurable benefit.

Related

• perf(agc): compare an import window's pages when the window closes #3874 added the window comparison and perf(agc): copy only nonzero pages before a watched import #5698 limited the copy to non-zero pages. This keeps both behaviours and drops the copy.
• Open fix(agc): run the unwatch hook before an import whose range leaves the write watch whatever the import does #7505, fix(agc): save pending storage images before a partly watched import unwatches them #7687 and fix(agc): run the unwatch hook when a watched import leaves the write watch #6308 change ImportWatched's unwatch and save paths near these lines (fix(agc): run the unwatch hook when a watched import leaves the write watch #6308's unwatch hook edits the same fallback block). None of them replaces the copy, so they don't overlap with this, but whichever lands second may have a textual conflict. I'll rebase if needed.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); this change adds no stubs
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source; no third-party code added
• Generic behaviour, not specific to one title
• Depends on: none
• GPUs tested: lavapipe (Mesa llvmpipe), Linux, and NVIDIA GeForce RTX 5070 Ti, driver 595.91.07, Linux (Nobara, kernel 7.2), for the tests; RTX 5070 Ti for the game runs
• AI-assisted: yes

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 4448	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 5021	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6186	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6187	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6188	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6192	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6209	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6868	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp core/libs/prx/libSceAgcDriver/tests/WriteTracking.cpp
PR 6946	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6947	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 6960	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp
PR 7201	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp core/libs/prx/libSceAgcDriver/tests/WriteTracking.cpp
PR 7505	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp core/libs/prx/libSceAgcDriver/tests/WriteTracking.cpp
PR 7604	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp core/libs/prx/libSceAgcDriver/tests/WriteTracking.cpp
PR 7642	core/libs/prx/libSceAgcDriver/Execution/src/GuestMemory.cpp

[MarcZX10]

One overlap detail for the integration note: the overlap check also flags open #6308 in GuestMemory.cpp. Its unwatch-hook change is separate from the digest change, so I do not see duplicate behavior in the diff, but both edit the ImportWatched fallback and may need a textual rebase or careful combination. Could you include #6308 in the related/rebase notes?

[fisherjoey]

#6308 was already in the related notes, but I've made it clearer that its unwatch hook edits the same ImportWatched fallback block, so a rebase there needs both changes combined by hand.

AI-assisted: yes

[fisherjoey]

A run on a current-main base, for honesty about where this matters.

Base: main 6762694 plus #6280, #6926, #7844 and a local stencil-export workaround (not in any PR), with only this PR's driver swapped in. Demon's Souls, RTX 5070 Ti, 900 s, without our local job-idle patch. The level load started, no crash, and the run went the full 900 s.

The largest 10 s host memory jump during the level load was +1.86 GB with this PR, against +1.69 GB and +2.08 GB in two runs of the same base without it. So at main's normal load speed, Demon's Souls does not show the multi-GB burst, and this run shows the change is safe there but not a measurable benefit. The +3 to +19 GB jumps in the description come from the faster loads with the job-idle patch. My guess is that several of the 0.5 to 3 GiB imports overlap there, but I have not measured that. The copy itself is still on main (the footprint test fails there with "held 64 MiB"), so a title or machine that loads faster can still hit it.

AI-assisted: yes

[MarcZX10]

Update: I discarded the first tracker-level approach because it conflicted with #6308 and #7505. The current local two-commit candidate instead filters coalesced runs in importRun when a run crosses existing write-watch gaps, leaving ImportWatched semantics unchanged; it has a Windows fixture for the reported layout. That now overlaps the importRun refactor in #6951, so I asked there about coordination. Neither version has been pushed or opened, and this alternative still needs a write-watch-capable Windows run.

[oneandonlydean]

Thanks for your patience on this. #8172 (the fix for the #8103 regression) has merged, and it also rewrites ImportWatched's window copy in GuestMemory.cpp, so this branch now conflicts there.

The two changes fit together: #8172 splits a partly watched range into runs of covered 64 KiB blocks that stay watched, and your per-page hashing can apply within those spans. Could you rebase onto current main and redo your change on top of #8172's version of that code? We'll review the rebase and run CI on it as soon as it's up.

[blawesom]

Rebase handoff — rebased onto current main (3fd1cc393, authorship unchanged). This PR and main evolved the same import-window path in different directions (main's 71fd1c0e5 copies only nonzero pages with 4-byte slots; you replace the copies with 8-byte xxHash64 page digests), so the resolution unites them:

• main's span-based flow is kept — partial coverage (an import over an unwatched block) still builds covered spans, drops unchanged ones via erase_if, and unwatches per span — but each page now records your HashPage digest instead of a slot plus a copied page. Your hash-based WindowCopy and HashPage (which merged cleanly) drive the re-walk.
• main's CheckImportWindowOverUnwatchedBlock test (the spans path) and your CheckImportWindowFootprint test are both kept and both pass.

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV): builds; agc_write_tracking passes; 1673/1676 — green apart from the three known main failures (#7584 BDA, #7784 guest-memory, tessellation), none from this PR; conventions pass.

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/7981
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is 0460baa6f.

[fisherjoey]

Rebased onto current main (75787d2), on top of #8172. New head is 4d893a9.

I used @blawesom's resolution as is. Thanks for doing the merge. It keeps #8172's span flow (covered 64 KiB runs, the erase_if, unwatching per span), and each page in a span now records its HashPage digest in place of a slot and a copied page. Main hasn't touched GuestMemory.cpp or WriteTracking.cpp since their base 3fd1cc3, so the commit moved over unchanged.

Built on Linux and check_conventions.py --base origin/main passes. agc_write_tracking (which runs CheckImportWindowFootprint and CheckImportWindowOverUnwatchedBlock) and agc_import_watch pass on lavapipe and on an RTX 5070 Ti.

AI-assisted: yes

[adriadam10]

Checked the diff against main (on top of #8172's span flow): HashPage digests replace the page copies, Holds compares digests, ZeroPage/WindowCopy::Zero have no remaining uses. CheckImportWindowFootprint pins a ≤ 1 MiB heap growth over a 64 MiB import (main copies the range, so it fails there) and the unchanged/written page decision; the full Build runs the test; merged.