Skip to content

perf(libkernel): find direct mappings that share physical memory through an index - #4321

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
oneandonlydean:up4/fix-3308-watch-mapping
Oct 12, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
oneandonlydean:up4/fix-3308-watch-mapping

Conversation

@oneandonlydean

@oneandonlydean oneandonlydean commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

What

Fixes #3308 (its stop 3, the first stop on main since #3428, #3465, #3455 and #3476).

WatchMapping (DirectMemory.cpp, Linux) registers a new direct mapping with the write watch and then unregisters every range whose physical memory is also mapped somewhere else. To find those it walked all of g_directMappings for each mapping added, so N mappings cost N(N+1)/2 visits. The report measured 23.2 s of 24.3 s there over 42,412 sceKernelBatchMap2 entries of 64 KiB, after which the title's watchdog stops it.

Direct mappings are now also indexed by physical range: g_physicalSpans cuts the physical offsets into disjoint spans, and each span lists the mappings that cover it. InsertMapping and RemoveMapping replace the eleven places that added to or erased from g_directMappings, so the index always matches it; adjacent spans with the same mappings are merged again on removal. WatchMapping reads the spans under the new mapping's physical range and gets exactly the mappings that overlap it, in address order as before, then runs the same loop body on them. The same ranges are registered and unregistered, in the same order; only the search changed. The cost of one map is now the number of mappings that really share its physical memory, whatever their sizes.

The index is Linux-only, like WatchMapping. ErasePhysMappings and DirectMemoryViews still walk every mapping (once per release, not per map) and are not changed here.

#4216 changes the same loop with an index by physical start that scans back by the longest mapping ever recorded. In the benchmark below it matches this change while every mapping is small, and with one live 4 GiB mapping it spends 38.8 s in WatchMapping (4.4 s on main, 0.04 s here).

@snoozedDev could you confirm on PPSA04029? I don't have the title.

Tested

Linux 7.2.3, GCC 16.2.1, Release, -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON (the Linux job's flags without the ccache launcher), RTX 4090, NVIDIA 610.57.04.

Benchmark without a device: one allocation of 2.6 GiB, then 42,412 sceKernelBatchMap2 calls of one OpMapDirect entry of 64 KiB each, no address given. A temporary counter and timer in WatchMapping (not in this pull request), base main 34a16b7:

visits in the search time in WatchMapping index upkeep whole run
main 899,410,078 4.41 s - 8.7 s (11.5 s and 25.5 s in two other runs on a busier machine)
this change 42,412 0.041 s 0.040 s 3.3 s
this change, one 4 GiB mapping live 42,412 0.043 s 0.045 s 3.5 s
  • Then 5,000 single aliases of those mappings: 224,562,500 visits and 6.4 s on main, 10,000 visits and 1.1 s here; on both, the same 37,412 of the first mappings stay watched and no alias is watched.
  • What is left of the whole run (about 3.2 s) is GuestArenaAllocate choosing each address (about 90 % of the samples in perf); it is outside WatchMapping and not changed here. With fixed addresses the walks are in GuestAllocationsCovers, GuestAllocationsOverlaps and GuestAllocationsRequireAvailable instead. The report's timers put everything outside WatchMapping at about 1.1 s for the title.
  • guest_memory gains two cases. One maps 16,384 separately allocated pages with one call (one direct mapping per page) and requires that it costs at most 48 times the same call over 1,024 pages, in thread CPU time, best of three. This is a relative timing bound, not a count: the tree has no operation counters for tests. On main the ratio is 150 to 240 and the case fails (5 of 5 runs); with this change it is 17 to 19 (24 runs, 12 of them with all 32 cores busy). The other checks what the write watch covers after aliases are partly unmapped, replaced by a fixed mapping and mapped again; it passes on main too.
  • The index code copied into a stand-alone program and compared with the full walk after every insert, with its invariants checked, under ASan and UBSan: 9 runs of 200,000 random inserts, removals and splits, no difference.
  • ctest -R "guest_|kernel|memory|write|direct": 325 of 325 pass on main 34a16b7; with this change 325 of 325 there and 326 of 326 after rebasing onto b42ef24. Full ctest with this change on 34a16b7: 925 of 925.
  • A 2D title we test with, 220 s through the menu into gameplay on base e013f8c: passes all picture checks, 32.7 fps in the menu and 38.1 in gameplay (dumping every frame), 0 skipped draws, 0 Xid.
  • tools/check_conventions.py --base origin/main and git diff --check pass.

Not tested: Cyberpunk 2077 (PPSA04029) itself; Windows and macOS (the index and WatchMapping are compiled on Linux only; the two helpers reduce to the previous emplace and erase elsewhere, not built here).

Checklist

  • Based on current main; no other open PR implements the same functions (perf(libkernel): find guest allocations and direct mappings by address #4216 changes the same loop with a different index, see above; this PR does not build on it, and whichever lands second rebases)
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes (Claude Code)

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#4216 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp L349-350, L357-359, L368-372, L375, L379, L402, L484, L948-949, L952, L954

@MarcZX10

Copy link
Copy Markdown
Contributor

The current conventions check has one actionable failure: “Based on current main; no other open PR implements the same functions” is unchecked. The description notes that #4216 changes the same loop, so please state the dependency/stacking plan and update that checklist response. The build and other checks reported here are passing.

@snoozedDev

Copy link
Copy Markdown
Contributor

Ran Cyberpunk 2077 (PPSA04029) on bare main d592d54, which contains 5b7cb8d: WatchMapping is gone from every stack sample, but the watchdog stop of #3308 still reproduces, three of three runs (cold, cold, warm), !!! EngineWatchdog timeout! 139 s after launch each time, the same as a bare-main run on d146ec1 just before this change. The sceKernelBatchMap2 threads are now in GuestAllocationsProtect_nid_postfix (the whole-map replaceRange copy), GuestAllocationsRequireAvailable_nid_postfix (walk from begin()) and the GuestAllocationsBegin_nid_postfix lock. With #4216 (327e3b8) cherry-picked on top, four of four cold runs still trip it, later (212 to 253 s), and the warm run passes it to the PA_CL_VS_OUT_CNTL refusal of #423. Runs, samples and setup in #4634.

AI-assisted: yes (Claude Code, Claude Fable 5.1), operating under my supervision; I am accountable for this comment and will respond to review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cyberpunk 2077 (PPSA04029) on Linux: sceKernelOpen("") opens a directory and the read throws EISDIR on main 961b25fe

3 participants