Skip to content

perf(libkernel): find guest allocations and direct mappings by address - #4216

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
Mouad-Scriptz-0:perf/guest-memory-registry-lookups
Oct 13, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
Mouad-Scriptz-0:perf/guest-memory-registry-lookups

Conversation

@Mouad-Scriptz-0

@Mouad-Scriptz-0 Mouad-Scriptz-0 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

What

sceKernelVirtualQuery, sceKernelMunmap, sceKernelReleaseDirectMemory and the reservation paths copied the whole allocation registry (or walked every direct mapping) under one lock on each call. Minecraft (PPSA17221) holds about 7,000 direct mappings and spent its world generation in these walks: free() → sceKernelVirtualQuery/sceKernelReleaseDirectMemory dominated every thread's stacks (eu-stack), and the render thread blocked on the registry lock behind them.

  • GuestAllocationsLookup_nid_postfix(address, next) answers "which range holds this address" (and the find-next flag of sceKernelVirtualQuery) from the registry's ordered map; sceKernelVirtualQuery uses it instead of acquiring a copy of all ranges.
  • Range replacement computes a RangePatch (ranges to erase, ranges to insert) and applies it to the live map instead of rebuilding the map; allocationRemains and Covers start from the containing range.
  • main's physical-span index (5b7cb8d, consulted by WatchMapping for shared backings) now also answers ErasePhysMappings (sceKernelReleaseDirectMemory) and DirectMemoryViews through one MappingsSharingPhys helper, so a release touches only the mappings over the released range; the index is kept on every platform for that instead of Linux only.
  • guest_memory gains two cases: sceKernelVirtualQuery finds the containing range, the split after sceKernelMprotect, and the next range after a hole (flags 1); releasing a physical range in the middle of 48 separately mapped pages plus an alias unmaps exactly those views.

Behaviour is unchanged; only the lookups are. With #4118 and the flip event PR, Minecraft goes from a crawl in world generation to playable.

Tested

Fedora 44 (kernel 7.2.8, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, Intel Arc Meteor Lake, Mesa 26.2.3), rebased onto main c36fd0a: guest_memory, guest_memory_pool_stats, guest_reallocalign, agc_write_tracking, agc_write_watch_coverage pass; tools/check_conventions.py --base origin/main and git diff --check pass. Minecraft (PPSA17221) and Zombie Cure Lab (PPSA22623) run with it (Linux). Not tested on Windows.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 12, 2026
@Mouad-Scriptz-0
Mouad-Scriptz-0 force-pushed the perf/guest-memory-registry-lookups branch from f28e12a to 327e3b8 Compare October 12, 2026 09:05
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 12, 2026
@Pacuka

Pacuka commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Data point from Minecraft (PPSA17221) on Linux / AMD: CachyOS kernel 7.2.2, RX 6650 XT (RADV, Mesa 26.2.4), GCC 16.2.1, Release. Base: main d592d54e1 + #4215 #4218 #454 #3633 #3604. Host import on.

Without this PR the render thread spent most of its time in HostImportFor. About 3% of the lookups found the registry generation changed, and then GuestAllocationsAcquire_nid_postfix copied all ~7,000 ranges. That took ~3.6 ms per lease and 111 s in total over a ~3 min run, about 125 us on average per in-place draw input. With this PR the same counter is 6 s over a comparable run. The in-game FPS counter went from ~2 to ~6 in a world.

@xXPBSXx

xXPBSXx commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Windows test of head 327e3b810 merged onto current main (2a5eca11c, clean merge; built together with two other open PRs, so the run is for the combination): Windows 10 (19045), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release, full build 0 errors, -j2.

guest_memory (the test this PR changes) passes. The 25 tests matching ^(guest_memory|guest_np_auth|windows_exceptions|guest_exception|guest_kernel) all pass.

Not checked: that the test fails without the change, Linux, a title run.

AI-assisted (Claude Code).

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Needs a rebase: docs/dev/TechnicalDebt.md was split into docs/dev/debt/, and that is the only conflict. Please drop the entry instead of moving it; it describes the index, not a gap.

sceKernelVirtualQuery, sceKernelMunmap, sceKernelReleaseDirectMemory and the
reservation paths walked a copy of the whole allocation registry or of every
direct mapping under one lock on each call. The registry now answers lookups
from its ordered map, applies range replacements as patches instead of
rebuilding the map, and direct mappings carry an index by physical offset so a
release finds only the mappings it covers. Minecraft (PPSA17221) holds about
7,000 direct mappings and spent its world generation in these walks.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@oneandonlydean
oneandonlydean force-pushed the perf/guest-memory-registry-lookups branch from 327e3b8 to df3d027 Compare October 13, 2026 10:28
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#1835 core/libs/prx/libc/src/GuestAllocations.cpp L427-428
#5053 core/libs/tests/GuestMemory.cpp L2151
#5073 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp L349-351

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Rebased onto main 77b9f72 and force-pushed. The only conflict was docs/dev/TechnicalDebt.md; the physical index entry is dropped rather than moved to docs/dev/debt/, since it describes the index, not a gap. No code changes; the commit keeps your authorship. guest_memory, guest_memory_pool_stats, guest_reallocalign, agc_write_tracking, agc_write_watch_coverage and the agc_/guest_ ctest set pass on Linux. Merging.

@oneandonlydean
oneandonlydean merged commit 50578eb into boykopovar:main Oct 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants