Skip to content

fix(libkernel): fall back below a mapping hint with no room above it - #5218

Merged
adriadam10 merged 3 commits into
boykopovar:mainfrom
YossiYad:fix/guest-arena-mapping-hint
Oct 14, 2026
Merged

adriadam10 merged 3 commits into
boykopovar:mainfrom
YossiYad:fix/guest-arena-mapping-hint

Conversation

@YossiYad

Copy link
Copy Markdown
Contributor

What

A mapping without MAP_FIXED treated its address as the lowest place it could go, and a hint at or above the end of the guest arena threw on the host (mapping address hint is above the guest address space arena). FreeBSD 11.4's vm_mmap_object places such mappings with vm_map_find_min, which takes the address as a hint only: it tries at or above it, then again from its minimum address.

  • GuestArenaAllocateFromHint does the same in the arena: at or above the hint when the hint lies below the arena's end and a range fits there, otherwise from the arena's start. MapPlaced (Linux) and mmap_aligned (Windows) use it for hinted mappings. GuestArenaAllocateAtOrAbove keeps its meaning for the callers that need a range above an address (CheckArenaHonoursHintsWithoutGoingBelowThem still passes).
  • guest_memory reserves with a hint above the arena and with a hint on the arena's last page for two pages; both land inside the arena, the second below its hint.
  • The debt entry records that FreeBSD's minimum is the end of the largest data segment rather than the arena's start.

Title. PS5CEMU-HAR (PPSA99360) reserves its 3DS JIT range with sceKernelReserveVirtualRange and a hint near its own code. With the title's earlier stops applied (#4514, #4469, #4467), launching a 3DS homebrew aborted the process with the throw above. With this PR the reservation succeeds and the title reports its own Azahar did not start: can't alloc and returns to its library instead of terminating. The emulator still cannot start, because the relinked executable lies outside the guest arena; that is #5217 and not changed here.

Refs #5217.

Tested

Linux (Bazzite 44, GCC 16.2.1, Ninja, Release, -DBUILD_TESTING=ON), based on main 0f2683cb:

  • Full build and ctest: 1054 of 1059 pass. The five others fail identically on unchanged main here: agc_driver_tessellation (segfault), guest_kernel_stack_protect, guest_module_directories, and guest_ampr_cpp_error / video_out_mouse (not built: the first does not link with this machine's non-static toolchain).
  • guest_memory fails with the previous call in MapPlaced, and with the fallback to the arena's start removed.
  • Title run (distrobox, RX 7700 XT, the three PRs above applied): before, what(): mapping address hint is above the guest address space arena and SIGABRT; after, the title's own error message and a running launcher.
  • tools/check_conventions.py --base upstream/main, git diff --check and tools/pr_overlap.py (no overlap) pass. Windows was not built here.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw; no silent stubs added
  • No notes, investigation .md files or images added to the repository
  • No new third-party code
  • Generic behaviour, not specific to one title
  • Depends on: none
  • AI-assisted: yes

@simonbronk

Copy link
Copy Markdown
Contributor

Tested on Windows 11 Pro 26200, WinLibs GCC 15.2.0 r7, Release, full build, this PR (3fbf4fb82) merged onto main 0aa4cb1c2 (AI-assisted: Claude Code). This exercises the Windows call site (mmap_aligned in DirectMemory.cpp), which the Linux run does not reach.

  • With main's DirectMemory.cpp and the rest of the PR kept: guest_memory aborts with what(): mapping address hint is above the guest address space arena (exit 0xc0000409).
  • With this PR: guest_memory passes 10/10 (--repeat until-fail:10), and the full ctest passes 1098/1098.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
A mapping without MAP_FIXED treated its address as the lowest place it
could go, and a hint at or above the end of the guest arena threw on the
host. FreeBSD 11.4's vm_mmap_object places such mappings with
vm_map_find_min, which takes the address as a hint only: it tries at or
above it, then again from its minimum address.

GuestArenaAllocateFromHint does the same in the arena: at or above the
hint when it lies below the arena's end and a range fits there, otherwise
from the arena's start. MapPlaced and the Windows mmap_aligned use it for
hinted mappings; GuestArenaAllocateAtOrAbove keeps its meaning for the
callers that need a range above an address.

PS5CEMU-HAR (PPSA99360) reserves its JIT range with
sceKernelReserveVirtualRange and a hint near its own code, a host address
above the arena here, and aborted with "mapping address hint is above the
guest address space arena" when launching a 3DS title.
guest_memory reserves with a hint above the guest arena and with a hint
on the arena's last page for two pages, and checks that both land inside
the arena, the second below its hint.
@YossiYad
YossiYad force-pushed the fix/guest-arena-mapping-hint branch from 3fbf4fb to aca7a3a Compare October 14, 2026 14:45
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@adriadam10 adriadam10 self-assigned this Oct 14, 2026
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the arena fallback against FreeBSD's vm_map_find_min retry from the minimum address, built and ran the libc/libkernel ctest scope (240 tests, all pass), and confirmed guest_memory fails with the code change reverted. Windows call site covered by @simonbronk's run above; merging.

@YossiYad

Copy link
Copy Markdown
Contributor Author

@adriadam10 thanks for checking it against vm_map_find_min, and @simonbronk for the Windows run. The branch still merges into current main (2b7e3ce69) without conflicts, so it can go in as it is.

@adriadam10
adriadam10 merged commit 9811e9d into boykopovar:main Oct 14, 2026
14 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Partly fixed by #5218: mapping hints with no room above them fall back to the arena start (covered by guest_memory); the relinked executable still lies outside the arena.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants