[Ragbie]

What

On Linux, mprotect and sceKernelMprotect over the host-only pages around a guest thread's stack now return 0 and leave those pages unchanged. Before, they returned -1 with EACCES. The sceKernelMprotect debt entry recorded the gap ("On Linux a range over the host-only pages around a guest thread's stack still fails with EACCES, where Windows returns 0"), and the #9882 discussion left the Linux return of the straddling range as a follow-up.

The same debt entry records the PS5 (firmware 13.60) measurement: mprotect and sceKernelMprotect change the mapped parts of a range, skip its gaps and return 0. On FreeBSD 11.4, libthr places the stack guard in the stack's own mapping (thr_stack.c), so on the console the pages around a thread stack are guest memory.

On Linux a guest thread's stack is a guest allocation (AllocateGuestStack) that the host thread runs on. The guard, the spare host stack below the guest stack and the host's frames above it are host-only. 9b6e7f6 keeps the guest from changing them by refusing any range that touches them. This PR treats them as gaps instead:

• GuestThreadStackHostRanges returns the host-only ranges a range meets. GuestThreadStackHostRangeOverlaps is built on it.
• DoMprotect cuts the request around those ranges and protects the parts beside them.
• A part inside a guest stack goes through the existing stack path, which works on host pages, so its precision is unchanged.
• Any other part (a GPU protection bit, or a thread that is starting or exiting) goes through the general path, which rounds to 16 KiB. Such a part is first shrunk away from any 16 KiB page that holds host bytes, so the rounding cannot reach them.
• The batch map and memory pool protect operations, which require a mapped start, return EINVAL when the range starts in a skipped page, as for an unmapped start.
• The general-path parts are handled in one registry mutation: every span of every part is checked for a pin before the first is protected, as the single-range path did.

The debt entry now states what is left: those pages are skipped where the console's guard takes the protection.

Tested

guest_kernel_stack_protect:

• Linux: a protect of the host guard page below a guest thread stack returns 0 and leaves the page unchanged (was -1).
• All hosts: the range that straddles the bottom of the stack returns 0 (Windows already checked this; the Linux return was unchecked) and leaves the page below the stack unchanged.
• Linux: a read-only protect of a three-page range starting below the stack makes the stack page inside it read-only.
• All hosts: the straddling range with a GPU read bit (0x11) leaves the page below the stack unchanged.
• Linux: a pinned later part. A guest page is mapped just below the stack's mapping and the stack's registry range is held, so it counts as pinned. A 0x11 protect from that page into the stack splits into two general-path parts around the host guard. The pin waiter hook records whether the earlier page already changed when the pin is found, then releases the pin. The test requires the earlier page unchanged at that point, and both parts protected after the call returns 0.

Linux (Ubuntu 24.04 container, GCC 13.3.0):

• Without the fix: GuestKernelStackProtect failed at line 235 (the host guard page returns -1).
• First version of the fix, which let the general path round over host pages: the GPU-bit case failed at line 268. Kept as the reason for the shrink.
• Per-part pin checks (4c52e758e0) with the new test: GuestKernelStackProtect failed at line 254. The waiter ran (line 253 passed), but the earlier part had already been protected.
• With the fix, rebased on b11c7bf84f: guest_kernel_stack_protect and guest_pthread_stack pass with --repeat until-fail:3. Of the 240 tests matching mprotect|memory|stack|pthread|mmap|thread|mtype|apr|ampr|flexible|pool|mlock|protection, 239 pass. guest_mmap_executable fails in this container on main too.

Windows 11 Pro, Intel Core i5-14600KF, NVIDIA GPU, WinLibs GCC 15.2.0, full Release build with libs:

• guest_kernel_stack_protect and guest_pthread_stack pass.
• Full ctest: 99% tests passed, 6 tests failed out of 4005. None of them is caused by this branch:
  • test_registration and select_tests_MissingCommandTests fail with WinError 1314, because this user cannot create symbolic links.
  • agc_driver_image_plane_address_on_line, agc_driver_image_sample_lod_clamp, agc_driver_image_sample_lod_clamp_shader_lane and agc_driver_direct_write_completion also fail with DirectMemory.cpp and Pthread/ restored to main.
• check_skips.py reports one skip, guest_fopen_errors, which skips itself here because this user cannot create symbolic links.

macOS: not tested. GuestThreadStackHostRanges returns no ranges there, so DoMprotect keeps its path; the GPU-bit case runs there as on Windows.

The three-page read-only case is Linux-only because Windows skips the whole straddling range: the host thread stack is not in the guest allocation registry, so the stack part is not protected either. That is a separate Windows gap and goes in its own PR.

Overlaps: #7149 and #7631 also change DoMprotect; both conflict with main already, and a trial merge with this branch conflicts in the same files only. #14466, #9684 and #7532 merge with this branch without conflicts (git merge-tree).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[mashingaan]

I found a pin-safety regression in the split protection path. The existing single-range path gathers every registered span and calls RequireUnpinned on all of them before the first protection change. Here each part is sent through ProtectGuestRange separately, and each recursive call checks and then protects its own spans. If a later part beyond a host-stack gap is pinned, the call can fail after an earlier part has already changed protection. Please preflight all surviving spans across every part before applying any protection, and add a focused regression for a pinned later span.

The affected call is core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp:1463 at head 4c52e758e0b9f2f6c39833871cc1bb6d4bdab525.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 9947	docs/dev/debt/libkernel.md L165-166

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 3986	core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4563	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 4630	docs/dev/debt/libkernel.md
PR 5021	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 6740	core/libs/prx/libkernel/Pthread/include/Pthread.hpp docs/dev/debt/libkernel.md
PR 6936	docs/dev/debt/libkernel.md
PR 7088	docs/dev/debt/libkernel.md
PR 7105	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp docs/dev/debt/libkernel.md
PR 7121	docs/dev/debt/libkernel.md
PR 7135	docs/dev/debt/libkernel.md
PR 7149	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 7192	docs/dev/debt/libkernel.md
PR 7447	docs/dev/debt/libkernel.md
PR 7464	docs/dev/debt/libkernel.md
PR 7631	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
PR 7804	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 8592	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 9156	docs/dev/debt/libkernel.md
PR 9199	docs/dev/debt/libkernel.md
PR 9293	docs/dev/debt/libkernel.md
PR 9297	docs/dev/debt/libkernel.md
PR 9302	docs/dev/debt/libkernel.md
PR 9389	docs/dev/debt/libkernel.md
PR 9557	docs/dev/debt/libkernel.md
PR 9566	docs/dev/debt/libkernel.md
PR 9843	docs/dev/debt/libkernel.md
PR 9938	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 10025	core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp docs/dev/debt/libkernel.md
PR 10319	docs/dev/debt/libkernel.md
PR 10442	docs/dev/debt/libkernel.md
PR 11896	docs/dev/debt/libkernel.md
PR 11939	docs/dev/debt/libkernel.md
PR 12104	docs/dev/debt/libkernel.md
PR 12130	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12544	core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12560	docs/dev/debt/libkernel.md
PR 12642	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12682	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 12688	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 12700	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 12965	core/libs/prx/libkernel/Pthread/include/Pthread.hpp core/libs/prx/libkernel/Pthread/src/Thread.cpp docs/dev/debt/libkernel.md
PR 12970	docs/dev/debt/libkernel.md
PR 13560	docs/dev/debt/libkernel.md
PR 14103	docs/dev/debt/libkernel.md
PR 14128	docs/dev/debt/libkernel.md
PR 14212	docs/dev/debt/libkernel.md
PR 14264	docs/dev/debt/libkernel.md
PR 14310	docs/dev/debt/libkernel.md
PR 14832	docs/dev/debt/libkernel.md
PR 14854	docs/dev/debt/libkernel.md
PR 14886	docs/dev/debt/libkernel.md
PR 14894	docs/dev/debt/libkernel.md

[Ragbie]

Thanks, you were right. Fixed in 0a8f377: the general-path parts now share one registry mutation in ProtectRegisteredRanges. Their spans are collected first, then all are checked with RequireUnpinned, and only then is anything protected. The recursion through DoMprotect is gone. 0325fd4 removes the per-part pin note from the debt entry.

The regression test is e8bd827 (guest_kernel_stack_protect, Linux). It maps a guest page just below a thread stack's mapping and holds the stack's registry range as a pin. A 0x11 protect then runs from that page into the stack, and the GPU bit sends both parts down the general path. The pin waiter hook records whether the earlier page had already changed when the pin was found, then releases the pin, so no timing is involved. With the per-part checks (4c52e758e0) it fails at line 254: the earlier part was already protected. With the fix it passes with --repeat until-fail:3. I also rebased onto current main. The PR description has the updated Linux and Windows runs.

[nikhilpanchal1]

@Ragbie I have a separate flexible-memory mtypeprotect safety fix ready that conflicts with your change in docs/dev/debt/libkernel.md. Would you prefer #15040 to merge first so I can rebase mine afterward? My branch is https://github.com/nikhilpanchal1/AnyPS5/tree/fix/kernel-mtypeprotect-flexible

[TechLevelZero]

✅ AMD GPU test: passed

PR head e8bd827 merged onto main b611e63 · tested 2026-11-05 02:46 UTC · 21 min

GPU	AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host	Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs	build and test logs for this run
Build	✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests	✅ 3,029 passed · 9 failed (0 new, 7 also fail on main, 2 unstable on main) · 59 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup	CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection	3,097 of 3,946 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation	✅ no new validation IDs vs main (1074 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers)	⚠️ agc_driver_pipeline_journal 120.2 s, ⚠️ agc_driver_image_sample_unnormalized 80.7 s, ⚠️ agc_driver_graphics_maps_fallback 71.8 s, ⚠️ agc_driver_lazy_resolved_abi 69.2 s, ⚠️ public_headers 61.2 s

Unstable on main (2, these also failed in recent main runs on this machine; not counted against this PR)

agc_driver_depth_fast_clear_submit: failed in 2 of the last 10 main runs
agc_driver_pipeline_journal: failed in 9 of the last 10 main runs

Failing on main too (7, not counted against this PR)

agc_driver_image_gather_volume
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_driver_submit_progress
agc_submit_progress
agc_submit_progress_no_timeline

Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)

agc_driver_emulated_color_compare: Timeout in the parallel run; also passed only on retry in 44 other runs in the last 7 days

Validation IDs already present on main (4 IDs in 624 tests)

VUID-VkDeviceCreateInfo-pNext-pNext (624 tests): agc_driver, agc_driver_address_draw_space_reuse, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, and 616 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (2 tests): agc_driver_prepared_exports, agc_driver_srgb8_color_target

Skipped (59)

agc_driver_bda_reads_shadowed_results: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_bda_reads_shadowed_results_per_build: skipped, the device reports watched host memory written when it imports or reads it
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_image_table_vouched_lookups: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a watched imported table cannot be vouched for
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_scattered_runs: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_sparse_dma_buf_import: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_driver_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_driver_unit_shadow_budget: skipped, the device does not import watched host memory for unit shadows
agc_driver_vertex_elected_atomic_max: skipped, the device runs vertex programs in subgroups, so they keep their cross-lane reductions
agc_host_import_sparse: skipped, host imports are unavailable: /dev/udmabuf is not accessible
agc_idle_publish_wait: skipped, the device reports 4 of 4 host-imported pages written after a GPU read, so a forced watch of the import reads its submissions as CPU stores
agc_import_settle_initial_dcc: skipped, the device has no deferred settle of host imports: imports under 1 MiB wait for one only on Windows
agc_shadowed_key_stores: skipped, the device imports no watched host memory under a unit shadow
agc_small_tiled_shadow_label: skipped, host imports are unavailable: the test block is not imported and watched
agc_storage_clear_hand_over: skipped, the device imports no watched host memory for the clear-key hand-over
agc_unit_shadow: skipped, the device does not import watched host memory for unit shadows
agc_unit_shadow_completion_store: skipped, the device reports watched host memory written when it imports or reads it
guest_http_uri_locale: 
guest_video_out_color_space: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_hdr: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_high_frame_rate: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_output_mode: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_resolution_status: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
guest_video_out_wqhd_detection: skipped, no display or Vulkan device: SDL_InitSubSystem(VIDEO | GAMECONTROLLER) failed: wayland,x11 not available
video_out_flip_aftergpu: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip_backtoback: skipped, no display or Vulkan device: DisplayWindow: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
video_out_flip

Reproduce

git fetch origin e8bd82771eea64506a12dd9686daf350ad2301ab b611e634f7fc024f0c13e342d96de744448ef1d9
git checkout b611e634f7fc024f0c13e342d96de744448ef1d9 && git merge --no-ff e8bd82771eea64506a12dd9686daf350ad2301ab
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

_{Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.}

[Ragbie]

@nikhilpanchal1 Yes, #15040 first, please; record Depends on: #15040 on yours and rebase once it lands. The libkernel.md change here is only the entry this PR rewrites, so the rebase should keep both texts.

Rebased onto main d23df92b38 (head f79dccff36) without conflicts; the previous Windows run was cancelled by the 45-minute job limit while linking, not by a test.

AI-assisted: yes (Claude Code)