Skip to content

fix(libkernel): map shared memory and files executable on Windows - #12339

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
medazizktata:fix/windows-exec-shared-mapping
Oct 28, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
medazizktata:fix/windows-exec-shared-mapping

Conversation

@medazizktata

@medazizktata medazizktata commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What

On Windows, a guest mmap with PROT_EXEC and MAP_SHARED of a shared-memory object or a regular file threw "an executable shared file mapping is not implemented on Windows" (#12337). FreeBSD 11.4 maps these calls, and so do the Linux and macOS branches of DoMapFileBacked_nid_no_patch:

  • shared memory: uipc_shm.c 878-879 gives execute to any descriptor opened for reading;
  • regular files: vfs_vnops.c vn_mmap allows execute unless the file system is mounted noexec.

Changes:

  • DirectMemory.cpp, Windows branch:
    • When execute is requested, the section is created on a second handle that ReOpenFile opens with GENERIC_READ | GENERIC_EXECUTE, plus GENERIC_WRITE for a read-write descriptor. Its protection is PAGE_EXECUTE_READ or PAGE_EXECUTE_READWRITE, as CreateFileMapping requires. The extra handle is closed once the section exists. Mappings without execute take the existing path.
    • A shared writable mapping of a read-only descriptor fails with EACCES, as shm_mmap does (885-887). Before, shared-memory descriptors skipped the access check in PosixMemory.cpp.
  • SharedMemory.cpp: the owner ACE of named objects grants GX with GR, so a readable named object can be reopened with execute.
  • Tests:
    • New ctest guest_mmap_executable (guest_memory_tests executable-mappings, Windows and Linux). It runs shared protections 0-7 against a regular file opened read-write, read-only and write-only, an anonymous shm object, a named shm object opened read-write and read-only, and a JIT descriptor. For each it runs code from executable mappings, writes through writable ones, and checks the errno of refusals.
    • The executable block of GuestShmOpen.cpp now runs on Windows. Its last line (a pwrite past the end of an empty object returning 0) stays off Windows: it fails there independently of this change.
  • docs/dev/debt/libkernel.md: the Windows "no PROT_EXEC" gap is removed. What still differs is stated:
    • named objects created by earlier builds lack execute until fchmod;
    • write-only descriptors still differ from FreeBSD;
    • a file whose ACL denies execute fails with EACCES where FreeBSD maps it.

Tested

At this head (dbd826bc8, on main 3a40945e6), details:

tests only (no fix) with this change
Windows 11 Pro 10.0.26200, i5-12600KF, WinLibs GCC 15.2.0 r7, Release: guest_shm_open, guest_mmap_executable both fail 3/3 (0xc0000409) both pass
Windows: ctest -E agc — (main: 1685/1685) 1686/1686, the only difference is the new guest_mmap_executable
Pop!_OS 24.04, GCC 13.3, Release: guest_shm_open, guest_mmap_executable not run pass 5/5
Pop!_OS: ctest -E agc not run 1591/1605 at -j16; the 14 failures (12 of them timeouts) all pass rerun one at a time

Earlier, on commit 46fe6f24374a:

tests only (no fix) with this change
Windows: guest_shm_open, guest_mmap_executable both fail (0xc0000409) both pass
Windows: ctest -E "^agc" 1576/1578, the two target tests fail 1578/1578
Pop!_OS: ctest -E "^agc" 1500/1500 1500/1500
  • Linux: the tests pass without the change because only Windows code changes. To show they can fail there, a deliberate refusal added to the Linux branch makes both fail, and restoring it passes.
  • Windows deliberate breaks:
    • removing the ReOpenFile path makes both tests fail;
    • removing the EACCES check makes guest_mmap_executable fail on the errno check;
    • restoring the file byte-for-byte passes again.
  • Build logs show each changed file recompiled in every run.
  • tools/check_conventions.py --base origin/main: 0 errors. Two debt-length warnings, on entries that were already over 1000 characters on main (1752 and 2861; now 1843 and 3214).

Not tested:

  • what a PS5 does for these mappings (no console measurement exists);
  • macOS (not built);
  • a file whose ACL denies execute;
  • closing the guest descriptor while an executable view is mapped;
  • write-watch or GPU aliases over executable file views.

Checklist

A guest mmap with PROT_EXEC and MAP_SHARED of a shared-memory object or a regular file threw on Windows. The section is now created from a handle reopened with GENERIC_EXECUTE, as PAGE_EXECUTE_READ or PAGE_EXECUTE_READWRITE, the owner ACE of named objects grants execute with read, and a writable shared mapping of a read-only descriptor fails with EACCES, as FreeBSD 11.4 shm_mmap and vn_mmap do.

Fixes boykopovar#12337
@oneandonlydean oneandonlydean self-assigned this Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 10645 docs/dev/debt/libkernel.md L177-179
PR 12170 docs/dev/debt/libkernel.md L289-290

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4449 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
core/libs/tests/GuestMemory.cpp
PR 4563 core/libs/prx/libkernel/CMakeLists.txt
PR 4630 docs/dev/debt/libkernel.md
PR 5021 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
core/libs/prx/libkernel/File/src/SharedMemory.cpp
core/libs/tests/GuestMemory.cpp
docs/dev/debt/libkernel.md
PR 6149 docs/dev/debt/libkernel.md
PR 6321 core/libs/tests/GuestMemory.cpp
PR 6936 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7088 docs/dev/debt/libkernel.md
PR 7105 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
core/libs/tests/GuestMemory.cpp
docs/dev/debt/libkernel.md
PR 7121 docs/dev/debt/libkernel.md
PR 7135 docs/dev/debt/libkernel.md
PR 7149 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
docs/dev/debt/libkernel.md
PR 7192 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7447 docs/dev/debt/libkernel.md
PR 7464 docs/dev/debt/libkernel.md
PR 7509 docs/dev/debt/libkernel.md
PR 7631 core/libs/prx/libkernel/DirectMemory/DirectMemory.cpp
core/libs/tests/GuestMemory.cpp
PR 7804 docs/dev/debt/libkernel.md
PR 8605 docs/dev/debt/libkernel.md
PR 9156 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9242 docs/dev/debt/libkernel.md
PR 9293 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9297 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9302 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9383 core/libs/prx/libkernel/CMakeLists.txt
PR 9389 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9557 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9566 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 10319 docs/dev/debt/libkernel.md
PR 10442 docs/dev/debt/libkernel.md
PR 11002 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11215 docs/dev/debt/libkernel.md
PR 11625 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11849 docs/dev/debt/libkernel.md
PR 11872 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 11969 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/File/src/SharedMemory.cpp
docs/dev/debt/libkernel.md

@medazizktata

Copy link
Copy Markdown
Contributor Author

Re-run on the current main 3a40945e6, on the same Windows machine as the description (Windows 11 Pro 10.0.26200, i5-12600KF, WinLibs GCC 15.2.0 r7, Release, --parallel 4):

guest_shm_open guest_mmap_executable ctest -E agc
main 3a40945e6 passes (executable block compiled out) not registered 1685/1685 passed
main + this PR's tests only (CMakeLists and test files) fails 3/3 (0xc0000409) fails 3/3 at GuestMemory.cpp:3219 not run
main + this PR (dbd826bc8) passes passes 1686/1686 passed

The only full-suite difference is the new guest_mmap_executable. Build logs show DirectMemory.cpp, SharedMemory.cpp, GuestMemory.cpp and GuestShmOpen.cpp recompiled.

The selection -E agc excludes every test whose name contains agc.

@MarcZX10

Copy link
Copy Markdown
Contributor

One overlap follow-up: the refreshed overlap report also shows #12296 editing the same Windows named-shared-memory and regular-file-mapping debt entries. The production changes appear complementary, so this is a documentation/rebase coordination issue rather than duplicate functionality. Please coordinate merge order with #12296’s author and declare it under Depends on if this PR needs that branch to land first; otherwise note the agreed order and rebase if required.

@github-actions github-actions Bot added the fix error Closes an issue (Fixes #N) label Oct 28, 2026

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the Windows section-creation and view-protection path, including the ReOpenFile access mask, named-object owner ACE, shared read-only write rejection, and the new protection/access matrix. I do not see a defect in those paths in this diff. The Windows and Linux test results in the description are from 46fe6f24374a, while the submitted head is dbd826bc88a03afc62cd9df46881fa06ea1d6be2; the description explicitly says the current-main application was not rerun. Please run at least guest_shm_open and guest_mmap_executable on Windows at this exact head (and the focused executable-mapping test on Linux if available), then update Tested. I have not approved pending that refreshed evidence; hosted checks currently cover conventions/intake/labels only.

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my review comment: I overlooked the author’s October 28 10:47 follow-up in this thread. It reports a Windows run on the exact current head dbd826bc8 (Windows 11, WinLibs GCC 15.2): guest_shm_open passes, guest_mmap_executable passes, and ctest -E agc passes 1686/1686; the control with only the PR tests also reproduces the failures on main. My statement that these results were only from the earlier 46fe6f commit was incorrect. The exact-head Windows test-evidence request is satisfied. I have not yet cleared the separate coordination point about the overlapping Windows-memory debt edits in #12296.

@medazizktata

Copy link
Copy Markdown
Contributor Author

Thanks @MarcZX10. On #12296 (@JJenkx): the overlap is more than the debt text, so here is what I found and what I propose.

Production overlap. Both PRs refuse a writable MAP_SHARED mapping of a read-only shared-memory descriptor with EACCES, at different layers:

mmap_nid_postfix is the only caller of DoMapFileBacked_nid_no_patch on main (text search for the symbol). So once #12296 is in, this PR's check can no longer be reached. #12296's check is also the better place for it, since it covers regular files and shared memory alike.

The rest does not overlap:

Proposed order: #12296 first, since it was opened earlier and is narrower. Then I rebase this PR on it, drop the now-redundant EACCES check from DoMapFileBacked, and merge the two Windows debt entries (shm_open, shm_unlink and Regular file mappings). @JJenkx, does that order work for you? Once you confirm, I will record it as Depends on: #12296 in this PR's description.

Current main: of the 167 commits since 3a40945e6 (now d4cb89300), only three touch this PR's files: they add or edit unrelated libkernel.md debt entries (sceKernelDebugRaiseException, the sanitizer replace tables, sceKernelMapDirectMemory with execute). None touches this PR's code files, PosixMemory.cpp, WindowsMappings.hpp, GuestArena.cpp or Jit.cpp, and the PR merges into main without conflicts. So the Windows results above still apply to the merged code.

Linux at this exact head dbd826bc8 (Pop!_OS 24.04, kernel 7.1.5, GCC 13.3.0, Intel Core Ultra 7 155H; incremental build with SharedMemory.cpp, DirectMemory.cpp, GuestShmOpen.cpp and GuestMemory.cpp recompiled):

  • guest_shm_open and guest_mmap_executable pass in each of 5 runs (--repeat until-fail:5).
  • ctest -E agc -j16 --timeout 300: 1591 of 1605 passed. 12 of the 14 failures were timeouts (10 public_headers_compile_*, performance_compare, guest_pthread_start), and the other two were conventions_function_bodies_shard_6 and vulkan_validation_gate_attribution. All 14 pass when rerun one at a time with --timeout 900: 13 through --rerun-failed -j1, and guest_pthread_start by name in each of 5 runs (0.7 s each). I did not capture why the two non-timeout failures failed in the parallel run.

The production change is Windows-only, so this Linux run shows that the new and changed tests build and pass on Linux. It does not exercise the fix itself.

@oneandonlydean
oneandonlydean merged commit 319621b into boykopovar:main Oct 28, 2026
5 checks passed
dkdhanda8 pushed a commit to dkdhanda8/AnyPS5 that referenced this pull request Oct 28, 2026
…c-shared-mapping

fix(libkernel): map shared memory and files executable on Windows
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs fix error Closes an issue (Fixes #N) libkernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

libkernel (Windows): mmap with PROT_EXEC of a shared-memory object or regular file fails

3 participants