Skip to content

feat(libkernel): map regular files with mmap on Linux - #5053

Merged
adriadam10 merged 3 commits into
boykopovar:mainfrom
YossiYad:feat/libkernel-mmap-regular-files
Oct 14, 2026
Merged

adriadam10 merged 3 commits into
boykopovar:mainfrom
YossiYad:feat/libkernel-mmap-regular-files

Conversation

@YossiYad

Copy link
Copy Markdown
Contributor

What

On Linux, mmap maps only shared memory descriptors (shm_open) and fails with EOPNOTSUPP for every other descriptor, including regular files. SQLite in WAL mode maps its -shm index file with mmap(MAP_SHARED), so that failure becomes disk I/O error: Dusklight (PPSA99305) logs Failed to set pragmas: disk I/O error and SQLite DB init failed for its GPU cache and pipeline cache at every start and runs without them. With strace, the guest's mmap of the 32 KiB -shm file stops at the descriptor check after the F_DUPFD_CLOEXEC/F_GETFL validation; no host mmap is attempted.

  • mmap accepts a regular file's descriptor (host fstat reports S_ISREG) besides shared memory descriptors, with the existing access checks (EACCES for a shared writable mapping of a read-only descriptor) and the existing DoMapFileBacked path, shared or private.
  • guest_memory maps a two-page file shared (reads both pages, a write reaches the file), private (a write stays in the mapping), at a page offset, and through a read-only descriptor (shared writable fails with EACCES, read-only works); a pipe still fails with EOPNOTSUPP.
  • A technical debt entry records the sources (FreeBSD 11 mmap(2), shadPS4's MemoryManager::MapFile for regular files), that console behaviour is not measured, that other descriptor kinds still fail, and that Windows maps no files.

Tested

Linux (Bazzite 44 host, Fedora 44 distrobox, GCC 16.2.1, Ninja, Release, -DBUILD_TESTING=ON), based on main bb9a4d8a:

  • Full build, libs, and full ctest: 1017 pass, 8 skip, 1 fail. The failure is agc_driver_tessellation (SEGFAULT), which fails the same way on unchanged main on this machine.
  • guest_memory against unchanged main's PosixMemory.cpp fails at the shared mapping of the regular file, and passes with the change.
  • Dusklight (PPSA99305, catalog v0.3.0, relinked with --trap-sdk-syscall-stubs; it needs the posix_spawn exports of feat(libSceLibcInternal): implement posix_spawn actions #3651 to load, so that PR was merged locally for the title run only): before the change the log has the two disk I/O error failures above; with it, SQLite cache init succeeded, dawn_cache.db-wal and pipeline_cache.db-wal are written, and no disk I/O line remains. Without a disc image the title stays at its menu, as expected.
  • tools/check_conventions.py --base upstream/main, git diff --check and tools/pr_overlap.py (no overlap) pass. Windows and macOS were not run; this change does not touch their paths.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository
  • No new third-party code
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none (Dusklight PPSA99305 logs the failure and continues without its caches)
  • AI-assisted: yes

@github-actions github-actions Bot added libkernel docs needs rebase Conflicts with main, rebase needed labels Oct 13, 2026
@YossiYad
YossiYad force-pushed the feat/libkernel-mmap-regular-files branch from b33341a to 3f12dce Compare October 13, 2026 12:47
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 13, 2026
@simonbronk

Copy link
Copy Markdown
Contributor

Tested on Linux x86-64 (Ubuntu 24.04 in a Colima VM with Rosetta on Apple M4, GCC 13.3), this PR (3f12dcef3) merged into main 39890031e, Release build with BUILD_TESTING=ON.

  • ctest: 1062/1069 pass, including the tests in core/libs/tests/GuestMemory.cpp. The only failures are the 7 that also fail on main in this environment (Rosetta and VM limits: guest_math, amd64_only_converter, agc_driver_depth_compare_copy, tls_function_coverage_register_address_loads, linux_executable_exports, linux_entry_argv, linux_input_program_headers). No new failures.

AI-assisted: yes (Claude Code).

@babyhuey

Copy link
Copy Markdown
Contributor

Linux title check (Ubuntu 24.04, GCC 13, Release, headless), head 3f12dce on its base: guest_memory passes 5/5 (--repeat until-fail:5). Dusklight (PPSA99305, catalog release): on main b868383 its GPU cache logs Failed to start transaction: disk I/O error and SQLite DB init failed right after it opens dawn_cache.db-shm. With this PR it logs SQLite cache init succeeded. Both runs then stop at the headless DisplayWindow throw, as far as a run without a display gets.

@YossiYad

YossiYad commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @simonbronk and @babyhuey for the runs. Dusklight (PPSA99305) going from disk I/O error / SQLite DB init failed on main to SQLite cache init succeeded with the PR is exactly the case this is for: Dawn's WAL index (dawn_cache.db-shm) is a regular-file mmap, which main can't serve. The headless DisplayWindow throw after is as far as a no-display run gets, so that's the expected stop.

On the overlap with #5186: it's docs-only and adjacent. #5186 edits the ioctl bullet in docs/dev/debt/libkernel.md, while this PR adds a separate "Regular file mappings" bullet two lines above it. There's no shared code, exports or new files (the bot's own table shows nothing in those columns), so there's no dependency between them, just a one-line textual conflict in that doc for whoever merges second.

@YossiYad
YossiYad force-pushed the feat/libkernel-mmap-regular-files branch from 3f12dce to 6b8ddfb Compare October 14, 2026 06:08
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 14, 2026
@adriadam10 adriadam10 self-assigned this Oct 14, 2026
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked on the PR merged onto current main: guest_memory and the scoped test run (166 tests) pass, and with PosixMemory.cpp reverted the new CheckRegularFileMapping fails, so it pins the change. Access checks, EOPNOTSUPP for a pipe and the debt entry look right; the Dusklight runs by @simonbronk and @babyhuey confirm the SQLite WAL case. Merging.

@YossiYad

Copy link
Copy Markdown
Contributor Author

Thanks @adriadam10 for checking it, including the revert of PosixMemory.cpp. It still merges cleanly onto the latest main (e464db9e3); on that tree guest_memory passes and the full ctest here shows only agc_driver_tessellation, which segfaults on this machine without any change.

@adriadam10
adriadam10 merged commit a628765 into boykopovar:main Oct 14, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants