Skip to content

feat(libc): implement sceLibcBacktraceSelf and GetBufferSize on Windows - #8137

Merged
oneandonlydean merged 8 commits into
boykopovar:mainfrom
hamb3r:feat/libc-backtrace-windows
Oct 20, 2026
Merged

oneandonlydean merged 8 commits into
boykopovar:mainfrom
hamb3r:feat/libc-backtrace-windows

Conversation

@hamb3r

@hamb3r hamb3r commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

What

During the startup and runtime of Unity Engine titles such as God of War: Sons of Sparta (PPSA28997), the IL2CPP runtime (Il2cppUserAssemblies.prx) and Boehm garbage collector query stack backtraces via sceLibcBacktraceSelf and sceLibcBacktraceGetBufferSize.

PR #7060 implemented these exports for Linux using backtrace() and dladdr(), leaving other platforms calling NotImplemented_nid_no_patch(__func__). On Windows, attempting to capture a backtrace aborts the process immediately with an unhandled runtime error:

[libc] unhandled exception 'St13runtime_error' thrown from 00007ffa2999b720
terminate called after throwing an instance of 'std::runtime_error'
  what():  sceLibcBacktraceSelf not implemented
FATAL: abort() (SIGABRT) on thread 48464
    #0 0x00007ffa298d2c98 libc.prx+0x12c98

This PR implements the missing Windows support for CaptureFrames in libc/src/Backtrace.cpp:

  • Stack unwinding: Walks stack frames using AnyPS5's unwinder (_Unwind_Backtrace_nid_postfix / _Unwind_GetIP_nid_postfix), which unwinds relinked guest and AnyPS5 .ehmeta/.ehfram frames and steps through foreign PE frames without exception/termination handlers via VirtualUnwind (Unwind.cpp:417).
  • Module identification & safe path handling: Resolves the module base handle via GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, ...) and queries the module path through internal helper BacktraceModulePath_nid_no_patch(handle, initialCapacity), which starts at MAX_PATH in production and grows exponentially up to 32 KiB on ERROR_INSUFFICIENT_BUFFER or length == capacity.
  • Module normalization: Strips path separators (/\\) and trims the .guest.prx suffix, producing the same clean module names as the Linux implementation.
  • RVA calculation: Stores the 32-bit RVA offset relative to the module base.
  • Automated test: Ports guest_backtrace (core/libs/tests/GuestBacktrace.cpp) to Windows (calling BacktraceModulePath_nid_no_patch(handle, 1) to exercise the buffer-growth loop directly without depending on a long checkout path) and registers it in core/libs/prx/libc/CMakeLists.txt restricted to Linux and WIN32.
  • Technical debt: Updates docs/dev/debt/libc.md noting that both Linux and Windows are supported and documenting that on Windows, unwinding stops at a frame without .ehmeta or .ehfram tables whose PE unwind data has an exception or termination handler, leaving the trace empty or cut short while returning 0.

Tested

  • Automated Test Suite:

    • guest_backtrace in ctest passes 100% on Windows (MinGW-w64 GCC 15, Release):
          Start 195: guest_backtrace
      195: Test command: E:\AnyPS5_backtrace\build\tests\guest_backtrace_tests.exe
      195: Working Directory: E:/AnyPS5_backtrace/build/core/libs/prx/libc
      1/1 Test #195: guest_backtrace ..................   Passed    0.01 sec
      
    • Without this PR on Windows, guest_backtrace aborts with:
      195: [libc] unhandled exception 'St13runtime_error' thrown from 00007ffa2999b720
      195: terminate called after throwing an instance of 'std::runtime_error'
      195:   what():  sceLibcBacktraceGetBufferSize_nid_postfix not implemented
      195: FATAL: abort() (SIGABRT) on thread 48464
      1/1 Test #195: guest_backtrace ..................Exit code 0xc0000409
      ***Exception:   1.74 sec
      
  • Title Verification (God of War: Sons of Sparta, PPSA28997):

    • Environment: Windows 11 x64, AMD Ryzen 7 7700, NVIDIA GeForce RTX 3080 (Driver 566.14 / 32.0.16.1714).
    • Standalone main + this PR: Because Il2cppUserAssemblies.prx imports 7 Dinkumware thread helpers directly from libc.prx (which on main are currently exported only from libSceAmpr.prx via feat(libSceAmpr): implement the std::_Pad thread launch helpers #3743 / libkernel: execv/execve and the setuid/setgroups/setrlimit family are absent (XPSemu, PS5MC); fix branch ready #3558), launching against standalone main stops at the ELF import resolution stage before reaching guest code:
      [time] scheduler tick raised to 0.5 ms (was 0.50 ms, now 0.50 ms, timed waits fine)
      [+0.000000 s] [src/HostCpu.cpp:193 operator()] host CPU AMD Ryzen 7 7700 8-Core Processor: has SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, LZCNT, BMI1, BMI2, MOVBE, CMPXCHG16B, LAHF/SAHF, AVX, AVX2, FMA, F16C, AES, PCLMULQDQ, RDRAND, RDSEED, ADX, SHA, CLFLUSHOPT, CLWB, RDPID, XSAVEOPT, XSAVEC, SSE4A, PREFETCHW, CLZERO, OS AVX state (XCR0); lacks none
      FAIL: unresolved ELF import dGYo9mE8K2A from libc.prx
      FAIL: unresolved ELF import xZqiZvmcp9k from libc.prx
      FAIL: unresolved ELF import gjLRZgfb3i0 from libc.prx
      FAIL: unresolved ELF import np6xXcXEnXE from libc.prx
      FAIL: unresolved ELF import W0j6vCxh9Pc from libc.prx
      FAIL: unresolved ELF import YvmY5Jf0VYU from libc.prx
      FAIL: unresolved ELF import a-z7wxuYO2E from libc.prx
      ...
      GetLastError: 127 - The specified procedure could not be found.
      
    • With the 7 libc.prx Dinkumware thread imports resolved (tested together with feat(libc): export Dinkumware thread primitives from libc.prx #8148):
      • Before this PR: Once static imports resolve and Unity boots, Il2cppUserAssemblies.prx invokes sceLibcBacktraceSelf and terminates immediately with std::runtime_error: sceLibcBacktraceSelf not implemented.
      • After this PR: sceLibcBacktraceSelf returns 0 and populates valid frame records (Il2cppUserAssemblies.prx, libkernel.prx, libc.prx, etc.), allowing the title to pass engine initialization into menus and in-game rendering.

Checklist

@github-actions github-actions Bot added the unblocks title Names a title ID on the Unblocks line label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 6044 core/libs/prx/libc/CMakeLists.txt
PR 6226 docs/dev/debt/libc.md
PR 6469 docs/dev/debt/libc.md
PR 6819 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7081 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7201 core/libs/prx/libc/CMakeLists.txt
PR 7473 docs/dev/debt/libc.md
PR 7648 docs/dev/debt/libc.md
PR 7715 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8150 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8520 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8550 docs/dev/debt/libc.md

@hamb3r
hamb3r force-pushed the feat/libc-backtrace-windows branch from 590c396 to 47591f2 Compare October 20, 2026 00:28
ycbuzi added a commit to ycbuzi/AnyPS5 that referenced this pull request Oct 20, 2026
@oneandonlydean oneandonlydean self-assigned this Oct 20, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
Comment thread core/libs/tests/GuestBacktrace.cpp Outdated
HMODULE callerHandle = nullptr;
const auto address = reinterpret_cast<std::uintptr_t>(handle) + records.front().offset;
Require(GetModuleHandleExA(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
reinterpret_cast<LPCSTR>(address), &callerHandle) != 0 && callerHandle == handle,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Windows assertion currently proves only that the first frame resolves to the same executable module. The Linux branch also checks the exact caller symbol (caller.dli_saddr == self.dli_saddr), so it would catch an off-by-one in the skipped frames that this Windows branch could miss. Since handle came from &CheckBacktrace, could this assert that reinterpret_cast<std::uintptr_t>(handle) + records.front().offset == reinterpret_cast<std::uintptr_t>(&CheckBacktrace) as well?

@hamb3r hamb3r Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 28d0b16: since records.front().offset is the call-site return RVA inside CheckBacktrace rather than the function entry &CheckBacktrace, the Windows branch now brackets sceLibcBacktraceSelf_nid_postfix with CurrentIp() (address > before && address < after) and checks that records[1].offset matches __builtin_return_address(0).

@wachi35676

Copy link
Copy Markdown
Contributor

Head 57c8c0118 merged onto main 02fb1ce53 (clean merge).

Windows 11 (10.0.26300), Ryzen 5 9600X, WinLibs GCC 15.2.0 UCRT, Release, BUILD_TESTING=ON:

  • guest_backtrace: passes 3 of 3. The build log has no warnings. With Backtrace.cpp reverted to main, the test does not link (BacktraceModulePath_nid_no_patch is missing), so there is no run to compare against.
  • The Windows branch of the test only checks that the first record lies in the test's module, where Linux checks the exact function (dli_saddr). So I checked the offsets with a separate program. Its call chain is main -> LevelOne -> LevelTwo -> LevelThree -> sceLibcBacktraceSelf, every function noinline and without destructors. The program is linked against this libc.prx and has no .ehmeta tables.
    • Main thread: 5 records: bt_probe.exe+0x181a (inside LevelThree, 0x1760 to 0x18f0), +0x18f9 (LevelTwo), +0x1909 (LevelOne), +0x924f (main), and +0x13ba (CRT startup). The count and size match sceLibcBacktraceGetBufferSize.
    • On a CreateThread thread: the same first three frames, then the thread procedure, then KERNEL32.DLL+0x2dda7.
    • So the first record is the caller, and the RVAs land in the right functions.
  • One case returns nothing. When the caller is a host frame with a C++ cleanup (in the probe, the function that calls sceLibcBacktraceSelf held a std::vector), both calls return 0 with count = 0 and no records. The libc unwinder stops at a frame whose PE unwind data has a handler, which is the known limit in debt/libc.md. The RtlCaptureStackBackTrace fallback runs only when the unwind returns zero frames, so it did not take over here. Relinked guest frames carry the project's tables, so this probably affects only native callers. Still, a silent empty trace is not recorded in the debt entry: falling back whenever no frame is left after the two skipped ones, or a line in the debt entry, would cover it.
  • Nit: the libc/CMakeLists.txt hunk adds two blank lines after BinaryWordIo.cpp that are unrelated to the change.

Ubuntu 24.04 (WSL2), GCC 13.3, Release, same merge: guest_backtrace passes 3 of 3 and the build has no warnings. The Linux path is unchanged.

AI-assisted: yes (Claude Code); results above were run on this machine.

@hamb3r

hamb3r commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @MarcZX10 and @wachi35676 for the review and catches! Addressed in the latest commit:

  1. core/libs/tests/GuestBacktrace.cpp: On Windows, bracket the sceLibcBacktraceSelf_nid_postfix call with a [[gnu::noinline]] CurrentIp() helper and verify that records.front().offset lies strictly between before and after inside CheckBacktrace, and that records[1].offset matches __builtin_return_address(0) (the caller of CheckBacktrace in main).
  2. core/libs/prx/libc/src/Backtrace.cpp: Changed the RtlCaptureStackBackTrace fallback check from if (captured == 0) to if (captured <= 2) so that if _Unwind_Backtrace_nid_postfix stops at a caller frame after only the two skipped internal libc frames (CaptureFrames and sceLibcBacktrace*), RtlCaptureStackBackTrace takes over instead of returning an empty frame list.
  3. core/libs/prx/libc/CMakeLists.txt: Removed the two stray blank lines after BinaryWordIo.cpp.

@wachi35676

Copy link
Copy Markdown
Contributor

Re-ran 28d0b1664 merged onto main 02fb1ce53 (clean merge; also merges cleanly onto d50ec76da). Windows 11, WinLibs GCC 15.2.0, and Ubuntu 24.04 (WSL2) GCC 13.3, both Release.

  • guest_backtrace: passes 3 of 3 on each OS, with no warnings in either build log. On Windows the new CurrentIp() range check and the records[1] return-address check both hold. With Backtrace.cpp at main, the Windows test can't load (0xc0000139, BacktraceModulePath_nid_no_patch is missing), and the Linux test still passes because its path is unchanged.
  • With only Backtrace.cpp set back to 57c8c0118, the test still passes, so the captured <= 2 change is untested.

The case of a native caller with a C++ cleanup frame is unchanged. I re-ran the probe 3 times, and the output was identical with Backtrace.cpp at 57c8c0118 and at 28d0b1664:

  • When the function that calls sceLibcBacktraceSelf holds a std::vector, the call returns 0 with count = 0.
  • In gdb, the fallback now runs (RtlCaptureStackBackTrace is called from CaptureFrames), but it returns 2. Both of those frames are then skipped. libc.prx is built with -fno-asynchronous-unwind-tables, and its .pdata has no entry for CaptureFrames or sceLibcBacktraceSelf_nid_postfix. So the system unwinder can't step out of CaptureFrames, and the fallback can't give caller frames from this build.
  • When the cleanup frame is higher up the stack, the trace is cut at that frame. A cleanup frame one level up gives 1 record, and one three levels up gives 3 records. Without a cleanup frame the full chain of 6 records comes back.

So the change doesn't fix the empty trace. Building Backtrace.cpp alone with -fasynchronous-unwind-tables doesn't assemble with this toolchain (Error: .seh_handlerdata used outside of .seh_proc block), so it isn't a one-line fix. A line in the debt/libc.md entry would cover it: behind a frame with a handler, the trace is empty or cut short, and the call still returns success. The stray blank lines in libc/CMakeLists.txt are gone.

AI-assisted: yes (Claude Code); results above were run on this machine.

@hamb3r

hamb3r commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @wachi35676 for digging into that in gdb! Since libc.prx is built with -fno-asynchronous-unwind-tables (.ehfram without .pdata entries for CaptureFrames / sceLibcBacktrace), RtlCaptureStackBackTrace indeed cannot step out of CaptureFrames. Removed the unreachable RtlCaptureStackBackTrace fallback from Backtrace.cpp and documented the PE unwind handler behavior in docs/dev/debt/libc.md.

@hamb3r

hamb3r commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Rebased / synced with main (50a4d5bfb3). cc @oneandonlydean

@oneandonlydean
oneandonlydean merged commit c8301bb into boykopovar:main Oct 20, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs libc unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants