Linux: when a title bundles its own libc.prx, both libc instances run the application-heap handshake, so the title initialises its main pool twice #693

Activity

1. jeffrey12cali commented on Oct 5, 2026

   

   jeffrey12cali

   on Oct 5, 2026

   ContributorAuthor

   More actions

   Confirmed, with the caller evidence pinned down (breaking by address, since name breakpoints do not
   resolve inside .prx files):

   ARENA_INIT from app0/sce_module/libc.prx.guest.prx+0xa288   thread 1
   ARENA_INIT from libs/libc.prx+0x37600                        thread 1   (ApplicationHeapInitialize's call_once body)
   

   So both libc instances really do drive the title's allocator initialisation, and the title's pool
   builder takes a hard-coded len=0x10000000, type=12, prot=0xf3 - only the number of runs can differ,
   which is why this shows up as two 256 MB arenas on Linux and one on Windows/Kyty.

   What we do locally, and it is worth an engine-side answer: after relinking, overwrite the first
   instruction of the bundled copy's DT_INIT and each DT_INIT_ARRAY entry with ret. Clearing
   DT_INIT itself is not a fix - the loader calls it regardless and lands in trap-filled padding
   (SIGTRAP before any output, both log files empty). Excluding the module is not a fix either: other
   bundled modules resolve real symbols from it (libresonanceaudio.prx.guest.prx: undefined symbol: ...).

   Result: exactly one 256 MB main pool, matching both references.

   Two notes for whoever picks this up. First, ApplicationHeapInitialize does run on Linux (traced via
   [heap] initialize params=0x... then [heap] register api=0x...), so the handshake itself is fine -
   it is the duplication that is wrong. Second, this issue alone was not the last blocker: the remaining
   fault after fixing it (plus #692 and #694) was that sceKernelRaiseException handed the title a
   fabricated mcontext, never filled PthreadPrivate::stackAddress on Linux, and treated a title thread
   handle and AnyPS5's adopted record for the same host thread as unrelated objects - so IL2CPP's
   stop-the-world returned success without pausing anything and the collector scanned a stack range it was
   told about wrongly. Happy to file that separately if it is useful.
2. ahir061 commented on Oct 7, 2026

   

   ahir061

   on Oct 7, 2026

   Contributor

   More actions

   I traced the current lifecycle paths on main. Both replacement libs/libc.prx and the converted bundled libc.prx.guest.prx remain distinct loaded images, and Linux republishes the bundled module's DT_INIT/DT_INIT_ARRAY, so the native loader runs its lifecycle independently. Replacement libc also reaches the application's allocator initialization through its own ApplicationHeapInitialize path, whose once guard only covers that runtime.
   Before implementing a fix, could you clarify the intended ownership model when both libc implementations coexist?
   Should both remain active runtimes but share a single application-heap initialization/finalization lifecycle, or is the bundled libc intended primarily as an export provider whose init/fini may be omitted?
   I can implement either direction with synthetic regression coverage, but I don't want to encode the title-specific ret workaround as generic policy without confirming the intended semantics.
3. jeffrey12cali commented on Oct 7, 2026

   

   jeffrey12cali

   on Oct 7, 2026

   ContributorAuthor

   More actions

   Answering the ownership question directly: the second option. A bundled copy of a system library should be an export provider whose lifecycle is suppressed by the loader, not a second active runtime that shares state with the replacement. I would not encode the ret patch as generic policy either - the behaviour is right, the place it is done is wrong.

   What we measured, which is what makes the first option (both active, one shared heap lifecycle) unattractive:

   • The two images are genuinely distinct runtimes: ApplicationHeapInitialize's guard is a std::once_flag per image, and the allocator registration is per-image state, so "share one heap lifecycle" needs process-wide shared state behind the replacement's API, not a once guard.
   • With both lifecycles live on PPSA28997 the title builds its main pool twice: the bundled copy completes the PS5 allocator handshake a second time and the second ApplicationHeapRegister either has to replace an active allocator or be ignored, and the title's own accounting then double-counts the arena. We saw two arenas where the reference trace has one.
   • The bundled image still has to load, because other bundled modules resolve symbols from it. Only its DT_INIT/DT_INIT_ARRAY/DT_FINI are the problem.

   So the general rule I would want in the engine: when the loader is about to initialise a converted module whose identity is a system library that is already present as a replacement (libs/<name>.prx), run the replacement's lifecycle and record the converted copy as initialised without calling into it - i.e. suppress init/fini by identity, in the loader, not by rewriting bytes in the file. That also covers the sce_module vs libs duplication for any other system library a title ships, and it keeps the "missing imports fail at startup" property intact, because the exports are still published by the copy that gets loaded.

   Our runner currently does the narrow thing your question anticipates: it writes ret over the bundled libc.prx.guest.prx init targets at prepare time, which is exactly a title-specific fix in the wrong layer, and I would be glad to delete it the moment the engine does the above. Happy to test any variant against this title, and I have the Windows reference trace of the same commit to compare heap and pool counts against - it shows one application-heap handshake, which is the number to match.
4. ahir061 commented on Oct 7, 2026

   

   ahir061

   on Oct 7, 2026

   Contributor

   More actions

   I opened a PR implementing the identity-based lifecycle ownership discussed above: #1156
   Bundled system libraries remain loaded for exports, while their init/fini lifecycle is suppressed when the matching AnyPS5 replacement is present. The change includes synthetic regression coverage for both Linux and Windows conversion paths.
   Native title testing would be very helpful if you’re able to check it against PPSA28997 and compare the application-heap/pool count with your reference trace.
5. jeffrey12cali commented on Oct 7, 2026

   

   jeffrey12cali

   on Oct 7, 2026

   ContributorAuthor

   More actions

   That is exactly the model I hoped someone would write, and the identity-based suppression is the right layer - thank you.

   I cannot test it as-is on this project today: our tree is a fork pinned to sos-v0.2.2 (c8830699), which is 18 commits behind main, and #1156 does not apply there (core/libs/CMakeLists.txt, core/relinker/CMakeLists.txt, both guest module writers and GuestImage all drift). Re-pinning onto main is already on our list, and #1156 is a reason to do it sooner rather than later; I will report the heap/pool comparison against our Windows reference trace as soon as I can build on a matching base.

   Two things worth knowing from our measurements, in case they help the review:

   • The symptom is exactly as you describe and is easy to confirm: with both lifecycles live, the title performs the PS5 application-heap handshake twice and builds two main pools where the reference trace shows one. Our stopgap is a prepare-time ret over the bundled copy's init targets, i.e. the same behaviour you are implementing, in the wrong layer - so when fix(relinker): suppress duplicate system library lifecycle #1156 lands we delete ours.
   • One overlap to flag for whoever reviews: fix(libkernel): pass module args and argp to runtime-loaded modules on Linux #1102 (the args/argp delivery for runtime-loaded modules, from sceKernelLoadStartModule discards the module parameter block, so modules that validate it fault at start-up #918) also edits LinuxGuestModuleWriter.cpp, in the same lifecycle area - it replaces the zeroing DT_INIT trampoline with one that reads the pending pair. Both PRs are correct and complementary, but they will conflict, and whichever lands second needs to keep the other's shape. Worth a line in whichever description gets updated first.

   Also, since you asked about native testing generally: PPSA28997 now boots through its managed bootstrap and renders an in-engine scene, so it is a usable target for lifecycle regressions - and I can compare allocator/pool counts against a Windows reference trace of the same commit, which is the specific check you asked for.
6. ahir061 commented on Oct 7, 2026

   

   ahir061

   on Oct 7, 2026

   Contributor

   More actions

   Thanks, this is really helpful. I’ve added a note about the #1102 overlap in the PR so both changes can be preserved if there is a merge conflict.

   Also, thanks for offering to test PPSA28997 later. The heap and pool count comparison against the Windows reference trace should be very useful once your branch is on a compatible base.
7. adriadam10

   closed this as completedin #1156on Oct 8, 2026
8. igordev23 commented on Oct 9, 2026

   

   igordev23

   on Oct 9, 2026

   More actions

   Still happening on Linux with PPSA28997 (God of War: Sons of Sparta) after #1515 reverted #1156. On current main the duplicate heap handshake now aborts at start-up instead of corrupting memory later.

   Build

   • main f1451396 (2026-10-09) + feat(libkernel): implement _umtx_op wait and wake operations #1482 (85c43cde, for _umtx_op, which the title imports)
   • plus a local workaround for sceKernelLoadStartModule discards the module parameter block, so modules that validate it fault at start-up #918 (guest module INIT stubs pass a zeroed placeholder in rsi instead of NULL); without it the title faults in call_init reading (%rsi)
   • native Linux output, AMD Ryzen APU (Barcelo, RADV), Ubuntu, GCC 15
   • the title bundles sce_module/libc.prx

   Result (right after opening /app0/Media/boot.config)

   [libc] unhandled exception 'St13runtime_error' thrown from 0x7cac0c9be567
   terminate called after throwing an instance of 'std::runtime_error'
     what():  RTLD application heap: cannot replace an active allocator
   

   Exit 134. The throw is the guard in registerApplicationHeapApi (core/libs/prx/libkernel/System/src/Rtld.cpp): sceKernelRtldSetApplicationHeapAPI is called a second time with a different allocator table, i.e. both libc instances run the application-heap handshake, as described in this issue.

   Older build without that guard (df16c4c, same title): no start-up error, but after running for a while glibc aborts with corrupted size vs. prev_size (the process then hangs and is killed by timeout). That fits the "everything downstream is inconsistent" description above.

   Could this issue be reopened, since the fix was reverted? Is there a planned follow-up that suppresses the second handshake without breaking Silksong (#1500)? I can test a branch on PPSA28997 / Linux and provide logs or a backtrace.
9. jeffrey12cali commented on Oct 11, 2026

   

   jeffrey12cali

   on Oct 11, 2026

   ContributorAuthor

   More actions

   Reopening this, because @igordev23 is right and I am the author so the obligation is mine to check.

   Why it is not resolved. #1156 was the fix. #1515 reverted #1156, and the reason given is sound: with the
   bundled sce_module/libc.prx loaded for exports but its init/fini suppressed, a title that still calls
   that copy's malloc gets NULL, because its mspace was never set up (#1500). So the suppression traded one
   failure for another and the duplicate handshake this issue describes is live again. The abort @igordev23
   reports is the guard in registerApplicationHeapApi (core/libs/prx/libkernel/System/src/Rtld.cpp), i.e. the
   same second registration, now failing fast instead of corrupting the heap later. That is a strict improvement in
   diagnosability and nothing else.

   A data point from a tree where this title boots, with the caveat attached. Our fork does not suppress the
   bundled libc's lifecycle at all. Instead, in core/libs/prx/libc/src/ApplicationHeap.cpp, it adopts a working
   allocator when none was registered:

   void adoptUnregisteredApi() {
   #ifdef __linux__
       if (heapApi[0] == nullptr && !heapFailure && !heapFinalized) heapApi = defaultApi();
   #endif
   }

   With that, PPSA28997 - which does bundle sce_module/libc.prx - reaches its main menu and holds 60 FPS
   through 15 minutes of play on Linux. The caveat matters: our base predates the registerApplicationHeapApi
   guard, so this is evidence that "make the second registration harmless rather than fatal or suppressed" can
   work for a title that bundles libc. It is not a tested patch against current main, and I am not claiming it
   would fix PPSA01474 or the #1500 case.

   The design constraint, stated as I read it. #1500 rules out any approach that leaves the bundled copy's
   allocator uninitialised, because titles call it by address. That leaves two families: give both copies a
   functioning allocator, or route them to one. Suppression is in neither family, which is why #1156 could not
   survive.

   @igordev23 one thing that will save you a step: #1102 merged on 2026-10-09, so the local workaround you carry
   for #918 (passing a zeroed placeholder in rsi for module INIT stubs) should no longer be needed - the
   parameter block is now forwarded on Linux and asserted by the guest_module_args test.
10. jeffrey12cali commented on Oct 11, 2026

    

    jeffrey12cali

    on Oct 11, 2026

    Last edited by jeffrey12cali

    ContributorAuthor

    More actions

    Correction to the first line of my comment above: I wrote "Reopening this" and then found I cannot. The reopen
    is refused for my account (Could not reopen the issue), so #693 is still showing as closed. The request stands
    and @adriadam10 you closed this, so the question is yours: given that #1515 reverted the fix and the duplicate handshake
    still aborts at start-up on PPSA28997, this issue is not resolved.