Skip to content

fix(loader): defer Linux runtime module starts until requested - #6586

Merged
oneandonlydean merged 14 commits into
boykopovar:mainfrom
JJenkx:fix/linux-runtime-module-start
Oct 24, 2026
Merged

oneandonlydean merged 14 commits into
boykopovar:mainfrom
JJenkx:fix/linux-runtime-module-start

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What

Double Dragon Gaiden (PPSA16018, 01.000.013) historically stopped before main when a linked Unity plugin started through Linux DT_INIT without engine context. This change keeps explicit runtime-directory modules mapped for bindings and starts each module when the title explicitly loads it, forwarding the caller's exact argument pair, including zero/null.

The Linux loader initializes each requested module once, returns the entry result to the load that performs initialization, handles concurrent/reentrant loads, and runs generated finalizers only after initialization completes. Standard SDK folders keep eager initialization, including browser conversions. Present zero-address DT_INIT entries retain their relocation; absent tags leave a null lifecycle slot. No title-specific contexts or default argument blocks are supplied.

Rebased onto main d6eafa54d0464258574fd2f71889754d05a5bc93, head fa3e35112131790755dcddadc54cf2c7b8dea37e. Prerequisite #6684 is merged; this branch contains only the fourteen topic commits, with no merge commits. The rebase preserves the production and test changes; range-diff differences are documentation context. A merge-tree check against newer main 7e2ba73e4c8c26b5d632b0d2f46166ffc1b8a29d is clean, and conventions pass against it. Build and test results below apply to the stated d6eafa5 base.

Review focus: initialization timing for linked runtime-directory modules and repeated-load result semantics. Uniform Windows deferral was accepted in the #8162 review; that is related design precedent, not a console measurement or approval of this Linux implementation. An additional real-kernel diagnostic on production-source head cea585242429fe0e20df3a92d670fa51e8a6437d observed entry result 0x5a17 for the first load and 0 for a repeated load, including a concurrent waiter; the entry and init array still ran once. Console repeated-load result semantics are unmeasured, so result retention across subsequent loads is not claimed. Earlier integration runs reached Double Dragon Gaiden character selection; gameplay is not established. Evergate remained black with and without the change. Those title observations are historical and were not rerun on this head.

Tested

Current-main refresh, 2026-10-23: head 2c98914fe4d95fc406b585c377412c7b48675bdc is a linear 14-commit rebase onto pinned main 4997726a5f6b8cd263461508945a4676647ef482. The only conflict was the dlopen load-failure block: this refresh preserves main's LoadFailure diagnostics and keeps deferred initialization after a successful load. The existing reentrancy, nested-handle, unload/reload and real condition-wait coverage is retained.

Fresh Linux GCC 16.2.1 Release, independent worktree/build with exact submodules: guest_runtime_lifecycle passes three consecutive repetitions, and relinker_conversion passes three consecutive repetitions, with no skips. The relinker and lifecycle probe are freshly built. Conventions and whitespace checks pass. These are CPU synthetic module/loader and conversion regressions; no new native title, Windows, GPU or performance result is claimed. The existing documented lifecycle/ABI limitations remain.

Arch Linux x86-64, GCC 16.2.1, Ryzen 9 7950X. Results below distinguish the public test head from the private runtime integration.

  • At public head fa3e35112131790755dcddadc54cf2c7b8dea37e, clean-configured and built the relinker, all runtime libraries (libs) and guest_runtime_lifecycle_tests. Passed 239/239 relinker tests and the real-kernel lifecycle wrapper covering all six modes: never-start, start, zero-start, concurrent, reentrant and reload. No failures or skips. The concurrent initializer remains blocked until the second loader thread enters pthread_cond_wait, and the test requires that wait to return. This wrapper is confined to the Linux test executable. Source and artifact hashes were independently checked; conventions and whitespace checks pass. This is not the complete runtime test suite.
  • Clean-rebase patch, range-diff, build log, test logs/JUnit and independent review, SHA-256 fcf647f928390b33f0fa21374c136d3ed72f8236687078e80a2d376fdd1cb1cd.
  • Darkest Dungeon II (PPSA19591) on that pinned private integration passed the narrated opening and prologue selection, rendered the stagecoach scene, and reached the party tutorial. The 573-second run exited 0 after the scheduled ordinary window close at 571 seconds; the temporary kernel limit was restored to 64 MiB. Screenshots at 300/450/550 seconds show those scenes. Combat, overall rendering correctness and performance remain unverified.
  • Waiter regression, test log and patch, native report, supervisor log and screenshots, SHA-256 3bfa3c0f62f8257ef7dd6980c52077b16442c182c4f426954f1abbfd1c3d0b80.
  • Historical GitHub CI at previous public head 624609bfe2: relinker jobs pass on Linux, Windows and both macOS configurations; relinker ASan passes. These do not establish full Windows/macOS runtime or runtime sanitizer coverage. Full Linux/Windows jobs were skipped in the automatic PR workflow. The current upstream workflow runs them on main or via manual workflow dispatch; marking this ready for review does not trigger them. A maintainer-run full build is still requested.

Earlier validation at its recorded revisions: 238/238 relinker tests passed; the reentrant negative control deadlocked when only the same-thread reentrancy return was removed; unload/reload coverage confirmed mapping removal and one finalizer sequence per initialized lifetime. Nested-handle regression and negative-control evidence, unload/reload evidence, and earlier full relinker validation.

No full runtime suite, Windows/macOS runtime, runtime sanitizer, GPU-validation or performance result is claimed.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none; fix(relinker): run a guest DT_INIT of 0 as the module initializer #6684 is merged
  • Fixes: none
  • Unblocks: none; Double Dragon Gaiden PPSA16018 evidence is historical
  • GPUs tested: n/a; loader and relinker fixtures
  • AI-assisted: yes

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 8717 core/libs/prx/libkernel/CMakeLists.txt L225-229
PR 9187 docs/dev/debt/libkernel.md L127
PR 9228 docs/dev/debt/relinker.md L7-8
PR 9882 docs/dev/debt/libkernel.md L127-128
PR 9947 docs/dev/debt/libkernel.md L127

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4563 core/libs/prx/libkernel/CMakeLists.txt
PR 4630 docs/dev/debt/libkernel.md
PR 5021 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Module/src/DynamicLoader.cpp
docs/dev/debt/libkernel.md
docs/dev/debt/relinker.md
docs/user/USAGE.md
PR 5925 docs/user/USAGE.md
PR 6044 core/libs/prx/libkernel/CMakeLists.txt
PR 6149 core/relinker/relinker/tests/test_guest_module_args.py
docs/dev/debt/libkernel.md
PR 6171 docs/user/USAGE.md
PR 6740 docs/dev/debt/libkernel.md
PR 6830 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 6924 docs/user/USAGE.md
PR 6936 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 7049 docs/dev/debt/libkernel.md
PR 7088 docs/dev/debt/libkernel.md
PR 7105 docs/dev/debt/libkernel.md
PR 7121 docs/dev/debt/libkernel.md
PR 7149 docs/dev/debt/libkernel.md
PR 7201 core/libs/prx/libkernel/CMakeLists.txt
core/libs/prx/libkernel/Module/src/DynamicLoader.cpp
PR 7509 docs/dev/debt/libkernel.md
PR 7511 docs/dev/debt/libkernel.md
PR 7642 docs/user/USAGE.md
PR 7653 docs/dev/debt/libkernel.md
PR 7918 docs/dev/debt/relinker.md
docs/user/USAGE.md
PR 7942 docs/user/USAGE.md
PR 8010 docs/dev/debt/libkernel.md
PR 8076 core/relinker/relinker/src/guest/GuestModuleBuilder.cpp
docs/dev/debt/relinker.md
PR 8520 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 8553 docs/dev/debt/libkernel.md
PR 9074 docs/dev/debt/libkernel.md
PR 9123 docs/dev/debt/libkernel.md
PR 9156 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9160 docs/dev/debt/libkernel.md
PR 9199 docs/dev/debt/libkernel.md
PR 9284 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9293 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9297 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9302 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9408 docs/dev/debt/relinker.md
docs/user/USAGE.md
PR 9415 core/libs/prx/libkernel/Module/src/DynamicLoader.cpp
PR 9508 docs/user/USAGE.md
PR 9557 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9561 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9566 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9575 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9693 core/libs/prx/libkernel/CMakeLists.txt
docs/dev/debt/libkernel.md
PR 9945 core/libs/prx/libkernel/CMakeLists.txt
PR 10193 docs/user/USAGE.md
PR 10319 docs/dev/debt/libkernel.md

@SandstromPL

Copy link
Copy Markdown
Contributor

Hi @JJenkx, a heads-up on how this interacts with #6684. That PR runs a guest DT_INIT of 0 as the module initializer, for the bundled libc.prx/libSceFios2.prx of PS4 Until Dawn (CUSA02636), where tag 12 is present with value 0. It adds GuestImage::HasInit, set when tag 12 is present, and the writers test it instead of Init != 0.

Merging the two:

  • The only textual conflict is passesModuleArgs in LinuxGuestModuleWriter.cpp (L32). I resolved it as image.HasInit && !deferInitialization.
  • One line that doesn't conflict also has to change: pointer(image.Init, image.Init != 0); (L98 at 8838a92bb) becomes pointer(image.Init, image.HasInit);. Otherwise a deferred module whose DT_INIT is 0 gets no relocation for lifecycle slot 2. InitializeLinuxGuestModule then sees table[2] == 0 (DynamicLoader.cpp L144) and skips the initializer, recording result 0 with no error.

I checked this on main 5d9dcc4 + #6684 + 8838a92 (relinker-only build) with a --module-dir plugins module whose DT_INIT is 0. With only the textual resolution, the slot at table+0x10 has no relocation. With the one-line change it gets R_X86_64_RELATIVE (0x4010, addend 0). The relinker-only suite passes 183/183 either way, so it won't catch this. Whichever PR lands second needs that line. If this one lands first, I'll add it to #6684.

AI Assisted

@JJenkx
JJenkx force-pushed the fix/linux-runtime-module-start branch from 8838a92 to 6c8ff59 Compare October 16, 2026 05:04
@JJenkx

JJenkx commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Addressed in 6c8ff59838b5dfcff2cf64cd32e71eb1dff415ce. The branch is refreshed onto current main 5d9dcc42d plus #6684 (ab9d21afcfbc, authorship retained), now declared as a prerequisite. The textual resolution uses image.HasInit && !deferInitialization, and lifecycle slot 2 now uses pointer(image.Init, image.HasInit).

I extended guest_zero_initializer with deferred Linux cases: DT_INIT zero with/without preinit, native initializer argument/return verification on Linux x86-64, and an absent-tag control. The new regression fails on the textual-resolution-only tree because table+0x10 has no relocation. With the one-line correction, the relinker-only build and all 183/183 tests pass; conventions/whitespace pass. No fresh full-runtime/title run on this head. The PR remains a draft for the existing initialization-timing review.

Windows CI then exposed a test-fixture portability issue (a Windows host path supplied as a Linux guest run path). Fixed in a17a8daf10acc9fbbf192b92e7b9ad0ae3e82bbf by using $ORIGIN/libs; the production initializer correction is unchanged. All 183 local tests pass again. The fresh GitHub relinker matrix now passes Linux, Windows and macOS on this exact head. Full runtime builds were skipped.

@JJenkx

JJenkx commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Temporarily closing this draft to free a slot under the project's 20-open-PR limit for the validated ATRAC9 vibration startup fix. The source branch fix/linux-runtime-module-start is retained at a17a8da for later reopening. The zero-DT_INIT feedback is addressed and its relinker checks pass; the initialization-timing review remains outstanding.

@JJenkx JJenkx closed this Oct 16, 2026
@JJenkx JJenkx reopened this Oct 19, 2026
@github-actions github-actions Bot added unblocks title Names a title ID on the Unblocks line needs rebase Conflicts with main, rebase needed labels Oct 19, 2026
@JJenkx
JJenkx force-pushed the fix/linux-runtime-module-start branch from a17a8da to f937385 Compare October 19, 2026 02:28
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 19, 2026
@SandstromPL

Copy link
Copy Markdown
Contributor

@JJenkx heads-up, since this carries #6684's first commit: with #6112 reverted, I've updated #6684 by merging main and adding two commits.

I replayed your branch onto main plus updated #6684 locally. The first five of your own commits apply cleanly, and 25266b497 has one conflict in test_guest_zero_initializer.py, where both add lines around the native load. After #6684 lands, the rebase should drop 2197de555 as already applied.

AI-assisted: yes (Claude Code)

@JJenkx
JJenkx force-pushed the fix/linux-runtime-module-start branch from f937385 to 85b281b Compare October 19, 2026 02:44
@JJenkx

JJenkx commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Updated to main 8a526d4 and carried all three topic commits from #6684 at f77a9ea, preserving authorship and omitting its merge commit. The HasInit dependency check and ASan host-loading option are retained. The deferred zero-initializer fixture now supplies libkernel.prx and checks dependency presence for present/absent DT_INIT.

At 85b281b: 149/149 relinker-directory CTests and all four focused loader/initializer tests pass without skips; conventions and whitespace pass. Local ASan execution was not tested. The PR remains draft for lifecycle-timing review. Thanks for identifying the dependency update.

@github-actions github-actions Bot removed the unblocks title Names a title ID on the Unblocks line label Oct 20, 2026
savvaniss added a commit to savvaniss/AnyPS5 that referenced this pull request Oct 20, 2026
The browser converter treats sce_module as a runtime directory, so the
Windows deferral of runtime modules the executable needs also deferred
libc.prx and libSceFios2.prx, which titles never load themselves.
Modules in sce_module, sce_modules and prx that the executable needs
are started at startup again, as the Linux deferral of boykopovar#6586 keeps
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
@JJenkx

JJenkx commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Windows applicability. The runtime change in DynamicLoader.cpp is entirely under Linux guards: a _WIN32 preprocessor view of the file is unchanged by this PR. guest_runtime_lifecycle is registered only on Linux. The Windows-relevant part is the relinker, which builds and converts modules on every host.

Run. Head fa3e351121 merged onto main ee8a7349b5 (clean, giving 7aeb681d33). Windows 11 Enterprise 10.0.22621, WinLibs GCC 15.2.0 UCRT, Release, BUILD_TESTING=ON, AMD Radeon RX 6700 XT (AMD proprietary driver 26.10.1). Runs had VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation (Khronos 1.4.309, never reported missing), ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1, used ctest -V and ran three times. The main baseline is ee8a7349b5, built the same way.

relinker_conversion, which includes the new runtimeLifecyclePolicy case (Media/Plugins deferred; sce_module, sce_modules and prx keep the mapping initializer), passes 3/3. main's version passes 3/3 too.

Not run on Windows: the Python test_guest_* relinker tests, which need the Linux guest module path, and a game run. Data only; I haven't touched the branch.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 23, 2026
@JJenkx
JJenkx force-pushed the fix/linux-runtime-module-start branch from fa3e351 to 2c98914 Compare October 23, 2026 15:26
@JJenkx

JJenkx commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed onto main 4997726a5f6b8cd263461508945a4676647ef482; new head 2c98914fe4d95fc406b585c377412c7b48675bdc keeps all 14 topic commits in a linear history. The conflict resolution retains upstream LoadFailure diagnostics, with Linux deferred initialization only after successful load.

Fresh Linux GCC 16.2.1 Release: rebuilt the relinker and lifecycle fixture; guest_runtime_lifecycle passes 3/3 repetitions and relinker_conversion passes 3/3, zero skips. Existing reentrant, nested-handle, unload/reload and real condition-wait coverage remains. Conventions and whitespace checks pass.

Exact-head receipt and full focused logs. This is synthetic CPU loader/conversion validation; no new native-title, Windows or performance claim, and the existing ABI/lifecycle debt remains.

AI-assisted: yes.

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 23, 2026
@JJenkx

JJenkx commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Additional native title diagnostic on exact main 4997726a5f6b8cd263461508945a4676647ef482 versus this PR 2c98914fe4d95fc406b585c377412c7b48675bdc, Space Adventure Cobra: The Awakening PPSA17337. One sequential pair on RX6650XT/RADV/Linux. Each relinker independently converts the same ten original ELF inputs with --registry --no-patches; each run uses its own coherent libraries, fresh writable seed and the same scripted input/settings. Inputs remain hash-identical.

  • Stockmain exits134 at16.0s: AGC graphics: storage texture has no committed guest pages, before the first scheduled capture.
  • Main+thisPR reaches the readable main menu (80s), then Mission1-1 story introduction (160s), and closes normally at173.0s (exit0). Saves/settings are restored and neither arm leaves a native process. The menu subtitle has a visible white overlay/clipping artifact.
  • Validation is actually loaded in both. Both report the same three VUID types: independentBlend00605, vertex-buffer00627 and index-buffer08784. This is not a validation-clean rendering result.

Sanitized pinned report, baseline failure excerpt and two reviewed captures.

This is one diagnostic pair, not a repeatability or causal claim: the baseline storage-texture stop may be timing-sensitive. It supports further title testing, not gameplay, complete title support, image accuracy or a loading/FPS gain. It complements the synthetic lifecycle tests without replacing them.

@JJenkx

JJenkx commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Cobra repeat qualification for my previous comment6369913549: the startup difference did not reproduce as a loader-specific improvement.

I repeated the same pinned main499/PR2c98914f conversions in reversed order, with empty private writable seeds and the same startup settings/input prefix. This time the loader arm exits134 before20s with the same storage texture has no committed guest pages error, while stockmain closes normally and reaches a rendered first-level desert/platform scene with a health HUD at305s (animated story at240s). Configuration/save cleanup succeeds in both.

Across these two diagnostic pairs, each revision has one failure and one success; the first arm fails in each pair. This small sample does not establish the cause or a failure rate. Please treat the original menu/mission captures as an observation only, not evidence that this PR unblocks Cobra. Interactive gameplay and complete title support remain unverified.

Reversed-repeat receipts and reviewed stockmain captures. I am continuing the shared startup/rendering issue on newer source separately; the synthetic lifecycle evidence for this PR is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants