Skip to content

feat(aot): discuss native ARM64 compiler architecture and integration - #1835

Draft
yabanana wants to merge 31 commits into
boykopovar:mainfrom
yabanana:feature/native-arm64-aot-compiler
Draft

yabanana wants to merge 31 commits into
boykopovar:mainfrom
yabanana:feature/native-arm64-aot-compiler

Conversation

@yabanana

@yabanana yabanana commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

What

Retain this PR as the architectural discussion and review record for the experimental offline x86-64 to native ARM64 compiler. The complete implementation is consolidated in #5021, including the runtime and runnable consumers. This draft preserves the original review context; it is not a second implementation to merge or a dependency that must land before #5021.

The architectural constraints remain native ARM64 AOT output, reuse of AnyPS5 platform services, explicit input/ABI/effect qualification and rejection of unsupported cases. The shipped executable contains no x86 interpreter, JIT or Rosetta dependency. The original proposal's typed native-call compiler and the broader guest-state compatibility backend have distinct qualified scopes; recovering native values and effects across caller chains remains the direction for further simplification and performance work.

The existing request for a runnable end-to-end consumer is retained in review history. The source-built hello_notify example, native provider startup and explicit Linux ARM64 target provide that reproduction path. They do not settle the maintainers' decision about architecture, dependency policy or in-tree acceptance. Further implementation review belongs in #5021; architecture and contributor coordination can remain here.

Tested

Discussion branch integrated with upstream 51acb79ac49eeabf2d0e1d377867f953eb1bcefa. All 466 local compiler/runtime tests and the conventions check passed on Apple M5 Max / macOS 27.2. Full combined consumer/runtime evidence is described on #5021 and must not be attributed to this narrower branch.

The source-built example and reproduction commands are in docs/dev/BUILD.md. Enable ANYPS5_RELINKER_ONLY, ANYPS5_NATIVE_AOT, ANYPS5_NATIVE_AOT_EXAMPLE and BUILD_TESTING. The bounded CI variant shortens the example delay; the runtime sleep behavior is unchanged. Full Linux/Windows builds still require maintainer dispatch; new-head hosted results are separate from local results and historical checks.

Checklist

@yabanana
yabanana force-pushed the feature/native-arm64-aot-compiler branch from 31cd065 to 7725ddb Compare October 9, 2026 08:12
@yabanana
yabanana marked this pull request as ready for review October 9, 2026 08:19
@yabanana

yabanana commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@boykopovar Could you review the architecture and integration of this initial compiler-only ARM64 AOT implementation, related to #1553? It is now ready for review at 7725ddba, rebased onto 6cc0503d. All 204 tests pass locally and the fresh native Apple ARM64 CI has passed: https://github.com/boykopovar/AnyPS5/actions/runs/37903536478. Full Linux/Windows and relinker validation is running separately. The pointer/ABI restrictions remain explicit in the description; the full runtime and complete game port are follow-up work.

@yabanana
yabanana force-pushed the feature/native-arm64-aot-compiler branch from 7725ddb to b7e3f8d Compare October 9, 2026 10:05
@yabanana

yabanana commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

@boykopovar Current head a2f4f368 is rebased onto main 745b775e, including merged #2099 and #2151. Both previous upstream CI blockers (SPIRV headers and Windows Clang without AMDGPU support) are addressed in this base. All 207 local CTest cases pass; the initial compiler-only scope and pointer/ABI limitations remain explicit. Under the new policy, could you dispatch Build on main with input pr=1835 for full Linux/Windows validation, and review the current head? Skipped full-build jobs are not counted as passes.

@yabanana
yabanana force-pushed the feature/native-arm64-aot-compiler branch from b7e3f8d to a2f4f36 Compare October 9, 2026 10:24
@adriadam10

Copy link
Copy Markdown
Collaborator

Thanks for the work and for keeping the scope explicit (compiler-only, limits recorded in BUILD/TechnicalDebt).

This adds a new area rather than a fix, so it needs a direction call before a code review:

  • ~5.4k lines of Python under core/aot/, two new third-party submodules (Capstone 5.0.6, pyelftools 0.32) and an opt-in ANYPS5_NATIVE_AOT CMake option.
  • A new CI workflow (.github/workflows/native-aot.yml) that only runs on Apple Silicon runners; CONTRIBUTING has no rule for that.
  • It is a parallel path to the macOS x86-64 port in macOS support #1, with no runtime consumer yet (no entry-point caller or import implementations).

@boykopovar @oneandonlydean: do you want this in-tree at this stage (new submodules, the Apple-only workflow, the long-term ARM64 direction from #1553), or should it live out of tree until a runtime exists? We can't build or run it on Linux, so the claims that depend on macOS/ARM64 (the 212 tests, the emitted code) remain unverified on our side.

@oneandonlydean

Copy link
Copy Markdown
Collaborator

I'd keep it out of tree until there's a runtime that runs something with it: we can't build or check it here, and it brings two submodules and an Apple-only workflow with no runnable output yet. The ARM64 direction (#1553) is the owner's call.

This was referenced Nov 4, 2026
@oberonlift

Copy link
Copy Markdown
Contributor

Hi all. On @Betr0x's point above, that the condition for taking an ARM64 path into the tree was a runtime that runs something: I have one that does, and it is now up as PRs.

ps5lift is a static recompiler for PS5 executables. It lifts a whole executable and its modules from x86-64 to LLVM IR and compiles them ahead of time to aarch64; there is no interpreter or JIT at run time. Its runtime (#15798) is what the recompiled program links: it maps the guest images, dispatches indirect calls, runs x87 code on Berkeley SoftFloat, answers cpuid as the PS5's CPU and hands every import to AnyPS5's own libraries, which build natively for arm64 with a new ANYPS5_GUEST_LIFTED option.

What it runs today, on an NVIDIA DGX Spark (GB10, aarch64 Linux):

  • EVO Player, the homebrew media player, lifted from its PS5 executable, runs to its home screen at 60 fps.
  • 2,288 of AnyPS5's own tests pass in the arm64 build of the libraries for lifted programs, on lavapipe.
  • x86-64 test programs, covering SSE through AVX2, FMA, F16C and x87, print the same output lifted on arm64 as natively on x86-64.
  • Commercial games lift, link and start, but none is playable yet.

The PRs, in merge order. The first eight make AnyPS5's libraries build and run for a lifted program; on x86-64 they leave behaviour unchanged, which I checked by comparing the compiled objects and by running AnyPS5's tests on x86-64 Linux before and after. The last three add the lifter:

  1. refactor: route the libraries' fenv, TSC, cpuid and pause instructions through helpers #15797 helpers for the libraries' CPU-specific instructions
  2. feat(ps5lift): add the runtime for lifted programs #15798 ps5lift's runtime
  3. feat(build): add ANYPS5_GUEST_LIFTED to build the libraries for ps5lift #15799 the ANYPS5_GUEST_LIFTED build option
  4. feat(libc): unwind guest frames and keep the host's exception runtime in lifted builds #15800 to feat: find a lifted program's guest images, load its modules and switch its fibers #15804 libc exceptions, va_list, x87 long double, libkernel threads, and modules and fibers
  5. build(tools): allow LLVM from the system #15805 lets the convention check accept the system LLVM the lifter builds against
  6. feat(ps5lift): add ps5lift, a static recompiler for PS5 executables #15806 the lifter
  7. feat(ps5lift): run lifted programs and titles on AnyPS5's libraries #15807 lifted programs and titles on AnyPS5's libraries

Branch ps5lift-preview has everything merged, with steps to try it.

This is a different approach from #5021, not a replacement for it, and I'd be glad to coordinate on directory layout, merge order or shared pieces. The first PR gathers the libraries' CPU-specific instructions in one header, which any ARM64 path can extend.

Native Apple Silicon support is coming along fast too. Lifted programs already run natively on an M4 Mac, and guest memory now sits away from the address ranges macOS reserves, so games get as far as rendering there. What remains is mostly GPU features MoltenVK lacks, such as shader clocks, which the shader recompiler will emulate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build ci .github docs libc libkernel needs rebase Conflicts with main, rebase needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants