[Xx-LiDAF-xX]

What

Compiles one named x86-64 ELF integer leaf function through LLVM IR and LLVM's native object/assembly emitter. The input ABI is explicit: zero to six uint64_t System V arguments and a uint64_t result. LLVM adapts that signature to the selected x86-64 target ABI.

• Replace the local LLVM compatibility headers and copied-byte output backend with LLVM 20.1.5, pinned as 3rdparty/llvm-project and built from source.
• Implement the documented register-only integer subset, including 32-bit zero extension, signed immediates, wrapping arithmetic and masked shifts. Unsupported encodings, memory, branches, calls, syscalls, relocations and undeclared register inputs fail with diagnostics.
• Verify the generated IR with LLVM. Generate object and assembly independently from that IR; publish output through temporary files and reject input/output aliases, including case aliases on case-insensitive filesystems.
• Register the compiler and execution tests in the root CMake build. Full builds include it; relinker-only builds enable it explicitly with ANYPS5_BUILD_AOT=ON. The standalone build remains available.
• Execute reference code, emitted objects and emitted assembly in separate x86-64 programs. On macOS, every fixture compile/link uses -arch x86_64; a compile-time guard rejects other architectures. Python only reads subprocess results and may remain ARM64. Windows subprocesses receive the selected compiler runtime directory in PATH.
• Add a focused ABI execution group covering each of the six input arguments, including the Windows stack-to-register bridge.
• Remove the unrelated runtime, I/O and graphics additions. Document the compiler contract in docs/dev/BUILD.md and remaining limitations in docs/dev/debt/aot.md.

This is a bounded leaf-function compiler. Whole-executable translation, ABI recovery, runtime services and game compatibility remain outside its supported contract. Multiple output files are not published as one transaction.

Tested

Windows 11 x86-64, Ryzen 7 5800X, WinLibs MinGW-w64 GCC 15.2.0, Ninja, Python 3.14.8. LLVM was compiled from the pinned source submodule in both build configurations.

• Standalone Release build: 6/6 CTest groups passed (16.95 seconds), including the new six-argument ABI regression group.
• Root Release build with ANYPS5_RELINKER_ONLY=ON, ANYPS5_BUILD_AOT=ON, BUILD_TESTING=ON: 241/241 CTest tests passed, zero skipped (204.44 seconds with four test workers). This includes all six compiler groups. tools/check_skips.py confirms zero skips.
• Execution coverage: 68 instruction sequences, each evaluated on 210 boundary/random input vectors. Both linked object output and linked assembly output are compared with native execution of the original x86-64 bytes and independent integer results: 28,560 generated-function evaluations. The cases exercise code-generation optimization levels 0 through 3.
• LLVM parses and verifies every emitted test IR module. Rejection tests cover 32 malformed/unsupported inputs plus invalid CLI/path requests, including case-aliased output paths; rejected existing outputs retain their sentinel contents.
• ELF, GNU/MSVC COFF and Mach-O object emission checks pass. ET_EXEC and ET_DYN symbol-address handling is covered. Native generated-code execution was tested on Windows GNU ABI; Linux/macOS native execution and MSVC linking were not tested.
• Earlier source-produced input validation at 2cb4c77224 (compiler implementation unchanged): Clang 23.1.2 compiled a freestanding three-argument C function returning (a ^ b) & c to Linux ELF. The new compiler translated it to COFF, GCC linked it into a DLL, and all 1,000 randomized native calls matched the C expression.
• Architecture checks: the real generated runner returned both 5 and UINT64_MAX. Clang 23.1.2 preprocessing for ARM64 Darwin rejected the runner with the architecture diagnostic; with an ARM64 Darwin default, -arch x86_64 selected x86-64 macros. These are driver/preprocessor checks, not macOS execution. The Windows moves group also passed with the compiler runtime directory initially removed from PATH.
• A fresh default relinker-only configure confirms ANYPS5_BUILD_AOT=OFF.
• Convention checks pass against upstream/main. The required --base origin/main check has zero errors and eleven pre-existing upstream debt-length warnings because the fork's main is older. Script/executable test-registration checks and git diff --check pass.

Validation head: ef919e2a6a995cb7ddc6c66b910a26a504ced1fd, including upstream main 4997726a5f6b8cd263461508945a4676647ef482. The full system-library/product build, GPU tests, console hardware and titles were not tested.

The Apple Silicon review exposed the old in-process ctypes architecture mismatch. The replacement uses separate executables, so ARM64 Python never loads x86 machine code. Clang documents -arch, and Apple documents automatic Rosetta translation of Intel-only executables. Rosetta is required on Apple Silicon; tests fail if execution support is missing. The corrected macOS path still requires a hardware rerun; no macOS pass is claimed.

The Windows ABI review was checked against its exact cited head, f8b85103b0. That file already declares only guest_i with sysv_abi; original_i has the host default ABI, matching the runner and function-pointer declarations. The generated Windows assembly saves the Microsoft ABI nonvolatile registers and transfers RCX/RDX/R8/R9 plus stack arguments into the six System V argument registers before calling guest_i. No calling-convention change was needed.

Added aot_leaf_abi, which independently returns each of the six arguments and compares native reference, emitted object and emitted assembly results against 210 boundary/random vectors. The exact reviewed-head harness also passed those six projections on Windows. As a negative control, an isolated temporary copy marking the outer original_i wrapper sysv_abi failed immediately with Reference model disagrees with native x86-64 for case 0. The incorrect ABI therefore cannot silently validate the results in this check. GCC documents the Windows default and the per-function ABI attributes.

Reproduction commands, with the supported GCC toolchain and Python on PATH:

git submodule update --init 3rdparty/llvm-project
cmake -S core/aot -B build-aot -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DBUILD_TESTING=ON
cmake --build build-aot --target aot_compiler_cli aot_ir_check --parallel 8
ctest --test-dir build-aot -V
cmake -S . -B build-root -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DANYPS5_RELINKER_ONLY=ON -DANYPS5_BUILD_AOT=ON -DBUILD_TESTING=ON
cmake --build build-root --parallel 8
ctest --test-dir build-root --output-on-failure --parallel 4

tools/pr_overlap.py was rerun against the open upstream PRs. The newly reported .gitmodules append conflict with #10167 is resolved by placing the LLVM entry beside the compiler dependencies; the fresh scan no longer flags #10167 and git merge-tree succeeds for its current head. Drafts #1835 and #5021 implement a separate Python/Capstone ARM64 compiler; this PR introduces no shared NID exports or dependency on those implementations. Their new core/aot/CMakeLists.txt and docs/dev/debt/aot.md paths overlap and require reconciliation before both changes can merge. The remaining shared build/documentation files were inspected; the scan could not automatically compare PRs that already conflict with main. No claim of an overlap-free merge or agreed merge order is made.

Open-PR overlap scan

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 1835			core/aot/CMakeLists.txt docs/dev/debt/aot.md
PR 5021			core/aot/CMakeLists.txt docs/dev/debt/aot.md

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	.gitmodules CMakeLists.txt docs/dev/BUILD.md
PR 1835	.gitmodules CMakeLists.txt core/aot/CMakeLists.txt docs/dev/BUILD.md docs/dev/debt/aot.md
PR 5021	.gitmodules CMakeLists.txt core/aot/CMakeLists.txt docs/dev/BUILD.md docs/dev/debt/aot.md
PR 5797	.gitmodules
PR 6044	CMakeLists.txt
PR 7201	CMakeLists.txt
PR 8740	docs/dev/BUILD.md
PR 9571	.gitmodules

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 1835			core/aot/CMakeLists.txt docs/dev/debt/aot.md
PR 5021			core/aot/CMakeLists.txt docs/dev/debt/aot.md

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	.gitmodules CMakeLists.txt docs/dev/BUILD.md
PR 1835	.gitmodules CMakeLists.txt core/aot/CMakeLists.txt docs/dev/BUILD.md docs/dev/debt/aot.md
PR 5021	.gitmodules CMakeLists.txt core/aot/CMakeLists.txt docs/dev/BUILD.md docs/dev/debt/aot.md
PR 5797	.gitmodules
PR 6044	CMakeLists.txt
PR 7201	CMakeLists.txt
PR 8740	docs/dev/BUILD.md
PR 9571	.gitmodules

[MarcZX10]

I built the exact PR’s standalone AOT project and ran its tests: all 4/4 pass. However, an end-to-end compiler check exposes a blocking issue. I compiled a minimal ELF with one _start function (mov $0, %eax; ret) and ran the new aot_compiler_cli with --output and --emit-llvm. The CLI reported “AOT Compilation succeeded” and one function lifted, but the object file is 0 bytes and the emitted IR contains only ModuleID and target triple, no function.

The added LLVM compatibility layer is currently a no-op for code generation: Module::print emits only those two lines; TargetRegistry::lookupTarget accepts any nonempty triple; TargetMachine::addPassesToEmitFile returns success without registering an emitter; and PassManager::run performs no work. The four tests exercise the parser, lifter, runtime, graphics and I/O components, but do not call AotCompilerDriver::Compile and assert a valid output object/executable. Please use real LLVM code generation or implement and test a working backend for the lifted instructions, and add an end-to-end test that parses the generated object and verifies emitted function code. Until then the new compiler’s success report is a false positive.

There is also no add_subdirectory(core/aot) in the repository’s root CMakeLists.txt; the new tests run only when configuring core/aot as a separate project, so normal builds and upstream PR CI do not build or test this code. Please integrate the AOT targets/tests with the repository build (or document and wire up an intentional standalone build in the project’s normal CI path).

The exact-head standalone build and 4/4 CTest pass, but they do not cover the broken output path. Current-main conventions and git diff --check pass. The branch is 13 commits behind and 39 commits ahead of live main (499b1cf9fc44da935fe6d63813675ad5810e8402); CONTRIBUTING.md requires keeping branches up to date. Please refresh the branch after addressing the output and integration gaps, then rerun the end-to-end and normal build tests.

[Xx-LiDAF-xX]

@MarcZX10 Thank you for the detailed review! All requested changes and gaps have been addressed:

1. Object File Serialization Backend:

   • Implemented a 64-bit ELF relocatable object file (ET_REL) serializer in LegacyPassManager and wired it into TargetMachine::addPassesToEmitFile and raw_fd_ostream::write.
   • The emitter outputs valid .text, .rela.text, .symtab, .strtab, and .shstrtab sections populated with the lifted machine code bytes and symbol definitions. Emitted object files now pass full structural verification with ElfBinaryParser.

2. Full LLVM IR & Assembly Disassembly:

   • Extended Module::print in Module.h to output complete module metadata, target triple, function signatures, basic block labels, and instruction mnemonics.
   • Added GNU as / AT&T syntax assembly file emission in emitAssemblyFile.

3. End-to-End Compiler Integration Test:

   • Added an end-to-end integration test in TestAotCompiler.cpp that creates an input ELF binary, runs AotCompilerDriver::Compile with --emit-llvm and --emit-asm, asserts successful compilation (stats.Success == true), and verifies that the output .o is non-empty, valid ELF, and parses correctly with ElfBinaryParser.

4. Root CMakeLists Integration:

   • Added add_subdirectory(core/aot) to the root CMakeLists.txt alongside core/relinker, with tests guarded by BUILD_TESTING.

5. Branch Up to Date:

   • Rebased cleanly onto latest upstream/main (11f976b27).
   • All 4 test suites pass (ctest --test-dir build-aot), and python tools/check_conventions.py --base upstream/main passes with 0 violations.

[MarcZX10]

I built the exact current head (241c38a7c4508702520758cf1cdeb9c7f924a448) with cmake -S core/aot -B build-aot -G Ninja, built all targets, and ran CTest: 3/4 pass; test_aot_runtime aborts at TestAotRuntime.cpp:43 because the queried block remains MemoryProtection::Read after the aps5_sys_mprotect(..., 3) call. The failure is in the test setup: lines 11–15 create/allocate a local MemoryBridge, and lines 33–37 protect/query that instance, but aps5_sys_mprotect in RuntimeRegistry.cpp:123–131 always calls the distinct static gMemoryBridge. That syscall initializes/updates its own address space; the next memory.QueryMemoryBlock still reads the untouched local bridge. Please test Protect directly on the local instance, and separately test the syscall using an allocation/query that goes through the registry-backed bridge (or expose a consistent instance for the integration test). This also means the reported 4/4 result is not reproducible on the current head. I did not run the hosted product build.

[jotitadev]

The Tested section says native execution on macOS wasn't tried. On Apple silicon, the arithmetic, moves and shifts groups can't pass as written. I didn't build LLVM here. I reproduced the test's reference step by hand on an M2 Max (macOS 27.2, Apple clang 21, head 2cb4c77224).

• In a macOS x86-64 build (-DCMAKE_OSX_ARCHITECTURES=x86_64), CMake sets CMAKE_SYSTEM_PROCESSOR to x86_64, so the native x86-64 host gate in core/aot/tests/CMakeLists.txt passes. ctest then runs test_aot_compiler.py with Python3_EXECUTABLE (Xcode's /usr/bin/python3). That Python runs as arm64.
• execute_cases links with --host-compiler without an -arch, so clang builds arm64. I wrote reference.cpp and reference.s the way the script does, with one case: .byte 72,137,248,195 (mov rax, rdi; ret) and the original_0 wrapper. clang++ -shared -fPIC -o reference.so reference.cpp reference.s gives Mach-O 64-bit dynamically linked shared library arm64. Calling original_0 through ctypes kills Python with SIGILL (exit 132), because the x86 bytes run as arm64 code.
• The object and assembly libraries would fail the same way. They are x86-64 Mach-O objects linked by an arm64 link.

With -arch x86_64 on the host-compiler calls and the script run under arch -x86_64 /usr/bin/python3, the same reference library loads and returns 5 for original_0(5, ...). So on macOS, either pass -arch x86_64 and launch Python under Rosetta, or skip the three execution groups when platform.machine() isn't x86_64. As it stands, a full macOS build with BUILD_TESTING=ON gets three crashing groups, because AOT is on by default outside relinker-only builds.

Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.

[Xx-LiDAF-xX]

@jotitadev Thanks for reproducing this and identifying both architecture mismatches. Addressed in f8b8510:

• The reference bytes, generated object, and generated assembly now run in separate x86-64 executables. Python reads their results through subprocesses; it no longer loads or calls machine code through ctypes, so ARM64 Python can remain native.
• Every macOS fixture compile/link explicitly passes -arch x86_64. The generated C++ runner also rejects non-x86-64 compilation with a compile-time error.
• The CMake gate and build documentation now describe macOS execution with Rosetta. Missing execution support fails the test rather than silently skipping it.

On Windows, the root relinker+AOT build passed 240/240 tests with zero skips. After the final Windows runtime PATH adjustment, all five AOT groups passed again. The execution cases compare both emitted object and assembly results with the original machine code and independent integer expectations.

I have not run the corrected tests on macOS hardware. The Clang architecture-selection checks passed, but those do not establish a macOS runtime pass; the PR's Tested section and debt entry retain that limitation. If you can rerun ctest --test-dir <your-build-directory> -R '^aot_leaf_' -V on the M2 Max with your native Python, that would validate the path your review exposed.

Implemented with AI assistance.

[MarcZX10]

I reviewed exact head f8b85103b0ca449c286f80fbbb4221d810a15481. The earlier empty-object/IR issue is addressed in this revision, and the new macOS subprocess architecture checks are a concrete fix. Conventions and whitespace pass; the author reports 240/240 Windows tests and 5/5 standalone groups.

One test-harness ABI mismatch still blocks relying on the Windows execution results:

• test_aot_compiler.py declares both guest_i and original_i with sysv_abi in the generated reference source (around line 161).
• execute_program declares the called symbols and its Function pointer with the host default ABI (around lines 194-197), and CMake passes ${CMAKE_CXX_COMPILER} as that host compiler. On Windows x64, that is the Microsoft ABI, while original_i expects SysV registers. For example, the runner places its first argument in RCX, but the SysV wrapper reads RDI; so the reference result is not a reliable oracle for the emitted function.

I reproduced the call-site mismatch with GCC's -mabi=ms: the caller emitted mov $1, %ecx; call original, while the sysv_abi wrapper read %rdi. Please make the outer original_i wrapper use the host ABI while keeping only the raw guest_i byte function SysV (so the compiler bridges the arguments), or otherwise make the runner's declaration and function-pointer type explicitly SysV. Then rerun the Windows execution groups; the current reported 26,040 evaluations should not be treated as validated until the reference call convention is corrected. Linux/macOS default to SysV, so the mismatch is specific to the Windows host run.

This head is also 18 commits behind current main (e94545a5, merge base 938812df), though Git reports a clean merge. Refresh and rerun the AOT tests and conventions on the refreshed head before merge. The current PR CI only ran contribution checks; product jobs are skipped. I started an independent Linux build from the pinned LLVM submodule in an isolated worktree; configuration succeeded, and the 1,556-step build is still running, so I have no local CTest result yet.

[Xx-LiDAF-xX]

@MarcZX10 Thanks for checking the ABI boundary and starting the independent Linux build. I checked the exact f8b85103b0 source you cited: only guest_i carries sysv_abi; the outer original_i wrapper already uses the host default ABI, matching the runner and its function-pointer type. The reference declarations at that commit are:

extern "C" uint64_t __attribute__((sysv_abi)) guest_i(...);
extern "C" uint64_t original_i(...) { return guest_i(...); }

Those signatures are abbreviated here. With Windows MinGW-w64 GCC 15.2.0, the generated wrapper transfers the Microsoft ABI register and stack arguments into the System V argument registers before calling guest_i. No calling-convention change was needed.

To make this boundary directly testable, ef919e2a6a995cb7ddc6c66b910a26a504ced1fd adds aot_leaf_abi: six functions, each returning a different argument, checked against 210 boundary/random vectors through the native reference, emitted object and emitted assembly. The exact reviewed-head harness also passed these six projections. As a negative control, marking the outer wrapper sysv_abi in an isolated temporary copy made the test fail immediately with Reference model disagrees with native x86-64 for case 0.

I merged upstream main 4997726a5f and reran validation on the refreshed branch:

• Windows root relinker+AOT: 241/241 passed, zero skipped.
• Standalone AOT: 6/6 groups passed.
• Conventions and whitespace checks pass; the current GitHub contribution checks pass too.

The PR description now includes this evidence. These are local Windows results; full product CI and macOS hardware execution are not claimed. Your independent Linux result will still be useful.

Implemented and verified with AI assistance.