Skip to content

fix(libkernel): end unwinding at the guest main stack's entry frame - #15691

Open
GiannisG1 wants to merge 1 commit into
boykopovar:mainfrom
GiannisG1:fix/libkernel-main-stack-unwind-end
Open

GiannisG1 wants to merge 1 commit into
boykopovar:mainfrom
GiannisG1:fix/libkernel-main-stack-unwind-end

Conversation

@GiannisG1

@GiannisG1 GiannisG1 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What

Since #7532, EnterGuestMainStack_nid_no_patch moves the stack pointer to the guest main stack and calls the guest entry from inline asm (Thread.cpp L263-L274). The compiler's unwind information for EnterGuestMainStack still describes its frame on the host stack. An unwinder that returns into it from the guest entry computes a CFA past the top of the guest main stack and reads there. libc's _Unwind_Backtrace faults in LibcUnwind::Step at 0x7eeffc0e8, 0xe8 above the console's main stack top. On main 04a9eb4fa2, vkQuake (PPSA99010) and RetroArch (PPSA99169) crash this way at startup. Before #7532, vkQuake exited with 0 and RetroArch stopped at sceSystemServiceLoadExec on this PC.

The stack switch is now a routine of its own, Aps5CallOnGuestMainStack_nid_no_patch, whose unwind information marks the return address undefined (.cfi_undefined %rip), as glibc's _start (start.S L57-L59: "Clearing frame pointer is insufficient, use CFI.") and clone (clone.S L92-L94) mark their outermost frames. libc's unwinder reads DW_CFA_undefined as a return address of 0 (Unwind.hpp L215, Unwind.cpp L654) and stops there (L676), so a walk from the guest main stack ends at the entry with _URC_END_OF_STACK. gdb's backtrace ends there too. The routine runs the instructions the inline asm ran.

guest_main_stack (both modes) now calls _Unwind_Backtrace from the entry on the guest main stack. It checks that the walk ends with _URC_END_OF_STACK, passes at least the entry and the routine, and that no frame's CFA is above the stack top.

Landing order: #15803 (with #15804 and #15807 stacked on it) wraps EnterGuestMainStack_nid_no_patch in ANYPS5_GUEST_LIFTED on the same lines. How the two compose is on #15803 (#15803 (comment)); whichever lands second rebases.

Tested

Linux Mint 22.3, kernel 7.0.0-38, glibc 2.39, GCC 13.3.0, Release, BUILD_TESTING=ON, head on main 04a9eb4fa2 (merges cleanly with current main):

  • guest_main_stack and guest_main_stack_adopted: 3/3 each. On 04a9eb4fa2 both fail 3/3 with a segfault in LibcUnwind::Step, at the instruction where the titles crash.
  • readelf --debug-dump=frames: the routine's CIE ends with DW_CFA_undefined: r16 (rip).
  • vkQuake (PPSA99010) and RetroArch (PPSA99169), catalog builds relinked by 04a9eb4fa2's relinker, GTX 1050 Ti under Xvfb: with 04a9eb4fa2's libkernel.prx both segfault in LibcUnwind::Step after 1 s and 2 s. With this libkernel.prx, vkQuake exits with 0 and RetroArch stops at sceSystemServiceLoadExec: /app0/picker/picker.bin without its converted executable, as before fix(linux): enter the guest entry on a main stack at the console's address #7532.
  • tools/suite.py split run against a --full run of 04a9eb4fa2: all 3828 tests on the GTX 1050 Ti, and the 932 that depend on the GPU on the HD Graphics 630 and llvmpipe. No new failure and no new skip; the HD Graphics 630 fails agc_driver_f64_rounding_denormal_modes and agc_driver_image_sample_lod_clamp on 04a9eb4fa2 too (Mesa).
  • check_conventions.py, check_test_registration.py, check_main_sync.py and check_compile.py are clean. pr_overlap.py --branch: the 28 open PRs that share files merge cleanly with this one, and the 9 it could not compare (they conflict with main) change other functions of Thread.cpp.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: PPSA34502 (Call of Duty: Black Ops II): its switch to the Zombies executable, which crashes in LibcUnwind::Step past the guest main stack top, per milescoviello's runs on an RTX 4070 Ti (fix(libkernel): end unwinding at the guest main stack's entry frame #15691 (comment)); PPSA99010 (vkQuake) and PPSA99169 (RetroArch), catalog homebrew, back to their state before fix(linux): enter the guest entry on a main stack at the console's address #7532
  • GPUs tested: n/a
  • AI-assisted: yes

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added unblocks title Names a title ID on the Unblocks line libkernel labels Nov 4, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

Hi GiannisG1, thanks for the focused unwind-boundary regression and the paired startup results for PPSA99010 and PPSA99169. Please remove the generated attribution footer below the checklist; the template checkbox can stay.

@GiannisG1

Copy link
Copy Markdown
Contributor Author

@oneandonlydean removed, thanks; the AI-assisted checkbox stays. I took the same footer out of my other open pull requests too.

@github-actions github-actions Bot added review queue Next in the maintainer review queue: checks pass, no open change request, no conflict and removed review queue Next in the maintainer review queue: checks pass, no open change request, no conflict labels Nov 4, 2026
@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 3109f4f merged onto main 5b1d381 · tested 2026-11-04 14:53 UTC · 27 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 2,966 passed · 8 failed (0 new, 7 also fail on main, 1 unstable on main) · 61 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 3,035 of 3,883 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (1065 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ agc_driver_pipeline_journal 120.6 s, ⚠️ public_headers 90.2 s, ⚠️ agc_driver_image_sample_unnormalized 44.8 s, ⚠️ conventions_pr_template 39.9 s, ⚠️ agc_shader_sampler_limits 37.3 s
Unstable on main (1, these also failed in recent main runs on this machine; not counted against this PR)
agc_driver_pipeline_journal: failed in 8 of the last 10 main runs
Failing on main too (7, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
agc_driver_submit_progress
agc_submit_progress
agc_submit_progress_no_timeline
guest_mmap_executable
Passed on retry (1, failed in the parallel run, passed when re-run with a 360 s timeout; not counted against this PR)
public_headers: Timeout in the parallel run; also passed only on retry in 15 other runs in the last 7 days
Validated but not compared (2, not in the main baseline's validation pass)
public_headers
public_headers_compile_18
Validation IDs already present on main (8 IDs in 626 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (626 tests): agc_driver, agc_driver_address_passes, agc_driver_address_passes_draw_transitions, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, and 618 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_srgb8_color_target
VUID-VkSamplerCreateInfo-addressModeU-01079 (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeU-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeV-parameter (1 test): agc_driver_image_gather_volume
VUID-VkSamplerCreateInfo-addressModeW-parameter (1 test): agc_driver_image_gather_volume
Skipped (61)
usage: check_skips.py [-h] [--job {linux,windows}] [--list] [--require-tests]
                      junit
check_skips.py: error: /home/ben/anyps5-gpu-ci/data/runs/pr-15691/3109f4f6daa4-5b1d381d5ad5/ctest.xml: testcase 'public_headers_compile_18' is notrun and was not run
Reproduce
git fetch origin 3109f4f6daa4d806fd6ba0143bd51063776ef66f 5b1d381d5ad5d9f923326b18479c3bd889cdd588
git checkout 5b1d381d5ad5d9f923326b18479c3bd889cdd588 && git merge --no-ff 3109f4f6daa4d806fd6ba0143bd51063776ef66f
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Hi GiannisG1,

We got this one wrong: the footer is your own text outside the template and nothing in AGENTS.md asks for it to go, so please keep it; we withdraw that request.

Thanks.

@GiannisG1

Copy link
Copy Markdown
Contributor Author

@oneandonlydean thanks for letting me know. I've put the footer back on this and my other open pull requests.

@github-actions github-actions Bot added review queue Next in the maintainer review queue: checks pass, no open change request, no conflict and removed review queue Next in the maintainer review queue: checks pass, no open change request, no conflict labels Nov 4, 2026

@ahmadmuj ahmadmuj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked head 3109f4f6daa4d806fd6ba0143bd51063776ef66f; PR base 5b1d381d5ad5d9f923326b18479c3bd889cdd588. Control: parent 04a9eb4fa2877eb6e40bf2c83adf972e40d89b24.

Linux builds and both guest_main_stack tests pass. I also ran a separate host libgcc probe through the actual EnterGuestMainStack call, in bounded initial/adopted child processes. dladdr confirms all three unwind APIs come from libgcc_s.so.1 (14.2).

Both head cases preserve the arguments and visit three nested guest frames, then Aps5CallOnGuestMainStack, then zero IP. They return _URC_END_OF_STACK; every CFA stays inside the guest stack. Both parent controls reach EnterGuestMainStack, then die with SIGSEGV.

The emitted FDE covers [0xbf8f0, 0xbf903), including return PC 0xbf901. Its CIE marks RIP undefined, consistent with GCC's outermost-frame rule and glibc startup code. Conventions pass.

No findings in this Linux boundary change. No title, console, or Windows/macOS runtime claim.

@github-actions github-actions Bot added review queue Next in the maintainer review queue: checks pass, no open change request, no conflict and removed review queue Next in the maintainer review queue: checks pass, no open change request, no conflict labels Nov 6, 2026
@milescoviello

Copy link
Copy Markdown
Contributor

Title data point: Call of Duty: Black Ops II (PPSA34502) on Linux, Ubuntu 26.04, RTX 4070 Ti, NVIDIA 615.71.09, Release libraries. Both runs were driven by hand from a fresh save.

  • main ab213704ee: choosing ZOMBIES from the main menu makes the game switch to the Zombies executable (sceSystemServiceLoadExec). The run ends there with a SIGSEGV (EXIT-139, 177 s in). I opened the cores of two earlier crashes at this point, from a build with an unrelated change. Both stop in LibcUnwind::Step past the top of the guest main stack, after the loader throws on the unconverted executable. That is the path this PR changes.
  • ab213704ee with this PR merged (828521c308): the same selection doesn't crash, and the run continues in codzm.elf. It shows the common front end again (intro, "Press options button", main menu). Choosing ZOMBIES there opens the BLACK OPS II ZOMBIES menu (PLAY ONLINE / LAN PARTY / LOCAL / OPTIONS / STORE) at 60 fps. I ended the run there at 411 s, with no fault.

AI-assisted: yes (Claude Code)

@GiannisG1

Copy link
Copy Markdown
Contributor Author

@milescoviello thanks for the Black Ops II runs; I added PPSA34502 to the description.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4563 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 5021 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 7149 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 7804 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 8592 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 9938 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12544 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12642 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12682 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12688 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12700 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 12965 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 15803 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 15804 core/libs/prx/libkernel/Pthread/src/Thread.cpp
PR 15807 core/libs/prx/libkernel/Pthread/src/Thread.cpp

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

libkernel review queue Next in the maintainer review queue: checks pass, no open change request, no conflict unblocks title Names a title ID on the Unblocks line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants