Skip to content

fix(agc): keep shader preparation temporaries off guest stacks - #2456

Closed
Mouhand-Kaddo wants to merge 1 commit into
boykopovar:mainfrom
Mouhand-Kaddo:fix/agc-shader-guest-stack
Closed

Mouhand-Kaddo wants to merge 1 commit into
boykopovar:mainfrom
Mouhand-Kaddo:fix/agc-shader-guest-stack

Conversation

@Mouhand-Kaddo

@Mouhand-Kaddo Mouhand-Kaddo commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

What

Shader registration and ABI preparation run on caller-owned guest fibers. QueueState embeds 48 KiB of constant RAM; local instances and copies made RegisterShader reserve about 99 KiB and PrepareRegistered about 51 KiB, even on paths that never used those locals. A 64 KiB guest fiber could therefore overwrite its neighbor.

Keep the five shader-preparation temporaries on the heap, retaining their initialization and copy behavior:

- QueueState state{}; / auto stageState = state;
+ auto storage = std::make_unique<QueueState>(...);
+ auto& state = *storage;

This changes storage duration/location only. No shader semantics, guest fiber sizes, title checks, or fallback behavior change.

Evidence

Tested

  • Before: agc_shader_preparation_stack fails on Windows and Arch with preparation wrote below its fiber context. The valid-registration case separately reproduces the remaining PrepareRegistered overflow after moving only the other four locals.
    After: both tests pass on both platforms. They use 32 KiB fiber contexts with a sentinel region below them, exercise existing input validation and successful synthetic compute registration/ABI preparation, and check that the sentinel survives. Each completes well under one second locally; the valid-registration test skips without Vulkan.
  • Built the full project and patched libs on main 19a0daa7 plus this commit. Windows 11, WinLibs GCC 15.2, RX 6900 XT: 527 tests, 523 passed, 1 skipped, 3 failed (36.82 s). Arch WSL2, GCC 16.2, lavapipe: 532 tests, 504 passed, 27 skipped, 1 timeout (41.82 s). Windows failures are agc_driver_pixel_interlock, agc_driver_sdwa_float_selectors, and agc_driver_sdwa_ldexp_f16; Arch times out in agc_driver_flat_store. All four were also reproduced after rebuilding the affected targets on unchanged main da122c90.
  • Conventions and test-registration checks pass.
  • A local guest startup probe confirmed the overflowing registration frame inside a 64 KiB fiber. The updated driver continues beyond that point; additional startup issues remain outside this PR. The probe uses a separate integration runtime with other startup changes, none of which are included here.

Overlap check reports a textual conflict with #1902, which moves null-pixel registration to another helper. It does not fix stack storage; if it lands first, retain this heap allocation in its new helper. No dependency on its behavior.

Merge Danger

Door: two-way

Blast Radius: preparation

Shader-preparation calls incur heap allocation and can propagate allocation failure. Existing registration publication, validation and transaction behavior remain intact.

Checklist

  • Based on main 19a0daa7; open PRs checked for the same stack fix (textual overlap with fix(agc): prepare the null pixel program at draw when its variant was not prepared #1902 described above)
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths retain their exceptions; no silent stubs added
  • No notes, investigation files or images added to the repository
  • No new third-party code
  • Generic behavior, not specific to one title
  • Depends on: none
  • AI-assisted: yes

@github-actions github-actions Bot added the intake paused Closed while PR intake is limited to authors with at least 10 PRs merged into main label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Thank you for your interest in the project! We are sorry, but we cannot accept more PRs while we work through the existing backlog. This PR has been automatically closed and labeled "intake paused". This is not a judgment of the quality of your changes.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
#1902 core/libs/prx/libSceAgcDriver/Execution/src/Driver/Shaders/ShaderRegistry.cpp L733-749

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

intake paused Closed while PR intake is limited to authors with at least 10 PRs merged into main

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant