Skip to content

fix(libSceVideoOut): refuse to unregister the buffer set on screen as the console does - #11196

Merged
adriadam10 merged 2 commits into
boykopovar:mainfrom
p3n1cillin:claude/videoout-unregister-busy
Oct 30, 2026
Merged

adriadam10 merged 2 commits into
boykopovar:mainfrom
p3n1cillin:claude/videoout-unregister-busy

Conversation

@p3n1cillin

@p3n1cillin p3n1cillin commented Oct 25, 2026 •

Copy link
Copy Markdown

What

sceVideoOutUnregisterBuffers released the buffer set holding the buffer on screen and returned 0. Two catalog titles measured that the console refuses this:

  • EVO Player (PPSA99039) records from a 2026-09-28 run that "the set on screen can't be unregistered (0x80290009 RESOURCE_BUSY)". Registering again then fails with 0x80290010 (SLOT_OCCUPIED), so the set stays registered.
  • Prospero Win (PPSA99995) gets the same code on firmware 12.02 when it closes after its flips have finished, and sceVideoOutClose then succeeds.

The change:

  • On-screen set: on a main-bus port (bus type 0), when the set holds flipStatus.currentBuffer, the call now returns RESOURCE_BUSY (0x80290009) and keeps the set and its buffers registered. Closing the port releases them, as before. Both titles open bus type 0 (sceVideoOutOpen(0xff, 0, 0, NULL) in EVO Player and Prospero Win). The overlay bus is unmeasured and still releases its on-screen set, which video_out_flip_overlay relies on.
  • Unchanged: a set that is not on screen still unregisters, and one with a pending flip still throws.
  • Debt: a new libSceVideoOut entry cites both measurements and records what isn't measured: the overlay bus, a set with a pending flip that is not yet on screen, and the answer after a blank flip.

video_out_flip_cpuflip and video_out_flip_cpuflip_in_order now check the refusal after their flips of buffer 0 have been shown. They also check that the set is still registered.

Tested

Current head 2188fef0aa on main 31646b6e17 (Windows 11 Pro 26300, WinLibs GCC 15.2.0, Release, full builds, NVIDIA GeForce RTX 5070):

  • All 36 tests libSceVideoOut registers pass on this head and on main, with the Windows driver on a separate desktop and VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1 and VK_LOADER_DEBUG=layer. In each log the loader inserts the layer (23 Insert instance layer lines), and neither has was not found but was requested or any VUID- error. guest_video_out_wqhd_detection_no_vulkan skips by design.
  • Before the main-bus commit, video_out_flip_overlay failed on this rebase ("an unregistered overlay buffer was composited"). That test, added to main on 2026-10-24, unregisters the overlay set while it is on screen.
  • tools/check_conventions.py --base origin/main passes.

Earlier runs, Windows 11 Pro (26300), WinLibs GCC 15.2.0, Release, NVIDIA, full build on main 1f68b1f:

  • libSceVideoOut tests: all 33 that libSceVideoOut registers pass. I ran them with ANYPS5_VIDEO_OUT_TEST_DRIVER=windows on a separate desktop, so the flip tests run instead of skipping on the offscreen driver; guest_video_out_wqhd_detection_no_vulkan skips by design.
  • Without the Buffer.cpp change: video_out_flip_cpuflip fails with "the set on screen was unregistered".
  • Validation layer: the same 33 tests were run with ctest -V and VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1, on main 1f68b1f and on this change, on the same machine and GPU (Vulkan SDK 1.4.363.0).
    • With VK_LOADER_DEBUG=layer the loader inserts the layer.
    • Neither log reports was not found but was requested by env var VK_INSTANCE_LAYERS, and neither has any VUID- error.
    • All 33 pass on both.
    • One run of this change had guest_video_out_high_frame_rate fail its timing check ("high-frame-rate mode did not double vblank frequency") while the machine was busy. That test doesn't unregister buffers. Three more runs each on main and the change pass, and so does a second full run.
  • Checks: tools/check_conventions.py --base origin/main passes. tools/pr_overlap.py finds no overlap; of the open PRs it couldn't compare, feat: run PS4 games on the AGC driver — Limbo (CUSA01664) in game (integration branch) #6977 and feat(libSceGnmDriver): build PS4 command buffers and submit them #9432 touch Buffer.cpp but not this function.
  • No title run: the catalog titles that hit this case only do so when they shut down.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: NVIDIA GeForce RTX 5070, driver 621.01, Windows 11 Pro 26300 (flip tests on the Windows SDL driver, with and without the validation layer)
  • AI-assisted: yes (Claude Code)

🤖 Generated with Claude Code

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The two independent catalog observations and the regression check give useful evidence for refusing the on-screen set, and I found no contradiction in the checked path. However, this is a VideoOut change, and CONTRIBUTING.md requires the focused tests on both main and the change with VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1, plus a comparison for newly introduced VUID- errors. The report describes the Windows flip-test run and a failing test without the code change, but not the required validation-layer comparison. Please add those same-machine/GPU results before approval; hosted product builds are skipped.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 4040 docs/dev/debt/libSceVideoOut.md
PR 5021 core/libs/prx/libSceVideoOut/tests/Flip.cpp
PR 6977 core/libs/prx/libSceVideoOut/src/Buffer.cpp
core/libs/prx/libSceVideoOut/tests/Flip.cpp
docs/dev/debt/libSceVideoOut.md
PR 9154 docs/dev/debt/libSceVideoOut.md
PR 9432 core/libs/prx/libSceVideoOut/src/Buffer.cpp
core/libs/prx/libSceVideoOut/tests/Flip.cpp
docs/dev/debt/libSceVideoOut.md
PR 9880 docs/dev/debt/libSceVideoOut.md
PR 9891 core/libs/prx/libSceVideoOut/tests/Flip.cpp
PR 9893 core/libs/prx/libSceVideoOut/tests/Flip.cpp
PR 9895 core/libs/prx/libSceVideoOut/tests/Flip.cpp
PR 10112 docs/dev/debt/libSceVideoOut.md
PR 11948 docs/dev/debt/libSceVideoOut.md
PR 12596 docs/dev/debt/libSceVideoOut.md
PR 12702 core/libs/prx/libSceVideoOut/tests/Flip.cpp
PR 12863 core/libs/prx/libSceVideoOut/tests/Flip.cpp
docs/dev/debt/libSceVideoOut.md

@p3n1cillin

Copy link
Copy Markdown
Author

Thanks. I've added the validation-layer comparison to the description:

  • Setup: the 33 libSceVideoOut tests with ctest -V, VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1, on main 1f68b1f and on this head. Same machine, NVIDIA GeForce RTX 5070, driver 621.01, Windows 11 Pro 26300.
  • Layer: it is loaded (VK_LOADER_DEBUG=layer shows it inserted). Neither log has the was not found message or any VUID- error.
  • Results: all 33 pass on both. One run of this head had a timing failure in guest_video_out_high_frame_rate while the machine was loaded. That test doesn't unregister buffers, and it passed three more times on each tree.

@p3n1cillin

Copy link
Copy Markdown
Author

@MarcZX10 could you take another look when you have time? The validation-layer comparison you asked for is in the description and in this comment: all 33 libSceVideoOut tests pass with VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1 and ANYPS5_REQUIRE_VULKAN=1 on main and on this head, with no VUID- errors (RTX 5070, driver 621.01).

@adriadam10 adriadam10 self-assigned this Oct 27, 2026

@adriadam10 adriadam10 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked against the current head: the validation-layer comparison @MarcZX10 asked for is in the description (33 libSceVideoOut tests on main and on this head with VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1, no VUID- errors, same machine and GPU), and the code, the Flip.cpp checks and the debt entry are fine. Nothing is left for the author.

@MarcZX10 the only thing holding the merge is your CHANGES_REQUESTED on this head; could you re-review or dismiss it?

@p3n1cillin
p3n1cillin force-pushed the claude/videoout-unregister-busy branch from 511b2b6 to 2188fef Compare October 28, 2026 14:16
@p3n1cillin

Copy link
Copy Markdown
Author

Rebased onto main 31646b6e17; the head is now 2188fef0aa. Rerunning the flip tests on the rebase found a conflict with a test main added on 2026-10-24, so this push has one new commit.

  • What broke: video_out_flip_overlay (from the overlay compositing change, ded043d552) flips an overlay-bus port to buffer 0 and then unregisters that set while it is on screen, expecting it to be released. This PR refused it with RESOURCE_BUSY, so the overlay stayed composited and the test failed with "an unregistered overlay buffer was composited".
  • The fix (2188fef0aa): the refusal now applies to main-bus ports (bus type 0) only. Both measurements are on bus type 0: EVO Player and Prospero Win each open with sceVideoOutOpen(0xff, 0, 0, NULL). Nothing has measured the overlay bus, so it keeps its behaviour from main. The debt entry now lists the overlay bus as unmeasured.

Rerun on 2188fef0aa and on main 31646b6e17, on the same machine and GPU (details in the updated description):

  • All 36 libSceVideoOut tests pass on both, including video_out_flip_overlay, video_out_flip_cpuflip and video_out_flip_cpuflip_in_order.
  • They ran with the Khronos validation layer inserted by the loader, and neither log has any VUID- error.
  • Conventions pass.

@MarcZX10 @adriadam10 Both your reviews on this PR are still recorded as changes requested; adriadam10's from 2026-10-27 said nothing was left for the author. Could you re-review with the new commit, or dismiss them if it's fine?

@adriadam10
adriadam10 dismissed MarcZX10’s stale review October 28, 2026 14:29

Dismissing a stale review: the requested changes were addressed after it (new head) and the merge queue blocks on it. Please request changes again if anything is still open.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 28, 2026

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — new head, changes still requested

The overlay-bus adjustment addresses the reported regression, but the rebase and validation evidence are stale against current main.

What I ran

  • Reviewed exact head 2188fef0aab9cebb9bbc2b61767d7f76cb07e4d6 against current main edf6804cf52c62eacafc592dd6142cb4efe5bd05.
  • Ran git merge-base, git merge-tree --write-tree, git diff --name-status, and git diff --check.
  • Checked the author’s rebase report and current hosted checks. I did not run the GPU tests locally.

Results

Check Result
PR merge base 31646b6e17e95e9124953d4008f0ce2e50ed153d
Current-main merge preview Conflicts in docs/dev/debt/libSceVideoOut.md
VideoOut paths changed on main since the reported test base Output.hpp, Output.cpp, Flip.cpp, and OutputModePolicy.cpp
PR paths Buffer.cpp, Flip.cpp, and the VideoOut debt file
Diff check Passed; no output
Hosted Windows / conventions / eligibility / labels / report Passed; Linux and specialized jobs skipped
Focused GPU validation Author reports 36 tests pass on main 31646b6e17 and head 2188fef0aa; that main is no longer current
Unedited command output
$ git merge-base upstream/main refs/review/upstream-11196
31646b6e17e95e9124953d4008f0ce2e50ed153d
$ git merge-tree --write-tree upstream/main refs/review/upstream-11196
8e74cb366dbd8ac5a73812ca24e09fe1f2238ebc
100644 d57e8243d19ae5d88698f18fd5ac247585de24fb 1 docs/dev/debt/libSceVideoOut.md
100644 527bc73b1251c6912063838d141437a0944bac80 2 docs/dev/debt/libSceVideoOut.md
100644 a0c42367343794db17238837f2179c395754692d 3 docs/dev/debt/libSceVideoOut.md

Auto-merging core/libs/prx/libSceVideoOut/tests/Flip.cpp
Auto-merging docs/dev/debt/libSceVideoOut.md
CONFLICT (content): Merge conflict in docs/dev/debt/libSceVideoOut.md

$ git diff --name-status 31646b6e17e95e9124953d4008f0ce2e50ed153d..upstream/main -- core/libs/prx/libSceVideoOut
M core/libs/prx/libSceVideoOut/include/Output.hpp
M core/libs/prx/libSceVideoOut/src/Output.cpp
M core/libs/prx/libSceVideoOut/tests/Flip.cpp
M core/libs/prx/libSceVideoOut/tests/OutputModePolicy.cpp
$ git diff --name-status upstream/main...refs/review/upstream-11196
M core/libs/prx/libSceVideoOut/src/Buffer.cpp
M core/libs/prx/libSceVideoOut/tests/Flip.cpp
M docs/dev/debt/libSceVideoOut.md
$ git diff --check upstream/main...refs/review/upstream-11196

Negative control

Reported control Outcome
Before the bus-type guard, video_out_flip_overlay on the rebased tree Author reports failure because the overlay buffer stayed composited; not independently reproduced here

Findings

Blocking

  • Please rebase onto current main, resolve the debt-file conflict, then rerun the focused VideoOut tests on both that exact main and the resulting PR head with the Vulkan validation settings in the report. The current main changed VideoOut implementation and test paths since 31646b6e17, so the posted 36-test comparison does not validate the current integration.

Non-blocking

  • The PR correctly limits the measured RESOURCE_BUSY behavior to the main bus and records the overlay bus as unmeasured.

Sources

Not verified

  • Current-main behavior after the VideoOut changes since 31646b6e17, and the author-reported test outcomes were not independently reproduced.

To get this to a good state

  • Rebase and resolve the debt overlap, then publish the same-scene validation-layer comparison with exact current-main and PR-head commits and all focused test results.

@p3n1cillin
p3n1cillin force-pushed the claude/videoout-unregister-busy branch from 2188fef to 040acf7 Compare October 29, 2026 08:47
@p3n1cillin

Copy link
Copy Markdown
Author

@MarcZX10 Rebased onto current main 4c7d275bef; the head is now 040acf798e. The libSceVideoOut.md conflict was main's new sceVideoOutIsOutputSupported entry from #11405, next to this PR's sceVideoOutUnregisterBuffers entry. Both are kept, and the diff against main adds only this PR's entry. The code and Flip.cpp changes are unchanged and applied cleanly over main's Output and Flip.cpp changes.

Rerun on that head and on main 4c7d275bef, on the same machine and GPU (Windows 11 Pro 26300, Ryzen 5 7600X, WinLibs GCC 15.2.0, Release, full build), NVIDIA GeForce RTX 5070:

  • 36 libSceVideoOut tests: run with ANYPS5_VIDEO_OUT_TEST_DRIVER=windows on a separate desktop. Settings: VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1 and VK_LOADER_DEBUG=layer.
  • Both runs: all 36 pass, including video_out_flip_overlay, video_out_flip_cpuflip and video_out_flip_cpuflip_in_order; guest_video_out_wqhd_detection_no_vulkan skips by design. The loader logs 23 Insert instance layer "VK_LAYER_KHRONOS_validation" lines, and neither log has was not found but was requested or any VUID- error.
  • Offscreen run of the same 36 on the head: all pass.
  • Conventions: tools/check_conventions.py --base origin/main passes.

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 29, 2026

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-check — changes requested\n\nThe new head addresses the overlay-bus regression and the reported 36-test validation-layer run on main/head 4c7d275 is useful. Current upstream main is now aaccd0b, and this head is 44 commits behind it; main has also changed the same VideoOut Flip.cpp test file since the reported base. Please merge current main, preserve the main-bus refusal and overlay regression coverage, resolve the overlapping test hunk, and rerun the focused VideoOut tests on current main and the exact updated head with validation enabled. I did not run the GPU tests locally.

p3nicillin and others added 2 commits October 29, 2026 12:18
The console answers RESOURCE_BUSY (0x80290009) when the set holding
the buffer on screen is unregistered, and the set stays registered:
EVO Player measured it with a re-registration then failing
SLOT_OCCUPIED, and Prospero Win on firmware 12.02 after its flips went
idle. The set was released and 0 returned instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…only

EVO Player and Prospero Win measured RESOURCE_BUSY on ports opened on
bus type 0. The overlay bus is unmeasured, so its on-screen set is
still released as before, which video_out_flip_overlay relies on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@p3n1cillin
p3n1cillin force-pushed the claude/videoout-unregister-busy branch from 040acf7 to ecadb60 Compare October 29, 2026 12:49
@p3n1cillin

Copy link
Copy Markdown
Author

@MarcZX10 Rebased onto current main 5454cd2e45; the head is now ecadb6068a. main's changes to tests/Flip.cpp since the last base (#11853's flip-master codes and the PS4 alignment message) don't touch testCpuFlipOrder, so the rebase applied without a conflict. The diff against main is still the two-line Buffer.cpp change, the debt line, and the testCpuFlipOrder check. Both pieces you asked about are kept:

  • Main-bus refusal: still checked in testCpuFlipOrder, which expects 0x80290009 and that the set stays registered.
  • Overlay bus: still covered by video_out_flip_overlay, which depends on the overlay's on-screen set being released.

Rerun on that exact head and on main 5454cd2e45 (Windows 11 Pro 26300, Ryzen 5 7600X, WinLibs GCC 15.2.0, Release, full build, NVIDIA GeForce RTX 5070):

  • Setup: the 36 libSceVideoOut tests ran with ANYPS5_VIDEO_OUT_TEST_DRIVER=windows on a separate desktop, using VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation, ANYPS5_NO_SHADER_CACHE=1, ANYPS5_REQUIRE_VULKAN=1 and VK_LOADER_DEBUG=layer.
  • Validation runs, head and main: all 36 pass, including video_out_flip_overlay, video_out_flip_cpuflip and video_out_flip_cpuflip_in_order; guest_video_out_wqhd_detection_no_vulkan skips by design. Each loader log has 23 Insert instance layer "VK_LAYER_KHRONOS_validation" lines and no was not found but was requested or VUID- lines.
  • Offscreen run on the head: the same 36 tests all pass.
  • Conventions: tools/check_conventions.py --base origin/main passes.

@adriadam10
adriadam10 merged commit 7d40066 into boykopovar:main Oct 30, 2026
10 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Re-checked on ecadb6068a: the diff is the same as the one reviewed (the Buffer.cpp main-bus guard, the testCpuFlipOrder checks that fail without it, the debt entry), rebased onto current main with the VideoOut tests rerun with the validation layer as asked above; merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants