[MarcZX10]

What

Run the Linux test suite with SDL on X11 under Xvfb so Vulkan-backed VideoOut tests can create a display surface in CI. Configure the Linux build for the X11 VideoOut test driver, use Mesa llvmpipe, and make Linux skip validation fail when a test skips only because no display is available. This addresses the coverage gap reported in #6899.

Tested

• Current rebased head acb8190919, based on current main 587bb343: python3 -I tools/tests/test_check_skips.py — 23 tests passed. python3 tools/check_conventions.py --base upstream/main --head HEAD and git diff --check upstream/main...HEAD pass.
• The Linux/Xvfb runtime is not available on this host: the existing video_out_flip_tests binary aborts at the CPU feature guard before assertions (Xeon E5-1620 v2 lacks LZCNT, BMI1/2, MOVBE, AVX2 and FMA). The repository's Linux runner is self-hosted; the PR Build event skips it, and workflow dispatch requires repository admin access that I do not have. No runtime pass is claimed on acb8190919.
• Historical supported-host report on prior PR head 7f60c8a72, merged into main f3bb6a4f8: under Xvfb with llvmpipe, Marco-Scheffler reported 1,631/1,631 tests and check_skips.py --job linux accepted 54 expected skips. On the RTX 4000 Ada run, the checker accepted both earlier and new JUnit files; the full suite was 1,630/1,631 due to an unrelated linux_gnu_segments parallel timeout, which passed 10/10 when run alone. This evidence predates the current rebase and is not presented as validation of acb8190919.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw or are listed in technical debt
• No notes or investigation files added to the repository
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: fix(agc): present EOP flips when draws are pipelined #6616
• Unblocks: none
• GPUs tested: Mesa llvmpipe (software Vulkan), Debian 12; tests aborted at CPU feature guard before GPU assertions
• AI-assisted: yes

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 6044	.github/workflows/build.yml
PR 7332	.github/workflows/build.yml

[MarcZX10]

I manually compared both .github/workflows/build.yml diffs against current main. #6044 reorganizes test registration/selection and replaces the Linux CTest command; #6932 adds X11/Xvfb setup and the X11 VideoOut test driver to the Linux job. These changes do not implement the same behavior, but they will need integration if #6044 merges first: I will rebase #6932 and apply the Xvfb wrapper to the resulting Linux CTest command while preserving the X11 configuration. No code overlap beyond the shared workflow file.

[marco-scheffler]

I ran this PR's Linux steps on a local host: Ubuntu 24.04.4, CMake/CTest 3.28.3, Mesa 25.2.8, base main bc3c0fc13, this PR at 3ef77906f. Environment as in the workflow: SDL_VIDEODRIVER=x11, ANYPS5_GPU=llvmpipe, ANYPS5_REQUIRE_VULKAN=1, GALLIUM_OVERRIDE_CPU_CAPS=avx, -DANYPS5_VIDEO_OUT_TEST_DRIVER=x11, full ctest under xvfb-run -a -s '-screen 0 1920x1080x24 -nolisten tcp', then this PR's check_skips.py --job linux.

Build	ctest	check_skips.py --job linux
main	1312/1315: video_out_flip_eop, _eop_main fail, _eop_aftergpu times out	fails
main + #6616	1314/1315: agc_async_log_fatal timed out under --parallel 24, passes 5/5 serial in 6.6 s	fails

Two things would turn the Linux job red after merge:

1. guest_video_out_high_frame_rate is reported as an unexpected skip. Under Xvfb it prints skipped, current display does not provide 120 Hz, but CTest also writes <skipped message="SKIP_RETURN_CODE=77"/>. skips() turns that attribute into a second skipped, SKIP_RETURN_CODE=77 reason, and high_refresh_limitation() requires every reason to match, so it returns false. main's check_skips.py fails the same way on this JUnit. CI didn't hit it so far because the test skipped with no display, which DEVICE_LIMITATION.search accepts. The fixtures in test_check_skips.py use <skipped/> or a reason as the message, never CTest's SKIP_RETURN_CODE= text. Ignoring a message that is only SKIP_RETURN_CODE=<n> would fix it.
2. Order: without fix(agc): present EOP flips when draws are pipelined #6616 the three EOP tests fail, so this PR needs Depends on: #6616.

The new tests in tools/tests pass. Side note for #6899: the NVIDIA proprietary driver (RTX 4000 SFF Ada, 595.99.02) presents on Xvfb, so all VideoOut tests also run on that GPU there, unlike ANV.

[MarcZX10]

Thanks for the exact JUnit report and main comparison. I fixed the SKIP_RETURN_CODE=77 metadata handling and added a regression for the CTest skip element plus the high-refresh message. On the rebased head, the checker suite passes 19/19, and a real local CTest fixture emitted that metadata and was accepted by the Linux checker with the 120 Hz reason. The fresh full CMake configure could not be repeated here because the worktree is missing 3rdparty/zstd/build/cmake, so I have not claimed a post-rebase VideoOut runtime pass. I rebased onto current main and added Depends on: #6616, which remains open.

[marco-scheffler]

Re-ran the updated head on the same host: main 84b0d2f24 + #6616 75f5beffa + this PR 962c9330c, full ctest under xvfb-run with the workflow's environment (ANYPS5_GPU=llvmpipe, -DANYPS5_VIDEO_OUT_TEST_DRIVER=x11), then check_skips.py --job linux.

• ctest: 1368/1368, all 22 VideoOut tests pass on llvmpipe.
• check_skips.py --job linux still exits 1 with error: guest_video_out_high_frame_rate skipped. Same on the RTX 4000 SFF Ada.

The SKIP_RETURN_CODE=77 filter works. The remaining cause is one I missed in my first report: the real test prints Vulkan log lines to stdout before its skip line, and high_refresh_limitation() treats every non-empty line as a reason:

<skipped message="SKIP_RETURN_CODE=77"/>
<system-out>Surface capabilities drawable=1152x648 min=1152x648 max=1152x648 minImages=3 maxImages=0 usage=0x8009f
[src/VulkanDevice.cpp:1972 VulkanDevice] Swapchain created swapchain=0x... extent=1152x648 images=3
[shader-disk-cache] .../build/tests/shader_cache/4bc6b6d4632b7637 (source version 4bc6b6d4632b7637, format 36)
skipped, current display does not provide 120 Hz
</system-out>

Counting only skipped, lines as reasons makes both real reports pass, but it breaks the 18 cases of test_high_refresh_reason_cannot_hide_additional_diagnostics (74e239408), which reject any extra output on purpose. So either the checker needs to tell log lines apart from failure detail, or the test must not print anything before its skip. I'll leave that choice to you. The new regression fixture would catch it if it carried the log lines above.

[MarcZX10]

Thanks for catching this with the full Xvfb rerun. I updated high_refresh_limitation() on head 0337bac to allow only the three observed Vulkan startup log formats (surface capabilities, swapchain creation, and shader-disk-cache), while still requiring the expected 120 Hz limitation and rejecting any other output. Added a fixture with those logs and kept the arbitrary-extra-diagnostic controls. python3 -I tools/tests/test_check_skips.py passes all 20 tests; conventions and diff checks pass. The Linux workflow’s updated checks are running now.

[MarcZX10]

CI has now completed on 0337bac71: eligibility, conventions, relinker on Ubuntu, Windows, macOS arm64 and macOS x86-64 under Rosetta, plus relinker ASan all pass. The broad Linux/Windows builds and register-context/write-watch jobs were skipped by path selection. The 20 local checker tests still pass; this run does not exercise the Linux video-out test suite itself.

[marco-scheffler]

Re-ran the Linux job on 0337bac71 with #6616, on main cb1d6e5bd, same environment as before (xvfb-run, x11, ANYPS5_GPU=llvmpipe, ANYPS5_REQUIRE_VULKAN=1, GALLIUM_OVERRIDE_CPU_CAPS=avx). ctest passes 1443/1443 with all VideoOut tests running, and tools/tests pass.

check_skips.py --job linux still fails on guest_video_out_high_frame_rate. The SKIP_RETURN_CODE=77 message is dropped now, but high_refresh_limitation() returns false for any other line, and the real test prints about 20 start-up lines besides the three in HIGH_REFRESH_STARTUP_LOGS, for example:

[src/HostCpu.cpp:193 operator()] host CPU Intel(R) Core(TM) i9-14900: has SSE, ...
[src/VulkanDevice.cpp:1016 VulkanDevice] Vulkan instance created instance=0x...
[src/VulkanDevice.cpp:1060 VulkanDevice] Physical device candidate=0x... name=llvmpipe (LLVM 20.1.2, 256 bits) ...
[write-watch] host imports resolve write protection: ...
[pipeline-cache] loaded 0.0 KiB from .../shader_cache/vk-pipelines-...

These lines depend on the host and driver, so an allow-list will keep missing some. An alternative would be to look only at the lines that start with skipped, and require all of them to be the 120 Hz reason, with the unexpected-diagnostics cases checked on those lines as well.

[marco-scheffler]

Sorry, my excerpt above was misleading: it was only the tail of system-out. The test prints 22 log lines before its skip line on llvmpipe (29 on the RTX), and my excerpt also cut the [src/VulkanDevice.cpp:1930 VulkanDevice] prefix off the surface capabilities line. So 0337bac71 still rejects both of my real reports: check_skips.py --job linux exits 1 on the same JUnit files, with error: guest_video_out_high_frame_rate skipped.

Full system-out of guest_video_out_high_frame_rate, llvmpipe under Xvfb (paths shortened)

[src/HostCpu.cpp:193 operator()] host CPU Intel(R) Core(TM) i9-14900: has SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, LZCNT, BMI1, BMI2, MOVBE, CMPXCHG16B, LAHF/SAHF, AVX, AVX2, FMA, F16C, AES, PCLMULQDQ, RDRAND, RDSEED, ADX, SHA, CLFLUSHOPT, CLWB, RDPID, XSAVEOPT, XSAVEC, PREFETCHW, OS AVX state (XCR0); lacks SSE4A, CLZERO
[src/VulkanDevice.cpp:966 VulkanDevice] VulkanDevice constructor window=0x7405acff7510
[src/VulkanDevice.cpp:975 VulkanDevice] Vulkan loader loaded library=0x7405a8002440
[src/VulkanDevice.cpp:991 VulkanDevice] Presentation window context=0x7405a83ac6f0 extent=1920x1080 extensions=2
[src/VulkanDevice.cpp:1013 VulkanDevice] Vulkan instance created instance=0x7405a83a5d00
[src/VulkanDevice.cpp:1018 VulkanDevice] Vulkan surface created surface=0x7405a83efc50 window=0x7405a83ac6f0
[src/VulkanDevice.cpp:1024 VulkanDevice] Physical device count=2
[src/VulkanDevice.cpp:1057 VulkanDevice] Physical device candidate=0x7405a83b8310 name=NVIDIA RTX 4000 SFF Ada Generation api=0x404149 type=2
[src/VulkanDevice.cpp:1057 VulkanDevice] Physical device candidate=0x7405a83b8290 name=llvmpipe (LLVM 20.1.2, 256 bits) api=0x40413e type=4
[src/VulkanDevice.cpp:1104 VulkanDevice] Physical device selected physical=0x7405a83b8290 queueFamily=0
[src/VulkanDevice.cpp:1117 VulkanDevice] Selected GPU name=llvmpipe (LLVM 20.1.2, 256 bits) vendor=0x10005 device=0x0 subgroup=8
[src/VulkanDevice.cpp:1119 VulkanDevice] Memory heap 0 size=128496.2 MiB flags=0x1
[src/VulkanDevice.cpp:1120 VulkanDevice] Memory type 0 heap=0 flags=0xf
[src/VulkanDevice.cpp:1189 VulkanDevice] Shader clock driver=13 narrowSubgroupClock=0
[src/VulkanDevice.cpp:1524 VulkanDevice] Vulkan features tessellationAvailable=1 mesh=1 depthClip=1 depthRangeUnrestricted=1 pipelineLibrary=1
[src/VulkanDevice.cpp:1886 VulkanDevice] Vulkan device ready device=0x7405a841fc20 queue=0x7405a84212d0 family=0
[write-watch] host imports resolve write protection: yes (0 of 4 scratch pages written after a GPU read, 4 after a GPU write, 0 after the import); imported ranges are compared
[write-watch] dma-buf imports keep write protection: yes (0 of 4 scratch pages written after a GPU read, 0 after a GPU write, 0 after the import, 1 seen after a CPU store); imported ranges stay watched
[pipeline-cache] loaded 0.0 KiB from .../build/tests/shader_cache/vk-pipelines-10005-0000-06402008-32352e322e382d307562756e7475302e.bin
[src/VulkanDevice.cpp:1930 VulkanDevice] Surface capabilities drawable=1152x648 min=1152x648 max=1152x648 minImages=3 maxImages=0 usage=0x8009f
[src/VulkanDevice.cpp:1972 VulkanDevice] Swapchain created swapchain=0x7405aa0e6c90 extent=1152x648 images=3
[shader-disk-cache] .../build/tests/shader_cache/4bc6b6d4632b7637 (source version 4bc6b6d4632b7637, format 36)
skipped, current display does not provide 120 Hz

The RTX run has a different set: more memory heap and type lines, a subgroup line, a [pipeline-journal] line and one [write-watch] line instead of two, and the GPU name, CPU flags, heap sizes and cache paths differ per host. So a fixed list of formats will likely keep breaking. Two options I can see: accept lines in the logger's [tag] message form alongside the 120 Hz reason (the unexpected failure detail cases in test_check_skips.py would still fail), or have the test print nothing but its skip line. Happy to re-run either on the same host.

[marco-scheffler]

To untangle my last three comments: they all come from the same host and the same JUnit output. The full picture is the 22 start-up lines in the last one; the three lines in my first comment were only its tail, which is why the allow-list in 0337bac71 still misses.

They also end with two different suggestions, sorry for that. Either works on the real output; the difference is what happens to the test_high_refresh_reason_cannot_hide_additional_diagnostics cases:

• Looking only at lines that start with skipped, passes both real reports, but extra detail on a line without that prefix would no longer reject the skip, so those cases need to change.
• Accepting lines in the logger's [tag] message form keeps them as they are, as long as the extra detail isn't printed in that form.

Your call. I'll re-run whichever you pick on this host with llvmpipe and the RTX.

[MarcZX10]

Thanks for the full llvmpipe and RTX logs. I pushed 7f60c8a72: the checker now validates each line marked skipped, as the precise 120 Hz limitation and ignores the variable startup diagnostics in system-out/system-err. The JUnit skip text is split into individual reasons, so extra text there still fails. I updated the tests to cover host-specific startup logs and additional skip reasons. The 20 checker tests pass and conventions pass; please rerun the new head on your two hosts.

[marco-scheffler]

Reran 7f60c8a72 on both hosts: Ubuntu 24.04.4, RTX 4000 SFF Ada Generation (595.99.02), lavapipe (Mesa 25.2.8, LLVM 20.1.2) on the same host, GCC 13.3.0, Release, under Xvfb with ANYPS5_VIDEO_OUT_TEST_DRIVER=x11. This PR merged onto main f3bb6a4f8 (clean merge), full ctest --parallel 24.

Check	llvmpipe	RTX 4000 Ada
check_skips.py --job linux on the JUnit files from my earlier run (the ones 0337bac71 rejected)	rc 0, 45 skips, all expected	rc 0, 2 skips, all expected
Full suite, new run	1631/1631	1630/1631, see below
check_skips.py --job linux on the new JUnit files	rc 0, 54 skips, all expected	rc 0, 2 skips, all expected
guest_video_out_high_frame_rate	skipped (SKIP_RETURN_CODE=77), 24 lines of system-out, one of them skipped, current display does not provide 120 Hz	skipped (SKIP_RETURN_CODE=77), 30 lines of system-out, one of them skipped, current display does not provide 120 Hz

So the checker now accepts the real output on both hosts, including the variable startup lines around the skipped, line. The checker unit tests pass: 21 tests here, not 20.

The one RTX failure was linux_gnu_segments, which hit its 20 s timeout while all 24 parallel jobs were starting. It passes 10/10 on its own (about 4.3 s each), and it passed in the llvmpipe run (7.2 s). This PR doesn't change it, so it is unrelated.

[MarcZX10]

Thanks for the rerun on llvmpipe and the RTX 4000 Ada. The skip checker accepts both new JUnit sets, and the full suite reports 1631/1631 on llvmpipe and 1630/1631 on the RTX host, with the remaining video test skipped because the display does not provide 120 Hz. That makes the prior rejected-skip-file issue reproducible and fixed while preserving the expected display limitation.

[oneandonlydean]

Thanks, the checker changes look right; tools/tests/test_check_skips.py passes 23/23 when this branch is merged into current main.

Before merging we need one more data point, because the Linux job only runs on pushes to main and on dispatches (which use main's workflow file), so this branch's change can't run in CI before it lands. The reruns above were on main f3bb6a4, which is now several hundred commits behind. Could you (or @marco-scheffler) rerun the Linux suite under Xvfb with ANYPS5_VIDEO_OUT_TEST_DRIVER=x11 and llvmpipe on this branch merged into current main, and post the check_skips.py result? With "no display" no longer accepted on Linux, any display test that skips or fails there would turn the first main build red.

[MarcZX10]

I rebased the four original MarcZX100-authored commits onto current main 587bb343; the new head is acb8190919. The 23 skip-checker tests, conventions, and diff checks pass on this head. The Xvfb runtime remains unverified here: the local CPU guard stops the test binary before assertions, and GitHub skips the self-hosted Linux job on PR events. The prior llvmpipe/RTX 4000 runs were on 7f60c8a72 merged into f3bb6a4f8, so I updated the PR description to mark them historical. If an equipped reviewer can run the Xvfb Linux suite on the current head merged with 587bb343, that would resolve the remaining validation request.

[MarcZX10]

Update on current head acb8190919164e22a74bf0fe5aba7d7d6974b615 (based on main 587bb343e8baf0f9c62bdb04c9dab5eb4b938dc9): hosted relinker checks have now completed successfully on Ubuntu, Windows, macOS ARM, and macOS x86_64, including the ASan relinker job. Conventions, labels, and eligibility also pass. The Linux/Xvfb check remains skipped for PR events by repository policy, so this does not provide the requested current-head Xvfb runtime result; the earlier 23/23 local checker result is unchanged.