Repository navigation
Windows: agc_shader_helpers intermittently hangs at exit in Driver::stop() under ctest #6791
Description
Failure
On Windows, agc_shader_helpers intermittently never exits. The test finishes its work and prints shader helper preparation tests passed, then the process stays alive. The test has no TIMEOUT property, so one full ctest -j 8 run (with #6379 merged, see below) waited on it for 38 minutes before I stopped it.
Reproduce
Windows 11 Pro 26200, NVIDIA GeForce RTX 4080 (driver 617.42), AMD Ryzen 7 5800X, WinLibs GCC 15.2.0 r7, Ninja, Release, bundled dependencies, BUILD_TESTING=ON. Current main 72c5eed1f does not build on Windows here without #6452, so the tree is main + #6452 (4fe92c9c2), no other change.
ctest --test-dir build -R '^agc_shader_helpers$' --timeout 60| Launch | Runs | Hung |
|---|---|---|
ctest as above, one test per ctest process |
10 | 3 (hit the 60 s limit) |
ctest, three loops stopped at their first hang |
4, 7 and 2 | hung on the 4th, the 7th and the 2nd run |
agc_shader_helpers_tests.exe started directly (PATH prepended with build/core/libs/libs/unpatched, working directory the source root, output redirected to files) |
31 | 0 |
The same happened with #6379's head merged on top (4 of 5 runs), so it is not tied to that PR.
Where it waits
gdb -p on a hung process: the main thread is the only thread left, apart from the one gdb injects.
Thread 1:
#0 ntdll!ZwWaitForSingleObject ()
#1 KernelBase!WaitForSingleObjectEx ()
#2 ?? () from libwinpthread-1.dll
#3 ?? () from libwinpthread-1.dll
#4 ?? () from libwinpthread-1.dll
#5 AgcDriver::DriverDetail::Driver::stop() () from libSceAgcDriver.prx
#6 ?? ()
Frame 5 returns from call pthread_mutex_lock at Driver::stop()+3366, preceded by lea 0x8(%rax),%rcx. So Driver::stop() (Execution/src/Driver/Lifecycle.cpp:98) is blocked locking a mutex that no remaining thread can release. The stack does not unwind past frame 6, and the build has no debug info, so I could not tell which of the locks in stop() (or in a function inlined into it) this is, nor whether stop() runs from ~Driver() or from the shutdown hook registered with LibcRegisterShutdown_nid_postfix.
Not verified: that the owner was a thread ended during process exit while holding the lock. That is consistent with the test having finished and with no other thread being left, but it is not measured. Not checked on Linux.
AI-assisted: yes (runs and gdb capture done with Claude Code).
Activity
LeandroLP23 commentedon Oct 16, 2026
Reproduces on AMD as well, and #6641 alone does not clear it here.
Windows 11 (26300), Radeon 860M (RDNA 3.5,
1002:1114, driver 32.0.13058.8001), Ryzen AI 7 350, WinLibs GCC 15.2.0, Release,ctest -R '^agc_shader_helpers$' --timeout 40, one test perctestprocess:main82ddae75b+ #6436 (only changes the SPIR-V words passed tovkCreateShaderModule)main82ddae75b+ #6641 (43830b93b, merged cleanly)Every timed-out run had already printed
shader helper preparation tests passed.agc_driver_pm4_ring_invalid, which timed out once inmain's Windows CI after printingPM4 ring-invalid test passed, passed 15 of 15 here.Not tested: #6641 together with #6814/#6827, Linux.
AI-assisted: yes (Claude Code).
LeandroLP23 commentedon Oct 16, 2026
Since it reproduces in about one run of two here, I'm building
libSceAgcDriverwith debug info to name the lockDriver::stop()waits on. I'll post the result here.AI-assisted: yes (Claude Code).
Amoenus commentedon Oct 16, 2026
Additional upstream CI evidence for the related PM4 test mentioned above: main66 Windows job, commit
66e290f1c04a6aa08271f106d0dc83ba06561bfc, runs CTest with--parallel 4.agc_driver_pm4_ring_invalidprintsPM4 ring-invalid test passed, then CTest times it out at 30.01s. It is the sole failed test among1321 results.agc_driver_graphicspasses in7.89s, so this is separate from the now-resolved #6792 fixture failure.This is inspection of the public CI log, not a new local reproduction. The success message followed by a timeout matches the symptom already reported here; no stack or lock attribution was captured in this CI run. AI-assisted: Codex.
Later CI comparison: main93 Windows job, commit
93e54cedd, also runs CTest with--parallel 4.agc_shader_helperspasses in0.04s andagc_driver_pm4_ring_invalidpasses in0.03s; the job reports zero failures among1321 results, with skipped tests excluded. This successful run is compatible with an intermittent exit hang; it does not establish a fix. No new local reproduction, stack or lock attribution.LeandroLP23 commentedon Oct 16, 2026
Named it: the main thread waits on locks that
ExitProcessleft held by the threads it ended, and there are three of them in a row. Onmainit's the async log's queue mutex (AsyncLog::Flush, inlined intoDriver::stopfrom~Driver, at offset 8 ofAsyncLog, which matches yourlea 0x8(%rax),%rcx). Bounding that flush moves the hang to~DeviceUseGate's condition variable, and keeping theDriveralive at detach moves it intolibkernel.prx's static destructors. So the fix is in the test's exit path: #6925 makesagc_shader_helpersrunLibcRunShutdown_nid_postfixbefore returning, like the other driver tests. On a Radeon 860M: 0 of 60 timed out, against 4 of 30 onmainwith the same build.AI-assisted: yes (Claude Code).
oneandonlydean commentedon Oct 17, 2026
More Windows CI data points, from a batch of 11 manual Build runs on 10-16 (Linux passed in all):
guest_shader_linkguest_shader_linkagc_shader_disk_cacheagc_shader_helpersThe
agc_shader_helpersone matches this issue (the assertions passed before the timeout). The others are different tests, each normally under 0.1 s, on PRs that can't reach them, so #6925 alone won't clear them. In the #4288 log the only output is the libc/kernel init lines at 23:50:51 and then nothing until CTest killed the test at 23:52:51, so that hang may be at process start-up rather than at exit.Two things that would help to tell the cases apart: a CTest
--timeoutthat dumps the stack of the hung process (procdump orgdb -pon the runner), and running the sameLibcRunShutdown_nid_postfixfix onguest_shader_link,agc_shader_disk_cacheandagc_driver_pm4_ring_invalidto see whether they stop timing out.MarcZX10 commentedon Oct 17, 2026
I checked the current
upstream/maintest paths.agc_shader_disk_cache(core/libs/prx/libSceAgcDriver/tests/ShaderDiskCache.cpp) andguest_shader_link(core/libs/tests/GuestShaderLink.cpp) do not explicitly callLibcRunShutdown_nid_postfix()before returning. The registeredagc_driver_pm4_ring_invalidmode does call it intests/Pm4.cppbefore returning, so that timeout likely needs a different explanation or a check that the registered mode reached that call. This is source inspection only; it does not establish that adding shutdown fixes either of the first two Windows timeouts.fix(tests): end Windows test processes through libc before DLL detach
oneandonlydean commentedon Oct 17, 2026
Update: #6986 (merged as 9bf01db) should stop these Windows timeouts.
Cause: on Windows, returning from a test's
maingoes throughexit()->ExitProcess, which kills every other thread first and only then runs each prx DLL's static destructors atDLL_PROCESS_DETACH. Detached host threads (the driver's async log thread, libkernel's file-watch and alarm threads) die holding locks, and the destructors then wait on them forever (for example~Driver->AsyncLog::Flush, or winpthreads'pthread_cond_destroy). Linux doesn't hit it, becauseexitruns destructors while those threads are still alive. That's why it moves between tests (agc_driver_pm4_ring_invalid,agc_interpolant_mapping,guest_shader_link, ...) and why calling the libc shutdown inside a test (#6925) didn't cover it.Fix: MinGW test executables are linked with
--wrap=main. After the test'smainreturns, the wrapper runs_cexit()and then ends the process through libc's terminate path, so no DLL static destructor runs after the threads are gone. It applies to every test executable, with no per-test changes.Evidence so far: two full Windows runs of the PR head ea33174 (https://github.com/boykopovar/AnyPS5/actions/runs/38992435677, https://github.com/boykopovar/AnyPS5/actions/runs/38993049149), each 1351/1351 passed with no timeouts (about 128 s test time). Before the fix, main hit a timeout in about 5 of 11 Windows runs on 10-16. We'll keep counting timeouts on main's Windows jobs from 9bf01db on and report back here. If a 120 s timeout still turns up on a build that contains 9bf01db, please post the run link.
MarcZX10 commentedon Oct 17, 2026
Post-merge main CI data point after #6986, on
9bf01db663445820687e85fdbff08959d5193a5a:agc_shader_helpersandagc_driver_pm4_ring_invalidboth passed on Windows.These are clean CI runs on main after the merged exit-wrapper fix, not a long-term intermittent-rate estimate. Continued Windows runs would still be useful for the rare timeout rate reported before #6986.
sync: upstream batch 13 and forced 1x MSAA resident view clamp (#528)
test(agc): shut libc down before agc_shader_helpers returns
feat(metal): sync upstream batch 14 and optimize Civ 7 Metal memory t…