Skip to content

ci(sanitizers): replace the fixed known failures with the current ones - #15165

Open
ljkx wants to merge 1 commit into
boykopovar:mainfrom
ljkx:ci/sanitizer-known-failures
Open

ljkx wants to merge 1 commit into
boykopovar:mainfrom
ljkx:ci/sanitizer-known-failures

Conversation

@ljkx

@ljkx ljkx commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

What

.github/sanitizer-known-failures.txt lists eight tests whose fixes have merged (#10973, #11555, #9240, #11059, #11480; guest_getrlimit now skips under ASan). The nightly Sanitizers runs of 2026-10-31, 2026-11-01 and 2026-11-02 report all eight under "Listed as known but passed". This removes them.

The 2026-11-02 run (41384606155) fails eight tests the list does not name. This adds seven with their reason and fix:

ngs2_filter is left out: it fails a bit-exact check (Ngs2Filter.cpp:235), not on a sanitizer report.

Tested

WSL2 Ubuntu 24.04, Python 3.12: python3 -m unittest tools.tests.test_sanitizer_report passes, including RepositoryKnownFailuresTests (entries parse and stay sorted). tools/check_conventions.py reports nothing for the change. The entries were checked against the reports of the three nightly runs above; no sanitizer build was run locally.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes

🤖 Generated with Claude Code

The eight entries name fixes that have merged (boykopovar#10973, boykopovar#11555, boykopovar#9240,
boykopovar#11059, boykopovar#11480), and the guest_getrlimit test skips under ASan; the
nightly runs of 2026-10-31, 2026-11-01 and 2026-11-02 list all eight as
known but passed. The 2026-11-02 run (41384606155) fails eight tests
that the list does not name. Six have an open fix: agc_frame_limit
(boykopovar#14995), guest_np_common_runtime (boykopovar#14996), nkwebkit_wtf (boykopovar#15003) and
the three pad_identify tests (boykopovar#14438). guest_webrtc_vad stops on a
negative left shift in the bundled libfvad, which upstream also has.
ngs2_filter is left out: it fails on a numeric mismatch, not on a
sanitizer report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — looks good

The replacement list matches the sanitizer reports: seven sanitizer failures are listed, while ngs2_filter remains a separate bit-exact failure.

What I ran

Reviewed exact head 08c1d42fb93b890a3fdd01bb093825b746ac7ab1 against base a4eca2a056bf3bbbec98ab42e9d28163e4ea9cb8; the isolated merge preview was clean. Environment: Debian Linux x86_64, Python 3.11.2, Git 2.39.5. I did not build AnyPS5 or run a sanitizer build locally.

python3 tools/pr_precheck.py --repo boykopovar/AnyPS5 --pr 15165
python3 -m unittest tools.tests.test_sanitizer_report
git diff --check a4eca2a056bf3bbbec98ab42e9d28163e4ea9cb8...08c1d42fb93b890a3fdd01bb093825b746ac7ab1
gh run download 41384606155 -R boykopovar/AnyPS5 --name sanitizer-tests --dir /tmp/anyps5-sanitizer-41384606155
gh run download 41270298089 -R boykopovar/AnyPS5 --name sanitizer-tests --dir /tmp/anyps5-sanitizer-41270298089
gh run download 41165638390 -R boykopovar/AnyPS5 --name sanitizer-tests --dir /tmp/anyps5-sanitizer-41165638390
python3 tools/sanitizer_report.py /tmp/anyps5-sanitizer-41384606155/ctest.xml --known /tmp/anyps5-sanitizer-known-15165.txt --sanitizer address --sanitizer undefined

The last command uses the PR's exact allowlist extracted from head 08c1d42. I also ran the same report against the Oct 31 and Nov 1 artifacts using the allowlist from each run's own head.

Results

Check Result Note
Diff check failed The final added row has a CRLF terminator.
Conventions and template passed Exact head.
Sanitizer report unit tests passed 28 tests.
Nightly artifact reconciliation passed Seven new entries match; ngs2_filter is a separate failure.
Hosted build jobs not run Linux and Windows jobs were skipped for this file-only change.
Test-quality heuristic not run Its helper was absent from the isolated snapshot; it does not apply to this text-only change.
Commands and unedited output

python3 tools/pr_precheck.py --repo boykopovar/AnyPS5 --pr 15165

/tmp/anyps5-pr-precheck/boykopovar-AnyPS5-pr-15165-08c1d42fb93b/report.md
/tmp/anyps5-pr-precheck/boykopovar-AnyPS5-pr-15165-08c1d42fb93b/report.json
Conclusion: checks failed

Precheck output for the failed diff check:

.github/sanitizer-known-failures.txt:7: trailing whitespace.
+pad_identify_rumble address Heap use after free in SDL_GameControllerClose: the test's PadInput outlives SDL_QuitSubSystem, which has already freed its controller; #14438 destroys the pad input first.

The precheck's conventions command printed:

All conventions checks passed.

Its test-quality helper was absent in the isolated snapshot:

/usr/bin/python3: can't open file '/tmp/anyps5-pr-precheck/boykopovar-AnyPS5-pr-15165-08c1d42fb93b/head-08c1d42fb93b-156c0660/tools/pr_test_quality.py': [Errno 2] No such file or directory

python3 -m unittest tools.tests.test_sanitizer_report

............................
----------------------------------------------------------------------
Ran 28 tests in 0.380s

OK

Latest artifact summary (run 41384606155, head 926f7dd492b0b432d9287f4d6faaaf1c9d3c86c5):

3152 tests: 2334 passed, 810 skipped, 8 failed (7 known).
| `ngs2_filter` |
| `agc_frame_limit` | Runs past its 30 s ctest TIMEOUT in the sanitizer build (timed out at 30.06 s); without sanitizers it takes about 1 s. Every check builds its failure message before checking, and each of those allocations is instrumented; #14995 builds the message only when a check fails. |
| `guest_np_common_runtime` | UBSan stops at Strings.cpp:670 (libSceNpCommon), where sceNpUtilParseJid passes the null data() of an empty JID piece to memcpy with size 0; #14996 copies the pieces with std::copy. |
| `guest_webrtc_vad` | The bundled libfvad left-shifts a negative value (3rdparty/libfvad/src/signal_processing/resample_by_2_internal.c:141); upstream libfvad has the same code. |
| `nkwebkit_wtf` | The test expects null from WTF::tryFastMalloc for an impossible size, but the ASan allocator aborts with allocation-size-too-big (libSceNKWebKit/src/Bmalloc.cpp:34) because allocator_may_return_null is 0; #15003 sets it for this test. |
| `pad_identify_connect` | Heap use after free in SDL_GameControllerClose: the test's PadInput outlives SDL_QuitSubSystem, which has already freed its controller; #14438 destroys the pad input first. |
| `pad_identify_off` | Heap use after free in SDL_GameControllerClose: the test's PadInput outlives SDL_QuitSubSystem, which has already freed its controller; #14438 destroys the pad input first. |
| `pad_identify_rumble` | Heap use after free in SDL_GameControllerClose: the test's PadInput outlives SDL_QuitSubSystem, which has already freed its controller; #14438 destroys the pad input first. |

The Nov 1 and Oct 31 reports each listed the previous eight entries as “Listed as known but passed.” The Nov 2 raw ctest.log says NGS2 check failed at line 235 for ngs2_filter, with no sanitizer diagnostic for that test.

Negative control: remove only guest_webrtc_vad from a temporary copy of the allowlist, then run the same reporter on the Nov 2 artifact:

3152 tests: 2334 passed, 810 skipped, 8 failed (6 known).
| `guest_webrtc_vad` | /home/runner/work/AnyPS5/AnyPS5/3rdparty/libfvad/src/signal_processing/resample_by_2_internal.c:141:37: runtime error: left shift of negative value -32490 |

Hosted checks on the PR head:

edge-watch	skipping
linux	skipping
register-context	skipping
relinker	skipping
relinker-asan	skipping
windows	skipping
write-watch	skipping
conventions	pass
eligibility	pass
intake	pass
labels	pass

Negative control

Assertion Change made Outcome
Each listed sanitizer failure is classified as known. Removed guest_webrtc_vad from a temporary allowlist copy. The report surfaced it as an unlisted sanitizer failure, as expected.

Findings

Blocking

None found.

Non-blocking

The precheck flags the final added row's CRLF terminator as trailing whitespace. Normalizing that row to LF would make git diff --check clean; it does not change the report's classification.

Sources

Not verified

I did not run a local ASan/UBSan build or console tests. The scheduled sanitizer evidence is from the linked Ubuntu GitHub Actions runs. All PR build jobs were skipped for this change.

To get this to a good state

No behavior change is requested. If the final line ending is normalized, I will rerun git diff --check, the sanitizer report unit tests, and the artifact-to-allowlist comparison on the new head.

@mashingaan mashingaan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 08c1d42. The replacement list distinguishes the seven reported sanitizer failures from the separate bit-exact ngs2_filter failure, removes eight resolved or skipped cases, and keeps reasons tied to fixes or upstream behavior. The exact-head Hosted CI conventions, intake and labels checks pass. A separate review records comparison with the three nightly sanitizer reports and the focused report test.

@mashingaan mashingaan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my earlier approval: the independent exact-head review reports that git diff --check fails because the final added row in .github/sanitizer-known-failures.txt has a CRLF terminator. Please normalize that row to LF. My earlier approval missed this whitespace failure.

@xXPBSXx

xXPBSXx commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Data on ngs2_filter, which this list leaves out because it fails a bit-exact check and not a sanitizer report. I did not change anything; this only collects what I measured.

It is red in every nightly Sanitizers run I read, from 2026-11-02 to 2026-11-05, with the same message each time, NGS2 check failed at line 235 (Require(SameBits(out, expected)) in TestLowPassAndTail, Ngs2Filter.cpp). Line numbers are those of the job log as printed by gh run view --job <id> --log:

Nightly (UTC) Run Head Job Test result line Message line Host CPU
2026-11-02 41384606155 926f7dd 125082564980 14280 14282 AMD EPYC 9V74
2026-11-03 41573880531 f6232fc 125712822689 16127 16129 AMD EPYC 7763
2026-11-04 41768893283 1c8c0ef 126373686835 16393 16395 Intel Xeon Platinum 8370C
2026-11-05 41963336767 1882108 127034561116 16663 16665 AMD EPYC 7763

The run of 2026-11-01 (41270298089) failed the same way and the run of 2026-10-31 (41165638390) passed; I wrote that up under #13787. All four runs above use the image ubuntu-26.04 with glibc 2.43 (libc-bin 2.43-2ubuntu2.4) and GCC 15.2.0, on three different CPU models.

I could not reproduce it with the project's own flags. WSL2 Ubuntu 24.04 (g++ 13.3, AMD Ryzen 7 5800X3D; virtualised kernel, not bare metal) on main b11c7bf, configured as in sanitizers.yml (CMAKE_BUILD_TYPE=Release, ANYPS5_ENABLE_ASAN=ON, -g1 -fsanitize=undefined -fno-sanitize=vptr) with the job's test environment (ASAN_OPTIONS=detect_leaks=0, UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1, APS5_SKIP_TIMING_TESTS=1): 12 of 12 runs of ctest -R '^ngs2_filter$' passed. The test object is compiled without -march or -ffp-contract there, and objdump finds no vfmadd in the test binary or in the library. Only Filter.cpp, PitchShift.cpp and Sampler.cpp of the library carry -ffp-contract=off (Ngs2Library.cmake), the test target does not.

What does reproduce it: I compiled the same test object by hand with the same flags plus -march=x86-64-v3 and linked it. The binary then has 28 vfmadd instructions and ngs2_filter failed 3 of 3 runs with NGS2 check failed at line 235. With -march=x86-64-v3 -ffp-contract=off it has 14 (the explicit std::fma calls) and passed 3 of 3 runs; with neither flag it has none and passed 3 of 3 runs.

What I cannot show: the job log prints no compile flags, so I do not know that the runner builds the test with x86-64-v3. The only statement about it is in the entry for guest_math in .github/sanitizer-known-failures.txt, that ubuntu-latest's GCC 15 targets x86-64-v3 and fuses multiply-adds into FMA. So the cause on the runner is a hypothesis that fits these measurements, not a measured fact.

Checked on 2026-11-05 at 05:21 UTC.

AI-assisted (Claude Code).

@TechLevelZero

Copy link
Copy Markdown
Contributor

✅ AMD GPU test: passed

PR head 08c1d42 merged onto main 4639eff · tested 2026-11-05 10:18 UTC · 15 min

GPU AMD Radeon RX 6300 (RADV NAVI24), driver Mesa 25.2.8-0ubuntu0.24.04.4, Vulkan 1.4.318
Host Ubuntu 26.04.1 LTS, kernel 7.0.0-42-generic, container ubuntu:24.04
Logs build and test logs for this run
Build ✅ passed (Release, -DANYPS5_ENABLE_SPIRV_TOOLS=ON)
Tests ✅ 1,076 passed · 3 failed (0 new, 3 also fail on main) · 7 skipped (ANYPS5_REQUIRE_VULKAN=1)
Test setup CPU-only tests without a GPU; GPU tests (and any CPU-only test that failed or skipped without one) once on the GPU with VK_LAYER_KHRONOS_validation and ANYPS5_NO_SHADER_CACHE=1; main ran the same way
Test selection 1,086 of 3,962 tests run: the ones this change can reach, chosen by tools/select_tests.py from main's build graph
Vulkan validation ✅ no new validation IDs vs main (546 tests run on the GPU)
Slow tests (>30 s, GPU tests timed with validation layers) ⚠️ public_headers 62.1 s
Failing on main too (3, not counted against this PR)
agc_driver_image_table
agc_driver_native_depth_bda_clear
agc_driver_native_multisample_depth
Validation IDs already present on main (4 IDs in 426 tests)
VUID-VkDeviceCreateInfo-pNext-pNext (426 tests): agc_driver_address_draw_space_reuse, agc_driver_address_passes, agc_driver_alias_copy_label, agc_driver_alpha_to_coverage, agc_driver_array_layer_target, agc_driver_bc_block_copy, agc_driver_bda_anchor_reads, agc_driver_bda_span_reads, and 418 more
VUID-vkCmdCopyImage-srcImage-01548 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-vkCmdCopyImage-srcImage-01551 (4 tests): agc_driver_depth_clear_hle, agc_driver_depth_storage_dispatch, agc_driver_depth_surface_from_storage, agc_driver_dispatch_move_depth
VUID-VkPipelineColorBlendStateCreateInfo-pAttachments-00605 (1 test): agc_driver_prepared_exports
Skipped (7)
agc_driver_copied_writer_label_order: skipped, the device reports a minStorageBufferOffsetAlignment of 4, so the dispatch binds its record at +0x1004 in place and nothing is written back; the test needs an alignment above 4
agc_driver_float16_unused_nan_payloads: skipped, the device has no native f16 conversions with RTE rounding and kept f16 denormals
agc_driver_gcn_display_present: skipped, no display: Vulkan support is either not configured in SDL or not available in current SDL video driver (offscreen) or platform
agc_driver_linear_target_direct_write_completion: skipped, the device does not keep host imports write-watched, so no completion stamp over the target is tracked
agc_driver_ngg_vertex_wave_reduction: skipped, s3 describes a 32-lane wave only on 32-lane subgroups, and the device has 64 lanes
agc_driver_recorded_draw_staging: skipped, the device does not keep host imports write-watched, so the target is never resident
Reproduce
git fetch origin 08c1d42fb93b890a3fdd01bb093825b746ac7ab1 4639eff7f4c23325a916875a0e37f1e312fa18f4
git checkout 4639eff7f4c23325a916875a0e37f1e312fa18f4 && git merge --no-ff 08c1d42fb93b890a3fdd01bb093825b746ac7ab1
git submodule update --init --recursive
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DANYPS5_ENABLE_SPIRV_TOOLS=ON
cmake --build build && cmake --build build --target libs
VK_DRIVER_FILES=/nonexistent/icd.json ctest --test-dir build -E '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 12 --output-on-failure
ANYPS5_REQUIRE_VULKAN=1 ANYPS5_NO_SHADER_CACHE=1 VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
  ctest --test-dir build -R '^(agc|shader|recompiler|videoout|video_out|guest_video_out|vulkan)|gpu' --timeout 120 --parallel 16 --output-on-failure
# failures are then re-run alone: --timeout 360 --parallel 2; new validation IDs: the test alone, twice, --parallel 1

Automated build and test on a contributor's AMD machine; it does not replace upstream CI. A new comment is posted for each new push. Full logs.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci .github

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants