Skip to content

test(tools): split conventions_wrapped_declarations by test class to stay under 30 s on Windows - #14412

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
roboterspinne:test/tools-split-conventions-wrapped-declarations
Nov 2, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
roboterspinne:test/tools-split-conventions-wrapped-declarations

Conversation

@roboterspinne

Copy link
Copy Markdown
Contributor

What

conventions_wrapped_declarations runs all 17 tests of tools/tests/test_check_conventions_wrapped_declarations.py as one ctest test, and 15 of them call check() 20 times in all, each building a two-commit git repository and running python -I check_conventions.py --base HEAD~1 --head HEAD as a subprocess (check(), lines 16-46 on 26fa121; line numbers below refer to it). On the Windows job of main Build runs it took 33.38 s (run 41288908159), 33.23 s (41288882563), 32.62 s (41292426874) and 35.03 s (41288949548), and the slow-test step (.github/workflows/build.yml:731-737) warned in each run, for example conventions_wrapped_declarations took 33.4 s on Windows; tests must finish in 30 s (CONTRIBUTING.md) in run 41288908159 (see Tested). CONTRIBUTING.md:19 requires each test to finish in 30 s on the CI runners and to be split or shrunk when it gets close instead of raising its timeout. The repository already splits Python test files by class: check_main_sync and select_tests read the class names with file(STRINGS ... REGEX "^class [A-Za-z0-9_]*Tests\\(") and register one test per class (CMakeLists.txt:306-318 and 319-330).

Tested

Windows 11, WinLibs GCC 15.2.0, relinker-only Release build (cmake -S . -B build-relinker -G Ninja -DCMAKE_BUILD_TYPE=Release -DANYPS5_RELINKER_ONLY=ON -DBUILD_TESTING=ON), Python 3.13.9 (Anaconda), Git for Windows; synthetic fixtures only, no title or homebrew.

  • CI on main, Build workflow, windows job (ctest --parallel 4): Test #49: conventions_wrapped_declarations ... Passed in 33.38 s (run 41288908159), 33.23 s (41288882563), 32.62 s (41292426874) and 35.03 s (41288949548), each with the warning conventions_wrapped_declarations took 33.4 s (33.2, 32.6, 35.0 s) on Windows; tests must finish in 30 s (CONTRIBUTING.md), read with gh run view <run> --log.
  • Counter-check on 26fa121: ctest --test-dir build-relinker -N -R conventions_wrapped lists one test (conventions_wrapped_declarations), and python tools/tests/test_check_conventions_wrapped_declarations.py AbiAliasTests fails with AttributeError: module '__main__' has no attribute 'AbiAliasTests'. On this branch it lists conventions_wrapped_declarations (... test_check_conventions_wrapped_declarations.py WrappedDeclarationTests) and conventions_wrapped_declarations_AbiAliasTests (... AbiAliasTests), both with TIMEOUT 30.
  • Same tests: the module of 26fa121 and of this branch both load 17 tests with the same names; with no argv the script runs Ran 17 tests ... OK, with WrappedDeclarationTests 10 and with AbiAliasTests 7, all OK.
  • Timing on a quiet host (CPU load 9-23 %), back to back on the same build: the whole file, which is the command of the old test, ran in 30.0 s and 27.6 s; ctest -R conventions_wrapped_declarations then took 15.26 s and 14.07 s, and 14.55 s and 13.64 s, for the two tests. With other builds loading the host to 100 %, the whole file took 222.3 s and 104.1 s and the two classes 92.3 s and 79.3 s, and 54.7 s and 44.4 s; under that load the old test took 179.93 s and 113.03 s in ctest on 26fa121, and both new tests and check_main_sync, check_main_sync_CheckMainSyncHistoryTests and check_main_sync_CheckMainSyncEncodingTests, which have the same TIMEOUT 30, timed out.
  • ctest --test-dir build-relinker -j 6 on this branch with other builds running: 553 tests, 542 passed, long_paths skipped. Failed: select_tests and its parts select_tests_SelectGraphStateTests, select_tests_SelectBuildTests and select_tests_SelectArgumentTests with the Windows symlink fixture's FileExistsError, as on main on this machine; timed out under the load: annotate_module, conventions_wrapped_declarations, conventions_wrapped_declarations_AbiAliasTests, check_main_sync, check_main_sync_CheckMainSyncHistoryTests and check_main_sync_CheckMainSyncEncodingTests. Rerun alone once the host was quiet, all nine annotate_module, conventions_wrapped_declarations* and check_main_sync* tests passed (13.70, 15.34, 15.23 and 3.03-9.71 s). test_registration and test_script_registration passed.
  • python tools/check_conventions.py --base 26fa1212b8 --head HEAD with this title and description passed.
  • Not tested on Linux or macOS; CI runs the tests on all three hosts.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes (Claude Code)

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 1835 CMakeLists.txt
PR 5021 CMakeLists.txt
PR 7642 CMakeLists.txt
PR 9539 CMakeLists.txt
PR 10193 CMakeLists.txt
PR 13578 CMakeLists.txt
PR 13928 CMakeLists.txt
PR 13929 CMakeLists.txt
PR 14212 CMakeLists.txt
PR 14264 CMakeLists.txt

@github-actions github-actions Bot added the build label Nov 1, 2026

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no blocking findings

Splitting the 17-test module into two test classes and registering the second class separately is a correct test-only refactor: the 17 tests run unchanged and pass at head, and the CMake class discovery matches the file.

What I ran

  • Exact head c186f7b2ccc36f75c392ecfe0587f667f6c0b7a9, base ba627802e2210187d417deb64c8f2fed8e33011f.
  • Environment: Linux, /usr/bin/python3 3.11.2, no build.
  • Precheck: python3 tools/pr_precheck.py --repo boykopovar/AnyPS5 --pr 14412 --output-dir /dev/shm/reviewer-run/precheck/14412
  • Head tools/ extracted with git archive to /dev/shm/reviewer-run/src2/14412full, then, from that tree:
    python3 -m unittest tools.tests.test_check_conventions_wrapped_declarations (no -I)
    python3 tools/tests/test_check_conventions_wrapped_declarations.py AbiAliasTests
    python3 tools/tests/test_check_conventions_wrapped_declarations.py WrappedDeclarationTests
  • CMake class discovery run in script mode (cmake -P), which generates no build outputs, using the same file(STRINGS ... REGEX) as CMakeLists.txt.

Results

Precheck report, checks table, unedited (/dev/shm/reviewer-run/precheck/14412/report.md):

Check Status Command / reason
build_and_compile not_applicable The precheck deliberately does not configure, build, or compile AnyPS5.
diff_check passed git diff --check ba627802e2210187d417deb64c8f2fed8e33011f...c186f7b2ccc36f75c392ecfe0587f667f6c0b7a9
conventions_and_pr_template passed /usr/bin/python3 tools/check_conventions.py --base ba627802e2210187d417deb64c8f2fed8e33011f --head c186f7b2ccc36f75c392ecfe0587f667f6c0b7a9 --title test(tools): split conventions_wrapped_declarations by test class to stay under 30 s on Windows --body /dev/shm/reviewer-run/precheck/14412/pr-body.md
changed_export_abi_audit not_applicable No changed source or public header under core/libs/prx.
python_syntax not_applicable No changed Python implementation files.
unit:tools/tests/test_check_conventions_wrapped_declarations.py passed /usr/bin/python3 -m unittest tools.tests.test_check_conventions_wrapped_declarations
python_test_registration passed /usr/bin/python3 tools/check_test_registration.py --scripts
live_dependency_and_mergeability_check passed /usr/bin/python3 /marc/AnyPS5/tools/pr_stack_check.py 14412 --repo boykopovar/AnyPS5 --output-dir /dev/shm/reviewer-run/precheck/14412/stack
build_dependent_cpp_tests not_applicable No C/C++ implementation or test files changed.
test_quality_heuristics failed /usr/bin/python3 /dev/shm/reviewer-run/precheck/14412/head-c186f7b2ccc3-9cae4046/tools/pr_test_quality.py --root /dev/shm/reviewer-run/precheck/14412/head-c186f7b2ccc3-9cae4046 --json CMakeLists.txt tools/tests/test_check_conventions_wrapped_declarations.py
negative_controls failed A pass means every candidate failed by assertion on base; a failure means at least one did not discriminate or could not run. See per-test results.

Manual results:

manual unittest output (head c186f7b)
Ran 17 tests in 3.550s

OK
Ran 7 tests in 1.617s

OK
Ran 10 tests in 1.603s

OK
cmake -P class discovery output
-- base test keeps class WrappedDeclarationTests
-- would add conventions_wrapped_declarations_AbiAliasTests

Test-quality helper: one heuristic finding at line 123 (test_alias_undefinition_redefinition_and_cycles_do_not_create_exports has no recognized assertion). That test calls self.assertNotRegex, so the finding is a false positive.

Negative control

Not a discriminating control for a refactor. The precheck run on base reported did_not_fail_on_base because the base module runs the same 17 tests. The discriminating check is the CTest registration (ctest -N), which needs a configure step and was not run.

Findings

Blocking: none.

Non-blocking:

  • Timing margin: the author's quiet-host numbers put the two classes at about 14-15 s each against the 30 s bound. The margin under the CI Windows runner is not measured here.

Sources

  • Registration at head:

    AnyPS5/CMakeLists.txt

    Lines 268 to 283 in c186f7b

    ${CMAKE_CURRENT_SOURCE_DIR}/tools/tests/test_performance_csv.py)
    set_tests_properties(performance_csv PROPERTIES TIMEOUT 20)
    add_test(NAME conventions_wrapped_declarations COMMAND ${Python3_EXECUTABLE}
    ${CMAKE_CURRENT_SOURCE_DIR}/tools/tests/test_check_conventions_wrapped_declarations.py WrappedDeclarationTests)
    set_tests_properties(conventions_wrapped_declarations PROPERTIES TIMEOUT 30)
    set(conventions_wrapped_declarations_source "${CMAKE_CURRENT_SOURCE_DIR}/tools/tests/test_check_conventions_wrapped_declarations.py")
    set_property(DIRECTORY APPEND PROPERTY CMAKE_CONFIGURE_DEPENDS "${conventions_wrapped_declarations_source}")
    file(STRINGS "${conventions_wrapped_declarations_source}" conventions_wrapped_declarations_classes REGEX "^class [A-Za-z0-9_]*Tests\\(")
    foreach(test_class IN LISTS conventions_wrapped_declarations_classes)
    string(REGEX REPLACE "^class ([A-Za-z0-9_]+).*" "\\1" test_class "${test_class}")
    if(NOT test_class STREQUAL "WrappedDeclarationTests")
    add_test(NAME conventions_wrapped_declarations_${test_class} COMMAND ${Python3_EXECUTABLE} "${conventions_wrapped_declarations_source}" ${test_class})
    set_tests_properties(conventions_wrapped_declarations_${test_class} PROPERTIES TIMEOUT 30)
    endif()
    endforeach()
    add_test(NAME conventions_ci_gate COMMAND ${Python3_EXECUTABLE}

Not verified

  • ctest -N registration and the Windows CI timing were not run (configure needs a build).
  • Hosted: windows in progress and macOS clang queued at review time.

@oneandonlydean oneandonlydean self-assigned this Nov 2, 2026
@oneandonlydean
oneandonlydean merged commit 589e34d into boykopovar:main Nov 2, 2026
13 checks passed
@roboterspinne
roboterspinne deleted the test/tools-split-conventions-wrapped-declarations branch November 2, 2026 21:16
oneandonlydean pushed a commit that referenced this pull request Nov 4, 2026
…n test

Each case builds a git fixture and starts check_conventions.py, so the
two per-class tests from #14412 still passed 30 s on the Windows runner
and timed out on main (9090972, f7d225d). Register every test_
method as its own test with a 20 s limit; the first keeps the
conventions_wrapped_declarations name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants