Repository navigation
[ci] Guard every test file against landing without a CI lane - #2084
Open
fv-superagent wants to merge 3 commits into
Open
fv-superagent wants to merge 3 commits into
fv-superagent wants to merge 3 commits into
Conversation
The test-collection contract checked directories, so a new file beside a partially listed directory could merge and never run. Resolve each CI pytest invocation and require every test file to be collected or listed in a shrink-only allowlist of the 50 files that are dark today.
Contributor
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
Fix five parser gaps found in review: -m "not gpu" now makes a gpu-only file dark (tests/attention/test_vsa_h3_heads_first_tile.py joins the allowlist, now 51 entries); a bare --ignore consumes its path; only a real pytest command word counts, so $pytest_rc no longer creates a phantom invocation; variable substitution is word-bounded; and quoted or exported assignments resolve.
Contributor
[Medium impact] The PR appears safe to merge; all previous findings are addressed and no new blocking issue was found.
|
Address Greptile review: the SSIM lane globs only test_*.py at the ssim root, so track those files instead of the directory; drop the redundant section labels; and cover continuations, both --ignore forms, variable expansion, gpu-only files and the phantom-invocation case with focused tests.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is by an automated agent, not a human.
Problem:
fastvideo/tests/contract/test_ci_test_collection.pychecked coverage at directory granularity, so a new file beside a directory that a lane lists only in part stayed invisible to CI.fastvideo/tests/AGENTS.mdalready warns about this under "CI Wiring", and the second-pass audit traced the gap totests/hooks,tests/audio, most oftests/layersandtests/distributed, and 16 MLX files: #2074 (comment).Change: resolve what each CI
pytestinvocation collects (shell and YAML line continuations, quoted or exported variable assignments such asgolden_root=./fastvideo/tests/..., both--ignoreforms,-m "not gpu", and the filesssim/ci_runner.pyglobs at the ssim root, which is not recursive), then require everytest_*.pyunderfastvideo/tests/to be collected or listed in a shrink-onlyDARK_TESTSallowlist. The allowlist holds the 51 files that are dark today, each with a reason, including one gpu-only file the unit lane deselects with-m "not gpu";test_dark_test_allowlist_stays_accuratefails when an entry is fixed or goes missing. The old directory-levelALLOWLISTand its two tests are replaced. Test-only.Verification:
python3 -m py_compile fastvideo/tests/contract/test_ci_test_collection.py: ok.pytestis not installed in this environment, so I ran the four tests directly withyamlstubbed:test_every_test_file_is_collected_or_allowlisted,test_dark_test_allowlist_stays_accurate,test_parser_handles_the_command_shapes_ci_uses(continuations, both--ignoreforms, quoted and exported variables, word-bounded expansion, gpu-only files, a phantom$pytest_rcline and non-recursive SSIM discovery) andtest_local_tests_stays_out_of_ci; all pass. The parser resolves 33 invocations, 508 test files and 51 allowlisted; the dark set equals the allowlist exactly, with no entry stale.fastvideo/tests/hooks/test_zz_tmp_dark.pyfails the guard and names that path; marking the collectedfastvideo/tests/contract/test_env_policy.pydark fails the accuracy test.1316cd77ran theunitlane (which collectsfastvideo/tests/contract/) green; thevaelane failed once on a pyxis enroot image-import error and passed on retry. The current head reruns all lanes.pre-commit run --files fastvideo/tests/contract/test_ci_test_collection.py: the hook cache path is read-only, and.pre-commit-config.yamlexcludesfastvideo/tests/anyway.Risk: test-only. No runtime code, public API or CI config changes. The parser does not resolve a lane that selects tests through a shell glob or a generated list; such files look dark and need a name or an allowlist entry, which the docstring and failure message state. Marker handling is limited to
-m "not gpu"with function- or module-level gpu markers. Theunitlane's existing coverage is unchanged.