Skip to content

ci: enforce Codecov project coverage with a ratchet (#977) - #999

Open
tschm wants to merge 2 commits into
sfu-db:mainfrom
tschm:rhiza_fix_977_20261008
Open

tschm wants to merge 2 commits into
sfu-db:mainfrom
tschm:rhiza_fix_977_20261008

Conversation

@tschm

@tschm tschm commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Right now codecov.yml sets the project status to informational: true, so a drop in coverage can never fail a PR. This PR switches it to a ratchet:

  • target: auto with threshold: 1%: the project status fails if total coverage falls more than 1% below the base commit. Using the base commit instead of a fixed number means nobody has to bump the target by hand as coverage improves (it's 44.8% on main today).
  • codecov.notify.after_n_builds: 2 (and the same for the PR comment): ci.yml uploads two reports per commit (unit-tests and integration-tests). Without this setting Codecov can calculate the status from whichever upload arrives first, show a false drop and fail.

The patch target (60%) is unchanged.

Verification

  • curl --data-binary @codecov.yml https://codecov.io/validate returns Valid!.
  • The new rule only takes effect once Codecov posts its status on this PR.

Not addressed

  • fail_ci_if_error: false on both upload steps in ci.yml stays as it is. If it were true, a Codecov outage or a fork PR without CODECOV_TOKEN would turn the whole test job red. With after_n_builds: 2, a missing upload means no project status gets posted, rather than a failing one.
  • Making the Codecov status a required check is a branch-protection setting, so a maintainer has to do it.

Closes #977

🤖 Generated with Claude Code

Replace informational: true with target: auto and a 1% threshold, so a
PR that drops total coverage by more than 1% fails the project status.
Wait for both coverage uploads before reporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov-commenter

codecov-commenter commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.09%. Comparing base (d0d8e15) to head (5fcf3d6).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #999   +/-   ##
=======================================
  Coverage   45.09%   45.09%           
=======================================
  Files          65       65           
  Lines        9552     9552           
=======================================
  Hits         4307     4307           
  Misses       5245     5245           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@saurabh500

Copy link
Copy Markdown
Collaborator

@tschm This is very cool. I didn't know about this technique.

@saurabh500

Copy link
Copy Markdown
Collaborator

@wangxiaoying I am tempted to merge this PR :) Atleast no PR can reduce coverage with this change.

I will let this comment stay for sometime, in case you have any concerns.

@tschm
tschm requested a review from saurabh500 October 13, 2026 04:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce a project coverage threshold instead of informational: true

3 participants