Skip to content

[PLA-2303] Report the human-review-policy status on PRs - #676

Open
devtools-agent[bot] wants to merge 2 commits into
masterfrom
pla-2303-human-review-policy
Open

devtools-agent[bot] wants to merge 2 commits into
masterfrom
pla-2303-human-review-policy

Conversation

@devtools-agent

@devtools-agent devtools-agent Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Linear: PLA-2303

What changed

Adds the two caller workflows for the shared human-review-policy check from rollbar/human-review-policy. That repo is public so SDK repos can use it; the private repos use the same code through rollbar/github-workflows#130.

  • .github/workflows/human-review-policy-trigger.yaml: a no-op that runs on PR events (opened, reopened, synchronize, ready_for_review, edited, closed) and review events (submitted, edited, dismissed).
  • .github/workflows/human-review-policy.yaml: runs on workflow_run of the trigger and calls the reusable workflow, which posts a human-review-policy commit status on the PR's head commit.

The status passes with 1 human approval besides the author on a human-authored PR and 2 distinct human approvals on a bot-authored one (e.g. devtools-agent[bot]). That way every production PR has two distinct humans on it. Approvals from bots and rollbar-circleci-machine don't count. Pushing commits to a PR doesn't stop someone from approving it.

Post-merge tracking: if a PR is merged without meeting the policy, e.g. through an admin break-glass bypass, it gets the needs-post-merge-review label and a comment saying how many approvals it still needs. Reviewers can still approve a merged PR. Once the policy is met, the label is removed and a closing comment is posted. To list what's still owed: is:pr is:merged label:needs-post-merge-review org:rollbar.

Why two files

pull_request workflows run the YAML from the PR's own branch, so a PR could rewrite the check that judges it. workflow_run workflows always run the copy on master. If a PR breaks the trigger, the status is never updated and the PR stays blocked.

Rollout order

  1. The first content of rollbar/human-review-policy has to be merged first. Until then the Human review policy runs here fail with "workflow not found". That's harmless, but noisy.
  2. Merge this PR. The status only starts appearing once human-review-policy.yaml is on master, because workflow_run only fires for workflows that are on the default branch.
  3. A repo admin then makes it required with the ruleset in ruleset.json: gh api -X POST repos/rollbar/rollbar-php/rulesets --input ruleset.json. Org and repo admins can bypass it through a PR for incidents (break glass); post-merge review is required in those cases.

Not part of this PR: the ruleset itself (step 3).

Pull requests from forks

These work. The trigger runs on the fork PR, and the evaluation runs in this repo with this repo's token, finding the PR by its head branch. If first-time contributors need approval before their workflows run, the status appears once a maintainer approves the trigger run.

Merge queues

The trigger also runs on merge_group, so in repos with a merge queue the status is written on each merge group's commit as well, and the queue isn't left waiting for it.

Validation

  • The workflow YAML parses. The shared logic has 41 unit tests and a read-only dry run against real rollbar/mox PRs.
  • Not yet run end to end here; that happens after steps 1 and 2.

🤖 Generated with Claude Code

Adds the two caller workflows for rollbar/human-review-policy: a no-op
trigger on PR and review events, and a workflow_run workflow (always run
from master) that posts the human-review-policy commit status: 1 human
approval besides the author on a human-authored PR, 2 on a bot-authored
one. PRs merged without meeting it are labelled needs-post-merge-review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

PLA-2303

CR-221

Trigger on merge_group so a merge queue isn't left waiting for the
required status, and write the workflow name unquoted so Prettier
accepts it whatever the repo's quote style.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rollbar-circleci-machine

Copy link
Copy Markdown

AI Agent Review LGTM (openai, openai-astra)

LGTM. No blocking findings were found.

LGTM, no findings. I read both added files and the existing workflows in the checkout. There are two caveats: the actual policy logic lives in a reusable workflow outside this repo, and nothing in the diff shows the workflows have been run.

What I checked

  • Trigger name matches. human-review-policy.yaml:11 listens for workflows: [Human review policy trigger], which is exactly the name: at human-review-policy-trigger.yaml:1. If the trigger were renamed, the workflow_run link would break without any error.
  • Trigger has no permissions and can't be injected into. permissions: {} is set (human-review-policy-trigger.yaml:14). The run step (:20) only interpolates github.event.pull_request.number and github.event_name, so attacker-controlled text never reaches the shell. On merge_group the PR number is just empty, which is harmless.
  • Policy workflow permissions are scoped to the job. They are set on the single uses: job (human-review-policy.yaml:16-19: contents: read, pull-requests: write, statuses: write). The file has no checkout or other steps of its own.
  • Covers PRs to every branch, by design. The trigger has no branches filter, so PRs into the next/1.x/**, next/2.x/** and next/3.x/** branches also get the status. The CI workflows, by contrast, are split by branch (ci.yml:19-21, ci-1.x.yml:20-22, ci-3.x.yml:21-23). Whether the status is actually required on those branches depends on the ruleset, which is out of scope here.
  • Default branch is master. This is consistent with update_js_snippet.yml:50 (base: master) and with the description's note that workflow_run only fires once the file is on the default branch.

Not verified (outside this checkout)

  • The reusable workflow. rollbar/human-review-policy/.github/workflows/human-review-policy.yaml@main is not checked out, so I could not check any of its claimed behaviour: approval counting, bot exclusion, finding fork PRs by head branch, the post-merge label and comments, merge-group handling, or whether it serialises concurrent runs. That last point matters because rapid review events (e.g. an approval followed immediately by a dismissal) start separate workflow_run runs. Unless the called workflow sets a concurrency group, they can finish out of order. Worth confirming over there.
  • It's pinned to @main. Any push to that repo's main changes the check for this repo without a PR here, and the code it runs gets pull-requests: write and statuses: write. That looks like a deliberate choice for central management, so I'm not flagging it. If you want changes to the policy reviewed per repo, pin to a tag or SHA.
  • Declared dependencies, not verified. The rollout dependency on rollbar/human-review-policy's first content landing and the private-repo counterpart rollbar/github-workflows#130 are both declared in the description. Neither is checked out, so I could not verify them.
  • Tests and runs. The diff contains no test or end-to-end run evidence. The description itself says the end-to-end run happens after rollout steps 1–2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant