Skip to content

Skip Vally evaluations on fork pull requests - #3302

Draft
Shraddha Jain (shrja-ms) with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-eval-job-failure-another-one
Draft

Skip Vally evaluations on fork pull requests#3302
Shraddha Jain (shrja-ms) with Copilot wants to merge 2 commits into
mainfrom
copilot/fix-eval-job-failure-another-one

Conversation

Copilot AI commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

The Vally evaluations / eval-job check fails on every pull request originating from a fork: all trials error with Session was not created with authentication info or custom provider.

GitHub does not expose repository secrets to pull_request runs from forks (job log: Secret source: None, and GITHUB_TOKEN limited to read scopes — the requested copilot-requests: write is not granted). COPILOT_GITHUB_TOKEN is therefore empty, the vally copilot-sdk executor cannot create a session, and all 45 trials error out. Fork runs can never pass; only same-repo runs can.

Changes (.github/workflows/vally-eval.yml)

  • Added a job-level IS_FORK_PR env expression derived from github.event.pull_request.head.repo.full_name != github.repository.
  • Guarded the vally-specific steps — CLI install, build_info.json, server build, eval generation, Run vally, and artifact upload — with if: ${{ env.IS_FORK_PR != 'true' }}. This also avoids ~5 minutes of pointless build time per fork PR.
  • Added a Report skipped evaluations step that emits a ::notice:: pointing maintainers to re-run the evaluations from a branch in this repository.

Same-repo pull requests and workflow_dispatch runs are unaffected.

    env:
      IS_FORK_PR: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}

Note

The evaluations still provide no coverage for fork PRs. If they need to run there, that requires a separate design (e.g. a pull_request_target-based or maintainer-triggered workflow with the usual untrusted-code safeguards) rather than wiring a token into this workflow.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

…ilable

Co-authored-by: shrja-ms <77041475+shrja-ms@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix failing GitHub Actions job for evaluations Skip Vally evaluations on fork pull requests Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants