Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/agent-contracts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@ jobs:
run: npm run typecheck
working-directory: evals

# The grader import scanner decides what gets staged into the container, and a
# naive one already failed the build on a prose sentence. These cases are the
# ones it must get right — including that a genuine bare import STILL throws,
# since the guard's entire value is its ability to fail.
- name: Self-test the import scanner
run: npm run imports:self-test
working-directory: evals

# Prove the graders still detect the regressions they claim to detect.
- name: Certify graders
run: npm run certify
Expand Down
8 changes: 8 additions & 0 deletions evals/graders/graderHarness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,14 @@ export const EXIT_GRADER_ERROR = 3;
*
* Which makes the marker more load-bearing, not less: red is now the not-applicable path, so
* the marker is what turns a red from "mysterious failure" into "known gap, here is the fix".
*
* That sentence is left here deliberately, and is load-bearing for something other than its
* meaning. `stage-graders.ts` once found imports with a regex over raw source, which read the
* words `from "mysterious failure"` as an import of a package by that name, could not find it
* in `node_modules`, and failed the build for everyone on a prose sentence. The scanner is now
* a real tokenizer that knows what a comment is, so this line is a **regression fixture**: if
* anyone ever swaps that tokenizer back for a regex, the build breaks here immediately rather
* than on whatever sentence somebody writes next. Do not "fix" it by rewording.
*/
export const NOT_APPLICABLE_EXIT_CODE = EXIT_GRADER_ERROR;

Expand Down
Loading
Loading