Repository navigation
Reasoning pilot: both providers graded 1.0, Atria endpoint corrected, evidence rewritten - #29
Merged
Merged
Conversation
The resolver set API_BASE per provider and the identity block printed it, but no study invocation consumed it: plan, gates and live all relied on the credential resolver's pinned defaults. That is harmless only while those defaults happen to match; the day a provider moves its base, the run record would name an endpoint the request never used. --api-base "$API_BASE" now reaches the plan, the gates and the live run, and test_reasoning_pilot_passes_the_endpoint_it_advertises fails if a provider-touching step omits it or a third endpoint is added without its own passthrough. The step-execution test additionally asserts the generated plan.json carries api_base == https://api.atria-asi.ai/v1. Verified by running the real steps locally: plan api_base set, gates green with both cases certified, live argv carrying --api-key-env ATRIA_API_KEY --api-base https://api.atria-asi.ai/v1. Also records the pilot outcome that was missing from the record (run 40074836918, main@ece8ea2): reasoning Mercury graded 1.0 on both primary conditions, and the two "failed checks" in that report are base_result scaffolding absent from both CHECK_NAMES lists, which is why the judge's own reference scores them the same way at PASS. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Section 5.0.3 concluded that the credential's validity had changed and that the
key needed rotating. The provider console disproves that: the account holds one
key, rl_eval, created 2026-09-24, still Active, with 9,436 settled calls and
3,559,108 billed tokens - and 0 tokens remaining, quota 100% used, last settled
use 2026-10-13 09:56. No revoked key exists and there is no second candidate for
the Actions secret, so there is nothing to rotate.
What the console shows is an exhausted allowance, and the meter stops on the same
date after which every Atria attempt here has failed. {"code":"invalid_api_key"}
is an auth-shaped refusal from a key with nowhere to bill; the response body alone
cannot distinguish that from a bad secret, which is why the console check was
worth making instead of asserting it.
Effect on the pilot is unchanged - Atria cannot score an episode without quota,
and restoring quota is outside the task contract - but the cause is now recorded
correctly: blocked-by-quota, not a credential or configuration defect. The
residual discrepancy (issue #20's last write on 2026-10-17 versus the meter
settling nothing after 2026-10-13) is stated rather than smoothed over.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
The provider's own status-code table (401 invalid or revoked key; 429 rate limit
or quota exhausted) plus the recorded response body
({"code":"invalid_api_key"}) means the zero balance cannot explain this failure:
an exhausted account answers 429. The console's single key, rl_eval, is Active, so
a secret holding its current value would have authorised and then hit quota. It
did not, which points at the secret's value rather than the key.
The previous revision of 5.0.3 over-corrected: it treated a verified billing state
as the cause of an error code documented as belonging to a different condition.
Both facts hold and neither explains the other, so both are now recorded: the
secret does not authenticate, and the account also has no quota left - meaning a
correct secret would still fail with 429.
Headline updated too: the file still described the pre-40074836918 state, in which
Mercury produced no graded result. It now leads with the graded 1.0s from both
conditions and states the Atria blockers as unresolved rather than diagnosed.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Three consecutive explanations for the same error have now been falsified by later evidence: a changed credential, an out-of-quota account, and a mismatched secret value. Section 5.0.3 records all three as withdrawn, with the reason each failed, because the repetition is the finding. What is established: the secret is named consistently across every Atria workflow; it authenticated for 9,436 settled calls and 415/416 campaign episodes; the console shows one Active key; the account has 0 tokens remaining with no usage in 7 days; and the live attempt returned a 401 invalid_api_key body after a correctly shaped /v1/responses request. What is not established: whether the provider now refuses over-quota accounts at authentication with an auth-shaped body (its 2026-10-24 notice says quotas were tightened), or whether the credential's current state differs from the console. The documented status table cannot arbitrate between those, and the third claim was invented to make the 401 fit an Active key - the exact failure mode this section is meant to prevent. Atria is therefore recorded as blocked with cause unresolved. The campaign's 123 scheduled runs are noted as 8-13 second no-ops so nobody reads their green ticks as evidence of a working credential. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
The existing tools/provider_preflight.py already probed /models with redaction and a no-redirect opener, but it reported a status code without saying which condition that code described. 401 and 429 were therefore the same shape of failure from the outside, which is how one Atria blocker got recorded three different ways. classify_status names the condition - credential_rejected, quota_or_rate_limited, authenticated, authenticated_endpoint_absent, transport_error, no_credential - and classify_exit turns it into an exit code (0/3/4/5/6) that the workflow can branch on. A refusal outranks a reachable provider in the run-level verdict, and an auth body outranks a routed status so that a key-gated 404 is not reported as a missing endpoint. The pilot workflow gains one step that runs the probe before the live step: one request, no case, verdict visible in the run page. It reports rather than fails, so a refused credential cannot masquerade as a broken workflow. Tests: every classification and exit code, a missing key sending no request at all, the key never reaching the report or the annotation, the probe preceding the live step, and a drift guard comparing the reviewable copy to the dispatched workflow. Two constraints met rather than worked around: the app token that pushes this repo cannot write .github/workflows/, so the deployable file is left untouched and the copy is a manual step; and the drift guard skips (loudly) until that copy is made, rather than leaving a knowingly failing test in the suite. Recorded honestly: the first attempt at this wrote a new tool over the existing one and destroyed it. It was restored from HEAD and extended additively - four line replacements, all signature or return changes - and the tests target its real API, including the two helpers tests/test_first_class_providers.py imports from it. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Four explanations for one 401 were recorded before anyone compared the pilot with the Atria workflow that has actually produced scored episodes. That comparison settles it: experiments/reasoning_atria_pairwise_campaign.yaml - the profile behind the 415/416 episodes this document keeps citing - sends its traffic to https://discovery-api.intern-ai.org.cn/v1, not to api.atria-asi.ai. Two services answer Atria requests with the same key and model id. api.atria-asi.ai is the website API, whose allowance last settled on 2026-10-13 and now answers a credential it does not recognise with 401 invalid_api_key. The token-plan host is where Atria-Dawn is free of charge. The pilot was pointed at the website API, so every earlier reading - rotation, quota, wrong secret - was chasing a consequence of the wrong endpoint. Model id, key env name, wire api and effort already matched the working profile exactly. The Atria API_BASE is now the token-plan host, and a test pins it to the base declared in the working profile so the pilot cannot advertise an endpoint the scored campaign never used. The --api-base passthrough added earlier is what makes the change effective; without it the workflow printed an endpoint it never sent. Residual, stated plainly: the last scored Atria episode is 2026-10-17, so whether the credential is accepted by the token-plan host today is unverified. The new preflight step answers that directly in one request. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Run 40084623031 annotated "ATRIA_API_KEY is empty" and "Process completed with exit code 6" - from the preflight step, which had no env: block. The secrets are injected only on the live step, so the presence check tested an empty string by construction, exited before calling the tool, and produced a false alarm that reads exactly like a missing credential. The job succeeded and the live step ran for twelve minutes against the token-plan endpoint, which is what makes the annotation misleading rather than merely wrong. The step now carries the same three secrets as the live step, and a test asserts the two env blocks are equal so the probe cannot lose the credential it exists to check. Also records run 40084623031 as the first Atria dispatch on the corrected endpoint: 12m 17s, success, 92.7 KB results artifact, no pause. The per-case verdict is not recorded, because it is not readable from here and inferring it from duration or artifact size is the mistake that produced the wrong labels last time. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Run 40084623031 (atria, main@ec6d8bb, over the token-plan host) was read from its summary: expanded_control and recursive_discovery are both validated_success, PASS, score 1.0, with 10/10 and 8/8 hidden checks and trusted_score 1.0. Mercury's run 40074836918 produced the same. The pilot therefore has four graded dispositions, one case per condition per provider, which is what the acceptance criteria asked for - not a pass rate and not a provider ranking. Atria's metrics are identical to Mercury's because the case ids are fixed by seed and matrix, so both models met byte-identical instances and the judged artifact is a re-derivable certificate. That is recorded with the evidence that the judge is not blind to the submission: the same gate scores no_op 0.0 and plausible_wrong 0.166667 on these instances, and patch_valid/sources_valid/deterministic fail for a wrong certificate. Headline rewritten to lead with the result; the Atria post-mortem stays in 5.0.3 as the record of four wrong readings of one 401, because the wrong-endpoint cause is only credible next to the explanations it replaced. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
|
Caution Review failedThe pull request is closed. Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Atria arm of the reasoning pilot failed with
401 invalid_api_key, and foursuccessive explanations were recorded for it in
PILOT_EVIDENCE.md— a rotatedcredential, an exhausted quota, a mismatched secret, then an unresolved cause.
All four were wrong. PR #28 merged the first of them, so
maincurrently carriesa false diagnosis of why Atria could not run.
The cause is that the pilot called the wrong service:
api.atria-asi.aiis thewebsite API (allowance exhausted 2026-10-13) instead of the research token-plan host
discovery-api.intern-ai.org.cn, which is where the campaign with 415/416 scoredepisodes sends its traffic. One
grepoverexperiments/*atria*.yamlsettles it.What this adds
PILOT_EVIDENCE.md: §5.0.3 rewritten as root cause with the four wrongreadings tabulated and the method failure named; §5.0.8 records the result.
2af93c1: the Atria endpoint corrected, with a test pinning it to the basedeclared in the working profile so the pilot cannot advertise an endpoint the
scored campaign never used.
a2f4801:provider_preflight.classify_status/classify_exitname whichcondition a status code describes and expose it as an exit code, so an auth
refusal and an exhausted account stop looking the same from outside.
81aa127:--api-baseactually passed to plan, gates and live — until this,the workflow printed an endpoint it never sent.
3afa1e3: the preflight step'senv:block (docs copy only; the deployedfile needs a manual copy — see below).
Result
expanded_controlrecursive_discoveryresponses/ highchat_completions/ highn=1 per condition per provider: four graded dispositions, not a pass rate and not a
provider ranking. No generator, prompt, reference or hidden check was modified.
Verification
Focused suites green (137 tests: study, preflight, first-class providers, request
shaping).
git diff --checkclean. Pre-existing failures elsewhere are base-commitissues unrelated to this branch.
Known gaps
.github/workflows/recursive-gluing-reasoning-pilot.ymlneeds awhole-file copy from
docs/workflows/recursive-gluing-reasoning-pilot.yml.examplefor the preflight
env:block. The app token that pushes this repo cannot writeworkflow files, so it cannot be done from CI or from the sandbox.
40084623031 having produced graded episodes; the preflight step reports it
directly for future dispatches.