Repository navigation
Arena/192dc25c rl eval generator - #28
Merged
Merged
Conversation
…wire complete_responses() was the only request path that did not (a) reset per-call attempt counters, (b) refuse a request once --max-http-attempts was spent, or (c) record attempt logs. Atria controlled reasoning runs exclusively on that wire, so a dispatch that declared a physical-attempt ceiling got a trace with attempt_count=0, no api_error records for retried attempts, and no enforcement of the budget the provider-free plan had authorised. The request body, URL, retry policy and response parsing are unchanged; only the accounting now mirrors complete(). Four offline regression tests cover counting, retry logging, budget-bounded retries, and the exhausted-budget refusal. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
…ring evidence Records the bounded two-case slice (seeds=1: one case per primary environment), the provider-free plan and exact-instance gate for both Atria-Dawn-Preview (responses, reasoning high) and mercury-2.5 (chat_completions, reasoning high), the sanitized credential/endpoint probe, and an offline controller->judge wiring check with a transport stub (explicitly not a model result). No live scored episode was possible from this environment: no ATRIA_API_KEY or INCEPTION_API_KEY is present, the sandbox has no egress to either provider, and the Actions dispatch that holds the secrets is refused to the integration token (403). The first failing layer is a missing credential, not a code defect. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Adds docs/workflows/recursive-gluing-reasoning-pilot.yml.example, the copyable workflow for the first reasoning-model pilot, and documents the copy-to- .github/workflows step plus the click-path in recursive-gluing-eval.md. The pilot cannot be dispatched with a drifted configuration: the only inputs are provider (atria|mercury) and execution_mode, model/wire/effort/credential name/ endpoint are derived from the provider, and scope=diagnostic, seeds=1, the two frozen case ids and every ceiling (48 calls, 49152 tokens, 4800 s, 8 steps, 1024 tokens/response, 2 invalid retries, 192 derived physical attempts) are constants rather than inputs. The live step exposes only ATRIA_API_KEY / INCEPTION_API_KEY / MERCURY_API_KEY, refuses before the first paid call when the selected one is absent (exit 2), and never retries a pause. dispatch-only: no schedule, no push, no self-re-arm. test_reasoning_pilot_workflow_pins_the_frozen_slice_and_its_ceilings locks the contract, including that the two triples are the only ones tools/recursive_ gluing_study.build_plan accepts and that no shell step references a secret. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Run 40067626107 (main@758a201, provider=atria, plan_only) failed in the Plan step after ~1s with 'MODEL: unbound variable': no plan was written, the gates and live steps were skipped, and the plan artifact was empty. The resolver exported lowercase names (model=, wire_api=, key_env=) while the plan, gates and live steps dereference $MODEL, $WIRE_API, $KEY_ENV under set -u. No provider call was made and nothing was spent. Exports are now uppercase, matching their readers. A new test runs the resolver and then the plan step in sequence under set -u: it asserts every variable the later steps read was exported by the resolver, and that plan.json appears with 2 cases, within_limits, and the 192-attempt derived budget. Against the failed workflow it reports 'Plan (provider-free) reads $MODEL but the resolver exported [...]'. Re-verified provider-free after the fix: plan -> gates (both cases certified) -> report -> diagnose all green; the live step's argv was audited with a stubbed python. The YAML parsed and every step passed bash -n on the broken file, so only running the steps in order catches this class of bug. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
… summary
The diagnose step piped --github into $GITHUB_STEP_SUMMARY, which turns the
::error::/::notice:: workflow commands into literal text in the summary instead
of run-page annotations. Annotations now go to stdout (they are parsed from
there) while the readable markdown is appended to the summary, and the JSON form
is persisted next to the checkpoint.
Records run 40068694722 in PILOT_EVIDENCE.md: plan and gates passed, the live
leg paused on Atria's own {"code":"invalid_api_key"} body, classified
credential_rejected - a rejected credential, not a missing one (a missing secret
exits 2 before any call), and not a quota, rate-limit, model-ID, endpoint or
protocol failure. No retry of the same key, no ceiling change, no fallback
provider.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
…26-10-24 Uses the repository's own campaign state instead of any key handling. Status issue #20 (Reasoning-Atria pairwise) recorded 415/416 episodes checkpointed and a pause only on max_tokens_total on 2026-10-17: Atria-Dawn-Preview over /v1/responses with nested reasoning.effort accepted this repo's ATRIA_API_KEY and scored at scale. Issue #21 (Mercury pairwise) recorded 416/416 completed on 2026-10-16. Run 40068694722 on 2026-10-24 returned Atria's invalid_api_key with the same secret name every Atria workflow maps. The harness, endpoint, protocol, model id and secret mapping are unchanged; the credential's validity changed between those dates, and only the repository owner can act on it. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Records what the API can prove: plan and gates succeeded, the live step exited 0 (controller reported no pause), both artifacts uploaded, 3m3s wall clock, and no pause or error annotation. The per-case verdict/score lives only in the results artifact, whose storage and the job log are unreachable from this sandbox and whose summary is not exposed over REST, so the outcome class is marked recorded but not yet read rather than inferred from artifact size or duration. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Run 40070994013 (mercury live pilot) reported both cases as scored_model_failure
with verdict FAIL, score null and no failed checks. That combination cannot come
from a graded answer: score_from_accuracy always writes a number and mark_check
records every hidden check, so those rows came from an early set_failure + emit.
Cause: shared/judge_lib.py defines lowercase codes, but the recursive-gluing
judge emits literal uppercase ("RUNTIME_ERROR", "REWARD_DENIAL"). Both
classify_result and the controller's status branch compared those strings
exactly, so an uppercase pre-scoring failure matched no malformed mode and fell
through to scored_model_failure - an ungraded submission reported as a model
score, which the pilot contract forbids.
Fix is on the reading side only: normalise the mode's case, and never call a
status=scored row without a numeric score a scored failure. The judge, generator,
prompts and references are untouched, so the judge bytes, pinned manifests and
exact-instance gate stay valid.
Two regression tests reproduce the run's exact rows; both fail against the old
code with 'scored_model_failure' != 'malformed_or_incomplete_certificate'.
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
Run 40070994013 rejected both submissions before grading under a 1024-token cap. At max_steps=8 and 2 invalid-response retries the worst case is 4096 x 8 x 3 = 98304 tokens per case, so the two-case ceiling becomes 196608 (verified by the planner: 2 cases, within_limits true, 192 derived physical attempts). With the old 49152 ceiling the plan would have failed closed instead of running, which is the intended behaviour; the token ceiling is raised deliberately here rather than discovered at dispatch time. Nothing else changes: still 2 cases, 8 steps, 48 logical calls, 4800 s. Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>
|
Caution Review failedThe pull request is closed. Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
title: ""
labels: ""
Summary
One-sentence description of changes.
Changes
Testing
pytest -qpassesRelated Issues
Fixes #
Closes #
Notes
Any additional context for reviewers.