AI Efficiency Benchmark

A benchmark for one question: when you add an "efficiency" layer to a coding agent, does the provider bill actually go down — without hurting task success?

Why this benchmark exists

Most context-optimization tools are judged by tokens removed. That number can mislead, for three reasons:

• Cache pricing. Most agent spend is prompt-cache traffic, billed at 0.1×–1.25× the input price. A removed token rarely saves a full token's price.
• Agent behavior. An agent that receives less context may search again, retry, or add turns — and every added turn re-transmits the whole context prefix.
• Success. A run that fails early is cheap. Cheap failures are not savings.

So this benchmark measures the thing that matters end to end: billed cost per successful execution, straight from the provider's own execution results.

How it works

• Every task runs under every arm (baseline + each optimization layer) from identical fresh working copies, in randomized order. Comparisons are paired, never pooled.
• Cost is the provider-returned total_cost_usd plus usage fields — never estimated from text length.
• Success is decided by deterministic judges, not by a model.
• Tasks, arms, judges, and the analysis plan are frozen in manifests before any spend.

What's in the repo

Path	What it holds
runner.py	The campaign runner (blocks × arms, spend cap, judges)
embed_sidecar.py	Local embedding sidecar the runner requires
deliverables/	Task manifests + the retained campaign datasets (per-run costs, ledgers, phase assignments)
analysis/	The statistics scripts (paired contrasts, task-clustered bootstrap)
docs/	Frozen methodology: pre-specifications, protocol, arm/run provenance
fixtures/	The datasets: synthetic-repo builders + pinned SHAs of the task repositories — see fixtures/README.md for what they are and why
component/	The component benchmarks (ContextBench, LoCoEval, InterCode proxy, structural retrieval, Caveman, grounding pilot, E2E pilot) — see component/README.md
fixtures_ground_truth.json	Retained copy of the synthetic-repo judge ground truth

No paper text and no result narratives live here — just what you need to run the benchmark or re-analyze its data.

Running it

Prerequisites: Python 3 with numpy; a Claude Code binary; the arm binaries and versions recorded in docs/ARM_PROVENANCE.md (including Headroom 0.27.0 for the proxy arm, which the runner starts itself).

1. Configure the environment.

# where runs and scratch worktrees go (default: /tmp/bvc-scratch)
export BVC_SCRATCH=/tmp/bvc-scratch
# where the synthetic repo is built (default: /tmp/bvc-synth)
export BVC_SYNTH=/tmp/bvc-synth
# your Claude Code binary
export CLAUDE_CODE_EXECPATH=/path/to/claude
# file with a line ANTHROPIC_API_KEY=... (default: ./.env.local)
export BVC_ENV_FILE=/path/to/.env.local
# dir prepended to PATH; must contain the headroom binary
# (default: ./.cache/venv/bin)
export BVC_VENV_BIN=/path/to/venv/bin
# dir with the frozen arm binaries rtk_v076 and rtk_gates_phase1_guarded,
# identified by sha256 in docs/ARM_PROVENANCE.md (default: ./bin)
export BVC_ARM_BIN=/path/to/arm/binaries

2. Fetch the task repositories at their pinned SHAs. (What these datasets are and why: fixtures/README.md.)

declare -A ORG=( [click]=pallets [cobra]=spf13 [express]=expressjs
                 [flask]=pallets [gin]=gin-gonic [requests]=psf )
for f in fixtures/repos/*.pinned_sha; do
  name=$(basename "$f" .pinned_sha)
  git clone "https://github.com/${ORG[$name]}/$name" "fixtures/repos/$name"
  git -C "fixtures/repos/$name" checkout "$(cat "$f")"
done

3. Build the synthetic repo (the synth tasks).

PYTHONPATH=fixtures python3 fixtures/build_synth.py     "$BVC_SYNTH"
PYTHONPATH=fixtures python3 fixtures/build_synth_ext.py "$BVC_SYNTH"

This writes the repo plus ground_truth.json (the retained reference copy is fixtures_ground_truth.json).

4. Start the embedding sidecar, then run a phase.

python3 embed_sidecar.py 8876 &

python3 runner.py --phase calibration --spent-so-far 0 --phase-cap 5.00

--phase is one of calibration, C1–C4, stress, holdout, e_cal, e_char_c1–e_char_c3, e_char_opus, e_holdout. --phase-cap is an absolute cumulative-dollar ceiling, enforced before every block; --spent-so-far carries the total across invocations.

Each run appends one row to deliverables/results.jsonl and deliverables/COST_LEDGER.csv — back up the retained datasets first if you want to keep your runs separate:

cp deliverables/results.jsonl deliverables/results.retained.jsonl

5. Analyze.

# paired contrasts + task-clustered bootstrap over one split
python3 analysis/stats.py  --results deliverables/results.jsonl \
    --split characterization --out stats_characterization.json

# expansion-aware variant (both splits, manifest-joined)
python3 analysis/stats2.py --results deliverables/results.jsonl \
    --manifest deliverables/MANIFEST_COMBINED.jsonl \
    --splits characterization,e_characterization --out stats2.json

# per-turn / per-tool-call ledgers from your own runs' transcripts
python3 analysis/extract_ledgers.py --results deliverables/results.jsonl \
    --runs-root "$BVC_SCRATCH/bvc_runs" --outdir deliverables/

Re-running bills real API spend. Re-analyzing the retained data in deliverables/ is free and offline.

Fine print

• Raw transcripts are withheld (they embed third-party repository content). results_metadata.jsonl is the full run record minus the transcript-derived result_text field.
• Tool-output token counts in the ledgers are local-BPE estimates, not provider counts.
• Third-party repositories are referenced by pinned SHA, not redistributed.

License: code (all .py files) under Apache-2.0 — see LICENSE-CODE; data and documentation under CC BY 4.0 — see LICENSE.