BreakPoint is a Python SDK and CLI for turning real LLM failures into regression eval datasets. Feed it failure traces, support tickets, red-team transcripts, RAG logs, or tool-call traces; it compiles them into versioned eval cases with expected answers, hidden traps, rubrics, adversarial variants, validation reports, and exportable runner formats.
The core idea is failure-to-eval compilation: one observed failure becomes a small failure neighborhood that can be replayed against future prompts, models, RAG pipelines, and agent/tool policies.
- Typed Python schemas for eval items, expected answers, hidden traps, rubrics, adversarial variants, model votes, validation reports, dataset bundles, suites, runs, reviews, clusters, and gates.
- Failure taxonomy for hallucination, instruction conflict, multi-hop reasoning, tool misuse, long-context retrieval, refusal boundaries, format violations, RAG contradiction, and prompt injection.
- BreakPointSpec YAML for declarative failure-family definitions.
- Trace2Eval ingestion for normalized failure traces, including RAG evidence, tool calls, original model output, expected behavior, provenance, and severity.
- Production ingestion adapters for OpenTelemetry/OpenInference spans, LangSmith traces, LiteLLM logs, retrieval logs, support tickets, thumbs-down feedback, red-team transcripts, and incident reports.
- Mutators for irrelevant context, reordered facts, renamed entities, paraphrased instructions, prompt attacks, and stale/contradictory evidence.
- Live judge adapters for OpenAI, Anthropic, Gemini, and an OpenAI-compatible local vLLM endpoint.
- Judge calibration, model-under-test comparison, risk-focused CI pack selection, and production regression-pack compilation.
- Export paths for JSONL, Hugging Face rows, DuckDB, OpenAI Evals-style YAML, lm-eval YAML/JSONL, CI reports, and FastAPI.
python -m venv .venv
.\.venv\Scripts\python -m pip install -r requirements-dev.txt
.\.venv\Scripts\python -m pytestCompile demo eval data:
.\.venv\Scripts\python -m breakpoint_eval.cli compile --items-per-category 4 --variants-per-item 2Compile sourced public failure traces:
.\.venv\Scripts\python -m breakpoint_eval.cli actual-dataCompile actual traces with live judges:
.\.venv\Scripts\python -m breakpoint_eval.cli actual-data --external-judges --max-live-cost-usd 2Generate live judge evaluation charts:
.\.venv\Scripts\python -m breakpoint_eval.cli judge-report --results artifacts/actual/trace2eval_results.json --out-dir artifacts/reportsBuild a production-style regression pack through the ingestion layer:
.\.venv\Scripts\python -m breakpoint_eval.cli ingest-traces --input data/actual/failure_traces.json --source incident_report
.\.venv\Scripts\python -m breakpoint_eval.cli production-pack --input artifacts/ingestion/traces.json --source incident_report --max-records 10 --external-judges --changed-files "app/rag/retriever.py,agents/tools/order_tool.py" --risk highfrom breakpoint_eval import BreakPoint, TraceBuilder
trace = TraceBuilder.rag_failure(
id="prod-rag-001",
question="Which policy is current?",
bad_answer="The cached summary says the limit is 7 days.",
expected_behavior="Use the newer official policy and answer 30 days.",
retrieved_docs=[
{"id": "cache", "content": "The limit is 7 days.", "effective_date": "2025-02-01", "reliability": 0.35},
{"id": "policy", "content": "The limit is 30 days.", "effective_date": "2026-07-01", "reliability": 0.95},
],
)
bp = BreakPoint(include_external_judges=False, variants_per_item=3)
build = bp.build_pack([trace], output_dir="artifacts/sdk_pack")
print(build.total_cases, build.artifact_paths)The current actual corpus lives at data/actual/failure_traces.json. It contains 12 sourced public incidents converted into normalized traces:
- Air Canada bereavement-fare chatbot hallucination
- NYC MyCity legal/compliance misadvice
- Mata v. Avianca fake legal citations
- DPD support chatbot manipulation
- Chevrolet dealership chatbot pricing manipulation
- CNET finance article errors
- NEDA Tessa harmful medical-adjacent advice
- Microsoft Tay social manipulation
- Pak'nSave Savey Mealbot unsafe recipe generation
- Google Bard JWST demo error
- Michael Cohen fake legal citations
- McDonald's AI drive-thru order failures
The latest live run produced:
| Metric | Value |
|---|---|
| Sourced traces | 12 |
| Accepted base items | 12 |
| Adversarial variants | 36 |
| Total eval cases | 48 |
| Live judge set | 10 live judges: OpenAI GPT-5.5, GPT-5.4, o3, GPT-5.4 mini; Anthropic Claude Sonnet 5, Claude Opus 4.8, Claude Fable 5; Gemini 2.5 Pro, Gemini 2.5 Flash; local vLLM |
| Estimated live judge cost | $0.3218 |
| Base validation passed | 12/12 |
| Human compiler-accepted labels | 8/12 |
| Human compiler-rejected labels | 4/12 |
These charts are generated from artifacts/actual/trace2eval_results.json, not from the synthetic category-quota demo. The score chart uses human labels as compiler-acceptance labels and penalizes false positives: judges lose credit when they accept eval cases that humans marked invalid, mislabeled, or needing repair. It is not a model leaderboard.
BreakPoint uses its own failure-oriented indices rather than broad competitor benchmark indexes:
- Failure Neighborhood Index
- Evidence Conflict Index
- Boundary Precision Index
- Instruction Attack Index
- Judge Consensus Index
- Source Tension Index
Latest index values:
| BreakPoint index | Score |
|---|---|
| Failure Neighborhood Index | 79.4 |
| Evidence Conflict Index | 84.6 |
| Boundary Precision Index | 73.2 |
| Instruction Attack Index | 80.5 |
| Judge Consensus Index | 86.7 |
| Source Tension Index | 66.3 |
calibrate-judges produced artifacts/calibration/judge_calibration_report.json, gold_labels.json, and calibrated_gate_policy.json from the 12 human-labeled actual traces. The final calibration treats wrong failure-family labels as compiler rejections, so the human calibration set is 8 accepted and 4 rejected items. It compares 10 live judges plus three deterministic local controls and promotes 28 judge/failure-family pairs. The filtering chart now ranks OpenAI GPT-5.5 first because it catches 3 of the 4 rejected compiler cases, while pass-everything judges such as Gemini 2.5 Flash and local vLLM are penalized for catching 0 of 4 rejected cases.
model-runs compares deterministic simulated baseline profiles against the compiled cases. These are not live model API outputs; they are controlled stress profiles used to verify that the suite exposes stale-RAG, over-refusal, and injection-prone behavior. The current 48-case run produced:
| Profile | Pass rate | Avg score | Cost per reliable pass |
|---|---|---|---|
| breakpoint-reference | 100.0% | 0.943 | $0.000100 |
| stale-rag-baseline | 10.4% | 0.442 | $0.000768 |
| over-refusal-baseline | 0.0% | 0.441 | $0.002400 |
| injection-prone-agent | 20.8% | 0.686 | $0.000576 |
The ingestion/production-pack route was validated by normalizing 12 incident-report traces and compiling the first 10 into a live-judged regression pack:
| Artifact | Value |
|---|---|
| Ingested traces | 12 |
| Production pack traces | 10 |
| Accepted base items | 10 |
| Total eval cases | 40 |
| Regression packs | 5 |
| Estimated live judge cost | $0.0855 |
.\.venv\Scripts\python -m breakpoint_eval.cli categories
.\.venv\Scripts\python -m breakpoint_eval.cli compile-spec
.\.venv\Scripts\python -m breakpoint_eval.cli trace2eval --traces path\to\traces.json
.\.venv\Scripts\python -m breakpoint_eval.cli actual-data --external-judges
.\.venv\Scripts\python -m breakpoint_eval.cli judge-report
.\.venv\Scripts\python -m breakpoint_eval.cli ingest-traces --input path\to\logs.json --source support_ticket
.\.venv\Scripts\python -m breakpoint_eval.cli production-pack --input artifacts\ingestion\traces.json
.\.venv\Scripts\python -m breakpoint_eval.cli calibrate-judges
.\.venv\Scripts\python -m breakpoint_eval.cli model-runs
.\.venv\Scripts\python -m breakpoint_eval.cli ci-packs --changed-files "app/rag/retriever.py"
.\.venv\Scripts\python -m breakpoint_eval.cli failuregym
.\.venv\Scripts\python -m breakpoint_eval.cli vllm-judge-server --port 8001actual-data writes:
artifacts/actual/source_traces.jsonartifacts/actual/trace2eval_results.jsonartifacts/actual/metrics.jsonartifacts/actual/manifest.jsonartifacts/actual/product.jsonartifacts/actual/cases.jsonlartifacts/actual/openai_evals.yamlartifacts/actual/lm_eval_task.yamlartifacts/actual/lm_eval_task.jsonlartifacts/actual/ci_report.jsonartifacts/actual/ci_report.md
judge-report writes:
artifacts/reports/live_judge_report.jsonartifacts/reports/judge_scoreboard.pngartifacts/reports/judge_confidence_matrix.pngartifacts/reports/failure_index_by_trace.pngartifacts/reports/cost_reliability_pareto.png
calibrate-judges, model-runs, ingest-traces, production-pack, and ci-packs write:
artifacts/calibration/judge_calibration_report.jsonartifacts/calibration/calibrated_gate_policy.jsonartifacts/model_runs/model_run_comparison.jsonartifacts/model_runs/model_pass_rates.pngartifacts/model_runs/family_model_matrix.pngartifacts/model_runs/cost_latency_tradeoff.pngartifacts/ingestion/traces.jsonartifacts/ingestion/report.jsonartifacts/production_pack/manifest.jsonartifacts/production_pack/cases.jsonlartifacts/production_pack/ci_report.jsonartifacts/ci/packs.json
BreakPoint is meant to sit inside existing Python evaluation and release workflows:
- Normalize real failures into
RawFailureTracerecords. - Compile traces into BreakPointSpec drafts and eval cases.
- Generate adversarial variants around each failure.
- Validate with local and live judges.
- Export cases to your eval runner.
- Gate regressions in CI using recent failure packs.
The repeatable local path uses deterministic judges. Live judges are opt-in through --external-judges, external_judges=True, or BREAKPOINT_EXTERNAL_JUDGES=1.
breakpoint_eval/ Python SDK, compiler, validators, reports, CLI
data/actual/ Sourced public failure traces
examples/ SDK examples for RAG, tool, support, and red-team traces
scripts/ Artifact and report generation helpers
tests/ Unit and API tests
artifacts/actual/ Generated actual-data outputs
artifacts/reports/ Generated live judge charts and report JSON
artifacts/model_runs/ Generated model-under-test comparisons and charts