Add cluster-bootstrap 95% CIs for the paper's AUC tables

[dangng2004]

Summary

This PR modifies benchmark code. It might be helpful to get some context from the paper https://arxiv.org/abs/2606.19749. It adds 95% confidence intervals around the conference study's pairwise-accuracy (AUC) tables, via a cluster bootstrap over papers.

Method

The resampling unit is the paper, not the comment, because comments within a paper are correlated. Each of 5000 draws resamples papers with replacement, stratified by (proxy, side), recomputes the pooled statistic, and the CI is the 2.5/97.5 percentiles. Point estimates match the committed compute_auc.py, so the CIs sit around numbers that are already trusted.

Functions → paper tables

Each table kind produces exactly one paper format:

kind	function	paper table	what it shows
comment_volume	run_comment_volume	Table 1	per-model mean comments (high/low), delta, and overall accuracy
by_proxy	run_by_proxy	Table 2	accuracy broken down by quality proxy, best model per system
by_severity	run_by_severity	Table 9	accuracy broken down by severity tier, per model

The full-cohort appendix tables are the same three kinds on the full 240-paper set.

Representative table outputs

These tables are examples of the formats supported by the script (numbers reproduce the paper):

Comment volume and overall accuracy, per model (Table 1)

Accuracy by quality proxy, best model per system (Table 2)

Accuracy by severity tier, per model (Table 9)

Tests

The config, result JSONs, and manifests are large and gitignored, so the runner needs local data and isn't executable in a fresh clone. Correctness is covered without data by tests/test_ci_auc.py, which exercises auc_from, cell_summary, and the bootstrap on tiny in-memory count tables (e.g. two low-quality papers with more comments than two high-quality ones gives AUC 1.0).

$ pytest tests/test_ci_auc.py -v

tests/test_ci_auc.py::test_auc_from_perfect_separation PASSED
tests/test_ci_auc.py::test_auc_from_reversed PASSED
tests/test_ci_auc.py::test_auc_from_ties_get_half_credit PASSED
tests/test_ci_auc.py::test_auc_from_empty_side_is_nan PASSED
tests/test_ci_auc.py::test_cell_summary_point_estimates PASSED
tests/test_ci_auc.py::test_bootstrap_tiers_ci_shape PASSED
tests/test_ci_auc.py::test_bootstrap_overall_and_per_proxy PASSED
tests/test_ci_auc.py::test_dispatch_covers_the_three_kinds PASSED

8 passed

Files

• ci_auc.py — new, config-driven AUC + CI runner (three kinds: comment_volume / by_proxy / by_severity)
• compute_auc.py — now exports the shared auc_from; point-estimate logic unchanged
• tests/test_ci_auc.py — unit tests on in-memory data

[joe32140]

minor issue.

[joe32140]

ci_auc.py:81 (bootstrap) — per-proxy CI crashes when a proxy is single-sided.

The draw loop guards if high.size == 0 or low.size == 0: continue, so for a proxy that has papers on only one side, per_lists[proxy_id] is never appended and stays []. The return then computes ci(per_lists[proxy_id]) → np.percentile([], [2.5, 97.5]), which raises IndexError: index -1 is out of bounds for axis 0 with size 0 (numpy 2.x), aborting the whole by_proxy table.

This is the same partial-data case the point-estimate paths already defend against — cell_summary skips with if not high_total or not low_total: continue, and auc_from returns nan for an empty side — so a (method, model) cell where one side of a proxy is empty (e.g. missing/failed result files for all high-quality papers of that proxy) is anticipated elsewhere but will crash here.

Suggested fix — mirror the point-estimate guard:

ci = lambda xs: tuple(np.percentile(xs, [2.5, 97.5])) if xs else (float("nan"), float("nan"))

[dangng2004]

Good catch, fixed in af3ffad. Guarded the empty case so an empty per_lists[proxy_id] returns (nan, nan) instead of crashing np.percentile, matching the nan the point-estimate paths already return for a single-sided proxy. Added a regression test (test_bootstrap_single_sided_proxy_no_crash).