Reference implementation and reproducibility artifact for "Argos: Learning When Not to Generalize in JSON Schema Induction."
Argos induces a JSON Schema by generating a family of candidate schemas that differ in how readily they generalize a rare key into a dictionary, and selecting among them per sample from held-out positives and synthetic negative probes.
There are no precomputed result CSVs. Every number in the paper is reproduced by running the artifact, and every driver checks its own output against the paper's stated value before exiting:
Table 1 -- clean-data accuracy
------------------------------------------------------------
PASS mean F1 got 0.9934 paper 0.9934 tol 0.0005
PASS datasets with P=1.000 got 17 paper 17 exact
PASS worst case got 0.9630 paper 0.9630 tol 0.0005
all 3 checks match the paper.
argos/expected.py holds all 92 claim constants as data; argos/verify.py
performs the comparison. A driver that cannot reproduce its figure exits
non-zero, so a stale number cannot pass silently — a property that shipping CSVs
does not give you.
argos/ inducer, harness and experiment drivers
argos.py candidate family and selector
vfi_core.py streaming validation-first core (VFI-Default, VFI-Aggr)
dict_core.py dataset-level cores (DictBranch, Structural)
strict_vfi.py diagnostic candidate at the closed end of the axis
runner.py splits, seeds, scoring -- one metric implementation
expected.py every paper claim, as data
verify.py recomputed-vs-paper checks
paths.py all input/output locations; no absolute paths anywhere
atomic.py crash-safe writes
poisoning_pkg/ PoisonedNYT construction and the contamination sweeps
SyntheticDatasetGen/ generators for the synthetic benchmark datasets
data/ inputs; see data/README.md (most are not redistributed)
docs/ CLAIMS.md, DATA.md, PITFALLS.md
env/hardware.txt the machine all reported timings were measured on
tests/ tests for the verification layer itself
out/ everything a run produces (git-ignored)
Python 3.10+. The inducer and harness use the standard library only, so
Tables 1–3 and Section 5.6 reproduce in an empty environment. scipy and
matplotlib are optional, for the reported statistics and Figure 2.
ReCG must be obtained and built separately; point ARGOS_RECG_BIN at the binary.
Stages needing it skip cleanly when it is unset.
make install # venv + optional extras
make test # tests for the verification layer./scripts/run_all.sh smoke # ~1 min one dataset, one seed; verifies the install
./scripts/run_all.sh core # Tables 1-3, Figure 2, Sections 5.3-5.6
./scripts/run_all.sh full # adds contamination, the repair, sweeps, scale-upOutputs land in out/, so a run never modifies the repository. Redirect with
ARGOS_OUT, and point ARGOS_DATA elsewhere if your inputs live outside the tree.
Start from docs/CLAIMS.md: it maps every table, figure and quantitative claim to
the driver that produces it and the constants it verifies.
docs/PITFALLS.md documents harness details that silently change results — above
all that the benchmark's negatives are wrapped in a provenance envelope which must
be unwrapped before validation. A harness that validates the envelope reports
precision exactly 1.000 on every dataset, and the failure is silent.
Report it. A mismatch between this code and the paper is a finding and we would
rather hear it. Do not edit expected.py to make a run agree — that file is the
paper's claim, not a tuning parameter. Open an issue with the failing line, your
env/hardware.txt, and your Python version.
- Code: https://github.com/kon0925/argos-vldb
- Benchmark datasets: https://doi.org/10.5281/zenodo.20072522 (CC-BY-4.0)
- Archived code release:
10.5281/zenodo.XXXXXXX
See CITATION.cff.
MIT — see LICENSE. Inputs carry their own terms; see docs/DATA.md.