Repository navigation
Commit 14da03a
## Summary
This fixes the 12 test failures that #47 tracks. Each fix is in `src/`
at the root cause, or in a test that called an API that never existed.
No test is skipped, deleted or loosened.
| # | Failure | Root cause | Fix |
|---|---|---|---|
| 1 | Statistics Parsing / Parse numeric statistics,
`runtests.jl:876–881` (3 fail + 3 `KeyError`) | `parse_statistics`
matched only a single-pair list `(:key value)` that closes at once.
Solver statistics come as one list of many pairs, and Z3 prints it over
several lines. | It now finds each `(:` keyword list and reads every
`:key value` pair in it. Quoted strings are consumed first, so a `(:…)`
inside an error message is not read as a list. Against real Z3 4.13.3
`(get-info :all-statistics)` output it returns 13 entries. |
| 2 | evaluate / Boolean logic, `runtests.jl:931–932` | Julia parses `p
&& q` and `p \|\| q` as `Expr(:&&)` / `Expr(:\|\|)`, not as calls. The
`:call` branch never ran, so the substituted `Expr` came back. |
`evaluate` computes `&&` and `\|\|` once every operand is a `Bool`. If
any operand is still unknown, it returns the substituted `Expr`. |
| 3 | Unsat core extraction (Z3), `runtests.jl:1153` | Two defects.
First, `set_option!(ctx, :produce_unsat_cores, true)` sent the option
name with underscores, which is not the SMT-LIB name (cvc5 1.1.2 answers
`unsupported`). Second, plain `check_sat(ctx)` never sent
`(get-unsat-core)`, even with the option on. | The `Symbol` method of
`set_option!` maps `_` to `-`. `check_sat`'s `get_unsat_core` now
defaults to the context's `produce-unsat-cores` option. An explicit
`get_unsat_core=true/false` behaves as before. Z3 now returns `[:pos,
:neg]`. |
| 4 | e2e `declare → assert → check_sat`, `e2e_test.jl:23`; e2e
`to_smtlib_script`, `e2e_test.jl:41`; property `assert!` count,
`property_test.jl:31` (3 `UndefVarError`) | **These tests were wrong.**
`declare_const` and `to_smtlib_script` have never existed in `src/`.
Commit `ed8eff1` added them only as calls, in the tests and in
`benches/benchmarks.jl`. | The tests and benches now use the exported,
documented API: `declare(ctx, name, sort)` (README: "`declare(ctx, name,
sort)` — Declare a constant or function") and `SMTLib.build_script(ctx,
get_model)`. The "Script Building" testset already exercises
`build_script`. |
**Why fix 4 changes the tests rather than adding `declare_const`.** #47
leaves the choice open. Adding the two functions would give the package
a second name for `declare` and a public wrapper around an internal
function, and only test code from a single commit would use them.
Keeping `declare` as the one public way to declare a constant is the
simpler choice. The alternative, adding and exporting
`declare_const(ctx, name, ::Symbol)` and `to_smtlib_script`, is easy to
do if the owner prefers it.
**No assertion is weakened, and two are stronger:**
- *"unknown sort raises an error"* passed vacuously before. The
`UndefVarError` for `declare_const` satisfied `@test_throws Exception`.
The test now asserts that an unmapped type (`Vector{Int}`) raises a
`MethodError` and that no declaration is recorded. I checked that
`declare(ctx, :z, Vector{Int})` really throws `MethodError: no method
matching smt_type(::Type{Vector{Int64}})`.
- The pipeline test now also checks that exactly three `(assert` forms
are emitted, as its own comment says.
`CHANGELOG.adoc` gains a `=== Fixed` section under `[Unreleased]`.
**Second commit (`318efef`), from CodeRabbit's review.** A quoted string
that is not a pair's value could leak a false pair. On `6604a9c`, `(:a 1
"stray :time 99")` gave `time => 99`; I reproduced this. CodeRabbit's
own example, `(:reason "message :time 99")`, was already parsed
correctly, because there the string is a value and is consumed whole.
The pair search now consumes any stray string whole as well.
The commit adds 9 regression tests:
- Z3's multi-line layout;
- quoted values containing parentheses and `:key value` text;
- the stray string;
- an error message containing `(:foo bar)`;
- a keyword value.
It also replaces the tautological `isempty(stats) || !isempty(stats)` in
"Empty output" with `isempty(stats)`.
**Third commit (`a6c6bc6`): make the CI matrix able to run the tests.**
#47's last acceptance criterion is *"`Pkg.test()` passes on the CI
matrix (1.10, 1.11, nightly × ubuntu/macos/windows)"*. Two things
stopped that, on `main` and on this branch before this commit:
- Every `test (*)` job stopped at **Build package** with `expected
package AcceleratorGate [59f742c7] to be registered`. `julia-buildpkg`
adds only General, and AcceleratorGate is in
`hyperpolymath/julia-professional-registry`. `CI.yml` now passes that
registry to `julia-buildpkg` through its existing `localregistry` input;
the test step reuses the same depot. The registry needs no credentials:
#50's Release run passed its "Add registries" step on a hosted runner.
- CI installs neither z3 nor cvc5. Four testsets (the three e2e
`SMTContext` testsets and the property `assert!` count) only build
SMT-LIB text, but they called `SMTContext()`, which looks a solver up on
`PATH` and raises `No SMT solver found. Install z3 or cvc5.` These calls
were already on `main`. They now pass `SMTContext` a solver record that
is never run (`SMTSolver(:z3, "/nonexistent/z3", "never run")`).
`declare`, `assert!` and `build_script` never read the context's solver;
only `run_solver` does. The first of them is titled "(no solver)" and
its comment says it "must not error even without a solver installed", so
this makes the test do what it says.
Closes #47. All nine `CI / test (*)` jobs on this head (`a6c6bc6`) are
green: run 37912491992. Eight passed on the first attempt. `test
(nightly, macos-latest)` first failed at **Setup Julia**, before any
package code ran, with `getaddrinfo ENOTFOUND
julialangnightlies-s3.julialang.org`. That is a runner DNS failure; the
host resolved and answered 200 from here. The job was re-run unchanged
and passed (job 113764403648, 2026-10-09T09:50Z). 1.11, nightly, macOS
and Windows are verified by CI alone; I ran 1.10 and 1.12 on Linux.
## Type of change
- [x] 🐛 Bug fix (non-breaking change that fixes an issue)
- [ ] ✨ New feature (non-breaking change that adds functionality). No
new public function.
- [ ] 💥 Breaking change (would change existing behaviour). There are two
small behaviour changes, both towards SMT-LIB-correct output. A `Symbol`
option key now has `_` mapped to `-`; the old underscore name was not a
valid SMT-LIB option name. Plain `check_sat` now requests the unsat core
when the context already enables `produce-unsat-cores`. String keys and
explicit keyword arguments are unchanged.
- [ ] 🕳️ Soundness fix. Not a checker or proof false-negative.
- [ ] 📖 Documentation. The only doc changes are docstrings and the
CHANGELOG entry.
- [ ] 🧹 Refactor / tech debt
- [ ] ⚡ Performance
- [x] 🔧 Build / CI / tooling. `CI.yml`: `julia-buildpkg` gains the
`localregistry` input (third commit).
## 📌 New pins
Head SHA: **a6c6bc6effde7ddb5cd5b4f8d173bd671e69a4c4**
No pins added or changed. The `julia-buildpkg` pin (`e3eb439f…`, v1.7.0)
is unchanged; only its `with:` input is new. There are no `actions.lock`
changes (the repo has none) and no `Project.toml` compat changes.
`Manifest.toml` is gitignored. `CI.yml` stays in block YAML, because
this repo's gates do not yet meet the KYAML preconditions.
## How has this been verified?
I ran `Pkg.test()` from a throwaway depot (General plus
`hyperpolymath/julia-professional-registry`), with z3 4.13.3 and cvc5
1.1.2 on `PATH`, so the solver-dependent testsets, including the Z3
unsat-core test, actually ran:
| Julia | `main` @ 487c7ab | 6604a9c (first commit) | **318efef (second
commit)** |
|---|---|---|---|
| 1.12.6 | 818 pass / 6 fail / 6 error (830) | 864 pass (864) | **873
pass** (873) |
| 1.10.10 | 818 pass / 6 fail / 6 error (830) | 864 pass (864) | **873
pass** (873) |
| 1.11 | not run, not installed on this machine | not run | not run |
From `main` to the first commit, the total rises by 34:
- The property test now runs all 30 iterations instead of erroring on
the first (+29).
- The e2e pipeline testset runs 4 checks instead of 1 error, including
the new assert count (+3).
- The e2e `build_script` testset runs 2 checks instead of 1 error (+1).
- The unknown-sort testset gains the "no declaration recorded" check
(+1).
The second commit adds 9 tests to "Statistics Parsing" (864 → 873). They
are described below.
I also checked `parse_statistics` directly against these inputs:
- real Z3 multi-line statistics (13 keys; `time` → `0.0`, `max-memory` →
`17.82`, `num-allocs` → `35972`);
- the test string;
- a quoted value containing parentheses (`(:name "Z3 (x)")`);
- a keyword value (`(:reason-unknown :incomplete)`);
- an error message containing `(:foo bar)`, which correctly yields
nothing.
The new `check_sat` default has one path that no test covers. When
`produce-unsat-cores` is on and the problem is satisfiable, plain
`check_sat` now also sends `(get-unsat-core)`. I ran this case directly
on both solvers:
- **z3:** `status=sat`, model `x => 6`, followed by `(error "… unsat
core is not available")`.
- **cvc5:** `status=sat`, model `x => 6`, followed by `(error "Cannot
get unsat core unless in unsat mode.")`.
In both cases the status and the model parse correctly.
**The CI condition: no solver on `PATH`.** For the third commit I also
ran `Pkg.test()` with a `PATH` that has neither z3 nor cvc5 (`command -v
z3 cvc5` finds nothing; `Sys.which` is the package's lookup):
| Julia, no solver | 318efef | **a6c6bc6 (head)** |
|---|---|---|
| 1.10.10 | 718 pass / 4 error (722), rc=1 | **757 pass** (757), rc=0 |
| 1.12.6 | not run | **757 pass** (757), rc=0 |
The 4 errors were the `SMTContext()` calls described above. 757 = 718 +
39: the four testsets now run their 5 + 2 + 2 + 30 checks. With z3 and
cvc5 on `PATH`, the head still gives **873 pass** (873) on 1.10.10, the
same as `318efef`.
CI note: the solver-dependent testsets ("Solver Integration", the Z3
optimisation and unsat-core tests) log `No SMT solver found, skipping …`
and run only where a solver is on `PATH`, so CI does not exercise them.
Installing a solver in CI is a separate change.
## Checklist
- [x] My commits are **signed**. All three commits are SSH-signed: `git
log --format=%G?` gives `G` for each. GitHub reported `verified: true`
(`valid`) for the first two, and the GraphQL commit `signature` reports
`VALID` (signer `hyperpolymath`) for the third.
- [x] I ran the project's own checks/tests locally and they pass.
`Pkg.test()` passes on 1.12.6 and 1.10.10, with and without solvers, as
above. 1.11 was not run. `actionlint` passes on `CI.yml`.
- [ ] New files carry the correct `SPDX-License-Identifier`. No new
files; every touched file keeps its existing header.
- [x] Docs are updated, and no public claim now overstates what the code
does. The docstrings for `parse_statistics`, `evaluate`, `check_sat` and
both `set_option!` methods describe the new behaviour. The two
`set_option!` methods had no docstring before.
- [x] I have not introduced a soundness hole (or I have flagged where I
might have). The `check_sat` default only adds a `(get-unsat-core)`
request when the user has turned unsat cores on.
## Notes for reviewers
- **Decision for the owner:** #47 offers two paths for fix 4. This PR
keeps `declare` as the only public way to declare a constant (see
above). Say if you want `declare_const` added instead.
- **Findings, not fixed here (out of scope):**
- CI installs no SMT solver, so the solver-dependent testsets are
skipped there (see the CI note).
- `parse_unsat_core` reads one line. cvc5 1.1.2 prints the core one
label per line (`(` / `pos` / `neg` / `)`), so cvc5 cores come back
empty. The test is Z3-only, so this is not one of the 12 failures.
- cvc5's statistics format, `(:all-statistics (("name" v) …))`, is a
nested list. `parse_statistics` deliberately skips it rather than
producing wrong values.
- The estate docstring scanner (`standards/.githooks/docstring-scan.sh`)
skips `.jl` files, so it reported no verdict (`skipped=4`). I checked
the docstrings by hand.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016XJoh8YA2sCWQmq93CF8RX
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 487c7ab commit 14da03a
7 files changed
Lines changed: 128 additions & 30 deletions
File tree
- .github/workflows
- benches
- src
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| 28 | + | |
| 29 | + | |
28 | 30 | | |
29 | 31 | | |
| 32 | + | |
| 33 | + | |
30 | 34 | | |
31 | 35 | | |
32 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
20 | 29 | | |
21 | 30 | | |
22 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
37 | | - | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
969 | 969 | | |
970 | 970 | | |
971 | 971 | | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
972 | 983 | | |
973 | | - | |
| 984 | + | |
974 | 985 | | |
975 | 986 | | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
976 | 993 | | |
977 | 994 | | |
978 | 995 | | |
| |||
982 | 999 | | |
983 | 1000 | | |
984 | 1001 | | |
985 | | - | |
| 1002 | + | |
986 | 1003 | | |
987 | 1004 | | |
988 | 1005 | | |
| |||
993 | 1010 | | |
994 | 1011 | | |
995 | 1012 | | |
996 | | - | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
997 | 1016 | | |
998 | 1017 | | |
999 | 1018 | | |
| |||
1004 | 1023 | | |
1005 | 1024 | | |
1006 | 1025 | | |
1007 | | - | |
| 1026 | + | |
| 1027 | + | |
1008 | 1028 | | |
1009 | 1029 | | |
1010 | 1030 | | |
| |||
1234 | 1254 | | |
1235 | 1255 | | |
1236 | 1256 | | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
1237 | 1271 | | |
1238 | 1272 | | |
1239 | 1273 | | |
1240 | 1274 | | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
1241 | 1281 | | |
1242 | 1282 | | |
1243 | 1283 | | |
1244 | | - | |
1245 | | - | |
1246 | | - | |
1247 | | - | |
1248 | | - | |
1249 | | - | |
1250 | | - | |
1251 | | - | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
1252 | 1295 | | |
1253 | | - | |
1254 | 1296 | | |
1255 | 1297 | | |
1256 | 1298 | | |
| |||
2292 | 2334 | | |
2293 | 2335 | | |
2294 | 2336 | | |
| 2337 | + | |
| 2338 | + | |
| 2339 | + | |
| 2340 | + | |
| 2341 | + | |
| 2342 | + | |
2295 | 2343 | | |
2296 | 2344 | | |
2297 | 2345 | | |
2298 | 2346 | | |
2299 | 2347 | | |
2300 | 2348 | | |
2301 | 2349 | | |
| 2350 | + | |
| 2351 | + | |
| 2352 | + | |
| 2353 | + | |
2302 | 2354 | | |
2303 | 2355 | | |
2304 | 2356 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
10 | 15 | | |
11 | 16 | | |
12 | 17 | | |
| |||
22 | 27 | | |
23 | 28 | | |
24 | 29 | | |
25 | | - | |
| 30 | + | |
26 | 31 | | |
27 | 32 | | |
28 | | - | |
29 | | - | |
| 33 | + | |
| 34 | + | |
30 | 35 | | |
31 | 36 | | |
32 | 37 | | |
33 | 38 | | |
34 | 39 | | |
35 | | - | |
| 40 | + | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
| 44 | + | |
39 | 45 | | |
40 | 46 | | |
41 | | - | |
42 | | - | |
43 | | - | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
44 | 50 | | |
45 | | - | |
| 51 | + | |
46 | 52 | | |
47 | 53 | | |
48 | 54 | | |
49 | 55 | | |
50 | 56 | | |
51 | | - | |
52 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
53 | 61 | | |
54 | 62 | | |
55 | 63 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
10 | 15 | | |
11 | 16 | | |
12 | 17 | | |
| |||
30 | 35 | | |
31 | 36 | | |
32 | 37 | | |
33 | | - | |
34 | | - | |
| 38 | + | |
| 39 | + | |
35 | 40 | | |
36 | 41 | | |
37 | 42 | | |
38 | 43 | | |
39 | | - | |
| 44 | + | |
40 | 45 | | |
41 | 46 | | |
42 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
893 | 893 | | |
894 | 894 | | |
895 | 895 | | |
896 | | - | |
| 896 | + | |
| 897 | + | |
| 898 | + | |
| 899 | + | |
| 900 | + | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
| 908 | + | |
| 909 | + | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
897 | 917 | | |
898 | 918 | | |
899 | 919 | | |
| |||
0 commit comments