Repository navigation
test(oils-spec): measure upstream Oils spec pass rate against real bash - #2640
Merged
Merged
Conversation
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | 8f40073 | Commit Preview URL Branch Preview URL |
Oct 08 2026, 11:20 AM |
chaliy
force-pushed
the
claude/oils-spec-rate
branch
from
October 8, 2026 11:07
d565387 to
ea41cbf
Compare
Add scripts/oils-spec/run.py (`just oils-spec`): fetches the Oils spec suite at a pinned commit (not vendored), runs every case of the 132 bash-column spec files through bashkit and real bash, scores the bash column with Oils' own assertion rules, and saves JSON/Markdown results under scripts/oils-spec/results/. Headline = bashkit passes among the cases real bash passes. First run: 74.2% (1997/2693); bash itself passes 2693/2759. Bash stand-ins for Oils' Python 2 helpers live in scripts/oils-spec/bin. The /benches page shows the latest rate, worst spec files and history. Knowledge: operations/oils-spec, performance-results, testing, log.
chaliy
force-pushed
the
claude/oils-spec-rate
branch
from
October 8, 2026 11:19
ea41cbf to
8f40073
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Михайло · project thread
What changed
scripts/oils-spec/run.py(stdlib Python): fetches the upstream Oils spec suite at a pinned commit (57d3f0d0, Apache-2.0, shallow-fetched at run time, not vendored), parsesspec/*.test.shwith Oils' ownsh_spec.pyassertion rules for the bash column, and runs every case against bashkit and real bash.scripts/oils-spec/results/oils-spec-<platform>-<stamp>.{json,md}(JSON lists every missed case, so runs diff case by case).just oils-specbuilds the CLI withrealfs, runs the suite, saves results, refreshes site data./benchesgets an Oils panel (latest rate, worst files, history).argv.pyetc.) underscripts/oils-spec/bin/; harness tests inscripts/tests/test_oils_spec.py.Why
Level-up roadmap, Step 5 "Measure real workloads": one headline number for "closer to real bash" that moves with each PR.
Before / After (verified)
First run (bash 5.2.21, 132 spec files that list bash, ~27 s with 4 jobs):
Most misses: builtin-completion 34, array 25, prompt 25, builtin-trap-bash 19, globignore 16, word-split 16. Full list in the committed results
.md.Risk
[ -r ], newer-bash features, locales) are excluded from the rate.Checklist
scripts/tests(158 tests),py_compile,cargo fmt --check,just check-okf,just check-doc-links, sitepnpm run buildoperations/oils-spec.md, performance-results, testing, log; THIRD_PARTY_LICENSES noteGenerated by Claude Code