test(oils-spec): measure upstream Oils spec pass rate against real bash

AGENTS.md

@@ -42,6 +42,7 @@ Fix root cause. Unsure: read more code; if stuck, ask w/ short options. Unrecogn
 | runtimes/python-builtin | Embedded Python via Monty, security, resource limits |
 | runtimes/cpython-wasm | Real CPython 3.14 on WASI (Wizer snapshot, Pulley AOT), WASI-on-VFS host, limits |
 | operations/eval | LLM eval study on the mira framework, dataset format, scoring |
+| operations/oils-spec | Upstream Oils spec suite pass rate vs real bash (`just oils-spec`) |
 | operations/maintenance | Pre-release maintenance requirements |
 | runtimes/python-package | Python package, PyPI wheels, platform matrix |
 | integrations/scripted-tool-orchestration | Compose ToolDef+callback pairs into OrchestratorTool via bash scripts |

THIRD_PARTY_LICENSES/README.md

@@ -16,7 +16,11 @@ Bashkit's design or whose test case formats have inspired our testing approach.
 
 2. **Test cases are original.** While our testing methodology was inspired by
    projects like Oils, the actual test cases are written specifically for
-   Bashkit.
+   Bashkit. `scripts/oils-spec/` additionally runs the upstream Oils spec suite
+   (Apache-2.0, Andy Chu and contributors, https://github.com/oils-for-unix/oils)
+   as a measurement; it is fetched at a pinned commit at run time, not
+   vendored. Its helper shims in `scripts/oils-spec/bin/` are bash
+   re-implementations of Oils' Python helpers, written for Bashkit.
 
 3. **Dependencies are via Cargo.** Rust dependencies (like jaq-core and
    jaq-std for jq support) are included via standard Cargo dependency

justfile

@@ -224,6 +224,15 @@ bash-oracle *ARGS:
     cargo build -q -p bashkit-cli
     python3 scripts/bash-oracle/run.py {{ARGS}}
 
+# Oils spec pass rate: the upstream Oils spec suite (bash column, fetched at a
+# pinned commit) through bashkit and real bash. Saves JSON/Markdown under
+# scripts/oils-spec/results/ and refreshes /benches data. Ad-hoc subsets:
+# python3 scripts/oils-spec/run.py SPEC... --fails FILE (not saved).
+oils-spec:
+    cargo build -q -p bashkit-cli --features realfs
+    python3 scripts/oils-spec/run.py --save
+    pnpm --dir site run data:performance
+
 # Debian-oracle scoreboard (pseudo-linus bench, fetched at a pinned commit).
 # Fails on a regression below scripts/debian-oracle/floors.tsv; pass
 # --update-floors after a fix raises a score. Optional TOOL args narrow the run.

knowledge/log.md

@@ -2,6 +2,7 @@
 
 ## 2026-10-08
 
+* **Test**: Oils spec pass rate. `just oils-spec` (`scripts/oils-spec/run.py`) fetches the upstream Oils spec suite at a pinned commit (`57d3f0d088c3`, not vendored), runs every case of the 132 bash-column spec files through bashkit (`-c`, `realfs` mounts) and real bash (the Oils way), and scores the bash column of each case with Oils' own assertion rules (`## OK/N-I/BUG bash` overrides win). First run: bashkit passes 1997 of the 2693 cases real bash 5.2.21 passes, **74.2%** (2003/2759 raw); bash itself passes 97.6%. Results go to `scripts/oils-spec/results/` and the `/benches` page. Python 2 helpers (`argv.py`, ...) are re-implemented in bash under `scripts/oils-spec/bin/`. See operations/oils-spec.
 * **Fix**: process substitution expands to bash's `/dev/fd/63`, counting down per command (`echo <(true) <(true)` is `/dev/fd/63 /dev/fd/62`), skipping fds still open (a caller's substitution, `exec 63>f`), and reusing 63 once the command, or the compound command whose words or redirects opened it, finishes. The fds live in a per-interpreter namespace (`ProcSubFs` wraps each shell's VFS view, TM-ISO-028) instead of `/dev/fd/proc_sub_<n>` files in the shared VFS, so tenants on one filesystem each get their own `/dev/fd/63`; a closed one is "No such file or directory". `set -C` no longer refuses `> >(cmd)` (a FIFO is not clobbered). Remaining gaps in [Limitations](operations/limitations.md). bash-oracle match 103 to 104, strict 101 to 102.
 * **Fix**: coprocesses run concurrently. `coproc NAME cmd` is a background job wired to the shell through two bounded in-process pipes: writes to `${NAME[1]}` feed its stdin, `exec {NAME[1]}>&-` delivers EOF, `read <&${NAME[0]}` / `read -u` take its output line by line, so `coproc UPPER { tr a-z A-Z; }` and request/response loops (`echo q >&${C[1]}; read r <&${C[0]}`) work; the body used to run once at `coproc` with empty input. fds follow bash (63/60, then 62/58), `$NAME_PID`/`$!`/`wait $NAME_PID` give the job, a second coproc while one runs prints bash's `still exists` warning, NAME and NAME_PID are unset and the fds closed once `wait` reaps it, and `exec()` closes the shell's ends before waiting for jobs. `<&$empty` is now an error instead of reading the shell's stdin. See foundations/parallel-execution "Coprocesses", L-PROC-005. bash-oracle match 102 to 103, strict 100 to 101.
 * **Fix**: `$SHLVL` follows bash's fork suppression. bash execs some simple commands in place: the last one of a `bash -c` string or `$( )` body (also the right side of a trailing `&&`/`||`/`;`; no redirections, no EXIT/ERR or signal trap set), the last one of a `( )`/`<( )` body (a sole one even with redirections), and any simple command run with `&`; a shell started there lowers `$SHLVL` first and so keeps the level. `exec bash` lowers it too, except directly inside `( )`; a substitution inside a pipeline stage does not. `bash -c 'bash -c "echo \$SHLVL"'` printed 3 in a script started at 1, now 2. A `bash`/`sh` child no longer inherits trap handlers (an ignored signal stays ignored), so `bash -c 'trap "echo bye" EXIT; bash -c true'` prints `bye` once. L-PROC-003 narrowed. bash-oracle match 101 to 102, strict 99 to 100.

knowledge/operations/index.md

@@ -7,4 +7,5 @@
 * [Release Process](release-process.md) - Versioning, validation, tagging, and publication to crates.io, PyPI, and npm.
 * [Performance Results](performance-results.md) - Benchmark harnesses, result locations, naming, and publication contract.
 * [Dependency Policy](dependencies.md) - Which third-party crates Bashkit keeps, which it trims, and which are deliberately not reimplemented in-house.
+* [Oils Spec Pass Rate](oils-spec.md) - Upstream Oils spec suite (bash column) run through bashkit and real bash, one headline pass rate per run.
 * [Evaluation Framework](eval.md) - LLM evaluation study design, dataset format, execution, and scoring.

knowledge/operations/oils-spec.md

@@ -0,0 +1,98 @@
+---
+type: Test Strategy
+title: Oils Spec Pass Rate
+description: Upstream Oils spec suite (bash column) run through bashkit and real bash, one headline pass rate per run.
+tags:
+  - bashkit
+  - testing
+  - compatibility
+  - oils
+---
+
+# Oils Spec Pass Rate
+
+## Status
+
+Implemented. Harness: `scripts/oils-spec/run.py` (`just oils-spec`).
+
+## Headline
+
+**74.2%**: bashkit passes 1997 of the 2693 Oils spec cases real bash 5.2.21
+passes (2759 cases in 132 spec files; bashkit passes 2003 of all cases,
+72.6%). Oils `57d3f0d088c3`, bashkit `f86bd7c81ae3`, 2026-10-08, 27 s with 4
+jobs. Report: `scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md`.
+
+Spec files with the most misses at that run: `builtin-completion` (34),
+`array` (25), `prompt` (25), `builtin-trap-bash` (19), `globignore` (16),
+`word-split` (16), `var-ref` (15), `builtin-cd` (14), `alias` (13),
+`dbracket` (13). 22 of the 132 files pass completely.
+
+## Why
+
+The in-repo spec cases (`crates/bashkit/tests/spec_cases/`) are written for
+Bashkit in an Oils-like format, so their pass rate measures what was chosen to
+be tested. The whole upstream Oils suite was written to pin bash, dash, mksh,
+zsh and OSH behavior, not bashkit's, so its pass rate is an outside measure of
+"closer to real bash" that moves with each PR (roadmap step 5, "Measure real
+workloads").
+
+## Decisions
+
+- **Fetched, not vendored.** Oils is Apache-2.0 but the suite is large and
+  upstream-owned. `run.py` pins `OILS_REV` (Oils pushes no release tags to
+  GitHub, so a master commit) and shallow-fetches it into
+  `target/oils-spec/oils` (`--cache` / `OILS_SPEC_CACHE` override). Bump the pin
+  deliberately and save a new result in the same change.
+- **Bash column only.** Files whose `## compare_shells` lists `bash` (or
+  `bash-4.4`) run; `## suite: disabled` and OSH/YSH-only files are skipped. The
+  parser ports Oils' `test/sh_spec.py` tokenizer (comment lines are dropped even
+  inside code and `STDOUT` blocks, as Oils runs them) and builds assertions as
+  it does for a `bash` label: `## OK|N-I|BUG bash` values win over the generic
+  `stdout`/`stderr`/`status`; unasserted status means 0; stderr is compared only
+  when asserted.
+- **Denominator is real bash.** Real bash runs each case the Oils way (code on
+  stdin, `SH=bash`, `TMP`, `REPO_ROOT`, `LC_ALL=C.UTF-8`, fresh cwd).
+  Cases real bash itself fails (no `python2` on the runner, root making `-r`/`-w`
+  true, bash version drift past 5.2, missing locales: 66 at the first run) are
+  out of the headline, so the number is about bashkit, not the runner.
+  `--no-bash` skips bash and reports bashkit over all cases.
+- **bashkit invocation.** `bashkit -c CODE bash` (so `$0` is `bash`, as with
+  stdin), stdin closed, `--timeout`. The CLI imports no host env, so
+  `export SH=bash TMP=/tmp REPO_ROOT=/oils ...;` is glued onto the first code
+  line, keeping `$LINENO` aligned. The Oils checkout and the helper shims are
+  mounted read-only, so the CLI must be built with `realfs`
+  (`cargo build -p bashkit-cli --features realfs`); the harness refuses a
+  binary without `--mount-ro`.
+- **Helper shims.** Oils' `spec/bin/argv.py`, `printenv.py`,
+  `stdout_stderr.py`, `read_from_fd.py` are Python 2. `scripts/oils-spec/bin/`
+  re-implements them in bash with identical output (argv.py renders a Python 2
+  list repr from `od` bytes) and both shells use them; bashkit runs no host
+  Python. A shim that bashkit mis-executes counts against bashkit, which is
+  fair: it is ordinary bash.
+- **Failure classes.** Each bashkit miss gets a keyword class: `signals`,
+  `processes`, `interactive` (completion, history, prompts, bind), `host`, or
+  `behavior`. The heuristic separates gaps bashkit keeps by design (see
+  [Known Limitations](limitations.md)) from plain behavior differences; all of
+  them still count in the rate. First run: behavior 530, interactive 110,
+  host 27, processes 19, signals 10.
+- **No floor, no CI gate yet.** Unlike the bash-oracle and Debian-oracle
+  scoreboards ([Testing Strategy](testing.md)), this is a headline metric:
+  bash-version drift on the runner moves the denominator. Saved results carry
+  the per-file missed-case list, so two runs diff case by case.
+
+## Results
+
+`just oils-spec` builds the CLI with `realfs`, runs the whole suite and saves
+`scripts/oils-spec/results/oils-spec-<platform>-<UTC stamp>.{json,md}`
+(see [Performance Results](performance-results.md)). `run.py --fails FILE`
+writes code, expected and actual output for every miss; positional spec names
+(`run.py arith var-sub`) run a subset, which is never saved. The `/benches`
+page shows the latest rate, the 12 spec files with the most misses and the
+run history. `scripts/tests/test_oils_spec.py` covers the parser rules, the
+scoring and the `argv.py` shim.
+
+## See also
+
+- [Testing Strategy](testing.md) - bash-oracle and Debian-oracle scoreboards
+- [Performance Results](performance-results.md) - result locations
+- [Known Limitations](limitations.md) - by-design gaps behind the failure classes

knowledge/operations/performance-results.md

@@ -28,6 +28,7 @@ directories:
 | Criterion benches | `crates/bashkit/benches/results/` | `criterion-*.md` |
 | `bashkit-eval` (archived) | `crates/bashkit-eval/results/` | `eval-*.json`, `scripting-eval-*.json`, plus matching `.md` reports |
 | `bashkit-replay` gap telemetry | `crates/bashkit-eval/results/gaps/` | `gaps-*.json` plus matching `gaps-*.md` |
+| Oils spec pass rate | `scripts/oils-spec/results/` | `oils-spec-*.json` plus matching `oils-spec-*.md` |
 
 Markdown files are the user-facing reports linked from `/benches`; JSON files
 are the aggregation input for benchmark and eval summaries.
@@ -43,7 +44,7 @@ are the aggregation input for benchmark and eval summaries.
 
 Default benchmark recipes that represent a real run MUST save artifacts in the
 directories above: `just bench`, `just bench-parallel`, `just bench-sqlite`, `just bench-python`,
-`just gaps`.
+`just gaps`, `just oils-spec`.
 
 The comparison harness resolves `bash` from `PATH` and requires Bash 4 or newer
 (case conversion and associative arrays are benchmarked). A missing or older
@@ -87,5 +88,11 @@ start table, warm `python_call/<runtime>/print`, and CPython
 point (CPython vs Monty) and the CPython history. `just bench-python` produces
 these reports.
 
+The transformer also emits `oilsSpec`: `runs` (one point per
+`oils-spec-*.json`: pass rate, bashkit passes among bash passes, Oils and
+bashkit revisions) and `latest` (that report plus its 12 spec files with the
+most misses). `/benches` renders both after the gap telemetry panels. See
+[Oils Spec Pass Rate](oils-spec.md).
+
 When changing result schemas, update the transformer and this spec in the same
 PR. Do not hand-edit `performance-timeline.json` except by running the script.

knowledge/operations/testing.md

@@ -384,6 +384,14 @@ process substitution, parameter expansion errors.
   representing a documented gap stays a gap rather than being deleted, so the
   corpus keeps measuring it (see [limitations](limitations.md)).
 
+## Oils Spec Pass Rate
+
+`scripts/oils-spec/run.py` (`just oils-spec`) runs the whole upstream Oils
+spec suite (bash column, ~2,760 cases) through bashkit and real bash and
+publishes one headline number, bashkit passes over real-bash passes. It is a
+measurement, not a gate: no floor, no CI job. See
+[Oils Spec Pass Rate](oils-spec.md).
+
 ## SQLite CLI Differential Tests
 
 CSV comparisons explicitly set the host CLI row separator to LF. SQLite

scripts/oils-spec/README.md

@@ -0,0 +1,50 @@
+# Oils spec pass rate
+
+Runs the upstream [Oils](https://github.com/oils-for-unix/oils) spec suite
+(Apache-2.0), bash column, through bashkit and through real bash, and reports
+one headline number:
+
+    pass rate = cases bashkit passes / cases real bash passes
+
+```bash
+just oils-spec                         # build CLI (realfs), run all, save results
+python3 scripts/oils-spec/run.py arith var-sub --fails /tmp/fails.txt
+python3 scripts/oils-spec/run.py --no-bash   # bashkit over all cases, no bash
+```
+
+`run.py` fetches Oils at the pinned `OILS_REV` into `target/oils-spec/oils`
+(override with `--cache` or `OILS_SPEC_CACHE`); the test files are never
+vendored. The bashkit CLI must be built with the `realfs` feature
+(`cargo build -p bashkit-cli --features realfs`), because the Oils checkout and
+`bin/` are mounted read-only into the sandbox.
+
+## How a case is scored
+
+The parser follows Oils' `test/sh_spec.py`: `#### name` starts a case,
+`## stdout:` / `## STDOUT: ... ## END` / `## stdout-json:` / `## status:` /
+`## stderr:` assert, and `## OK bash`, `## N-I bash`, `## BUG bash` overrides
+win over the generic values. Status defaults to 0; stderr is checked only when
+asserted. Files without `bash` in `## compare_shells`, and `## suite: disabled`
+files, are skipped.
+
+- Real bash runs the Oils way: code on stdin, `SH=bash`, `TMP`, `REPO_ROOT`,
+  `LC_ALL=C.UTF-8`, a fresh temp cwd. Cases it fails (for example ones that
+  call `python2`) leave the denominator.
+- bashkit runs `bashkit -c CODE bash` with stdin closed. The same variables are
+  exported by a prefix glued onto the first code line, so `$LINENO` matches.
+- `bin/` holds bash stand-ins for Oils' Python 2 helpers (`argv.py`,
+  `printenv.py`, `stdout_stderr.py`, `read_from_fd.py`, `foo=bar`), with the
+  same output. Both shells use them.
+- Each bashkit miss is classed `signals`, `processes`, `interactive`, `host` or
+  `behavior` by keyword, so by-design gaps stand apart. All still count.
+
+## Results
+
+`--save` (what `just oils-spec` passes) writes
+`results/oils-spec-<platform>-<UTC stamp>.json` and `.md`: totals, failure
+classes, and per spec file the counts plus every case bashkit misses, so two
+runs diff case by case. A subset run is never saved. The `/benches` site page
+shows the latest headline and history.
+
+See `knowledge/operations/oils-spec.md` for the decisions and the current
+number.

scripts/oils-spec/bin/argv.py

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Shell stand-in for Oils' spec/bin/argv.py (a Python 2 script): prints the
+# arguments as a Python 2 list repr, byte for byte. Written in bash so the same
+# helper runs under real bash and inside the bashkit sandbox, which does not
+# execute host Python. Bytes come from od, so non-UTF-8 and multibyte
+# arguments render as Python 2 would ('\xce\xbc').
+out='['
+sep=''
+for arg in "$@"; do
+  q="'"
+  if [[ $arg == *"'"* && $arg != *'"'* ]]; then
+    q='"'
+  fi
+  s=''
+  for h in $(printf '%s' "$arg" | od -An -v -tx1); do
+    n=$((16#$h))
+    if ((n == 92)); then
+      s+='\\'
+    elif ((n == 39)) && [[ $q == "'" ]]; then
+      s+="\\'"
+    elif ((n == 9)); then
+      s+='\t'
+    elif ((n == 10)); then
+      s+='\n'
+    elif ((n == 13)); then
+      s+='\r'
+    elif ((n < 32 || n > 126)); then
+      s+="\\x$h"
+    else
+      s+=$(printf "\\x$h")
+    fi
+  done
+  out+="$sep$q$s$q"
+  sep=', '
+done
+printf '%s]\n' "$out"

scripts/oils-spec/bin/foo=bar

@@ -0,0 +1,2 @@
+#!/bin/sh
+echo HI

scripts/oils-spec/bin/printenv.py

@@ -0,0 +1,10 @@
+#!/usr/bin/env bash
+# Shell stand-in for Oils' spec/bin/printenv.py: one line per name, the
+# exported value or None.
+for name in "$@"; do
+  if [[ $name =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] && [[ -v $name ]]; then
+    printf '%s\n' "${!name}"
+  else
+    echo None
+  fi
+done

scripts/oils-spec/bin/read_from_fd.py

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Shell stand-in for Oils' spec/bin/read_from_fd.py: for each fd, prints
+# "<fd>: " followed by what can be read from it.
+for fd in "$@"; do
+  if ! { true <&"$fd"; } 2>/dev/null; then
+    echo "FATAL: Error reading from fd $fd" >&2
+    exit 1
+  fi
+  printf '%d: ' "$fd"
+  cat <&"$fd"
+done

scripts/oils-spec/bin/stdout_stderr.py

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Shell stand-in for Oils' spec/bin/stdout_stderr.py: prints $1 (STDOUT) to
+# stdout, $2 (STDERR) to stderr, exits with $3 (0).
+printf '%s\n' "${1-STDOUT}"
+printf '%s\n' "${2-STDERR}" >&2
+exit "${3-0}"