diff --git a/AGENTS.md b/AGENTS.md index c07ac31fe..04cc444fc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,6 +42,7 @@ Fix root cause. Unsure: read more code; if stuck, ask w/ short options. Unrecogn | runtimes/python-builtin | Embedded Python via Monty, security, resource limits | | runtimes/cpython-wasm | Real CPython 3.14 on WASI (Wizer snapshot, Pulley AOT), WASI-on-VFS host, limits | | operations/eval | LLM eval study on the mira framework, dataset format, scoring | +| operations/oils-spec | Upstream Oils spec suite pass rate vs real bash (`just oils-spec`) | | operations/maintenance | Pre-release maintenance requirements | | runtimes/python-package | Python package, PyPI wheels, platform matrix | | integrations/scripted-tool-orchestration | Compose ToolDef+callback pairs into OrchestratorTool via bash scripts | diff --git a/THIRD_PARTY_LICENSES/README.md b/THIRD_PARTY_LICENSES/README.md index bd1312bb8..2d0da3060 100644 --- a/THIRD_PARTY_LICENSES/README.md +++ b/THIRD_PARTY_LICENSES/README.md @@ -16,7 +16,11 @@ Bashkit's design or whose test case formats have inspired our testing approach. 2. **Test cases are original.** While our testing methodology was inspired by projects like Oils, the actual test cases are written specifically for - Bashkit. + Bashkit. `scripts/oils-spec/` additionally runs the upstream Oils spec suite + (Apache-2.0, Andy Chu and contributors, https://github.com/oils-for-unix/oils) + as a measurement; it is fetched at a pinned commit at run time, not + vendored. Its helper shims in `scripts/oils-spec/bin/` are bash + re-implementations of Oils' Python helpers, written for Bashkit. 3. **Dependencies are via Cargo.** Rust dependencies (like jaq-core and jaq-std for jq support) are included via standard Cargo dependency diff --git a/justfile b/justfile index 572dd6deb..75b665658 100644 --- a/justfile +++ b/justfile @@ -224,6 +224,15 @@ bash-oracle *ARGS: cargo build -q -p bashkit-cli python3 scripts/bash-oracle/run.py {{ARGS}} +# Oils spec pass rate: the upstream Oils spec suite (bash column, fetched at a +# pinned commit) through bashkit and real bash. Saves JSON/Markdown under +# scripts/oils-spec/results/ and refreshes /benches data. Ad-hoc subsets: +# python3 scripts/oils-spec/run.py SPEC... --fails FILE (not saved). +oils-spec: + cargo build -q -p bashkit-cli --features realfs + python3 scripts/oils-spec/run.py --save + pnpm --dir site run data:performance + # Debian-oracle scoreboard (pseudo-linus bench, fetched at a pinned commit). # Fails on a regression below scripts/debian-oracle/floors.tsv; pass # --update-floors after a fix raises a score. Optional TOOL args narrow the run. diff --git a/knowledge/log.md b/knowledge/log.md index 53c861c84..e773870c6 100644 --- a/knowledge/log.md +++ b/knowledge/log.md @@ -2,6 +2,7 @@ ## 2026-10-08 +* **Test**: Oils spec pass rate. `just oils-spec` (`scripts/oils-spec/run.py`) fetches the upstream Oils spec suite at a pinned commit (`57d3f0d088c3`, not vendored), runs every case of the 132 bash-column spec files through bashkit (`-c`, `realfs` mounts) and real bash (the Oils way), and scores the bash column of each case with Oils' own assertion rules (`## OK/N-I/BUG bash` overrides win). First run: bashkit passes 1997 of the 2693 cases real bash 5.2.21 passes, **74.2%** (2003/2759 raw); bash itself passes 97.6%. Results go to `scripts/oils-spec/results/` and the `/benches` page. Python 2 helpers (`argv.py`, ...) are re-implemented in bash under `scripts/oils-spec/bin/`. See operations/oils-spec. * **Fix**: process substitution expands to bash's `/dev/fd/63`, counting down per command (`echo <(true) <(true)` is `/dev/fd/63 /dev/fd/62`), skipping fds still open (a caller's substitution, `exec 63>f`), and reusing 63 once the command, or the compound command whose words or redirects opened it, finishes. The fds live in a per-interpreter namespace (`ProcSubFs` wraps each shell's VFS view, TM-ISO-028) instead of `/dev/fd/proc_sub_` files in the shared VFS, so tenants on one filesystem each get their own `/dev/fd/63`; a closed one is "No such file or directory". `set -C` no longer refuses `> >(cmd)` (a FIFO is not clobbered). Remaining gaps in [Limitations](operations/limitations.md). bash-oracle match 103 to 104, strict 101 to 102. * **Fix**: coprocesses run concurrently. `coproc NAME cmd` is a background job wired to the shell through two bounded in-process pipes: writes to `${NAME[1]}` feed its stdin, `exec {NAME[1]}>&-` delivers EOF, `read <&${NAME[0]}` / `read -u` take its output line by line, so `coproc UPPER { tr a-z A-Z; }` and request/response loops (`echo q >&${C[1]}; read r <&${C[0]}`) work; the body used to run once at `coproc` with empty input. fds follow bash (63/60, then 62/58), `$NAME_PID`/`$!`/`wait $NAME_PID` give the job, a second coproc while one runs prints bash's `still exists` warning, NAME and NAME_PID are unset and the fds closed once `wait` reaps it, and `exec()` closes the shell's ends before waiting for jobs. `<&$empty` is now an error instead of reading the shell's stdin. See foundations/parallel-execution "Coprocesses", L-PROC-005. bash-oracle match 102 to 103, strict 100 to 101. * **Fix**: `$SHLVL` follows bash's fork suppression. bash execs some simple commands in place: the last one of a `bash -c` string or `$( )` body (also the right side of a trailing `&&`/`||`/`;`; no redirections, no EXIT/ERR or signal trap set), the last one of a `( )`/`<( )` body (a sole one even with redirections), and any simple command run with `&`; a shell started there lowers `$SHLVL` first and so keeps the level. `exec bash` lowers it too, except directly inside `( )`; a substitution inside a pipeline stage does not. `bash -c 'bash -c "echo \$SHLVL"'` printed 3 in a script started at 1, now 2. A `bash`/`sh` child no longer inherits trap handlers (an ignored signal stays ignored), so `bash -c 'trap "echo bye" EXIT; bash -c true'` prints `bye` once. L-PROC-003 narrowed. bash-oracle match 101 to 102, strict 99 to 100. diff --git a/knowledge/operations/index.md b/knowledge/operations/index.md index 356435094..6eb7cc04c 100644 --- a/knowledge/operations/index.md +++ b/knowledge/operations/index.md @@ -7,4 +7,5 @@ * [Release Process](release-process.md) - Versioning, validation, tagging, and publication to crates.io, PyPI, and npm. * [Performance Results](performance-results.md) - Benchmark harnesses, result locations, naming, and publication contract. * [Dependency Policy](dependencies.md) - Which third-party crates Bashkit keeps, which it trims, and which are deliberately not reimplemented in-house. +* [Oils Spec Pass Rate](oils-spec.md) - Upstream Oils spec suite (bash column) run through bashkit and real bash, one headline pass rate per run. * [Evaluation Framework](eval.md) - LLM evaluation study design, dataset format, execution, and scoring. diff --git a/knowledge/operations/oils-spec.md b/knowledge/operations/oils-spec.md new file mode 100644 index 000000000..99168260a --- /dev/null +++ b/knowledge/operations/oils-spec.md @@ -0,0 +1,98 @@ +--- +type: Test Strategy +title: Oils Spec Pass Rate +description: Upstream Oils spec suite (bash column) run through bashkit and real bash, one headline pass rate per run. +tags: + - bashkit + - testing + - compatibility + - oils +--- + +# Oils Spec Pass Rate + +## Status + +Implemented. Harness: `scripts/oils-spec/run.py` (`just oils-spec`). + +## Headline + +**74.2%**: bashkit passes 1997 of the 2693 Oils spec cases real bash 5.2.21 +passes (2759 cases in 132 spec files; bashkit passes 2003 of all cases, +72.6%). Oils `57d3f0d088c3`, bashkit `f86bd7c81ae3`, 2026-10-08, 27 s with 4 +jobs. Report: `scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md`. + +Spec files with the most misses at that run: `builtin-completion` (34), +`array` (25), `prompt` (25), `builtin-trap-bash` (19), `globignore` (16), +`word-split` (16), `var-ref` (15), `builtin-cd` (14), `alias` (13), +`dbracket` (13). 22 of the 132 files pass completely. + +## Why + +The in-repo spec cases (`crates/bashkit/tests/spec_cases/`) are written for +Bashkit in an Oils-like format, so their pass rate measures what was chosen to +be tested. The whole upstream Oils suite was written to pin bash, dash, mksh, +zsh and OSH behavior, not bashkit's, so its pass rate is an outside measure of +"closer to real bash" that moves with each PR (roadmap step 5, "Measure real +workloads"). + +## Decisions + +- **Fetched, not vendored.** Oils is Apache-2.0 but the suite is large and + upstream-owned. `run.py` pins `OILS_REV` (Oils pushes no release tags to + GitHub, so a master commit) and shallow-fetches it into + `target/oils-spec/oils` (`--cache` / `OILS_SPEC_CACHE` override). Bump the pin + deliberately and save a new result in the same change. +- **Bash column only.** Files whose `## compare_shells` lists `bash` (or + `bash-4.4`) run; `## suite: disabled` and OSH/YSH-only files are skipped. The + parser ports Oils' `test/sh_spec.py` tokenizer (comment lines are dropped even + inside code and `STDOUT` blocks, as Oils runs them) and builds assertions as + it does for a `bash` label: `## OK|N-I|BUG bash` values win over the generic + `stdout`/`stderr`/`status`; unasserted status means 0; stderr is compared only + when asserted. +- **Denominator is real bash.** Real bash runs each case the Oils way (code on + stdin, `SH=bash`, `TMP`, `REPO_ROOT`, `LC_ALL=C.UTF-8`, fresh cwd). + Cases real bash itself fails (no `python2` on the runner, root making `-r`/`-w` + true, bash version drift past 5.2, missing locales: 66 at the first run) are + out of the headline, so the number is about bashkit, not the runner. + `--no-bash` skips bash and reports bashkit over all cases. +- **bashkit invocation.** `bashkit -c CODE bash` (so `$0` is `bash`, as with + stdin), stdin closed, `--timeout`. The CLI imports no host env, so + `export SH=bash TMP=/tmp REPO_ROOT=/oils ...;` is glued onto the first code + line, keeping `$LINENO` aligned. The Oils checkout and the helper shims are + mounted read-only, so the CLI must be built with `realfs` + (`cargo build -p bashkit-cli --features realfs`); the harness refuses a + binary without `--mount-ro`. +- **Helper shims.** Oils' `spec/bin/argv.py`, `printenv.py`, + `stdout_stderr.py`, `read_from_fd.py` are Python 2. `scripts/oils-spec/bin/` + re-implements them in bash with identical output (argv.py renders a Python 2 + list repr from `od` bytes) and both shells use them; bashkit runs no host + Python. A shim that bashkit mis-executes counts against bashkit, which is + fair: it is ordinary bash. +- **Failure classes.** Each bashkit miss gets a keyword class: `signals`, + `processes`, `interactive` (completion, history, prompts, bind), `host`, or + `behavior`. The heuristic separates gaps bashkit keeps by design (see + [Known Limitations](limitations.md)) from plain behavior differences; all of + them still count in the rate. First run: behavior 530, interactive 110, + host 27, processes 19, signals 10. +- **No floor, no CI gate yet.** Unlike the bash-oracle and Debian-oracle + scoreboards ([Testing Strategy](testing.md)), this is a headline metric: + bash-version drift on the runner moves the denominator. Saved results carry + the per-file missed-case list, so two runs diff case by case. + +## Results + +`just oils-spec` builds the CLI with `realfs`, runs the whole suite and saves +`scripts/oils-spec/results/oils-spec--.{json,md}` +(see [Performance Results](performance-results.md)). `run.py --fails FILE` +writes code, expected and actual output for every miss; positional spec names +(`run.py arith var-sub`) run a subset, which is never saved. The `/benches` +page shows the latest rate, the 12 spec files with the most misses and the +run history. `scripts/tests/test_oils_spec.py` covers the parser rules, the +scoring and the `argv.py` shim. + +## See also + +- [Testing Strategy](testing.md) - bash-oracle and Debian-oracle scoreboards +- [Performance Results](performance-results.md) - result locations +- [Known Limitations](limitations.md) - by-design gaps behind the failure classes diff --git a/knowledge/operations/performance-results.md b/knowledge/operations/performance-results.md index 89be6b5e4..15ecdf1d7 100644 --- a/knowledge/operations/performance-results.md +++ b/knowledge/operations/performance-results.md @@ -28,6 +28,7 @@ directories: | Criterion benches | `crates/bashkit/benches/results/` | `criterion-*.md` | | `bashkit-eval` (archived) | `crates/bashkit-eval/results/` | `eval-*.json`, `scripting-eval-*.json`, plus matching `.md` reports | | `bashkit-replay` gap telemetry | `crates/bashkit-eval/results/gaps/` | `gaps-*.json` plus matching `gaps-*.md` | +| Oils spec pass rate | `scripts/oils-spec/results/` | `oils-spec-*.json` plus matching `oils-spec-*.md` | Markdown files are the user-facing reports linked from `/benches`; JSON files are the aggregation input for benchmark and eval summaries. @@ -43,7 +44,7 @@ are the aggregation input for benchmark and eval summaries. Default benchmark recipes that represent a real run MUST save artifacts in the directories above: `just bench`, `just bench-parallel`, `just bench-sqlite`, `just bench-python`, -`just gaps`. +`just gaps`, `just oils-spec`. The comparison harness resolves `bash` from `PATH` and requires Bash 4 or newer (case conversion and associative arrays are benchmarked). A missing or older @@ -87,5 +88,11 @@ start table, warm `python_call//print`, and CPython point (CPython vs Monty) and the CPython history. `just bench-python` produces these reports. +The transformer also emits `oilsSpec`: `runs` (one point per +`oils-spec-*.json`: pass rate, bashkit passes among bash passes, Oils and +bashkit revisions) and `latest` (that report plus its 12 spec files with the +most misses). `/benches` renders both after the gap telemetry panels. See +[Oils Spec Pass Rate](oils-spec.md). + When changing result schemas, update the transformer and this spec in the same PR. Do not hand-edit `performance-timeline.json` except by running the script. diff --git a/knowledge/operations/testing.md b/knowledge/operations/testing.md index e8ca96836..a998769cb 100644 --- a/knowledge/operations/testing.md +++ b/knowledge/operations/testing.md @@ -384,6 +384,14 @@ process substitution, parameter expansion errors. representing a documented gap stays a gap rather than being deleted, so the corpus keeps measuring it (see [limitations](limitations.md)). +## Oils Spec Pass Rate + +`scripts/oils-spec/run.py` (`just oils-spec`) runs the whole upstream Oils +spec suite (bash column, ~2,760 cases) through bashkit and real bash and +publishes one headline number, bashkit passes over real-bash passes. It is a +measurement, not a gate: no floor, no CI job. See +[Oils Spec Pass Rate](oils-spec.md). + ## SQLite CLI Differential Tests CSV comparisons explicitly set the host CLI row separator to LF. SQLite diff --git a/scripts/oils-spec/README.md b/scripts/oils-spec/README.md new file mode 100644 index 000000000..571ba3e3d --- /dev/null +++ b/scripts/oils-spec/README.md @@ -0,0 +1,50 @@ +# Oils spec pass rate + +Runs the upstream [Oils](https://github.com/oils-for-unix/oils) spec suite +(Apache-2.0), bash column, through bashkit and through real bash, and reports +one headline number: + + pass rate = cases bashkit passes / cases real bash passes + +```bash +just oils-spec # build CLI (realfs), run all, save results +python3 scripts/oils-spec/run.py arith var-sub --fails /tmp/fails.txt +python3 scripts/oils-spec/run.py --no-bash # bashkit over all cases, no bash +``` + +`run.py` fetches Oils at the pinned `OILS_REV` into `target/oils-spec/oils` +(override with `--cache` or `OILS_SPEC_CACHE`); the test files are never +vendored. The bashkit CLI must be built with the `realfs` feature +(`cargo build -p bashkit-cli --features realfs`), because the Oils checkout and +`bin/` are mounted read-only into the sandbox. + +## How a case is scored + +The parser follows Oils' `test/sh_spec.py`: `#### name` starts a case, +`## stdout:` / `## STDOUT: ... ## END` / `## stdout-json:` / `## status:` / +`## stderr:` assert, and `## OK bash`, `## N-I bash`, `## BUG bash` overrides +win over the generic values. Status defaults to 0; stderr is checked only when +asserted. Files without `bash` in `## compare_shells`, and `## suite: disabled` +files, are skipped. + +- Real bash runs the Oils way: code on stdin, `SH=bash`, `TMP`, `REPO_ROOT`, + `LC_ALL=C.UTF-8`, a fresh temp cwd. Cases it fails (for example ones that + call `python2`) leave the denominator. +- bashkit runs `bashkit -c CODE bash` with stdin closed. The same variables are + exported by a prefix glued onto the first code line, so `$LINENO` matches. +- `bin/` holds bash stand-ins for Oils' Python 2 helpers (`argv.py`, + `printenv.py`, `stdout_stderr.py`, `read_from_fd.py`, `foo=bar`), with the + same output. Both shells use them. +- Each bashkit miss is classed `signals`, `processes`, `interactive`, `host` or + `behavior` by keyword, so by-design gaps stand apart. All still count. + +## Results + +`--save` (what `just oils-spec` passes) writes +`results/oils-spec--.json` and `.md`: totals, failure +classes, and per spec file the counts plus every case bashkit misses, so two +runs diff case by case. A subset run is never saved. The `/benches` site page +shows the latest headline and history. + +See `knowledge/operations/oils-spec.md` for the decisions and the current +number. diff --git a/scripts/oils-spec/bin/argv.py b/scripts/oils-spec/bin/argv.py new file mode 100755 index 000000000..6a56afef4 --- /dev/null +++ b/scripts/oils-spec/bin/argv.py @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Shell stand-in for Oils' spec/bin/argv.py (a Python 2 script): prints the +# arguments as a Python 2 list repr, byte for byte. Written in bash so the same +# helper runs under real bash and inside the bashkit sandbox, which does not +# execute host Python. Bytes come from od, so non-UTF-8 and multibyte +# arguments render as Python 2 would ('\xce\xbc'). +out='[' +sep='' +for arg in "$@"; do + q="'" + if [[ $arg == *"'"* && $arg != *'"'* ]]; then + q='"' + fi + s='' + for h in $(printf '%s' "$arg" | od -An -v -tx1); do + n=$((16#$h)) + if ((n == 92)); then + s+='\\' + elif ((n == 39)) && [[ $q == "'" ]]; then + s+="\\'" + elif ((n == 9)); then + s+='\t' + elif ((n == 10)); then + s+='\n' + elif ((n == 13)); then + s+='\r' + elif ((n < 32 || n > 126)); then + s+="\\x$h" + else + s+=$(printf "\\x$h") + fi + done + out+="$sep$q$s$q" + sep=', ' +done +printf '%s]\n' "$out" diff --git a/scripts/oils-spec/bin/foo=bar b/scripts/oils-spec/bin/foo=bar new file mode 100755 index 000000000..9be84c615 --- /dev/null +++ b/scripts/oils-spec/bin/foo=bar @@ -0,0 +1,2 @@ +#!/bin/sh +echo HI diff --git a/scripts/oils-spec/bin/printenv.py b/scripts/oils-spec/bin/printenv.py new file mode 100755 index 000000000..867643be2 --- /dev/null +++ b/scripts/oils-spec/bin/printenv.py @@ -0,0 +1,10 @@ +#!/usr/bin/env bash +# Shell stand-in for Oils' spec/bin/printenv.py: one line per name, the +# exported value or None. +for name in "$@"; do + if [[ $name =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] && [[ -v $name ]]; then + printf '%s\n' "${!name}" + else + echo None + fi +done diff --git a/scripts/oils-spec/bin/read_from_fd.py b/scripts/oils-spec/bin/read_from_fd.py new file mode 100755 index 000000000..d07d37bd4 --- /dev/null +++ b/scripts/oils-spec/bin/read_from_fd.py @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# Shell stand-in for Oils' spec/bin/read_from_fd.py: for each fd, prints +# ": " followed by what can be read from it. +for fd in "$@"; do + if ! { true <&"$fd"; } 2>/dev/null; then + echo "FATAL: Error reading from fd $fd" >&2 + exit 1 + fi + printf '%d: ' "$fd" + cat <&"$fd" +done diff --git a/scripts/oils-spec/bin/stdout_stderr.py b/scripts/oils-spec/bin/stdout_stderr.py new file mode 100755 index 000000000..7be08c5c9 --- /dev/null +++ b/scripts/oils-spec/bin/stdout_stderr.py @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +# Shell stand-in for Oils' spec/bin/stdout_stderr.py: prints $1 (STDOUT) to +# stdout, $2 (STDERR) to stderr, exits with $3 (0). +printf '%s\n' "${1-STDOUT}" +printf '%s\n' "${2-STDERR}" >&2 +exit "${3-0}" diff --git a/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.json b/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.json new file mode 100644 index 000000000..c73253911 --- /dev/null +++ b/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.json @@ -0,0 +1,6128 @@ +{ + "schema": 1, + "oils_repo": "https://github.com/oils-for-unix/oils", + "oils_rev": "57d3f0d088c36340bc4b3038208e58ea02acf15c", + "bashkit_commit": "f86bd7c81ae3", + "bash_version": "5.2.21(1)-release", + "timestamp": "2026-10-08T10:32:35Z", + "platform": "linux-x86_64", + "jobs": 4, + "timeout_secs": 10, + "elapsed_secs": 27.3, + "files_run": 132, + "files_skipped": 91, + "total": { + "cases": 2759, + "bashkit_pass": 2003, + "bashkit_pass_pct": 72.6, + "bash_pass": 2693, + "bash_pass_pct": 97.6, + "bashkit_pass_of_bash": 1997, + "pass_rate": 74.2 + }, + "failure_classes": { + "behavior": 530, + "host": 27, + "interactive": 110, + "processes": 19, + "signals": 10 + }, + "files": { + "builtin-completion": { + "summary": { + "cases": 51, + "bashkit_pass": 17, + "bashkit_pass_pct": 33.3, + "bash_pass": 51, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 17, + "pass_rate": 33.3 + }, + "missed": [ + { + "case": 0, + "name": "complete with no args and complete -p both print completion spec", + "class": "interactive", + "on": "stdout" + }, + { + "case": 1, + "name": "complete -F f is usage error", + "class": "interactive", + "on": "stdout" + }, + { + "case": 2, + "name": "complete with nonexistent function", + "class": "interactive", + "on": "stdout" + }, + { + "case": 3, + "name": "complete with no action", + "class": "interactive", + "on": "stdout" + }, + { + "case": 5, + "name": "Invalid syntax", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "how compgen calls completion functions", + "class": "interactive", + "on": "stdout" + }, + { + "case": 7, + "name": "complete -o -F (git)", + "class": "interactive", + "on": "status" + }, + { + "case": 8, + "name": "compopt with invalid syntax", + "class": "interactive", + "on": "stdout" + }, + { + "case": 9, + "name": "compopt fails when not in completion function", + "class": "interactive", + "on": "status" + }, + { + "case": 14, + "name": "compgen -v P", + "class": "interactive", + "on": "stdout" + }, + { + "case": 15, + "name": "compgen -e with global/local exported vars", + "class": "interactive", + "on": "stdout" + }, + { + "case": 18, + "name": "compgen -e P", + "class": "interactive", + "on": "stdout" + }, + { + "case": 19, + "name": "compgen with actions: function / variable / file", + "class": "interactive", + "on": "stdout" + }, + { + "case": 20, + "name": "compgen with actions: alias, setopt", + "class": "interactive", + "on": "stdout" + }, + { + "case": 21, + "name": "compgen with actions: shopt", + "class": "interactive", + "on": "stdout" + }, + { + "case": 22, + "name": "compgen with action and suffix: helptopic", + "class": "interactive", + "on": "stdout" + }, + { + "case": 25, + "name": "compgen -A user", + "class": "interactive", + "on": "status" + }, + { + "case": 28, + "name": "compgen -A command completes builtins and keywords", + "class": "interactive", + "on": "stdout" + }, + { + "case": 29, + "name": "compgen -k shows the same keywords as bash", + "class": "processes", + "on": "stdout" + }, + { + "case": 31, + "name": "compgen -k completes reserved shell keywords", + "class": "interactive", + "on": "stdout" + }, + { + "case": 32, + "name": "-o filenames and -o nospace have no effect with compgen", + "class": "interactive", + "on": "stdout" + }, + { + "case": 33, + "name": "-o plusdirs and -o dirnames with compgen", + "class": "interactive", + "on": "stdout" + }, + { + "case": 34, + "name": "compgen -o default completes files and dirs", + "class": "interactive", + "on": "stdout" + }, + { + "case": 35, + "name": "compgen doesn't respect -X for user-defined functions", + "class": "interactive", + "on": "stdout" + }, + { + "case": 36, + "name": "compgen -W words -X filter", + "class": "interactive", + "on": "stdout" + }, + { + "case": 37, + "name": "compgen -f -X filter -- $cur", + "class": "interactive", + "on": "stdout" + }, + { + "case": 39, + "name": "compgen -W 'one two three'", + "class": "interactive", + "on": "stdout" + }, + { + "case": 40, + "name": "compgen -W evaluates code in $()", + "class": "interactive", + "on": "stdout" + }, + { + "case": 41, + "name": "compgen -W uses IFS, and delimiters are escaped with \\", + "class": "interactive", + "on": "stdout" + }, + { + "case": 42, + "name": "Parse errors for compgen -W and complete -W", + "class": "interactive", + "on": "stdout" + }, + { + "case": 43, + "name": "Runtime errors for compgen -W", + "class": "interactive", + "on": "stdout" + }, + { + "case": 46, + "name": "compgen -A builtin", + "class": "interactive", + "on": "stdout" + }, + { + "case": 47, + "name": "complete -C vs. compgen -C", + "class": "interactive", + "on": "stdout" + }, + { + "case": 50, + "name": "compgen -F with scalar COMPREPLY", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "array": { + "summary": { + "cases": 78, + "bashkit_pass": 53, + "bashkit_pass_pct": 67.9, + "bash_pass": 78, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 53, + "pass_rate": 67.9 + }, + "missed": [ + { + "case": 3, + "name": "space before ( in array initialization", + "class": "behavior", + "on": "status" + }, + { + "case": 5, + "name": "array with invalid token", + "class": "behavior", + "on": "status" + }, + { + "case": 9, + "name": "Negative index", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "Negative index and sparse array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Negative index and sparse array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Length after unset", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Retrieve index that is a command sub", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "${!a[1]} is named ref in bash", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "All elements unquoted", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "$*", + "class": "behavior", + "on": "stdout" + }, + { + "case": 25, + "name": "Exporting array doesn't do anything, not even first element", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "Arrays can't be used as env bindings", + "class": "behavior", + "on": "stdout" + }, + { + "case": 28, + "name": "Associative arrays can't be used as env bindings either", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "Set array item to array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "Slice of array with [@]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "Negative slice begin", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "Slice with arithmetic", + "class": "behavior", + "on": "stdout" + }, + { + "case": 59, + "name": "Slice of sparse array with [@]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 63, + "name": "Dynamic parsing of LHS a[$code]=value", + "class": "behavior", + "on": "stdout" + }, + { + "case": 65, + "name": "Is element set? test -v a[i]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 67, + "name": "test -v a[i] with arith expressions", + "class": "behavior", + "on": "stdout" + }, + { + "case": 71, + "name": "Regression: Negative out-of-range index in [[ -v a[index] ]]", + "class": "behavior", + "on": "stderr" + }, + { + "case": 73, + "name": "Regression: unset a[-2]: out-of-bound negative index should cause error", + "class": "behavior", + "on": "stderr" + }, + { + "case": 75, + "name": "Regression: Array length after unset", + "class": "behavior", + "on": "stdout" + }, + { + "case": 77, + "name": "Regression: silent out-of-bound negative index in ${a[-2]} and $((a[-2]))", + "class": "behavior", + "on": "stderr" + } + ], + "bash_fails": [] + }, + "prompt": { + "summary": { + "cases": 33, + "bashkit_pass": 4, + "bashkit_pass_pct": 12.1, + "bash_pass": 29, + "bash_pass_pct": 87.9, + "bashkit_pass_of_bash": 4, + "pass_rate": 13.8 + }, + "missed": [ + { + "case": 2, + "name": "literal escapes", + "class": "interactive", + "on": "stdout" + }, + { + "case": 4, + "name": "PS1 evaluation order", + "class": "interactive", + "on": "stdout" + }, + { + "case": 5, + "name": "PS1 evaluation order 2", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "\\1004", + "class": "interactive", + "on": "stdout" + }, + { + "case": 7, + "name": "\\001 octal literals are supported", + "class": "interactive", + "on": "stdout" + }, + { + "case": 8, + "name": "\\555 is beyond max octal byte of \\377 and wrapped to m", + "class": "interactive", + "on": "stdout" + }, + { + "case": 11, + "name": "Escaped backslash", + "class": "interactive", + "on": "stdout" + }, + { + "case": 12, + "name": "\\0001 octal literals are not supported", + "class": "interactive", + "on": "stdout" + }, + { + "case": 13, + "name": "\\u0001 unicode literals not supported", + "class": "interactive", + "on": "stdout" + }, + { + "case": 15, + "name": "hostname", + "class": "interactive", + "on": "stdout" + }, + { + "case": 16, + "name": "username", + "class": "interactive", + "on": "stdout" + }, + { + "case": 17, + "name": "current working dir", + "class": "interactive", + "on": "stdout" + }, + { + "case": 18, + "name": "\\W is basename of working dir", + "class": "interactive", + "on": "stdout" + }, + { + "case": 19, + "name": "\\t for 24h time (HH:MM:SS)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 20, + "name": "\\T for 12h time (HH:MM:SS)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 21, + "name": "\\@ for 12h time (HH:MM AM/PM)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 22, + "name": "\\A for 24h time (HH:MM)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 23, + "name": "\\d for date", + "class": "interactive", + "on": "stdout" + }, + { + "case": 24, + "name": "\\D{%H:%M} for strftime", + "class": "interactive", + "on": "stdout" + }, + { + "case": 25, + "name": "\\D{} for locale specific strftime", + "class": "interactive", + "on": "stdout" + }, + { + "case": 26, + "name": "\\s for shell, \\v for major.minor version, and \\V for full version", + "class": "interactive", + "on": "stdout" + }, + { + "case": 27, + "name": "\\j for number of jobs", + "class": "signals", + "on": "stdout" + }, + { + "case": 28, + "name": "\\l for TTY device basename", + "class": "interactive", + "on": "stdout" + }, + { + "case": 29, + "name": "\\! for history number", + "class": "interactive", + "on": "stdout" + }, + { + "case": 30, + "name": "\\# for command number", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [ + 0, + 1, + 3, + 32 + ] + }, + "builtin-trap-bash": { + "summary": { + "cases": 23, + "bashkit_pass": 4, + "bashkit_pass_pct": 17.4, + "bash_pass": 23, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 17.4 + }, + "missed": [ + { + "case": 3, + "name": "trap DEBUG ignores $?", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "but trap DEBUG respects errexit", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "trap DEBUG with 'return'", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "trap DEBUG with 'exit'", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "trap DEBUG with non-compound commands", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "trap DEBUG and control flow", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "trap DEBUG and command sub / subshell", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "trap DEBUG not run in forked interpreter for first pipeline part", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "One 'echo' in first pipeline part - why does bash behave differently from case above?", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "trap DEBUG and pipeline (lastpipe difference)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "trap DEBUG function call", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "trap DEBUG case", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "trap DEBUG for each", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "trap DEBUG for expr", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "trap DEBUG if while", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "Compare trap DEBUG vs. trap ERR", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "Combine DEBUG trap and USR1 trap", + "class": "signals", + "on": "stdout" + }, + { + "case": 21, + "name": "Combine ERR trap and USR1 trap", + "class": "signals", + "on": "stdout" + }, + { + "case": 22, + "name": "Combine DEBUG trap and ERR trap", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "globignore": { + "summary": { + "cases": 18, + "bashkit_pass": 2, + "bashkit_pass_pct": 11.1, + "bash_pass": 18, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 11.1 + }, + "missed": [ + { + "case": 0, + "name": "Don't glob flags on file system with GLOBIGNORE", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "Ignore *.txt", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "Ignore ?.txt", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "Ignore *.o:*.h", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "Ignore single file src/__main__.py", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "Ignore dirs dist/*:node_modules/*", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "find files in subdirectory but not the ignored pattern", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "Ignore globs with char patterns like [!ab]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "Ignore globs with char classes like [[:alnum:]]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Ignore *", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "treat escaped patterns literally", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "resetting globignore reverts to default behaviour", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Ignore .:..", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "Quoting GLOBIGNORE", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "When GLOBIGNORE is set, glob may become empty (nullglob too)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "Extended glob expansion combined with GLOBIGNORE", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "word-split": { + "summary": { + "cases": 55, + "bashkit_pass": 37, + "bashkit_pass_pct": 67.3, + "bash_pass": 53, + "bash_pass_pct": 96.4, + "bashkit_pass_of_bash": 37, + "pass_rate": 69.8 + }, + "missed": [ + { + "case": 19, + "name": "empty literals are not elided", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "default value can yield multiple words", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "default value can yield multiple words with part joining", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "default value with unquoted IFS char", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "IFS='' with ${!prefix@} and ${!prefix*} (bug #627)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 40, + "name": "IFS='' with ${!a[@]} and ${!a[*]} (bug #627)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 41, + "name": "Bug #628 split on : with : in literal word", + "class": "behavior", + "on": "stdout" + }, + { + "case": 43, + "name": "Bug #1664, \\\\ with noglob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 46, + "name": "4 x 3 table: (default IFS, IFS='', IFS=zx) x ( $* \"$*\" $@ \"$@\" )", + "class": "behavior", + "on": "stdout" + }, + { + "case": 47, + "name": "4 x 3 table - with for loop", + "class": "behavior", + "on": "stdout" + }, + { + "case": 48, + "name": "IFS=x and '' and $@ - same bug as spec/toysh-posix case #12", + "class": "behavior", + "on": "stdout" + }, + { + "case": 49, + "name": "IFS=x and '' and $@ (#2)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 50, + "name": "IFS=x and '' and $@ (#3)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 51, + "name": "\"\"$A\"\" - empty string on both sides - derived from spec/toysh-posix #15", + "class": "behavior", + "on": "stdout" + }, + { + "case": 53, + "name": "Regression: ${!v} should be split", + "class": "behavior", + "on": "stdout" + }, + { + "case": 54, + "name": "Regression: \"${v:-AxBxC}\"x should not be split", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 24, + 25 + ] + }, + "var-ref": { + "summary": { + "cases": 31, + "bashkit_pass": 15, + "bashkit_pass_pct": 48.4, + "bash_pass": 30, + "bash_pass_pct": 96.8, + "bashkit_pass_of_bash": 15, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 2, + "name": "${!undef:-}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "${!a[@]-'default'} is legal but fails with more than one element", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "var ref: 1, @, *", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "var ref TO array var", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "var ref TO array var, with subscripts", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "var ref TO assoc array a[key]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "var ref TO array with arbitrary subscripts", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "Indirect expansion TO fancy expansion features bash disallows", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "Bad var ref", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "Bad var ref 2", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "Var Ref Code Injection $(tee PWNED)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "${!array_ref:-set} and ${!array_ref:=assign}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 28, + "name": "Array indirect expansion with suffix operators", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "Array indirect expansion with replacements", + "class": "behavior", + "on": "stdout" + }, + { + "case": 30, + "name": "Array indirect expansion with @? conversion", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 12 + ] + }, + "builtin-cd": { + "summary": { + "cases": 30, + "bashkit_pass": 14, + "bashkit_pass_pct": 46.7, + "bash_pass": 28, + "bash_pass_pct": 93.3, + "bashkit_pass_of_bash": 14, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 1, + "name": "cd BAD/..", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "cd with 2 or more args - with strict_arg_parse", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "cd with 2 or more args is allowed (strict_arg_parse disabled)", + "class": "behavior", + "on": "status" + }, + { + "case": 4, + "name": "cd - without OLDPWD", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "$OLDPWD", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "pwd with symlink and -P", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "setting $PWD doesn't affect the value of 'pwd' builtin", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "unset PWD; then pwd", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "pwd in symlinked dir on shell initialization", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "cd permits double bare dash", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "cd to symlink with -L and -P", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "cd to relative path with -L and -P", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "CDPATH is respected", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "Survey of getcwd() syscall", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 24, + 25 + ] + }, + "alias": { + "summary": { + "cases": 48, + "bashkit_pass": 35, + "bashkit_pass_pct": 72.9, + "bash_pass": 48, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 35, + "pass_rate": 72.9 + }, + "missed": [ + { + "case": 0, + "name": "Usage of builtins", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "define and use alias on a single line", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "Syntax error after expansion", + "class": "behavior", + "on": "status" + }, + { + "case": 23, + "name": "Loop split across alias in another way", + "class": "behavior", + "on": "stdout" + }, + { + "case": 24, + "name": "Loop split across both iterative and recursive aliases", + "class": "behavior", + "on": "stdout" + }, + { + "case": 25, + "name": "Alias with a quote in the middle is a syntax error", + "class": "behavior", + "on": "status" + }, + { + "case": 30, + "name": "Alias can be defined and used on a single line", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "alias with environment bindings works", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "alias with line continuation in the middle", + "class": "behavior", + "on": "stdout" + }, + { + "case": 35, + "name": "alias for left brace", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "alias for left paren", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "here doc inside alias", + "class": "behavior", + "on": "stdout" + }, + { + "case": 43, + "name": "Alias and command sub (bug regression)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "dbracket": { + "summary": { + "cases": 49, + "bashkit_pass": 36, + "bashkit_pass_pct": 73.5, + "bash_pass": 49, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 36, + "pass_rate": 73.5 + }, + "missed": [ + { + "case": 20, + "name": "[[ with op variable (compare with test-builtin.test.sh)", + "class": "behavior", + "on": "status" + }, + { + "case": 23, + "name": "[[ with env prefix doesn't work", + "class": "behavior", + "on": "status" + }, + { + "case": 25, + "name": "Argument that looks like a real operator", + "class": "behavior", + "on": "status" + }, + { + "case": 33, + "name": "[[ '(' foo ]] is syntax error", + "class": "behavior", + "on": "status" + }, + { + "case": 35, + "name": "[[ -z ]] is syntax error", + "class": "behavior", + "on": "status" + }, + { + "case": 37, + "name": "[[ -z '>' a ]] is syntax error", + "class": "behavior", + "on": "status" + }, + { + "case": 39, + "name": "[[ ]] is syntax error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 40, + "name": "[[ && ]] is syntax error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 41, + "name": "[[ a 3< b ]] doesn't work (bug regression)", + "class": "behavior", + "on": "status" + }, + { + "case": 44, + "name": "tilde expansion with =~ (confusing)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 45, + "name": "[[ ]] with redirect", + "class": "behavior", + "on": "stdout" + }, + { + "case": 46, + "name": "special chars", + "class": "behavior", + "on": "stdout" + }, + { + "case": 47, + "name": "\\(\\) in pattern (regression)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-fc": { + "summary": { + "cases": 14, + "bashkit_pass": 2, + "bashkit_pass_pct": 14.3, + "bash_pass": 14, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 14.3 + }, + "missed": [ + { + "case": 0, + "name": "fc -l lists history commands", + "class": "interactive", + "on": "stdout" + }, + { + "case": 1, + "name": "fc -ln lists history commands without numbers", + "class": "interactive", + "on": "stdout" + }, + { + "case": 2, + "name": "fc -lr lists history commands in reverse order", + "class": "interactive", + "on": "stdout" + }, + { + "case": 3, + "name": "fc -lnr lists history commands without numbers in reverse order", + "class": "interactive", + "on": "stdout" + }, + { + "case": 4, + "name": "fc -l lists history commands with default page size", + "class": "interactive", + "on": "stdout" + }, + { + "case": 5, + "name": "fc -l [first] where first is an index", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "fc -l [first] where first is an offset from current command", + "class": "interactive", + "on": "stdout" + }, + { + "case": 7, + "name": "fc -l [first] [last] where first and last are indexes", + "class": "interactive", + "on": "stdout" + }, + { + "case": 8, + "name": "fc -l [first] [last] where first and last are offsets from current command", + "class": "interactive", + "on": "stdout" + }, + { + "case": 9, + "name": "fc -l [first] [last] where first and last are reversed indexes", + "class": "interactive", + "on": "stdout" + }, + { + "case": 10, + "name": "fc -lr [first] [last] where first and last are reversed indexes does not undo reverse", + "class": "interactive", + "on": "stdout" + }, + { + "case": 12, + "name": "fc errors out on too many args with strict_arg_parse", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-history": { + "summary": { + "cases": 17, + "bashkit_pass": 5, + "bashkit_pass_pct": 29.4, + "bash_pass": 17, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 29.4 + }, + "missed": [ + { + "case": 0, + "name": "history -a", + "class": "interactive", + "on": "stdout" + }, + { + "case": 1, + "name": "history -w writes out the in-memory history to the history file", + "class": "interactive", + "on": "stdout" + }, + { + "case": 2, + "name": "history -r reads from the history file, and appends it to the current history", + "class": "interactive", + "on": "stdout" + }, + { + "case": 3, + "name": "history -n reads *new* commands from the history file, and appends them to the current history", + "class": "interactive", + "on": "stdout" + }, + { + "case": 4, + "name": "history -c clears in-memory history", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "history -d to delete history from end", + "class": "interactive", + "on": "stdout" + }, + { + "case": 9, + "name": "HISTFILE set to array", + "class": "interactive", + "on": "stdout" + }, + { + "case": 11, + "name": "history usage", + "class": "interactive", + "on": "stdout" + }, + { + "case": 12, + "name": "HISTSIZE shrinks the in-memory history when changed", + "class": "interactive", + "on": "stdout" + }, + { + "case": 13, + "name": "HISTFILESIZE shrinks the history file when changed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "recording history can be toggled with set -o/+o history", + "class": "interactive", + "on": "stdout" + }, + { + "case": 16, + "name": "shopt histappend - osh ignores shopt and appends, bash sometimes overwrites", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-type-bash": { + "summary": { + "cases": 31, + "bashkit_pass": 19, + "bashkit_pass_pct": 61.3, + "bash_pass": 31, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 19, + "pass_rate": 61.3 + }, + "missed": [ + { + "case": 1, + "name": "type -t -> alias", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "type -t doesn't find non-executable (like command -v)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "type -a -P gives multiple files", + "class": "host", + "on": "stdout" + }, + { + "case": 14, + "name": "type -P builtin -> not a file but file found", + "class": "host", + "on": "stdout" + }, + { + "case": 16, + "name": "type -f builtin -> function and file exists", + "class": "host", + "on": "stdout" + }, + { + "case": 19, + "name": "type -a -> alias; prints alias definition", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "type -ap -> alias", + "class": "behavior", + "on": "status" + }, + { + "case": 24, + "name": "type -a -> file", + "class": "host", + "on": "stdout" + }, + { + "case": 25, + "name": "type -ap -> file; abbreviated", + "class": "host", + "on": "stdout" + }, + { + "case": 26, + "name": "type -a -> builtin and file", + "class": "host", + "on": "stdout" + }, + { + "case": 27, + "name": "type -a -> builtin and file and shell function", + "class": "host", + "on": "stdout" + }, + { + "case": 28, + "name": "type -ap -> builtin and file; doesn't print builtin or function", + "class": "host", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "interactive": { + "summary": { + "cases": 18, + "bashkit_pass": 7, + "bashkit_pass_pct": 38.9, + "bash_pass": 18, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 38.9 + }, + "missed": [ + { + "case": 0, + "name": "'exit' in oshrc (regression)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 2, + "name": "interactive shell loads rcfile (when combined with -c)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 7, + "name": "interactive shell runs PROMPT_COMMAND after each command", + "class": "interactive", + "on": "stdout" + }, + { + "case": 10, + "name": "Error message with bad oshrc file (currently ignored)", + "class": "interactive", + "on": "stdout" + }, + { + "case": 11, + "name": "PROMPT_COMMAND can see $?, like bash", + "class": "interactive", + "on": "stdout" + }, + { + "case": 12, + "name": "PROMPT_COMMAND that writes to BASH_REMATCH", + "class": "interactive", + "on": "stdout" + }, + { + "case": 13, + "name": "NO ASSERTIONS: Are startup files sourced before or after job control?", + "class": "processes", + "on": "status" + }, + { + "case": 14, + "name": "HISTFILE is written in interactive shell", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "HISTFILE default value", + "class": "interactive", + "on": "stdout" + }, + { + "case": 16, + "name": "HISTFILE=my-history loads history from that file, and writes back to it", + "class": "interactive", + "on": "stdout" + }, + { + "case": 17, + "name": "HISTFILE=my-history with history -a", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "sh-options": { + "summary": { + "cases": 39, + "bashkit_pass": 28, + "bashkit_pass_pct": 71.8, + "bash_pass": 39, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 28, + "pass_rate": 71.8 + }, + "missed": [ + { + "case": 1, + "name": "$- with pipefail", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "$- with interactive shell", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "pass shopt options like sh -O nullglob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "vi and emacs are mutually exclusive", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "interactive shell starts with emacs mode on", + "class": "interactive", + "on": "stdout" + }, + { + "case": 17, + "name": "shopt with no flags prints options", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "noclobber on <>", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "noclobber on &> >", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "stubbed out bash options", + "class": "behavior", + "on": "stdout" + }, + { + "case": 35, + "name": "Unimplemented options - print, query, set, unset", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "Unimplemented options - OSH shopt -s ignore_shopt_not_impl", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "vars-special": { + "summary": { + "cases": 42, + "bashkit_pass": 29, + "bashkit_pass_pct": 69.0, + "bash_pass": 40, + "bash_pass_pct": 95.2, + "bashkit_pass_of_bash": 29, + "pass_rate": 72.5 + }, + "missed": [ + { + "case": 1, + "name": "$PWD is not only set, but exported", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "$PATH is set if unset at startup", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "$HOME is NOT set", + "class": "interactive", + "on": "stdout" + }, + { + "case": 4, + "name": "Vars set interactively only: $HISTFILE", + "class": "interactive", + "on": "stdout" + }, + { + "case": 5, + "name": "Some vars are set, even without startup file, or env: PATH, PWD", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "UID EUID PPID can't be changed", + "class": "processes", + "on": "stdout" + }, + { + "case": 13, + "name": "$BASHPID DOES change with subshell and command sub", + "class": "processes", + "on": "stdout" + }, + { + "case": 25, + "name": "$LINENO in ((", + "class": "behavior", + "on": "stdout" + }, + { + "case": 28, + "name": "$LINENO in for (( loop", + "class": "behavior", + "on": "stdout" + }, + { + "case": 37, + "name": "$_ with assignments, arrays, etc.", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "$_ is not undefined on first use", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 34, + 40 + ] + }, + "array-assign": { + "summary": { + "cases": 11, + "bashkit_pass": 1, + "bashkit_pass_pct": 9.1, + "bash_pass": 11, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 1, + "pass_rate": 9.1 + }, + "missed": [ + { + "case": 1, + "name": "Indexed LHS with spaces", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "Nested a[i[0]]=0", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "Multiple LHS array words", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "LHS array is protected with shopt -s eval_unsafe_arith, e.g. 'a[$(echo 2)]'", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "file named a[ is not executed", + "class": "host", + "on": "stdout" + }, + { + "case": 6, + "name": "More fragments like a[ a[5 a[5 + a[5 + 3]", + "class": "host", + "on": "stdout" + }, + { + "case": 7, + "name": "Are quotes allowed?", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "Tricky parsing - a[ a[0]=1 ]=X a[ a[0]+=1 ]+=X", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "argv.py a[1 + 2]=", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "declare builtin doesn't allow spaces", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "array-sparse": { + "summary": { + "cases": 40, + "bashkit_pass": 30, + "bashkit_pass_pct": 75.0, + "bash_pass": 40, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 30, + "pass_rate": 75.0 + }, + "missed": [ + { + "case": 11, + "name": "unset -v a[i] with out-of-bound negative index", + "class": "behavior", + "on": "stderr" + }, + { + "case": 12, + "name": "unset -v a[i] for max index", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "[[ -v a[i] ]] with invalid negative index", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "((sp[i])) and ((sp[i]++)) with invalid negative index", + "class": "behavior", + "on": "stderr" + }, + { + "case": 18, + "name": "${sp[i]} with negative invalid index", + "class": "behavior", + "on": "stderr" + }, + { + "case": 19, + "name": "${a[@]:offset:length}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 25, + "name": "${a[@]@P}, ${a[@]@Q}, and ${a[@]@a}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "${a-}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 28, + "name": "${!a[0]}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 31, + "name": "compgen -F _set_COMPREPLY", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-sub-quote": { + "summary": { + "cases": 41, + "bashkit_pass": 31, + "bashkit_pass_pct": 75.6, + "bash_pass": 41, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 31, + "pass_rate": 75.6 + }, + "missed": [ + { + "case": 11, + "name": "Multiple words: no outer quotes, inner single quotes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Multiple words: no outer quotes, inner double quotes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "Mixed inner quotes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "part_value tree with multiple words", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "part_value tree on RHS", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "Multiple words: no outer quotes, inner double quotes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "No outer quotes, Multiple internal quotes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "\"${undef-'c d'}\" and \"${foo%'c d'}\" are parsed differently", + "class": "behavior", + "on": "stdout" + }, + { + "case": 38, + "name": "Right Brace as argument (similar to #702)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "Var substitution with newlines (#2492)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "array-assoc": { + "summary": { + "cases": 42, + "bashkit_pass": 27, + "bashkit_pass_pct": 64.3, + "bash_pass": 36, + "bash_pass_pct": 85.7, + "bashkit_pass_of_bash": 27, + "pass_rate": 75.0 + }, + "missed": [ + { + "case": 7, + "name": "coerce to string with ${A[*]}, etc.", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "lookup with unquoted $key and quoted \"$i$i\"", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "Indexed array as key of associative array coerces to string (without shopt -s strict_array)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 30, + "name": "unset -v and assoc array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 31, + "name": "nameref and assoc array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "${!ref} and assoc array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "printf -v and assoc array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "test -v assoc[key]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 37, + "name": "test -v with dynamic parsing", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 1, + 2, + 21, + 26, + 34, + 41 + ] + }, + "builtin-bind": { + "summary": { + "cases": 9, + "bashkit_pass": 0, + "bashkit_pass_pct": 0.0, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 0, + "pass_rate": 0.0 + }, + "missed": [ + { + "case": 0, + "name": "bind -l should report readline functions", + "class": "interactive", + "on": "stdout" + }, + { + "case": 1, + "name": "bind -p -P to print function names and key bindings", + "class": "interactive", + "on": "stdout" + }, + { + "case": 2, + "name": "bind -s -S accepted", + "class": "interactive", + "on": "status" + }, + { + "case": 3, + "name": "bind -v -V accepted", + "class": "interactive", + "on": "stdout" + }, + { + "case": 4, + "name": "bind -q", + "class": "interactive", + "on": "stdout" + }, + { + "case": 5, + "name": "bind -X", + "class": "interactive", + "on": "stdout" + }, + { + "case": 6, + "name": "bind -m with bind -x/-X", + "class": "interactive", + "on": "stdout" + }, + { + "case": 7, + "name": "bind -r", + "class": "interactive", + "on": "stdout" + }, + { + "case": 8, + "name": "bind -r of bind -x commands", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-read": { + "summary": { + "cases": 64, + "bashkit_pass": 55, + "bashkit_pass_pct": 85.9, + "bash_pass": 64, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 55, + "pass_rate": 85.9 + }, + "missed": [ + { + "case": 33, + "name": "read -t 0 tests if input is available", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "read -u", + "class": "behavior", + "on": "stdout" + }, + { + "case": 38, + "name": "read -u -s", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "read -u 3 -d 5", + "class": "behavior", + "on": "stdout" + }, + { + "case": 40, + "name": "read -u 3 -d b -N 6", + "class": "behavior", + "on": "stdout" + }, + { + "case": 41, + "name": "read -N doesn't respect delimiter, while read -n does", + "class": "behavior", + "on": "stdout" + }, + { + "case": 48, + "name": "mapfile from directory (bash doesn't handle errors)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 54, + "name": "empty input and splitting", + "class": "behavior", + "on": "stdout" + }, + { + "case": 63, + "name": "read bash bug", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-trap": { + "summary": { + "cases": 33, + "bashkit_pass": 24, + "bashkit_pass_pct": 72.7, + "bash_pass": 33, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 24, + "pass_rate": 72.7 + }, + "missed": [ + { + "case": 0, + "name": "traps are not active inside subshells $() () trap | cat", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "trap EXIT calling exit", + "class": "behavior", + "on": "status" + }, + { + "case": 20, + "name": "trap EXIT with PARSE error and explicit exit", + "class": "behavior", + "on": "status" + }, + { + "case": 24, + "name": "exit codes for traps are isolated", + "class": "signals", + "on": "stdout" + }, + { + "case": 25, + "name": "traps are cleared in subshell (started with &)", + "class": "signals", + "on": "stdout" + }, + { + "case": 26, + "name": "trap USR1, sleep, SIGINT: non-interactively", + "class": "behavior", + "on": "stdout" + }, + { + "case": 28, + "name": "trap EXIT, sleep, SIGINT: non-interactively", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "Remove trap with an unsigned integer", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "trap with command.NoOp - check internal invariant", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "introspect": { + "summary": { + "cases": 13, + "bashkit_pass": 3, + "bashkit_pass_pct": 23.1, + "bash_pass": 12, + "bash_pass_pct": 92.3, + "bashkit_pass_of_bash": 3, + "pass_rate": 25.0 + }, + "missed": [ + { + "case": 0, + "name": "${FUNCNAME[@]} array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "FUNCNAME with source (scalar or array)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "BASH_SOURCE and BASH_LINENO scalar or array (e.g. for virtualenv)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "$((BASH_LINENO)) (scalar form in arith)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "${BASH_SOURCE[@]} with source and function name", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "${BASH_LINENO[@]} is a stack of line numbers for function calls", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "Locations with temp frame", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Locations when sourcing", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Sourcing inside function grows the debug stack", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 8 + ] + }, + "nul-bytes": { + "summary": { + "cases": 16, + "bashkit_pass": 7, + "bashkit_pass_pct": 43.8, + "bash_pass": 16, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 43.8 + }, + "missed": [ + { + "case": 1, + "name": "printf - literal NUL in format string", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "printf - NUL byte in value (OSH and zsh agree)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "NUL bytes with echo $'\\0' (OSH and zsh agree)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "NUL bytes with test -n", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "NUL bytes with test -f", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "NUL bytes with ${#s} (OSH and zsh agree)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "Compare \\x00 byte versus \\x01 byte - read builtin", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Compare \\x00 byte versus \\x01 byte - read -n", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Compare \\x00 byte versus \\x01 byte - mapfile builtin", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "redirect-multi": { + "summary": { + "cases": 13, + "bashkit_pass": 4, + "bashkit_pass_pct": 30.8, + "bash_pass": 13, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 30.8 + }, + "missed": [ + { + "case": 0, + "name": "File redirects with glob args (bash and zsh only)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "File redirect without matching any file, with failglob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "File redirect to $var with glob char", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "File redirect that globs to more than one file (bash and zsh only)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "File redirect with extended glob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Extended glob that doesn't match anything", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "Non-file redirects don't respect glob args (we differe from bash)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Redirect with brace expansion isn't allowed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "File redirects have word splitting too!", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "regex": { + "summary": { + "cases": 37, + "bashkit_pass": 28, + "bashkit_pass_pct": 75.7, + "bash_pass": 37, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 28, + "pass_rate": 75.7 + }, + "missed": [ + { + "case": 10, + "name": "Regex with == and not =~ is parse error, different lexer mode required", + "class": "behavior", + "on": "status" + }, + { + "case": 12, + "name": "Malformed regex", + "class": "behavior", + "on": "status" + }, + { + "case": 16, + "name": "Unquoted { is a regex parse error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "Quoted { and +", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "unquoted [a b] as pattern, [a b|c]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 31, + "name": "Operator chars ; & but not |", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "Quotes '' \"\" $'' $\"\" in pattern", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "Parse error with 2 words", + "class": "behavior", + "on": "status" + }, + { + "case": 35, + "name": "make a lisp example", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "tilde": { + "summary": { + "cases": 14, + "bashkit_pass": 5, + "bashkit_pass_pct": 35.7, + "bash_pass": 14, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 35.7 + }, + "missed": [ + { + "case": 1, + "name": "~ expansion in readonly assignment", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "No tilde expansion in word that looks like assignment but isn't", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "${undef:-~}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "${x//~/~root}", + "class": "host", + "on": "stdout" + }, + { + "case": 8, + "name": "x=foo:~ has tilde expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "a[x]=foo:~ has tilde expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "tilde expansion an assignment keyword", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "x=${undef-~:~}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "temp assignment x=~ env", + "class": "host", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-op-bash": { + "summary": { + "cases": 27, + "bashkit_pass": 17, + "bashkit_pass_pct": 63.0, + "bash_pass": 26, + "bash_pass_pct": 96.3, + "bashkit_pass_of_bash": 17, + "pass_rate": 65.4 + }, + "missed": [ + { + "case": 10, + "name": "${!prefix@} ${!prefix*} yields sorted array of var names", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "${!prefix@} matches var name (regression)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "undef and @P @Q @a", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "assoc array and @P @Q @a", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "${!A@a} and ${!A[@]@a}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "undef vs. empty string in var ops", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "-o nounset with var ops", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "${!r@a} with r='a[0]' (attribute for indirect expansion of an array element)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "Array expansion with nullary var op @a", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 3 + ] + }, + "var-op-patsub": { + "summary": { + "cases": 28, + "bashkit_pass": 19, + "bashkit_pass_pct": 67.9, + "bash_pass": 28, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 19, + "pass_rate": 67.9 + }, + "missed": [ + { + "case": 11, + "name": "Confusing unquoted slash matches bash (and ash)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "Synthesized ${x///} bug (similar to above)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "Replacement with special chars (bug fix)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Replace backslash", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "When LC_ALL=C, pattern ? doesn't match multibyte character", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "patsub with [^]]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 24, + "name": "[a-z] Invalid range end is syntax error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "Chromium from http://www.oilshell.org/blog/2016/11/07.html", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "${x//~homedir/}", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "ysh-builtin-private": { + "summary": { + "cases": 23, + "bashkit_pass": 13, + "bashkit_pass_pct": 56.5, + "bash_pass": 22, + "bash_pass_pct": 95.7, + "bashkit_pass_of_bash": 13, + "pass_rate": 59.1 + }, + "missed": [ + { + "case": 9, + "name": "builtin sleep behaves like external sleep", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "builtin cat", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "builtin cat usage", + "class": "behavior", + "on": "stdout" + }, + { + "case": 14, + "name": "builtin cat non nonexistent file", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "builtin cat accept - for stdin", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "builtin rm: usage, removing files", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "builtin rm -f - ignores arguments that don't exist", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "builtin rm always fails on directories (regardless of -f)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "compgen -A builtin doesn't find private builtins", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [ + 18 + ] + }, + "arith": { + "summary": { + "cases": 74, + "bashkit_pass": 66, + "bashkit_pass_pct": 89.2, + "bash_pass": 74, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 66, + "pass_rate": 89.2 + }, + "missed": [ + { + "case": 0, + "name": "Side Effect in Array Indexing", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "Arith sub within arith sub", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Backticks within arith sub", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "Integer constant validation", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "Increment undefined variables with nounset", + "class": "behavior", + "on": "stdout" + }, + { + "case": 40, + "name": "nounset with arithmetic", + "class": "behavior", + "on": "stdout" + }, + { + "case": 50, + "name": "Comment not allowed in the middle of multiline arithmetic", + "class": "behavior", + "on": "stdout" + }, + { + "case": 70, + "name": "undef[0] with nounset", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "assign-extended": { + "summary": { + "cases": 39, + "bashkit_pass": 27, + "bashkit_pass_pct": 69.2, + "bash_pass": 35, + "bash_pass_pct": 89.7, + "bashkit_pass_of_bash": 27, + "pass_rate": 77.1 + }, + "missed": [ + { + "case": 4, + "name": "declare -F with shopt -s extdebug prints more info", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "declare -F with shopt -s extdebug and main file", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "ble.sh: eval -- \"$(declare -p var arr)\"", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "eval -- \"$(declare -p arr)\" (restore arrays w/ unset elements)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "declare -p UNDEF (and typeset) -- prints something to stderr", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "syntax error in array assignment", + "class": "behavior", + "on": "stdout" + }, + { + "case": 31, + "name": "myvar=typeset (another form of dynamic assignment)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 38, + "name": "unset and shell funcs", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 7, + 8, + 10, + 11 + ] + }, + "brace-expansion": { + "summary": { + "cases": 55, + "bashkit_pass": 47, + "bashkit_pass_pct": 85.5, + "bash_pass": 55, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 47, + "pass_rate": 85.5 + }, + "missed": [ + { + "case": 11, + "name": "double expansion with simple var -- bash bug", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "double expansion with literal and simple var", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "{ in expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "Empty alternative with empty string suffix", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "Two kinds of tilde expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 30, + "name": "Tilde expansion come before var expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "regression: 0 step disallowed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 42, + "name": "Mixed case char expansion is invalid", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "bugs": { + "summary": { + "cases": 29, + "bashkit_pass": 21, + "bashkit_pass_pct": 72.4, + "bash_pass": 29, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 21, + "pass_rate": 72.4 + }, + "missed": [ + { + "case": 5, + "name": "assign readonly -- multiple lines -- set -o posix", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Function names", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "file with NUL byte", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "'echo' and printf fail on writing to full disk", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "other builtins fail on writing to full disk", + "class": "processes", + "on": "stdout" + }, + { + "case": 19, + "name": "command execution $(echo 42 | tee PWNED) not allowed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "unset doesn't allow command execution", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "Crash after changing $[] to be alias of $(( ))", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-op-test": { + "summary": { + "cases": 37, + "bashkit_pass": 29, + "bashkit_pass_pct": 78.4, + "bash_pass": 37, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 29, + "pass_rate": 78.4 + }, + "missed": [ + { + "case": 15, + "name": "Nix idiom ${!hooksSlice+\"${!hooksSlice}\"} - was workaround for obsolete bash 4.3 bug", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "array and - and +", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "$@ (\"\" \"\") and - and +", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "$* (\"\" \"\") and - and + (IFS=)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "op-test for ${a} and ${a[0]}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "op-test for ${!array} with array=\"a\" and array=\"a[0]\"", + "class": "behavior", + "on": "stdout" + }, + { + "case": 35, + "name": "op-test for ${!array} with array=\"a[@]\" or array=\"a[*]\"", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "op-test for unquoted ${a[*]:-empty} with IFS=", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "array-literal": { + "summary": { + "cases": 19, + "bashkit_pass": 12, + "bashkit_pass_pct": 63.2, + "bash_pass": 19, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 12, + "pass_rate": 63.2 + }, + "missed": [ + { + "case": 0, + "name": "Tilde expansions in RHS of [k]=v (BashArray)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "Tilde expansions in RHS of [k]=v (BashAssoc)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "append to element (BashAssoc)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "non-index forms of element (BashAssoc)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "Evaluation order (1)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Evaluation order (2)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "Evaluation order (3)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "background": { + "summary": { + "cases": 27, + "bashkit_pass": 20, + "bashkit_pass_pct": 74.1, + "bash_pass": 27, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 20, + "pass_rate": 74.1 + }, + "missed": [ + { + "case": 1, + "name": "wait -n with arguments - arguments are respected", + "class": "processes", + "on": "stdout" + }, + { + "case": 5, + "name": "wait with invalid arg", + "class": "processes", + "on": "status" + }, + { + "case": 7, + "name": "wait for N parallel jobs and check failure", + "class": "processes", + "on": "stdout" + }, + { + "case": 11, + "name": "Start background pipeline, wait %job_spec", + "class": "processes", + "on": "stdout" + }, + { + "case": 12, + "name": "Wait for job and PIPESTATUS", + "class": "processes", + "on": "stdout" + }, + { + "case": 13, + "name": "Wait for job and PIPESTATUS - cat", + "class": "processes", + "on": "stdout" + }, + { + "case": 26, + "name": "Signal message for killed background job", + "class": "signals", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "ble-idioms": { + "summary": { + "cases": 26, + "bashkit_pass": 19, + "bashkit_pass_pct": 73.1, + "bash_pass": 26, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 19, + "pass_rate": 73.1 + }, + "missed": [ + { + "case": 5, + "name": "recursive arith: recursion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "dynamic arith varname: copy/add", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "is-array with ${var@a}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "shift unshift reverse", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "shopt -u expand_aliases and eval", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "Issue #1069 [53] BUG: a[1 + 1]=2, etc. fails", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "Issue #1069 [53] - LHS array parsing a[1 + 2]=3 (see spec/array-assign for more)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-bracket": { + "summary": { + "cases": 52, + "bashkit_pass": 43, + "bashkit_pass_pct": 82.7, + "bash_pass": 50, + "bash_pass_pct": 96.2, + "bashkit_pass_of_bash": 43, + "pass_rate": 86.0 + }, + "missed": [ + { + "case": 27, + "name": "-k for sticky bit", + "class": "behavior", + "on": "stdout" + }, + { + "case": 36, + "name": "-G and -O for effective user ID and group ID", + "class": "behavior", + "on": "stdout" + }, + { + "case": 37, + "name": "-u for setuid, -g too", + "class": "host", + "on": "stdout" + }, + { + "case": 38, + "name": "-v to test variable (bash)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "test -o for options", + "class": "behavior", + "on": "stdout" + }, + { + "case": 41, + "name": "-ef", + "class": "behavior", + "on": "stdout" + }, + { + "case": 44, + "name": "test -c", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 25, + 26 + ] + }, + "builtin-eval-source": { + "summary": { + "cases": 23, + "bashkit_pass": 14, + "bashkit_pass_pct": 60.9, + "bash_pass": 21, + "bash_pass_pct": 91.3, + "bashkit_pass_of_bash": 14, + "pass_rate": 66.7 + }, + "missed": [ + { + "case": 1, + "name": "eval accepts/ignores --", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "eval usage", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "source accepts/ignores --", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Eval in does tilde expansion", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "Eval in bash does tilde expansion in array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "source looks in PATH for files", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "source finds files in PATH before current dir", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 13, + 14 + ] + }, + "command_": { + "summary": { + "cases": 16, + "bashkit_pass": 9, + "bashkit_pass_pct": 56.2, + "bash_pass": 16, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 9, + "pass_rate": 56.2 + }, + "missed": [ + { + "case": 3, + "name": "Name too long", + "class": "behavior", + "on": "status" + }, + { + "case": 8, + "name": "$PATH lookup", + "class": "host", + "on": "stdout" + }, + { + "case": 9, + "name": "filling $PATH cache, then insert the same command earlier in cache", + "class": "host", + "on": "stdout" + }, + { + "case": 10, + "name": "filling $PATH cache, then deleting command", + "class": "host", + "on": "stdout" + }, + { + "case": 11, + "name": "Non-executable on $PATH", + "class": "host", + "on": "stdout" + }, + { + "case": 12, + "name": "hash without args prints the cache", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "hash with args", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "redirect": { + "summary": { + "cases": 41, + "bashkit_pass": 34, + "bashkit_pass_pct": 82.9, + "bash_pass": 41, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 34, + "pass_rate": 82.9 + }, + "missed": [ + { + "case": 0, + "name": ">& and <& are the same", + "class": "behavior", + "on": "stderr" + }, + { + "case": 23, + "name": "&> redirects stdout and stderr", + "class": "processes", + "on": "stdout" + }, + { + "case": 24, + "name": ">&word redirects stdout and stderr when word is not a number or -", + "class": "processes", + "on": "stdout" + }, + { + "case": 26, + "name": "1>&2- to move file descriptor", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "1>&2- (Bash bug: fail to restore closed fd)", + "class": "processes", + "on": "stdout" + }, + { + "case": 29, + "name": "<> for read/write named pipes", + "class": "host", + "on": "stdout" + }, + { + "case": 30, + "name": "&>> appends stdout and stderr", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "sh-usage": { + "summary": { + "cases": 24, + "bashkit_pass": 17, + "bashkit_pass_pct": 70.8, + "bash_pass": 24, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 17, + "pass_rate": 70.8 + }, + "missed": [ + { + "case": 2, + "name": "sh +c is accepted", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "sh - and sh -- stop flag processing", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Set LC_ALL LC_CTYPE LC_COLLATE LANG - affects glob ?", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "LC_CTYPE=invalid", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "sh -c -- 'echo hi' does not run anything (#2637)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 22, + "name": "'sh -c -z' does not try to run -z", + "class": "behavior", + "on": "stdout" + }, + { + "case": 23, + "name": "sh -c -x 'echo hi' - order is reversed", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-printf": { + "summary": { + "cases": 63, + "bashkit_pass": 55, + "bashkit_pass_pct": 87.3, + "bash_pass": 58, + "bash_pass_pct": 92.1, + "bashkit_pass_of_bash": 52, + "pass_rate": 89.7 + }, + "missed": [ + { + "case": 5, + "name": "dynamic declare instead of %s", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "%06s is no-op", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "printf %6.0s and %0.0s", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "printf %6.s and %0.s", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "printf octal backslash escapes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 51, + "name": "bash truncates long strftime string at 128", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 26, + 27, + 46, + 47, + 48 + ] + }, + "builtin-process": { + "summary": { + "cases": 26, + "bashkit_pass": 19, + "bashkit_pass_pct": 73.1, + "bash_pass": 24, + "bash_pass_pct": 92.3, + "bashkit_pass_of_bash": 18, + "pass_rate": 75.0 + }, + "missed": [ + { + "case": 3, + "name": "exec builtin accepts --", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "exec -- 2>&1", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "exec -a sets argv[0]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "arg that would overflow 64 bits is detected", + "class": "processes", + "on": "stdout" + }, + { + "case": 21, + "name": "ulimit -f 1 prevents files larger 512 bytes", + "class": "processes", + "on": "stdout" + }, + { + "case": 25, + "name": "ulimit -n limits file descriptors", + "class": "processes", + "on": "stdout" + } + ], + "bash_fails": [ + 9, + 22 + ] + }, + "builtin-vars": { + "summary": { + "cases": 41, + "bashkit_pass": 33, + "bashkit_pass_pct": 80.5, + "bash_pass": 39, + "bash_pass_pct": 95.1, + "bashkit_pass_of_bash": 33, + "pass_rate": 84.6 + }, + "missed": [ + { + "case": 16, + "name": "Make an existing local variable readonly", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "assign to readonly variable - errexit", + "class": "behavior", + "on": "status" + }, + { + "case": 22, + "name": "Unset a function without -f", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "unset -v assoc (related to issue #661)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 35, + "name": "Unset array member with dynamic parsing", + "class": "behavior", + "on": "stdout" + }, + { + "case": 39, + "name": "unset a[-1] (bf.bash regression)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 26, + 32 + ] + }, + "glob": { + "summary": { + "cases": 39, + "bashkit_pass": 31, + "bashkit_pass_pct": 79.5, + "bash_pass": 37, + "bash_pass_pct": 94.9, + "bashkit_pass_of_bash": 31, + "pass_rate": 83.8 + }, + "missed": [ + { + "case": 5, + "name": "looks like glob at the start, but isn't", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "looks like glob plus negation at the start, but isn't", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "glob can expand to command and arg", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "glob with escaped - in char class", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "Glob of unescaped [[] and []]", + "class": "behavior", + "on": "stdout" + }, + { + "case": 38, + "name": "shopt -u globskipdots shows . and ..", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 23, + 31 + ] + }, + "here-doc": { + "summary": { + "cases": 36, + "bashkit_pass": 30, + "bashkit_pass_pct": 83.3, + "bash_pass": 36, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 30, + "pass_rate": 83.3 + }, + "missed": [ + { + "case": 3, + "name": "Here doc from another input file descriptor", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "Multiple here docs with different descriptors", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Here doc line continuation", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Here doc with line continuation, then pipe. Syntax error.", + "class": "behavior", + "on": "status" + }, + { + "case": 32, + "name": "Multiple here docs in pipeline", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "Multiple here docs in pipeline on multiple lines", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "sh-options-bash": { + "summary": { + "cases": 9, + "bashkit_pass": 3, + "bashkit_pass_pct": 33.3, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 3, + "pass_rate": 33.3 + }, + "missed": [ + { + "case": 0, + "name": "SHELLOPTS is updated when options are changed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "SHELLOPTS is readonly", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "SHELLOPTS and BASHOPTS are non-empty", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "SHELLOPTS reflects flags like sh -x", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "export SHELLOPTS does cross-process tracing", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "export SHELLOPTS does cross-process tracing with bash", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "toysh-posix": { + "summary": { + "cases": 23, + "bashkit_pass": 17, + "bashkit_pass_pct": 73.9, + "bash_pass": 23, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 17, + "pass_rate": 73.9 + }, + "missed": [ + { + "case": 11, + "name": "IFS - http://landley.net/notes.html#05-03-2020", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "IFS=x and '' and unquoted $@ - reduction of case above - copied into spec/word-split", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "for loop parsing - http://landley.net/notes.html#04-03-2020", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "IFS - http://landley.net/notes.html#15-02-2020 (TODO: osh)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "IFS 2 - copied into spec/word-split", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "IFS 4", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "xtrace": { + "summary": { + "cases": 19, + "bashkit_pass": 13, + "bashkit_pass_pct": 68.4, + "bash_pass": 19, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 13, + "pass_rate": 68.4 + }, + "missed": [ + { + "case": 0, + "name": "unset PS4", + "class": "interactive", + "on": "stderr" + }, + { + "case": 1, + "name": "set -o verbose prints unevaluated code", + "class": "behavior", + "on": "stderr" + }, + { + "case": 2, + "name": "xtrace with unprintable chars", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "Assignments and assign builtins", + "class": "behavior", + "on": "stderr" + }, + { + "case": 10, + "name": "[[ ]]", + "class": "behavior", + "on": "stderr" + }, + { + "case": 17, + "name": "Regression: xtrace for \"declare -a a+=(v)\"", + "class": "behavior", + "on": "stderr" + } + ], + "bash_fails": [] + }, + "arith-context": { + "summary": { + "cases": 16, + "bashkit_pass": 11, + "bashkit_pass_pct": 68.8, + "bash_pass": 16, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 11, + "pass_rate": 68.8 + }, + "missed": [ + { + "case": 6, + "name": "Ambiguous colon in slice", + "class": "behavior", + "on": "stdout" + }, + { + "case": 9, + "name": "$[ is a synonym for $((", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "$[$var is a synonym for $(($var (#2426)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "$[$undefined] is a synonym for $(($undefined (#2566)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "Empty expression a[]", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "assign": { + "summary": { + "cases": 48, + "bashkit_pass": 40, + "bashkit_pass_pct": 83.3, + "bash_pass": 45, + "bash_pass_pct": 93.8, + "bashkit_pass_of_bash": 40, + "pass_rate": 88.9 + }, + "missed": [ + { + "case": 8, + "name": "Escaped = in command name", + "class": "behavior", + "on": "stdout" + }, + { + "case": 25, + "name": "Test above without 'local' (which is not POSIX)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 30, + "name": "assignment using dynamic keyword (splits in most shells, not in zsh/osh)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 32, + "name": "assign and glob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "declare and glob", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 24, + 27, + 46 + ] + }, + "blog1": { + "summary": { + "cases": 9, + "bashkit_pass": 4, + "bashkit_pass_pct": 44.4, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 44.4 + }, + "missed": [ + { + "case": 1, + "name": "${###}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "${####}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "${##2}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "${###2}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "${#1#'###'}", + "class": "behavior", + "on": "status" + } + ], + "bash_fails": [] + }, + "builtin-meta": { + "summary": { + "cases": 18, + "bashkit_pass": 13, + "bashkit_pass_pct": 72.2, + "bash_pass": 18, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 13, + "pass_rate": 72.2 + }, + "missed": [ + { + "case": 3, + "name": "command -v doesn't find non-executable file", + "class": "host", + "on": "stdout" + }, + { + "case": 5, + "name": "command -V", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "command -V nonexistent", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "command -p (override existing program)", + "class": "host", + "on": "stdout" + }, + { + "case": 11, + "name": "command -p (hide tool in custom path)", + "class": "host", + "on": "status" + } + ], + "bash_fails": [] + }, + "builtin-meta-assign": { + "summary": { + "cases": 11, + "bashkit_pass": 6, + "bashkit_pass_pct": 54.5, + "bash_pass": 11, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 6, + "pass_rate": 54.5 + }, + "missed": [ + { + "case": 0, + "name": "builtin declare a=(x y) is allowed", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "export, builtin export", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "\\builtin declare - ble.sh relies on it", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "\\command readonly - similar issue", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "Dynamic $c readonly - bash and dash change behavior, mksh bug", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "loop": { + "summary": { + "cases": 29, + "bashkit_pass": 24, + "bashkit_pass_pct": 82.8, + "bash_pass": 29, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 24, + "pass_rate": 82.8 + }, + "missed": [ + { + "case": 14, + "name": "continue in subshell", + "class": "behavior", + "on": "stdout" + }, + { + "case": 15, + "name": "continue in subshell aborts with errexit", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "too many args to continue", + "class": "behavior", + "on": "stdout" + }, + { + "case": 18, + "name": "break in condition of loop", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "break in condition of nested loop", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "nameref": { + "summary": { + "cases": 32, + "bashkit_pass": 27, + "bashkit_pass_pct": 84.4, + "bash_pass": 32, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 27, + "pass_rate": 84.4 + }, + "missed": [ + { + "case": 15, + "name": "-n attribute on array is hard error, not a warning", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "exported nameref", + "class": "behavior", + "on": "stdout" + }, + { + "case": 26, + "name": "a[expr] in nameref", + "class": "behavior", + "on": "stdout" + }, + { + "case": 27, + "name": "a[@] in nameref", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "bad mutation through nameref: ref[0]= where ref is array[0]", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "paren-ambiguity": { + "summary": { + "cases": 9, + "bashkit_pass": 4, + "bashkit_pass_pct": 44.4, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 44.4 + }, + "missed": [ + { + "case": 0, + "name": "(( closed with ) ) after multiple lines is command - #2337", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "$(( closed with ) ) after multiple lines is command - #2337", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "(((grep example - 4+ instances in regtest/aports - #2337", + "class": "processes", + "on": "status" + }, + { + "case": 5, + "name": "((gzip example - zdiff package - #2337", + "class": "behavior", + "on": "stdout" + }, + { + "case": 7, + "name": "((test example - liblo package - #2337", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "parse-errors": { + "summary": { + "cases": 27, + "bashkit_pass": 20, + "bashkit_pass_pct": 74.1, + "bash_pass": 25, + "bash_pass_pct": 92.6, + "bashkit_pass_of_bash": 20, + "pass_rate": 80.0 + }, + "missed": [ + { + "case": 11, + "name": "} on the second line", + "class": "behavior", + "on": "status" + }, + { + "case": 16, + "name": "misplaced parentheses are not a subshell", + "class": "behavior", + "on": "status" + }, + { + "case": 20, + "name": "empty clause in [[", + "class": "behavior", + "on": "status" + }, + { + "case": 22, + "name": "array literal inside array is a parse error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 25, + "name": "%foo=() is parse error (regression)", + "class": "behavior", + "on": "status" + } + ], + "bash_fails": [ + 0, + 1 + ] + }, + "ble-features": { + "summary": { + "cases": 9, + "bashkit_pass": 5, + "bashkit_pass_pct": 55.6, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 55.6 + }, + "missed": [ + { + "case": 1, + "name": "[bash_unset] local-unset / dynamic-unset for localvar (mutated from tempenv)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "[bash_unset] local-unset / dynamic-unset for tempenv", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "[bash_unset] function call with tempenv vs tempenv-eval", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "[bash_unset] localvar-inherit from tempenv", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "ble-unset": { + "summary": { + "cases": 5, + "bashkit_pass": 1, + "bashkit_pass_pct": 20.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 1, + "pass_rate": 20.0 + }, + "missed": [ + { + "case": 1, + "name": "[bash_unset] local-unset / dynamic-unset for localvar on nested-context", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "[bash_unset] dynamic-unset for nested localvars", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "[bash_unset] dynamic-unset for nested tempenvs", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "[bash_unset] local-unset for nested tempenvs", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "bool-parse": { + "summary": { + "cases": 8, + "bashkit_pass": 4, + "bashkit_pass_pct": 50.0, + "bash_pass": 8, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 0, + "name": "test builtin - Unexpected trailing word '--' (#2409)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "test builtin: ( = ) is confusing: equality test or non-empty string test", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "test builtin: ( == ) is confusing: equality test or non-empty string test", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "Not allowed: [[ ) ]] and [[ ( ]]", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-dirs": { + "summary": { + "cases": 18, + "bashkit_pass": 14, + "bashkit_pass_pct": 77.8, + "bash_pass": 18, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 14, + "pass_rate": 77.8 + }, + "missed": [ + { + "case": 1, + "name": "pushd usage", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "popd usage error", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "pushd does not take more than one argument", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "dirs does not take arguments", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-kill": { + "summary": { + "cases": 20, + "bashkit_pass": 16, + "bashkit_pass_pct": 80.0, + "bash_pass": 20, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 16, + "pass_rate": 80.0 + }, + "missed": [ + { + "case": 8, + "name": "kill -l 10 TERM translates between names and numbers", + "class": "signals", + "on": "stdout" + }, + { + "case": 9, + "name": "kill -L checks for invalid input", + "class": "signals", + "on": "stdout" + }, + { + "case": 12, + "name": "kill -l 0 returns EXIT", + "class": "signals", + "on": "stdout" + }, + { + "case": 13, + "name": "kill -l 0 INT lists both signals", + "class": "signals", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-set": { + "summary": { + "cases": 24, + "bashkit_pass": 20, + "bashkit_pass_pct": 83.3, + "bash_pass": 24, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 20, + "pass_rate": 83.3 + }, + "missed": [ + { + "case": 6, + "name": "set -u with undefined var in interactive shell does NOT exit the interpreter", + "class": "interactive", + "on": "stdout" + }, + { + "case": 12, + "name": "'set' and 'eval' round trip", + "class": "behavior", + "on": "stdout" + }, + { + "case": 16, + "name": "A single + is an ignored flag; not an argument", + "class": "behavior", + "on": "stdout" + }, + { + "case": 17, + "name": "set - + and + -", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-special": { + "summary": { + "cases": 12, + "bashkit_pass": 8, + "bashkit_pass_pct": 66.7, + "bash_pass": 12, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 8, + "pass_rate": 66.7 + }, + "missed": [ + { + "case": 1, + "name": "Prefix assignments persist after special builtins, like : (set -o posix)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "Prefix assignments persist after readonly, but NOT exported (set -o posix)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "Special builtins can't be redefined as shell functions (set -o posix)", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "bash 'type' gets confused - says 'function', but runs builtin", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "assign-dialects": { + "summary": { + "cases": 4, + "bashkit_pass": 1, + "bashkit_pass_pct": 25.0, + "bash_pass": 4, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 1, + "pass_rate": 25.0 + }, + "missed": [ + { + "case": 1, + "name": "test -v with strings", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "test -v with arrays", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "test -v with assoc arrays", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "blog2": { + "summary": { + "cases": 8, + "bashkit_pass": 5, + "bashkit_pass_pct": 62.5, + "bash_pass": 8, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 62.5 + }, + "missed": [ + { + "case": 4, + "name": "-a -a -a -a -a", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "-a -a -a -a -a -a", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "-a -a -a -a -a -a -a", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-bash": { + "summary": { + "cases": 13, + "bashkit_pass": 10, + "bashkit_pass_pct": 76.9, + "bash_pass": 13, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 10, + "pass_rate": 76.9 + }, + "missed": [ + { + "case": 0, + "name": "help", + "class": "behavior", + "on": "stderr" + }, + { + "case": 1, + "name": "bad help topic", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "mapfile (delimiter): -d '' (null-separated)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-misc": { + "summary": { + "cases": 7, + "bashkit_pass": 4, + "bashkit_pass_pct": 57.1, + "bash_pass": 7, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 57.1 + }, + "missed": [ + { + "case": 0, + "name": "history builtin usage", + "class": "interactive", + "on": "stdout" + }, + { + "case": 1, + "name": "Print shell strings with weird chars: set and printf %q and ${x@Q}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "time pipeline", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-trap-err": { + "summary": { + "cases": 23, + "bashkit_pass": 19, + "bashkit_pass_pct": 82.6, + "bash_pass": 22, + "bash_pass_pct": 95.7, + "bashkit_pass_of_bash": 19, + "pass_rate": 86.4 + }, + "missed": [ + { + "case": 6, + "name": "trap ERR pipelines without simple commands", + "class": "behavior", + "on": "stdout" + }, + { + "case": 12, + "name": "set -o errtrace: trap ERR with &", + "class": "processes", + "on": "stdout" + }, + { + "case": 16, + "name": "trap ERR with \"atoms\": assignment (( [[", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 18 + ] + }, + "builtin-umask": { + "summary": { + "cases": 24, + "bashkit_pass": 21, + "bashkit_pass_pct": 87.5, + "bash_pass": 24, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 21, + "pass_rate": 87.5 + }, + "missed": [ + { + "case": 10, + "name": "'umask 0002' sets the umask", + "class": "host", + "on": "stdout" + }, + { + "case": 11, + "name": "set umask with symbolic mode: g-w,o-w", + "class": "host", + "on": "stdout" + }, + { + "case": 13, + "name": "umask with too many arguments (i.e. extra spaces)", + "class": "host", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "extglob-files": { + "summary": { + "cases": 23, + "bashkit_pass": 20, + "bashkit_pass_pct": 87.0, + "bash_pass": 23, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 20, + "pass_rate": 87.0 + }, + "missed": [ + { + "case": 15, + "name": "Extended glob in same word as array", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "no match", + "class": "behavior", + "on": "stdout" + }, + { + "case": 21, + "name": "noglob", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "fatal-errors": { + "summary": { + "cases": 5, + "bashkit_pass": 2, + "bashkit_pass_pct": 40.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 40.0 + }, + "missed": [ + { + "case": 1, + "name": "Unrecoverable: divide by zero in conditional word", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "Unrecoverable: ${undef?message}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "${undef} with nounset", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "func-parsing": { + "summary": { + "cases": 15, + "bashkit_pass": 12, + "bashkit_pass_pct": 80.0, + "bash_pass": 15, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 12, + "pass_rate": 80.0 + }, + "missed": [ + { + "case": 8, + "name": "= in function name", + "class": "behavior", + "on": "status" + }, + { + "case": 9, + "name": "Function name with $", + "class": "behavior", + "on": "status" + }, + { + "case": 10, + "name": "Function name with command sub", + "class": "behavior", + "on": "status" + } + ], + "bash_fails": [] + }, + "nix-idioms": { + "summary": { + "cases": 6, + "bashkit_pass": 3, + "bashkit_pass_pct": 50.0, + "bash_pass": 6, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 3, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 0, + "name": "var ref to array 'preHooks[@]'", + "class": "behavior", + "on": "stdout" + }, + { + "case": 1, + "name": "Similar to above with set -u", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "${!ref} to undefined string var is fatal, INCONSISTENT with array", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "quote": { + "summary": { + "cases": 35, + "bashkit_pass": 32, + "bashkit_pass_pct": 91.4, + "bash_pass": 35, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 32, + "pass_rate": 91.4 + }, + "missed": [ + { + "case": 27, + "name": "$'' octal escapes don't have leading 0", + "class": "behavior", + "on": "stdout" + }, + { + "case": 29, + "name": "OSH allows invalid backslashes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 33, + "name": "$'' supports \\cA escape for Ctrl-A - mask with 0x1f", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "redirect-command": { + "summary": { + "cases": 23, + "bashkit_pass": 20, + "bashkit_pass_pct": 87.0, + "bash_pass": 23, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 20, + "pass_rate": 87.0 + }, + "missed": [ + { + "case": 8, + "name": "Redirect in the middle of two assignments", + "class": "behavior", + "on": "stdout" + }, + { + "case": 11, + "name": "Redirect in function body is evaluated multiple times", + "class": "behavior", + "on": "stdout" + }, + { + "case": 13, + "name": "redirect bash extensions: [[ (( for ((", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "serialize": { + "summary": { + "cases": 10, + "bashkit_pass": 7, + "bashkit_pass_pct": 70.0, + "bash_pass": 10, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 70.0 + }, + "missed": [ + { + "case": 3, + "name": "printf %q unprintable", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "printf %q invalid unicode", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "${var@Q}", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "toysh": { + "summary": { + "cases": 8, + "bashkit_pass": 5, + "bashkit_pass_pct": 62.5, + "bash_pass": 8, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 62.5 + }, + "missed": [ + { + "case": 3, + "name": "char class / extglob", + "class": "behavior", + "on": "stdout" + }, + { + "case": 5, + "name": "Brace Expansion", + "class": "host", + "on": "stdout" + }, + { + "case": 7, + "name": "slice of @ and @ - http://landley.net/notes.html#23-04-2020", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-op-len": { + "summary": { + "cases": 9, + "bashkit_pass": 6, + "bashkit_pass_pct": 66.7, + "bash_pass": 8, + "bash_pass_pct": 88.9, + "bashkit_pass_of_bash": 5, + "pass_rate": 62.5 + }, + "missed": [ + { + "case": 3, + "name": "String length with incomplete utf-8", + "class": "behavior", + "on": "stdout" + }, + { + "case": 4, + "name": "String length with invalid utf-8 continuation bytes", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "Length of undefined variable with nounset", + "class": "behavior", + "on": "status" + } + ], + "bash_fails": [ + 8 + ] + }, + "arg-parse": { + "summary": { + "cases": 3, + "bashkit_pass": 1, + "bashkit_pass_pct": 33.3, + "bash_pass": 3, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 1, + "pass_rate": 33.3 + }, + "missed": [ + { + "case": 1, + "name": "shift 1 extra", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "continue 1 extra, break, etc.", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "array-basic": { + "summary": { + "cases": 5, + "bashkit_pass": 3, + "bashkit_pass_pct": 60.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 3, + "pass_rate": 60.0 + }, + "missed": [ + { + "case": 1, + "name": "${a[@]} and ${a[*]}", + "class": "behavior", + "on": "stdout" + }, + { + "case": 2, + "name": "4 ways to interpolate empty array", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "assign-deferred": { + "summary": { + "cases": 9, + "bashkit_pass": 7, + "bashkit_pass_pct": 77.8, + "bash_pass": 9, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 77.8 + }, + "missed": [ + { + "case": 7, + "name": "is 'builtin' prefix and array allowed? OSH is smarter", + "class": "behavior", + "on": "stdout" + }, + { + "case": 8, + "name": "is 'command' prefix and array allowed? OSH is smarter", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-getopts": { + "summary": { + "cases": 31, + "bashkit_pass": 29, + "bashkit_pass_pct": 93.5, + "bash_pass": 31, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 29, + "pass_rate": 93.5 + }, + "missed": [ + { + "case": 6, + "name": "getopts with invalid variable name", + "class": "behavior", + "on": "stdout" + }, + { + "case": 19, + "name": "flag and arg: -c10", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "builtin-type": { + "summary": { + "cases": 6, + "bashkit_pass": 4, + "bashkit_pass_pct": 66.7, + "bash_pass": 6, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 66.7 + }, + "missed": [ + { + "case": 1, + "name": "type -> alias external", + "class": "host", + "on": "stdout" + }, + { + "case": 2, + "name": "type of relative path", + "class": "host", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "command-sub": { + "summary": { + "cases": 30, + "bashkit_pass": 26, + "bashkit_pass_pct": 86.7, + "bash_pass": 28, + "bash_pass_pct": 93.3, + "bashkit_pass_of_bash": 26, + "pass_rate": 92.9 + }, + "missed": [ + { + "case": 20, + "name": "Quoting \\ within ``", + "class": "behavior", + "on": "stdout" + }, + { + "case": 24, + "name": "Quoting non-special characters within ``", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 11, + 12 + ] + }, + "divergence": { + "summary": { + "cases": 4, + "bashkit_pass": 2, + "bashkit_pass_pct": 50.0, + "bash_pass": 4, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 1, + "name": "!( as negation and subshell versus extended glob - #2463", + "class": "behavior", + "on": "stdout" + }, + { + "case": 3, + "name": "builtin cat crashes a subshell (#2530)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "errexit": { + "summary": { + "cases": 35, + "bashkit_pass": 33, + "bashkit_pass_pct": 94.3, + "bash_pass": 35, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 33, + "pass_rate": 94.3 + }, + "missed": [ + { + "case": 33, + "name": "set -e in function #2", + "class": "behavior", + "on": "stdout" + }, + { + "case": 34, + "name": "Command sub exit code is lost", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "exit-status": { + "summary": { + "cases": 11, + "bashkit_pass": 9, + "bashkit_pass_pct": 81.8, + "bash_pass": 11, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 9, + "pass_rate": 81.8 + }, + "missed": [ + { + "case": 7, + "name": "If subshell true WITH ARGUMENT", + "class": "behavior", + "on": "stdout" + }, + { + "case": 10, + "name": "More test cases with empty argv", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "glob-bash": { + "summary": { + "cases": 8, + "bashkit_pass": 6, + "bashkit_pass_pct": 75.0, + "bash_pass": 8, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 6, + "pass_rate": 75.0 + }, + "missed": [ + { + "case": 3, + "name": "shopt -s failglob in array literal context", + "class": "behavior", + "on": "stdout" + }, + { + "case": 6, + "name": "shopt -s failglob behavior on single line with semicolon", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-op-slice": { + "summary": { + "cases": 22, + "bashkit_pass": 20, + "bashkit_pass_pct": 90.9, + "bash_pass": 22, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 20, + "pass_rate": 90.9 + }, + "missed": [ + { + "case": 9, + "name": "Slice undefined", + "class": "behavior", + "on": "stdout" + }, + { + "case": 20, + "name": "${array[@]:} vs ${array[@]: } - bash and zsh inconsistent", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "append": { + "summary": { + "cases": 20, + "bashkit_pass": 19, + "bashkit_pass_pct": 95.0, + "bash_pass": 20, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 19, + "pass_rate": 95.0 + }, + "missed": [ + { + "case": 11, + "name": "Try to append list to element", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "dparen": { + "summary": { + "cases": 15, + "bashkit_pass": 13, + "bashkit_pass_pct": 86.7, + "bash_pass": 14, + "bash_pass_pct": 93.3, + "bashkit_pass_of_bash": 13, + "pass_rate": 92.9 + }, + "missed": [ + { + "case": 11, + "name": "(( )) with redirect", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 10 + ] + }, + "errexit-osh": { + "summary": { + "cases": 35, + "bashkit_pass": 34, + "bashkit_pass_pct": 97.1, + "bash_pass": 35, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 34, + "pass_rate": 97.1 + }, + "missed": [ + { + "case": 30, + "name": "command_sub_errexit and command sub in array", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "extglob-match": { + "summary": { + "cases": 29, + "bashkit_pass": 28, + "bashkit_pass_pct": 96.6, + "bash_pass": 29, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 28, + "pass_rate": 96.6 + }, + "missed": [ + { + "case": 21, + "name": "Turning extglob on changes the meaning of [[ !(str) ]] in bash", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "for-expr": { + "summary": { + "cases": 9, + "bashkit_pass": 7, + "bashkit_pass_pct": 77.8, + "bash_pass": 8, + "bash_pass_pct": 88.9, + "bashkit_pass_of_bash": 7, + "pass_rate": 87.5 + }, + "missed": [ + { + "case": 2, + "name": "Accepts { } syntax too", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 6 + ] + }, + "history-expand": { + "summary": { + "cases": 1, + "bashkit_pass": 0, + "bashkit_pass_pct": 0.0, + "bash_pass": 1, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 0, + "pass_rate": 0.0 + }, + "missed": [ + { + "case": 0, + "name": "!! expansion", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "pipeline": { + "summary": { + "cases": 26, + "bashkit_pass": 24, + "bashkit_pass_pct": 92.3, + "bash_pass": 25, + "bash_pass_pct": 96.2, + "bashkit_pass_of_bash": 24, + "pass_rate": 96.0 + }, + "missed": [ + { + "case": 6, + "name": "Initial value of PIPESTATUS is empty string", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 10 + ] + }, + "process-sub": { + "summary": { + "cases": 9, + "bashkit_pass": 8, + "bashkit_pass_pct": 88.9, + "bash_pass": 8, + "bash_pass_pct": 88.9, + "bashkit_pass_of_bash": 7, + "pass_rate": 87.5 + }, + "missed": [ + { + "case": 2, + "name": "Process sub from shell to stdin", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 4 + ] + }, + "smoke": { + "summary": { + "cases": 18, + "bashkit_pass": 17, + "bashkit_pass_pct": 94.4, + "bash_pass": 18, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 17, + "pass_rate": 94.4 + }, + "missed": [ + { + "case": 11, + "name": "Here doc with redirect", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "unicode": { + "summary": { + "cases": 7, + "bashkit_pass": 1, + "bashkit_pass_pct": 14.3, + "bash_pass": 2, + "bash_pass_pct": 28.6, + "bashkit_pass_of_bash": 1, + "pass_rate": 50.0 + }, + "missed": [ + { + "case": 0, + "name": "OSH source code doesn't have to be valid Unicode (like other shells)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [ + 2, + 3, + 4, + 5, + 6 + ] + }, + "var-op-strip": { + "summary": { + "cases": 29, + "bashkit_pass": 28, + "bashkit_pass_pct": 96.6, + "bash_pass": 29, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 28, + "pass_rate": 96.6 + }, + "missed": [ + { + "case": 26, + "name": "Strip Right Brace (#702)", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "var-sub": { + "summary": { + "cases": 6, + "bashkit_pass": 5, + "bashkit_pass_pct": 83.3, + "bash_pass": 6, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 83.3 + }, + "missed": [ + { + "case": 4, + "name": "Descriptor redirect to bad \"$@\"", + "class": "behavior", + "on": "status" + } + ], + "bash_fails": [] + }, + "vars-bash": { + "summary": { + "cases": 1, + "bashkit_pass": 0, + "bashkit_pass_pct": 0.0, + "bash_pass": 1, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 0, + "pass_rate": 0.0 + }, + "missed": [ + { + "case": 0, + "name": "$SHELL is set to what is in /etc/passwd", + "class": "interactive", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "word-eval": { + "summary": { + "cases": 8, + "bashkit_pass": 7, + "bashkit_pass_pct": 87.5, + "bash_pass": 8, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 87.5 + }, + "missed": [ + { + "case": 5, + "name": "Default values -- more cases", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "zsh-idioms": { + "summary": { + "cases": 3, + "bashkit_pass": 2, + "bashkit_pass_pct": 66.7, + "bash_pass": 3, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 66.7 + }, + "missed": [ + { + "case": 2, + "name": "zsh var sub is rejected at runtime", + "class": "behavior", + "on": "stdout" + } + ], + "bash_fails": [] + }, + "arith-dynamic": { + "summary": { + "cases": 4, + "bashkit_pass": 4, + "bashkit_pass_pct": 100.0, + "bash_pass": 4, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "array-compat": { + "summary": { + "cases": 12, + "bashkit_pass": 12, + "bashkit_pass_pct": 100.0, + "bash_pass": 12, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 12, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "builtin-echo": { + "summary": { + "cases": 27, + "bashkit_pass": 27, + "bashkit_pass_pct": 100.0, + "bash_pass": 27, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 27, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "builtin-times": { + "summary": { + "cases": 1, + "bashkit_pass": 1, + "bashkit_pass_pct": 100.0, + "bash_pass": 1, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 1, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "case_": { + "summary": { + "cases": 13, + "bashkit_pass": 13, + "bashkit_pass_pct": 100.0, + "bash_pass": 13, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 13, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "command-parsing": { + "summary": { + "cases": 5, + "bashkit_pass": 5, + "bashkit_pass_pct": 100.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "comments": { + "summary": { + "cases": 2, + "bashkit_pass": 2, + "bashkit_pass_pct": 100.0, + "bash_pass": 2, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "empty-bodies": { + "summary": { + "cases": 3, + "bashkit_pass": 3, + "bashkit_pass_pct": 100.0, + "bash_pass": 3, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 3, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "explore-parsing": { + "summary": { + "cases": 5, + "bashkit_pass": 5, + "bashkit_pass_pct": 100.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "globstar": { + "summary": { + "cases": 5, + "bashkit_pass": 5, + "bashkit_pass_pct": 100.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "if_": { + "summary": { + "cases": 5, + "bashkit_pass": 5, + "bashkit_pass_pct": 100.0, + "bash_pass": 5, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 5, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "known-differences": { + "summary": { + "cases": 2, + "bashkit_pass": 2, + "bashkit_pass_pct": 100.0, + "bash_pass": 2, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "let": { + "summary": { + "cases": 2, + "bashkit_pass": 2, + "bashkit_pass_pct": 100.0, + "bash_pass": 2, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "nocasematch-match": { + "summary": { + "cases": 6, + "bashkit_pass": 6, + "bashkit_pass_pct": 100.0, + "bash_pass": 6, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 6, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "posix": { + "summary": { + "cases": 15, + "bashkit_pass": 15, + "bashkit_pass_pct": 100.0, + "bash_pass": 15, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 15, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "print-source-code": { + "summary": { + "cases": 4, + "bashkit_pass": 4, + "bashkit_pass_pct": 100.0, + "bash_pass": 4, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "sh-func": { + "summary": { + "cases": 12, + "bashkit_pass": 12, + "bashkit_pass_pct": 100.0, + "bash_pass": 12, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 12, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "strict-options": { + "summary": { + "cases": 17, + "bashkit_pass": 15, + "bashkit_pass_pct": 88.2, + "bash_pass": 15, + "bash_pass_pct": 88.2, + "bashkit_pass_of_bash": 15, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [ + 3, + 5 + ] + }, + "subshell": { + "summary": { + "cases": 2, + "bashkit_pass": 2, + "bashkit_pass_pct": 100.0, + "bash_pass": 2, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 2, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "temp-binding": { + "summary": { + "cases": 4, + "bashkit_pass": 4, + "bashkit_pass_pct": 100.0, + "bash_pass": 4, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 4, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "type-compat": { + "summary": { + "cases": 7, + "bashkit_pass": 5, + "bashkit_pass_pct": 71.4, + "bash_pass": 5, + "bash_pass_pct": 71.4, + "bashkit_pass_of_bash": 5, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [ + 3, + 5 + ] + }, + "var-num": { + "summary": { + "cases": 7, + "bashkit_pass": 7, + "bashkit_pass_pct": 100.0, + "bash_pass": 7, + "bash_pass_pct": 100.0, + "bashkit_pass_of_bash": 7, + "pass_rate": 100.0 + }, + "missed": [], + "bash_fails": [] + }, + "whitespace": { + "summary": { + "cases": 5, + "bashkit_pass": 0, + "bashkit_pass_pct": 0.0, + "bash_pass": 0, + "bash_pass_pct": 0.0, + "bashkit_pass_of_bash": 0, + "pass_rate": 0.0 + }, + "missed": [], + "bash_fails": [ + 0, + 1, + 2, + 3, + 4 + ] + } + } +} diff --git a/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md b/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md new file mode 100644 index 000000000..774337a36 --- /dev/null +++ b/scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md @@ -0,0 +1,167 @@ +# Oils spec pass rate + +- Oils revision: [`57d3f0d088c3`](https://github.com/oils-for-unix/oils/tree/57d3f0d088c36340bc4b3038208e58ea02acf15c) +- bashkit commit: `f86bd7c81ae3` +- bash: 5.2.21(1)-release +- Spec files: 132 (bash column; 91 without one skipped) +- Runtime: 27.3 s, 4 jobs, 10 s per-case timeout + +## Headline + +**74.2%** — bashkit passes 1997 of the 2693 cases real bash passes. + +| Shell | Pass | Cases | % | +|-------|-----:|------:|--:| +| bash | 2693 | 2759 | 97.6 | +| bashkit | 2003 | 2759 | 72.6 | + +## bashkit failures by class + +Heuristic keyword classes; every case still counts in the rate. + +| Class | Failing cases | +|-------|--------------:| +| behavior | 530 | +| interactive | 110 | +| host | 27 | +| processes | 19 | +| signals | 10 | + +## Per spec file + +Sorted by cases bashkit misses among those bash passes. + +| Spec file | Cases | bash | bashkit | Missed | Rate % | +|-----------|------:|-----:|--------:|-------:|-------:| +| builtin-completion | 51 | 51 | 17 | 34 | 33.3 | +| array | 78 | 78 | 53 | 25 | 67.9 | +| prompt | 33 | 29 | 4 | 25 | 13.8 | +| builtin-trap-bash | 23 | 23 | 4 | 19 | 17.4 | +| globignore | 18 | 18 | 2 | 16 | 11.1 | +| word-split | 55 | 53 | 37 | 16 | 69.8 | +| var-ref | 31 | 30 | 15 | 15 | 50.0 | +| builtin-cd | 30 | 28 | 14 | 14 | 50.0 | +| alias | 48 | 48 | 35 | 13 | 72.9 | +| dbracket | 49 | 49 | 36 | 13 | 73.5 | +| builtin-fc | 14 | 14 | 2 | 12 | 14.3 | +| builtin-history | 17 | 17 | 5 | 12 | 29.4 | +| builtin-type-bash | 31 | 31 | 19 | 12 | 61.3 | +| interactive | 18 | 18 | 7 | 11 | 38.9 | +| sh-options | 39 | 39 | 28 | 11 | 71.8 | +| vars-special | 42 | 40 | 29 | 11 | 72.5 | +| array-assign | 11 | 11 | 1 | 10 | 9.1 | +| array-sparse | 40 | 40 | 30 | 10 | 75.0 | +| var-sub-quote | 41 | 41 | 31 | 10 | 75.6 | +| array-assoc | 42 | 36 | 27 | 9 | 75.0 | +| builtin-bind | 9 | 9 | 0 | 9 | 0.0 | +| builtin-read | 64 | 64 | 55 | 9 | 85.9 | +| builtin-trap | 33 | 33 | 24 | 9 | 72.7 | +| introspect | 13 | 12 | 3 | 9 | 25.0 | +| nul-bytes | 16 | 16 | 7 | 9 | 43.8 | +| redirect-multi | 13 | 13 | 4 | 9 | 30.8 | +| regex | 37 | 37 | 28 | 9 | 75.7 | +| tilde | 14 | 14 | 5 | 9 | 35.7 | +| var-op-bash | 27 | 26 | 17 | 9 | 65.4 | +| var-op-patsub | 28 | 28 | 19 | 9 | 67.9 | +| ysh-builtin-private | 23 | 22 | 13 | 9 | 59.1 | +| arith | 74 | 74 | 66 | 8 | 89.2 | +| assign-extended | 39 | 35 | 27 | 8 | 77.1 | +| brace-expansion | 55 | 55 | 47 | 8 | 85.5 | +| bugs | 29 | 29 | 21 | 8 | 72.4 | +| var-op-test | 37 | 37 | 29 | 8 | 78.4 | +| array-literal | 19 | 19 | 12 | 7 | 63.2 | +| background | 27 | 27 | 20 | 7 | 74.1 | +| ble-idioms | 26 | 26 | 19 | 7 | 73.1 | +| builtin-bracket | 52 | 50 | 43 | 7 | 86.0 | +| builtin-eval-source | 23 | 21 | 14 | 7 | 66.7 | +| command_ | 16 | 16 | 9 | 7 | 56.2 | +| redirect | 41 | 41 | 34 | 7 | 82.9 | +| sh-usage | 24 | 24 | 17 | 7 | 70.8 | +| builtin-printf | 63 | 58 | 55 | 6 | 89.7 | +| builtin-process | 26 | 24 | 19 | 6 | 75.0 | +| builtin-vars | 41 | 39 | 33 | 6 | 84.6 | +| glob | 39 | 37 | 31 | 6 | 83.8 | +| here-doc | 36 | 36 | 30 | 6 | 83.3 | +| sh-options-bash | 9 | 9 | 3 | 6 | 33.3 | +| toysh-posix | 23 | 23 | 17 | 6 | 73.9 | +| xtrace | 19 | 19 | 13 | 6 | 68.4 | +| arith-context | 16 | 16 | 11 | 5 | 68.8 | +| assign | 48 | 45 | 40 | 5 | 88.9 | +| blog1 | 9 | 9 | 4 | 5 | 44.4 | +| builtin-meta | 18 | 18 | 13 | 5 | 72.2 | +| builtin-meta-assign | 11 | 11 | 6 | 5 | 54.5 | +| loop | 29 | 29 | 24 | 5 | 82.8 | +| nameref | 32 | 32 | 27 | 5 | 84.4 | +| paren-ambiguity | 9 | 9 | 4 | 5 | 44.4 | +| parse-errors | 27 | 25 | 20 | 5 | 80.0 | +| ble-features | 9 | 9 | 5 | 4 | 55.6 | +| ble-unset | 5 | 5 | 1 | 4 | 20.0 | +| bool-parse | 8 | 8 | 4 | 4 | 50.0 | +| builtin-dirs | 18 | 18 | 14 | 4 | 77.8 | +| builtin-kill | 20 | 20 | 16 | 4 | 80.0 | +| builtin-set | 24 | 24 | 20 | 4 | 83.3 | +| builtin-special | 12 | 12 | 8 | 4 | 66.7 | +| assign-dialects | 4 | 4 | 1 | 3 | 25.0 | +| blog2 | 8 | 8 | 5 | 3 | 62.5 | +| builtin-bash | 13 | 13 | 10 | 3 | 76.9 | +| builtin-misc | 7 | 7 | 4 | 3 | 57.1 | +| builtin-trap-err | 23 | 22 | 19 | 3 | 86.4 | +| builtin-umask | 24 | 24 | 21 | 3 | 87.5 | +| extglob-files | 23 | 23 | 20 | 3 | 87.0 | +| fatal-errors | 5 | 5 | 2 | 3 | 40.0 | +| func-parsing | 15 | 15 | 12 | 3 | 80.0 | +| nix-idioms | 6 | 6 | 3 | 3 | 50.0 | +| quote | 35 | 35 | 32 | 3 | 91.4 | +| redirect-command | 23 | 23 | 20 | 3 | 87.0 | +| serialize | 10 | 10 | 7 | 3 | 70.0 | +| toysh | 8 | 8 | 5 | 3 | 62.5 | +| var-op-len | 9 | 8 | 6 | 3 | 62.5 | +| arg-parse | 3 | 3 | 1 | 2 | 33.3 | +| array-basic | 5 | 5 | 3 | 2 | 60.0 | +| assign-deferred | 9 | 9 | 7 | 2 | 77.8 | +| builtin-getopts | 31 | 31 | 29 | 2 | 93.5 | +| builtin-type | 6 | 6 | 4 | 2 | 66.7 | +| command-sub | 30 | 28 | 26 | 2 | 92.9 | +| divergence | 4 | 4 | 2 | 2 | 50.0 | +| errexit | 35 | 35 | 33 | 2 | 94.3 | +| exit-status | 11 | 11 | 9 | 2 | 81.8 | +| glob-bash | 8 | 8 | 6 | 2 | 75.0 | +| var-op-slice | 22 | 22 | 20 | 2 | 90.9 | +| append | 20 | 20 | 19 | 1 | 95.0 | +| dparen | 15 | 14 | 13 | 1 | 92.9 | +| errexit-osh | 35 | 35 | 34 | 1 | 97.1 | +| extglob-match | 29 | 29 | 28 | 1 | 96.6 | +| for-expr | 9 | 8 | 7 | 1 | 87.5 | +| history-expand | 1 | 1 | 0 | 1 | 0.0 | +| pipeline | 26 | 25 | 24 | 1 | 96.0 | +| process-sub | 9 | 8 | 8 | 1 | 87.5 | +| smoke | 18 | 18 | 17 | 1 | 94.4 | +| unicode | 7 | 2 | 1 | 1 | 50.0 | +| var-op-strip | 29 | 29 | 28 | 1 | 96.6 | +| var-sub | 6 | 6 | 5 | 1 | 83.3 | +| vars-bash | 1 | 1 | 0 | 1 | 0.0 | +| word-eval | 8 | 8 | 7 | 1 | 87.5 | +| zsh-idioms | 3 | 3 | 2 | 1 | 66.7 | +| arith-dynamic | 4 | 4 | 4 | 0 | 100.0 | +| array-compat | 12 | 12 | 12 | 0 | 100.0 | +| builtin-echo | 27 | 27 | 27 | 0 | 100.0 | +| builtin-times | 1 | 1 | 1 | 0 | 100.0 | +| case_ | 13 | 13 | 13 | 0 | 100.0 | +| command-parsing | 5 | 5 | 5 | 0 | 100.0 | +| comments | 2 | 2 | 2 | 0 | 100.0 | +| empty-bodies | 3 | 3 | 3 | 0 | 100.0 | +| explore-parsing | 5 | 5 | 5 | 0 | 100.0 | +| globstar | 5 | 5 | 5 | 0 | 100.0 | +| if_ | 5 | 5 | 5 | 0 | 100.0 | +| known-differences | 2 | 2 | 2 | 0 | 100.0 | +| let | 2 | 2 | 2 | 0 | 100.0 | +| nocasematch-match | 6 | 6 | 6 | 0 | 100.0 | +| posix | 15 | 15 | 15 | 0 | 100.0 | +| print-source-code | 4 | 4 | 4 | 0 | 100.0 | +| sh-func | 12 | 12 | 12 | 0 | 100.0 | +| strict-options | 17 | 15 | 15 | 0 | 100.0 | +| subshell | 2 | 2 | 2 | 0 | 100.0 | +| temp-binding | 4 | 4 | 4 | 0 | 100.0 | +| type-compat | 7 | 5 | 5 | 0 | 100.0 | +| var-num | 7 | 7 | 7 | 0 | 100.0 | +| whitespace | 5 | 0 | 0 | 0 | 0.0 | diff --git a/scripts/oils-spec/run.py b/scripts/oils-spec/run.py new file mode 100755 index 000000000..c7c905f45 --- /dev/null +++ b/scripts/oils-spec/run.py @@ -0,0 +1,646 @@ +#!/usr/bin/env python3 +"""Oils spec pass rate: the upstream Oils spec suite, bash column, through bashkit. + +Oils (https://github.com/oils-for-unix/oils, Apache-2.0) keeps ~2,800 shell +spec cases with expected output per shell. This harness fetches the suite at a +pinned commit into a cache outside the repo (never vendored), runs every case +whose file compares against bash through bashkit and through real bash, and +publishes one headline number: + + pass rate = cases bashkit passes / cases real bash passes + +A case passes for a shell when its status, stdout and any asserted stderr match +the bash column of the case: `## OK bash`, `## N-I bash` and `## BUG bash` +overrides win over the generic `## stdout:` / `## status:` values, exactly as +Oils' own runner (`test/sh_spec.py`) builds assertions for a `bash` label. + +Important decisions: +- Real bash runs the Oils way: code on stdin, env `SH`, `TMP`, `REPO_ROOT`, + `LC_ALL=C.UTF-8`, cwd a fresh temp dir. Its pass count is a sanity check of + this parser against Oils' published bash numbers, and the denominator. +- bashkit runs `bashkit -c CODE bash` (`$0` is `bash`, as with stdin) with + stdin closed. The CLI does not import host env, so the same variables are + exported by a prefix glued onto the first code line, keeping `$LINENO` + identical. The Oils checkout and the helper shims are mounted read-only + (`--mount-ro`, needs the CLI `realfs` feature). +- Oils' helpers in `spec/bin` are Python 2; `bin/` here holds bash stand-ins + with the same output, used by both shells, because bashkit runs no host + Python and the runner may have no Python 2. +- A failing case is classified (signals, processes, interactive, host) by a + keyword heuristic so gaps bashkit keeps by design are visible, but every + case still counts in the rate. +""" + +import argparse +import concurrent.futures +import datetime +import json +import os +import pathlib +import platform +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import time + +HERE = pathlib.Path(__file__).resolve().parent +ROOT = HERE.parent.parent +HELPERS = HERE / "bin" +RESULTS = HERE / "results" + +OILS_REPO = "https://github.com/oils-for-unix/oils" +# Oils pushes no release tags to GitHub; pin a master commit (2026-10-06). +OILS_REV = "57d3f0d088c36340bc4b3038208e58ea02acf15c" + +# VFS paths inside bashkit for the read-only mounts. +VFS_REPO = "/oils" +VFS_HELPERS = "/oils-spec-bin" + + +# --- Oils spec file format (port of test/sh_spec.py, bash column only) ------ + +KEY_VALUE_RE = re.compile( + rb""" + [#][#] \s+ + (?: (OK(?:-\d)? | BUG(?:-\d)? | N-I) \s+ ([\w+/]+) \s+ )? + ([\w\-]+) + : + \s* (.*) +""", + re.VERBOSE, +) +END_MULTILINE_RE = re.compile(rb"[#][#] \s+ END", re.VERBOSE) + +CASE_BEGIN, KEY_VALUE, KEY_VALUE_MULTILINE, END_MULTILINE, PLAIN, EOF = range(6) + + +class Tokenizer: + """Modal line lexer, same rules as sh_spec.py. + + Comment lines are dropped everywhere, including inside code and STDOUT + blocks; Oils runs the code that way, so line numbers match it. + """ + + def __init__(self, lines): + self.lines = lines + self.pos = 0 + self.cursor = None + self.next() + + def _classify(self, line, raw): + if line is None: + return EOF, b"" + if not raw and not line.strip(): + return None + if line.startswith(b"####"): + return CASE_BEGIN, line[4:].strip().decode("utf-8", "replace") + m = KEY_VALUE_RE.match(line) + if m: + qualifier, shells, name, value = m.groups() + name = name.decode() + if name in ("stdout", "stderr"): + value += b"\n" + kind = KEY_VALUE_MULTILINE if name in ("STDOUT", "STDERR") else KEY_VALUE + q = qualifier.decode() if qualifier else None + s = shells.decode() if shells else None + return kind, (q, s, name, value) + if END_MULTILINE_RE.match(line): + return END_MULTILINE, None + if line.startswith(b"##"): + raise ValueError(f"invalid ## line {line!r}") + if line.lstrip().startswith(b"#"): + return None + return PLAIN, line + + def next(self, raw=False): + while True: + line = self.lines[self.pos] if self.pos < len(self.lines) else None + self.pos += 1 + tok = self._classify(line, raw) + if tok is not None: + self.cursor = tok + return tok + + def peek(self): + return self.cursor + + +def _add_meta(case, qualifier, shells, name, value): + for shell in shells.split("/"): + d = case.setdefault("by_shell", {}).setdefault(shell, {}) + d[name] = value + d["qualifier"] = qualifier + + +def _parse_key_values(tokens, case): + while True: + kind, item = tokens.peek() + if kind == KEY_VALUE_MULTILINE: + qualifier, shells, name, _ = item + lines = [] + while True: + kind2, item2 = tokens.next(raw=True) + if kind2 != PLAIN: + break + lines.append(item2) + value = b"".join(lines) + name = name.lower() + if qualifier: + _add_meta(case, qualifier, shells, name, value) + else: + case[name] = value + if kind2 == END_MULTILINE: + tokens.next() + elif kind == KEY_VALUE: + qualifier, shells, name, value = item + if qualifier: + _add_meta(case, qualifier, shells, name, value) + else: + case[name] = value + tokens.next() + else: + return + + +def parse_spec(path): + """Return (file metadata, cases) for one `*.test.sh` file.""" + lines = path.read_bytes().splitlines(keepends=True) + tokens = Tokenizer(lines) + meta = {} + while True: + kind, item = tokens.peek() + if kind != KEY_VALUE: + break + meta[item[2]] = item[3].decode().strip() + tokens.next() + cases = [] + while True: + kind, item = tokens.peek() + if kind == EOF: + break + if kind != CASE_BEGIN: + raise ValueError(f"{path.name}: expected ####, got {item!r}") + tokens.next() + case = {"desc": item} + _parse_key_values(tokens, case) + if "code" not in case: + code = [] + if tokens.peek()[0] != PLAIN: + raise ValueError(f"{path.name}: case {item!r} has no code") + while tokens.peek()[0] == PLAIN: + code.append(tokens.peek()[1]) + tokens.next(raw=True) + case["code"] = b"".join(code) + _parse_key_values(tokens, case) + cases.append(case) + return meta, cases + + +def _string_assertion(d, key, out, qualifier): + found = False + if key in d: + out.append((key, d[key], qualifier)) + found = True + if key + "-json" in d: + value = json.loads(d[key + "-json"].decode("utf-8")).encode("utf-8") + out.append((key, value, qualifier)) + found = True + return found + + +def bash_assertions(case): + """(key, expected, qualifier) triples for the `bash` column of a case.""" + out = [] + got = {"stdout": False, "stderr": False, "status": False} + over = case.get("by_shell", {}).get("bash") + if over: + q = over["qualifier"] + got["stdout"] = _string_assertion(over, "stdout", out, q) + got["stderr"] = _string_assertion(over, "stderr", out, q) + if "status" in over: + out.append(("status", int(over["status"]), q)) + got["status"] = True + if not got["stdout"]: + _string_assertion(case, "stdout", out, None) + if not got["stderr"]: + _string_assertion(case, "stderr", out, None) + if not got["status"]: + out.append(("status", int(case.get("status", b"0")), None)) + return out + + +def check(assertions, actual): + """None when every assertion holds, else the first failing key.""" + if actual.get("timeout"): + return "timeout" + for key, expected, _ in assertions: + if actual[key] != expected: + return key + if b"Traceback (most recent" in actual["stderr"]: + return "stderr" + return None + + +def bash_qualifier(case): + over = case.get("by_shell", {}).get("bash") + return over["qualifier"] if over else "PASS" + + +def in_bash_column(meta): + shells = meta.get("compare_shells", "").split() + return any(s == "bash" or s.startswith("bash-") for s in shells) + + +# --- classification of bashkit failures ------------------------------------- + +# Heuristic, ordered: first match wins. These are areas bashkit keeps out of +# the sandbox by design (knowledge/operations/limitations.md); a failing case +# that touches one is labeled so the gap list separates them from plain +# behavior differences. They still count in the pass rate. +CLASSES = [ + ( + "signals", + re.compile(rb"\bkill\b|\bSIG[A-Z]+\b|\btrap\b[^\n]*\b(INT|TERM|USR1|USR2|HUP|QUIT|CHLD|ALRM|PIPE|WINCH)\b"), + ), + ( + "processes", + re.compile( + rb"\bwait\b|\bjobs\b|\bfg\b|\bbg\b|\bdisown\b|\$!|\$\$|\$PPID|\$BASHPID|/proc/|\bulimit\b|\btimes\b|\bps\b|\bcoproc\b|\bsleep\b[^\n]*&" + ), + ), + ( + "interactive", + re.compile(rb"\$SH -i\b| -i -c\b|\bPS[0-4]\b|\bbind\b|\bhistory\b|\bfc\b|\bcomplete\b|\bcompgen\b|\bcompopt\b"), + ), + ( + "host", + re.compile( + rb"/dev/tty|/etc/passwd|~root\b|\bumask\b|\bhostname\b|\$UID\b|\$EUID\b|\bid -|\bchmod\b|\bmkfifo\b" + ), + ), +] + + +def classify(case): + code = case["code"] + for name, rx in CLASSES: + if rx.search(code): + return name + return "behavior" + + +# --- running ----------------------------------------------------------------- + + +def run_proc(argv, stdin_bytes, cwd, env, timeout, executable=None): + proc = subprocess.Popen( + argv, + executable=executable, + stdin=subprocess.PIPE if stdin_bytes is not None else subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=cwd, + env=env, + start_new_session=True, + ) + try: + out, err = proc.communicate(stdin_bytes, timeout=timeout) + return {"stdout": out, "stderr": err, "status": proc.returncode} + except subprocess.TimeoutExpired: + try: + os.killpg(proc.pid, signal.SIGKILL) + except ProcessLookupError: + pass + try: + proc.communicate(timeout=5) + except subprocess.TimeoutExpired: + pass + return {"stdout": b"", "stderr": b"", "status": -9, "timeout": True} + except BrokenPipeError: + out, err = proc.communicate(timeout=timeout) + return {"stdout": out, "stderr": err, "status": proc.returncode} + + +def bashkit_prefix(legacy_tmp): + prefix = f"export SH=bash TMP=/tmp REPO_ROOT={VFS_REPO} LC_ALL=C.UTF-8 PATH={VFS_HELPERS}:$PATH; " + if legacy_tmp: + prefix += "mkdir -p _tmp; " + return prefix.encode() + + +def run_bashkit(binary, repo, code, timeout, legacy_tmp): + with tempfile.TemporaryDirectory(prefix="oils-bk-") as work: + argv = [ + binary, + "--mount-ro", + f"{repo}:{VFS_REPO}", + "--mount-ro", + f"{HELPERS}:{VFS_HELPERS}", + "--timeout", + str(timeout), + "-c", + (bashkit_prefix(legacy_tmp) + code).decode("utf-8", "surrogateescape"), + "bash", + ] + env = {"PATH": os.environ.get("PATH", "/usr/bin:/bin"), "LC_ALL": "C.UTF-8"} + return run_proc(argv, None, work, env, timeout + 5) + + +def run_bash(bash, repo, code, timeout, legacy_tmp): + with tempfile.TemporaryDirectory(prefix="oils-sh-") as work: + if legacy_tmp: + os.mkdir(os.path.join(work, "_tmp")) + env = { + "PATH": f"{HELPERS}:{os.path.dirname(bash)}:{os.environ.get('PATH', '/usr/bin:/bin')}", + "LC_ALL": "C.UTF-8", + # A bare name, as Oils passes it: bash then reports errors as + # "bash: line N:", which is what stderr assertions expect. + "SH": "bash", + "TMP": work, + "REPO_ROOT": str(repo), + } + return run_proc(["bash"], code, work, env, timeout, executable=bash) + + +def fetch_oils(cache): + src = cache / "oils" + head = subprocess.run(["git", "-C", str(src), "rev-parse", "HEAD"], capture_output=True, text=True, check=False) + if head.returncode == 0 and head.stdout.strip() == OILS_REV: + return src + if src.exists(): + shutil.rmtree(src) + src.mkdir(parents=True) + for argv in ( + ["git", "init", "-q", str(src)], + ["git", "-C", str(src), "fetch", "-q", "--depth", "1", OILS_REPO, OILS_REV], + ["git", "-C", str(src), "checkout", "-q", "FETCH_HEAD"], + ): + subprocess.run(argv, check=True) + return src + + +def supports_mounts(binary): + probe = subprocess.run( + [binary, "--mount-ro", f"{HELPERS}:/probe", "-c", "test -e /probe/argv.py"], + stdin=subprocess.DEVNULL, + capture_output=True, + timeout=30, + check=False, + ) + return probe.returncode == 0 + + +# --- reporting --------------------------------------------------------------- + + +def pct(num, den): + return round(100.0 * num / den, 1) if den else 0.0 + + +def summarize(rows, with_bash): + total = len(rows) + bk = sum(1 for r in rows if r["bashkit"] is None) + out = {"cases": total, "bashkit_pass": bk, "bashkit_pass_pct": pct(bk, total)} + if with_bash: + sh = [r for r in rows if r["bash"] is None] + both = sum(1 for r in sh if r["bashkit"] is None) + out.update( + { + "bash_pass": len(sh), + "bash_pass_pct": pct(len(sh), total), + "bashkit_pass_of_bash": both, + "pass_rate": pct(both, len(sh)), + } + ) + else: + out["pass_rate"] = out["bashkit_pass_pct"] + return out + + +def write_markdown(path, report): + t = report["total"] + lines = [ + "# Oils spec pass rate", + "", + f"- Oils revision: [`{report['oils_rev'][:12]}`]({OILS_REPO}/tree/{report['oils_rev']})", + f"- bashkit commit: `{report['bashkit_commit']}`", + f"- bash: {report['bash_version'] or 'not run'}", + f"- Spec files: {report['files_run']} (bash column; {report['files_skipped']} without one skipped)", + f"- Runtime: {report['elapsed_secs']} s, {report['jobs']} jobs, {report['timeout_secs']} s per-case timeout", + "", + "## Headline", + "", + ] + if "bash_pass" in t: + lines += [ + ( + f"**{t['pass_rate']}%** — bashkit passes {t['bashkit_pass_of_bash']} of the " + f"{t['bash_pass']} cases real bash passes." + ), + "", + "| Shell | Pass | Cases | % |", + "|-------|-----:|------:|--:|", + f"| bash | {t['bash_pass']} | {t['cases']} | {t['bash_pass_pct']} |", + f"| bashkit | {t['bashkit_pass']} | {t['cases']} | {t['bashkit_pass_pct']} |", + "", + ] + else: + lines += [ + f"**{t['pass_rate']}%** — bashkit passes {t['bashkit_pass']} of {t['cases']} cases (bash not run).", + "", + ] + lines += [ + "## bashkit failures by class", + "", + "Heuristic keyword classes; every case still counts in the rate.", + "", + "| Class | Failing cases |", + "|-------|--------------:|", + ] + for name, n in sorted(report["failure_classes"].items(), key=lambda kv: -kv[1]): + lines.append(f"| {name} | {n} |") + lines += [ + "", + "## Per spec file", + "", + "Sorted by cases bashkit misses among those bash passes.", + "", + "| Spec file | Cases | bash | bashkit | Missed | Rate % |", + "|-----------|------:|-----:|--------:|-------:|-------:|", + ] + for name, f in report["files"].items(): + s = f["summary"] + missed = s.get("bash_pass", s["cases"]) - s.get("bashkit_pass_of_bash", s["bashkit_pass"]) + lines.append( + f"| {name} | {s['cases']} | {s.get('bash_pass', '-')} | {s['bashkit_pass']} | {missed} | {s['pass_rate']} |" + ) + lines.append("") + path.write_text("\n".join(lines)) + + +def git_commit(): + r = subprocess.run( + ["git", "-C", str(ROOT), "rev-parse", "--short=12", "HEAD"], capture_output=True, text=True, check=False + ) + return r.stdout.strip() or "unknown" + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--bin", default=os.environ.get("BASHKIT_BIN"), help="bashkit CLI built with realfs") + ap.add_argument("--cache", default=os.environ.get("OILS_SPEC_CACHE", str(ROOT / "target" / "oils-spec"))) + ap.add_argument("--no-bash", action="store_true", help="skip real bash; rate becomes bashkit/all cases") + ap.add_argument("--jobs", type=int, default=os.cpu_count() or 4) + ap.add_argument("--timeout", type=int, default=10, help="per-case seconds") + ap.add_argument("--save", action="store_true", help="write results/oils-spec-*.{json,md}") + ap.add_argument("--fails", help="write failing cases (code, expected, actual) to this file") + ap.add_argument("specs", nargs="*", help="spec names to run (e.g. arith var-sub); default all") + args = ap.parse_args() + + if args.save and args.specs: + sys.exit("oils-spec: --save records the whole suite; drop the spec names") + binary = args.bin or str(ROOT / "target" / "debug" / "bashkit") + if not os.access(binary, os.X_OK): + sys.exit(f"oils-spec: no bashkit binary at {binary} (cargo build -p bashkit-cli --features realfs)") + if not supports_mounts(binary): + sys.exit(f"oils-spec: {binary} lacks --mount-ro; build with: cargo build -p bashkit-cli --features realfs") + bash = None + bash_version = None + if not args.no_bash: + bash = shutil.which("bash") + if not bash: + sys.exit("oils-spec: no bash on PATH (or pass --no-bash)") + bash_version = subprocess.run( + [bash, "-c", "echo $BASH_VERSION"], capture_output=True, text=True, check=False + ).stdout.strip() + + repo = fetch_oils(pathlib.Path(args.cache)) + specs = sorted((repo / "spec").glob("*.test.sh")) + if args.specs: + want = {s.removesuffix(".test.sh") for s in args.specs} + specs = [p for p in specs if p.name.removesuffix(".test.sh") in want] + + work = [] + skipped = 0 + for path in specs: + meta, cases = parse_spec(path) + if not in_bash_column(meta) or meta.get("suite") == "disabled": + skipped += 1 + continue + name = path.name.removesuffix(".test.sh") + for i, case in enumerate(cases): + work.append((name, i, case, bool(meta.get("legacy_tmp_dir")))) + + def run_one(item): + name, i, case, legacy_tmp = item + asserts = bash_assertions(case) + bk = run_bashkit(binary, repo, case["code"], args.timeout, legacy_tmp) + sh = run_bash(bash, repo, case["code"], args.timeout, legacy_tmp) if bash else None + return name, i, case, asserts, bk, sh + + started = time.time() + rows_by_file = {} + fails = [] + done = 0 + with concurrent.futures.ThreadPoolExecutor(max_workers=args.jobs) as pool: + for name, i, case, asserts, bk, sh in pool.map(run_one, work): + row = { + "case": i, + "name": case["desc"], + "qualifier": bash_qualifier(case), + "bashkit": check(asserts, bk), + "bash": check(asserts, sh) if sh else None, + } + if row["bashkit"] is not None: + row["class"] = classify(case) + if args.fails and row["bash"] is None: + fails.append((name, case, asserts, bk)) + rows_by_file.setdefault(name, []).append(row) + done += 1 + if done % 200 == 0: + print(f"oils-spec: {done}/{len(work)}", file=sys.stderr) + elapsed = round(time.time() - started, 1) + + with_bash = bash is not None + all_rows = [r for rows in rows_by_file.values() for r in rows] + files = {} + for name, rows in rows_by_file.items(): + s = summarize(rows, with_bash) + files[name] = { + "summary": s, + # bashkit misses among cases bash passes: the list a PR shrinks. + "missed": [ + {"case": r["case"], "name": r["name"], "class": r["class"], "on": r["bashkit"]} + for r in rows + if r["bashkit"] is not None and (r["bash"] is None or not with_bash) + ], + "bash_fails": [r["case"] for r in rows if with_bash and r["bash"] is not None], + } + key = ( + (lambda kv: (-(kv[1]["summary"]["bash_pass"] - kv[1]["summary"]["bashkit_pass_of_bash"]), kv[0])) + if with_bash + else (lambda kv: (-(kv[1]["summary"]["cases"] - kv[1]["summary"]["bashkit_pass"]), kv[0])) + ) + files = dict(sorted(files.items(), key=key)) + classes = {} + for r in all_rows: + if r["bashkit"] is not None and (not with_bash or r["bash"] is None): + classes[r["class"]] = classes.get(r["class"], 0) + 1 + + report = { + "schema": 1, + "oils_repo": OILS_REPO, + "oils_rev": OILS_REV, + "bashkit_commit": git_commit(), + "bash_version": bash_version, + "timestamp": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "platform": f"{platform.system().lower()}-{platform.machine()}", + "jobs": args.jobs, + "timeout_secs": args.timeout, + "elapsed_secs": elapsed, + "files_run": len(files), + "files_skipped": skipped, + "total": summarize(all_rows, with_bash), + "failure_classes": classes, + "files": files, + } + + t = report["total"] + if with_bash: + print( + f"oils-spec: {t['pass_rate']}% ({t['bashkit_pass_of_bash']}/{t['bash_pass']} of cases real bash passes); " + f"bash {t['bash_pass']}/{t['cases']}, bashkit {t['bashkit_pass']}/{t['cases']}; {elapsed}s" + ) + else: + print(f"oils-spec: bashkit {t['bashkit_pass']}/{t['cases']} ({t['pass_rate']}%); {elapsed}s") + + if args.fails: + with open(args.fails, "w", encoding="utf-8", errors="replace") as fh: + for name, case, asserts, bk in fails: + fh.write(f"==== {name} #{case['desc']}\n") + fh.write(case["code"].decode("utf-8", "replace")) + for key, expected, _ in asserts: + if bk.get("timeout"): + fh.write("-- TIMEOUT\n") + break + if bk[key] != expected: + fh.write(f"-- expected {key}: {expected!r}\n-- got {key}: {bk[key]!r}\n") + if not bk.get("timeout"): + fh.write(f"-- stderr: {bk['stderr'][:400]!r}\n") + fh.write("\n") + + if args.save: + RESULTS.mkdir(exist_ok=True) + stamp = datetime.datetime.now(datetime.timezone.utc).strftime("%Y%m%dT%H%M%SZ") + base = RESULTS / f"oils-spec-{report['platform']}-{stamp}" + base.with_suffix(".json").write_text(json.dumps(report, indent=1) + "\n") + write_markdown(base.with_suffix(".md"), report) + print(f"oils-spec: wrote {base}.json and .md") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tests/test_oils_spec.py b/scripts/tests/test_oils_spec.py new file mode 100644 index 000000000..d71b2c03a --- /dev/null +++ b/scripts/tests/test_oils_spec.py @@ -0,0 +1,122 @@ +"""Tests for scripts/oils-spec: the Oils spec-file parser, bash-column +assertions, scoring, and the argv.py shim. + +The parser decides the denominator of the headline number, so the Oils rules +it ports (bash overrides win, comment lines dropped, default status 0) each +get a case. No network and no bashkit binary needed. +""" + +import importlib.util +import pathlib +import shutil +import subprocess +import tempfile +import unittest + +REPO = pathlib.Path(__file__).resolve().parents[2] +SPEC = importlib.util.spec_from_file_location("oils_spec_run", REPO / "scripts" / "oils-spec" / "run.py") +run = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(run) + +SAMPLE = b"""\ +## compare_shells: dash bash mksh + +#### plain +echo hi +# dropped comment +echo there +## STDOUT: +hi +there +## END + +#### bash override wins +echo x; exit 1 +## stdout: y +## status: 0 +## OK bash stdout: x +## OK bash status: 1 + +#### N-I for bash, json stdout +printf '%s' '' +## stdout-json: "a\\u00e9" +## N-I bash/mksh stdout-json: "" + +#### stderr only when asserted +echo e >&2 +## status: 0 +""" + + +def parse(text): + with tempfile.TemporaryDirectory() as d: + path = pathlib.Path(d) / "sample.test.sh" + path.write_bytes(text) + return run.parse_spec(path) + + +def ok(stdout=b"", status=0, stderr=b""): + return {"stdout": stdout, "stderr": stderr, "status": status} + + +class ParseTests(unittest.TestCase): + def setUp(self): + self.meta, self.cases = parse(SAMPLE) + + def test_metadata_and_bash_column(self): + self.assertEqual(self.meta["compare_shells"], "dash bash mksh") + self.assertTrue(run.in_bash_column(self.meta)) + self.assertFalse(run.in_bash_column({"compare_shells": "dash mksh"})) + self.assertTrue(run.in_bash_column({"compare_shells": "bash-4.4 zsh"})) + self.assertEqual(len(self.cases), 4) + + def test_comment_lines_dropped_from_code(self): + self.assertEqual(self.cases[0]["code"], b"echo hi\necho there\n") + a = run.bash_assertions(self.cases[0]) + self.assertIsNone(run.check(a, ok(b"hi\nthere\n"))) + self.assertEqual(run.check(a, ok(b"hi\n")), "stdout") + + def test_bash_override_wins(self): + a = run.bash_assertions(self.cases[1]) + self.assertIsNone(run.check(a, ok(b"x\n", 1))) + self.assertEqual(run.check(a, ok(b"y\n", 0)), "stdout") + self.assertEqual(run.bash_qualifier(self.cases[1]), "OK") + + def test_shared_override_and_json(self): + a = run.bash_assertions(self.cases[2]) + self.assertIsNone(run.check(a, ok(b""))) + self.assertEqual(run.check(a, ok("aé".encode())), "stdout") + self.assertEqual(run.bash_qualifier(self.cases[2]), "N-I") + + def test_unasserted_stderr_and_default_status(self): + a = run.bash_assertions(self.cases[3]) + self.assertIsNone(run.check(a, ok(stderr=b"e\n"))) + self.assertEqual(run.check(a, ok(status=1)), "status") + self.assertEqual(run.check(a, {**ok(), "timeout": True}), "timeout") + + def test_bad_metadata_line_rejected(self): + with self.assertRaises(ValueError): + parse(b"#### x\necho\n##stdout: x\n") + + +class ClassifyTests(unittest.TestCase): + def test_classes(self): + self.assertEqual(run.classify({"code": b"kill -TERM $pid\n"}), "signals") + self.assertEqual(run.classify({"code": b"sleep 1 &\nwait\n"}), "processes") + self.assertEqual(run.classify({"code": b"complete -F f cmd\n"}), "interactive") + self.assertEqual(run.classify({"code": b"echo hi\n"}), "behavior") + + +@unittest.skipUnless(shutil.which("bash") and shutil.which("od"), "needs bash and od") +class ArgvShimTests(unittest.TestCase): + def test_python2_repr(self): + args = ["a b", "it's", 'q"x', "μ", "t\tn\n", "back\\slash", "both'\"", ""] + out = subprocess.run(["bash", str(run.HELPERS / "argv.py"), *args], capture_output=True, check=True).stdout + self.assertEqual( + out, + b"['a b', \"it's\", 'q\"x', '\\xce\\xbc', 't\\tn\\n', 'back\\\\slash', 'both\\'\"', '']\n", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/site/scripts/build-performance-data.mjs b/site/scripts/build-performance-data.mjs index 185b61668..6383cae18 100644 --- a/site/scripts/build-performance-data.mjs +++ b/site/scripts/build-performance-data.mjs @@ -13,6 +13,7 @@ const benchDir = path.join(repoRoot, "crates/bashkit-bench/results"); const criterionDir = path.join(repoRoot, "crates/bashkit/benches/results"); const evalDir = path.join(repoRoot, "crates/bashkit-eval/results"); const gapDir = path.join(evalDir, "gaps"); +const oilsSpecDir = path.join(repoRoot, "scripts/oils-spec/results"); const benchmarkCategoryDescriptions = { arithmetic: "Integer math, substitutions, and expression-heavy shell snippets.", @@ -378,6 +379,53 @@ async function buildGapTelemetry() { }; } +// Oils spec pass rate: one point per `scripts/oils-spec/run.py --save` report +// (upstream Oils spec suite, bash column). Headline = bashkit passes among the +// cases real bash passes; the latest report's worst spec files are listed. +async function buildOilsSpec() { + let files; + try { + files = (await listFiles(oilsSpecDir, ".json")).filter((file) => file.startsWith("oils-spec-")); + } catch { + return { runs: [], latest: null }; + } + const runs = []; + for (const file of files) { + const report = await readJson(path.join(oilsSpecDir, file)); + const source = `scripts/oils-spec/results/${file}`; + const total = report.total; + runs.push({ + id: file.replace(/\.json$/, ""), + date: dateLabel(report.timestamp), + timestamp: report.timestamp, + source, + reportSource: await existingMarkdownReport(source), + oilsRev: report.oils_rev.slice(0, 12), + bashkitCommit: report.bashkit_commit, + bashVersion: report.bash_version, + cases: total.cases, + bashPass: total.bash_pass ?? null, + bashkitPass: total.bashkit_pass, + bashkitPassOfBash: total.bashkit_pass_of_bash ?? null, + passRate: total.pass_rate, + topFiles: Object.entries(report.files) + .slice(0, 12) + .map(([name, entry]) => ({ + name, + cases: entry.summary.cases, + bashPass: entry.summary.bash_pass ?? entry.summary.cases, + bashkitPass: entry.summary.bashkit_pass_of_bash ?? entry.summary.bashkit_pass, + passRate: entry.summary.pass_rate, + })), + }); + } + const sorted = runs.toSorted((a, b) => new Date(a.timestamp) - new Date(b.timestamp)); + return { + runs: sorted.map(({ topFiles, ...point }) => point), + latest: sorted.at(-1) ?? null, + }; +} + async function buildEvalRuns() { const files = await listFiles(evalDir, ".json"); const runs = []; @@ -508,6 +556,7 @@ const criterionRuns = await buildCriterionRuns(); const evalRuns = await buildEvalRuns(); const pythonStartup = await buildPythonStartup(); const gapTelemetry = await buildGapTelemetry(); +const oilsSpec = await buildOilsSpec(); const newestSourceTimestamp = latest([...benchRuns, ...criterionRuns, ...evalRuns])?.timestamp ?? null; const payload = { @@ -517,6 +566,7 @@ const payload = { criterion: "crates/bashkit/benches/results/*.md", evals: "crates/bashkit-eval/results/*.json", gaps: "crates/bashkit-eval/results/gaps/gaps-*.json", + oilsSpec: "scripts/oils-spec/results/oils-spec-*.json", }, summary: { benchRuns: benchRuns.length, @@ -536,6 +586,7 @@ const payload = { evalRuns, pythonStartup, gapTelemetry, + oilsSpec, modelTrends: buildModelTrends(evalRuns), milestones: buildMilestones({ benchRuns, criterionRuns, evalRuns }), }; diff --git a/site/src/data/performance-timeline.json b/site/src/data/performance-timeline.json index 86df6e67a..3cdadbf9c 100644 --- a/site/src/data/performance-timeline.json +++ b/site/src/data/performance-timeline.json @@ -1,14 +1,15 @@ { - "generatedAt": "2026-10-07T03:57:14.000Z", + "generatedAt": "2026-10-07T19:23:22.000Z", "sources": { "bench": "crates/bashkit-bench/results/*.json", "criterion": "crates/bashkit/benches/results/*.md", "evals": "crates/bashkit-eval/results/*.json", - "gaps": "crates/bashkit-eval/results/gaps/gaps-*.json" + "gaps": "crates/bashkit-eval/results/gaps/gaps-*.json", + "oilsSpec": "scripts/oils-spec/results/oils-spec-*.json" }, "summary": { "benchRuns": 10, - "criterionRuns": 17, + "criterionRuns": 18, "evalRuns": 38, "latestBench": { "id": "bench-mykhailosmac-macos-aarch64-1788629833", @@ -1874,6 +1875,27 @@ "us": 16.35 }, "bestImprovement": null + }, + { + "id": "criterion-parallel-vm-linux-x86_64-1791401002", + "kind": "criterion", + "family": "parallel", + "label": "Criterion Parallel Execution Benchmark", + "date": "2026-10-07", + "timestamp": "2026-10-07T19:23:22.000Z", + "source": "crates/bashkit/benches/results/criterion-parallel-vm-linux-x86_64-1791401002.md", + "reportSource": "crates/bashkit/benches/results/criterion-parallel-vm-linux-x86_64-1791401002.md", + "cases": 33, + "medianUs": 17490, + "p95Us": 247426, + "medianChangePct": null, + "meanChangePct": null, + "bestChangePct": null, + "fastestCase": { + "name": "single_bash_new", + "us": 68.29 + }, + "bestImprovement": null } ], "evalRuns": [ @@ -5152,6 +5174,126 @@ ] } }, + "oilsSpec": { + "runs": [ + { + "id": "oils-spec-linux-x86_64-20261008T103235Z", + "date": "2026-10-08", + "timestamp": "2026-10-08T10:32:35Z", + "source": "scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.json", + "reportSource": "scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md", + "oilsRev": "57d3f0d088c3", + "bashkitCommit": "f86bd7c81ae3", + "bashVersion": "5.2.21(1)-release", + "cases": 2759, + "bashPass": 2693, + "bashkitPass": 2003, + "bashkitPassOfBash": 1997, + "passRate": 74.2 + } + ], + "latest": { + "id": "oils-spec-linux-x86_64-20261008T103235Z", + "date": "2026-10-08", + "timestamp": "2026-10-08T10:32:35Z", + "source": "scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.json", + "reportSource": "scripts/oils-spec/results/oils-spec-linux-x86_64-20261008T103235Z.md", + "oilsRev": "57d3f0d088c3", + "bashkitCommit": "f86bd7c81ae3", + "bashVersion": "5.2.21(1)-release", + "cases": 2759, + "bashPass": 2693, + "bashkitPass": 2003, + "bashkitPassOfBash": 1997, + "passRate": 74.2, + "topFiles": [ + { + "name": "builtin-completion", + "cases": 51, + "bashPass": 51, + "bashkitPass": 17, + "passRate": 33.3 + }, + { + "name": "array", + "cases": 78, + "bashPass": 78, + "bashkitPass": 53, + "passRate": 67.9 + }, + { + "name": "prompt", + "cases": 33, + "bashPass": 29, + "bashkitPass": 4, + "passRate": 13.8 + }, + { + "name": "builtin-trap-bash", + "cases": 23, + "bashPass": 23, + "bashkitPass": 4, + "passRate": 17.4 + }, + { + "name": "globignore", + "cases": 18, + "bashPass": 18, + "bashkitPass": 2, + "passRate": 11.1 + }, + { + "name": "word-split", + "cases": 55, + "bashPass": 53, + "bashkitPass": 37, + "passRate": 69.8 + }, + { + "name": "var-ref", + "cases": 31, + "bashPass": 30, + "bashkitPass": 15, + "passRate": 50 + }, + { + "name": "builtin-cd", + "cases": 30, + "bashPass": 28, + "bashkitPass": 14, + "passRate": 50 + }, + { + "name": "alias", + "cases": 48, + "bashPass": 48, + "bashkitPass": 35, + "passRate": 72.9 + }, + { + "name": "dbracket", + "cases": 49, + "bashPass": 49, + "bashkitPass": 36, + "passRate": 73.5 + }, + { + "name": "builtin-fc", + "cases": 14, + "bashPass": 14, + "bashkitPass": 2, + "passRate": 14.3 + }, + { + "name": "builtin-history", + "cases": 17, + "bashPass": 17, + "bashkitPass": 5, + "passRate": 29.4 + } + ] + } + }, "modelTrends": [ { "model": "anthropic/claude-haiku-4-5", @@ -5983,6 +6125,15 @@ "detail": "python_call/monty/pass at 16.35 us median", "metric": 5663.25, "source": "crates/bashkit/benches/results/criterion-python-vm-linux-x86_64-1791345434.md" + }, + { + "date": "2026-10-07", + "timestamp": "2026-10-07T19:23:22.000Z", + "kind": "Criterion", + "title": "parallel", + "detail": "single_bash_new at 68.29 us median", + "metric": 17490, + "source": "crates/bashkit/benches/results/criterion-parallel-vm-linux-x86_64-1791401002.md" } ] } diff --git a/site/src/pages/benches.astro b/site/src/pages/benches.astro index 01dc607fb..d2a3ba351 100644 --- a/site/src/pages/benches.astro +++ b/site/src/pages/benches.astro @@ -45,6 +45,10 @@ const gapLatest = data.gapTelemetry?.latest ?? null; const gapRuns = (data.gapTelemetry?.runs ?? []).toSorted( (a, b) => new Date(b.timestamp).getTime() - new Date(a.timestamp).getTime(), ); +const oilsLatest = data.oilsSpec?.latest ?? null; +const oilsRuns = (data.oilsSpec?.runs ?? []).toSorted( + (a, b) => new Date(b.timestamp).getTime() - new Date(a.timestamp).getTime(), +); const gapKindLabel: Record = { missing_command: "missing command", unknown_option: "unknown option", @@ -120,7 +124,7 @@ const evalResultsUrl = "https://github.com/everruns/bashkit/tree/main/crates/bashkit-eval/results"; const pageDescription = - "Explore Bashkit benchmark, Python startup, gap telemetry, Criterion bench, and LLM eval results with latest runtime snapshots, category scores, raw reports, and historical artifact links over time."; + "Explore Bashkit benchmark, Python startup, gap telemetry, Oils spec pass rate, Criterion bench, and LLM eval results with latest runtime snapshots, category scores, raw reports, and historical artifact links over time."; --- @@ -305,6 +309,74 @@ const pageDescription = ) } + { + oilsLatest && ( +
+
+
+ Oils spec suite +

{`${oilsLatest.passRate}% of what real bash passes`}

+ + Open report ({oilsLatest.date}) + +
+ + + + + + { + oilsLatest.topFiles.map((file) => ( + + + + + + )) + } + +
Spec file (most misses)bashkitbash
+ {file.name} + {`${file.passRate}% of ${file.cases} cases`} + {file.bashkitPass}{file.bashPass}
+
+ +
+
+ Oils spec history +

Cases passed vs real bash

+
+ + + + + + { + oilsRuns.map((run) => ( + + + + + + )) + } + +
RunRatebashkit / bash
+ + {run.date} + + {`Oils ${run.oilsRev}, bashkit ${run.bashkitCommit}`} + {`${run.passRate}%`}{`${compactNumber(run.bashkitPassOfBash ?? run.bashkitPass)} / ${compactNumber(run.bashPass ?? run.cases)}`}
+
+
+ ) + } +