Skip to content

README.md

Latest commit

 

History

History
348 lines (271 loc) · 13.3 KB

README.md

File metadata and controls

348 lines (271 loc) · 13.3 KB
Raw

Switchcraft Scripts

This directory holds developer-facing helpers that are not part of the shipping Swift package: model conversion, tokenizer-fixture regeneration, and reference-score generation for the cross-implementation parity tests.

convert-xtr-to-coreml.py

Converts the google/xtr-base-en PyTorch encoder into a .mlpackage suitable for T5CoreMLEmbedder. See README.md (project root) for the end-to-end "Getting Started" instructions and the SWITCHCRAFT_XTR_MLPACKAGE env-var contract.

quantize-mlpackage-int8w.py

Post-processes the FP32 .mlpackage produced by convert-xtr-to-coreml.py into an INT8 weight-only sibling (~430 MB → ~110 MB). Compute precision stays at FP32; only Linear-op weight storage changes. Runs an INT8w-vs-FP32 CoreML parity check (mean cosine similarity ≥ 0.998 by default) before exiting non-zero.

The resulting asset is opt-in via SWITCHCRAFT_XTR_MLPACKAGE_INT8W and MUST be initialised with a distinct modelIdentifier (recommended "google/xtr-base-en@v1-int8w") to avoid silent collisions in ChunkRecord.model mismatch detection. See ADR 010(i) for the full contract and python3 scripts/quantize-mlpackage-int8w.py --help for the workflow.

regenerate-tokenizer-fixtures.py

Regenerates the cached HuggingFace tokenizer fixture used by TokenizerTests. Re-run after upgrading the upstream tokenizer JSON.

fetch-nfcorpus.sh — populating $SWITCHCRAFT_NFCORPUS_DIR

NFCorpusBenchmarkTests (Tests/SwitchcraftTests/) computes NDCG@10 against the NFCorpus test split and asserts the result lands in Witchcraft's published [0.31, 0.33] band. The dataset is not committed to this repository: NFCorpus is academic-use-only and incompatible with Switchcraft's Apache 2.0 release intent. Developers who want to run the benchmark obtain NFCorpus locally under whatever upstream terms they accept.

This script is one developer convenience. It pulls four .zst files from upstream Witchcraft's pinned commit, decompresses them in place, and writes plaintext into the directory referenced by $SWITCHCRAFT_NFCORPUS_DIR.

Prerequisites

  • bash, curl, zstd (on macOS: brew install zstd).

Usage

export SWITCHCRAFT_NFCORPUS_DIR="$HOME/datasets/nfcorpus"
mkdir -p "$SWITCHCRAFT_NFCORPUS_DIR"
./scripts/fetch-nfcorpus.sh

The script writes three required plaintext files plus one optional file into $SWITCHCRAFT_NFCORPUS_DIR:

  • nfcorpus.tsv — corpus rows (docid \t title \t body)
  • questions.test.tsv — dev queries (query-id \t query)
  • qrels.test.json — pytrec_eval-style nested-dict relevance judgments
  • collection_map.json — optional; ignored by the Switchcraft suite (Switchcraft indexes by the original MED-XXXX doc-id directly)

Total size ~5–6 MB plaintext. With both SWITCHCRAFT_XTR_MLPACKAGE and SWITCHCRAFT_NFCORPUS_DIR set, run the benchmark with:

swift test --filter NFCorpusBenchmark

Expect ~6 minutes on Apple Silicon for the one-time index build.

License caveat

NFCorpus is third-party data; Switchcraft does not redistribute it and takes no position on its licensing terms. By running this script you are downloading data hosted by upstream Witchcraft — confirm that your intended use is permitted before proceeding.

witchcraft-fact-dump.patch — regenerating Tests/Fixtures/facts_corpus.json

FactsCorpusParityTests (Tests/SwitchcraftTests/) compares Switchcraft's SearchEngine.search output against reference scores produced by upstream Witchcraft for the same 33-fact corpus and same three queries. The reference scores live in Tests/Fixtures/facts_corpus.json. This patch regenerates that file.

One-time setup

You need a working Rust toolchain and the Witchcraft assets (~62 MB quantized GGUF + ~2 MB tokenizer + config). The Switchcraft Apple-side toolchain is not required.

# Pinned Witchcraft commit. The fixture file's
# `provenance.witchcraftCommit` field must match this exactly.
WITCHCRAFT_COMMIT=6ad59e51cfc89bcfb20756e3f05cf9429b7cb55f

git clone https://github.com/dropbox/witchcraft /tmp/witchcraft
cd /tmp/witchcraft
git checkout "$WITCHCRAFT_COMMIT"

# Build assets/xtr.gguf, assets/config.json, assets/tokenizer.json.
# Drives uv + transformers + safetensors + the in-tree quantize-tool.
make download

Built and tested against Cargo 1.92.0 (Homebrew) and Python 3.13+. Apple Silicon picks up the metal feature automatically; Intel Macs and Linux pick up fbgemm. Either is fine for fixture generation.

Apply the patch and run

# From inside the Witchcraft checkout, with the same commit checked out:
cd /tmp/witchcraft
git apply /path/to/switchcraft/scripts/witchcraft-fact-dump.patch

# The new test is `#[ignore]` so a normal `cargo test` does not run it.
WITCHCRAFT_COMMIT=$(git rev-parse HEAD) cargo test \
    --features t5-quantized \
    --lib \
    tests::tests::dump_facts_corpus_fixture \
    -- --nocapture --include-ignored

The test prints a JSON document on stdout, framed by the literal lines ---SWITCHCRAFT-FIXTURE-BEGIN--- / ---SWITCHCRAFT-FIXTURE-END---. Extract that block and write it to the Switchcraft repo at Tests/Fixtures/facts_corpus.json:

WITCHCRAFT_COMMIT=$(git rev-parse HEAD) cargo test \
    --features t5-quantized --lib \
    tests::tests::dump_facts_corpus_fixture \
    -- --nocapture --include-ignored 2>&1 \
| awk '/^---SWITCHCRAFT-FIXTURE-BEGIN---$/{flag=1; next} \
       /^---SWITCHCRAFT-FIXTURE-END---$/{flag=0} flag' \
> /path/to/switchcraft/Tests/Fixtures/facts_corpus.json

WITCHCRAFT_COMMIT must be set on this command too — the helper reads the env var directly when emitting the provenance.witchcraftCommit field, and falls back to the literal string "UNKNOWN" if unset.

Then:

cd /path/to/switchcraft
python3 -m json.tool < Tests/Fixtures/facts_corpus.json > /dev/null

If json.tool is silent the file parsed cleanly. Commit it.

Why the patch is committed (instead of a standalone Cargo binary)

tests.rs already owns the canonical FACTS and QUERIES constants; duplicating them in a separate binary would split the source of truth. The patch adds one #[ignore]d #[test] and reuses the existing constants verbatim.

What the patch does, in one sentence

Adds a single #[ignore]d test that indexes the 33-fact corpus exactly the way test_end_to_end does, runs every query through crate::match_centroids (the pure-vector pipeline — no FTS, no RRF, no post-processing roll), and prints the captured (docId, score) lists between the marker lines for the runner to capture.

witchcraft-fixture-export.patch — regenerating cross-stack reference fixtures

Three Switchcraft tests (KMeansTests.referenceCentroidsParity, Q4CodecTests.referenceResidualsParity, CrossStackEmbeddingParityTests.referenceEmbeddingsParity) compare Switchcraft's k-means / Q4 codec / CoreML embedder against ground-truth reference data produced by upstream Witchcraft. The reference data lives in:

  • Tests/Fixtures/reference_centroids.{bin,json}
  • Tests/Fixtures/reference_residuals.{bin,json}
  • Tests/Fixtures/reference_embeddings.{bin,json}

This patch regenerates all three fixture pairs in one Witchcraft test run.

See adrs/013-reference-fixture-provenance.md for the full provenance, tolerance policy, and rationale for the patch-based generator.

Setup

Same one-time setup as witchcraft-fact-dump.patch above:

WITCHCRAFT_COMMIT=6ad59e51cfc89bcfb20756e3f05cf9429b7cb55f
git clone https://github.com/dropbox/witchcraft /tmp/witchcraft
cd /tmp/witchcraft
git checkout "$WITCHCRAFT_COMMIT"
make download                       # ~62 MB GGUF + tokenizer + config
git apply /path/to/switchcraft/scripts/witchcraft-fixture-export.patch

The fact-dump patch and the fixture-export patch insert at the same anchor in src/tests.rs. Apply only one at a time. To regenerate both fact and fixture data, apply one, run, git stash or git checkout src/tests.rs, then apply the other.

Trait method calls: as of the pinned Witchcraft commit, to_q4_bytes and from_companded_q4_bytes are trait methods in src/packops.rs. The patch imports PackOps via use crate::packops::PackOps as _ to bring them into scope. If the trait is renamed or the method signatures change, adjust the use statement accordingly. The dump_reference_centroids_fixture SQL also assumes embedding.value and bucket.center are the input/ output BLOB columns — adjust the SELECT statements if the schema has moved.

GGUF format note: make download produces assets/xtr.gguf in GGUF v2 format (the pinned Candle rev 5bd5618 emits v2). The Switchcraft reader accepts both v2 and v3 — see ADR 016.

linear.weight carve-out: the patch also patches tools/quantize-tool/src/main.rs to preserve the 2_Dense projection (linear.weight) as FP32 rather than quantising it to Q4_K. If the quantize-tool hunk does not apply cleanly due to context drift, manually add name != "linear.weight" && before the tensor.rank() == 2 condition on the Q4_K gate. See docs/porting/ggml-t5.md for the upstream PR tracking note.

Run

The patch adds three #[ignore]d tests:

Fixture Test name
reference_centroids.{bin,json} dump_reference_centroids_fixture
reference_residuals.{bin,json} dump_reference_residuals_fixture
reference_embeddings.{bin,json} dump_reference_embeddings_fixture

Each test prints two stdout sections, framed by markers:

---SWITCHCRAFT-<NAME>-JSON-BEGIN---
…JSON metadata…
---SWITCHCRAFT-<NAME>-JSON-END---
---SWITCHCRAFT-<NAME>-BIN-BASE64-BEGIN---
…base64-encoded raw FP32 LE blob…
---SWITCHCRAFT-<NAME>-BIN-BASE64-END---

Run all three at once. Rust's libtest harness only accepts a single filter string, so we pass the common prefix dump_reference_ and let it match all three tests:

WITCHCRAFT_COMMIT=$(git rev-parse HEAD) cargo test \
    --features t5-quantized \
    --lib \
    dump_reference_ \
    -- --nocapture --include-ignored \
    > /tmp/switchcraft-fixtures.out 2>&1

Alternatively run them as three separate commands (one filter each) and concatenate the outputs into /tmp/switchcraft-fixtures.out.

Extract

Per-fixture extraction follows the same awk-between-markers pattern as witchcraft-fact-dump.patch. JSON regions go to .json files, base64 regions decode into .bin files:

DEST=/path/to/switchcraft/Tests/Fixtures
SRC=/tmp/switchcraft-fixtures.out

for NAME in CENTROIDS RESIDUALS EMBEDDINGS; do
    LOWER=$(echo "$NAME" | tr A-Z a-z)
    awk -v b="---SWITCHCRAFT-${NAME}-JSON-BEGIN---" \
        -v e="---SWITCHCRAFT-${NAME}-JSON-END---" \
        '$0==b{flag=1; next} $0==e{flag=0} flag' "$SRC" \
        > "$DEST/reference_${LOWER}.json"
    awk -v b="---SWITCHCRAFT-${NAME}-BIN-BASE64-BEGIN---" \
        -v e="---SWITCHCRAFT-${NAME}-BIN-BASE64-END---" \
        '$0==b{flag=1; next} $0==e{flag=0} flag' "$SRC" \
        | openssl base64 -d -A \
        > "$DEST/reference_${LOWER}.bin"
done

openssl base64 -d -A is portable across macOS and Linux. If you prefer the base64 CLI, use base64 -D on macOS / BSD or base64 --decode on GNU coreutils (the flags differ; the form above avoids the split).

Validate:

cd /path/to/switchcraft
for f in reference_centroids reference_residuals reference_embeddings; do
    python3 -m json.tool < "Tests/Fixtures/${f}.json" > /dev/null
    test -s "Tests/Fixtures/${f}.bin"
done
swift test --filter "Reference"

If swift test is silent on the new tests, double-check the JSON files contain a valid provenance.witchcraftCommit matching the pin above — the loaders fail-fast on missing provenance fields.

Why patch-based instead of a Cargo binary

ADR 013(b) is the long form. Short version: Witchcraft's kmeans, Q4 codec, and bucket-centroid SQLite accessors are pub(crate) and not reachable from an external crate. Adding #[ignore]d tests inside upstream src/tests.rs reuses the existing FACTS constants and private-API access without requiring upstream to publish anything.

Per-fixture tolerance summary

ADR 013(d) is the canonical reference. Quick recap so the runner knows what each Swift parity test is asserting against the regenerated fixture:

  • reference_centroids.bin: Swift k-means on the embedded inputs region must produce centroids that, for each Swift centroid, find a reference centroid with cosine ≥ 0.99 (ADR 013(d)). The bound is loose because Lloyd's iteration on the small 33-fact corpus is init-sensitive and Swift / Rust don't share an RNG.
  • reference_residuals.bin: byte-exact encoder parity (Swift encodeResiduals ≡ Witchcraft to_q4_bytes) plus MSE ≤ 1e-3 decoder parity, both per ADR 003(a).
  • reference_embeddings.bin: per-token absolute diff ≤ 0.025 between Switchcraft FP32 CoreML output and Witchcraft Q4K GGUF output, per ADR 010(h). The bound is the measured FP32-vs-Q4K drift on the XTR T5 graph; matched-precision FP16-vs-Q4K is not achievable on this graph (ADR 010(c)).

If a regenerated fixture pushes any of these tolerances, the runner re-records provenance.witchcraftCommit (and provenance.computeUnits for the embeddings fixture) and the tolerance discussion happens in the PR that bumps the pin.