[puretensor]

The three known performance issues at 20k+ chunks. Measurement showed the biggest cost is different from the one listed first: it is the server's in-memory bm25s rebuild after every write (7.5 s at 50k chunks), not the FTS5 optimize (0.5 s).

Measured

Linux (fox-n1, unloaded), synthetic vault of 5-chunk notes, ~170 Zipf-distributed words per chunk, 1024-dim vectors. v0.1.4 vs this branch, same machine and index:

	20k chunks	50k chunks
next search after a save (serve reload)	2.86 s → 0.28 s	7.53 s → 0.79 s
end of each watch run	0.22 s → 0.002 s	0.50 s → 0.002 s
compaction, once per quiet spell	0.21 s	0.54 s
first load + keyword index	3.05 s → 2.58 s	7.63 s → 6.24 s
keyword query (2–20 words)	0.2–0.8 ms → 7–11 ms	0.4–2 ms → 14–27 ms

The keyword-query cost is the trade: scoring happens at query time over all postings (np.isin(kind="table")), instead of in a precomputed score matrix. +25 ms is small next to embedding the query and reranking. If it matters later, a term-sorted layout costs about 0.25 s more per reload and makes queries O(postings).

(tensor-core numbers were unusable: under host memory pressure, with swap 35 of 39 GB, the reload stalled in kernel page reclaim for minutes in any process that allocates a few hundred MB.)

1. Reload: incremental keyword index (retrieval.py, service.py)

• Lexicon replaces bm25s.BM25. It is BM25 scored at query time, exactly as bm25s scores it: Lucene idf/tfc, k1 1.5, b 0.75, document length = tokens after stopwords, repeated query tokens counted again, unknown tokens dropped. It is built from per-chunk (term id, count) arrays with a few vectorised numpy operations.
• LexiconBuilder (one per Service, used under its lock) keeps those arrays for the last snapshot's chunks, keyed by a blake2b digest of the text. Chunk rowids are reused after a delete, so the key can't be the chunk id. A reload tokenizes only chunks it hasn't seen. It prunes text that left the index, copies the vocab dict when it grows (so a snapshot still being searched keeps its own), and renumbers to drop unused words once they make up half the vocabulary. A failed build leaves its state unchanged.
• bm25s.tokenize is unchanged, so tokens are identical. Scores match bm25s within float32 rounding; ties are now ordered by position (bm25s's argpartition order is arbitrary).
• Store.load_index joins the vector blobs into one buffer (and refuses mixed lengths with a --rebuild hint).

2. Watch: deferred FTS5 compaction (indexer.py, store.py, watcher.py, cli.py)

• index_vault(purge=False) records purge_pending in meta and only checkpoints the WAL. The text of deleted or edited chunks still leaves the tables, and is zeroed, at the end of the run; only their words stay in the FTS5 segments until optimize.
• cmd_watch compacts:
  • on its first run (paying a debt left by a crashed watch);
  • when watch() calls its new quiet hook, 60 s after a run with no change since;
  • at the latest 300 s after the first deferral, for a vault that never goes quiet;
  • on exit.
• puremind index still compacts every run, and also pays a pending debt. purge() and clear() clear the record.
• README and SECURITY.md state the new timing.

3. Failed model loads (models.py)

• _LoadOnce wraps both models' loaders. It remembers a ModelLoadError for LOAD_RETRY_AFTER (60 s) per instance. Callers within that window, including threads queued on the lock, get a fresh ModelLoadError with the same message and cause, without touching resolve_model or the disk. The cause chain is kept, so the indexer's isinstance(exc.__cause__, ModelLoadError) check is unchanged. A success clears the memory.

Validation

• pytest -q: 621 passed, 6 skipped. ruff check src tests bench (0.17.1): clean.
• New tests:
  • bm25s score parity on 6 random corpora (empty, stopword-only and CJK documents; queries with repeated, unknown and stopword-only words);
  • incremental build vs fresh build;
  • only new text is tokenized; pruning; vocabulary compaction with an older lexicon still answering; a failed build; tie order;
  • a Service reload after one save tokenizes exactly one chunk;
  • the purge_pending lifecycle;
  • deferred compaction in index_vault (the canary text is gone after the paying run);
  • cmd_watch compaction on the first run, when quiet, on exit, and at the cap;
  • the quiet hook firing once per quiet spell;
  • the model-load cool-down (both models, retry after it, success afterwards) and overlapping callers trying once.
• The real-models job was started on this branch (run 39066192192), since fix 3 touches the model-loading path.

Known, not fixed here

• tests/test_watcher.py::test_a_burst_of_writes_gives_one_run flaked once (3 runs instead of 2) in the build and installed wheel job of run 39066192192, and passed in every other job. That test calls watch() without the new hook, so it runs code this PR leaves unchanged. With a 0.2 s debounce, a loaded runner can split the 16-write burst into two runs. Pre-existing timing sensitivity.
• The macOS failures on the first push were the new quiet-hook test assuming exactly one startup run; FSEvents can report the vault's creation after watching starts. The test now tolerates that (second commit).

🤖 Generated with Claude Code

https://claude.ai/code/session_011zd8G1cibJoQPVbZEpVLYs