v0.1.5: large vaults: incremental keyword index, deferred compaction under watch, remembered model-load failures

CHANGELOG.md

@@ -1,5 +1,13 @@
 # Changelog
 
+## 0.1.5 (unreleased)
+
+Faster with large vaults. Measured on Linux with a synthetic vault of 50,000 chunks (20,000 in brackets), against 0.1.4:
+
+- After a note changes, a running `puremind serve` reloads the index on its next search in 0.8 s (0.3 s), down from 7.5 s (2.9 s). The keyword index is no longer built again from all the text: only chunks the previous snapshot did not hold are tokenized, and the BM25 scores are computed when a search asks for them. This also applies to every search during `puremind index --rebuild`, which changes the index note by note. Scores are the same as before; chunks with exactly equal scores are now listed in index order, and a keyword search costs 15 to 30 ms instead of a few. The first search of a new server process takes about the same time as before.
+- `puremind watch` no longer compacts the keyword index after every save, which took 0.5 s (0.2 s) and rewrote the whole keyword index each time. It now compacts once the vault has been quiet for a minute, at the latest five minutes after the change, and when it stops. The text of deleted and edited notes still leaves the index file at the end of the run that notices it; only their words stay in the keyword index until the compaction. A watch that crashes leaves a note in the index, so the next `puremind index` or `puremind watch` compacts. `puremind index` compacts at the end of every run, as before.
+- A model that fails to load is not tried again for a minute. Every search used to try again, which could mean waiting for the Hugging Face Hub or reading the weights each time, and searches that overlapped tried one after another. They now get the same error at once.
+
 ## 0.1.4 (2026-10-17)
 
 - A new embedding model, or a new `embedder_revision`, now needs `puremind index --rebuild`. `puremind index` without it stops with an error and changes nothing; 0.1.3 and earlier emptied the index and embedded every note again. A `puremind watch` still running on the old configuration, and a fresh `puremind index` on the new one, used to wipe each other's index in turn, with a full re-embed each time. Now the stale watch stops with an error and a hint to restart it, and the index stays as it is. An index that holds no notes takes a new model without `--rebuild`.

README.md

@@ -106,7 +106,7 @@ The reranker costs about 0.2 seconds per candidate on a CPU, so with the default
 
 Once both models are in the Hugging Face cache, pureMind makes no network requests: models load from the cache at their pinned revisions, and Hugging Face telemetry is turned off unless you have set `HF_HUB_DISABLE_TELEMETRY` yourself. The only network use is the one-time download, which sends your Hugging Face token if you have one configured. Set `HF_HUB_OFFLINE=1` to forbid downloads altogether; a model that is not downloaded then fails at once with a clear message. A cached copy that lacks a file the models need (after an interrupted download, or one made by another tool) is completed on the next run with network access, and reported by name when offline.
 
-The index is a second copy of your notes' text. It is created readable only by you (mode 0600 in a 0700 directory), and an index, `~/.puremind` directory or `config.toml` that an earlier version left more open is tightened when pureMind next uses it. When a note is deleted or edited, the index run that notices it removes the old text from the index file and its `-wal` file; if another process is reading the index at that moment, the `-wal` file is emptied by a later run. Text deleted while you used 0.1.1 or earlier is removed when 0.1.2 first opens the index. Copies of the index in backups or snapshots still hold it; to remove everything, delete `~/.puremind/index.sqlite` and the `-wal` and `-shm` files beside it, then run `puremind index` again.
+The index is a second copy of your notes' text. It is created readable only by you (mode 0600 in a 0700 directory), and an index, `~/.puremind` directory or `config.toml` that an earlier version left more open is tightened when pureMind next uses it. When a note is deleted or edited, the index run that notices it removes the old text from the index file and its `-wal` file; if another process is reading the index at that moment, the `-wal` file is emptied by a later run. The keyword index keeps the old words a little longer under `puremind watch`: it is compacted once the vault has been quiet for a minute, and at the latest five minutes after the change, rather than after every save. A watch that stops first compacts on its way out, and one that crashes leaves a note in the index so the next `puremind index` or `puremind watch` does it. `puremind index` compacts at the end of every run. Text deleted while you used 0.1.1 or earlier is removed when 0.1.2 first opens the index. Copies of the index in backups or snapshots still hold it; to remove everything, delete `~/.puremind/index.sqlite` and the `-wal` and `-shm` files beside it, then run `puremind index` again.
 
 ## Command line
 

SECURITY.md

@@ -20,7 +20,7 @@ Security fixes go into the latest 0.1.x release. Upgrade to it before reporting.
 - **What `puremind_get` serves.** Only the notes the indexer reads: regular `.md` and `.markdown` files of at most 2 MiB that, after resolving symlinks, lie inside the vault, outside hidden folders, `node_modules` and `__pycache__`. Absolute paths, `..`, NUL characters and paths over 1,024 characters are refused before the file system is touched, and refusals never contain absolute paths. A path is tried as given and then without surrounding white space, and each spelling must pass every check.
 - **Network use.** The only network access is the one-time download of the two models from Hugging Face, at pinned revisions, which sends your Hugging Face token if one is configured. Once they are cached, models load from the cache and nothing contacts the network. pureMind turns Hugging Face telemetry off unless you have set `HF_HUB_DISABLE_TELEMETRY` yourself. Set `HF_HUB_OFFLINE=1` to forbid downloads altogether. A cached copy that lacks a file the models need is completed by a download, or refused when offline, so a partial copy is never loaded as if it were complete.
 - **Model loading.** A model name is a Hugging Face id unless it starts with `/`, `./`, `../` or `~`, so a directory in the working directory is never loaded by accident. Models load with `trust_remote_code` off, pureMind downloads only safetensors weights (no pickle files), and `puremind serve` runs from the pureMind home directory. `sentence-transformers` 5.6.0 or newer is required because older releases could run code from a local model directory (CVE-2026-68770).
-- **The index holds your notes' text.** `~/.puremind/index.sqlite` is a second copy of every note's text. It and its `-wal` and `-shm` files are created mode 0600 in a 0700 directory, and existing files you own, including `~/.puremind` and `config.toml`, are tightened when pureMind uses them. The index run that notices a deleted or edited note removes the old text from the file and its `-wal` file (SQLite zeroes the freed pages, and the keyword index is compacted so it drops the deleted words). If another process is reading the index at that moment, the `-wal` file is emptied by a later run. Text deleted under 0.1.1 or earlier is removed when 0.1.2 first opens the index. Backups and snapshots of the index keep whatever they captured. To purge everything, delete the three files and re-index.
+- **The index holds your notes' text.** `~/.puremind/index.sqlite` is a second copy of every note's text. It and its `-wal` and `-shm` files are created mode 0600 in a 0700 directory, and existing files you own, including `~/.puremind` and `config.toml`, are tightened when pureMind uses them. The index run that notices a deleted or edited note removes the old text from the file and its `-wal` file (SQLite zeroes the freed pages). The keyword index is compacted so it drops the deleted words at the end of that run under `puremind index`; under `puremind watch` this waits until the vault has been quiet for a minute, five minutes at most, or until the watch stops, and a watch that crashes leaves a note in the index so the next `index` or `watch` run does it. If another process is reading the index at that moment, the `-wal` file is emptied by a later run. Text deleted under 0.1.1 or earlier is removed when 0.1.2 first opens the index. Backups and snapshots of the index keep whatever they captured. To purge everything, delete the three files and re-index.
 - **Hostile vault content.** The indexer only reads the vault. Symlink loops, pipes, undecodable names, oversized files, a cap of 10,000 chunks per note and a 200-character cap on each heading in a breadcrumb bound the work a vault can cause. A note is checked again just before it is read, so a symlink swapped in during a run is not followed.
 - **Notes are untrusted input to an agent.** Note text returned to an agent is data, not instructions, but a model may still follow instructions written in a note. Keep untrusted material out of the vault you connect to your agents.
 - **MCP SDK advisories.** The published advisories for the `mcp` package releases this project allows concern its HTTP, SSE and WebSocket transports, or its experimental task handlers (CVE-2026-52870, mcp 1.23.0 to 1.27.1). pureMind uses stdio only and never enables tasks, so none of them apply.

src/puremind_mac/__init__.py

@@ -1,3 +1,3 @@
 """pureMind for Mac: local memory for AI coding agents, served over MCP."""
 
-__version__ = "0.1.4"
+__version__ = "0.1.5"

src/puremind_mac/cli.py

@@ -18,6 +18,7 @@
 import sqlite3
 import sys
 import threading
+import time
 from collections.abc import Mapping, Sequence
 from pathlib import Path
 
@@ -114,6 +115,13 @@ def cmd_index(args, cfg: Config) -> int:
     return 0
 
 
+# Under `puremind watch` the keyword index is compacted (Store.purge: about half a second at 50,000
+# chunks) once the vault has been quiet this long after a run, not after every save...
+PURGE_WHEN_QUIET = 60.0
+# ...but at the latest this long after the first run that left words of removed text in it.
+PURGE_AT_LEAST_EVERY = 300.0
+
+
 def cmd_watch(args, cfg: Config) -> int:
     from .indexer import ModelMismatch, VaultMismatch, index_vault
     from .watcher import watch
@@ -122,25 +130,46 @@ def cmd_watch(args, cfg: Config) -> int:
     svc = _service(dataclasses.replace(cfg, vault=vault))
     stop = threading.Event()
     refused: list[ValueError] = []
+    owed_since: list[float | None] = [None]  # when a run first left the keyword index to compact
+    first = [True]
 
     def run() -> None:  # always called on this thread, one run at a time
+        # The first run compacts, and so pays what a watch that crashed left owing.
+        now = time.monotonic()
+        purge = first[0] or (owed_since[0] is not None and now - owed_since[0] >= PURGE_AT_LEAST_EVERY)
+        first[0] = False
         try:
-            stats = index_vault(vault, svc.store, svc.embedder)
+            stats = index_vault(vault, svc.store, svc.embedder, purge=purge)
         except (VaultMismatch, ModelMismatch) as exc:
             # A symlinked vault that now names another directory, or an index rebuilt for another
             # model since this watch read its config: every later run would fail the same way.
             refused.append(exc)
             stop.set()
             return
+        if purge:
+            owed_since[0] = None
+        elif stats.purge_deferred and owed_since[0] is None:
+            owed_since[0] = now
         print(stats.summary(), flush=True)
         for item in stats.skipped:
             print(f"  skipped: {_one_line(item)}", file=sys.stderr, flush=True)
 
+    def compact() -> None:
+        if svc.store.purge_pending():
+            svc.store.purge()
+        owed_since[0] = None
+
     print(f"watching {_one_line(str(vault.resolve()))} (Ctrl-C to stop)", flush=True)
     try:
-        watch(vault, run, delay=args.debounce, stop=stop)
+        watch(vault, run, delay=args.debounce, stop=stop, quiet=compact, quiet_after=PURGE_WHEN_QUIET)
     finally:
-        svc.store.close()
+        try:
+            compact()  # skipped by a crash or power cut: the record in the index carries it to the next run
+        except Exception as exc:  # noqa: BLE001 - must not hide why watching ended
+            print(f"warning: the keyword index was not compacted ({_one_line(exc)}); the next run does it",
+                  file=sys.stderr)
+        finally:
+            svc.store.close()
     if refused:
         if isinstance(refused[0], ModelMismatch):
             return _fail(

src/puremind_mac/indexer.py

@@ -61,6 +61,7 @@ class IndexStats:
     skipped: list[str] = field(default_factory=list)
     removal_skipped: str = ""  # why notes missing from the scan were not removed from the index
     stopped: bool = False  # the run was asked to stop before it finished
+    purge_deferred: bool = False  # the keyword index still holds words of text this run removed
 
     def summary(self) -> str:
         s = (
@@ -420,6 +421,7 @@ def index_vault(
     rebuild: bool = False,
     *,
     should_stop: Callable[[], bool] | None = None,
+    purge: bool = True,
 ) -> IndexStats:
     """Bring the index in line with the vault.
 
@@ -429,6 +431,12 @@ def index_vault(
     vectors from two models cannot be searched together. Notes cut by an earlier chunker are cut
     and embedded again. ``should_stop`` is polled between notes; once it returns True the run
     ends early without removing anything.
+
+    Text a run removes or replaces leaves the tables at once. Its words leave the keyword index
+    when the index is compacted (``Store.purge``), which takes about half a second at 50,000
+    chunks. With ``purge`` the run compacts at its end, also paying a compaction an earlier run
+    put off; without it, the run only records that one is owed (``stats.purge_deferred``) and the
+    caller pays it later, as ``puremind watch`` does.
     """
     vault = vault.expanduser().resolve()
     if not vault.is_dir():
@@ -488,7 +496,16 @@ def index_vault(
         for rel in gone:
             store.delete_file(rel)
             stats.removed += 1
-    if (stats.removed or stats.updated) and not store.purge():  # clear() purges by itself
+    emptied = True
+    if stats.removed or stats.updated:  # clear() purges by itself
+        if purge:
+            emptied = store.purge()
+        else:
+            emptied = store.defer_purge()
+            stats.purge_deferred = True
+    elif purge and store.purge_pending():  # owed by a `puremind watch` that stopped before paying it
+        emptied = store.purge()
+    if not emptied:
         log.warning(
             "another process was reading the index, so text deleted or replaced in this run stays in "
             "%s-wal until the next run", store.path,