feat(mcp): ephemeral-primary + persistent-attached two-database model (#29)

* refactor(mcp): two-database engine model + platform-default persistent path

Begins migrating hyperdb-mcp toward an ephemeral-primary, persistent-attached
session model. Each engine now holds two .hyper files at all times (when not
in --ephemeral-only mode):

- Ephemeral primary: $TMPDIR/hyperdb-mcp-<pid>-<seq>/scratch.hyper
  Created fresh per-engine, deleted on Drop. The connection is bound here;
  unqualified SQL routes here.
- Persistent attachment: at the platform-default location (or user-supplied
  path), attached under the reserved alias "persistent" during Engine::new.
  Survives across sessions.

Per-engine sequence number lets multiple Engines coexist in the same PID
(parallel test runners, restart-after-ConnectionLost) without colliding.

New module:
- paths: cross-platform resolution for the persistent-db default location
  using `dirs::data_dir()`. Override via HYPERDB_PERSISTENT_DB env var.

Engine API changes:
- workspace_path() -> ephemeral_path() and new persistent_path()
- is_persistent() -> has_persistent() + persistent_was_just_created()
- new resolve_target_db() resolves an optional database alias for tools
- PERSISTENT_ALIAS = "persistent" const exposed
- Drop always cleans up ephemeral; persistent stays where the user put it

attach::reset_search_path now respects the always-on persistent attachment:
re-pins to the primary's name instead of issuing RESET (which would leave
the persistent attachment without a working unqualified-name resolver).

Tests are all passing except for two clusters being addressed in later
iterations:
- saved_queries_tests (2 failures): WorkspaceStore still targets primary;
  iteration 7 will move it to the persistent attachment.
- table_catalog_tests (2 failures): catalog ensure_exists still targets
  primary; iteration 5 will make it presence-conditional and per-DB.

This is the second commit in a multi-step migration; --bare and --workspace
CLI flags still work (deprecation comes in iteration 3).

* feat(mcp): CLI surface for ephemeral-primary model + remove --bare

CLI changes:
- New `--persistent-db <PATH>`: replaces `--workspace`. Defaults to the
  platform data dir (or HYPERDB_PERSISTENT_DB env var). The deprecated
  `--workspace` is hidden but still accepted with a stderr warning;
  passing both is an error.
- New `--ephemeral-only`: skip persistent attachment entirely. Saved
  queries fall back to in-memory storage.
- Removed `--bare`. Catalog creation is now uniform (always seed when
  MCP creates a fresh .hyper file; never touch an existing file's
  catalog), so the opt-out flag became redundant. Users wanting a
  pristine .hyper for export can DROP TABLE _table_catalog after
  creation; subsequent opens won't recreate it.

HyperMcpServer:
- new() and with_no_daemon() drop the `bare` parameter.
- is_bare() removed.
- Saved-query store selection becomes "persistent if available, session
  otherwise" — same logic, but driven by --ephemeral-only instead of
  --bare.
- AttachRegistry no longer takes a catalog policy; `seed_catalog_on_create`
  becomes the constant default.

`_table_catalog` is now treated as an internal table by `is_internal_table`
so it doesn't appear in user-visible `describe_tables` output or
`total_rows`. The catalog's own `table_present`/`user_tables` helpers go
through the raw Catalog directly to bypass that filter.

attach::reset_search_path keeps the primary-name pin when a default
persistent attachment is in place — RESET to "$single" would break
unqualified resolution while persistent stays attached.

Tests updated:
- All `HyperMcpServer::new(path, ro, bare)` callsites lose their bare arg.
- Two `--bare`-specific tests deleted (bare_server_does_not_create_catalog,
  is_bare_reflects_constructor_argument).
- One detach-search-path test updated to expect the primary-name pin
  instead of "$single".
- Test helpers writing to "the workspace" updated to write into the
  persistent attachment via fully-qualified SQL.
- `table_exists` test helper goes through Catalog directly so it can see
  internal tables (which the new is_internal_table now filters out).

Five tests still failing in saved_queries and table_catalog:
- saved_queries: WorkspaceStore still targets primary; iteration 7.
- table_catalog: ensure_exists/reconcile/upsert_stub still target primary;
  iteration 5 will route them to the persistent attachment.

* refactor(mcp): drop seed_catalog_on_create from AttachRegistry

Now that --bare is gone, AttachRegistry's seed_catalog_on_create flag is
always true. Remove the field, the with_catalog_policy() constructor,
and the conditional in attach() — seeding now depends only on whether
MCP just created the file (file_was_created), matching the same uniform
policy the engine uses for the default persistent attachment.

The registry continues to hold *user-attached* databases only. The
default persistent attachment is owned by Engine itself and isn't
tracked in the registry; replay-on-reconnect re-issues only the user
attaches.

Tests:
- on_missing_create_does_not_seed_under_bare_policy deleted (the bare
  policy no longer exists; the matching positive test for non-bare
  policy stays as the canonical create-+-seed regression).

* feat(mcp): route _table_catalog to persistent attachment

The catalog tracks tables the user wants to keep around — i.e. tables
in the persistent attachment. Ephemeral scratch tables aren't worth
catalogging because the database is replaced every session. So:

- ensure_exists / list / get / upsert_stub / set_metadata / delete_for /
  reconcile / refresh_row_count all qualify SQL with
  "persistent"."public"."_table_catalog".
- When no persistent attachment is present (--ephemeral-only), all
  catalog operations no-op gracefully (Ok with empty/None) instead of
  erroring; set_metadata is the only path that surfaces a clear
  ReadOnlyViolation since the user's intent there is mutation.
- user_tables / table_present / row_count_of probe persistent's
  pg_catalog.pg_tables directly via fully-qualified SQL.
- Removed HYPERDB_NAMED_INTERNAL_TABLES from is_internal_table — the
  catalog never appears in describe_tables now (which only enumerates
  the connection's primary, i.e. ephemeral) so no filter is needed.

Tests: catalog tests update their fixtures to seed user tables in
"persistent"."public". (sed-driven batch update — every CREATE
TABLE / INSERT INTO in those tests now qualifies the target db).
attach_tests.copy_create_stubs_table_catalog_on_primary_workspace
asserts catalog presence via persistent's pg_tables.

Two saved_queries tests still failing pending iteration 7.

* feat(mcp): saved queries persist in the persistent attachment

WorkspaceStore now writes _hyperdb_saved_queries into the persistent
attachment ("persistent"."public"."_hyperdb_saved_queries") instead
of the connection's primary database. This matches user expectations:
saved queries are reference material that should outlive a single
session, and the persistent attachment is where curated, long-lived
data lives in the new model.

build_store remains driven by 'persistent path was supplied' — the
behavior is unchanged from the user's perspective. --ephemeral-only
sessions still get SessionStore (in-memory, dies with the process).

* test(mcp): regression tests for ephemeral-primary two-DB model

Eight new tests in tests/two_db_model_tests.rs covering the core
contracts of the new model:

- Engine::new(Some(path)) attaches the file as 'persistent' and
  has_persistent() reports true.
- Engine::new(None) produces an ephemeral-only engine; the 'persistent'
  alias is genuinely absent.
- Each Engine gets a distinct ephemeral path even when multiple Engines
  coexist in the same PID (parallel test runners, embedded uses).
- Persistent writes survive engine drop and are visible on recreate.
- Ephemeral writes are discarded on drop (the entire point).
- resolve_target_db routes None -> primary, 'persistent' -> persistent
  when present, errors with InvalidArgument when --ephemeral-only.
- Engine::status() exposes both database paths and the has_persistent
  flag. ephemeral_only mode reports persistent_path = null.

* docs(mcp): describe ephemeral-primary + persistent-attached model

README:
- Operating Modes section now leads with the two-database concept and
  explains how to target either DB from SQL via fully-qualified names.
- New 'Database storage' table documents --persistent-db default,
  --ephemeral-only, and the deprecated --workspace alias.
- Saved-queries persistence note updated (queries now land in the
  persistent attachment automatically).
- CLI Reference rewritten to match the actual flag set; --bare removed.
- Examples migrated from --workspace to --persistent-db.

DEVELOPMENT.md:
- 'Workspace Modes Internals' replaced with 'Two-Database Engine
  Model' covering the per-engine ephemeral path naming, the
  schema_search_path pin to the primary, and the catalog/saved-queries
  routing helpers.

CHANGELOG.md:
- Added entries for the two-database engine model, platform-default
  persistent path, --persistent-db / --ephemeral-only flags, and the
  catalog / saved-queries persistence move.
- Added 'Removed' section documenting --bare retirement.

* perf(mcp): cache _table_catalog presence on the engine

table_catalog::table_present previously ran a fresh
"persistent".pg_catalog.pg_tables probe on every catalog read/write
(get, list, delete_for, set_metadata, upsert_stub via ensure_exists).
For workloads that touch the catalog frequently — every ingest, every
DDL — that's one round-trip per call after the catalog has clearly
been created.

Add a catalog_present_cache: Mutex<Option<bool>> on Engine. The cache:

- Lives for the engine's lifetime, which is the right TTL: a
  ConnectionLost reconnect builds a fresh Engine, naturally resetting
  the cache.
- Short-circuits to Ok(false) in --ephemeral-only mode without
  running the probe at all.
- Gets primed to Some(true) by ensure_exists immediately after CREATE
  TABLE IF NOT EXISTS so the next catalog op skips the probe.
- Returns the cached value on every subsequent call until the engine
  is dropped.

Two new tests in two_db_model_tests verify the cache:
- catalog_presence_probe_is_cached: probe runs once, mark_catalog_present
  flips to true without running the probe again.
- catalog_presence_short_circuits_in_ephemeral_only: probe never runs
  when has_persistent() is false.

* fix(mcp): watchers auto-recover after hyperd restart

When the daemon restarts hyperd, every connection in a watcher's pool
becomes invalid. Previously the watcher would route every subsequent
file to 'failed/' until the user noticed and re-issued
watch_directory.

Now:
- The watcher's pool lives behind Arc<RwLock<Arc<Pool>>> so it can be
  swapped atomically.
- Each per-file ingest reads the current pool from the slot, calls
  ingest_one_ready_file, and inspects the result.
- If the result is a connection-lost error (per is_connection_lost),
  rebuild_watcher_pool builds a fresh Pool from the engine's *current*
  endpoint and the ingest retries exactly once on the new pool.
- Persistent failures (the retry also fails) fall through to the
  existing 'failed/' move logic so a permanently-broken file doesn't
  keep the watcher pinned in retry loops.
- The initial-sweep path uses the same recovery wrapper so a watcher
  registered just after a hyperd hiccup still ingests the backlog.

Internal refactor:
- Pure ingest path extracted into ingest_one_ready_file (returns
  Result<u64, McpError> with no file-system side effects).
- process_ready_with_recovery wraps it: handles symlink rejection,
  the retry-on-connection-lost loop, and the success/failure file
  moves. The old process_ready_async is gone in favor of these two.

The DEVELOPMENT.md known-limitation entry for watchers is updated to
reflect the new behavior; CHANGELOG gets a Fixed entry.

Author: StefanSteiner

Cargo.lock

@@ -31,6 +31,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
   attempts are rate-limited to 3 per 60 seconds; exceeding the limit
   triggers daemon shutdown so the user sees the failure clearly
   rather than spinning silently.
+- **Two-database engine model.** Every session now has both an
+  ephemeral primary (created fresh per-session, deleted on exit) AND
+  a persistent attachment under the alias `"persistent"`. Unqualified
+  SQL routes to the ephemeral primary (the LLM's scratch space);
+  fully-qualified SQL like
+  `INSERT INTO "persistent"."public"."customers" ...` writes to
+  storage that survives across sessions.
+- **Platform-default persistent path.** When `--persistent-db` is
+  unset, the persistent file lives at the platform data directory:
+  `~/Library/Application Support/hyperdb/workspace.hyper` on macOS,
+  `~/.local/share/hyperdb/workspace.hyper` on Linux,
+  `%APPDATA%\hyperdb\workspace.hyper` on Windows. Override with
+  `HYPERDB_PERSISTENT_DB`.
+- **New CLI flags:** `--persistent-db <PATH>` (replaces `--workspace`,
+  which is kept as a deprecated alias with a stderr warning); and
+  `--ephemeral-only` to skip the persistent attachment entirely.
+- The `_table_catalog` and `_hyperdb_saved_queries` meta-tables now
+  live in the persistent attachment instead of the connection's
+  primary, so saved queries automatically persist across sessions
+  without any flag toggling.
+
+### Removed
+
+- **`--bare` flag.** Catalog seeding is now uniform: created when MCP
+  creates a fresh `.hyper`, never touched on existing files. Users
+  who want a pristine `.hyper` for export can `DROP TABLE _table_catalog`
+  once after creation; subsequent opens won't recreate it.
+
+### Performance
+
+- **Per-engine `_table_catalog` presence cache.** Catalog reads/writes
+  used to round-trip a `pg_catalog.pg_tables` probe on every call;
+  now the existence check is cached for the engine's lifetime and
+  primed immediately after `CREATE TABLE IF NOT EXISTS`.
+
+### Fixed
+
+- **Watcher recovery after hyperd restart.** The watcher's connection
+  pool now auto-rebuilds when a per-file ingest hits a connection-lost
+  error (typically after the daemon restarts hyperd). Each ingest gets
+  one retry on a fresh pool; persistent failures still flow into the
+  standard `failed/` move so a single broken file can't pin the
+  watcher in retry loops.
 
 ## [0.1.1] - 2026-05-13
 

hyperdb-mcp/CHANGELOG.md

@@ -41,6 +41,7 @@ notify = "8"
 tokio-util = { version = "0.7", features = ["rt"] }
 tempfile = { workspace = true }
 sqlformat = "0.5.0"
+dirs = "5"
 
 [target.'cfg(unix)'.dependencies]
 libc = "0.2"

hyperdb-mcp/Cargo.toml

@@ -196,12 +196,16 @@ Derived fields are merged into the serialized JSON so callers get a self-contain
 
 ---
 
-## Workspace Modes Internals
+## Two-Database Engine Model
 
-- **Ephemeral** — `Engine::new(None)` creates a temp directory (`$TMPDIR/hyperdb-mcp-<pid>/`) with a `workspace.hyper` file. Cleaned up on process exit. In daemon mode, `Engine::drop` issues `DETACH DATABASE` (releasing Hyper's file lock — required on Windows) before `remove_dir_all` deletes the temp directory.
-- **Persistent** — `Engine::new(Some(path))` uses the caller-supplied path. Parent directories are created automatically. `~` is expanded via `$HOME` (no shell crate dependency).
+Every `Engine` holds:
 
-Logs (both `hyperd` server logs and the MCP client log) land in the same directory as the workspace file. The `status` tool reports log paths so operators know where to look.
+- **Ephemeral primary** at `$TMPDIR/hyperdb-mcp-<pid>-<seq>/scratch.hyper`. Created fresh on each `Engine::new` (`<seq>` is a process-wide atomic so parallel test runners and restart-after-ConnectionLost reconnects get distinct files). The connection is *bound* to this database — unqualified SQL routes here. `Engine::drop` always deletes the per-engine temp directory.
+- **Persistent attachment** under the alias `"persistent"`. When `Engine::new(Some(path))` is called, the engine runs `CREATE DATABASE IF NOT EXISTS '<path>'` (creating the file if missing) and `ATTACH DATABASE '<path>' AS "persistent"`. Then `SET schema_search_path = '<primary_db>'` keeps unqualified resolution routing into the ephemeral primary. `Engine::new(None)` skips this step (the `--ephemeral-only` mode); `engine.has_persistent()` reflects which mode was selected.
+
+Catalog and saved-queries meta-tables live in the persistent attachment, qualified via `"persistent"."public"."_table_catalog"` etc. When the engine has no persistent attachment, those operations no-op. See `crate::table_catalog::qualified_catalog` and `crate::saved_queries::WorkspaceStore::qualified_table` for the routing helpers.
+
+Logs land next to the persistent file when one is supplied (so users find them in a stable per-project location); ephemeral-only sessions log to `$TMPDIR/hyperdb-mcp-<pid>/`. The `status` tool reports `ephemeral_path`, `persistent_path`, and `has_persistent` so operators can confirm where data lives.
 
 ## Daemon Mode Internals
 
@@ -239,7 +243,7 @@ Two new code paths fire `report_hyperd_error_to_daemon` (best-effort, 200ms time
 ### Known limitations
 
 - **Hung-but-alive `hyperd`** (TCP listening, but unresponsive to queries) is NOT detected. The monitor's `try_wait()` returns `None` for a hung process; client tool calls hang on the read side without producing a `ConnectionLost` error. Operator recovery is `hyperdb-mcp daemon stop` followed by reconnect.
-- **Watchers (background tasks holding `AsyncConnection` handles in `WatcherRegistry`)** do not currently auto-reconnect after a hyperd restart. They will go quiet until the user re-issues `watch_directory`.
+- **Watchers** auto-recover from hyperd restarts: when an ingest fails with a connection-lost error, the watcher rebuilds its connection pool against the engine's current endpoint and retries the file once. Persistent failures (the second attempt also fails) fall through to the standard `failed/` move so a single broken file can't keep the watcher pinned in retry loops.
 
 See `src/daemon/{mod,discovery,health,run,spawn}.rs` for the full implementation.