Skip to content

feat: DB-side fuzzy job search (drop the in-memory index) - #8

Merged
felipebalbi merged 15 commits into
mainfrom
feat/db-fuzzy-search
Jun 17, 2026
Merged

felipebalbi merged 15 commits into
mainfrom
feat/db-fuzzy-search

Conversation

@felipebalbi

Copy link
Copy Markdown
Owner

Summary

Moves job fuzzy search — plus the "↑ N new" pill and the live poller's
change-detection — off the full-history in-memory JobIndex in paavo-web
and into SQLite, so the web process no longer holds the whole job history in
RAM just to search it. SQLite drives iteration and returns only the requested
page.

  • Ranking + membership in SQL. A custom fuzzy_score SQLite scalar function
    (SkimMatcherV2, registered on every connection) ranks matches; a subsequence
    LIKE '%a%l%m%c%x%' pre-filter decides membership. So almcx still matches
    and ranks alice … mcxa266, identically to the old in-memory matcher — but the
    expensive UDF runs only on matched rows, and the pagination COUNT(*) calls it
    zero times.
  • JobIndex deleted. /api/jobs reads SQLite directly (fuzzy page +
    search_count, or an as_of-pinned time-ordered page for a blank query); the
    "N new" pill is a COUNT(*) WHERE submitted_at > as_of; the poller's
    change-detection switches to a bounded GROUP BY state activity_digest
    (no full-table materialization). fuzzy-matcher moves paavo-web → paavo-db.
  • SPA debounce 150 → 250 ms, since each committed keystroke now triggers a
    DB-side scan.

No schema migration, no paavo-proto / wire change, no paavod change, no
/api/events contract change.
Design + plan: docs/superpowers/specs/2026-06-17-db-side-fuzzy-search-design.md,
docs/superpowers/plans/2026-06-17-db-side-fuzzy-search.md.

Notable design points / trade-offs

  • Count ≡ page is structural: search_count and search_index_page use a
    byte-identical lowercased haystack and the same subsequence pattern, so a page
    can never reference a row the count excludes. A divergence-guard test asserts
    the LIKE membership set equals the fuzzy_score IS NOT NULL set.
  • Accepted (documented) trade-offs: search is an honest O(n) table scan per
    committed (debounced) keystroke — fine at lab scale; the activity_digest can
    miss two offsetting same-tick state transitions and self-heals on the next poll
    (it's a refetch hint, never a correctness input).
  • Cross-branch note: if feat/dashboard-sql-aggregates lands after this, its
    "Recent activity" list should read JobRow::list_index_page(None, 0, 8) rather
    than reintroduce JobIndex.

Test plan

  • cargo fmt --all -- --check — clean
  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo test --workspace — green (362 passed, 0 failed; 6 hardware/tooling-gated ignores)
  • New coverage: db-level fuzzy match/rank, list ordering + as_of pin,
    count_newer boundary, activity_digest change-detection, the
    LIKE≡fuzzy_score divergence guard, search pagination; /api/jobs
    end-to-end through SQL (api_jobs); digest-driven SSE bump (api_events).
  • trunk build --release (crates/paavo-web-ui) — builds (workspace-excluded wasm32)
  • Manual smoke (human): PAAVO_FAKE_RUNNER=1 daemon + paavo-web; on
    /jobs, typing almcx ranks an alice … mcxa266 row first, clearing the
    box returns to the time-ordered list, the "↑ N new" pill updates on new
    submissions, and the query fires only ~250 ms after typing stops.

@felipebalbi
felipebalbi enabled auto-merge June 17, 2026 17:51
Resolves the spec section 9 cross-branch conflict: the dashboard work that landed on main reads live.index for its Recent activity list, but this branch removes the in-memory JobIndex. Repoint api/dashboard.rs::get recent_jobs to WebDb::jobs_list_page(None, 0, 8) (SQL, newest-first) and refresh the now-stale in-memory-index comments in api_dashboard.rs.
@felipebalbi
felipebalbi merged commit 144b90f into main Jun 17, 2026
3 checks passed
@felipebalbi
felipebalbi deleted the feat/db-fuzzy-search branch June 17, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant