Skip to content

feat(search-pipeline): compose search indexing as a pipeline instance#561

Merged
ddeboer merged 0 commit into
feat/search-rebuild-writersfrom
feat/search-pipeline
Jul 7, 2026
Merged

feat(search-pipeline): compose search indexing as a pipeline instance#561
ddeboer merged 0 commit into
feat/search-rebuild-writersfrom
feat/search-pipeline

Conversation

@ddeboer

@ddeboer ddeboer commented Jul 6, 2026

Copy link
Copy Markdown
Member

Slice 3 of #534, stacked on #560: the new @lde/search-pipeline package that makes search indexing an @lde/pipeline Configurable Pipeline instance.

What it does

searchIndexWriter({ schema, writer }) is the one type-changing step (quad → document), shared across engines – which is why it lives here and not inside an engine adapter:

  • buffers each dataset’s extracted CONSTRUCT quads until the dataset flushes;
  • frames and projects them with @lde/search’s projectGraph (engine-agnostic);
  • writes the documents to the engine writer’s run before forwarding the flush, so an In-place stale sweep never races its own documents;
  • passes the run lifecycle through unchanged (run context, per-dataset flush outcome, reset for the dump-fallback discard, commit/abort) – the engine writer’s update mode governs, and the pipeline never branches on it.

With this, a search index rebuild is a plain Pipeline with an extraction SparqlConstructReader and writers: searchIndexWriter({ schema, writer: new BlueGreenRebuild(...) }) – reusing Discovery → Selection → change-gate (skip-unchanged) instead of the bespoke full-rebuild indexer.

Scope notes

Releasing

New package: after merge, the first version must be published manually per the npm-bootstrap-package flow (npm view @lde/search-pipeline → 404 today).

Part of #534 (slice 3 of 4; next: per-reference label sources + facet-by-name search).

@ddeboer ddeboer force-pushed the feat/search-pipeline branch from 9ef3ce7 to cf1d24b Compare July 6, 2026 19:12
@ddeboer ddeboer force-pushed the feat/search-rebuild-writers branch 2 times, most recently from c2f4868 to bee9710 Compare July 7, 2026 17:29
@ddeboer ddeboer force-pushed the feat/search-pipeline branch from cf1d24b to f6ee061 Compare July 7, 2026 17:29
@ddeboer ddeboer merged commit 46be36a into feat/search-rebuild-writers Jul 7, 2026
@ddeboer ddeboer force-pushed the feat/search-rebuild-writers branch from bee9710 to 46be36a Compare July 7, 2026 17:45
@ddeboer ddeboer deleted the feat/search-pipeline branch July 7, 2026 17:45
@ddeboer ddeboer force-pushed the feat/search-pipeline branch from f6ee061 to 46be36a Compare July 7, 2026 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant