Skip to content

Add shared sparse vector support - #2274

Open
tomz-alt wants to merge 1 commit into
cocoindex-io:mainfrom
tomz-alt:agent/generic-sparse-vectors
Open

Add shared sparse vector support#2274
tomz-alt wants to merge 1 commit into
cocoindex-io:mainfrom
tomz-alt:agent/generic-sparse-vectors

Conversation

@tomz-alt

Copy link
Copy Markdown
Contributor

Summary

  • add a canonical shared SparseVector value type, SparseVectorSchema, and connector-kit schema resolver
  • add sparse-vector support for PostgreSQL/pgvector and shared-value interop for zvec and Qdrant
  • fail fast in unsupported connectors and reject sparse mappings before they can be written as dense vector keys
  • preserve nullable and nested Annotated metadata precedence while centralizing validation
  • document the shared API and add connector, resolver, normalization, and regression coverage

Why

Sparse vectors were represented differently by each connector. That left nullable annotations, mapping-form values, schema conflicts, and backend overrides with inconsistent behavior. In the worst case, a sparse mapping could be accepted as a dense vector and only its keys would be persisted.

This change establishes one validated interchange format and one schema-resolution contract, then makes each connector either support it explicitly or reject it before writing.

Impact

PostgreSQL can now create and write sparsevec(n) columns, including HNSW indexes. zvec and Qdrant accept the same canonical sparse values, while dense-only connectors provide actionable errors. Existing zvec list-based dense vectors and legacy sparse definitions remain supported.

Validation

  • uv run pytest -q python/: 1,155 passed, 180 skipped
  • uv run mypy: no issues in 281 files
  • focused Ruff checks and git diff --check
  • docs build: 98 pages plus agent-output validation
  • real on-disk zvec insert/update/query coverage
  • real pgvector PostgreSQL sparse round-trip and HNSW index test
  • live Qdrant 1.15.5 dense, sparse, and hybrid query test

@ZhiHanZ

ZhiHanZ commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

I think it is ready for review @badmonster0 could you help me open it from the draft

@badmonster0
badmonster0 marked this pull request as ready for review July 14, 2026 00:25
@badmonster0
badmonster0 requested a review from georgeh0 July 14, 2026 00:26
@badmonster0

Copy link
Copy Markdown
Member

marked and tagged @georgeh0 ty!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants