Skip to content

feat: Add SemanticResolver component for entity resolution using TEI (#10985) - #13627

Closed
tech00exploere wants to merge 9 commits into
deepset-ai:mainfrom
tech00exploere:feat/semantic-resolver-component
Closed

tech00exploere wants to merge 9 commits into
deepset-ai:mainfrom
tech00exploere:feat/semantic-resolver-component

Conversation

@tech00exploere

Copy link
Copy Markdown

Related Issues

Proposed Changes

Adds a new SemanticResolver component under haystack/components/preprocessors/ to identify and merge semantically equivalent entity names in document metadata using the Hugging Face Text Embeddings Inference (TEI) /similarity endpoint.

The component includes:

  • TEI integration: Uses the /similarity endpoint to compare entity names in batches. Supports authentication through Haystack's Secret type.
  • Entity type isolation: By default, entities are grouped by entity_type (group_by_type=True), preventing entities from different categories from being merged.
  • Entity clustering: Uses a Union-Find (disjoint-set) algorithm to group similar entities. Comparisons between entities already connected through transitivity are skipped.
  • Canonicalization strategies: Supports most_frequent, shortest, longest, and first strategies to choose a canonical name for each cluster.
  • Metadata preservation: Returns new Document instances without modifying the input documents. Preserves document IDs, blobs, embeddings, and unrelated metadata.
  • Alias tracking: Stores original entity names under original_<entity_key>, preserving previously collected aliases across repeated runs.
  • Entity deduplication: Removes duplicate entities within each document based on their canonical name and entity type.
  • Pipeline support: Implements synchronous (run) and asynchronous (run_async) execution, context management, and component serialization/deserialization through to_dict and from_dict.
  • Documentation: Adds component documentation under docs-website/ and a Reno release note.

How did you test it?

Added 30 unit and integration tests in test/components/preprocessors/test_semantic_resolver.py, covering:

  • Default and custom initialization parameters and input validation.
  • Component serialization and deserialization, including pipeline integration.
  • Synchronous and asynchronous execution.
  • Transitive clustering, candidate pruning, and similarity threshold boundaries.
  • Canonicalization strategies and entity type isolation.
  • Document immutability and metadata preservation across repeated runs.
  • Alias deduplication within individual documents.
  • Handling of HTTP errors and malformed TEI responses.

Checks performed:

  • Unit tests: 30 passed.
  • ruff check: passed.
  • ruff format: passed.
  • mypy: passed with no errors.

Notes for the reviewer

  • Entity matching is performed within entity-type groups by default. This can be configured through group_by_type.
  • The resolver uses transitive clustering, so entities connected through qualifying similarity relationships are placed in the same cluster.
  • Original entity names are retained in metadata to avoid losing alias information during resolution.
  • The implementation is designed to work with both synchronous and asynchronous Haystack pipelines.

Checklist

AI disclosure: ### AI Disclosure

I investigated and implemented this feature myself and used an ai assistant only to help investigate the issue, explore possible approaches, and discuss design considerations. The implementation, tests, and changes in this pr are my own work.

@tech00exploere
tech00exploere requested a review from a team as a code owner November 9, 2026 06:32
@tech00exploere
tech00exploere requested review from anakin87 and removed request for a team November 9, 2026 06:32
@vercel

vercel Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

@priyanshuANDcoad is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 9, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added topic:tests P2 Medium priority, add to the next sprint if no P1 available labels Nov 9, 2026
@tech00exploere
tech00exploere force-pushed the feat/semantic-resolver-component branch from 10b0543 to 417ac42 Compare November 9, 2026 06:50
@github-actions github-actions Bot added P2 Medium priority, add to the next sprint if no P1 available and removed P2 Medium priority, add to the next sprint if no P1 available labels Nov 20, 2026
@anakin87

Copy link
Copy Markdown
Member

Hello, and thanks for contributing.

I'm closing this PR based on #10985 (comment).

(Additionally, a component based on HF TEI should live in Core Integrations)

@anakin87 anakin87 closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium priority, add to the next sprint if no P1 available topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add New SemanticResolver component

3 participants