Skip to content

Add TwelveLabs Marengo embedding provider - #2296

Open
mohit-twelvelabs wants to merge 1 commit into
SciPhi-AI:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

Add TwelveLabs Marengo embedding provider#2296
mohit-twelvelabs wants to merge 1 commit into
SciPhi-AI:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

What this adds

A new opt-in embedding provider, TwelveLabsEmbeddingProvider, backed by TwelveLabs Marengo. Marengo produces 512-dimensional embeddings in a shared multimodal space, so text queries can retrieve over video/image/audio content embedded with the same model. This PR implements the text path used by R2R's search and ingestion pipelines (sync + async), mirroring the existing Ollama/OpenAI providers.

Why it helps R2R

R2R already supports several text embedding backends. Marengo extends R2R toward multimodal RAG: the same provider can embed text queries into the same vector space TwelveLabs uses for video/audio, which is a natural fit for indexing rich media alongside documents.

How to use it (opt-in, non-breaking)

[embedding]
provider = "twelvelabs"
base_model = "marengo3.0"
base_dimension = 512

Set TWELVELABS_API_KEY in the environment. Defaults and all existing providers are unchanged; nothing is wired in unless you select provider = "twelvelabs". The twelvelabs SDK import is lazy, so it's only required when the provider is actually instantiated.

Changes

  • core/providers/embeddings/twelvelabs.py — new provider (sync + async)
  • Registered in the embedding factory, supported_providers, and provider __init__ exports
  • twelvelabs >=1.2.8 added to the core optional dependencies
  • Config reference example in all_possible_config.toml
  • tests/unit/test_twelvelabs_embedding.py — config/validation unit tests (no network) plus an API-key-gated live test asserting a 512-dim embedding

Testing

  • Ran the focused suite in a Python 3.12 env with the full [core] extras installed: all 4 tests pass, including the live Marengo call returning a 512-dim vector through the provider class.
  • ruff format and ruff check pass on all changed files.

Per CONTRIBUTING, this is opened directly for review; happy to also file an issue or iterate on anything. You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

Adds an opt-in embedding provider backed by TwelveLabs Marengo, which
produces 512-dimensional embeddings in a shared multimodal space. This
lets text queries retrieve over video/image/audio content embedded with
the same model.

- New TwelveLabsEmbeddingProvider (sync + async), wired into the
  embedding provider factory and registries
- Selected via [embedding] provider = "twelvelabs"; requires
  TWELVELABS_API_KEY. Default model marengo3.0
- Existing providers and defaults are unchanged
- Unit tests for config/validation plus an API-key-gated live test
  asserting a 512-dim embedding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants