Add TwelveLabs Marengo embedding provider - #2296
Open
mohit-twelvelabs wants to merge 1 commit into
Open
Conversation
Adds an opt-in embedding provider backed by TwelveLabs Marengo, which produces 512-dimensional embeddings in a shared multimodal space. This lets text queries retrieve over video/image/audio content embedded with the same model. - New TwelveLabsEmbeddingProvider (sync + async), wired into the embedding provider factory and registries - Selected via [embedding] provider = "twelvelabs"; requires TWELVELABS_API_KEY. Default model marengo3.0 - Existing providers and defaults are unchanged - Unit tests for config/validation plus an API-key-gated live test asserting a 512-dim embedding
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
A new opt-in embedding provider,
TwelveLabsEmbeddingProvider, backed by TwelveLabs Marengo. Marengo produces 512-dimensional embeddings in a shared multimodal space, so text queries can retrieve over video/image/audio content embedded with the same model. This PR implements the text path used by R2R's search and ingestion pipelines (sync + async), mirroring the existing Ollama/OpenAI providers.Why it helps R2R
R2R already supports several text embedding backends. Marengo extends R2R toward multimodal RAG: the same provider can embed text queries into the same vector space TwelveLabs uses for video/audio, which is a natural fit for indexing rich media alongside documents.
How to use it (opt-in, non-breaking)
Set
TWELVELABS_API_KEYin the environment. Defaults and all existing providers are unchanged; nothing is wired in unless you selectprovider = "twelvelabs". ThetwelvelabsSDK import is lazy, so it's only required when the provider is actually instantiated.Changes
core/providers/embeddings/twelvelabs.py— new provider (sync + async)supported_providers, and provider__init__exportstwelvelabs >=1.2.8added to thecoreoptional dependenciesall_possible_config.tomltests/unit/test_twelvelabs_embedding.py— config/validation unit tests (no network) plus an API-key-gated live test asserting a 512-dim embeddingTesting
[core]extras installed: all 4 tests pass, including the live Marengo call returning a 512-dim vector through the provider class.ruff formatandruff checkpass on all changed files.Per CONTRIBUTING, this is opened directly for review; happy to also file an issue or iterate on anything. You can grab a free API key at https://twelvelabs.io — there's a generous free tier.