WEC RAG Studio

A configurable, multi-tenant RAG (Retrieval-Augmented Generation) backend built with FastAPI. WEC uses this to build independent RAG pipelines for each of their clients — every aspect of the pipeline (chunking strategy, embedding provider, retrieval blend, reranking) is configurable at runtime via API, no redeploys needed.

Architecture

                         ┌─────────────────────────────────────────────────┐
                         │                   FastAPI Backend                │
                         │                                                   │
  ┌──────────┐  upload   │  ┌──────────────────────────────────────────┐   │
  │  WEC UI  │──────────►│  │           Ingestion Pipeline             │   │
  └──────────┘           │  │  LiteParse → Chunker → Embedder ─────┐  │   │
                         │  └──────────────────────────────────────┼──┘   │
                         │                                          │       │
                         │                               ┌──────────▼────┐  │
                         │                               │               │  │
                         │                          ┌────► Qdrant (VDB)  │  │
                         │                          │    │               │  │
                         │  ┌──────────────────┐   │    └───────────────┘  │
  ┌──────────┐  query    │  │ Retrieval Pipeline│   │                       │
  │  WEC UI  │──────────►│  │                  │   │    ┌───────────────┐  │
  └──────────┘           │  │ QueryTransformer  │   │    │   OpenSearch  │  │
                         │  │ Vector + BM25 ───►│───┘    │   (BM25)     │  │
                         │  │ RRF Fusion        │        └───────────────┘  │
                         │  │ Dedup + Reranker  │                           │
                         │  └──────────────────┘        ┌───────────────┐  │
                         │                               │  PostgreSQL   │  │
                         │                               │  (metadata +  │  │
                         │                               │   configs)    │  │
                         │                               └───────────────┘  │
                         │                                                   │
                         │                               ┌───────────────┐  │
                         │                               │     MinIO     │  │
                         │                               │  (raw files)  │  │
                         │                               └───────────────┘  │
                         └─────────────────────────────────────────────────┘

Multi-tenancy

All data is scoped to a tenant_id. There is one OpenSearch index (rag_chunks) and per-embedding-dimension Qdrant collections shared across tenants — the default-dim collection keeps the bare QDRANT_COLLECTION name (e.g. rag_chunks) and other dimensions auto-route to {base}_{dim} (e.g. rag_chunks_1536, rag_chunks_3072). Isolation is enforced by filtering on tenant_id in every query. PostgreSQL tables use a foreign-key tenant_id column with cascade deletes. The documents.embedding_dimensions column records which collection each doc lives in so DELETE can route to the right one.

Configuration system

Pipelines are split into two independent rows in PostgreSQL — ingestion_configs (chunking + embedding) and retrieval_configs (query transform, search blend, fusion, dedup, reranker, top_k). A retrieval config links to one or more ingestion configs via a join table; all linked configs must share the same embedding model + dimensions so the query embedder produces vectors of the right shape for every corpus the retrieval searches.

// ingestion_configs.config
{
  "chunking":  { "strategy": "recursive", "chunk_size": 512, ... },
  "embedding": { "provider": "zeroentropy", "model": "zembed-1", "dimensions": 2560, ... }
}

// retrieval_configs.config
{
  "query_transformation": {...}, "vector_search": {...}, "keyword_search": {...},
  "fusion": {...}, "deduplication": {...}, "reranker": {...}, "top_k": 5
}

Configs are addressed by ID at runtime — there is no "active config" concept. Every document upload specifies which ingestion config to use (POST /tenants/{tid}/ingestion-configs/{icid}/documents) and every query specifies which retrieval config (POST /tenants/{tid}/retrieval-configs/{rcid}/query). A tenant can hold any number of ingestion and retrieval configs in parallel; callers pick the one they want per request.

Startup checks

On FastAPI startup the app provisions the data stores it depends on (idempotent, safe to run repeatedly):

MinIO bucket created if missing (when STORAGE_BACKEND=minio)
Default Qdrant collection (QDRANT_COLLECTION, default rag_chunks) created with QDRANT_VECTOR_SIZE dimension. Additional per-dim collections are created lazily on first upsert.
OpenSearch index rag_chunks created with the BM25 + English-stemmer mapping

Failures are logged as warnings rather than crashing the process, so partial deploys (e.g. missing MinIO during local tests) don't block boot.

Latency & connection pooling

Hot-path latency tuning landed in this release. Knobs to know:

Singleton Qdrant + OpenSearch clients. QdrantVectorStore and OpenSearchBM25 instances are cached process-wide (keyed by embedding dimension for Qdrant; single instance for OpenSearch) via get_qdrant_store() / get_opensearch_store(). Previously every run_retrieval call constructed fresh clients and paid a TLS handshake against Qdrant Cloud + AWS OpenSearch (~100-200ms each). Lifespan shutdown closes them cleanly via close_all_qdrant_stores() / close_opensearch_store().
Long-lived TEI httpx clients. TEIEmbedder and TEIReranker hold a per-instance httpx.AsyncClient (with keep-alive limits) instead of async with-ing one per call, so repeated embeds/reranks reuse the same TCP/TLS connection.
Explicit LiteLLM defaults. Every litellm.acompletion / aembedding call site passes timeout=30 (and num_retries=1 for completions) so a wedged provider call can't hang the request.
Bounded chat output. ChatRequest.max_tokens defaults to 2048 (caller-overridable, range 1-8192). Without this, the JSON /chat endpoint could ramble to the model's max and pin TTFT to full generation.
Parallel citation presigning. /chat and /chat/stream fan out generate_download_url calls via asyncio.gather, so adding more citations doesn't multiply round-trips against MinIO / customer S3.
Streaming chat. POST .../chat/stream returns SSE so the UI gets first-token in ~retrieval-latency + ~600ms instead of waiting for the full JSON response. See #### Streaming chat under the Chat (RAG) section.

For production throughput, the entrypoint should also bump uvicorn workers (--workers $N --loop uvloop); each worker holds its own singleton clients.

Observability & cost

Every ingestion and retrieval run writes a row to the executions table — status, timings, and a metrics JSONB with token counts, *_ms phases, and cost. Costs are computed best-effort via LiteLLM get_model_info; providers without published pricing (ZeroEntropy, HuggingFace) leave *_cost_usd null with a warning rather than fabricate zero.

Two cost surfaces complement the executions log:

Pre-flight prediction — POST /ingestion-configs/{icid}/preview-cost accepts a sample file or sample_text and returns the cost the config would incur, without any DB writes. Use this at chunking-config-selection time.
Post-flight roll-up — GET /ingestion-configs/{icid} carries a cost_summary block aggregating across all executions for that config (total_cost_usd, executions_count, successful_count, failed_count, latest_execution_at, avg_cost_per_document_usd).

File size & contextual chunking limits

Format allow-list — only PDF, DOCX, TXT, MD, CSV are accepted. Other types return 415 Unsupported Media Type with a structured error.
30 MB upload cap — enforced at the HTTP layer (POST /tenants/{tid}/ingestion-configs/{icid}/documents returns 413 Content Too Large) and again at parse time in the ingestion pipeline as defense in depth. Configurable via MAX_UPLOAD_BYTES.
Contextual chunking warning — when an ingestion config uses chunking.strategy=contextual, every relevant API response (config create/update, document upload) carries a warnings array explaining the per-chunk Claude cost.
Auto-split for oversize docs — if a document's token count exceeds 85% of the configured LLM's context window, the contextual chunker splits the text at a paragraph boundary into halves and processes each independently. Resulting chunks carry auto_split: true and suggested_models: [...] in their metadata so the UI can recommend a larger-context alternative.

Requirements

Python 3.11+
Node.js ≥ 18 (required by LiteParse for document parsing)
Docker + Docker Compose

Quick Start

There are two deploy targets, picked by which compose file you use:

Target	Compose file	What it brings up
Local-bundled	`docker-compose.yml`	Postgres + Qdrant + OpenSearch + MinIO + app (all in containers). Single-command for a from-scratch laptop / VM.
Hosted-services	`docker-compose.hosted.yml`	Only the app. Talks to externally-hosted Postgres, Qdrant, AWS OpenSearch, and object storage (S3 or hosted MinIO). Use this for production.

Both compose files build the same Dockerfile; the difference is purely which services are included and which env vars are required.

Option A — Local-bundled (laptop / VM)

cp .env.example .env
# Generate the three required secrets and paste them into .env:
python -c "from cryptography.fernet import Fernet; print('ENCRYPTION_KEY=' + Fernet.generate_key().decode())"
python -c "import secrets; print('STATIC_API_KEY=wec_sk_' + secrets.token_hex(32))"
python -c "import secrets; print('ADMIN_API_KEY=wec_admin_' + secrets.token_hex(32))"

docker compose up --build -d
curl http://localhost:8000/health

Option B — Hosted-services (production)

cp .env.hosted.example .env
# Fill in real connection URLs/creds for each external service, plus the
# three secrets (ENCRYPTION_KEY / STATIC_API_KEY / ADMIN_API_KEY).
docker compose -f docker-compose.hosted.yml up --build -d
curl http://localhost:8000/health

The entrypoint runs alembic upgrade head against your hosted Postgres before starting uvicorn, so a fresh DB is migrated automatically. AWS OpenSearch (basic auth + TLS) is supported via OPENSEARCH_USE_SSL=true; Qdrant Cloud is supported via QDRANT_API_KEY + QDRANT_USE_HTTPS=true.

Option C — Local dev (infra in Docker, app on host)

docker compose up -d postgres qdrant opensearch minio
cp .env.example .env   # fill in secrets as above

pip install -e ".[dev]"
npm install -g @llamaindex/liteparse   # LiteParse requires Node.js ≥ 18
# Verify: `liteparse --help` should print usage. Without this CLI, every
# document upload fails at the parse step. For scanned PDFs needing OCR,
# also install Tesseract or point LiteParse at an OCR server (see
# https://developers.llamaindex.ai/liteparse/).

alembic upgrade head
uvicorn app.main:app --reload --port 8000

Interactive docs at http://localhost:8000/docs.

Generating the secrets

# ENCRYPTION_KEY — Fernet base64 key for stored provider API keys
from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())

# STATIC_API_KEY — single-user bearer (authorizes every route, including admin)
import secrets
print("wec_sk_" + secrets.token_hex(32))

# ADMIN_API_KEY — only needed if you want a separate admin token alongside
# the per-tenant auth flow. Optional if STATIC_API_KEY is set.
print("wec_admin_" + secrets.token_hex(32))

Environment Variables

All variables are read from .env (or the real environment). Every variable has a default so the service starts without a .env in development — only ENCRYPTION_KEY and an auth secret (STATIC_API_KEY or ADMIN_API_KEY) are required.

Variable	Default	Description
`APP_ENV`	`development`	Set to `production` to silence SQL echo
`APP_SECRET_KEY`	`change-me`	Application secret
`LOG_LEVEL`	`INFO`	Python logging level
`DATABASE_URL`	`postgresql+asyncpg://ragstudio:ragstudio@localhost:5432/ragstudio`	Async PostgreSQL DSN
`QDRANT_HOST`	`localhost`	Qdrant host
`QDRANT_PORT`	`6333`	Qdrant gRPC/REST port
`QDRANT_COLLECTION`	`rag_chunks`	Base name for Qdrant collections. The default-dim collection uses this name verbatim; non-default dims auto-route to `{base}_{dim}` (e.g. `rag_chunks_1536`).
`QDRANT_VECTOR_SIZE`	`2560`	Dimension of the default (bare-name) Qdrant collection. Other dimensions are handled automatically — see "Multi-tenancy" above.
`OPENSEARCH_HOST`	`localhost`	OpenSearch host
`OPENSEARCH_PORT`	`9200`	OpenSearch port
`OPENSEARCH_INDEX`	`rag_chunks`	OpenSearch index name
`OPENSEARCH_USER`	`admin`	OpenSearch username
`OPENSEARCH_PASSWORD`	`admin`	OpenSearch password
`MINIO_ENDPOINT`	`localhost:9000`	MinIO endpoint (no scheme)
`MINIO_ACCESS_KEY`	`minioadmin`	MinIO access key
`MINIO_SECRET_KEY`	`minioadmin`	MinIO secret key
`MINIO_BUCKET`	`rag-documents`	Bucket for uploaded documents
`MINIO_SECURE`	`false`	Set to `true` for HTTPS
`STORAGE_BACKEND`	`minio`	`minio` or `s3` — picks the storage adapter
`AWS_REGION`	`us-east-1`	AWS region (when `STORAGE_BACKEND=s3`)
`AWS_ACCESS_KEY_ID`	(empty)	AWS access key. Empty → boto3 default credential chain (env / IAM role)
`AWS_SECRET_ACCESS_KEY`	(empty)	AWS secret key
`S3_BUCKET`	`rag-documents`	S3 bucket name (when `STORAGE_BACKEND=s3`)
`MAX_UPLOAD_BYTES`	`31457280` (30 MiB)	Hard cap on upload size; enforced at HTTP and again at parse time
`ENCRYPTION_KEY`	(required)	Fernet base64 key for encrypting stored provider API keys
`STATIC_API_KEY`	(empty)	Single-user mode. One bearer that authorizes every tenant-scoped route AND counts as admin (wildcard tenant). Leave blank to require the dynamic per-tenant key flow.
`ADMIN_API_KEY`	(empty)	Separate admin token for the multi-tenant bootstrap flow (mint per-tenant keys via `POST /v1/tenants/{tid}/auth-keys`). Optional if `STATIC_API_KEY` is set.
`OPENSEARCH_USE_SSL`	`false`	Use TLS for OpenSearch (set `true` for AWS OpenSearch Service).
`OPENSEARCH_VERIFY_CERTS`	`true`	Verify TLS certs against the system trust store. Only consulted when `OPENSEARCH_USE_SSL=true`.
`QDRANT_API_KEY`	(empty)	API key for hosted Qdrant (e.g. Qdrant Cloud). Blank = unauthenticated, which is correct for local single-node.
`QDRANT_USE_HTTPS`	`false`	Use HTTPS for Qdrant. Set `true` for Qdrant Cloud / any hosted deployment.

Infrastructure Services

Started via docker-compose up -d:

Service	Image	Ports	Purpose
PostgreSQL	`postgres:16-alpine`	`5432`	Relational store — tenants, documents, chunks, ingestion_configs, retrieval_configs, API keys
Qdrant	`qdrant/qdrant:latest`	`6333`, `6334`	Vector similarity search
OpenSearch	`opensearchproject/opensearch:2.17.0`	`9200`, `9600`	BM25 keyword search
MinIO	`minio/minio:latest`	`9000` (API), `9001` (console)	Object storage for raw document files

MinIO console: http://localhost:9001 (admin / minioadmin)

API Reference

Authentication

Every /v1/tenants/{tid}/... route requires Authorization: Bearer <token>. Open routes (no auth): GET /health, GET /v1/providers.

There are two modes:

Single-user mode (simplest). Set STATIC_API_KEY in .env. That one token authorizes every tenant-scoped route (the request's tenant_id is not checked) AND counts as admin. Use this if you're the only user.

curl http://localhost:8000/v1/tenants/{any-uuid}/documents \
  -H "Authorization: Bearer $STATIC_API_KEY"

Multi-tenant mode. Set ADMIN_API_KEY (a separate admin token). Admin creates tenants and mints the first auth key for each; tenants use their own key thereafter. Cross-tenant tokens get 403.

# 1. Admin creates a tenant.
curl -X POST http://localhost:8000/v1/tenants \
  -H "Authorization: Bearer $ADMIN_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"Acme Corp"}'

# 2. Admin mints the FIRST auth key for that tenant.
curl -X POST http://localhost:8000/v1/tenants/{tid}/auth-keys \
  -H "Authorization: Bearer $ADMIN_API_KEY" -H "Content-Type: application/json" \
  -d '{"name":"production"}'
# Response: {"key": "wec_pk_<prefix>_<secret>", "metadata": {...}} — raw key shown ONCE.

# 3. Tenant uses that key for everything.
curl http://localhost:8000/v1/tenants/{tid}/documents \
  -H "Authorization: Bearer wec_pk_..."

Health

GET /health

Tenants

POST   /v1/tenants                                                  [admin] Create a tenant
GET    /v1/tenants/{tenant_id}                                      Get tenant by ID
POST   /v1/tenants/{tenant_id}/api-keys                             Store (or replace) a named provider API key

(See Pipeline Configs below for ingestion/retrieval config endpoints, and Auth keys for tenant bearer-token management.)

Create tenant (admin token required):

POST /v1/tenants
{ "name": "Acme Corp" }

Store API key:

POST /v1/tenants/{tenant_id}/api-keys
{
  "provider": "openai",   // openai | anthropic | google | voyage | zeroentropy | cohere | huggingface | aws_s3
  "api_key": "sk-...",    // for aws_s3, this is a JSON string: {"access_key_id":"...","secret_access_key":"...","region":"..."}
  "name": "api_key"       // optional — defaults to "api_key"; use a custom name to store multiple keys per provider
}

The key is AES-256 encrypted before storage. Submitting again with the same (provider, name) pair overwrites the previous key. Different names for the same provider are stored independently.

Signups (self-service tenant creation)

Open endpoint — no auth required, same posture as /health and /v1/providers. Creates a tenant scoped to an email and mints its first API key in one shot. The admin-gated POST /v1/tenants still works for ops use.

POST /v1/signups        Open — create tenant + mint first API key in one request

POST /v1/signups
{
  "email": "acme@example.com",
  "name": "Acme Corp"
}

201 Created →
{
  "tenant_id": "<uuid>",
  "email": "acme@example.com",
  "name": "Acme Corp",
  "api_key": "wec_pk_<prefix>_<secret>",   // raw key — shown ONCE
  "key_metadata": {...}
}

email is validated as RFC-format (Pydantic EmailStr) and is globally unique. Status codes:

Code	When
201	Success
409	Email already registered
422	Invalid email format, missing `name`, or extra fields

Auth keys

POST   /v1/tenants/{tenant_id}/auth-keys                            Mint a new tenant API key (raw key returned ONCE)
GET    /v1/tenants/{tenant_id}/auth-keys                            List this tenant's auth-key metadata (no raw values)
DELETE /v1/tenants/{tenant_id}/auth-keys/{key_id}                   Revoke (sets revoked_at)

The very first POST for a freshly-created tenant accepts either an admin token (bootstrap path) or any valid tenant key. Subsequent calls require a tenant key. Raw keys are formatted wec_pk_{8-char-prefix}_{32-char-secret}. The hash uses SHA-256 + hmac.compare_digest; raw keys are stored only as their hash — losing the raw value means re-minting.

S3 configs (tenant-scoped credentials)

Customer-bucket pulls authenticate with AWS creds stored per-tenant. Prefer this dedicated entity over the legacy api_keys.provider="aws_s3" flow.

POST   /v1/tenants/{tenant_id}/s3-configs                Create (encrypts secret_access_key with Fernet)
GET    /v1/tenants/{tenant_id}/s3-configs                List metadata (no secrets)
GET    /v1/tenants/{tenant_id}/s3-configs/{config_id}    Get single (no secret)
PUT    /v1/tenants/{tenant_id}/s3-configs/{config_id}    Partial update; if `secret_access_key` is present, re-encrypts
DELETE /v1/tenants/{tenant_id}/s3-configs/{config_id}    Delete

POST /v1/tenants/{tid}/s3-configs
{
  "name": "acme-prod",
  "access_key_id": "AKIA...",
  "secret_access_key": "...",
  "region": "us-east-1",
  "default_bucket": "acme-rag",     // optional
  "default_prefix": "docs/"         // optional
}

secret_access_key is never returned by any GET — only access_key_id, region, and the optional default-bucket/prefix metadata. Multiple named credential sets per tenant are supported (e.g. acme-prod, acme-staging). The new POST /run-from-s3 endpoint references these by name.

Backwards compat: the legacy POST /v1/tenants/{tid}/api-keys with provider="aws_s3" still works; the /documents/from-s3 endpoint resolves credentials by trying s3_configs first and falling back to api_keys.

Pinning credentials to an ingestion-config

You can save the credential default directly on the ingestion-config so /run-from-s3 doesn't need s3_config_name in every call. Add a source block to the config:

POST /v1/tenants/{tid}/ingestion-configs
{
  "name": "wec-dev-pipeline",
  "config": {
    "chunking": {"strategy": "basic", "token_size": 512, "token_overlap": 50},
    "embedding": {"provider": "zeroentropy", "model": "zembed-1", "dimensions": 2560},
    "source": {
      "type": "aws_s3",
      "s3_config_id": "<s3-config-uuid>"
    }
  }
}

source is a discriminated union with two variants today:

{"type": "upload"} — explicit marker that this pipeline only accepts multipart uploads via /run. (Same effect as omitting source entirely.)
{"type": "aws_s3", "s3_config_id": "<uuid>", "store_files": true} — credential default for /run-from-s3. The s3_config_id must belong to the same tenant; that's validated at POST/PUT time (422 with a clear message if not). store_files controls whether files are copied into our backing storage (managed mode, default) or kept in place as references (linked mode) — see Storage modes below.

When /run-from-s3 is called against a config with source.type=aws_s3:

Body sends {uri, kind} only → creds come from the saved source.
Body also sends s3_config_name → body wins (per-call override; useful for ad-hoc runs without editing the config).
Body has neither s3_config_name nor a saved source → 400 with the "set one or the other" message.
Saved s3_config_id was deleted → 404 telling the caller to update the ingestion-config.

URI stays per-call regardless — the same config feeds many runs against different paths over time.

Default-bucket mode (shorter body)

When the s3-config row has default_bucket set, /run-from-s3 accepts a leaner body that omits the s3://bucket/ part:

POST /run-from-s3
{
  "s3_config_name": "wec-rag-dev",   // s3-config with default_bucket="wec-rag-dev"
  "key": "docs/jan-2026/",           // path inside that bucket
  "kind": "prefix"
}

The server assembles s3://wec-rag-dev/docs/jan-2026/ from the config + body. Rules:

key and uri are mutually exclusive (422 if both are sent).
key requires s3_config_name + kind.
The s3-config must have default_bucket set (422 with a clear message otherwise).
Default-bucket mode only works with new-style s3-configs (not legacy api_keys.aws_s3).

Source presets (saved S3 sources for the dropdown)

Reusable named S3 source bundles per tenant. The UI lists them as a dropdown; POST /run-from-s3 resolves a preset_id to its saved spec at run time.

POST   /v1/tenants/{tenant_id}/source-presets                Create
GET    /v1/tenants/{tenant_id}/source-presets                List (sorted by last_used_at desc) — drives the UI dropdown
GET    /v1/tenants/{tenant_id}/source-presets/{preset_id}    Get single
PUT    /v1/tenants/{tenant_id}/source-presets/{preset_id}    Update name and/or spec
DELETE /v1/tenants/{tenant_id}/source-presets/{preset_id}    Delete

spec is a discriminated union — S3 sources only (file upload is direct multipart and doesn't need a preset):

// aws_s3 — saved s3 URI/prefix tied to an s3_config
POST /v1/tenants/{tid}/source-presets
{
  "name": "jan-2026-batch",
  "spec": {
    "type": "aws_s3",
    "s3_config_id": "<s3-config-uuid>",
    "uri": "s3://acme-rag/jan-2026/",
    "kind": "prefix"
  }
}

// presigned — saved batch of pre-signed URLs (no creds needed)
POST /v1/tenants/{tid}/source-presets
{
  "name": "q1-share",
  "spec": {
    "type": "presigned",
    "urls": ["https://...", "https://..."]
  }
}

s3_config_id is validated at write time — it must reference a real s3_configs row owned by the same tenant. Each successful /run-from-s3 against a preset bumps its last_used_at so the dropdown can sort by recency.

To run with a saved preset:

curl -X POST .../ingestion-configs/{ic_id}/run-from-s3 \
  -H "Content-Type: application/json" \
  -d '{"preset_id": "8881fe5f-..."}'

preset_id is mutually exclusive with the inline source fields (uri / kind / s3_config_name / presigned_urls) — sending both gets a 422 with a clear message.

Documents

# New unified entry points (recommended):
POST   /v1/tenants/{tenant_id}/ingestion-configs/{ic_id}/run              Multipart: upload file(s) + trigger ingestion
POST   /v1/tenants/{tenant_id}/ingestion-configs/{ic_id}/run-from-s3      JSON: pull from s3 URI / prefix / presigned URLs

# Legacy entry points (still supported, identical behavior):
POST   /v1/tenants/{tenant_id}/ingestion-configs/{ic_id}/documents          Multipart upload
POST   /v1/tenants/{tenant_id}/ingestion-configs/{ic_id}/documents/from-s3  S3 pull

# Browse / get / delete (tenant-scoped):
GET    /v1/tenants/{tenant_id}/documents                                    List all documents (across configs)
GET    /v1/tenants/{tenant_id}/documents/{document_id}                      Get + ingestion status
DELETE /v1/tenants/{tenant_id}/documents/{document_id}                      Delete document + chunks + all indexes

POST /run is the new entry point: pick a source per request (upload via multipart, or S3 via JSON), and the same downstream pipeline (size guard → format guard → storage → Document row → background ingestion) runs in both cases. The legacy /documents and /documents/from-s3 continue to work unchanged.

# /run with upload source (multipart)
curl -X POST .../ingestion-configs/{ic_id}/run \
  -F "files=@a.pdf" -F "files=@b.docx"

# /run-from-s3 — single object, using stored s3-config
curl -X POST .../ingestion-configs/{ic_id}/run-from-s3 \
  -H "Content-Type: application/json" \
  -d '{"s3_config_name": "acme-prod",
       "uri": "s3://acme-rag/docs/quarterly.pdf",
       "kind": "uri"}'

# /run-from-s3 — bulk pull every supported file under a prefix
curl -X POST .../ingestion-configs/{ic_id}/run-from-s3 \
  -H "Content-Type: application/json" \
  -d '{"s3_config_name": "acme-prod",
       "uri": "s3://acme-rag/docs/",
       "kind": "prefix"}'

# /run-from-s3 — presigned URLs (no creds needed in our system)
curl -X POST .../ingestion-configs/{ic_id}/run-from-s3 \
  -H "Content-Type: application/json" \
  -d '{"presigned_urls": ["https://...", "https://..."]}'

Both /run and /run-from-s3 return 202 with a list[DocumentResponse]. Each entry's metadata includes a source tag ("upload" / "s3" / "presigned") and, for S3-sourced docs, the original source_key. The response also carries linked: bool and linked_s3_config_id: UUID | None — see Storage modes below.

Storage modes

For S3 sources, files can either be copied into our backing storage (managed mode, default) or kept in place as references (linked mode). Toggle per-call via store_files in the request body, or save the default on the ingestion-config's source.store_files.

Mode	`store_files`	`file_key`	`linked_s3_config_id`	Re-fetch on parse	Delete touches customer bucket?
Managed (default)	`true`	managed path (`{tenant_id}/{uuid}/{filename}`)	NULL	From our storage	n/a (managed)
Linked	`false`	original `s3://bucket/key`	s3-config row id	From customer bucket using stored creds	No — only the local row + index entries are removed

# Linked mode: keep the file in the customer bucket, never copy.
curl -X POST .../ingestion-configs/{ic_id}/run-from-s3 \
  -H "Content-Type: application/json" \
  -d '{"s3_config_name": "wec-rag-dev",
       "uri": "s3://wec-rag-dev/docs/contract.pdf",
       "kind": "uri",
       "store_files": false}'

Rules:

store_files=false + presigned_urls is rejected with 422 — presigned URLs have no stable customer-side reference for re-fetch.
store_files=false requires creds from the new s3_configs table (not legacy api_keys.aws_s3).
If the linked s3-config is later deleted, ingestion fails with a "re-link or re-ingest with store_files=true" error.

The legacy endpoints (no behavior change):

Upload is nested under an ingestion config because every document is tied to the specific pipeline used to process it. Browse / get / delete remain at the tenant level so callers can list documents across all their configs.

Multipart uploads use a repeatable files field — send one or more files in a single request. Each file becomes its own Document row with its own background ingestion task (so files are processed in parallel). The 30 MB cap applies per file; if any file in the batch is over the limit the entire request fails with 413 before any storage writes happen. Response is a list[DocumentResponse] in upload order; each entry carries an optional warnings array (e.g. when contextual chunking is in use) plus ingestion_config_id and ingestion_config_name.

Supported file types: PDF, DOCX, TXT, MD, CSV. Other types return 415 Unsupported Media Type with a structured error listing the accepted MIME types and extensions.

Ingestion runs in the background; poll GET /documents/{id} and watch status:

Status	Meaning
`pending`	Uploaded, queued
`processing`	Parse/chunk/embed in progress
`indexed`	All indexes updated, ready to query
`failed`	Error in `error_msg` field

Retrieval

POST /v1/tenants/{tenant_id}/retrieval-configs/{retrieval_config_id}/query

The query is nested under a retrieval config — pick which one to use per request. The retrieval config links to one or more ingestion configs; all linked configs share the same embedding model + dimensions, so the query embedder produces vectors of the right shape for every corpus the retrieval covers. Searches are filtered to chunks belonging to the linked ingestion-config set.

{
  "query": "What are the key risks in the contract?",
  "retrieval_config_override": {            // optional — overrides specific fields for this request
    "top_k": 10,
    "reranker": { "enabled": true, "provider": "zeroentropy", "model": "zerank-2", "top_k": 10 }
  }
}

Default response (group_by="chunk"):

{
  "query": "What are the key risks in the contract?",
  "group_by": "chunk",
  "total": 5,
  "results": [
    {
      "chunk_id": "...",
      "document_id": "...",
      "content": "Clause 14 limits liability to...",
      "score": 0.0147,
      "metadata": { "page": 3, "source": "contract.pdf" }
    }
  ]
}

Grouped-by-document response (group_by="document"):

{
  "query": "What are the key risks in the contract?",
  "group_by": "document",
  "total": 3,
  "results": [
    {
      "document_id": "...",
      "document_name": "contract.pdf",
      "score": 0.0147,
      "chunks": [
        { "chunk_id": "...", "content": "...", "score": 0.0147, "metadata": {...} }
      ]
    }
  ]
}

The retrieval config controls these filters that shape the response:

document_ids — static list of document UUIDs to restrict the search to. Every id must belong to this tenant AND to one of the retrieval config's linked ingestion configs. Empty list rejected (422); null = no filter (default).
metadata_filters — dict[str, str|int|bool|list] filter on per-chunk metadata. Keys must be in the union of every linked ingestion config's declared chunking.metadata_fields (422 with the offending keys otherwise). Scalar values produce a term match; list values produce an any-of match. Chunks from a config that didn't declare the key just won't match — no error.
group_by — "chunk" (default, flat list of RetrievedChunk) or "document" (one entry per source doc with up to chunks_per_doc chunks nested).
chunks_per_doc — int ≥ 1, default 3. Only consulted when group_by = "document".

Providers Catalog

GET /v1/providers

Returns all supported embedding, reranker, and LLM providers enriched with dimensions, context windows, and pricing pulled live from LiteLLM's model_prices_and_context_window.json. ZeroEntropy is hardcoded since it's not in LiteLLM's catalog. Each entry carries a source: "litellm" | "hardcoded" flag.

{
  "embedding": [
    {
      "provider": "openai", "model": "text-embedding-3-small",
      "default_dimensions": 1536,
      "supported_dimensions_min": 256, "supported_dimensions_max": 1536,
      "max_input_tokens": 8191,
      "cost_per_1m_input_tokens": 0.02,
      "source": "litellm"
    },
    {
      "provider": "zeroentropy", "model": "zembed-1",
      "default_dimensions": 2560,
      "supported_dimensions": [40, 80, 160, 320, 640, 1280, 2560],
      "max_input_tokens": 8192,
      "source": "hardcoded"
    }
  ],
  "llm": [
    {
      "provider": "anthropic", "model": "claude-haiku-4-5",
      "context_window": 200000, "max_output_tokens": 8192,
      "cost_per_1m_input_tokens": 0.8, "cost_per_1m_output_tokens": 4.0,
      "supports_prompt_caching": true,
      "source": "litellm"
    }
  ],
  "reranker": [
    {
      "provider": "cohere", "model": "rerank-v3.5",
      "max_input_tokens": 4096,
      "cost_per_1m_input_tokens": 2.0,
      "source": "litellm"
    }
  ]
}

Pipeline Configs

The pipeline is split into two resources: an ingestion config (chunking + embedding, used at index time) and a retrieval config (query transformation + search blend + fusion + dedup + reranker + top_k, used at query time). A retrieval config links to one or more ingestion configs through a join table; all linked configs must share the same embedding (provider + model + dimensions), validated at POST/PUT time. A single query embedding then searches every linked corpus in one shot, filtered to chunks whose ingestion_config_id is in the linked set.

There is no "active config" concept — every upload and every query specifies the config to use via the URL. A tenant can hold any number of ingestion and retrieval configs in parallel. GET responses include the full config JSONB blob.

Ingestion configs (chunking + embedding):

GET    /v1/tenants/{tenant_id}/ingestion-configs                       List ingestion configs (each carries `cost_summary`)
GET    /v1/tenants/{tenant_id}/ingestion-configs/{config_id}           Get single (full `config`, `warnings`, `cost_summary`)
POST   /v1/tenants/{tenant_id}/ingestion-configs                       Create (config nullable)
PUT    /v1/tenants/{tenant_id}/ingestion-configs/{config_id}           Update name and/or config
POST   /v1/tenants/{tenant_id}/ingestion-configs/{config_id}/preview-cost   Predict ingestion cost from a sample file or text (no DB writes)

The cost_summary block on GET responses is computed live from the executions table:

"cost_summary": {
  "total_cost_usd": 0.0421,
  "executions_count": 12,
  "successful_count": 11,
  "failed_count": 1,
  "latest_execution_at": "2026-05-12T10:30:00Z",
  "avg_cost_per_document_usd": 0.0035
}

preview-cost accepts either a multipart sample_file or a multipart form field sample_text. Returns {estimated_chunks, estimated_tokens, embedding_cost_usd, llm_cost_usd, total_cost_usd, warnings}. Providers without published per-token pricing (ZeroEntropy, HuggingFace) return null for cost fields with a warning, rather than fabricate zero.

Retrieval configs (query-time settings; FK to an ingestion config):

GET    /v1/tenants/{tenant_id}/retrieval-configs                       List retrieval configs
GET    /v1/tenants/{tenant_id}/retrieval-configs/{config_id}           Get single (full `config`)
POST   /v1/tenants/{tenant_id}/retrieval-configs                       Create (body must include non-empty `ingestion_config_ids: list[UUID]`)
PUT    /v1/tenants/{tenant_id}/retrieval-configs/{config_id}           Update

Create an empty ingestion slot (no config yet):

POST /v1/tenants/{tenant_id}/ingestion-configs
{ "name": "My Pipeline" }

config is null by default. Fill it in later with PUT.

Create a fully-configured ingestion config:

POST /v1/tenants/{tenant_id}/ingestion-configs
{
  "name": "Recursive 512 + ZE zembed-1",
  "config": {
    "chunking": {
      "strategy": "recursive",
      "chunk_size": 512,
      "chunk_overlap": 64,
      "separators": ["\n\n", "\n", " ", ""],
      "context_window_provider": "anthropic",
      "context_window_llm": "claude-haiku-4-5",
      "metadata_fields": ["source", "page"],
      "semantic": { "breakpoint_percentile": 95, "min_chunk_size": 100, "max_chunk_size": 1500 }
    },
    "embedding": {
      "provider": "zeroentropy",
      "model": "zembed-1",
      "dimensions": 2560
    }
  }
}

Create a retrieval config linking one or more ingestion configs. Every config in ingestion_config_ids must share the same embedding model + dimensions — the API rejects mismatches with a 422. A single-element list is valid and behaves identically to a single-corpus retrieval; multi-element lists let a single query span N corpora ingested under different parsing / chunking strategies. PUT with a non-empty ingestion_config_ids REPLACES the link set wholesale; omit the field to leave links unchanged.

POST /v1/tenants/{tenant_id}/retrieval-configs
{
  "name": "Hybrid + ZE rerank",
  "ingestion_config_ids": ["<ingestion-config-uuid-a>", "<ingestion-config-uuid-b>"],
  "config": {
    "query_transformation": { "enabled": true, "strategy": "rewrite", "llm_model": "gpt-4o-mini" },
    "vector_search":        { "enabled": true, "top_k": 20 },
    "keyword_search":       { "enabled": true, "top_k": 20 },
    "fusion":               { "method": "rrf", "rrf_k": 60, "vector_weight": 0.7, "keyword_weight": 0.3 },
    "deduplication":        { "enabled": true, "similarity_threshold": 0.95 },
    "reranker":             { "enabled": true, "provider": "zeroentropy", "model": "zerank-2", "top_k": 10 },
    "top_k": 5,
    "document_ids":         ["doc-uuid-1", "doc-uuid-2"],
    "metadata_filters":     { "source": "helpdesk", "page": [3, 4, 5] },
    "group_by":             "document",
    "chunks_per_doc":       3
  }
}

Executions (observability)

Every ingestion and every retrieval run writes a row to the executions table — status, timings, metrics JSONB, and the FK to whichever config drove the run.

GET  /v1/tenants/{tenant_id}/executions                                    List runs (paginated)
GET  /v1/tenants/{tenant_id}/executions/{execution_id}                     Single run with full metrics
POST /v1/tenants/{tenant_id}/executions/{execution_id}/cancel              Cooperative cancel (202 — best-effort)

Query params on the list endpoint: pipeline_type (ingestion|retrieval), status (running|succeeded|failed), document_id, retrieval_config_id, limit (1–500, default 50), offset.

Every response carries the config names alongside the IDs — ingestion_config_name for ingestion rows, retrieval_config_name for retrieval rows. Both are null if the FK'd config was deleted (ON DELETE SET NULL).

Metrics JSONB shape (fields conditionally present):

Ingestion: chunks_count, embed_tokens, embed_cost_usd, llm_cost_usd (contextual only), total_cost_usd, parser_ms, chunker_ms, embed_ms, index_ms, parsed_pages, parsed_chars, chunking_strategy, embedding_provider, embedding_model, embedding_dimensions.
Retrieval: top_k, vector_hits, keyword_hits, fused_count, unique_documents (when grouped), query_transform_used, reranker_used.
Either: cancel_requested: true and cancel_requested_at: <iso> when a cancel was issued.

Costs are best-effort via LiteLLM get_model_info. Providers without published pricing leave *_cost_usd as null with a warning rather than fabricate zero.

Cancelling a running execution

POST /v1/tenants/{tid}/executions/{exec_id}/cancel   → 202 Accepted

Cancellation is cooperative — the endpoint flags metrics.cancel_requested=true and the running pipeline polls between stages (parse → chunk → between embed batches → before indexing). When it sees the flag, it raises ExecutionCancelled, which the top-level handler translates to status="failed" with error_msg="cancelled by request". No partial chunks are committed.

Code	When
202	Cancel requested; pipeline will observe at the next stage boundary
409	Execution is already `succeeded` or `failed`
404	Execution not found for this tenant

Retrieval cancellation is best-effort. Most retrieval calls complete in well under a second, so the cancel flag is usually flipped after the run has already returned.

Playground (A/B retrieval compare)

Fire the same query against two retrieval-configs in parallel and get both result sets back with per-leg latency + agreement metrics. Drives a side-by-side compare UI.

POST /v1/tenants/{tenant_id}/playground/compare

{
  "query": "What are the key risks in the contract?",
  "retrieval_config_id_a": "<uuid>",
  "retrieval_config_id_b": "<uuid>",
  "top_k_for_metrics": 10               // optional; default 10
}

Response shape:

{
  "query": "...",
  "a": {"retrieval_config_id":"...","latency_ms":245,"group_by":"chunk","results":[...],"total":5},
  "b": {"retrieval_config_id":"...","latency_ms":312,"group_by":"chunk","results":[...],"total":5},
  "comparison": {
    "top_k_for_metrics": 10,
    "chunk_overlap_jaccard": 0.40,
    "document_overlap_jaccard": 0.75,
    "score_concordance_spearman": 0.62
  },
  "warnings": []
}

The three comparison metrics:

Metric	What it measures
`chunk_overlap_jaccard`	Top-K chunk IDs that appear in BOTH configs' results / union of both.
`document_overlap_jaccard`	Same, but keyed by `document_id` — useful when the two configs use different chunking.
`score_concordance_spearman`	Rank correlation of scores on the intersection of A's and B's top-K. `null` when < 2 chunks overlap.

Both legs run via the existing run_retrieval, so each gets its own Execution row written by the tracker — full audit trail for free. Sending the same id for a and b is allowed but emits a warnings entry (metrics will trivially be 1.0 / 1.0 / 1.0).

Chat (RAG)

POST /v1/tenants/{tenant_id}/retrieval-configs/{retrieval_config_id}/chat

Runs the retrieval pipeline, calls an OpenAI mini model with the retrieved chunks as context, and returns the answer plus citations with freshly-generated presigned URLs to each source document. URLs aren't stored — every call regenerates them with the requested TTL.

POST /v1/tenants/{tid}/retrieval-configs/{rc_id}/chat
{
  "query": "What are the key risks in the contract?",
  "llm_provider": "openai",             // fixed for v1
  "llm_model": "gpt-4o-mini",           // any LiteLLM-supported openai chat model
  "top_k_for_context": 5,               // 1–20; chunks fed to the LLM
  "max_tokens": 2048,                   // 1–8192; hard cap on LLM output (default 2048)
  "presign_expires_in": 3600            // 60–86400 seconds
}

200 OK →
{
  "query": "...",
  "answer": "Per [1] clause 14, liability is capped at...",
  "citations": [
    {
      "document_id": "<uuid>",
      "filename": "contract.pdf",
      "download_url": "https://...?X-Amz-Signature=...",
      "expires_at": "2026-05-17T13:45:00Z",
      "score": 0.91,
      "chunk_ids": ["<chunk-uuid>", "..."]
    }
  ],
  "retrieval_config_id": "<uuid>",
  "llm_provider": "openai",
  "llm_model": "gpt-4o-mini",
  "latency_ms": 1840,
  "tokens": {"input": 1240, "output": 187, "total": 1427},
  "uncited": false
}

Citation behavior:

One citation per unique document_id cited (chunks from the same doc roll up to a single citation with all contributing chunk_ids).
[N] markers in answer index unique documents, never chunks — multiple chunks from the same source document share one [N].
Marker hygiene — the API sanitises the LLM output before returning: hallucinated or out-of-range [N] markers (e.g. [5] when only 2 unique docs are in context) are stripped from the answer; surviving valid markers are kept verbatim with no renumbering. citations is then filtered to only the documents whose marker actually appears in the cleaned answer.
No-citation fallback — when the model emits zero valid markers (it answered without grounding in any single source), uncited is set to true and citations returns every retrieved document so the UI can still display what was looked at. Distinguish "model cited these N sources" from "model didn't cite anything" via the uncited flag.
download_url is generated on demand: managed docs presign against our MinIO/S3 backing; linked docs presign against the customer bucket using the stored s3-config creds. Presigning runs in parallel via asyncio.gather so adding more citations doesn't multiply round-trips.
Citation order follows the order docs first appeared in the retrieval results.

Status codes: 200 success, 404 retrieval-config not found, 422 missing OpenAI API key, 502 LLM provider error.

The OpenAI API key is resolved via the existing POST /api-keys flow (with provider="openai").

Streaming chat — `POST .../chat/stream`

Same retrieval semantics as /chat, but the answer streams as Server-Sent Events. Time-to-first-token drops from "wait for full LLM response" (~7-8s on a 200-token answer) to retrieval-latency + ~600ms (~2.5-3s); total wall time is roughly the same as /chat. Use this from any UI that wants to paint tokens as they arrive.

Request body is identical to /chat (same ChatRequest schema — max_tokens, top_k_for_context, presign_expires_in, etc).

Wire format: Content-Type: text/event-stream, frames separated by blank lines.

event: meta
data: {"retrieval_latency_ms": 150, "context_chunks": 5,
       "doc_order": ["uuid", "..."]}

event: token
data: {"text": "The "}

event: token
data: {"text": "answer "}
...

event: final
data: {"answer": "<sanitised>", "citations": [...], "uncited": false,
       "tokens": {"input": 272, "output": 37, "total": 309},
       "latency_ms": 3093}

meta fires the moment retrieval finishes (so the UI can paint "found N sources" before the LLM speaks). doc_order lists the unique document IDs in [N] order.
token carries the next text delta. Stream is raw text — may include hallucinated [N] markers. UI choices: show streamed tokens live for snappiness OR buffer-and-display only the sanitised answer from final. Same marker hygiene + citation filtering as the JSON endpoint runs on the buffered text at end.
final carries {answer, citations, uncited, tokens, latency_ms} once at the end. Citation presigning runs in parallel with the token stream so final arrives ~immediately after the last token.
error may fire (with {detail: ...}) if the LLM call blows up mid-stream; the HTTP status is already 200 by that point so callers must handle it on the event channel.
Validation errors (404, 422 missing key, schema 422) are returned as a normal HTTP error response before the stream starts — they never appear as SSE events.

Configuration Reference

Chunking strategies

Strategy	Key	Description
Basic / Token	`basic` (alias `token`)	Fixed token-window split using tiktoken (cl100k_base). Fast, no dependencies.
Recursive	`recursive`	Splits on a hierarchy of separators (`\n\n` → `\n` → → `""`). Good general-purpose default.
Character	`character`	Splits by raw character count. Ignores token boundaries.
Pages	`pages`	One chunk per parsed page. Best for PDFs where page context matters. Won NVIDIA's chunking benchmark.
Semantic	`semantic`	Splits at topic boundaries via percentile-based embedding similarity drops (95th percentile by default). Embeds every sentence — slower but boundary-aware.
Contextual	`contextual`	Recursive split + per-chunk situating context generated by Claude (Anthropic) with prompt caching. Highest retrieval quality, costs ~$3 per 1k chunks with Haiku. Requires `anthropic` API key. Auto-splits oversized docs at paragraph boundary.
Hierarchical	`hierarchical`	Creates small child chunks for retrieval and stores the parent chunk in metadata for wider context.

Per-strategy parameters

Each strategy declares only its own parameters. Sending a field that belongs to another strategy returns 422 Unprocessable Entity with extra_forbidden — there's no shared chunk_size / chunk_overlap across all strategies. Match the param names below exactly.

`strategy`	Required / optional fields	Defaults
`basic`	`token_size` (int), `token_overlap` (int)	`512`, `64`
`token`	`token_size` (int), `token_overlap` (int) — alias of `basic`	`512`, `64`
`recursive`	`separators` (list[str])	`["\n\n", "\n", " ", ""]`
`character`	`chunk_size` (int), `chunk_overlap` (int)	`512`, `64`
`pages`	(no params)	—
`semantic`	`breakpoint_percentile` (int), `min_chunk_size` (int), `max_chunk_size` (int)	`95`, `100`, `1500`
`hierarchical`	`parent_size` (int), `chunk_size` (int), `chunk_overlap` (int)	`2048`, `512`, `64`
`contextual`	`character_size` (int), `character_overlap` (int), `llm_provider` (`openai` \| `anthropic`), `llm_model` (str), `prompt_template` (object)	`512`, `64`, `anthropic`, `claude-haiku-4-5`, default template

For contextual, the prompt_template object accepts:

Field	Type	Required	Notes
`template`	`"default"` \| `"custom"`	yes	Built-in or user-supplied.
`template_name`	string	no	Free-form label, metadata only.
`prompt`	string	required iff `template = "custom"`	Used in place of the built-in prompt. Should include the `{chunk_content}` placeholder — that's where the chunk text gets substituted.

Examples — minimal valid bodies:

// basic / token
{"strategy": "basic", "token_size": 512, "token_overlap": 64}

// recursive
{"strategy": "recursive", "separators": ["\n\n", "\n"]}

// character
{"strategy": "character", "chunk_size": 512, "chunk_overlap": 64}

// hierarchical
{"strategy": "hierarchical", "parent_size": 2048, "chunk_size": 512, "chunk_overlap": 64}

// pages
{"strategy": "pages"}

// semantic
{"strategy": "semantic", "breakpoint_percentile": 95, "min_chunk_size": 100, "max_chunk_size": 1500}

// contextual (default prompt)
{"strategy": "contextual", "character_size": 512, "character_overlap": 64,
 "llm_provider": "anthropic", "llm_model": "claude-haiku-4-5"}

// contextual (custom prompt)
{"strategy": "contextual", "character_size": 512, "character_overlap": 64,
 "llm_provider": "anthropic", "llm_model": "claude-haiku-4-5",
 "prompt_template": {"template": "custom", "template_name": "mine",
                     "prompt": "Situate <chunk>{chunk_content}</chunk> in the document."}}

Common pitfall. Sending chunk_size / chunk_overlap with strategy: "basic" (or recursive, or semantic) will 422 — those keys belong to character and hierarchical. Use token_size / token_overlap for basic, or change the strategy to character.

Embedding providers

Provider	Key	Models	Dimensions
OpenAI	`openai`	`text-embedding-3-small`, `text-embedding-3-large`	1536, 3072
Google	`google`	`gemini-embedding-001`	3072
Voyage AI	`voyage`	`voyage-3`, `voyage-3-large`, `voyage-4`, `voyage-4-lite`	512–2048
Zero Entropy	`zeroentropy`	`zembed-1`	40–2560 (Matryoshka)
HuggingFace	`huggingface`	Any TEI-compatible model	Varies

Note: QDRANT_VECTOR_SIZE sets the dimension of the default Qdrant collection (kept at the bare QDRANT_COLLECTION name for backwards compatibility). Switching to a different embedding dimension automatically routes upserts and queries to a {base}_{dim} collection that's created lazily — no manual collection management needed. The documents.embedding_dimensions column records each doc's collection so DELETE routes correctly.

Reranker providers

Provider	Key	Models
Cohere	`cohere`	`rerank-v3.5`, `rerank-v3`
Voyage AI	`voyage`	`rerank-2.5`, `rerank-2.5-lite`
Zero Entropy	`zeroentropy`	`zerank-2`, `zerank-2-nano`
HuggingFace	`huggingface`	Any self-hosted cross-encoder

Query transformation strategies

Strategy	LLM calls	Description
`none`	0	Pass original query directly to search
`rewrite`	1	Rewrite query for better retrieval specificity
`expand`	N (`num_llm_calls`)	Generate N diverse query variants, search all
`hyde`	1	Generate a hypothetical document, embed it as the query vector

Database Schema

Eight tables: the original five (tenants, documents, chunks, ingestion_configs, retrieval_configs, api_keys) plus platform_api_keys (shared cross-tenant provider defaults), executions (pipeline run log), and tenant_auth_keys (bearer tokens for the auth layer).

What changed in this release

#	Change	Migration	Reason
1	New table `executions` + ENUMs `pipeline_type`, `execution_status`	`a7b8c9d0e1f2_add_executions`	Every ingestion + retrieval run is now logged with timings + metrics (chunks, tokens, cost, *_ms phases). Drives the `cost_summary` roll-up on ingestion configs and the `GET /executions` endpoints.
2	New table `tenant_auth_keys`	`b8c9d0e1f2a3_add_tenant_auth_keys`	Bearer-token auth for every `/v1/tenants/{tid}/...` route. Separate from `api_keys` (which holds provider creds like OpenAI). Raw keys are SHA-256 hashed; the `key_prefix` column is the lookup handle.
3	`provider_type` enum gains `aws_s3`	`c9d0e1f2a3b4_add_aws_s3_provider`	Lets tenants store AWS S3 credentials under `api_keys` so the new `POST /documents/from-s3` endpoint can pull objects from a customer bucket.
4	New table `s3_configs`	`d0e1f2a3b4c5_add_s3_configs`	First-class tenant-scoped AWS S3 credentials with `default_bucket` / `default_prefix`. Replaces the legacy `api_keys.aws_s3` JSON blob for new clients (legacy still works).
5	New table `source_presets`	`e1f2a3b4c5d6_add_source_presets`	Saved named S3 source bundles (`s3_config_id` + `uri` + `kind` OR presigned URLs). Drives the UI's "pick a saved source" dropdown. `last_used_at` updates on each `/run-from-s3` call that resolves a preset.
6	`tenants.email` nullable column + unique constraint	`f2a3b4c5d6e7_add_tenants_email`	Powers self-service signups via `POST /v1/signups`. Existing admin-created tenants keep `email IS NULL`.
7	Hosted-DB compat: convert `api_keys.provider` enum from legacy `api_provider` to `provider_type`	`a3b4c5d6e7f8_fix_api_keys_provider_enum`	Some hosted Postgres instances bootstrapped with the wrong enum name; reads against `api_keys` 500'd with `operator does not exist: api_provider = provider_type`. Idempotent — no-op on databases already using `provider_type`.
8	`documents.linked_s3_config_id` nullable FK → `s3_configs.id` ON DELETE SET NULL	`0a3b4c5d6e7f_add_documents_linked_s3_config`	Tags documents that reference the customer bucket directly (linked mode — see Storage modes). NULL on managed copies (existing rows).

All migrations are purely additive — no rewrites of existing rows. The a3b4c5d6e7f8 enum-fix migration is the only one with conditional logic; it inspects the current column type and only transforms it if the legacy name is present.

Entity Relationship Diagram

┌────────────────────────────────────────────────────────────────────────────────────┐
│                                   PostgreSQL                                         │
│                                                                                      │
│  ┌──────────────┐                                                                    │
│  │   tenants    │                                                                    │
│  ├──────────────┤                                                                    │
│  │ id      UUID │◄───────┬────────┬────────┬────────┬────────┬────────┬────────┐    │
│  │ name VARCHAR │        │        │        │        │        │        │        │    │
│  │ created_at   │        │        │        │        │        │        │        │    │
│  └──────────────┘        │        │        │        │        │        │        │    │
│                          │ * (FK tenant_id on every child, ON DELETE CASCADE) │    │
│                          ▼        ▼        ▼        ▼        ▼        ▼        ▼    │
│  ┌──────────────┐  ┌──────────┐  ┌──────────────┐  ┌─────────┐  ┌──────────┐  ┌─────┐│
│  │  documents   │  │  chunks  │  │ingestion_cfgs│  │ api_keys│  │executions│  │auth ││
│  ├──────────────┤  ├──────────┤  ├──────────────┤  ├─────────┤  ├──────────┤  │keys ││
│  │ id           │  │ id       │  │ id           │  │ id      │  │ id       │  ├─────┤│
│  │ tenant_id    │  │ doc_id   ├──┤ tenant_id    │  │tenant_id│  │tenant_id │  │id   ││
│  │ ic_id ───────┼──┘          │  │ name         │  │provider │  │pipeline_ │  │tid  ││
│  │ filename     │  │tenant_id │  │ config JSONB │  │ name    │  │  type    │  │name ││
│  │ file_key     │  │ content  │  │ created_at   │  │encrypt_ │  │status    │  │pre- ││
│  │ mime_type    │  │ chunk_ix │  │ updated_at   │  │  key    │  │doc_id ──►│  │fix  ││
│  │ status       │  │ metadata │  └──────┬───────┘  │created  │  │ic_id  ──►│  │hash ││
│  │ error_msg    │  │created_at│         │ ▲         └─────────┘  │rc_id  ──►│  │exp/ ││
│  │ metadata     │  └──────────┘         │ │ FK ic_id              │started_at│  │rev/ ││
│  │ embed_dim    │       ▲               │ │ ON DELETE RESTRICT    │finished  │  │used ││
│  │ created_at   │       │ FK doc_id     │ │                       │duration  │  │_at  ││
│  │ updated_at   │       │ id = qdrant pt│ │                       │error_msg │  │crea-││
│  └─────┬────────┘                       │ │                       │metrics   │  │ted  ││
│        ▲                                 │ │                       │  JSONB   │  └─────┘│
│        │ FK doc_id          ┌────────────┴─┴──┐                    │created   │        │
│        │ ON DELETE CASCADE  │ retrieval_configs│                   │updated   │        │
│        │                    ├──────────────────┤                   └──────────┘        │
│        │                    │ id          UUID │      ▲ FK ic_id, rc_id                │
│        │                    │ tenant_id        │      │ ON DELETE SET NULL             │
│        │                    │ ic_id ───────────┼──────┘                                │
│        │                    │ name             │  (executions keep history when        │
│        │                    │ config     JSONB │   the config they reference is        │
│        │                    │ created_at       │   deleted)                            │
│        │                    │ updated_at       │                                       │
│        │                    └──────────────────┘                                       │
│        │                                                                                │
│        └──────────── FK doc_id (executions.document_id) ON DELETE CASCADE              │
│                                                                                          │
│   platform_api_keys: cross-tenant provider defaults (no tenant FK)                       │
│   - (provider, name) unique; consulted when get_api_key finds no tenant row.            │
└────────────────────────────────────────────────────────────────────────────────────────┘

documents.ingestion_config_id links each doc to the ingestion config that processed it (ON DELETE RESTRICT — can't delete a config that still has docs). executions.ingestion_config_id and executions.retrieval_config_id use ON DELETE SET NULL so deleting a config preserves its run history.

Tables

`tenants`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`name`	`VARCHAR(255)`	`NOT NULL`	Human-readable tenant label
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`

`documents`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`	Scopes document to a tenant
`ingestion_config_id`	`UUID`	nullable, `FK → ingestion_configs.id ON DELETE RESTRICT`	The ingestion config used to process this document. RESTRICT prevents deleting a config that still has docs attached. NULL on legacy rows.
`filename`	`VARCHAR(512)`	`NOT NULL`	Original upload filename
`file_key`	`VARCHAR(1024)`	`NOT NULL`	MinIO object path: `{tenant_id}/{uuid}/{filename}`
`mime_type`	`VARCHAR(128)`	`NOT NULL`	e.g. `application/pdf`
`status`	`ENUM`	`NOT NULL`, default `pending`	`pending` → `processing` → `indexed` \| `failed`
`error_msg`	`TEXT`	nullable	Populated on `failed`; contains exception message
`metadata`	`JSONB`	`NOT NULL`, default `{}`	User-supplied key-value pairs attached at upload time
`embedding_dimensions`	`INTEGER`	nullable	Set at ingest from the embedding config; used by DELETE to route to the correct per-dim Qdrant collection. NULL on legacy rows ingested before the per-dim refactor.
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`
`updated_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`, auto-updated

Indexes: ix_documents_tenant_id on (tenant_id)

`chunks`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`	Also used as the Qdrant point ID — the link between PG and the vector store
`document_id`	`UUID`	`NOT NULL`, `FK → documents.id ON DELETE CASCADE`	Parent document
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`	Denormalised for fast tenant-scoped queries
`content`	`TEXT`	`NOT NULL`	Raw chunk text as indexed
`chunk_index`	`INTEGER`	`NOT NULL`	0-based position within the source document
`metadata`	`JSONB`	`NOT NULL`, default `{}`	Page number, source filename, parent chunk ref (hierarchical), strategy tag, etc.
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`

Indexes: ix_chunks_tenant_id on (tenant_id), ix_chunks_document_id on (document_id)

`ingestion_configs`

Pipeline configs were split into two tables: ingestion_configs (chunking + embedding, used at index time) and retrieval_configs (search blend + reranker + top_k, used at query time).

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`
`name`	`VARCHAR(255)`	`NOT NULL`	Human label, e.g. `"Recursive 512 + ZE zembed-1"`
`config`	`JSONB`	nullable	`{chunking, embedding}` blob, `null` for empty slots
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`
`updated_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`, auto-updated

Indexes: ix_ingestion_configs_tenant_id on (tenant_id)

`retrieval_configs`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`
`name`	`VARCHAR(255)`	`NOT NULL`
`config`	`JSONB`	nullable	`{query_transformation, vector_search, keyword_search, fusion, deduplication, reranker, top_k, document_ids, metadata_filters, group_by, chunks_per_doc}` blob, `null` for empty slots
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`
`updated_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`, auto-updated

Indexes: ix_retrieval_configs_tenant_id on (tenant_id)

Ingestion-config link is stored in the retrieval_config_ingestion_configs join table (below), not as a column on this row — a retrieval config can search across one or more ingestion configs.

`retrieval_config_ingestion_configs`

Many-to-many join between retrieval configs and the ingestion configs they search across.

Column	Type	Constraints	Notes
`retrieval_config_id`	`UUID`	`PRIMARY KEY` (part 1), `FK → retrieval_configs.id ON DELETE CASCADE`	Deleting a retrieval config drops its links.
`ingestion_config_id`	`UUID`	`PRIMARY KEY` (part 2), `FK → ingestion_configs.id ON DELETE RESTRICT`	Cannot delete an ingestion config that any retrieval config still references.
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`	When this link was added; on PUT-replace the row's `created_at` becomes the time of the replace.

Indexes: composite PK on (retrieval_config_id, ingestion_config_id), plus ix_rcic_ingestion_config_id on (ingestion_config_id) for the reverse lookup.

Embedding compatibility (same provider + model + dimensions) across all rows for a given retrieval_config_id is enforced at the API layer on POST/PUT, NOT by a database constraint.

`api_keys`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`
`provider`	`ENUM`	`NOT NULL`	`openai` \| `anthropic` \| `google` \| `voyage` \| `zeroentropy` \| `cohere` \| `huggingface` \| `aws_s3`
`name`	`VARCHAR(255)`	`NOT NULL`, default `'api_key'`	Label for the key — multiple keys per provider are allowed when names differ
`encrypted_key`	`TEXT`	`NOT NULL`	AES-256 Fernet-encrypted API key. For `aws_s3`, this is a JSON string `{access_key_id, secret_access_key, region}`.
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`

Unique constraint: (tenant_id, provider, name) — upsert behaviour: submitting the same (provider, name) pair overwrites the existing key.

`executions`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`
`pipeline_type`	`ENUM`	`NOT NULL`	`ingestion` \| `retrieval`
`status`	`ENUM`	`NOT NULL`, default `running`	`running` → `succeeded` \| `failed`
`document_id`	`UUID`	nullable, `FK → documents.id ON DELETE CASCADE`	Populated for ingestion rows; `null` for retrieval.
`ingestion_config_id`	`UUID`	nullable, `FK → ingestion_configs.id ON DELETE SET NULL`	Preserves history when a config is deleted.
`retrieval_config_id`	`UUID`	nullable, `FK → retrieval_configs.id ON DELETE SET NULL`
`started_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`
`finished_at`	`TIMESTAMPTZ`	nullable	Populated when the row transitions out of `running`.
`duration_ms`	`INTEGER`	nullable	Wall-clock between `started_at` and `finished_at`.
`error_msg`	`TEXT`	nullable	Populated on `failed`.
`metrics`	`JSONB`	`NOT NULL`, default `{}`	Free-form: `chunks_count`, `embed_tokens`, `_cost_usd`, `_ms` timings, etc.
`created_at`, `updated_at`	`TIMESTAMPTZ`	`NOT NULL`

Indexes: (tenant_id, pipeline_type, started_at DESC), (document_id), (retrieval_config_id, started_at DESC).

`tenant_auth_keys`

Column	Type	Constraints	Notes
`id`	`UUID`	`PRIMARY KEY`, default `gen_random_uuid()`
`tenant_id`	`UUID`	`NOT NULL`, `FK → tenants.id ON DELETE CASCADE`
`name`	`VARCHAR(128)`	`NOT NULL`	Human label, e.g. `"production"`, `"ci-runner"`.
`key_prefix`	`VARCHAR(32)`	`NOT NULL`	Visible portion of the key for lookup, e.g. `wec_pk_a1b2c3d4`.
`key_hash`	`VARCHAR(256)`	`NOT NULL`	SHA-256 hash of the full raw key; compared via `hmac.compare_digest`.
`expires_at`	`TIMESTAMPTZ`	nullable	Optional expiry.
`revoked_at`	`TIMESTAMPTZ`	nullable	Set by `DELETE /auth-keys/{key_id}`.
`last_used_at`	`TIMESTAMPTZ`	nullable	Updated on successful auth.
`created_at`	`TIMESTAMPTZ`	`NOT NULL`, default `now()`

Unique constraint: (tenant_id, name). Indexes: (key_prefix), (tenant_id, revoked_at).

Raw key format: wec_pk_{8-char prefix}_{32-char secret}. The raw value is returned only once at creation time.

Relationships summary

Relationship	Type	Cascade
`tenants` → `documents`	one-to-many	delete tenant → delete all documents
`tenants` → `chunks`	one-to-many (denormalised)	delete tenant → delete all chunks
`tenants` → `ingestion_configs`	one-to-many	delete tenant → delete all ingestion configs
`tenants` → `retrieval_configs`	one-to-many	delete tenant → delete all retrieval configs
`ingestion_configs` ↔ `retrieval_configs`	many-to-many via `retrieval_config_ingestion_configs`	CASCADE on the retrieval side; RESTRICT on the ingestion side — cannot delete an ingestion config that any retrieval config still links to
`ingestion_configs` → `documents`	one-to-many	RESTRICT — cannot delete an ingestion config that still has docs
`tenants` → `api_keys`	one-to-many	delete tenant → delete all keys
`tenants` → `executions`	one-to-many	delete tenant → delete all rows
`tenants` → `tenant_auth_keys`	one-to-many	delete tenant → delete all auth keys
`documents` → `chunks`	one-to-many	delete document → delete all its chunks
`documents` → `executions`	one-to-many	delete document → delete all its execution rows
`ingestion_configs` → `executions`	one-to-many	SET NULL — delete config → executions keep history
`retrieval_configs` → `executions`	one-to-many	SET NULL

Deleting a document via DELETE /v1/tenants/{id}/documents/{doc_id} also removes the corresponding points from Qdrant and documents from the OpenSearch index before the PostgreSQL row is deleted.

Running Tests

# All tests
pytest

# With verbose output
pytest -v

# Specific file
pytest tests/test_rrf.py -v

# Only unit tests (fast, no external services needed)
pytest tests/test_chunking.py tests/test_embeddings.py tests/test_rrf.py -v

The test suite uses mocks for all external services (Qdrant, OpenSearch, OpenAI, etc.) — no running infrastructure needed for unit tests.

Project Structure

wec_rag_studio/
├── app/
│   ├── main.py                        # FastAPI app entry point
│   ├── core/
│   │   ├── config.py                  # pydantic-settings (env vars)
│   │   ├── database.py                # SQLAlchemy async engine + session
│   │   └── deps.py                    # FastAPI DI helpers
│   ├── models/                        # SQLAlchemy ORM models
│   ├── schemas/                       # Pydantic v2 request/response models
│   ├── api/v1/                        # FastAPI routers
│   │   ├── tenants.py
│   │   ├── signups.py                 # POST /v1/signups (open) — email + first key
│   │   ├── documents.py               # legacy /documents/from-s3 + browse/get/delete
│   │   ├── run.py                     # POST /run (multipart) + /run-from-s3 (json)
│   │   ├── retrieval.py
│   │   ├── chat.py                    # POST /chat — retrieval + LLM + citations; POST /chat/stream — SSE variant
│   │   ├── playground.py              # POST /playground/compare — A/B retrieval
│   │   ├── ingestion_configs.py       # incl. /preview-cost endpoint
│   │   ├── retrieval_configs.py
│   │   ├── s3_configs.py              # tenant-scoped AWS credentials
│   │   ├── source_presets.py          # saved S3 source bundles (dropdown)
│   │   ├── providers.py               # GET /v1/providers — catalog enriched live from LiteLLM
│   │   ├── platform.py                # admin-only cross-tenant provider keys
│   │   ├── auth_keys.py               # tenant bearer-token CRUD
│   │   └── executions.py              # GET + POST /cancel
│   ├── modules/                       # Modular, swappable components
│   │   ├── parser/                    # LiteParse wrapper
│   │   ├── chunking/                  # 7 chunking strategies (recursive, basic/token, character, pages, hierarchical, semantic, contextual)
│   │   ├── embeddings/                # 5 embedding providers (ZeroEntropy direct, others via LiteLLM)
│   │   ├── vdb/                       # Qdrant — per-dim collections, lazy create
│   │   ├── keyword_search/            # OpenSearch BM25
│   │   ├── reranker/                  # 4 reranker providers (ZeroEntropy direct, others via LiteLLM)
│   │   └── storage/                   # MinIO / S3 object storage
│   ├── pipelines/
│   │   ├── ingestion.py               # parse → chunk → embed → index
│   │   └── retrieval.py               # transform → search → fuse → rerank
│   ├── core/
│   │   └── security.py                # HTTPBearer + require_api_key / tenant_match / admin deps
│   └── services/
│       ├── rrf.py                     # Reciprocal Rank Fusion
│       ├── query_transformer.py       # HyDE / rewrite / expand
│       ├── config_service.py          # Load active configs from DB
│       ├── embedder_factory.py        # Build embedder from config
│       ├── reranker_factory.py        # Build reranker from config
│       ├── crypto.py                  # API key encrypt/decrypt
│       ├── auth_service.py            # Mint / verify / revoke tenant API keys
│       ├── cost_estimator.py          # LiteLLM-driven cost prediction + actuals
│       ├── execution_tracker.py       # Writes the executions table + cancel poll
│       ├── retrieval_compare.py       # Jaccard + Spearman metrics for /playground
│       ├── document_bytes.py          # Managed-vs-linked fetch dispatch
│       ├── presign.py                 # Generate citation download URLs
│       ├── s3_config_service.py       # s3-config lookups + cred decryption
│       ├── source_preset_service.py   # source-preset lookups + validation
│       └── s3_source.py               # Pull files from a customer S3 bucket
├── migrations/                        # Alembic async migrations
├── tests/                             # ~205 tests, mocked infra
├── docker-compose.yml                 # local-bundled: postgres + qdrant + opensearch + minio + app
├── docker-compose.hosted.yml          # hosted: app-only, talks to external infra
├── Dockerfile                         # Python 3.11 + Node 20 + LiteParse CLI
├── entrypoint.sh                      # `alembic upgrade head` then uvicorn
├── pyproject.toml
└── .env.example

Adding a New Embedding Provider

Create app/modules/embeddings/my_provider.py implementing BaseEmbedder
Add it to app/modules/embeddings/__init__.py
Add a branch in app/services/embedder_factory.py
Add the provider name to the ProviderType enum in app/models/api_key.py
Store the API key via POST /v1/tenants/{id}/api-keys

The pattern is identical for rerankers (BaseReranker → reranker_factory.py).

Adding a New Chunking Strategy

Create app/modules/chunking/my_strategy.py implementing BaseChunker
Add it to STRATEGY_MAP in app/modules/chunking/__init__.py
Reference it by key in a ChunkingConfig JSON blob

License

Private — WEC internal use.

Name		Name	Last commit message	Last commit date
app		app
infra		infra
migrations		migrations
scripts		scripts
tests		tests
.env.example		.env.example
.env.hosted.example		.env.hosted.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
README.md		README.md
alembic.ini		alembic.ini
docker-compose.hosted.yml		docker-compose.hosted.yml
docker-compose.yml		docker-compose.yml
eng.traineddata		eng.traineddata
entrypoint.sh		entrypoint.sh
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation