feat: add LiteLLM gateway support for chat + embeddings

[prodmanpd]

Summary

Adds optional support for routing notesGPT's LLM calls through a
LiteLLM proxy (or any
OpenAI-compatible gateway), so users can point the chat + embedding calls at
100+ providers (OpenAI, Azure, Anthropic, Bedrock, Gemini, ...) behind a single
URL. Together AI stays the default - existing deployments keep working with
only TOGETHER_API_KEY set. This is purely additive: no behavior changes unless
you set the new env vars.

Motivation

Today the provider is hardcoded in convex/together.ts
(baseURL: 'https://api.together.xyz/v1' + hardcoded model strings), so there's
no way to run notesGPT against a different LLM without editing source. Routing
through a LiteLLM proxy is the standard way to swap providers, add spend
limits/fallbacks, or use models you already have keys for - without touching app
code.

Changes

• convex/llm.ts (new) - central provider config read from env with Together AI
  defaults (LLM_BASE_URL, LLM_API_KEY -> falls back to TOGETHER_API_KEY,
  LLM_CHAT_MODEL, LLM_EMBEDDING_MODEL, LLM_INSTRUCTOR_MODE) + a
  createLLMClient() factory.
• convex/together.ts - uses the shared client + config instead of the hardcoded
  new OpenAI({ baseURL: '...together...' }) and hardcoded model literals. The
  extraction and both embedding call sites now honor the config.
• README.md - documents the optional vars and a copy-pasteable LiteLLM proxy
  example.

Transcription (convex/whisper.ts) intentionally stays on Together's Whisper
endpoint: it relies on Together's file-URL upload extension (handles >1GB /
30-min audio without chunking), which the generic OpenAI transcription contract
doesn't cover. This PR gateways the chat + embedding inference.

Cross-provider fix caught while testing

Instructor's mode: 'JSON_SCHEMA' (previously hardcoded) emits Together's
proprietary response_format.schema field, which other providers reject
(Azure returned Unknown parameter: 'response_format.schema'). So a naive base-
URL swap would 400 on OpenAI/Azure/Anthropic. The fix: LLM_INSTRUCTOR_MODE
(default JSON_SCHEMA for Together back-compat) - set it to TOOLS to drive
structured output through cross-provider tool calling. Verified below.

Tests

This repo has no test harness, so here's typecheck + formatter + a real
end-to-end run.

1. Typecheck - npx tsc --noEmit -> exit 0 (clean).

2. Formatting - npx prettier --check convex/llm.ts convex/together.ts README.md -> all files use Prettier code style.

3. Live E2E - configured client -> local LiteLLM proxy -> Azure (gpt via
Azure AI Foundry), structured extraction with LLM_INSTRUCTOR_MODE=TOOLS:

base URL   : http://localhost:4000
model      : chat

=== Structured extraction (tool call through LiteLLM -> Azure) ===
finish_reason: tool_calls
usage        : {"completion_tokens":102,"prompt_tokens":186,"total_tokens":288,...}
arguments    :
{
  "title": "Tomorrow's Tasks and Errands",
  "summary": "Reschedule Thursday's dentist appointment, complete the Q3 budget spreadsheet before the 2 p.m. standup, and pick up oat milk on the way home.",
  "actionItems": [
    "Call the dentist tomorrow to reschedule Thursday's appointment.",
    "Finish the Q3 budget spreadsheet before the 2 p.m. standup tomorrow.",
    "Buy oat milk on the way home tomorrow."
  ]
}

Proxy config used (Azure behind a chat alias):

model_list:
  - model_name: chat
    litellm_params:
      model: azure_ai/<deployment>
      api_base: https://<resource>.services.ai.azure.com
      api_key: os.environ/AZURE_API_KEY
litellm_settings:
  drop_params: true

This proves the full chain: createLLMClient() -> OpenAI SDK -> LiteLLM proxy ->
Azure -> structured JSON matching the existing NoteSchema. Embeddings run
through the same configured client (togetherai.embeddings.create) and the same
base URL; I didn't have an embedding deployment on the test resource, so that
path wasn't exercised live, but it's the identical client + config.

Risk / Compatibility

• Additive only. With no new env vars set, behavior is byte-for-byte the same
  (Together AI, same models, JSON_SCHEMA mode).
• No new dependencies - reuses the existing openai SDK.
• Transcription path untouched.

Example usage

1. Define a LiteLLM proxy config (litellm.config.yaml) that fronts whatever
providers you want behind one endpoint:

model_list:
  - model_name: chat # -> LLM_CHAT_MODEL
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: chat-claude # swap in with one env change
    litellm_params:
      model: anthropic/claude-3-5-sonnet-latest
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: embed # -> LLM_EMBEDDING_MODEL
    litellm_params:
      model: openai/text-embedding-3-small
      api_key: os.environ/OPENAI_API_KEY
litellm_settings:
  drop_params: true

2. Start the proxy (Python) - exposes an OpenAI-compatible API on :4000:

# pip install "litellm[proxy]"
# litellm --config litellm.config.yaml --port 4000
#
# Sanity-check it the same way notesGPT talks to it - plain OpenAI SDK:
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-litellm-anything")
resp = client.chat.completions.create(
    model="chat",
    messages=[{"role": "user", "content": "Say OK"}],
)
print(resp.choices[0].message.content)  # -> OK

3. Point notesGPT at it - set these Convex env vars (all optional; unset =
Together AI as before):

LLM_BASE_URL=http://localhost:4000
LLM_API_KEY=sk-litellm-...
LLM_CHAT_MODEL=chat            # or chat-claude, etc.
LLM_EMBEDDING_MODEL=embed
LLM_INSTRUCTOR_MODE=TOOLS      # for non-Together providers

That's it - convex/together.ts picks the client/models up automatically:

// convex/llm.ts (new) - single place all provider config lives
export function createLLMClient() {
  return new OpenAI({ apiKey: llmApiKey, baseURL: llmBaseURL });
}

// convex/together.ts - unchanged call shape, now provider-agnostic
const togetherai = createLLMClient();
const client = Instructor({ client: togetherai, mode: instructorMode });

await client.chat.completions.create({
  messages: [
    { role: 'system', content: 'Extract a title, summary, and action items...' },
    { role: 'user', content: transcript },
  ],
  model: chatModel, // Qwen on Together by default; "chat" via the proxy
  response_model: { schema: NoteSchema, name: 'SummarizeNotes' },
});

[vercel]

@prodmanpd is attempting to deploy a commit to the Together AI Team on Vercel.

A member of the Team first needs to authorize it.

[liuzhao1225]

I'm concerned about the supply-chain boundary of this LiteLLM integration. The upstream incident report confirms compromised PyPI releases 1.82.7/1.82.8 in March 2026; those releases were removed.

This reviewed change uses a proxy-compatible integration; it does not itself add a Python LiteLLM package installation. The deployment questions are who operates the proxy, which verified server version runs there, and whether credentials remain isolated from other providers. Please document those trust boundaries for users.

This is one of 49 observed LiteLLM integration PRs from the same account, whose author acknowledged the cross-project effort. The pattern prompted this review; it does not establish malicious intent or connect the author to the incident. The consolidated questions and corrections are in YouDub #130; this PR's review snapshot preserves the revision and scope.