Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/docs/extraction/api-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ On Windows PowerShell you can use `$env:NVIDIA_API_KEY = "nvapi-..."`.

The SDK and CLI do not load a `.env` file automatically. For a full list of related variables and how to source a `.env` file into the shell, refer to [Environment configuration variables](environment-config.md).

For `LiteLLMClient` and `LLMJudge` with a `nvidia_nim/...` model, pass `api_key="os.environ/NVIDIA_API_KEY"`. LiteLLM does not read `NVIDIA_API_KEY` for that provider.

Hosted object-detection NIMs (Page Elements, Table Structure, Graphic Elements) cap inline base64 image payloads at about 180,000 characters (roughly 180 KB). This key authorizes those hosted inference calls. For size limits and what to do when an image exceeds the cap, refer to [Hosted Page Elements NIM image size limits](troubleshoot.md#hosted-page-elements-nim-image-size-limits).

!!! note
Expand All @@ -39,6 +41,8 @@ api_key="os.environ/NVIDIA_API_KEY"

Use the provider's own variable name, for example `os.environ/OPENAI_API_KEY` for an OpenAI model. The reference is stored in graph JSON and resolved only when the operator is constructed or invoked on the worker.

For LiteLLM `nvidia_nim/...` models, use `os.environ/NVIDIA_API_KEY`. The worker resolves that reference and passes the value as `api_key`. If you omit `api_key`, LiteLLM looks up `NVIDIA_NIM_API_KEY` instead.

Literal keys remain available for non-persisted local execution, but attempting to serialize one raises an error. This prevents graph persistence from silently substituting an NVIDIA credential for another provider's key.

For how persisted graphs store credential references, refer to [Persisted graphs are trusted configuration](nemo-retriever-api-reference.md#persisted-graphs-are-trusted-configuration) in the Python API guide.
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/extraction/nemo-retriever-api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,8 @@ QAGenerationOperator(
)
```

For LiteLLM `nvidia_nim/...` models, including the default `LiteLLMClient` and `LLMJudge` models, use `os.environ/NVIDIA_API_KEY`. Refer to [Authentication and API keys](api-keys.md#credential-references-in-persisted-graphs).

Serializing a graph containing a literal API key fails with a contextual error instead of guessing which provider credential should be used on a worker.


Expand Down
6 changes: 6 additions & 0 deletions nemo_retriever/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,10 @@ uv pip install "nemo-retriever[llm]"
export NVIDIA_API_KEY=nvapi-...
```

The default Live RAG model uses LiteLLM's `nvidia_nim` provider. LiteLLM does not
read `NVIDIA_API_KEY` for that provider. Pass `api_key="os.environ/NVIDIA_API_KEY"`
so the same key is forwarded on each request.

Single-query live RAG. Point `vdb_kwargs["uri"]` at any table built above; the
embedding model in `embed_kwargs` must match the one used during ingestion so
query vectors land in the same embedding space as the stored chunks.
Expand All @@ -501,6 +505,7 @@ retriever = Retriever(
)
llm = LiteLLMClient.from_kwargs(
model="nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5",
api_key="os.environ/NVIDIA_API_KEY",
temperature=0.0,
max_tokens=512,
)
Expand All @@ -522,6 +527,7 @@ from nemo_retriever.models.llm import LLMJudge

judge = LLMJudge.from_kwargs(
model="nvidia_nim/nvidia/llama-3.3-nemotron-super-49b-v1.5",
api_key="os.environ/NVIDIA_API_KEY",
temperature=0.1,
max_tokens=4096,
)
Expand Down