Skip to content

docs: add OrcaRouter as an OpenAI-compatible gateway example #2509

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
XiaoHuo888-hue wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: add OrcaRouter as an OpenAI-compatible gateway example #2509

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/extraction/api-keys.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Persisted pipeline graphs never contain literal API keys. Configure a graph with
api_key="os.environ/NVIDIA_API_KEY"
```

Use the provider's own variable name, for example `os.environ/OPENAI_API_KEY` for an OpenAI model. The reference is stored in graph JSON and resolved only when the operator is constructed or invoked on the worker.
Use the provider's own variable name, for example `os.environ/OPENAI_API_KEY` for an OpenAI model. For an OpenAI-compatible gateway such as [OrcaRouter](https://www.orcarouter.ai), use `os.environ/ORCAROUTER_API_KEY`. The reference is stored in graph JSON and resolved only when the operator is constructed or invoked on the worker.

Literal keys remain available for non-persisted local execution, but attempting to serialize one raises an error. This prevents graph persistence from silently substituting an NVIDIA credential for another provider's key.

Expand Down
21 changes: 21 additions & 0 deletions docs/docs/extraction/nemo-retriever-api-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,27 @@ titles = GenericGenerationOperator(
).run(pd.DataFrame({"style": ["concise"], "document": ["Quarterly results"]}))
```

To use a different OpenAI-compatible gateway, set `model` to an `openai/`-prefixed
model ID and pass the gateway base URL as `api_base`. For example, route through
[OrcaRouter](https://www.orcarouter.ai) with an `ORCAROUTER_API_KEY` (keys start
with `sk-orca-`):

```python
orcarouter_params = TextGenerationParams.from_kwargs(
model="openai/orcarouter/auto",
api_base="https://api.orcarouter.ai/v1",
api_key="os.environ/ORCAROUTER_API_KEY",
temperature=0.0,
max_tokens=512,
)
summaries = SummarizationOperator(orcarouter_params).run(
pd.DataFrame({"text": ["A long document to summarize."]})
)
```

Any OrcaRouter model ID works in the `openai/`-prefixed form, for example
`openai/anthropic/claude-sonnet-4.6` or `openai/deepseek/deepseek-v4-pro`.

`SummarizationOperator` defaults to `text`, `summary`, `summary_latency_s`, `summary_model`, and `summary_error`. `GenericGenerationOperator` maps each named prompt placeholder to a physical DataFrame column and derives the metadata column names from `output_column`. Prompt contracts are validated when the operator is constructed, before any provider request runs.

To define another one-request/one-text-result task, subclass `TextGenerationTask`, declare `required_inputs`, and implement `build_request()`. Then construct it from a `TextGenerationOperator` subclass with explicit logical-input-to-DataFrame-column mappings. This abstraction is intentionally text-only; use a separate operator family for embeddings, captioning, tools, streaming, or structured domain results.
Expand Down
20 changes: 20 additions & 0 deletions docs/docs/extraction/workflow-agentic-retrieval.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,26 @@ agentic:
request_timeout_s: 1800
```

For example, to route the agentic LLM through an OpenAI-compatible gateway such
as [OrcaRouter](https://www.orcarouter.ai), set `llm_model` to a model on that
gateway, point `invoke_url` at its chat-completions endpoint, and export the
gateway key in the service process environment. OrcaRouter uses the
`ORCAROUTER_API_KEY` environment variable, and its keys start with `sk-orca-`.

```yaml
Comment on lines +71 to +73

@greptile-apps greptile-apps Bot Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Agentic gateway key is ignored

When users follow this example and export ORCAROUTER_API_KEY, the agentic service does not read that variable; it reuses the embedding credential resolved from NVIDIA_API_KEY or NGC_API_KEY. OrcaRouter consequently receives an empty or NVIDIA bearer token and rejects the chat-completion request.

Rule Used: When this PR changes user-facing code, configurati... (source)

Knowledge Base Used: Query Pipeline

Prompt To Fix With AI 
This is a comment left during a code review.
Path: docs/docs/extraction/workflow-agentic-retrieval.md
Line: 71-73

Comment:
**Agentic gateway key is ignored**

When users follow this example and export `ORCAROUTER_API_KEY`, the agentic service does not read that variable; it reuses the embedding credential resolved from `NVIDIA_API_KEY` or `NGC_API_KEY`. OrcaRouter consequently receives an empty or NVIDIA bearer token and rejects the chat-completion request.

**Rule Used:** When this PR changes user-facing code, configurati... ([source](.greptile))

**Knowledge Base Used:** [Query Pipeline](https://app.greptile.com/nvidia-public-github/-/custom-context/knowledge-base/nvidia/nemo-retriever/-/docs/query-pipeline.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

agentic:
enabled: true
llm_model: orcarouter/auto
invoke_url: https://api.orcarouter.ai/v1/chat/completions
reasoning_effort: high
backend_top_k: 20
react_max_steps: 50
request_timeout_s: 1800
```

You can use any OrcaRouter model ID as `llm_model`, for example
`anthropic/claude-sonnet-4.6` or `deepseek/deepseek-v4-pro`.

The VectorDB process owns the LanceDB volume and executes the agentic workflow.
Start it with matching `--agentic`, `--agentic-llm-model`, and
`--agentic-invoke-url` options. The LLM and embedding credentials are resolved
Expand Down