Skip to content

Commit de610e2

Browse files
dasiknbilgeyucel
andauthored
Add Privatemode integration page (#737)
* Add Privatemode integration page * Update integrations/privatemode.md Co-authored-by: Bilge Yücel <bilge.yucel@deepset.ai> --------- Co-authored-by: Bilge Yücel <bilge.yucel@deepset.ai>
1 parent 5196238 commit de610e2

2 files changed

Lines changed: 170 additions & 0 deletions

File tree

‎integrations/privatemode.md‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
---
2+
# SPDX-License-Identifier: CC0-1.0
3+
layout: integration
4+
name: Privatemode
5+
description: Use open-weight models from Privatemode in Haystack, end-to-end encrypted and based on confidential computing
6+
authors:
7+
- name: Edgeless Systems
8+
socials:
9+
github: edgelesssys
10+
linkedin: https://www.linkedin.com/company/edgeless-systems
11+
pypi: https://pypi.org/project/haystack-ai/
12+
repo: https://github.com/edgelesssys/privatemode-public
13+
type: Model Provider
14+
report_issue: https://github.com/edgelesssys/privatemode-public/issues
15+
logo: /logos/privatemode.svg
16+
version: Haystack 2.0
17+
toc: true
18+
---
19+
20+
### **Table of Contents**
21+
22+
- [Overview](#overview)
23+
- [Setup](#setup)
24+
- [Usage](#usage)
25+
26+
## Overview
27+
28+
[Privatemode](https://www.privatemode.ai/) is an inference API for open-weight models with end-to-end encryption.
29+
Privatemode processes prompts and responses only inside hardware-isolated Confidential Computing Environments.
30+
By design, neither Edgeless Systems as the operator nor the infrastructure provider can access them.
31+
Before any data leaves your environment, the client verifies the service through remote attestation.
32+
Edgeless Systems, a company based in Germany, operates Privatemode on infrastructure in the EU.
33+
34+
The API is OpenAI-compatible, so Haystack's OpenAI components work with Privatemode for chat generation, embeddings, and agents with tool calling.
35+
Privatemode serves chat models such as GLM, and the Qwen3 embedding model.
36+
For the current lineup, see the [models page](https://docs.privatemode.ai/models/overview).
37+
38+
## Setup
39+
40+
The [Privatemode proxy](https://docs.privatemode.ai/api/proxy-configuration) runs next to your application.
41+
It verifies the service through remote attestation, encrypts your prompts, and decrypts the responses.
42+
Haystack talks to the proxy like it would to any OpenAI-compatible endpoint.
43+
44+
1. Create an API key in the [Privatemode portal](https://portal.privatemode.ai).
45+
2. Start the proxy with your API key:
46+
47+
```bash
48+
docker run -p 127.0.0.1:8080:8080 ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --api-key <your-api-key>
49+
```
50+
51+
3. Install Haystack:
52+
53+
```bash
54+
pip install haystack-ai
55+
```
56+
57+
The proxy adds your API key to each request.
58+
Haystack's OpenAI components still require an `api_key` value, so the examples pass a placeholder.
59+
60+
## Usage
61+
62+
The examples use [GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) with `reasoning_effort` set to `high`.
63+
See [Choose a model and reasoning effort](#choose-a-model-and-reasoning-effort) to change either.
64+
65+
### Chat generation with `OpenAIChatGenerator`
66+
67+
```python
68+
from haystack.components.generators.chat import OpenAIChatGenerator
69+
from haystack.dataclasses import ChatMessage
70+
from haystack.utils import Secret
71+
72+
generator = OpenAIChatGenerator(
73+
api_key=Secret.from_token("placeholder"), # The proxy adds your API key.
74+
api_base_url="http://localhost:8080/v1",
75+
model="glm-flash-latest",
76+
generation_kwargs={"reasoning_effort": "high"},
77+
)
78+
79+
messages = [ChatMessage.from_user("Explain confidential computing in one sentence.")]
80+
response = generator.run(messages=messages)
81+
print(response["replies"][0].text)
82+
```
83+
84+
### Retrieval-augmented generation with embeddings
85+
86+
Embedding documents and queries through Privatemode keeps the documents encrypted as well, not only the prompts.
87+
88+
```python
89+
from haystack import Document, Pipeline
90+
from haystack.components.builders import ChatPromptBuilder
91+
from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
92+
from haystack.components.generators.chat import OpenAIChatGenerator
93+
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
94+
from haystack.dataclasses import ChatMessage
95+
from haystack.document_stores.in_memory import InMemoryDocumentStore
96+
from haystack.utils import Secret
97+
98+
PRIVATEMODE_URL = "http://localhost:8080/v1"
99+
API_KEY = Secret.from_token("placeholder") # The proxy adds your API key.
100+
101+
document_store = InMemoryDocumentStore()
102+
document_embedder = OpenAIDocumentEmbedder(
103+
api_key=API_KEY, api_base_url=PRIVATEMODE_URL, model="qwen3-embedding-4b"
104+
)
105+
documents = [
106+
Document(content="Employees may work remotely for up to three days per week."),
107+
Document(content="Travel expenses must be submitted within 30 days."),
108+
Document(content="The office is closed on public holidays in Berlin."),
109+
]
110+
document_store.write_documents(document_embedder.run(documents)["documents"])
111+
```
112+
113+
### Use with a Haystack Agent
114+
115+
```python
116+
from haystack.components.agents import Agent
117+
from haystack.components.generators.chat import OpenAIChatGenerator
118+
from haystack.dataclasses import ChatMessage
119+
from haystack.tools import tool
120+
from haystack.utils import Secret
121+
122+
123+
@tool
124+
def weather(city: str) -> str:
125+
"""Return the weather for a city."""
126+
return f"The weather in {city} is sunny and 22 degrees"
127+
128+
129+
agent = Agent(
130+
chat_generator=OpenAIChatGenerator(
131+
api_key=Secret.from_token("placeholder"), # The proxy adds your API key.
132+
api_base_url="http://localhost:8080/v1",
133+
model="glm-flash-latest",
134+
generation_kwargs={"reasoning_effort": "high"},
135+
),
136+
tools=[weather],
137+
)
138+
result = agent.run(messages=[ChatMessage.from_user("What's the weather in Munich?")])
139+
print(result["last_message"].text)
140+
```
141+
142+
### Choose a model and reasoning effort
143+
144+
Set `model="glm-latest"` to use the larger GLM-5.3 model instead of GLM-5.3-Flash.
145+
146+
GLM models accept `none`, `low`, `high`, and `max` for `reasoning_effort` and default to `max`.
147+
Lower values reduce latency, and `none` switches reasoning off.
148+
See [reasoning](https://docs.privatemode.ai/models/overview#reasoning) for details.
149+
150+
### Evaluate without the proxy
151+
152+
To try Privatemode without running the proxy, use the [proxyless API](https://docs.privatemode.ai/api/proxyless-api).
153+
Set `api_base_url="https://proxyless-api.privatemode.ai/v1"` and pass your API key with `Secret.from_env_var("PRIVATEMODE_API_KEY")`.
154+
155+
Use the proxy if you need cryptographically verifiable protection against access by Edgeless Systems.

‎logos/privatemode.svg‎

Lines changed: 15 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)