|
| 1 | +--- |
| 2 | +# SPDX-License-Identifier: CC0-1.0 |
| 3 | +layout: integration |
| 4 | +name: Privatemode |
| 5 | +description: Use open-weight models from Privatemode in Haystack, end-to-end encrypted and based on confidential computing |
| 6 | +authors: |
| 7 | + - name: Edgeless Systems |
| 8 | + socials: |
| 9 | + github: edgelesssys |
| 10 | + linkedin: https://www.linkedin.com/company/edgeless-systems |
| 11 | +pypi: https://pypi.org/project/haystack-ai/ |
| 12 | +repo: https://github.com/edgelesssys/privatemode-public |
| 13 | +type: Model Provider |
| 14 | +report_issue: https://github.com/edgelesssys/privatemode-public/issues |
| 15 | +logo: /logos/privatemode.svg |
| 16 | +version: Haystack 2.0 |
| 17 | +toc: true |
| 18 | +--- |
| 19 | + |
| 20 | +### **Table of Contents** |
| 21 | + |
| 22 | +- [Overview](#overview) |
| 23 | +- [Setup](#setup) |
| 24 | +- [Usage](#usage) |
| 25 | + |
| 26 | +## Overview |
| 27 | + |
| 28 | +[Privatemode](https://www.privatemode.ai/) is an inference API for open-weight models with end-to-end encryption. |
| 29 | +Privatemode processes prompts and responses only inside hardware-isolated Confidential Computing Environments. |
| 30 | +By design, neither Edgeless Systems as the operator nor the infrastructure provider can access them. |
| 31 | +Before any data leaves your environment, the client verifies the service through remote attestation. |
| 32 | +Edgeless Systems, a company based in Germany, operates Privatemode on infrastructure in the EU. |
| 33 | + |
| 34 | +The API is OpenAI-compatible, so Haystack's OpenAI components work with Privatemode for chat generation, embeddings, and agents with tool calling. |
| 35 | +Privatemode serves chat models such as GLM, and the Qwen3 embedding model. |
| 36 | +For the current lineup, see the [models page](https://docs.privatemode.ai/models/overview). |
| 37 | + |
| 38 | +## Setup |
| 39 | + |
| 40 | +The [Privatemode proxy](https://docs.privatemode.ai/api/proxy-configuration) runs next to your application. |
| 41 | +It verifies the service through remote attestation, encrypts your prompts, and decrypts the responses. |
| 42 | +Haystack talks to the proxy like it would to any OpenAI-compatible endpoint. |
| 43 | + |
| 44 | +1. Create an API key in the [Privatemode portal](https://portal.privatemode.ai). |
| 45 | +2. Start the proxy with your API key: |
| 46 | + |
| 47 | + ```bash |
| 48 | + docker run -p 127.0.0.1:8080:8080 ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --api-key <your-api-key> |
| 49 | + ``` |
| 50 | + |
| 51 | +3. Install Haystack: |
| 52 | + |
| 53 | + ```bash |
| 54 | + pip install haystack-ai |
| 55 | + ``` |
| 56 | + |
| 57 | +The proxy adds your API key to each request. |
| 58 | +Haystack's OpenAI components still require an `api_key` value, so the examples pass a placeholder. |
| 59 | +
|
| 60 | +## Usage |
| 61 | +
|
| 62 | +The examples use [GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) with `reasoning_effort` set to `high`. |
| 63 | +See [Choose a model and reasoning effort](#choose-a-model-and-reasoning-effort) to change either. |
| 64 | +
|
| 65 | +### Chat generation with `OpenAIChatGenerator` |
| 66 | +
|
| 67 | +```python |
| 68 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 69 | +from haystack.dataclasses import ChatMessage |
| 70 | +from haystack.utils import Secret |
| 71 | +
|
| 72 | +generator = OpenAIChatGenerator( |
| 73 | + api_key=Secret.from_token("placeholder"), # The proxy adds your API key. |
| 74 | + api_base_url="http://localhost:8080/v1", |
| 75 | + model="glm-flash-latest", |
| 76 | + generation_kwargs={"reasoning_effort": "high"}, |
| 77 | +) |
| 78 | +
|
| 79 | +messages = [ChatMessage.from_user("Explain confidential computing in one sentence.")] |
| 80 | +response = generator.run(messages=messages) |
| 81 | +print(response["replies"][0].text) |
| 82 | +``` |
| 83 | +
|
| 84 | +### Retrieval-augmented generation with embeddings |
| 85 | +
|
| 86 | +Embedding documents and queries through Privatemode keeps the documents encrypted as well, not only the prompts. |
| 87 | +
|
| 88 | +```python |
| 89 | +from haystack import Document, Pipeline |
| 90 | +from haystack.components.builders import ChatPromptBuilder |
| 91 | +from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder |
| 92 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 93 | +from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever |
| 94 | +from haystack.dataclasses import ChatMessage |
| 95 | +from haystack.document_stores.in_memory import InMemoryDocumentStore |
| 96 | +from haystack.utils import Secret |
| 97 | +
|
| 98 | +PRIVATEMODE_URL = "http://localhost:8080/v1" |
| 99 | +API_KEY = Secret.from_token("placeholder") # The proxy adds your API key. |
| 100 | +
|
| 101 | +document_store = InMemoryDocumentStore() |
| 102 | +document_embedder = OpenAIDocumentEmbedder( |
| 103 | + api_key=API_KEY, api_base_url=PRIVATEMODE_URL, model="qwen3-embedding-4b" |
| 104 | +) |
| 105 | +documents = [ |
| 106 | + Document(content="Employees may work remotely for up to three days per week."), |
| 107 | + Document(content="Travel expenses must be submitted within 30 days."), |
| 108 | + Document(content="The office is closed on public holidays in Berlin."), |
| 109 | +] |
| 110 | +document_store.write_documents(document_embedder.run(documents)["documents"]) |
| 111 | +``` |
| 112 | +
|
| 113 | +### Use with a Haystack Agent |
| 114 | +
|
| 115 | +```python |
| 116 | +from haystack.components.agents import Agent |
| 117 | +from haystack.components.generators.chat import OpenAIChatGenerator |
| 118 | +from haystack.dataclasses import ChatMessage |
| 119 | +from haystack.tools import tool |
| 120 | +from haystack.utils import Secret |
| 121 | +
|
| 122 | +
|
| 123 | +@tool |
| 124 | +def weather(city: str) -> str: |
| 125 | + """Return the weather for a city.""" |
| 126 | + return f"The weather in {city} is sunny and 22 degrees" |
| 127 | +
|
| 128 | +
|
| 129 | +agent = Agent( |
| 130 | + chat_generator=OpenAIChatGenerator( |
| 131 | + api_key=Secret.from_token("placeholder"), # The proxy adds your API key. |
| 132 | + api_base_url="http://localhost:8080/v1", |
| 133 | + model="glm-flash-latest", |
| 134 | + generation_kwargs={"reasoning_effort": "high"}, |
| 135 | + ), |
| 136 | + tools=[weather], |
| 137 | +) |
| 138 | +result = agent.run(messages=[ChatMessage.from_user("What's the weather in Munich?")]) |
| 139 | +print(result["last_message"].text) |
| 140 | +``` |
| 141 | +
|
| 142 | +### Choose a model and reasoning effort |
| 143 | +
|
| 144 | +Set `model="glm-latest"` to use the larger GLM-5.3 model instead of GLM-5.3-Flash. |
| 145 | +
|
| 146 | +GLM models accept `none`, `low`, `high`, and `max` for `reasoning_effort` and default to `max`. |
| 147 | +Lower values reduce latency, and `none` switches reasoning off. |
| 148 | +See [reasoning](https://docs.privatemode.ai/models/overview#reasoning) for details. |
| 149 | +
|
| 150 | +### Evaluate without the proxy |
| 151 | +
|
| 152 | +To try Privatemode without running the proxy, use the [proxyless API](https://docs.privatemode.ai/api/proxyless-api). |
| 153 | +Set `api_base_url="https://proxyless-api.privatemode.ai/v1"` and pass your API key with `Secret.from_env_var("PRIVATEMODE_API_KEY")`. |
| 154 | +
|
| 155 | +Use the proxy if you need cryptographically verifiable protection against access by Edgeless Systems. |
0 commit comments