From f74a8228dc31f144a951cd2e48b01446ed3ed7a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuela=20Z=C3=BCger?= Date: Fri, 8 May 2026 16:55:57 +0200 Subject: [PATCH 1/2] docs/website: Improve partial evaluation / data filtering documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * added a simple data filtering example to the filtering overview page to better showcase its purpose * explained the metadata annotation for unknowns and how it links to the database table and field names * added a tutorial page to provide a quick demo Fixes: #8316 Signed-off-by: Manuela Züger --- docs/docs/filtering/fragment.md | 30 ++- docs/docs/filtering/index.md | 67 ++++++ docs/docs/filtering/tutorial-sql-filtering.md | 196 ++++++++++++++++++ 3 files changed, 290 insertions(+), 3 deletions(-) create mode 100644 docs/docs/filtering/tutorial-sql-filtering.md diff --git a/docs/docs/filtering/fragment.md b/docs/docs/filtering/fragment.md index f88e8f8734f..d5eae127996 100644 --- a/docs/docs/filtering/fragment.md +++ b/docs/docs/filtering/fragment.md @@ -17,7 +17,7 @@ Not every construct is supported for every target. For a step-by-step walkthrough of evaluating a Rego policy _partially_, see [Evaluating a data filter policy](./partial-evaluation). ::: -## What is Partial Evaluation? +## What is Partial Evaluation? {#what-is-partial-evaluation} The translation of data policies into queries (like SQL WHERE clauses) is driven by _partial evaluation (PE)_ of a Rego query. @@ -31,7 +31,7 @@ When only _known_ values are used, **you can use all of Rego.** ## Example Preamble -In our running example, we'll assume a table `fruits` with columns `name`, `colour`, and `price`. These **unknown values** are represented with `input..` e.g. `input.fruits.name` +In our running example, we'll assume a table `fruits` with columns `name`, `colour`, and `price`. ```mermaid erDiagram @@ -42,7 +42,29 @@ erDiagram } ``` -Our data filters also depend on user information. These **known values** are represented with `input.user` +## Context data for Partial Evaluation +### Unknowns: database rows + +Database rows are **unknown** at policy evaluation time — OPA does not have access to the database. They are represented in Rego using the convention `input.
.`, e.g. `input.fruits.name` refers to the `name` column of the `fruits` table. + +The **METADATA annotation** on the policy package declares which `input` paths are unknown. OPA uses this to know which parts of the policy to leave as conditions rather than evaluate: + +```rego title="policy.rego" +package filters + +# METADATA +# scope: document +# compile: +# unknowns: [input.fruits] + +include if input.fruits.name == "banana" +``` + +With `input.fruits` declared as unknown, OPA will not try to resolve `input.fruits.name` during partial evaluation — instead it becomes a column reference in the output SQL. + +### Known values: request context + +Our data filters also depend on user information. These **known values** are sent to OPA as input at query time and will be substituted during partial evaluation: ```json { @@ -53,6 +75,8 @@ Our data filters also depend on user information. These **known values** are rep } ``` +They are referenced in the policy as `input.user`, e.g. `input.user.budget`. Because they are not listed in `unknowns`, OPA resolves them to their concrete values during partial evaluation. + ## Simple comparisons The fragment supports simple comparisons, such as `==`, `!=`, `<`, `>`, `<=`, `>=`, between _unknown_ and _known_ values. diff --git a/docs/docs/filtering/index.md b/docs/docs/filtering/index.md index edad39bc02c..a355d2c9e7d 100644 --- a/docs/docs/filtering/index.md +++ b/docs/docs/filtering/index.md @@ -34,3 +34,70 @@ sequenceDiagram Database-->>Application: Filtered employees Application-->>User: Filtered employees ``` + +## A quick example + +Consider an `employees` database table with salary information. The question is: **whose salaries can a Director see?** + +The rule is: Directors may see the salaries of employees in their own department. When Alice (Engineering Director) lists employees, the highlighted rows are what she should see: + +
+ + + + + + + + + + +
namedepartmentrolesalary
Aliceengineeringdirector130000
Bobengineeringengineer90000
Carolengineeringengineer85000
Davemarketingdirector120000
Evemarketingmanager95000
+ +OPA can be used to derive the needed SQL filter at run time, leveraging OPA's [partial evaluation](./filtering/partial-evaluation) feature. + +**1. Input passed to OPA** + +Alice is a _Director_ of the _Engineering_ department. The application sends her user context to OPA: + +```json title="input.json" +{ + "user": { + "name": "Alice", + "role": "director", + "department": "engineering" + } +} +``` + +**2. OPA evaluates the policy** + +```rego title="policy.rego" +package authz + +# METADATA +# scope: document +# compile: +# unknowns: [input.employees] + +include if { + input.user.role == "director" # known: true for Alice, consumed + input.employees.department == input.user.department # ¹unknown == ²known → SQL condition +} +``` + +¹ The value of `input.employees.department` is _unknown_ during partial policy evaluation — it refers to a table column in the database. + +² The value of `input.user.department` is known during partial policy evaluation — it resolves to the value `"engineering"` from the `input` document. + +**3. OPA returns a SQL filter** + +```sql title="SQL filter for Alice" +WHERE employees.department = 'engineering' +``` + +**4. Application Runs Query** + +The application can then query the database using this filter and process or display the returned data. + +For a hands-on walkthrough, see the [SQL Data Filtering Tutorial](./filtering/tutorial-sql-filtering). diff --git a/docs/docs/filtering/tutorial-sql-filtering.md b/docs/docs/filtering/tutorial-sql-filtering.md new file mode 100644 index 00000000000..8a47632b3a2 --- /dev/null +++ b/docs/docs/filtering/tutorial-sql-filtering.md @@ -0,0 +1,196 @@ +--- +title: "Tutorial: SQL Data Filtering" +sidebar_position: 6 +--- + +This tutorial demonstrates end-to-end data filtering with OPA around a concrete question: **whose salaries can a Director see?** + +You will write an authorization policy, use OPA's partial evaluation to derive a SQL `WHERE` clause, and apply that filter to a real database query. + +## Prerequisites + +- [OPA installed](../#1-download-opa) +- [sqlite3](https://sqlite.org/index.html) (pre-installed on macOS and most Linux distributions) +- `curl` and `jq` + +## Steps + +### 1. Create and populate the database + +We'll work with the following dataset: + +| name | department | role | salary | +|-------|-------------|-----------|--------| +| Alice | engineering | director | 130000 | +| Bob | engineering | engineer | 90000 | +| Carol | engineering | engineer | 85000 | +| Dave | marketing | director | 120000 | +| Eve | marketing | manager | 95000 | + +Save the following SQL to a file named `employees.sql`: + +```sql title="employees.sql" +CREATE TABLE employees (name TEXT, department TEXT, role TEXT, salary INTEGER); +INSERT INTO employees VALUES ('Alice', 'engineering', 'director', 130000); +INSERT INTO employees VALUES ('Bob', 'engineering', 'engineer', 90000); +INSERT INTO employees VALUES ('Carol', 'engineering', 'engineer', 85000); +INSERT INTO employees VALUES ('Dave', 'marketing', 'director', 120000); +INSERT INTO employees VALUES ('Eve', 'marketing', 'manager', 95000); +``` + +Then create the database by loading that file: + +```shell +sqlite3 company.db < employees.sql +``` + +### 2. Write the policy + +The rule is: Directors may see the salaries of employees in their own department. + +`input.employees` is declared as *unknown* — it represents database rows that OPA has not seen yet. `input.user` is *known* at query time and its values will be substituted during partial evaluation. + +Save the following Rego code to a file named `policy.rego`: + +```rego title="policy.rego" +package authz + +# METADATA +# scope: document +# compile: +# unknowns: [input.employees] + +include if { + input.user.role == "director" + input.employees.department == input.user.department +} +``` + +### 3. Start OPA + +```shell +opa run --server policy.rego +``` + +OPA is now listening on `http://localhost:8181`. + +### 4. Ask OPA for a SQL filter + +In another terminal, call the compile endpoint with the logged-in user as input. Alice is a Director in Engineering: + +```shell +curl -s -X POST http://localhost:8181/v1/compile/authz/include \ + -H "Content-Type: application/json" \ + -H "Accept: application/vnd.opa.sql.sqlite+json" \ + -d '{"input": {"user": {"name": "alice", "role": "director", "department": "engineering"}}}' +``` + +OPA partially evaluates the policy: + +- `input.user.role == "director"` — both sides are known; the condition is true, so it is consumed. +- `input.employees.department == input.user.department` — the left hand side is unknown; the known right hand side (`"engineering"`) is substituted, yielding the SQL condition. + +The response: + +```json +{ + "result": { + "query": "WHERE employees.department = 'engineering'" + } +} +``` + +### 5. Query the database + +Extract the filter and use it in a SQL query: + +```shell +FILTER=$(curl -s -X POST http://localhost:8181/v1/compile/authz/include \ + -H "Content-Type: application/json" \ + -H "Accept: application/vnd.opa.sql.sqlite+json" \ + -d '{"input": {"user": {"name": "alice", "role": "director", "department": "engineering"}}}' \ + | jq -r '.result.query') + +sqlite3 company.db "SELECT name, salary FROM employees $FILTER;" +``` + +Output — Alice sees all Engineering salaries: + +| name | salary | +|-------|--------| +| Alice | 130000 | +| Bob | 90000 | +| Carol | 85000 | + +Dave is a Director in Marketing, so he gets a different filter from the same policy: + +```shell +FILTER=$(curl -s -X POST http://localhost:8181/v1/compile/authz/include \ + -H "Content-Type: application/json" \ + -H "Accept: application/vnd.opa.sql.sqlite+json" \ + -d '{"input": {"user": {"name": "dave", "role": "director", "department": "marketing"}}}' \ + | jq -r '.result.query') + +sqlite3 company.db "SELECT name, salary FROM employees $FILTER;" +``` + +Output — Dave sees all Marketing salaries: + +| name | salary | +|------|--------| +| Dave | 120000 | +| Eve | 95000 | + +### 6. Non-Directors are denied + +Bob is an Engineer, not a Director. The `input.user.role == "director"` condition is known and false, so no rule body can ever be satisfied — the policy unconditionally denies: + +```shell +curl -s -X POST http://localhost:8181/v1/compile/authz/include \ + -H "Content-Type: application/json" \ + -H "Accept: application/vnd.opa.sql.sqlite+json" \ + -d '{"input": {"user": {"name": "bob", "role": "engineer", "department": "engineering"}}}' +``` + +Response — the `query` key is absent: + +```json +{} +``` + +An absent `query` means unconditional deny. The application should return zero rows without issuing a database query. + +:::warning Ensure safe defaults +OPA returns the filter — it does not enforce it. The application is responsible to use it as intended. + +In this example, if the user is not a Director, no rule body can be satisfied and OPA returns an unconditional deny — represented as a missing `query` key in the result — meaning the application should safely return zero rows. +::: + +## What partial evaluation did + +OPA evaluated the policy with `input.user` fully known. The expressions that involved only known values (`input.user.role == "director"`) were fully evaluated and consumed — they do not appear in the output. Only expressions involving the unknown `input.employees` survived as residual conditions, which OPA then translated into SQL. + +The application never needs to know _how_ the policy decides which salaries are visible. It sends user context and receives a SQL filter (or a deny) to act on. + +## Handling unconditional results + + +| OPA response | Meaning | Application action | +|----------------------------|---------------------|--------------------------------| +| `{ "query": "WHERE ..." }` | Conditional allow | Append filter to SQL query | +| `{ "query": "" }` | Unconditional allow | Run query with no `WHERE` | +| `{}` | Unconditional deny | Return zero rows, skip query | + +## Clean up + +Stop the OPA server with `Ctrl+C` in the terminal where it is running, then remove the files created during this tutorial: + +```shell +rm employees.sql policy.rego company.db +``` + +## Next steps + +- [Evaluating a Data Filter Policy](./partial-evaluation) — a step-by-step walkthrough of partial evaluation +- [Writing valid Data Filtering Policies](./fragment) — which Rego constructs are supported as filter conditions +- [Language SDKs](/ecosystem#languages) — in a production setup, using a language SDK is recommended over raw `curl` calls. The ecosystem page lists SDKs for Go, Java, Python, JavaScript, and more, all of which provide typed clients for the compile API used in this tutorial. From a17069fe2093a9856b8e9c2f49e97c04b68bd518 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Manuela=20Z=C3=BCger?= Date: Fri, 8 May 2026 17:07:16 +0200 Subject: [PATCH 2/2] docs/website: fix formatting on partial eval documentation changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit references: #8316 Signed-off-by: Manuela Züger --- docs/docs/filtering/fragment.md | 1 + docs/docs/filtering/index.md | 8 ++-- docs/docs/filtering/tutorial-sql-filtering.md | 37 +++++++++---------- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/docs/filtering/fragment.md b/docs/docs/filtering/fragment.md index d5eae127996..1b07a3bbd20 100644 --- a/docs/docs/filtering/fragment.md +++ b/docs/docs/filtering/fragment.md @@ -43,6 +43,7 @@ erDiagram ``` ## Context data for Partial Evaluation + ### Unknowns: database rows Database rows are **unknown** at policy evaluation time — OPA does not have access to the database. They are represented in Rego using the convention `input..`, e.g. `input.fruits.name` refers to the `name` column of the `fruits` table. diff --git a/docs/docs/filtering/index.md b/docs/docs/filtering/index.md index a355d2c9e7d..0beab4a1896 100644 --- a/docs/docs/filtering/index.md +++ b/docs/docs/filtering/index.md @@ -62,10 +62,10 @@ Alice is a _Director_ of the _Engineering_ department. The application sends her ```json title="input.json" { - "user": { - "name": "Alice", - "role": "director", - "department": "engineering" + "user": { + "name": "Alice", + "role": "director", + "department": "engineering" } } ``` diff --git a/docs/docs/filtering/tutorial-sql-filtering.md b/docs/docs/filtering/tutorial-sql-filtering.md index 8a47632b3a2..f19fcb282f5 100644 --- a/docs/docs/filtering/tutorial-sql-filtering.md +++ b/docs/docs/filtering/tutorial-sql-filtering.md @@ -19,13 +19,13 @@ You will write an authorization policy, use OPA's partial evaluation to derive a We'll work with the following dataset: -| name | department | role | salary | -|-------|-------------|-----------|--------| -| Alice | engineering | director | 130000 | -| Bob | engineering | engineer | 90000 | -| Carol | engineering | engineer | 85000 | -| Dave | marketing | director | 120000 | -| Eve | marketing | manager | 95000 | +| name | department | role | salary | +| ----- | ----------- | -------- | ------ | +| Alice | engineering | director | 130000 | +| Bob | engineering | engineer | 90000 | +| Carol | engineering | engineer | 85000 | +| Dave | marketing | director | 120000 | +| Eve | marketing | manager | 95000 | Save the following SQL to a file named `employees.sql`: @@ -48,7 +48,7 @@ sqlite3 company.db < employees.sql The rule is: Directors may see the salaries of employees in their own department. -`input.employees` is declared as *unknown* — it represents database rows that OPA has not seen yet. `input.user` is *known* at query time and its values will be substituted during partial evaluation. +`input.employees` is declared as _unknown_ — it represents database rows that OPA has not seen yet. `input.user` is _known_ at query time and its values will be substituted during partial evaluation. Save the following Rego code to a file named `policy.rego`: @@ -117,10 +117,10 @@ sqlite3 company.db "SELECT name, salary FROM employees $FILTER;" Output — Alice sees all Engineering salaries: | name | salary | -|-------|--------| +| ----- | ------ | | Alice | 130000 | -| Bob | 90000 | -| Carol | 85000 | +| Bob | 90000 | +| Carol | 85000 | Dave is a Director in Marketing, so he gets a different filter from the same policy: @@ -137,9 +137,9 @@ sqlite3 company.db "SELECT name, salary FROM employees $FILTER;" Output — Dave sees all Marketing salaries: | name | salary | -|------|--------| +| ---- | ------ | | Dave | 120000 | -| Eve | 95000 | +| Eve | 95000 | ### 6. Non-Directors are denied @@ -174,12 +174,11 @@ The application never needs to know _how_ the policy decides which salaries are ## Handling unconditional results - -| OPA response | Meaning | Application action | -|----------------------------|---------------------|--------------------------------| -| `{ "query": "WHERE ..." }` | Conditional allow | Append filter to SQL query | -| `{ "query": "" }` | Unconditional allow | Run query with no `WHERE` | -| `{}` | Unconditional deny | Return zero rows, skip query | +| OPA response | Meaning | Application action | +| -------------------------- | ------------------- | ---------------------------- | +| `{ "query": "WHERE ..." }` | Conditional allow | Append filter to SQL query | +| `{ "query": "" }` | Unconditional allow | Run query with no `WHERE` | +| `{}` | Unconditional deny | Return zero rows, skip query | ## Clean up