Merge pull request #1446 from cipherstash/feat/eql-v4-json-tree-match

feat(eql): one-parameter jsonpath predicates on json_tree documents (json_tree_match, @@, match)

Author: coderdan

.changeset/eql-json-tree-match.md

@@ -0,0 +1,9 @@
+---
+"@cipherstash/eql": minor
+---
+
+EQL v4 `json_tree`: a whole filter as one parameter, `eql_v4.json_tree_match(col, $1::eql_v4.query_json_predicate)` and `col @@ $1::eql_v4.query_json_predicate` (#1441).
+
+An application that compiles a filter language into SQL targets `jsonb_path_match(col, $1, $2)` on a plaintext column: one predicate, one set of values. On a `public.eql_v4_json_search` column each question is its own operator, so the same filter had to be assembled by hand, one operand per leaf. The new domain `eql_v4.query_json_predicate` is a boolean tree (`and`, `or`, `not`) whose leaves are the operands those operators already take (a `contains` needle, an `exists` selector, `cmp` ordering bounds and an `elements` predicate), each checked by its own domain's rule, and `eql_v4.json_tree_match` evaluates it with SQL's three-valued logic, the answer the hand-written WHERE gives. Stack Encrypt's `match` query operator compiles a jsonpath predicate in the subset the terms can answer into the tree. The tree is evaluated inside a function, so it filters a scan and engages no index; the operators written out per leaf are the fast path. The operand must carry its type: an untyped operand on the right of `@@` is refused as ambiguous, and `@@` with a `jsonpath` is still blocked.
+
+`eql_v4.json_tree_element_match` now reads an opened document through `eql_v4_internal.json_tree_element_match_opened`, which `json_tree_match` shares; its answers are unchanged.

languages/golang/encrypt/eql/eql_gen.go

@@ -268,6 +268,35 @@ mod tests {
         ));
     }
 
+    /// The JSON document's query domains this guest names, and so the JSON
+    /// operators the Go SDK is offered: the leaf operand, for the orderings
+    /// at a path, and not the element-scoped or compiled predicate, whose
+    /// operators (`element_match`, `match`) the Go side does not parse yet.
+    /// Naming one is a change to the Go SDK's operator list: update this
+    /// test with the Go side that takes the operator.
+    #[cfg(feature = "eql")]
+    #[test]
+    fn a_document_names_the_leaf_domain_and_not_the_predicate_domains() {
+        let targets = resolver().targets();
+        let document = targets
+            .iter()
+            .find(|t| t.name == "JsonSearch")
+            .expect("JsonSearch");
+        assert_eq!(
+            document.leaf_query_sql_domain.as_deref(),
+            Some("eql_v4.query_json_leaf")
+        );
+        assert_eq!(document.elements_query_sql_domain, None);
+        assert_eq!(document.predicate_query_sql_domain, None);
+        let operators = document.operators();
+        for absent in [
+            stack_encrypt::dynamic::OpSpec::ElementMatch,
+            stack_encrypt::dynamic::OpSpec::Match,
+        ] {
+            assert!(!operators.contains(&absent), "{absent:?}: {operators:?}");
+        }
+    }
+
     #[cfg(feature = "eql")]
     #[test]
     fn eql_refusals_convert_to_the_engines_with_the_field_left_to_the_lowering() {

languages/golang/encrypt/guest/src/targets.rs

@@ -60,7 +60,7 @@ Where the two versions' SQL must differ, the source holds both forms in **versio
 
 #### Files confined to one EQL version
 
-A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `elements.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json`, `eql_v4.query_json_leaf` and `eql_v4.query_json_elements`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
+A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `elements.sql`, `predicate.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json`, `eql_v4.query_json_leaf`, `eql_v4.query_json_elements` and `eql_v4.query_json_predicate`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
 
 Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) reads each file's rendered copy from `build/render/eql_v3/`. A new tool that parses `src/v3` must do the same, and depend on `build`; do not substitute the placeholders a second way.
 

packages/eql/AGENTS.md

@@ -164,6 +164,18 @@ impl Target {
             .map(|_| crate::v4::json_tree::JsonTreeElementsQuery::sql_domain_static())
     }
 
+    /// The PostgreSQL domain of the type's compiled predicate, for a type
+    /// that stores a JSON document (`JsonSearch`:
+    /// `eql_v4.query_json_predicate`); `None` for every other type. Stack
+    /// Encrypt binds `match` as it
+    /// (`TargetDescriptor::with_predicate_query_sql_domain`), and answers
+    /// `match` only on a type that names it.
+    pub fn predicate_query_sql_domain(&self) -> Option<&'static str> {
+        use crate::v4::DomainType;
+        document(self.name)
+            .map(|_| crate::v4::json_tree::JsonTreePredicateQuery::sql_domain_static())
+    }
+
     /// The options this type's match index is built under, for a type that
     /// carries one (`TextMatch`): [`EqlMatch`](super::EqlMatch)'s, the one
     /// match configuration every EQL type's filter is built under. `None`

packages/eql/crates/eql-bindings/src/encryption/targets.rs

@@ -107,6 +107,7 @@ pub fn all_query() -> Vec<Box<dyn DomainType>> {
         Box::new(PhantomData::<super::json_tree::JsonTreeQuery>),
         Box::new(PhantomData::<super::json_tree::JsonTreeLeafQuery>),
         Box::new(PhantomData::<super::json_tree::JsonTreeElementsQuery>),
+        Box::new(PhantomData::<super::json_tree::JsonTreePredicateQuery>),
     ]
 }
 /// Every v4 RANGE query operand (`eql_v4.query_<family>_range_<term>`),

packages/eql/crates/eql-bindings/src/v4/inventory.rs

@@ -219,6 +219,57 @@ pub struct JsonTreeElementsQuery {
     pub m: Vec<JsonTreeElementsMember>,
 }
 
+/// The bounds of an ordering at one path: one or two ordering operands at
+/// the path's selector, `gt` or `gte` the lower and `lt` or `lte` the upper.
+/// The `cmp` leaf of a [`JsonTreePredicateQuery`]. The domain's CHECK also
+/// holds what this type does not: at least one bound, at most one of each
+/// side, and every bound at one selector.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
+#[serde(deny_unknown_fields)]
+pub struct JsonTreeBounds {
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub gt: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub gte: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub lt: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub lte: Option<JsonTreeLeafQuery>,
+}
+
+/// `eql_v4.query_json_predicate`: a compiled predicate, a boolean tree
+/// whose leaves are the family's other query operands, which
+/// `eql_v4.json_tree_match` and `col @@ $1` evaluate. What Stack Encrypt's
+/// `match` operator compiles a jsonpath predicate into. Each node is an
+/// object of one key naming its kind: `{"and": [node, …]}`, `{"or": […]}`,
+/// `{"not": node}`, `{"contains": needle}`, `{"exists": selector}`,
+/// `{"cmp": bounds}` and `{"elements": predicate}`. The domain's CHECK also
+/// holds what this type does not: at least one child under `and` and `or`.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
+#[serde(rename_all = "snake_case", deny_unknown_fields)]
+pub enum JsonTreePredicateQuery {
+    /// True when every child is; false when one is false; else NULL.
+    And(Vec<JsonTreePredicateQuery>),
+    /// True when one child is; false when every child is false; else NULL.
+    Or(Vec<JsonTreePredicateQuery>),
+    /// The child's negation, NULL of NULL.
+    Not(Box<JsonTreePredicateQuery>),
+    /// The document contains the needle, `col @> needle`.
+    Contains(JsonTreeQuery),
+    /// The document has a node at the path, `json_tree_path_exists`.
+    Exists(JsonTreeSelector),
+    /// The first entry at the bounds' selector against each bound,
+    /// `(col -> s) > gt AND (col -> s) < lt`.
+    Cmp(JsonTreeBounds),
+    /// One element of an array satisfies every member,
+    /// `json_tree_element_match`.
+    Elements(JsonTreeElementsQuery),
+}
+
 /// The entry `col -> selector` returned, with the selector that asked for
 /// it: opens to the scalar the entry holds, or to the whole document for a
 /// sealed root asked for at `$`. The only constructor,
@@ -330,6 +381,7 @@ json_tree_domain_type!(JsonTreeEntry, "public.eql_v4_json_entry");
 json_tree_domain_type!(JsonTreeQuery, "eql_v4.query_json");
 json_tree_domain_type!(JsonTreeLeafQuery, "eql_v4.query_json_leaf");
 json_tree_domain_type!(JsonTreeElementsQuery, "eql_v4.query_json_elements");
+json_tree_domain_type!(JsonTreePredicateQuery, "eql_v4.query_json_predicate");
 
 // =============================================================================
 // Conversions to and from the engine's types, through their one wire form
@@ -594,6 +646,47 @@ mod tests {
             JsonTreeElementsQuery::sql_domain_static(),
             "eql_v4.query_json_elements"
         );
+        assert_eq!(
+            JsonTreePredicateQuery::sql_domain_static(),
+            "eql_v4.query_json_predicate"
+        );
+    }
+
+    #[test]
+    fn a_predicate_query_is_a_tree_of_one_key_nodes() {
+        let leaf = serde_json::json!({"s": b64(0xaa), "op": "AQI="});
+        let needle = serde_json::json!({"sv": [{"s": b64(1), "vs": b64(2)}], "el": []});
+        let elements = serde_json::json!({"m": [{"t": [b64(1)], "gt": leaf}]});
+        let query = serde_json::json!({"and": [
+            {"exists": b64(0xaa)},
+            {"not": {"cmp": {"gte": leaf, "lt": leaf}}},
+            {"or": [{"contains": needle}, {"elements": elements}]},
+        ]});
+        let parsed: JsonTreePredicateQuery = serde_json::from_value(query.clone()).unwrap();
+        let JsonTreePredicateQuery::And(children) = &parsed else {
+            panic!("the root is an and: {parsed:?}")
+        };
+        assert!(
+            matches!(children[0], JsonTreePredicateQuery::Exists(_)),
+            "{children:?}"
+        );
+        assert_eq!(
+            serde_json::to_value(&parsed).unwrap(),
+            query,
+            "a tree reads and writes as itself"
+        );
+        for wrong in [
+            serde_json::json!({"exists": b64(1), "not": {"exists": b64(1)}}),
+            serde_json::json!({"xor": []}),
+            serde_json::json!({"cmp": {"eq": leaf}}),
+            serde_json::json!({"exists": 1}),
+            serde_json::json!([]),
+        ] {
+            assert!(
+                serde_json::from_value::<JsonTreePredicateQuery>(wrong.clone()).is_err(),
+                "{wrong} should not deserialize"
+            );
+        }
     }
 
     #[test]

packages/eql/crates/eql-bindings/src/v4/json_tree.rs

@@ -1084,6 +1084,7 @@ pub fn render_v4_inventory_rs() -> String {
             eql_domains::DocumentRole::Query,
             eql_domains::DocumentRole::LeafQuery,
             eql_domains::DocumentRole::ElementsQuery,
+            eql_domains::DocumentRole::PredicateQuery,
         ]);
         quote! { #query_entries #documents }
     };

packages/eql/crates/eql-codegen/src/bindings.rs

@@ -189,6 +189,9 @@ fn document_types(out: &mut String) {
                 DocumentRole::ElementsQuery => {
                     "element-scoped predicate, which eql_v4.json_tree_element_match tests"
                 }
+                DocumentRole::PredicateQuery => {
+                    "compiled predicate, a boolean tree of the other operands, which eql_v4.json_tree_match and `@@` evaluate"
+                }
                 DocumentRole::Entry => {
                     "entry: one node of a document, as `->` and the walks return it"
                 }
@@ -275,6 +278,7 @@ mod tests {
             "JSONSearchQuery",
             "JSONSearchLeafQuery",
             "JSONSearchElementsQuery",
+            "JSONSearchPredicateQuery",
             "JSONEntry",
         ] {
             assert_eq!(

packages/eql/crates/eql-codegen/src/go_eql.rs

@@ -567,6 +567,9 @@ pub enum DocumentRole {
     /// The element-scoped predicate: fields of one array element, each
     /// with its read tokens and its bounds.
     ElementsQuery,
+    /// The compiled predicate: a boolean tree whose leaves are the other
+    /// query operands.
+    PredicateQuery,
     /// One node, as `->` and the walks return it.
     Entry,
 }
@@ -625,8 +628,10 @@ impl DocumentFamily {
 /// `JsonSearch` over `public.eql_v4_json_search`, its needle
 /// `JsonSearchQuery` (`eql_v4.query_json`), its ordering operand
 /// `JsonSearchLeafQuery` (`eql_v4.query_json_leaf`), its element-scoped
-/// predicate `JsonSearchElementsQuery` (`eql_v4.query_json_elements`) and
-/// its entry `JsonEntry` (`public.eql_v4_json_entry`). The storage-only `Json`
+/// predicate `JsonSearchElementsQuery` (`eql_v4.query_json_elements`), its
+/// compiled predicate `JsonSearchPredicateQuery`
+/// (`eql_v4.query_json_predicate`) and its entry `JsonEntry`
+/// (`public.eql_v4_json_entry`). The storage-only `Json`
 /// (`public.eql_v4_json`) is the `json` family's. The Rust structs keep the
 /// implementation's name, as v3's `SteVecDocument` does.
 pub const JSON_TREE: DocumentFamily = DocumentFamily {
@@ -658,6 +663,12 @@ pub const JSON_TREE: DocumentFamily = DocumentFamily {
             rust: "JsonTreeElementsQuery",
             sql_domain: "eql_v4.query_json_elements",
         },
+        DocumentDomain {
+            role: DocumentRole::PredicateQuery,
+            name: "JsonSearchPredicateQuery",
+            rust: "JsonTreePredicateQuery",
+            sql_domain: "eql_v4.query_json_predicate",
+        },
         DocumentDomain {
             role: DocumentRole::Entry,
             name: "JsonEntry",