Merge pull request #1445 from cipherstash/feat/eql-v4-json-tree-element-match

feat(eql): test equality and ranges on one array element of a json_tree document

Author: coderdan

.changeset/eql-json-tree-element-match.md

@@ -0,0 +1,9 @@
+---
+"@cipherstash/eql": minor
+---
+
+EQL v4 `json_tree`: a predicate on one array element, `eql_v4.json_tree_element_match(col, $1::eql_v4.query_json_elements)` (#1440).
+
+`exists($.contacts[*] ? (@.relationship == "Sibling" && @.calls > 5))` asks for one contact that is both. On a `public.eql_v4_json_search` column the nearest query, `@>` for the relationship and an ordering over `eql_v4.json_tree_path_query` for the calls, can be answered by two different contacts, so it returned more rows than the plaintext query. The new function answers the element-scoped question with one operand, the new domain `eql_v4.query_json_elements`, which Stack Encrypt's `element_match` query operator derives: for each field of the element, the tokens of a subtree read of its path and its bounds. Every field must be at a path the column orders, and equality is the ordering term's, so text compares case- and accent-folded and integers past 2^53 can tie. A row whose array is longer than the bound the operand was derived to raises `invalid_parameter_value` rather than answer over part of the array.
+
+`eql_v4.json_tree_subtree_indexed(col, tokens, budget)` is the subtree read with `k`, the place in `tokens` of the token whose node each row sits in, which pairs the fields of one element. `eql_v4.json_tree_subtree` now returns its rows without `k`, and is otherwise unchanged.

languages/golang/encrypt/eql/eql_gen.go

@@ -60,7 +60,7 @@ Where the two versions' SQL must differ, the source holds both forms in **versio
 
 #### Files confined to one EQL version
 
-A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json` and `eql_v4.query_json_leaf`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
+A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Three surfaces do today. The generated range surface: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7). And the hand-written JSON tree family under `src/v3/json_tree/` (`types.sql`, `functions.sql`, `operators.sql`, `elements.sql`, `blockers.sql`): `public.eql_v4_json_search`, `public.eql_v4_json_entry`, `eql_v4.query_json`, `eql_v4.query_json_leaf` and `eql_v4.query_json_elements`, the document Stack Encrypt's JSON index writes, under the names v3 gives its searchable JSON document but not a port of the SteVec surface (design: `docs/plans/2026-10-17-json-tree-index.md` at the repository root; bindings: `crates/eql-bindings/src/v4/json_tree.rs`, hand-written like the SteVec structs; catalog: `eql_domains::DOCUMENT_FAMILIES`, whose stored type is the target `JsonSearch`). Both are `-- ONLY EQL VERSION: 4`. The third surface goes the other way: the SteVec SQL under `src/v3/json/` and the generated `json_entry_<T>` cross files under `src/v3/scalars/<T>/` are `-- ONLY EQL VERSION: 3`, because in v4 their domain names belong to the JSON tree family. EQL v3 is what cipherstash-client writes, nothing there produces a range or a JSON tree document, and its SQL must not change: the v3 bundle differs from an unconfined build only by those directive comment lines. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs the whole v4 install order (`build/render/eql_v4/deps-ordered.txt`) in one rolled-back transaction, so every statement of the confined files runs after what it requires, besides running each file of the v3 render on its own. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the files confined to either version; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
 
 Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) reads each file's rendered copy from `build/render/eql_v3/`. A new tool that parses `src/v3` must do the same, and depend on `build`; do not substitute the placeholders a second way.
 

packages/eql/AGENTS.md

@@ -152,6 +152,18 @@ impl Target {
         document(self.name).map(|_| crate::v4::json_tree::JsonTreeLeafQuery::sql_domain_static())
     }
 
+    /// The PostgreSQL domain of the type's element-scoped predicate, for a
+    /// type that stores a JSON document (`JsonSearch`:
+    /// `eql_v4.query_json_elements`); `None` for every other type. Stack
+    /// Encrypt binds `element_match` as it
+    /// (`TargetDescriptor::with_elements_query_sql_domain`), and answers
+    /// `element_match` only on a type that names it.
+    pub fn elements_query_sql_domain(&self) -> Option<&'static str> {
+        use crate::v4::DomainType;
+        document(self.name)
+            .map(|_| crate::v4::json_tree::JsonTreeElementsQuery::sql_domain_static())
+    }
+
     /// The options this type's match index is built under, for a type that
     /// carries one (`TextMatch`): [`EqlMatch`](super::EqlMatch)'s, the one
     /// match configuration every EQL type's filter is built under. `None`

packages/eql/crates/eql-bindings/src/encryption/targets.rs

@@ -106,6 +106,7 @@ pub fn all_query() -> Vec<Box<dyn DomainType>> {
         Box::new(PhantomData::<super::double::DoubleOrdOpeQuery>),
         Box::new(PhantomData::<super::json_tree::JsonTreeQuery>),
         Box::new(PhantomData::<super::json_tree::JsonTreeLeafQuery>),
+        Box::new(PhantomData::<super::json_tree::JsonTreeElementsQuery>),
     ]
 }
 /// Every v4 RANGE query operand (`eql_v4.query_<family>_range_<term>`),

packages/eql/crates/eql-bindings/src/v4/inventory.rs

@@ -177,6 +177,48 @@ pub struct JsonTreeLeafQuery {
     pub op: JsonTreeOrderingTerm,
 }
 
+/// A read token, in [`Base64Text`]: sixteen bytes a client derives from a
+/// concrete path alone, which `eql_v4.json_tree_subtree` and its indexed
+/// form take to find the node at that path in every row.
+#[derive(Clone, Debug, PartialEq, Eq, Hash, Serialize, Deserialize, TS, JsonSchema)]
+pub struct JsonTreeToken(pub Base64Text);
+
+/// One member of an element-scoped predicate: the tokens of a read of one
+/// field's path through the array's wildcard, one per element below the
+/// bound in element order and then the probe, and the field's bounds,
+/// each an ordering operand. `gt` or `gte` is the lower bound, `lt` or
+/// `lte` the upper; equality is `gte` and `lte` at one value.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
+#[serde(deny_unknown_fields)]
+pub struct JsonTreeElementsMember {
+    pub t: Vec<JsonTreeToken>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub gt: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub gte: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub lt: Option<JsonTreeLeafQuery>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    #[ts(optional)]
+    pub lte: Option<JsonTreeLeafQuery>,
+}
+
+/// `eql_v4.query_json_elements`: an element-scoped predicate, `{"m":
+/// [member, …]}`, true of a row where some one element of the array
+/// satisfies every member (`eql_v4.json_tree_element_match`). What Stack
+/// Encrypt's `element_match` operator derives. The domain's CHECK also
+/// holds what this type does not: at least one member, each with at least
+/// one bound and at most one of each side, all its bounds at one selector,
+/// and every member's tokens as many as every other's.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, TS, JsonSchema)]
+#[serde(deny_unknown_fields)]
+pub struct JsonTreeElementsQuery {
+    pub m: Vec<JsonTreeElementsMember>,
+}
+
 /// The entry `col -> selector` returned, with the selector that asked for
 /// it: opens to the scalar the entry holds, or to the whole document for a
 /// sealed root asked for at `$`. The only constructor,
@@ -287,6 +329,7 @@ json_tree_domain_type!(JsonTreeDocument, "public.eql_v4_json_search");
 json_tree_domain_type!(JsonTreeEntry, "public.eql_v4_json_entry");
 json_tree_domain_type!(JsonTreeQuery, "eql_v4.query_json");
 json_tree_domain_type!(JsonTreeLeafQuery, "eql_v4.query_json_leaf");
+json_tree_domain_type!(JsonTreeElementsQuery, "eql_v4.query_json_elements");
 
 // =============================================================================
 // Conversions to and from the engine's types, through their one wire form
@@ -547,6 +590,38 @@ mod tests {
             JsonTreeLeafQuery::sql_domain_static(),
             "eql_v4.query_json_leaf"
         );
+        assert_eq!(
+            JsonTreeElementsQuery::sql_domain_static(),
+            "eql_v4.query_json_elements"
+        );
+    }
+
+    #[test]
+    fn an_elements_query_is_members_of_tokens_and_bounds() {
+        let leaf = serde_json::json!({"s": b64(0xaa), "op": "AQI="});
+        let query = serde_json::json!({"m": [
+            {"t": [b64(1), b64(2)], "gte": leaf, "lte": leaf},
+            {"t": [b64(3), b64(4)], "gt": leaf},
+        ]});
+        let parsed: JsonTreeElementsQuery = serde_json::from_value(query.clone()).unwrap();
+        assert_eq!(parsed.m.len(), 2, "one member per field");
+        assert!(parsed.m[1].lt.is_none(), "an absent bound is none");
+        assert_eq!(
+            serde_json::to_value(&parsed).unwrap(),
+            query,
+            "absent bounds are left out"
+        );
+        for wrong in [
+            serde_json::json!({"m": [{"t": [b64(1)], "eq": leaf}]}),
+            serde_json::json!({"m": [{"gt": leaf}]}),
+            serde_json::json!({"m": [], "x": 1}),
+            serde_json::json!({"m": [{"t": ["not base64"], "gt": leaf}]}),
+        ] {
+            assert!(
+                serde_json::from_value::<JsonTreeElementsQuery>(wrong.clone()).is_err(),
+                "{wrong} should not deserialize"
+            );
+        }
     }
 
     #[test]

packages/eql/crates/eql-bindings/src/v4/json_tree.rs

@@ -1061,7 +1061,7 @@ pub fn render_v4_inventory_rs() -> String {
         })
         .collect();
     // The document families' hand-written types: the stored document in
-    // `all()`, its two query operands in `all_query()`. The entry `->`
+    // `all()`, its query operands in `all_query()`. The entry `->`
     // returns is neither a column nor an operand, so it is in neither.
     let document = |roles: &'static [eql_domains::DocumentRole]| -> TokenStream {
         eql_domains::DOCUMENT_FAMILIES
@@ -1083,6 +1083,7 @@ pub fn render_v4_inventory_rs() -> String {
         let documents = document(&[
             eql_domains::DocumentRole::Query,
             eql_domains::DocumentRole::LeafQuery,
+            eql_domains::DocumentRole::ElementsQuery,
         ]);
         quote! { #query_entries #documents }
     };

packages/eql/crates/eql-codegen/src/bindings.rs

@@ -186,6 +186,9 @@ fn document_types(out: &mut String) {
                 DocumentRole::LeafQuery => {
                     "ordering operand at a path, compared with an entry `->` extracts"
                 }
+                DocumentRole::ElementsQuery => {
+                    "element-scoped predicate, which eql_v4.json_tree_element_match tests"
+                }
                 DocumentRole::Entry => {
                     "entry: one node of a document, as `->` and the walks return it"
                 }
@@ -271,6 +274,7 @@ mod tests {
             "JSONSearch",
             "JSONSearchQuery",
             "JSONSearchLeafQuery",
+            "JSONSearchElementsQuery",
             "JSONEntry",
         ] {
             assert_eq!(

packages/eql/crates/eql-codegen/src/go_eql.rs

@@ -554,7 +554,7 @@ pub const CATALOG: &[DomainFamily] = &[
 ];
 
 /// The role a domain of a [`DocumentFamily`] plays: the column type, the
-/// two query operands, or the entry the SQL returns from a document.
+/// query operands, or the entry the SQL returns from a document.
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub enum DocumentRole {
     /// The stored document: the column type, and the target a data plan
@@ -564,6 +564,9 @@ pub enum DocumentRole {
     Query,
     /// The ordering operand at a path.
     LeafQuery,
+    /// The element-scoped predicate: fields of one array element, each
+    /// with its read tokens and its bounds.
+    ElementsQuery,
     /// One node, as `->` and the walks return it.
     Entry,
 }
@@ -621,8 +624,9 @@ impl DocumentFamily {
 /// gives its own (`public.eql_v3_json_search`, the SteVec document):
 /// `JsonSearch` over `public.eql_v4_json_search`, its needle
 /// `JsonSearchQuery` (`eql_v4.query_json`), its ordering operand
-/// `JsonSearchLeafQuery` (`eql_v4.query_json_leaf`) and its entry
-/// `JsonEntry` (`public.eql_v4_json_entry`). The storage-only `Json`
+/// `JsonSearchLeafQuery` (`eql_v4.query_json_leaf`), its element-scoped
+/// predicate `JsonSearchElementsQuery` (`eql_v4.query_json_elements`) and
+/// its entry `JsonEntry` (`public.eql_v4_json_entry`). The storage-only `Json`
 /// (`public.eql_v4_json`) is the `json` family's. The Rust structs keep the
 /// implementation's name, as v3's `SteVecDocument` does.
 pub const JSON_TREE: DocumentFamily = DocumentFamily {
@@ -648,6 +652,12 @@ pub const JSON_TREE: DocumentFamily = DocumentFamily {
             rust: "JsonTreeLeafQuery",
             sql_domain: "eql_v4.query_json_leaf",
         },
+        DocumentDomain {
+            role: DocumentRole::ElementsQuery,
+            name: "JsonSearchElementsQuery",
+            rust: "JsonTreeElementsQuery",
+            sql_domain: "eql_v4.query_json_elements",
+        },
         DocumentDomain {
             role: DocumentRole::Entry,
             name: "JsonEntry",

packages/eql/crates/eql-domains/src/lib.rs

@@ -1633,6 +1633,7 @@ mod document_families {
                 DocumentRole::Stored,
                 DocumentRole::Query,
                 DocumentRole::LeafQuery,
+                DocumentRole::ElementsQuery,
                 DocumentRole::Entry,
             ] {
                 assert_eq!(
@@ -1645,7 +1646,9 @@ mod document_families {
             for domain in family.domains {
                 let schema = match domain.role {
                     DocumentRole::Stored | DocumentRole::Entry => "public.eql_v4_",
-                    DocumentRole::Query | DocumentRole::LeafQuery => "eql_v4.query_",
+                    DocumentRole::Query | DocumentRole::LeafQuery | DocumentRole::ElementsQuery => {
+                        "eql_v4.query_"
+                    }
                 };
                 assert!(
                     domain.sql_domain.starts_with(schema),
@@ -1677,6 +1680,11 @@ mod document_families {
                 "JsonSearchLeafQuery",
                 "eql_v4.query_json_leaf",
             ),
+            (
+                DocumentRole::ElementsQuery,
+                "JsonSearchElementsQuery",
+                "eql_v4.query_json_elements",
+            ),
             (DocumentRole::Entry, "JsonEntry", "public.eql_v4_json_entry"),
         ] {
             let domain = JSON_TREE.domain(role).unwrap();