feat(stack-encrypt): make two-letter words such as "Ng" searchable in the match index

.changeset/eql-v4-match-short-words.md

@@ -0,0 +1,11 @@
+---
+'@cipherstash/eql': minor
+---
+
+**EQL v4's match types index each two-letter word whole, so a two-letter surname such as "Ng" is searchable** (#1309). `eql-bindings`' `EqlMatch`, the match configuration of `TextMatch`, `TextSearch` and `TextSearchOre`, turns on Stack Encrypt's `MatchOptions::short_words`: beside the 3-grams of the whole text, every word of exactly two characters (words split on space, comma, semicolon, colon and exclamation mark) is one more token, the word itself, hashed under a domain of its own and folded into the same 2048-bit filter by the same 6 positions. "Ng" now stores into `TextMatch` (it was refused, cutting no 3-gram), a `TextSearch` holding "Ng" stores a filter rather than the empty one, and `@@` with the query "ng" finds the rows holding the word "Ng" ("Ng", "Mike Ng", "NG, Anna"), not "Nguyen". A query's two-letter words must be whole words of the row, so "Mike Ng" does not find "Mike Nguyen". Text that still cuts no token (one letter, or none) is refused by `TextMatch` and stored with the empty filter by the search types, as before.
+
+**The v4 match format now includes short-word tokens.** A filter of text holding a two-letter word gains that word's bits; text with none derives the filter it did. EQL v4 is unreleased, so this is the format it ships with: a v4 value written by an earlier build that holds a two-letter word lacks the word's bits, and a query naming that word does not find it until the value is re-encrypted.
+
+**What it reveals.** One more token per two-letter word, under the same PRF, in the same filter, by the same containment rule, so no new class of leakage. The token is specific to the word: someone who clusters tokens across rows can tell the rows holding the word "Ng" from those holding "Nguyen", where before only the boundary trigram `" ng"` they share was visible. No bigram of a longer word is indexed.
+
+No SQL function changes (one doc comment in the bundle is reworded), and the TypeScript types do not change.

languages/golang/CHANGELOG.md

@@ -12,6 +12,19 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
 
 ### Added
 
+- **A two-letter word is searchable in `TextMatch`, `TextSearch` and
+  `TextSearchOre` (#1309).** The EQL v4 match types index each two-letter
+  word whole, beside the 3-grams of the text, so a two-letter surname such
+  as "Ng" stores into a `TextMatch` field (it was refused with `ErrTerm`),
+  and `Matches` with "ng" is a query (it was refused) that finds the rows
+  holding the word "Ng", not "Nguyen". A query's two-letter words must be
+  whole words of the row: "Mike Ng" does not find "Mike Nguyen". Text that
+  cuts no token, an empty string or one letter, is still refused by
+  `TextMatch` and stored with the empty filter by the search types. The
+  word token is one more token per two-letter word, in the same filter, so
+  someone who clusters tokens across rows can tell "Ng" from "Nguyen".
+  A native `index=match` field takes no options yet (#1112), so this
+  reaches Go through the EQL types only.
 - **Decimals, dates and timestamps take an order index (#1236).** An
   `encrypt.Decimal`, `encrypt.Date` or `time.Time` field takes `ore` and
   `ope`, and the generated code gives it `Lt`, `Lte`, `Gt`, `Gte` and

languages/golang/cmd/stashgen/README.md

@@ -199,7 +199,8 @@ The ordered date, timestamp and numeric types are refused: the engine derives no
 An ordered type has no `Between` method yet, because a `BETWEEN` operand needs EQL v4's range query domains.
 
 Any built-in Go integer field takes an integer type (ADR-0002) and seals at the type's width: an `int16` field always fits an `Integer`, and an `int64` field is range checked value by value, `Encrypt` refusing a value that does not fit with `encrypt.ErrEncoding`. `encrypt.Int128` and `encrypt.Uint128` are not converted. A float's width must match: a `float64` field is not a `Real`.
-A `TextMatch` field refuses text shorter than three characters, such as an empty string or a two-letter code, because it has no 3-grams to index; its query refuses such text too.
+A `TextMatch` field indexes the 3-grams of its text and each two-letter word whole, so a two-letter surname such as "Ng" is stored and found by the query "ng"; a query's two-letter words must be whole words of the row.
+A `TextMatch` field refuses text that cuts no token, such as an empty string or one letter; its query refuses such text too.
 A `TextSearch` field stores such text with an empty match filter, so its `Eq`, `Lt`, `Lte`, `Gt` and `Gte` queries find it and no `Matches` query does; those queries take such text too, and its `Matches` query refuses it.
 Put the column on the left of `@@` (`WHERE name @@ $1::eql_v4.query_text_match`): the other operand order finds nothing until #1171 is fixed.
 `encrypt_into` with any other type is refused with the type's name and the reason it is not produced yet.

languages/golang/encrypt/README.md

@@ -490,7 +490,7 @@ The EQL types the engine produces, and the Go field types each takes:
 | EQL type | Go field | Queries |
 |---|---|---|
 | `TextEq` | `string` | `=`, `<>` |
-| `TextMatch` | `string` | `@@` (substring match, 3-grams, case-insensitive) |
+| `TextMatch` | `string` | `@@` (substring match, 3-grams and two-letter words, case-insensitive) |
 | `TextOrd`, `TextOrdOpe`, `TextOrdOre` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY` |
 | `TextSearch`, `TextSearchOre` | `string` | `=`, `<>`, `<`, `<=`, `>`, `>=`, `ORDER BY`, `@@` |
 | `Text` | `string` | none: stored and read only |
@@ -527,13 +527,19 @@ with `ErrEncoding`, naming the field's kind and not the value.
 `encrypt.Int128` and `encrypt.Uint128` are not converted, and floats take
 exactly their own width.
 
-A `TextMatch` value must be at least three characters long: shorter text has
-no 3-grams to index and is refused. A `TextSearch` value of such text, an
-empty string included, is stored with an empty match filter: `Eq`, `Lt`,
-`Lte`, `Gt` and `Gte` find it through its other terms, and no `Matches`
-query does. A `TextSearch` query carries only the term its operator reads,
-so `Eq` and `Lt` on two-letter text work and `Matches` on it is refused; EQL
-v4 raises if a query lacks the term its operator compares. The number
+A `TextMatch` value is indexed by its 3-grams and by each two-letter word
+whole, so a two-letter surname such as "Ng" is stored, and `Matches` with
+"ng" finds the rows holding the word "Ng" ("Ng", "Mike Ng"), not "Nguyen".
+A query's two-letter words must be whole words of the row, so "Mike Ng" does
+not find "Mike Nguyen". The word token reveals one more token per two-letter
+word, in the same filter: someone who clusters tokens across rows can tell
+"Ng" from "Nguyen", where before only the trigram they share was visible.
+Text that cuts no token, such as an empty string or one letter, is refused. A
+`TextSearch` value of such text is stored with an empty match filter: `Eq`,
+`Lt`, `Lte`, `Gt` and `Gte` find it through its other terms, and no
+`Matches` query does. A `TextSearch` query carries only the term its
+operator reads, so `Eq` and `Lt` on one-letter text work and `Matches` on it
+is refused; EQL v4 raises if a query lacks the term its operator compares. The number
 equality types compare as
 PostgreSQL compares the plain types: an integer equals itself whatever its
 width, and for floats `-0.0 = 0.0` and `NaN = NaN`; the stored value still

languages/golang/encrypt/eql_v4_test.go

@@ -279,13 +279,24 @@ func TestV4QueriesMatchTheRustFixture(t *testing.T) {
 		t.Errorf("TargetLimit with a histogram = %d (bounded %v), want at least 10", rows, bounded)
 	}
 
-	// A needle too short for a 3-gram has no filter bits, and an empty
-	// filter would be a subset of every row's: the engine refuses it.
-	for _, short := range []string{"", "al"} {
+	// A needle that cuts no token (one letter, or none) has no filter bits,
+	// and an empty filter would be a subset of every row's: the engine
+	// refuses it.
+	for _, short := range []string{"", "a"} {
 		if q, err := testusers.ProfileFields.Name.Matches(ctx, cipher, short); err == nil {
 			t.Errorf("Name.Matches(%q) = %s, want a refusal", short, q)
 		}
 	}
+	// A two-letter word is a token of its own in EQL v4's match types, so a
+	// two-letter surname is a query, and Verify keeps the rows holding it as
+	// a word.
+	ng, err := testusers.ProfileFields.Name.Matches(ctx, cipher, "Ng")
+	if err != nil {
+		t.Fatalf("Name.Matches(%q): %v", "Ng", err)
+	}
+	if ok, err := ng.Verify(ctx, []string{"Ng", "Mike Ng", "Nguyen"}); err != nil || !ok[0] || !ok[1] || ok[2] {
+		t.Errorf("Verify(Ng, Mike Ng, Nguyen) = %v, %v, want true, true, false", ok, err)
+	}
 }
 
 // rowsFixture is v4_rows.json: the profiles above as the generated code seals

languages/golang/encrypt/eql_v4_text_search_test.go

@@ -14,9 +14,10 @@ import (
 
 // EQL v4's TextSearch, sealed and queried through generated code over the
 // deterministic guest build with the EQL types. A stored value carries all
-// three terms, short text with the empty filter; a query carries only the
-// one its operator reads (ADR-0002, amended 2026-10-19), so Eq and Lt take a
-// two-letter name and Matches refuses one.
+// three terms, text that cuts no match token with the empty filter; a query
+// carries only the one its operator reads (ADR-0002, amended 2026-10-19), so
+// Eq and Lt take a one-letter name and Matches refuses one. A two-letter
+// word is a match token of its own, so a two-letter name is matched too.
 // packages/eql/tests/encryption/tests/v4_text_search.rs holds the same rules
 // against PostgreSQL.
 
@@ -82,14 +83,14 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) {
 		if string(terms[tc.term]) != string(stored[tc.term]) {
 			t.Errorf("%s query's %s is not the stored one", tc.method, tc.term)
 		}
-		// Text too short for a 3-gram still has an equality and an order
+		// Text that cuts no match token still has an equality and an order
 		// term.
-		short, err := tc.query("Al")
+		short, err := tc.query("A")
 		if err != nil {
-			t.Fatalf("%s on two-letter text: %v", tc.method, err)
+			t.Fatalf("%s on one-letter text: %v", tc.method, err)
 		}
 		if keys, _ := keysAndTerms(t, short); keys != strings.Join(sortedWith("i", "v", tc.term), ",") {
-			t.Errorf("%s query keys on two-letter text %s", tc.method, keys)
+			t.Errorf("%s query keys on one-letter text %s", tc.method, keys)
 		}
 	}
 
@@ -108,19 +109,36 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) {
 		t.Errorf("Verify = %v, %v", ok, err)
 	}
 
-	// A match query on text too short for a 3-gram is refused, never sent
-	// as an empty filter.
-	if q, err := f.Matches(ctx, cipher, "Al"); !errors.Is(err, encrypt.ErrTerm) {
-		t.Errorf("Matches on two-letter text = %v, %v, want ErrTerm", q, err)
+	// A match query on text that cuts no token is refused, never sent as an
+	// empty filter.
+	if q, err := f.Matches(ctx, cipher, "A"); !errors.Is(err, encrypt.ErrTerm) {
+		t.Errorf("Matches on one-letter text = %v, %v, want ErrTerm", q, err)
+	}
+	// A two-letter word is one token: "Ng" stores a filter, and the query of
+	// "ng" carries the same bits.
+	ng, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 4, Name: "Ng"}})
+	if err != nil {
+		t.Fatalf("storing a two-letter name in a TextSearch: %v", err)
+	}
+	_, ngTerms := keysAndTerms(t, ng[0].Name)
+	if string(ngTerms["bf"]) == "[]" {
+		t.Error(`"Ng" stored the empty filter, want its word token`)
+	}
+	ngQuery, err := f.Matches(ctx, cipher, "ng")
+	if err != nil {
+		t.Fatalf("Matches(%q): %v", "ng", err)
+	}
+	if _, q := keysAndTerms(t, ngQuery.Query()); string(q["bf"]) != string(ngTerms["bf"]) {
+		t.Errorf(`the query "ng"'s bf %s is not the stored "Ng"'s %s`, q["bf"], ngTerms["bf"])
 	}
 	// Storing such text, an empty string included, keeps every term, the
 	// match term as the empty filter (where TextMatch refuses the text): Eq
 	// and Lt find the row through hm and op, and no Matches query does.
-	short, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 2, Name: "Al"}, {ID: 3, Name: ""}})
+	short, err := testusers.EncryptEntry(ctx, cipher, []testusers.Entry{{ID: 2, Name: "A"}, {ID: 3, Name: ""}})
 	if err != nil {
 		t.Fatalf("storing short text in a TextSearch: %v", err)
 	}
-	for i, name := range []string{"Al", ""} {
+	for i, name := range []string{"A", ""} {
 		keys, terms := keysAndTerms(t, short[i].Name)
 		if keys != "bf,c,hm,i,op,v" {
 			t.Errorf("stored keys for %q %s, want bf,c,hm,i,op,v", name, keys)
@@ -147,7 +165,7 @@ func TestV4TextSearchStoresEveryTermAndQueriesCarryOne(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if opened[0].Name != "Al" || opened[1].Name != "" {
+	if opened[0].Name != "A" || opened[1].Name != "" {
 		t.Errorf("opened %q and %q", opened[0].Name, opened[1].Name)
 	}
 }

languages/golang/encrypt/internal/testusers/directory.go

@@ -4,7 +4,7 @@ package testusers
 
 // Entry stores a name as EQL v4's TextSearch: one column that answers `=`,
 // the orderings and `@@`. Its query carries only the term its operator
-// reads, so `=` and `<` work on a two-letter name while `@@` on one is
+// reads, so `=` and `<` work on a one-letter name while `@@` on one is
 // refused (ADR-0002, amended 2026-10-19).
 type Entry struct {
 	_    struct{} `stash:"context=directory"`

packages/eql/crates/eql-bindings/src/encryption.rs

@@ -295,8 +295,23 @@ impl<P, Ctx> DecryptField<P, Ctx> for Hmac256 {
     }
 }
 
-/// The match configuration of EQL v4's `TextMatch`: 3-grams of the
-/// lower-cased text, `k = 6` positions per token in a 2048-bit filter.
+/// The match configuration of EQL v4's match types (`TextMatch`,
+/// `TextSearch`, `TextSearchOre`): 3-grams of the lower-cased text, plus each
+/// two-letter word whole as one more token (Stack Encrypt's
+/// `MatchOptions::short_words`), `k = 6` positions per token in a 2048-bit
+/// filter.
+///
+/// The short-word tokens are what make a two-letter surname searchable: "Ng"
+/// cuts no 3-gram, so without them it would be no `TextMatch` value at all,
+/// and the query "ng" would be refused. With them "Ng" stores one token, and
+/// `@@` with the query "ng" finds the rows holding the word "Ng" ("Ng",
+/// "Mike Ng", "Ng, Mike"), not "Nguyen". A query's two-letter words must be
+/// whole words of the row, so "Mike Ng" does not find "Mike Nguyen". What it
+/// reveals: one more token per two-letter word, under the same PRF (in a
+/// domain of its own), in the same filter, by the same containment rule, so
+/// no new class of leakage; the token is specific to the word, so someone who
+/// clusters tokens across rows can tell "Ng" from "Nguyen", where before only
+/// the boundary trigram `" ng"` they share was visible.
 ///
 /// Pinned here rather than taken from Stack Encrypt's default, because it is
 /// part of the stored format: a stored filter and a query filter agree only
@@ -316,6 +331,7 @@ impl MatchConfig for EqlMatch {
         MatchOptions {
             tokenizer: Tokenizer::Ngram { length: 3 },
             downcase: true,
+            short_words: true,
             k: 6,
             m: 2048,
         }
@@ -328,8 +344,9 @@ impl MatchConfig for EqlMatch {
 /// cipherstash-client's parameters is not a goal, because cipherstash-client
 /// never writes an `eql_v4` column.
 ///
-/// Text that cuts no token (empty, separator-only, or shorter than a 3-gram)
-/// is refused with [`TermError::EmptyTermText`], stored or queried. Stack
+/// Text that cuts no token (empty, separator-only, or shorter than a 3-gram
+/// with no two-letter word, such as one letter) is refused with
+/// [`TermError::EmptyTermText`], stored or queried. Stack
 /// Encrypt stores such a value as the empty filter, but this type is the
 /// filter of every match query, and a query's filter must never be empty;
 /// it is also `TextMatch`'s stored filter, and a `TextMatch` value with an
@@ -462,6 +479,10 @@ mod tests {
         let options = EqlMatch::options();
         assert!(matches!(options.tokenizer, Tokenizer::Ngram { length: 3 }));
         assert!(options.downcase);
+        assert!(
+            options.short_words,
+            "the v4 match types index each two-letter word whole"
+        );
         assert_eq!(options.k, 6);
         assert_eq!(options.m, 2048);
         assert!(

packages/eql/crates/eql-bindings/src/v4/mod.rs

@@ -98,8 +98,9 @@ pub mod terms {
     /// The `bf` term of a stored search value (`TextSearch`,
     /// `TextSearchOre`): a [`BloomFilter`] that may be empty.
     ///
-    /// Text too short to cut a 3-gram (empty, separator-only, or shorter than
-    /// three characters) has no match tokens. A search column stores it with
+    /// Text that cuts no match token (empty, separator-only, or shorter than
+    /// three characters with no two-letter word, such as one letter) has no
+    /// match tokens. A search column stores it with
     /// the empty filter, Stack Encrypt's answer for a stored value, so `=`
     /// and `<` still find it through its `hm` and `op`, and no `@@` query
     /// does: EQL's `matches` finds a stored filter only when it contains the

packages/eql/src/v3/scalars/functions.sql

@@ -70,9 +70,10 @@ $$ LANGUAGE plpgsql;
 --! as no row.
 --!
 --! A term whose value is JSON `null` is missing, and so is an empty array: a
---! `bf` with no positions is the filter of text too short for a 3-gram, which
---! a stored filter of the same text would contain, so a query carrying it
---! would match exactly the stored values no match query should find.
+--! `bf` with no positions is the filter of text that cuts no match token (one
+--! letter, or none), which a stored filter of the same text would contain, so
+--! a query carrying it would match exactly the stored values no match query
+--! should find.
 --!
 --! The raise happens where the operand is evaluated, not where the statement
 --! is planned. A literal operand is folded at plan time and raises even over

packages/eql/tests/encryption/tests/typed_queries.rs

@@ -462,7 +462,7 @@ async fn a_typed_query_refuses_what_the_data_path_refuses() {
         .build()
         .unwrap();
     let short = cipher
-        .query(op::Matches, "al")
+        .query(op::Matches, "a")
         .keyset("tenant")
         .on(&plan)
         .await;