[coderdan]

Summary

A two-letter surname such as "Ng" can now be stored and searched. Stack Encrypt's match index (the encrypted full-text index: text is cut into 3-character pieces, "trigrams", each piece is hashed under a secret key, and the hashes set bits in a fixed-size bit array, a Bloom filter) gains an option, MatchOptions::short_words. With it on, every word too short to hold a trigram (exactly two characters under the default 3-grams) becomes one more token, the word itself, beside the trigrams of the whole text. So "Ng" stores one token instead of an empty filter, and the query "Ng" is accepted and finds "Ng", "Mike Ng" and "NG, Anna", but not "Nguyen".

EQL v4's match types (TextMatch, TextSearch, TextSearchOre; EQL is the SQL library that stores and queries encrypted values in PostgreSQL) turn the option on, so the EQL v4 match format now includes short-word tokens. "Ng" was refused as a TextMatch value before; it now stores, and @@ with the query "ng" finds it.

This differs from the fix #1309 proposed (one whole-text token, only when the whole text is shorter than a trigram). The per-word token also makes "Mike Ng" findable by "ng", which the whole-text token would not. A comment on #1309 says the same.

What it reveals. One more token per short word, hashed by the same keyed function (under a label of its own), in the same filter, matched by the same rule, so no new class of leakage. The token is specific to the word: someone who clusters tokens across rows can tell the rows holding the word "Ng" from those holding "Nguyen", where before only the boundary trigram " ng" they share was visible. No bigram of a longer word is ever indexed.

Changes

Stack Encrypt (packages/stack-encrypt)

• MatchOptions::short_words: bool, default off. A word is a run of characters between the separators Tokenizer::Standard splits on (space, comma, semicolon, colon, exclamation mark), so a short word is exactly a token the standard tokenizer would cut; length is counted in characters of the NFC text. The comma matters for names written "Ng, Mike".
• The word token is hashed under its own domain, stack-encrypt/sem/match/word/v1, framed with PAE (the crate's length-prefixed encoding of hash inputs) beside the column context, as every other term kind is. A word token and a trigram of the same characters never share a hash.
• Refused (TermError::InvalidOptions) beside the standard tokenizer, which already emits whole words, and beside n-grams shorter than 3, where no word is short enough.
• Part of the index's identity: a query asked under the other setting is refused (PlanError::IndexOptions), as a mismatched k is.
• Data-plan wire form: {"match": {"short_words": true, ...}}, written only when on, so every other plan serializes to the same bytes as before. The IndexOptions error payload follows the same rule.
• A query's verifier checks each kind of token against the same kind in the row. The expected_tokens fetch prior's docs say it counts word tokens.
• The sem module docs gain a "What a match index reveals" section; MatchOptions::short_words's rustdoc states the leakage; CHANGELOG entry.

EQL (packages/eql)

• EqlMatch (the pinned match configuration of the v4 match types) sets short_words: true. Text with no two-letter word derives the filter it did; the pinned "alice" positions do not move.
• Tests that used two-letter text as "too short for a match term" now use one letter. New tests store and find sixteen real two-letter surnames through the typed path and the by-name dispatch, and a PostgreSQL test runs @@ over TextMatch and TextSearch columns.
• One SQL doc comment reworded; no SQL function changes. Changeset: @cipherstash/eql minor.

Go SDK (languages/golang)

• No Go code change: the EQL guest links eql-bindings by path, so a field typed encrypt_into=TextMatch / TextSearch / TextSearchOre gets the word token. Tests updated to match ("Ng" stores and is a query; one letter is refused), README and CHANGELOG updated.
• A native index=match field still takes no options: stashgen refuses every index option until the guest derives query terms under a field's own options (Go SDK: stashgen keeps its own copy of engine rules (index options, index outputs, a second lowering) — ask the engine instead #1112). So the option reaches Go through the EQL types only.

Not touched: the TypeScript packages carry no Stack Encrypt match options (they use EQL v3 through protect-ffi), so there is nothing to plumb there. No skill covers EQL v4 or Stack Encrypt, so no skills/ change.

Verification

All run locally on this branch:

• cargo fmt --check (root and EQL workspaces): clean.
• RUSTFLAGS=-D warnings cargo clippy --workspace --tests, with and without --all-features: clean.
• mise x --env test -- cargo nextest run --workspace --all-features: 1525 passed, 1 skipped.
• mise run test:doc: 90 doc tests passed. mise run doc: clean.
• New Stack Encrypt tests: tokenizer cases (two-letter words at the start, middle and end; one-letter words ignored; 4-grams taking 2- and 3-letter words; separators; two-character words outside ASCII counted by characters; NFC), proptests (every short word is exactly one word token and no other word is; the option moves no trigram; the stored filter holds the query filter of any run of its whole words; the two domains never derive one hash), the round trip ("Ng" finds "Ng" and "Mike Ng", not "Nguyen"), the option-mismatch refusal, the one-letter refusal, a pinned PRF framing (a word token and a trigram of the same characters share no position), and a pin of the word token's positions.
• False-positive calibration (3000 random rows, 60 queries of each kind, k = 3, m = 256): a one-word query matched falsely at 0.00445 (801 of 180000 row checks), a one-trigram query at 0.00456 (820 of 179910), against 0.00430 predicted from the filters' fill; both set 2.98 bits on average. A five-letter control query matched falsely 0 times.
• EQL mise run test:encryption: 527 passed. mise run test:encryption:postgres against a fresh database on localhost:5400: 43 passed, including postgres_finds_two_letter_surnames.
• Go: wasm:guest:build, wasm:guest:build:eql and both deterministic builds, wasm:auth-guest:build; mise run go:test passed; mise run go:lint: 0 issues; go generate ./... changed nothing; mise run go:test:postgres passed.
• cargo metadata --locked in the root, EQL, protect-ffi and both guest workspaces: clean.
• cargo mutants --in-diff over stack-auth and stack-encrypt (the CI gate): 31 caught, 6 unviable, 1 missed on the first push (Tokens::zeroize replaced with a no-op). A test now pins that zeroizing clears both kinds of token, and a local rerun of that mutant reports it caught.

Related

Closes #1309

• Go SDK: stashgen keeps its own copy of engine rules (index options, index outputs, a second lowering) — ask the engine instead #1112: the Go SDK's index-option limit, which is why a native Go index=match field does not take the option.
• #1326: searchable symmetric encryption as a lower-leakage alternative to Bloom-filter match terms.
• match index: "Mike Ng" no longer finds "Mike Nguyen" with short words; cut query word tokens only when the query has no n-gram #1398 (follow-up): cut a query's word tokens only when it cuts no n-gram, so "Mike Ng" finds "Mike Nguyen" again. Query-side only; no stored filter moves.
• match index: "Ng." and "Ng-Smith" hold no word "ng"; add a character filter to the text pipeline #1399 (follow-up): a character filter, so "Ng." and "Ng-Smith" hold the word ng.

Review notes

• Design difference from Match: text shorter than the 3-gram length can't be stored in TextMatch or found by match #1309: the issue proposed a whole-text token for text shorter than a trigram. This PR uses a per-word token instead, so "Mike Ng" is found by "ng" as well as "Ng" alone. The cost is a narrowing for mixed queries: a query's two-letter words must be whole words of the row, so "Mike Ng" does not find "Mike Nguyen" (it did with the option off, through trigrams alone). The option's rustdoc and the changeset say so.
• Stored format: EQL v4 is unreleased, so the v4 match format simply includes short-word tokens from here. A v4 value written by an earlier build that holds a two-letter word lacks the word's bits, and a query naming that word does not find it until re-encrypted. Text with no two-letter word is unchanged.
• Deferred, deliberately: how a text-processing pipeline (tokenizer and filters) should be carried through EQL in general, so a column could choose its own, is an open question the owner has deferred; this PR only turns the one option on for the v4 match types.
• MatchOptions gains a public field, so a struct literal naming every field must name it. stack-encrypt 0.3.0 is unreleased (crates.io has 0.2.0), so this lands in the same release.
• The separator set is the standard tokenizer's, not Unicode whitespace: a tab, newline, full stop or hyphen does not end a word, as for Tokenizer::Standard, so "Ng." and "Ng-Smith" carry no ng token. Widening it, and the query-side narrowing above, are flagged in review threads as owner decisions; one alternative raised there is to cut a query's word tokens only when the query cuts no n-gram.

https://claude.ai/code/session_01N7bCqa3ZVJMcEE99RzVqEu

[changeset-bot]

🦋 Changeset detected

Latest commit: 93fd33e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@cipherstash/eql	Minor
stash	Patch
@cipherstash/stack-prisma	Patch
@cipherstash/basic-example	Patch
@cipherstash/e2e	Patch
@cipherstash/prisma-example	Patch
@cipherstash/stack-drizzle	Patch
@cipherstash/stack-supabase	Patch
@cipherstash/stack	Patch
@cipherstash/wizard	Patch
@cipherstash/bench	Patch
@cipherstash/test-kit	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI; Enforced overrides: Organization
• Review profile: CHILL
• Plan: Advanced
• Run ID: c08b48fe-92c6-4819-a9c2-fa20263567fd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Mutation testing (cargo-mutants, --in-diff, stack-auth + stack-encrypt)

caught	missed	unviable	timeout
39	0	6	0

Every mutant in the changed lines was caught by a test.

[coderdan]

Human review pass.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	🔄 Running since 2026-10-26T11:15:19.509839Z	5eaabb5	Draft marked ready
🔒 Security Review	✅ Completed 2026-10-26T11:19:42.735997Z	5eaabb5	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cipherstash-agent]

Review attestation — source: Claude Code /code-review medium followed by company-skills:cross-check-review-findings, run against head 5eaabb57.

Critical / blocking: none.

New findings: none. Reviewed stack-encrypt sem/mod.rs, sem/tokenize.rs, dynamic/{term,record,query,kind}.rs, plan/error.rs, the eql-bindings change and the new tests. I checked these against the head commit:

• Word tokens use their own PRF domain (stack-encrypt/sem/match/word/v1), and a_word_token_and_an_ngram_of_the_same_characters_share_no_position pins that.
• validate() refuses short_words beside the standard tokenizer and beside n-grams shorter than 3.
• The wire form writes short_words only when it is on, so existing plans keep their bytes.
• Empty-token handling in match_term and bloom_positions still skips the PRF call.
• QueryTokens::verify compares each kind of token against the same kind in the row.

Already covered by existing threads (no repeat posted):

• Query-side word tokens narrow substring queries, e.g. @@ 'Mike Ng' no longer finds "Mike Nguyen". Open, deferred to the owner (tokenize.rs ~L167).
• The separator set splits only on ,;:!, so "Ng.", "Ng-Smith" and newline-separated text get no word token (tokenize.rs ~L182).
• Plaintext short words on the error path (fixed in 5eaabb57e).
• Stale README and op.rs labels (fixed).
• The Go README "substring match" wording (fixed).
• The test that asserted only verify, not the filter (fixed).
• Length counted after case folding (doc fixed).
• The duplicated split logic (fixed with a shared words helper).

The two substantive threads, query-side narrowing and the separator set, are still open and need an owner decision before merge. They are a deliberate trade-off documented in the PR, not a defect I found independently.

---

Generated by Claude Code