feat: a null has an equality term, and a sealed null sits beside one

vitaminc-normalize 0.8 derives an equality term for a null: one term per
context, shared by every null, typed or plain. #1327 dropped it to keep
the old behaviour; this keeps it (#1298).

- stack-encrypt: `EqualityTerm` of a `None`, and `IndexSpec::Equality`
  on a null, now derive that term. No other index has one.
- Data plans: `"null": "sealed"` takes a field whose only index is
  equality, on the field or on the EQL type it names. The null is sealed
  as long as a value and stores the null term beside it, so the absent
  rows are one equality group, as any repeated value is, and the column
  holds no SQL NULL. An `eq` query whose input is a null derives that
  term (`query_term`, `record::query`); a null query of any other field
  or operator is the new `Reason::NullQuery`.
- eql-bindings: a type whose only term is equality's (`DateEq`,
  `IntegerEq`, ...) and its query twin take an `Option` of the plaintext,
  as storage-only types do, and the by-name dispatch encrypts, opens and
  queries them through it.
- Go: stashgen defaults to `null=sealed` for a fixed-width field whose
  only index is equality, and generates `EqNull`, the `eq` query of nil,
  through the new `gensupport.Field.NullTerm`.

Verified in PostgreSQL: `=` with the query of a null finds exactly the
rows of a DateEq column whose value was a sealed null.

Refs #1298.

Claude-Session: https://claude.ai/code/session_01XQN1QpaWVDGjHEg39KEJF3

Author: coderdan

.changeset/eql-bindings-optional-storage-only.md

@@ -2,4 +2,4 @@
 "@cipherstash/eql": patch
 ---
 
-`eql-bindings` (the Rust crate released with this package): a storage-only EQL v4 type (`Date`, `Integer`, `Text`, and the other types with no term) now also implements `EncryptFrom` and `DecryptInto` for an `Option` of its plaintext. A `None` seals as a typed null as long as a value (vitaminc 0.8), so an optional column does not show which rows are absent, and opens as `None`. A value seals exactly as before. The by-name dispatch (`encryption::targets`) reads and opens these types through the `Option`, so a null frame encrypts and a stored null decrypts to a null; a type with a term still refuses a null. eql-bindings' own `Date`, `Timestamp` and `Decimal` plaintexts declare their fixed width to vitaminc, so their `None` is padded too. The SQL bundle and the TypeScript types do not change.
+`eql-bindings` (the Rust crate released with this package): an EQL v4 type whose terms a null has, a storage-only type (`Date`, `Integer`, `Text`, ...) or one whose only term is equality's (`DateEq`, `IntegerEq`, `TextEq`, ...), now also implements `EncryptFrom` and `DecryptInto` for an `Option` of its plaintext, and its query twin `EncryptFrom` for one. A `None` seals as a typed null as long as a value (vitaminc 0.8), so an optional column does not show which rows are absent, and opens as `None`; its `hm` is the one term every null shares, and the query of a `None` finds the stored nulls with `=`. A value seals exactly as before. The by-name dispatch (`encryption::targets`) reads, opens and queries these types through the `Option`; a type with an order or match term still refuses a null. eql-bindings' own `Date`, `Timestamp` and `Decimal` plaintexts declare their fixed width to vitaminc, so their `None` is padded too. The SQL bundle and the TypeScript types do not change.

languages/golang/CHANGELOG.md

@@ -20,12 +20,15 @@ listed under **Breaking**. Releases are tagged `languages/golang/vX.Y.Z`.
   so the column does not show which rows have no value; `null=sql` stores
   nothing, so the field's columns are SQL `NULL` and its member in the
   generated type is a pointer. `stashgen` picks `null=sealed` for a field
-  with no index whose type has a fixed width (a bool, a number,
-  `encrypt.Date`, `time.Time`, `encrypt.Decimal`, the 128-bit integers),
-  and asks for `null=sql` to be written on any other: a null has no index
-  term yet, and text shows its length. A stored null read into a field that
-  is not a pointer is an error from `Decrypt`. `-model` does not map a
-  `null=sql` field yet.
+  whose type has a fixed width (a bool, a number, `encrypt.Date`,
+  `time.Time`, `encrypt.Decimal`, the 128-bit integers) and whose only
+  index, if any, is equality, and asks for `null=sql` to be written on any
+  other: a null has an equality term and no other, and text shows its
+  length. Beside equality a sealed null stores the term every nil of the
+  field shares, and a generated `EqNull` method derives its `eq` query,
+  which finds the rows whose value is absent. A stored null read into a
+  field that is not a pointer is an error from `Decrypt`. `-model` does not
+  map a `null=sql` field yet.
 
 ### Breaking
 

languages/golang/cmd/stashgen/README.md

@@ -113,11 +113,12 @@ A sealed field declared over a pointer to one scalar (`*encrypt.Date`, `*int32`,
 
 | `null=` | Stored for `nil` | What a reader of the column sees |
 |---|---|---|
-| `sealed` | a sealed null, as long as a sealed value of the field's type | nothing: an absent value looks like any other |
+| `sealed` | a sealed null, as long as a sealed value of the field's type, and beside an equality index the one term every null of the field shares | nothing in the ciphertext; in an equality column, that the absent rows are one group, as rows holding one repeated value are |
 | `sql` | nothing: the field's columns are `NULL`, and its member in the generated type is `nil` | which rows have no value |
 
-A sealed null hides that the value is absent only where its length matches a value's and nothing beside it shows it, so it is the default for a field with no index whose type has a fixed width: a `bool`, a number, `encrypt.Date`, `time.Time`, `encrypt.Decimal`, `encrypt.Int128` and `encrypt.Uint128`, sealed with `encrypt` or into an EQL type with no index (`Date`, `Integer`, `Boolean`, ...).
-A field with an index, and a field of text or bytes, has to write `null=sql`: a null has no index term yet, and text shows its length, so a sealed null would hide nothing there, and `stashgen` will not choose a visible `NULL` for you.
+A sealed null hides that the value is absent only where its length matches a value's and no index beside it shows it, so it is the default for a field whose type has a fixed width (a `bool`, a number, `encrypt.Date`, `time.Time`, `encrypt.Decimal`, `encrypt.Int128` and `encrypt.Uint128`) and whose only index, if it has one, is `equality`: sealed with `encrypt`, `encrypt,index=equality`, or into an EQL type with no index or with equality alone (`Date`, `DateEq`, `Integer`, `IntegerEq`, ...).
+A null has an equality term, so its rows are one equality group and a query finds them: the generated `EqNull` derives the `eq` query of `nil`, which matches every row whose value is absent.
+A null has no order or match term, so a field with any other index, and a field of text or bytes, has to write `null=sql`: an order or match column would show which values are absent, and text shows its length, and `stashgen` will not choose a visible `NULL` for you.
 `null=sealed` on such a field stops `stashgen` with the reason.
 
 A `null=sql` field's member in the generated type is a pointer (`*eql.DateEq`, `*EncryptedUserScore`), `nil` when the value was; a `null=sealed` field's is the same type as a non-optional field's, and its column is never `NULL`.

languages/golang/encrypt/gensupport/declaration.go

@@ -86,8 +86,9 @@ type NullMode string
 const (
 	// NullSealed seals nil like a value, as a null as long as a value of
 	// the field's kind, so the stored record does not show which rows are
-	// absent. Decrypt gives nil. The field derives no index, and its kind
-	// has a fixed width.
+	// absent. Decrypt gives nil. An equality index stores the term every nil
+	// shares, which [Field.NullTerm] queries; the field has no other index,
+	// and its kind has a fixed width.
 	NullSealed NullMode = "sealed"
 	// NullSQL stores nothing for nil: the field's columns hold SQL NULL,
 	// which shows that the value is absent. Decrypt gives nil.
@@ -170,7 +171,8 @@ func (d Declaration) EncryptInto(name string, kind Kind, eqlType string) Declara
 
 // Null says how the named field, declared over a pointer, stores nil. The
 // engine decides whether the field can, when the declaration is first used:
-// a sealed null needs a field with no index, of a kind of fixed width.
+// a sealed null needs a field whose only index is equality, of a kind of
+// fixed width.
 func (d Declaration) Null(name string, mode NullMode) Declaration {
 	if d.err != nil {
 		return d

languages/golang/encrypt/gensupport/field.go

@@ -100,6 +100,18 @@ func (f Field[T]) Range(ctx context.Context, c *encrypt.Cipher, operator string,
 	return terms[0], terms[1], nil
 }
 
+// NullTerm derives an exact operator's term for nil: for eq on a field that
+// stores nil as a sealed null, the term every such nil shares, so it finds
+// the rows whose value is absent. The engine refuses it for any other field
+// or operator: nothing else stores a null's term.
+func (f Field[T]) NullTerm(ctx context.Context, c *encrypt.Cipher, operator string) ([]byte, error) {
+	terms, err := f.query(ctx, c, operator, nil, 1)
+	if err != nil {
+		return nil, err
+	}
+	return terms[0], nil
+}
+
 // query derives an exact operator's terms, and refuses an answer of another
 // shape than the generated method's: a probabilistic term has a query type
 // of its own (Matches). Each operand is converted to the field's kind as a

languages/golang/encrypt/guest/tests/native_ops.rs

@@ -352,15 +352,16 @@ fn unsupported_term_inputs_are_encoding_errors() {
     let ctx = encode(s("f"));
 
     // Bools have no equality domain and no order term; match is text-only;
-    // containers have no term semantics; kinds outside the table and empty
-    // contexts are rejected.
+    // containers have no term semantics; a null has an equality term
+    // (#1298) and no order term; kinds outside the table and empty contexts
+    // are rejected.
     for (value, kind) in [
         (Tree::Bool(true), TERM_EQUALITY),
         (Tree::Bool(true), TERM_ORE),
         (Tree::Bool(false), TERM_OPE),
         (Tree::UInt32(1), TERM_MATCH),
         (Tree::Array(vec![]), TERM_ORE),
-        (Tree::Null, TERM_EQUALITY),
+        (Tree::Null, TERM_ORE),
         (Tree::UInt32(1), 99),
     ] {
         assert_eq!(

languages/golang/encrypt/internal/testusers/people.go

@@ -6,18 +6,20 @@ import "github.com/cipherstash/stack/languages/golang/encrypt"
 //go:generate go run github.com/cipherstash/stack/languages/golang/cmd/stashgen -type Birthday -name Birthday
 
 // Person has optional fields, each declared over a pointer, in both ways a
-// field stores nil (#1298): DOB and Age as a sealed null, as long as a
-// value, which the generator picks for a type of fixed width with no index;
-// Born, Score and Nick as SQL NULL, which an indexed field and text have to
-// ask for.
+// field stores nil (#1298). DOB, Born and Age store it as a sealed null, as
+// long as a value, which the generator picks for a type of fixed width whose
+// only index is equality: Born's and Age's equality terms of nil are the one
+// term every nil shares, which EqNull queries. Joined, Score and Nick store
+// SQL NULL, which an order index and text have to ask for.
 type Person struct {
-	_     struct{}      `stash:"context=people"`
-	ID    int64         `stash:"id,passthrough"`
-	DOB   *encrypt.Date `stash:"dob,encrypt_into=Date"`
-	Born  *encrypt.Date `stash:"born,encrypt_into=DateEq,null=sql"`
-	Age   *int32        `stash:"age,encrypt"`
-	Score *int64        `stash:"score,encrypt,index=ore,null=sql"`
-	Nick  *string       `stash:"nick,encrypt,null=sql"`
+	_      struct{}      `stash:"context=people"`
+	ID     int64         `stash:"id,passthrough"`
+	DOB    *encrypt.Date `stash:"dob,encrypt_into=Date"`
+	Born   *encrypt.Date `stash:"born,encrypt_into=DateEq"`
+	Joined *encrypt.Date `stash:"joined,encrypt_into=DateEq,null=sql"`
+	Age    *int32        `stash:"age,encrypt,index=equality"`
+	Score  *int64        `stash:"score,encrypt,index=ore,null=sql"`
+	Nick   *string       `stash:"nick,encrypt,null=sql"`
 }
 
 // Birthday reads Person's dob column into a field that is not a pointer,