Skip to content

Commit 56260ac

Browse files
committed
feat(golang): declarations from a policy, and protosource
Add encrypt/policy, stashgen.Generate and encrypt/policy/protosource: the path for a type that cannot carry stash tags, such as a protobuf message whose field options hold its data categories. A Source gives the facts about each field, the first rule that matches a field decides it, and Generate writes the same file the tag path writes, into a package of the user's own, with pointer-taking functions. A decision spells itself in the tag grammar (Outcome.Tag returns what the field's stash tag would say) and Generate parses it with the tag parser, so the two ways in share one grammar, one reader and one emitter, which is what SDK principle 6 asks of a second way in. That also keeps policy free of an import of stashgen, which Generate needs to import; the index values are policy.Equality, policy.Match() and friends rather than the plan's encrypt.Equality, because package encrypt does not exist in this tree and policy must not import a generator package. Identity joins the declaration as a field a policy alone sets; the tag grammar has no word for it. protosource reads the descriptor through protoreflect, records each extension set on a field's options under the extension's full name, and derives the Go field name the way protoc-gen-go does; Generate matches facts to struct fields through the protobuf tag's name= first and the Go name second. Its fixture is the plan's protoc-gen-go output, renamed. This adds google.golang.org/protobuf to the module. The policy test asserts that the policy path and the tag path write the same body for the same declaration, so the one emitter stays one. Claude-Session: https://claude.ai/code/session_01V3WFXwax4J3uecpFEJ6yHc
1 parent 5487252 commit 56260ac

21 files changed

Lines changed: 1807 additions & 44 deletions

File tree

‎languages/golang/cmd/stashgen/README.md‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,37 @@ The error names the type and the field, and never a value.
138138
- an embedded struct from another package with no tag;
139139
- a struct whose every field is left out.
140140

141+
## Declarations from a policy
142+
143+
A type that a schema generates, such as a protobuf message, cannot carry tags.
144+
For such a type, rules decide how each field is encrypted from what the schema says about it, and `stashgen.Generate` writes the same file from the rules.
145+
The rules live in `github.com/cipherstash/stack/languages/golang/encrypt/policy`; `encrypt/policy/protosource` reads a protobuf message's fields and their options.
146+
147+
```go
148+
var category = policy.Key("classification.data_categories")
149+
150+
var Individuals = policy.ForMessage(&pb.Individual{}, policy.Context("individuals"),
151+
policy.FirstOf(
152+
policy.When(policy.Field("id"), policy.Passthrough()),
153+
policy.When(category.Under("user.government_id"), policy.EncryptInto("TextEq")),
154+
policy.When(category.Under("user.contact.email"), policy.EncryptIndex(policy.Equality, policy.Match())),
155+
policy.When(category.Under("user"), policy.Encrypt()),
156+
),
157+
)
158+
159+
//go:generate go run ../cmd/genencrypt
160+
func main() {
161+
err := stashgen.Generate(protosource.New(), rules.Individuals,
162+
stashgen.Output("../individuals/individual_stash.go"))
163+
...
164+
}
165+
```
166+
167+
The first rule that matches a field decides it.
168+
Every field needs a decision: a field no rule decides stops the generator with the field's name and its annotations.
169+
`Name` sets the column name, and `Identity` keeps the field's context when its column is renamed.
170+
The generated file goes in a package of your own, and the functions take and return pointers to the message.
171+
141172
## Printing
142173

143174
A generated type hides its sealed fields when a program prints or logs it.
@@ -149,4 +180,4 @@ No warning, error or log line holds a plaintext value.
149180

150181
The generator is built and tested against a static stand-in for the engine.
151182
`stashgen.GuestEngine`, which runs the WASI guest the SDK embeds, is not wired yet, so the command stops with `ErrEngineUnavailable` until it is.
152-
The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`.
183+
The library, `github.com/cipherstash/stack/languages/golang/stashgen`, takes any `Engine`; `stashgen.Generate` takes one with `WithEngine`.
Lines changed: 353 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
// Package policy decides how a type that cannot carry stash tags is
2+
// encrypted, from what its schema says about each field.
3+
//
4+
// Tags are the way to declare a type that you write. A policy is for a type
5+
// that a schema generates, such as a protobuf message whose field options hold
6+
// the field's data categories. A [Source] gives the facts about each field; a
7+
// rule has a [Matcher] and a [Decision]; the first rule that matches a field
8+
// decides it. stashgen.Generate runs the rules and writes the same generated
9+
// file that it writes from tags. Only the generate program runs a policy.
10+
//
11+
// Every field of the message needs a decision. A field that no rule decides
12+
// stops the generator with the field's name and its annotations. A policy has
13+
// an [Otherwise] rule only when its author writes one.
14+
package policy
15+
16+
import (
17+
"fmt"
18+
"strings"
19+
)
20+
21+
// Fact is what a source knows about one field: its name, its Go name, its
22+
// kind and its annotations. A policy reads facts and never learns where they
23+
// came from.
24+
type Fact struct {
25+
// Message is the full name of the message, for errors.
26+
Message string
27+
// Name is the field's name in the schema, such as the proto field name.
28+
// It is the column name unless a rule gives another with [Name].
29+
Name string
30+
// GoName is the field of the Go struct the schema generated.
31+
GoName string
32+
// Kind is the field's kind in the schema's own words, such as "string" or
33+
// "int64". Rules may match on it.
34+
Kind string
35+
// Annotations are what the schema says about the field, such as its
36+
// data categories.
37+
Annotations []Annotation
38+
}
39+
40+
// Annotation is one annotation on a field: a key and its values. For a
41+
// protobuf field option the key is the option's full name.
42+
type Annotation struct {
43+
Key string
44+
Values []string
45+
}
46+
47+
// Values returns every value under key, in order.
48+
func (f Fact) Values(key string) []string {
49+
var out []string
50+
for _, a := range f.Annotations {
51+
if a.Key == key {
52+
out = append(out, a.Values...)
53+
}
54+
}
55+
return out
56+
}
57+
58+
// String spells the field and its annotations, for the error that names a
59+
// field no rule decides.
60+
func (f Fact) String() string {
61+
var b strings.Builder
62+
fmt.Fprintf(&b, "%s.%s (%s", f.Message, f.Name, f.Kind)
63+
for _, a := range f.Annotations {
64+
fmt.Fprintf(&b, "; %s = %s", a.Key, strings.Join(a.Values, ", "))
65+
}
66+
b.WriteString(")")
67+
return b.String()
68+
}
69+
70+
// Source gives the facts about each field of a message.
71+
type Source interface {
72+
// Facts returns one fact for each field of the message, in declared
73+
// order. The message is the value given to [ForMessage].
74+
Facts(message any) ([]Fact, error)
75+
}
76+
77+
// SourceFunc is a Source made from a function.
78+
type SourceFunc func(message any) ([]Fact, error)
79+
80+
// Facts calls the function.
81+
func (s SourceFunc) Facts(message any) ([]Fact, error) { return s(message) }
82+
83+
// Matcher selects the fields a rule applies to.
84+
type Matcher interface {
85+
Match(Fact) bool
86+
}
87+
88+
// MatcherFunc is a Matcher made from a function.
89+
type MatcherFunc func(Fact) bool
90+
91+
// Match calls the function.
92+
func (m MatcherFunc) Match(f Fact) bool { return m(f) }
93+
94+
// Field matches the field with this schema name.
95+
func Field(name string) Matcher {
96+
return MatcherFunc(func(f Fact) bool { return f.Name == name })
97+
}
98+
99+
// Key is an annotation key, such as the full name of a protobuf field option.
100+
type Key string
101+
102+
// Under matches a field with a value under key that is prefix, or starts
103+
// with prefix and a dot: Key("c").Under("user") matches "user" and
104+
// "user.contact.email", and not "username".
105+
func (k Key) Under(prefix string) Matcher {
106+
return MatcherFunc(func(f Fact) bool {
107+
for _, v := range f.Values(string(k)) {
108+
if v == prefix || strings.HasPrefix(v, prefix+".") {
109+
return true
110+
}
111+
}
112+
return false
113+
})
114+
}
115+
116+
// Is matches a field with a value under key equal to value.
117+
func (k Key) Is(value string) Matcher {
118+
return MatcherFunc(func(f Fact) bool {
119+
for _, v := range f.Values(string(k)) {
120+
if v == value {
121+
return true
122+
}
123+
}
124+
return false
125+
})
126+
}
127+
128+
// Decision is what a rule decides for a field: one of the tag verbs, or a
129+
// refusal. A decision spells itself in the tag grammar, so the generator
130+
// reads a policy's decision through the same parser as a tag.
131+
type Decision struct {
132+
verb string
133+
indexes []IndexSpec
134+
eqlType string
135+
fail bool
136+
reason string
137+
}
138+
139+
// Encrypt seals the field with no index.
140+
func Encrypt() Decision { return Decision{verb: "encrypt"} }
141+
142+
// EncryptIndex seals the field and derives each index beside it.
143+
func EncryptIndex(indexes ...IndexSpec) Decision {
144+
return Decision{verb: "encrypt,index", indexes: indexes}
145+
}
146+
147+
// Index derives the indexes alone, with no ciphertext.
148+
func Index(indexes ...IndexSpec) Decision { return Decision{verb: "index", indexes: indexes} }
149+
150+
// EncryptInto seals the field into one EQL value of this type.
151+
func EncryptInto(eqlType string) Decision { return Decision{verb: "encrypt_into", eqlType: eqlType} }
152+
153+
// Passthrough stores the field as it is.
154+
func Passthrough() Decision { return Decision{verb: "passthrough"} }
155+
156+
// Omit leaves the field out.
157+
func Omit() Decision { return Decision{verb: "-"} }
158+
159+
// Fail refuses the field: the generator stops with the reason.
160+
func Fail(reason string) Decision { return Decision{fail: true, reason: reason} }
161+
162+
// IndexName is one of the tag words for an index: equality, match, ore, ope
163+
// or json.
164+
type IndexName string
165+
166+
// IndexOption is one option of an index, as the tag writes it in parentheses.
167+
type IndexOption struct {
168+
Key string
169+
Value string
170+
}
171+
172+
// IndexSpec is one index with its options, as a rule names it: the engine's
173+
// own word for the data form of an index.
174+
type IndexSpec struct {
175+
Name IndexName
176+
Options []IndexOption
177+
}
178+
179+
// The indexes that take no options.
180+
var (
181+
Equality = IndexSpec{Name: "equality"}
182+
Ore = IndexSpec{Name: "ore"}
183+
Ope = IndexSpec{Name: "ope"}
184+
)
185+
186+
// Match is the match index, with its options.
187+
func Match(options ...IndexOption) IndexSpec { return IndexSpec{Name: "match", Options: options} }
188+
189+
// JSON is the json index, with its options.
190+
func JSON(options ...IndexOption) IndexSpec { return IndexSpec{Name: "json", Options: options} }
191+
192+
// String spells the index as a tag does: match or match(k=3).
193+
func (i IndexSpec) String() string {
194+
if len(i.Options) == 0 {
195+
return string(i.Name)
196+
}
197+
opts := make([]string, len(i.Options))
198+
for n, o := range i.Options {
199+
opts[n] = o.Key
200+
if o.Value != "" {
201+
opts[n] += "=" + o.Value
202+
}
203+
}
204+
return string(i.Name) + "(" + strings.Join(opts, ",") + ")"
205+
}
206+
207+
// Outcome is a decision applied to one field: the decision, and the name
208+
// and identity a rule's options set.
209+
type Outcome struct {
210+
Decision Decision
211+
// Name is the column name, or "" for the field's schema name.
212+
Name string
213+
// Identity is the field's part of its context, or "" for the column
214+
// name. A field whose column is renamed keeps its identity, so data
215+
// written before the change still decrypts.
216+
Identity string
217+
}
218+
219+
// Tag spells the outcome for a field in the tag grammar: what the field's
220+
// stash tag would say. ok is false for a refusal, which [Outcome.Reason]
221+
// explains, and for the zero Outcome.
222+
func (o Outcome) Tag(name string) (tag string, ok bool) {
223+
d := o.Decision
224+
if d.fail || d.verb == "" {
225+
return "", false
226+
}
227+
if d.verb == "-" {
228+
return "-", true
229+
}
230+
if o.Name != "" {
231+
name = o.Name
232+
}
233+
switch d.verb {
234+
case "encrypt_into":
235+
return name + ",encrypt_into=" + d.eqlType, true
236+
case "encrypt,index", "index":
237+
names := make([]string, len(d.indexes))
238+
for i, idx := range d.indexes {
239+
names[i] = idx.String()
240+
}
241+
return name + "," + d.verb + "=" + strings.Join(names, ";"), true
242+
}
243+
return name + "," + d.verb, true
244+
}
245+
246+
// Reason is the reason of a Fail decision, or "".
247+
func (o Outcome) Reason() string {
248+
if !o.Decision.fail {
249+
return ""
250+
}
251+
return o.Decision.reason
252+
}
253+
254+
// Option modifies a rule's outcome.
255+
type Option func(*Outcome)
256+
257+
// Name sets the field's column name.
258+
func Name(column string) Option { return func(o *Outcome) { o.Name = column } }
259+
260+
// Identity keeps the field's context when its column gets a new name.
261+
func Identity(identity string) Option { return func(o *Outcome) { o.Identity = identity } }
262+
263+
// Rule decides a field, or reports that it does not match.
264+
type Rule interface {
265+
Decide(Fact) (Outcome, bool)
266+
}
267+
268+
// When is a rule: the fields the matcher selects get the decision.
269+
func When(m Matcher, d Decision, options ...Option) Rule {
270+
return rule{m: m, outcome: outcome(d, options)}
271+
}
272+
273+
// Otherwise decides every field that no earlier rule matched.
274+
func Otherwise(d Decision, options ...Option) Rule {
275+
return rule{m: MatcherFunc(func(Fact) bool { return true }), outcome: outcome(d, options)}
276+
}
277+
278+
func outcome(d Decision, options []Option) Outcome {
279+
o := Outcome{Decision: d}
280+
for _, opt := range options {
281+
opt(&o)
282+
}
283+
return o
284+
}
285+
286+
type rule struct {
287+
m Matcher
288+
outcome Outcome
289+
}
290+
291+
func (r rule) Decide(f Fact) (Outcome, bool) {
292+
if !r.m.Match(f) {
293+
return Outcome{}, false
294+
}
295+
return r.outcome, true
296+
}
297+
298+
// Rules are rules tried in order; the first that matches decides.
299+
type Rules []Rule
300+
301+
// FirstOf returns the rules, tried in order.
302+
func FirstOf(rules ...Rule) Rules { return Rules(rules) }
303+
304+
// Decide tries each rule in order.
305+
func (r Rules) Decide(f Fact) (Outcome, bool) {
306+
for _, rule := range r {
307+
if rule == nil {
308+
continue
309+
}
310+
if o, ok := rule.Decide(f); ok {
311+
return o, true
312+
}
313+
}
314+
return Outcome{}, false
315+
}
316+
317+
// OrElse returns the rules, then other for the fields they do not match.
318+
func (r Rules) OrElse(other Rule) Rules {
319+
return append(append(Rules{}, r...), other)
320+
}
321+
322+
// Context is the context of every field of a message, as the `context=` tag
323+
// gives it for a struct.
324+
type Context string
325+
326+
// Message is one message with its context and its rules, as [ForMessage]
327+
// builds it and stashgen.Generate reads it.
328+
type Message struct {
329+
message any
330+
context Context
331+
rules Rule
332+
}
333+
334+
// ForMessage binds a message to its context and the rules that decide its
335+
// fields. The message is a value of the generated Go type, such as
336+
// &pb.Individual{}; the source reads its schema and the generator its type.
337+
func ForMessage(message any, context Context, rules Rule) Message {
338+
return Message{message: message, context: context, rules: rules}
339+
}
340+
341+
// Message returns the message value given to ForMessage.
342+
func (m Message) Message() any { return m.message }
343+
344+
// Context returns the message's context.
345+
func (m Message) Context() string { return string(m.context) }
346+
347+
// Decide runs the rules on one field.
348+
func (m Message) Decide(f Fact) (Outcome, bool) {
349+
if m.rules == nil {
350+
return Outcome{}, false
351+
}
352+
return m.rules.Decide(f)
353+
}

0 commit comments

Comments
 (0)