Skip to content

feat(ess): add study trial fields to work units - #1683

Draft
VLukiantsev wants to merge 1 commit into
feat/hpo-batch-create-work-units-b558652203from
feat/hpo-work-unit-trial-fields-b558652203
Draft

VLukiantsev wants to merge 1 commit into
feat/hpo-batch-create-work-units-b558652203from
feat/hpo-work-unit-trial-fields-b558652203

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add the study trial fields WorkUnit.parameters (field 11, google.protobuf.Struct, IMMUTABLE) and WorkUnit.trial_id (field 12, IMMUTABLE, set by the controller). They are persisted through migration 020 on both backends. BatchCreateWorkUnits requires both fields on STUDY experiments and validates the parameters against the study config; unary CreateWorkUnit rejects them. This is layer 3 of a 4-PR stack.

Motivation & Context

Each trial carries the optimization engine's suggested parameter values and the engine's trial ID. The actuator will inject the parameters into containers in the follow-up dispatch task. Validating them against StudyConfig at the API boundary keeps malformed or out-of-range suggestions out of storage.

Sub-slice: HPO M1 S1.3. First production caller: the upcoming controller trial-dispatch task.

Key Changes

  • Protos: parameters = 11 and trial_id = 12 on WorkUnit, with struct_proto added to BUILD deps.
  • Migration 020 (same number on both backends; TestSchemaVersionParity passes):
    • Spanner: WorkUnits.Parameters BYTES(MAX) (proto-marshaled Struct) and TrialId STRING(MAX).
    • Postgres: BYTEA and TEXT.
    • All columns are nullable, with matching down migrations.
    • The proto bundle is not changed: Struct stays unbundled. The bundled WorkUnit descriptor, which backs the WorkUnitType column and Operations.Response, keeps the new fields as unknown fields.
  • Storage: ess-schema.sql, workunitschema (selectable, not filterable) and converters on both backends handle NULL, empty and corrupt bytes. Every read path, including the paused-task joins, returns the new columns on both backends.
  • Validation (internal/validator/studytrial.go):
    • trial_id is required, printable, valid UTF-8, at most 128 bytes, and unique within the batch.
    • parameters must have exactly the study's parameter keys.
    • Per-type rules: DOUBLE is finite and in range; INTEGER is integral and in range; DISCRETE is a feasible value; CATEGORICAL is a declared category.
    • Both fields are forbidden on non-study work units. Unary CreateWorkUnit rejects them with "set only through BatchCreateWorkUnits on a STUDY experiment".
  • Docs: ESS README (field contract, no cross-batch trial_id dedup) and the work unit lifecycle doc.

Assumptions

  • Store the parameters as marshaled bytes, not JSON/JSONB — keeps backend parity and avoids a proto-bundle change — the values can't be queried in SQL; analytics in a later milestone may need a JSON column.
  • A fresh-schema Spanner CREATE PROTO BUNDLE with a WorkUnit descriptor that references google.protobuf.Struct behaves like the existing Timestamp references — it was verified only on an upgraded live database and the emulator — if a fresh database rejects it, the bundle needs the Struct types or a schema split.
  • trial_id is unique only within one batch — the controller owns cross-batch dedup — duplicates across ticks are possible if the controller misbehaves.
  • Gate matrix: same as layer 2; live on Spanner, Postgres covered by repository and integration tests — the Postgres DDL was not live-verified — schema parity is enforced by TestSchemaVersionParity and the repository tests.

Verification & Testing

  • Commands Executed: ./.githooks/pre-commit; bazel test on the ESS server, internal/workunit, internal/validator, internal/storage/... and internal/experiment/... (76/76 pass); run_affected_tests.py at the stack tip (155/155 pass).
  • Coverage & Test Status:
    • Validator: NaN and ±Inf; string, list, null, bool and unset kinds; multibyte keys, invalid-UTF-8 keys and keys over 128 bytes; case-different keys; empty or missing params; bounds; invalid-UTF-8 trial IDs; duplicate trial IDs.
    • Converter round trips on both backends (NULL, empty Struct, corrupt bytes, marshal error).
    • Repository read-back on the Spanner emulator and Postgres.
    • 100% coverage of new validator and storage lines.
  • Live (GKE dev cluster, Spanner, stable-2026b profile, stack tip after the sync to main):
    • Migration 020 applied: ddl describe shows Parameters BYTES(MAX), TrialId STRING(MAX); SchemaMigrations is version 20, not dirty.
    • The controller's batch persisted trialId/parameters, visible in GetWorkUnit, ListWorkUnits and the Spanner rows.
    • Invalid params were rejected with InvalidArgument and nothing was created: missing key, unknown key, out of range, duplicate trial_id.

Related Issues / Bugs

BUG=b/558652203

TAG=agy
CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae

@VLukiantsev
VLukiantsev added this pull request to stack #1685 November 13, 2026 10:42

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces study trial fields (parameters and trial_id) to the WorkUnit message for STUDY experiments. It adds database columns and migrations for both PostgreSQL and Spanner, implements validation logic to verify trial parameters against the study configuration, and updates the RPC handlers and tests accordingly. The review feedback suggests a minor optimization in the batch validation logic to avoid storing empty trial IDs in the tracking map.

Comment thread internal/validator/requests.go Resolved
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-work-unit-trial-fields-b558652203 branch from fcad1fb to 0c5b863 Compare November 17, 2026 12:08
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-work-unit-trial-fields-b558652203 branch from 0c5b863 to 39165e8 Compare November 17, 2026 12:59
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces study trial fields (parameters and trial_id) to WorkUnit and implements validation, storage, and retrieval of these fields across both Postgres and Spanner backends. The BatchCreateWorkUnits RPC now validates trial parameters against the study configuration, while CreateWorkUnit rejects them. Feedback on the changes highlights two validation issues in internal/validator/studytrial.go: a potential validation bypass when duplicate parameter specs exist in StudyConfig, and a potential nil-pointer panic if a parameter value is nil.

Comment thread internal/validator/studytrial.go Resolved
Comment thread internal/validator/studytrial.go Resolved
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-work-unit-trial-fields-b558652203 branch from 39165e8 to d1b2471 Compare November 17, 2026 13:28
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for study trial fields (parameters and trial_id) on WorkUnit objects. It updates the protobuf definitions, implements validation logic against StudyConfig parameters, adds database schema migrations and storage support for both PostgreSQL and Spanner, and updates the BatchCreateWorkUnits RPC to handle these fields. The review feedback focuses on defensive programming, suggesting the addition of nil checks for WorkUnit and ParameterSpec pointers in the validation and database mapping layers to prevent potential nil pointer dereference panics.

return fmt.Errorf("work_unit.parameters is required on work units of STUDY experiments: %w", grpcerr.ErrInvalidArgument)
}
fields := params.GetFields()
for _, spec := range specs {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: specs can potentially contain nil elements if the study configuration is malformed or not fully validated. Adding a nil check for spec prevents a potential nil pointer dereference panic when calling spec.GetParameterId().

	for _, spec := range specs {
		if spec == nil {
			continue
		}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No panic is possible here. specs comes from the experiment's stored StudyConfig, which ESS validated at CreateExperiment time and which is immutable. Proto unmarshalling never produces nil elements in a repeated message field. Even for a nil *ParameterSpec, the generated getters (GetParameterId(), GetParameterValueSpec()) are nil-receiver-safe and return zero values. Such a spec would just never match a key, and the value would fail as a type mismatch (INVALID_ARGUMENT). Adding explicit nil branches would add dead code without changing behaviour.

Comment on lines +88 to +89
func validateParameterValue(spec *essmsgpb.ParameterSpec, value *structpb.Value) error {
switch s := spec.GetParameterValueSpec().(type) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: Ensure spec is not nil before accessing its fields or calling GetParameterValueSpec() to prevent a potential nil pointer dereference panic.

func validateParameterValue(spec *essmsgpb.ParameterSpec, value *structpb.Value) error {
	if spec == nil {
		return fmt.Errorf("parameter spec is nil: %w", grpcerr.ErrInvalidArgument)
	}
	switch s := spec.GetParameterValueSpec().(type) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No panic is possible here. specs comes from the experiment's stored StudyConfig, which ESS validated at CreateExperiment time and which is immutable. Proto unmarshalling never produces nil elements in a repeated message field. Even for a nil *ParameterSpec, the generated getters (GetParameterId(), GetParameterValueSpec()) are nil-receiver-safe and return zero values. Such a spec would just never match a key, and the value would fail as a type mismatch (INVALID_ARGUMENT). Adding explicit nil branches would add dead code without changing behaviour.

// message is deterministic at linear cost.
func firstUnknownKey(fields map[string]*structpb.Value, specs []*essmsgpb.ParameterSpec) (string, bool) {
known := make(map[string]bool, len(specs))
for _, spec := range specs {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: Guard against nil elements in specs to prevent a potential nil pointer dereference panic when calling spec.GetParameterId().

	for _, spec := range specs {
		if spec == nil {
			continue
		}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No panic is possible here. specs comes from the experiment's stored StudyConfig, which ESS validated at CreateExperiment time and which is immutable. Proto unmarshalling never produces nil elements in a repeated message field. Even for a nil *ParameterSpec, the generated getters (GetParameterId(), GetParameterValueSpec()) are nil-receiver-safe and return zero values. Such a spec would just never match a key, and the value would fail as a type mismatch (INVALID_ARGUMENT). Adding explicit nil branches would add dead code without changing behaviour.

Comment on lines +209 to +210
func setTrialFields(wu *essmsgpb.WorkUnit, parameters []byte, trialID *string) error {
if parameters != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: Add a nil check for wu to prevent a potential nil pointer dereference panic if a nil work unit is passed to setTrialFields.

func setTrialFields(wu *essmsgpb.WorkUnit, parameters []byte, trialID *string) error {
	if wu == nil {
		return fmt.Errorf("work unit is nil")
	}
	if parameters != nil {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not adding a nil check: this is a private converter helper, and a nil work unit can't reach it.

  • Row-to-proto converters (WorkUnitRowToProto and the paused-task converter): they call it on a WorkUnit they have just allocated.
  • WorkUnitProtoToRow: it first validates wu.GetCurrentWorkUnitExecution().GetName(). The getters are nil-safe, so a nil work unit fails that validation with an error before trialFieldsToColumns is reached.

A nil check here would be dead code.

Comment on lines +225 to +226
func trialFieldsToColumns(wu *essmsgpb.WorkUnit) ([]byte, *string, error) {
parameters, err := MarshalNullableProto(wu.GetParameters())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: Add a nil check for wu to prevent a potential nil pointer dereference panic if a nil work unit is passed to trialFieldsToColumns.

Suggested change
func trialFieldsToColumns(wu *essmsgpb.WorkUnit) ([]byte, *string, error) {
parameters, err := MarshalNullableProto(wu.GetParameters())
func trialFieldsToColumns(wu *essmsgpb.WorkUnit) ([]byte, *string, error) {
if wu == nil {
return nil, nil, fmt.Errorf("work unit is nil")
}
parameters, err := MarshalNullableProto(wu.GetParameters())

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not adding a nil check: this is a private converter helper, and a nil work unit can't reach it.

  • Row-to-proto converters (WorkUnitRowToProto and the paused-task converter): they call it on a WorkUnit they have just allocated.
  • WorkUnitProtoToRow: it first validates wu.GetCurrentWorkUnitExecution().GetName(). The getters are nil-safe, so a nil work unit fails that validation with an error before trialFieldsToColumns is reached.

A nil check here would be dead code.

Comment on lines +73 to +74
func readTrialFields(row *spanner.Row, wu *pb.WorkUnit) error {
var parameters []byte

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Defensive programming: Add a nil check for wu to prevent a potential nil pointer dereference panic if a nil work unit is passed to readTrialFields.

func readTrialFields(row *spanner.Row, wu *pb.WorkUnit) error {
	if wu == nil {
		return fmt.Errorf("work unit is nil")
	}
	var parameters []byte

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not adding a nil check: this is a private converter helper, and a nil work unit can't reach it.

  • Row-to-proto converters (WorkUnitRowToProto and the paused-task converter): they call it on a WorkUnit they have just allocated.
  • WorkUnitProtoToRow: it first validates wu.GetCurrentWorkUnitExecution().GetName(). The getters are nil-safe, so a nil work unit fails that validation with an error before trialFieldsToColumns is reached.

A nil check here would be dead code.

@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for study trial parameters and trial IDs by adding parameters (as google.protobuf.Struct) and trial_id (as string) fields to the WorkUnit proto. The changes include updating the storage schemas for both Postgres and Spanner, implementing validation logic in the experiment state server to ensure these fields are correctly handled during BatchCreateWorkUnits, and updating relevant tests and documentation. I have no feedback to provide.

Study trials must carry their hyperparameter values and the optimization
engine's trial ID, and ESS must reject values outside the study's search
space before they reach a workload.

- API: immutable WorkUnit.parameters (google.protobuf.Struct) and
  WorkUnit.trial_id.
- Storage: migration 020 adds nullable WorkUnits.Parameters (proto
  bytes, no proto bundle change) and TrialId on Spanner and PostgreSQL;
  converters write them and every read path returns them, including the
  paused-task joins.
- Validation: BatchCreateWorkUnits requires trial_id (at most 128
  printable bytes) and parameters with exactly one feasible value per
  study_config parameter, and rejects duplicate trial IDs within a
  batch; CreateWorkUnit rejects both fields.
- Docs: ESS README STUDY section and the batch create flow.

BUG=b/558652203
TAG=agy
CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-work-unit-trial-fields-b558652203 branch from d1b2471 to b9340bd Compare November 17, 2026 14:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant