Skip to content

Commit fcad1fb

Browse files
committed
feat(ess): add study trial fields to work units
Study trials must carry their hyperparameter values and the optimization engine's trial ID, and ESS must reject values outside the study's search space before they reach a workload. - API: immutable WorkUnit.parameters (google.protobuf.Struct) and WorkUnit.trial_id. - Storage: migration 020 adds nullable WorkUnits.Parameters (proto bytes, no proto bundle change) and TrialId on Spanner and PostgreSQL; converters write them and every read path returns them, including the paused-task joins. - Validation: BatchCreateWorkUnits requires trial_id (at most 128 printable bytes) and parameters with exactly one feasible value per study_config parameter, and rejects duplicate trial IDs within a batch; CreateWorkUnit rejects both fields. - Docs: ESS README STUDY section and the batch create flow. BUG=b/558652203 TAG=agy CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae
1 parent 980cece commit fcad1fb

43 files changed

Lines changed: 832 additions & 81 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎api/protos/experiment/BUILD.bazel‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ proto_library(
5656
"@com_google_googleapis//google/api:field_behavior_proto",
5757
"@com_google_googleapis//google/api:resource_proto",
5858
"@com_google_googleapis//google/rpc:status_proto",
59+
"@com_google_protobuf//:struct_proto",
5960
"@com_google_protobuf//:timestamp_proto",
6061
],
6162
)
@@ -132,6 +133,7 @@ proto_library(
132133
"@com_google_googleapis//google/rpc:status_proto",
133134
"@com_google_protobuf//:empty_proto",
134135
"@com_google_protobuf//:field_mask_proto",
136+
"@com_google_protobuf//:struct_proto",
135137
"@com_google_protobuf//:timestamp_proto",
136138
],
137139
)

‎api/protos/experiment/work_unit.proto‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ package cloud.xmanager.v1;
1717

1818
import "google/api/field_behavior.proto";
1919
import "google/api/resource.proto";
20+
import "google/protobuf/struct.proto";
2021
import "google/protobuf/timestamp.proto";
2122
import "xmanager_cloud/experiment_state_server/proto/priority.proto";
2223
import "xmanager_cloud/experiment_state_server/proto/resources.proto";
@@ -63,6 +64,15 @@ message WorkUnit {
6364
// The current work unit execution.
6465
WorkUnitExecution current_work_unit_execution = 10
6566
[(google.api.field_behavior) = OUTPUT_ONLY];
67+
// The hyperparameter values of a study trial, keyed by the parameter_id of
68+
// each parameter of the study_config. Required on work units of STUDY
69+
// experiments and rejected on all others.
70+
google.protobuf.Struct parameters = 11
71+
[(google.api.field_behavior) = IMMUTABLE];
72+
// The ID of the trial in the optimization engine, supplied by the study
73+
// controller. Required on work units of STUDY experiments and rejected on all
74+
// others.
75+
string trial_id = 12 [(google.api.field_behavior) = IMMUTABLE];
6676
}
6777

6878
message Executable {

‎cmd/experiment_state_server/README.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -115,7 +115,8 @@ it, and skips a warning that is already present.
115115

116116
A study's work units are its trials, which only the study controller creates. `CreateWorkUnit` on a
117117
`STUDY` experiment returns `FAILED_PRECONDITION` without writing anything
118-
([`internal/workunit/createworkunit.go`](../../internal/workunit/createworkunit.go)).
118+
([`internal/workunit/createworkunit.go`](../../internal/workunit/createworkunit.go)), and rejects
119+
`parameters` and `trial_id` with `INVALID_ARGUMENT`.
119120

120121
#### Creating Trials: `BatchCreateWorkUnits`
121122

@@ -133,17 +134,19 @@ experiment's largest work unit ID. ESS checks, in order
133134
| The caller's verified email equals the `hpo` component parameter `studyControllerServiceAccount` (ASCII case-insensitive). XID ACLs are not consulted. A missing `hpo` component or an empty parameter rejects every caller. This check runs before the experiment is read or the payload is validated. | `PERMISSION_DENIED` |
134135
| The experiment exists. | `NOT_FOUND` |
135136
| The experiment is a `STUDY`; other types are not supported yet. | `UNIMPLEMENTED` |
136-
| The request holds 1 to 500 requests, and each work unit is valid, as for `CreateWorkUnit`. | `INVALID_ARGUMENT` |
137+
| The request holds 1 to 500 requests, each work unit is valid, as for `CreateWorkUnit`, and trial IDs are unique in the batch. | `INVALID_ARGUMENT` |
138+
| Each work unit has a `trial_id` (at most 128 bytes, printable) and `parameters` with exactly one feasible value per `study_config` parameter: a finite number in range for `DOUBLE`, an integral one for `INTEGER`, one of the values for `DISCRETE`, and one of the category strings for `CATEGORICAL`. | `INVALID_ARGUMENT` |
137139

138140
ESS then writes all work units in one transaction
139141
([`internal/workunit/batchcreateworkunits.go`](../../internal/workunit/batchcreateworkunits.go)): it
140142
checks the caller against the `hpo` component again, reads the experiment once, allocates the IDs with a
141143
single next-ID read, inserts the work units in one write, and updates the experiment's denormalized
142144
status and resources once. After the commit it records one creation operation per work unit and
143145
enqueues each creation task. A work unit whose operation or task cannot be created gets the same
144-
`FATAL` status message as in `CreateWorkUnit`, and the RPC returns `INTERNAL` naming those work units;
145-
the other work units are scheduled. The work units are already stored, so the study controller must
146-
not create them again when it retries.
146+
`FATAL` status message as in `CreateWorkUnit`, and the RPC returns `INTERNAL` naming those work
147+
units; the other work units are scheduled. Because the trials are already stored, the study
148+
controller must not create them again when it retries; ESS does not deduplicate `trial_id` across
149+
batches. `parameters` and `trial_id` are immutable and returned by `GetWorkUnit` and `ListWorkUnits`.
147150

148151
---
149152

‎cmd/experiment_state_server/server/batchcreateworkunits.go‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,8 @@ import (
3333
)
3434

3535
// BatchCreateWorkUnits creates the trials of a STUDY experiment for the study controller: it
36-
// persists all of them in one transaction, records one creation operation per work unit, and
37-
// enqueues their creation tasks.
36+
// validates every work unit against the study config, persists all of them in one transaction,
37+
// records one creation operation per work unit, and enqueues their creation tasks.
3838
func (h *WorkUnitHandler) BatchCreateWorkUnits(ctx context.Context, req *msgpb.BatchCreateWorkUnitsRequest) (*msgpb.BatchCreateWorkUnitsResponse, error) {
3939
ctx = logging.InjectOperation(ctx, "")
4040
logging.Info(ctx, "Batch create work units", "parent", req.GetParent(), "count", len(req.GetRequests()))
@@ -58,8 +58,8 @@ func (h *WorkUnitHandler) BatchCreateWorkUnits(ctx context.Context, req *msgpb.B
5858
return nil, fmt.Errorf("BatchCreateWorkUnits validation failed: %w", err)
5959
}
6060

61-
// 6. Assemble each work unit.
62-
wus, err := h.assembleStudyTrials(ctx, xid, req.GetRequests())
61+
// 6. Validate the trial fields and assemble each work unit.
62+
wus, err := h.assembleStudyTrials(ctx, xid, exp.GetStudyConfig(), req.GetRequests())
6363
if err != nil {
6464
return nil, err
6565
}
@@ -115,11 +115,15 @@ func (h *WorkUnitHandler) fetchBatchCreateParent(ctx context.Context, xid int64)
115115
return exp, nil
116116
}
117117

118-
// assembleStudyTrials returns copies of the requested work units assembled for creation in
119-
// experiment xid.
120-
func (h *WorkUnitHandler) assembleStudyTrials(ctx context.Context, xid int64, reqs []*msgpb.CreateWorkUnitRequest) ([]*msgpb.WorkUnit, error) {
118+
// assembleStudyTrials validates the trial fields of each requested work unit against cfg and
119+
// returns copies assembled for creation in experiment xid.
120+
func (h *WorkUnitHandler) assembleStudyTrials(ctx context.Context, xid int64, cfg *msgpb.StudyConfig, reqs []*msgpb.CreateWorkUnitRequest) ([]*msgpb.WorkUnit, error) {
121121
wus := make([]*msgpb.WorkUnit, len(reqs))
122122
for i, req := range reqs {
123+
if err := validator.ValidateWorkUnitTrialFields(req.GetWorkUnit(), cfg); err != nil {
124+
logging.Warn(ctx, "BatchCreateWorkUnits validation failed", "index", i, "error", err)
125+
return nil, fmt.Errorf("BatchCreateWorkUnits validation failed: requests[%d]: %w", i, err)
126+
}
123127
wu, err := h.assembleNewWorkUnit(ctx, xid, 0, proto.Clone(req.GetWorkUnit()).(*msgpb.WorkUnit))
124128
if err != nil {
125129
return nil, fmt.Errorf("requests[%d]: %w", i, err)

‎cmd/experiment_state_server/server/batchcreateworkunits_test.go‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,8 @@ func batchRequest(n int, fn func(req *msgpb.BatchCreateWorkUnitsRequest)) *msgpb
8383
req.Requests = append(req.Requests, &msgpb.CreateWorkUnitRequest{WorkUnit: &msgpb.WorkUnit{
8484
WorkUnitType: msgpb.WorkUnit_STANDARD,
8585
Executable: testExecutable,
86-
Notes: fmt.Sprintf("trial-%d", i+1),
86+
Parameters: experimenttest.TrialParameters(),
87+
TrialId: fmt.Sprintf("trial-%d", i+1),
8788
}})
8889
}
8990
return prototest.Modify(req, fn)
@@ -112,14 +113,14 @@ func TestBatchCreateWorkUnits_CreatesTrialsInOneTransaction(t *testing.T) {
112113
var gotNames, gotTrials []string
113114
for _, wu := range resp.GetWorkUnits() {
114115
gotNames = append(gotNames, wu.GetName())
115-
gotTrials = append(gotTrials, wu.GetNotes())
116+
gotTrials = append(gotTrials, wu.GetTrialId())
116117
}
117118
wantNames := []string{"experiments/123/workUnits/4", "experiments/123/workUnits/5", "experiments/123/workUnits/6"}
118119
if diff := cmp.Diff(wantNames, gotNames); diff != "" {
119120
t.Errorf("created work unit names mismatch (-want +got):\n%s", diff)
120121
}
121122
if diff := cmp.Diff([]string{"trial-1", "trial-2", "trial-3"}, gotTrials); diff != "" {
122-
t.Errorf("created work unit notes mismatch (-want +got):\n%s", diff)
123+
t.Errorf("created trial IDs mismatch (-want +got):\n%s", diff)
123124
}
124125
if st.txCalls != 1 || st.UpdatedExperiment == nil || len(st.InsertedWorkUnits) != 3 {
125126
t.Errorf("transactions = %d, experiment updated = %v, inserted = %d; want 1, true, 3", st.txCalls, st.UpdatedExperiment != nil, len(st.InsertedWorkUnits))
@@ -204,6 +205,10 @@ func TestBatchCreateWorkUnits_Rejections(t *testing.T) {
204205
st.ComponentRepository = &storage.MockComponentRepository{GetActiveComponentErr: errors.New("catalog down")}
205206
return st
206207
}, wantCode: codes.Internal},
208+
{name: "parameters do not match the study", req: batchRequest(2, func(r *msgpb.BatchCreateWorkUnitsRequest) {
209+
r.Requests[1].WorkUnit.Parameters.Fields["momentum"] = r.Requests[1].WorkUnit.Parameters.Fields["dropout"]
210+
}), wantCode: codes.InvalidArgument, wantErrSub: "requests[1]"},
211+
{name: "missing trial ID", req: batchRequest(1, func(r *msgpb.BatchCreateWorkUnitsRequest) { r.Requests[0].WorkUnit.TrialId = "" }), wantCode: codes.InvalidArgument},
207212
{name: "unknown execution environment", opts: []testServerOpt{withConfig(&config.Config{DefaultExecutionEnvironment: "unknown"})}, wantCode: codes.Internal, wantErrSub: "requests[0]"},
208213
{name: "storage failure", storage: func() *batchStorage {
209214
st := newBatchStorage("test@example.com")

‎cmd/experiment_state_server/server/createworkunit.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,11 @@ func (h *WorkUnitHandler) CreateWorkUnit(ctx context.Context, req *msgpb.CreateW
6161
logging.Warn(ctx, "CreateWorkUnit validation failed", "error", err)
6262
return nil, fmt.Errorf("CreateWorkUnit validation failed: %w", err)
6363
}
64+
// Study trial fields are only set by BatchCreateWorkUnits; this RPC rejects STUDY experiments.
65+
if err := validator.ValidateWorkUnitTrialFields(req.GetWorkUnit(), nil); err != nil {
66+
logging.Warn(ctx, "CreateWorkUnit validation failed", "error", err)
67+
return nil, fmt.Errorf("CreateWorkUnit validation failed: %w", err)
68+
}
6469

6570
if req.GetWorkUnitId() == 0 {
6671
logging.Warn(ctx, "work_unit_id not provided, will be generated by server")

‎cmd/experiment_state_server/server/createworkunit_test.go‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ import (
3333
xidmsgpb "xmanager_cloud/api/protos/xid"
3434
"xmanager_cloud/cmd/experiment_state_server/config"
3535
"xmanager_cloud/internal/auth"
36+
"xmanager_cloud/internal/experiment/experimenttest"
3637
"xmanager_cloud/internal/grpcerr"
3738
"xmanager_cloud/internal/security/securitytest"
3839
"xmanager_cloud/internal/storage"
@@ -529,3 +530,31 @@ func TestCreateWorkUnit_StudyExperimentRejected(t *testing.T) {
529530
t.Errorf("CreateWorkUnit() on a study enqueued a task: %v", taskQueueClient.LastEnqueuedAction)
530531
}
531532
}
533+
534+
func TestCreateWorkUnit_RejectsStudyTrialFields(t *testing.T) {
535+
tests := []struct {
536+
name string
537+
wu *msgpb.WorkUnit
538+
}{
539+
{name: "parameters", wu: &msgpb.WorkUnit{WorkUnitType: msgpb.WorkUnit_STANDARD, Executable: testExecutable, Parameters: experimenttest.TrialParameters()}},
540+
{name: "trial_id", wu: &msgpb.WorkUnit{WorkUnitType: msgpb.WorkUnit_STANDARD, Executable: testExecutable, TrialId: "trial-1"}},
541+
}
542+
for _, tt := range tests {
543+
t.Run(tt.name, func(t *testing.T) {
544+
st := &storage.MockStorage{Experiment: &msgpb.Experiment{Name: "experiments/123"}}
545+
server := newTestServer(t,
546+
withConfig(&config.Config{DefaultExecutionEnvironment: "gke"}),
547+
withXidClient(&mockXidClient{
548+
checkUserAuthorizationResp: &xidmsgpb.CheckUserAuthorizationResponse{Authorized: proto.Bool(true)},
549+
}),
550+
withStorage(st),
551+
)
552+
553+
_, err := server.CreateWorkUnit(context.Background(), &msgpb.CreateWorkUnitRequest{Parent: "experiments/123", WorkUnit: tt.wu})
554+
555+
if status.Code(err) != codes.InvalidArgument || st.InsertedWorkUnit != nil {
556+
t.Errorf("CreateWorkUnit(%s) = %v, inserted %v; want InvalidArgument and no insert", tt.name, err, st.InsertedWorkUnit)
557+
}
558+
})
559+
}
560+
}

‎docs/flows/work_unit_lifecycle.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -133,8 +133,8 @@ sequenceDiagram
133133
134134
SC->>ESS: 1. BatchCreateWorkUnits(parent, N requests)
135135
ESS->>DB: 2. Read hpo component (studyControllerServiceAccount)
136-
ESS->>DB: 3. FetchExperiment (STUDY, priority)
137-
ESS->>ESS: 4. Validate and assemble each work unit
136+
ESS->>DB: 3. FetchExperiment (STUDY, study_config, priority)
137+
ESS->>ESS: 4. Validate parameters and trial_id, and assemble each work unit
138138
ESS->>DB: 5. One transaction: re-check caller, next WID, insert N work units, update experiment once
139139
ESS->>DB: 6. AddWorkUnitOperation per work unit
140140
ESS->>Redis: 7. XADD signed payload per work unit

‎internal/experiment/experimenttest/BUILD.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ go_library(
2828
"//api/protos/experiment:messages_go_proto",
2929
"//pkg/components/store",
3030
"@org_golang_google_protobuf//proto",
31+
"@org_golang_google_protobuf//types/known/structpb",
3132
],
3233
)
3334

‎internal/experiment/experimenttest/studyconfig.go‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
// Package experimenttest provides experiment test fixtures shared across packages.
1616
package experimenttest
1717

18-
import essmsgpb "xmanager_cloud/api/protos/experiment"
18+
import (
19+
structpb "google.golang.org/protobuf/types/known/structpb"
20+
essmsgpb "xmanager_cloud/api/protos/experiment"
21+
)
1922

2023
// StudyConfig returns a new valid single-objective study config that uses every parameter type.
2124
// Derive variants with prototest.Modify.
@@ -47,3 +50,13 @@ func StudyConfig() *essmsgpb.StudyConfig {
4750
MaxParallelTrials: 2,
4851
}
4952
}
53+
54+
// TrialParameters returns new valid trial parameters for StudyConfig, one value per parameter.
55+
func TrialParameters() *structpb.Struct {
56+
return &structpb.Struct{Fields: map[string]*structpb.Value{
57+
"learning_rate": structpb.NewNumberValue(0.01),
58+
"batch_size": structpb.NewNumberValue(32),
59+
"optimizer": structpb.NewStringValue("adam"),
60+
"dropout": structpb.NewNumberValue(0.2),
61+
}}
62+
}

0 commit comments

Comments
 (0)