Repository navigation
feat(ess): add BatchCreateWorkUnits for study trials - #1682
VLukiantsev wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the BatchCreateWorkUnits RPC, allowing the study controller to create up to 500 trials of a STUDY experiment in a single transaction. The changes span API proto definitions, server-side handler implementation, refactoring of work unit assembly, batch insert support in Spanner and Postgres storage layers, and updates to the HPO component Helm charts to support the studyControllerServiceAccount parameter. The review feedback suggests several optimizations: adding defensive nil checks for the component record and its spec in AuthorizeStudyController to prevent nil pointer dereferences, and implementing early return checks for empty work unit slices in BatchCreateWorkUnits and Spanner's InsertWorkUnits to avoid unnecessary transactions and allocations.
7fc81da to
259dc30
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the BatchCreateWorkUnits RPC to support the batch creation of up to 500 work units (trials) for STUDY experiments in a single transaction. The changes span protobuf definitions, server implementation, database repositories (both Spanner and PostgreSQL), Helm charts, and documentation, alongside comprehensive tests. The review feedback suggests adding defensive nil checks in assembleStudyTrials, validateBatchedCreateWorkUnitRequest, and BatchCreateWorkUnits to prevent potential nil pointer dereference panics.
259dc30 to
5ba381b
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces BatchCreateWorkUnits to support creating multiple work units for STUDY experiments in a single transaction, improving efficiency for study controllers. It includes the necessary API definitions, server-side logic, database repository updates for batch insertion, and comprehensive testing. The review highlighted a potential performance bottleneck in the error handling of actuateCreatedWorkUnits due to sequential database calls and identified a security risk regarding case-insensitive email comparison for authorization, which requires canonicalization.
The study controller needs to create the trials of a STUDY experiment atomically and cheaply. BatchCreateWorkUnits (AIP-233) creates up to 500 work units in one transaction with contiguous IDs and records one operation per work unit. Only the study controller may call it, on STUDY experiments only. - API: BatchCreateWorkUnits RPC (POST :batchCreate, synchronous response in request order); work_unit_id must be unset because the server assigns contiguous IDs. - Storage: new InsertWorkUnits primitive: one Spanner BufferWrite, or one experiment lock plus one pgx batch on PostgreSQL (pgdb.DBTX gains SendBatch); InsertWorkUnit delegates to it. Both backends reject nil entries and batches spanning experiments with INVALID_ARGUMENT, and return early for an empty batch, before any lock or buffered write. - Domain: internal/workunit BatchCreateWorkUnits rejects an empty batch or a nil work unit with INVALID_ARGUMENT before starting a transaction, re-checks the caller, reads the experiment once, allocates IDs with a single next-ID read, and updates the experiment status and resources once. Concurrent batches get disjoint contiguous IDs on both backends. - Authorization: the caller's verified email must equal the hpo component parameter studyControllerServiceAccount (case-insensitive). It is checked right after authentication and the parent name check, before the experiment is read or the payload is validated, and again in the transaction; XID ACLs are not consulted, and a missing component, missing record, or empty parameter denies every caller. STANDARD experiments return UNIMPLEMENTED. - Enqueue: one task per work unit; a work unit whose operation or task cannot be created gets a FATAL status message and the RPC returns INTERNAL naming it. - hpo chart (stable-2026b profile): parameters.studyControllerServiceAccount, validated by the profile schema (email or empty), rendered into the Component CR and, from the hpoDeployment value xmc-bridge passes, as the Workload Identity annotation of the xmc-study-controller ServiceAccount. - Docs: ESS README, work unit lifecycle, flows overview, ESS component RPC list, hpo and stable-2026b profile READMEs. BUG=b/558652203 TAG=agy CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae
5ba381b to
f6c70b5
Compare
Summary
Add
BatchCreateWorkUnits, an AIP-233 batch create RPC that only the configured study controller may call. It creates all trials of a reconcile tick in one transaction with contiguous work unit IDs. This is layer 2 of a 4-PR stack; the trial fields and pipelined enqueue follow in the next two layers.Motivation & Context
The study controller dispatches up to N trials per reconcile tick. Doing that with N unary calls costs N transactions, N parent-row updates and N Redis round trips, and it can interleave IDs with other writers. This RPC writes the whole batch atomically. It also adds the principal guard: on a
STUDYexperiment, authorization is a match against the hpo component'sstudyControllerServiceAccount, and the user XID ACL is not consulted.Sub-slice: HPO M1 S1.3 (trials run and the study finishes). The first production caller is the upcoming controller trial-dispatch task; until then the RPC is reachable through the public gRPC/REST API and its tests.
Key Changes
API (
api.proto):rpc BatchCreateWorkUnits(POST/v1/{parent=experiments/*}/workUnits:batchCreate). The request takesparentplusrepeated CreateWorkUnitRequest requests(1..500; inner parent empty or equal to the outer one;work_unit_idmust be unset). The synchronous response returnswork_unitsin request order.Storage:
InsertWorkUnitsonWorkUnitTxRepositoryfor both backends. Spanner uses a singleBufferWrite; Postgres uses one experiment-row lock plus apgx.Batch(pgdb.DBTXgainsSendBatch). Both reject nil entries and mixed-experiment batches withINVALID_ARGUMENT. The caller owns the transaction.Domain (
internal/workunit): one transaction that runs the authoritative in-txn principal check, reads the experiment once, does oneFetchNextWorkUnitIDplus offsets for contiguous IDs, inserts all rows, records execution history per work unit, foldsPrepareWorkUnitCreationin memory, and updates the experiment once. Helpers are shared with unaryCreateWorkUnit.Handler, in check order:
PERMISSION_DENIEDbefore the experiment is read or the payload is validatedNOT_FOUND; non-STUDY→UNIMPLEMENTEDFailures to add an Operation or enqueue mark the affected work unit with a
FATALmessage (unary pattern), and the RPC returnsINTERNALnaming the affected work units.hpo chart (
stable-2026bprofile; the GA-lockedstable-2026ais untouched): newparameters.studyControllerServiceAccount(default empty; schema: email or empty). It is rendered into the Component CR only when set, and the same value becomes the controller KSA'siam.gke.io/gcp-service-accountannotation. No new rendered objects, so no bridge RBAC change.Docs: ESS README (contract, check order, errors, and a Trust Boundary section explaining why experiment owners can't create trials and which principals can pass the check), hpo README ("Securing the Study Controller Identity" operator guidance),
docs/flows/work_unit_lifecycle.md(batch create section and sequence diagram),docs/flows/README.md,docs/architecture/components.md.Assumptions
tx.Components— the pre-check gives a fastPERMISSION_DENIEDwithout leaking existence or validation detail; the in-txn check closes the TOCTOU window — if the component changes between the two checks, the transaction result wins.studyControllerServiceAccount, or a missing hpo component ⇒PERMISSION_DENIED— fail closed — operators must set the param before the controller can create trials.BatchCreateWorkUnitson aSTANDARDexperiment ⇒UNIMPLEMENTED— not needed in this milestone — exposing it to users later is a separate decision.hpo/vizierenabled + acknowledged + Ready (enabled + unacknowledged is rejected at admission); Spanner and Postgres covered by tests, live on Spanner; metrics store ClickHouse present but not used by this RPC — dependency-unready was not exercised live — covered by the active-only authorization rule above.Verification & Testing
./.githooks/pre-commit;bazel teston the touched packages (ESS server,internal/workunit, Spanner/Postgres work unit storage, validator, task queue, helm profile tests, component linter);run_affected_tests.pyat the stack tip (155/155 pass).STANDARDexperiment ⇒PERMISSION_DENIED;STANDARD⇒UNIMPLEMENTED; validation errors; partial enqueue and Operation failures.stable-2026bprofile with all components labelledxmc.dev/profile: stable-2026b; study controller Ready via its Redis/readyz):PermissionDenied.CREATE, requester = controller), actuator consumed both tasks, and 2 JobSets ran to completion.STANDARDexperiment ⇒Unimplemented.DependenciesReady=True; an EXPERIMENTAL component without acknowledgement was rejected at admission.Related Issues / Bugs
BUG=b/558652203
TAG=agy
CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae