Skip to content

feat(ess): add BatchCreateWorkUnits for study trials - #1682

Draft
VLukiantsev wants to merge 1 commit into
feat/hpo-reject-unary-study-wu-b558652203from
feat/hpo-batch-create-work-units-b558652203
Draft

VLukiantsev wants to merge 1 commit into
feat/hpo-reject-unary-study-wu-b558652203from
feat/hpo-batch-create-work-units-b558652203

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add BatchCreateWorkUnits, an AIP-233 batch create RPC that only the configured study controller may call. It creates all trials of a reconcile tick in one transaction with contiguous work unit IDs. This is layer 2 of a 4-PR stack; the trial fields and pipelined enqueue follow in the next two layers.

Motivation & Context

The study controller dispatches up to N trials per reconcile tick. Doing that with N unary calls costs N transactions, N parent-row updates and N Redis round trips, and it can interleave IDs with other writers. This RPC writes the whole batch atomically. It also adds the principal guard: on a STUDY experiment, authorization is a match against the hpo component's studyControllerServiceAccount, and the user XID ACL is not consulted.

Sub-slice: HPO M1 S1.3 (trials run and the study finishes). The first production caller is the upcoming controller trial-dispatch task; until then the RPC is reachable through the public gRPC/REST API and its tests.

Key Changes

  • API (api.proto): rpc BatchCreateWorkUnits (POST /v1/{parent=experiments/*}/workUnits:batchCreate). The request takes parent plus repeated CreateWorkUnitRequest requests (1..500; inner parent empty or equal to the outer one; work_unit_id must be unset). The synchronous response returns work_units in request order.

  • Storage: InsertWorkUnits on WorkUnitTxRepository for both backends. Spanner uses a single BufferWrite; Postgres uses one experiment-row lock plus a pgx.Batch (pgdb.DBTX gains SendBatch). Both reject nil entries and mixed-experiment batches with INVALID_ARGUMENT. The caller owns the transaction.

  • Domain (internal/workunit): one transaction that runs the authoritative in-txn principal check, reads the experiment once, does one FetchNextWorkUnitID plus offsets for contiguous IDs, inserts all rows, records execution history per work unit, folds PrepareWorkUnitCreation in memory, and updates the experiment once. Helpers are shared with unary CreateWorkUnit.

  • Handler, in check order:

    1. authenticate (verified token email)
    2. parent name
    3. principal pre-check → PERMISSION_DENIED before the experiment is read or the payload is validated
    4. experiment read → NOT_FOUND; non-STUDY → UNIMPLEMENTED
    5. payload validation
    6. transaction
    7. one Operation per work unit
    8. enqueue per work unit, using the existing path; layer 4 pipelines it

    Failures to add an Operation or enqueue mark the affected work unit with a FATAL message (unary pattern), and the RPC returns INTERNAL naming the affected work units.

  • hpo chart (stable-2026b profile; the GA-locked stable-2026a is untouched): new parameters.studyControllerServiceAccount (default empty; schema: email or empty). It is rendered into the Component CR only when set, and the same value becomes the controller KSA's iam.gke.io/gcp-service-account annotation. No new rendered objects, so no bridge RBAC change.

  • Docs: ESS README (contract, check order, errors, and a Trust Boundary section explaining why experiment owners can't create trials and which principals can pass the check), hpo README ("Securing the Study Controller Identity" operator guidance), docs/flows/work_unit_lifecycle.md (batch create section and sequence diagram), docs/flows/README.md, docs/architecture/components.md.

Assumptions

  • The principal is checked twice: a handler pre-check plus an authoritative in-txn check against tx.Components — the pre-check gives a fast PERMISSION_DENIED without leaking existence or validation detail; the in-txn check closes the TOCTOU window — if the component changes between the two checks, the transaction result wins.
  • An empty or missing studyControllerServiceAccount, or a missing hpo component ⇒ PERMISSION_DENIED — fail closed — operators must set the param before the controller can create trials.
  • The hpo component only needs to be active, not Ready, for authorization — readiness is enforced when the study is created — a controller could create trials while hpo is temporarily unready.
  • BatchCreateWorkUnits on a STANDARD experiment ⇒ UNIMPLEMENTED — not needed in this milestone — exposing it to users later is a separate decision.
  • ESS does not deduplicate batches across calls — the controller owns retry and dedup — a controller bug could create duplicate trials.
  • Experiment priority comes from the pre-transaction read — priority does not change during a batch — a concurrent priority update could apply only to later work units.
  • Gate matrix: components enabled; hpo/vizier enabled + acknowledged + Ready (enabled + unacknowledged is rejected at admission); Spanner and Postgres covered by tests, live on Spanner; metrics store ClickHouse present but not used by this RPC — dependency-unready was not exercised live — covered by the active-only authorization rule above.

Verification & Testing

  • Commands Executed: ./.githooks/pre-commit; bazel test on the touched packages (ESS server, internal/workunit, Spanner/Postgres work unit storage, validator, task queue, helm profile tests, component linter); run_affected_tests.py at the stack tip (155/155 pass).
  • Coverage & Test Status:
    • Handler tests: a single transaction; contiguous IDs; one Operation per work unit with the requester; the principal matrix (match, case-insensitive match, mismatch, empty param, hpo absent, catalog error); an unauthorized caller with an invalid payload, a missing experiment or a STANDARD experiment ⇒ PERMISSION_DENIED; STANDARD ⇒ UNIMPLEMENTED; validation errors; partial enqueue and Operation failures.
    • Cross-backend integration test (Spanner and Postgres): two batches get IDs 1..5, and 4 concurrent batches of 3 get disjoint contiguous IDs whose union is 1..12.
    • Domain retry idempotency with a re-run transaction closure. Chart tests for the CR param, the WI annotation, and schema rejection.
  • Live (GKE dev cluster, Spanner, stack tip after the sync to main; stable-2026b profile with all components labelled xmc.dev/profile: stable-2026b; study controller Ready via its Redis /readyz):
    • Ordinary caller ⇒ PermissionDenied.
    • Configured controller service account ⇒ 2 work units with IDs 1–2, 2 Operations (CREATE, requester = controller), actuator consumed both tasks, and 2 JobSets ran to completion.
    • STANDARD experiment ⇒ Unimplemented.
    • Components gate passed: clickhouse, vizier and hpo Ready; hpo DependenciesReady=True; an EXPERIMENTAL component without acknowledgement was rejected at admission.

Related Issues / Bugs

BUG=b/558652203

TAG=agy
CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae

@VLukiantsev
VLukiantsev added this pull request to stack #1685 November 13, 2026 10:42

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the BatchCreateWorkUnits RPC, allowing the study controller to create up to 500 trials of a STUDY experiment in a single transaction. The changes span API proto definitions, server-side handler implementation, refactoring of work unit assembly, batch insert support in Spanner and Postgres storage layers, and updates to the HPO component Helm charts to support the studyControllerServiceAccount parameter. The review feedback suggests several optimizations: adding defensive nil checks for the component record and its spec in AuthorizeStudyController to prevent nil pointer dereferences, and implementing early return checks for empty work unit slices in BatchCreateWorkUnits and Spanner's InsertWorkUnits to avoid unnecessary transactions and allocations.

Comment thread internal/workunit/studycontroller.go Resolved
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-batch-create-work-units-b558652203 branch 2 times, most recently from 7fc81da to 259dc30 Compare November 17, 2026 12:59
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the BatchCreateWorkUnits RPC to support the batch creation of up to 500 work units (trials) for STUDY experiments in a single transaction. The changes span protobuf definitions, server implementation, database repositories (both Spanner and PostgreSQL), Helm charts, and documentation, alongside comprehensive tests. The review feedback suggests adding defensive nil checks in assembleStudyTrials, validateBatchedCreateWorkUnitRequest, and BatchCreateWorkUnits to prevent potential nil pointer dereference panics.

Comment thread internal/validator/requests.go Resolved
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-batch-create-work-units-b558652203 branch from 259dc30 to 5ba381b Compare November 17, 2026 13:28
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces BatchCreateWorkUnits to support creating multiple work units for STUDY experiments in a single transaction, improving efficiency for study controllers. It includes the necessary API definitions, server-side logic, database repository updates for batch insertion, and comprehensive testing. The review highlighted a potential performance bottleneck in the error handling of actuateCreatedWorkUnits due to sequential database calls and identified a security risk regarding case-insensitive email comparison for authorization, which requires canonicalization.

Comment thread internal/workunit/studycontroller.go Resolved
The study controller needs to create the trials of a STUDY experiment
atomically and cheaply. BatchCreateWorkUnits (AIP-233) creates up to 500
work units in one transaction with contiguous IDs and records one
operation per work unit. Only the study controller may call it, on STUDY
experiments only.

- API: BatchCreateWorkUnits RPC (POST :batchCreate, synchronous response
  in request order); work_unit_id must be unset because the server
  assigns contiguous IDs.
- Storage: new InsertWorkUnits primitive: one Spanner BufferWrite, or
  one experiment lock plus one pgx batch on PostgreSQL (pgdb.DBTX gains
  SendBatch); InsertWorkUnit delegates to it. Both backends reject nil
  entries and batches spanning experiments with INVALID_ARGUMENT, and
  return early for an empty batch, before any lock or buffered write.
- Domain: internal/workunit BatchCreateWorkUnits rejects an empty batch
  or a nil work unit with INVALID_ARGUMENT before starting a
  transaction, re-checks the
  caller, reads the experiment once, allocates IDs with a single next-ID
  read, and updates the experiment status and resources once. Concurrent
  batches get disjoint contiguous IDs on both backends.
- Authorization: the caller's verified email must equal the hpo
  component parameter studyControllerServiceAccount (case-insensitive).
  It is checked right after authentication and the parent name check,
  before the experiment is read or the payload is validated, and again
  in the transaction; XID ACLs are not consulted, and a missing
  component, missing record, or empty parameter denies every caller.
  STANDARD experiments return UNIMPLEMENTED.
- Enqueue: one task per work unit; a work unit whose operation or task
  cannot be created gets a FATAL status message and the RPC returns
  INTERNAL naming it.
- hpo chart (stable-2026b profile): parameters.studyControllerServiceAccount,
  validated by the profile schema (email or empty), rendered into the
  Component CR and, from the hpoDeployment value xmc-bridge passes, as
  the Workload Identity annotation of the xmc-study-controller
  ServiceAccount.
- Docs: ESS README, work unit lifecycle, flows overview, ESS component
  RPC list, hpo and stable-2026b profile READMEs.

BUG=b/558652203
TAG=agy
CONV=4b859498-5191-4c6a-9c79-00403f7cb8ae
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-batch-create-work-units-b558652203 branch from 5ba381b to f6c70b5 Compare November 17, 2026 14:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant