Repository navigation
feat(study-controller): renew per-study reconcile leases with lost-lease fencing - #1708
VLukiantsev wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces distributed lease renewal for the study controller's reconcile lock, allowing long-running reconciliations to periodically extend their lease. It adds an Extend method to redislock.Lease backed by a Lua script, implements a background renewLoop in the worker handler, and introduces a new internal/redistest package to run integration tests against real Redis containers using testcontainers-go. Feedback on the changes suggests refining the error handling in the renewLoop to avoid logging false-positive warnings when the context is cancelled or times out during normal worker shutdown or timeout boundaries.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements distributed Redis lease lock renewal to prevent locks from expiring during long-running study reconciliations. It adds an Extend method to the redislock package, implements a background renewal loop in the worker handler, and introduces a redistest package using testcontainers-go for integration testing. The reviewer identified an issue where misleading warnings could be logged during normal worker shutdown or context timeouts in the renewal loop, and suggested checking renewCtx.Err() directly to distinguish expected cancellations from genuine failures.
…ase fencing Extend per-study Redis reconcile leases periodically while reconciliation is in progress so ticks running up to the 60s reconcile timeout hold their 15s lease without risking premature expiration, while immediately fencing work if ownership is lost. - Add (*redislock.Lease).Extend using a token-checked Lua PEXPIRE script that returns ErrLeaseLost if the key expired or changed owner. - Run a 5s (TTL/3) heartbeat renewal loop inside worker.Handler using context.WithCancelCause to cancel in-flight reconciliation immediately on renewal failure, skipping unlock when ErrLeaseLost proves another worker may own the key and ignoring benign context.Canceled races during normal reconcile teardown. - Add internal/redistest to manage disposable redis:6-alpine containers with explicit Ryuk-free termination (TESTCONTAINERS_RYUK_DISABLED=true), plus real-Redis integration suites verifying token-fenced lease semantics and multi-replica contention, cross-study isolation, and multi-renewal ticks. BUG=b/583731472 TAG=agy CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
…cellation Distinguish outer renewCtx cancellation and reconcile timeouts from genuine Redis lease renewal failures in renewLoop using renewCtx.Err(), avoiding misleading renewal warnings during graceful shutdown or reconcile deadline expiry while preserving hung-Redis and ErrLeaseLost fencing. Comment link: #1708 (comment) Comment link: #1708 (comment) TAG=agy CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
8f59529 to
df331d4
Compare
Summary
Adds token-checked Lua
PEXPIRElease renewal ((*redislock.Lease).Extend), a5s(TTL/3) background heartbeat ticker withcontext.WithCancelCauselost-lease fencing inxmc-study-controller, and atestcontainers-goreal-Redis integration suite (Sub-slice S1.3: trials run and the study finishes — random search withmax_trial_budget=5runs 5 JobSets with injected parameters and transitions toFINISHED).Motivation & Context
Stacked on #1707 (
feat/hpo-reconcile-lock-core-b583731472, Layer 1 of 2) so the core non-blocking per-study lock and metrics land within repository PR sizing limits (344prod lines in Layer 1;228prod lines in Layer 2) with zero dead code in either layer.Keeping
leaseTTL = 15sensures a hard-crashed controller pod's orphaned lock expires quickly (<= 15srather than stalling promoted 10s ticks until the 5-minute stream PEL reclamation window), while reconciles that invoke remote ESS, Vizier, and ClickHouse RPCs can run up toreconcileTimeout = 60s. Renewing the lease every5sextends long reconciles safely and immediately cancels in-flight work (context.WithCancelCause) if the lease is lost or Redis becomes unresponsive (5sper-renewal timeout gives a5ssafety cushion before the15sTTL expires).Key Changes
internal/redislock: Added(*Lease).Extend(ctx)backed by an atomic token-checked LuaGET/PEXPIREscript, returningErrLeaseLostwhen the key expired or changed owner.cmd/study_controller/worker:HandlerParams.RenewInterval(default5s=TTL/3) andrenewLoop, bounding eachExtendcall byrenewInterval, cancellingreconcileCtxviacontext.WithCancelCauseon any renewal failure, suppressing expected outerrenewCtxcancellations/timeouts (renewCtx.Err() != nil && errors.Is(err, renewCtx.Err())) so graceful shutdown and reconcile deadline expiry log zero false renewal warnings while genuineErrLeaseLostor hung-RedisextendCtxtimeouts still fence and retry, awaitingwg.Wait()beforereleaseLeasesoExtendandReleasenever race, and skipping redundantReleasewhenErrLeaseLostwas already observed.internal/redistest& Real-Redis Integration Tests (tags = ["local"]):testonlyinternal/redistest(RunMain) launchingredis:6-alpineviatestcontainers-goand explicitly callingtestcontainers.TerminateContaineron a fresh15scleanup context (TESTCONTAINERS_RYUK_DISABLED=trueCI safe).internal/redislock:redislock_redis_test(LuaAcquire/Extend/Release, stale-holder token fencing after expiry + re-acquisition, 8-goroutine mutual exclusion) andcmd/study_controller/worker:worker_redis_test(2-replica stream contention ACK without overlap, cross-studyxidnon-blocking concurrency, and live lease extension past initial TTL).deploy/profiles/stable-2026b/components/hpo/README.mdanddocs/architecture/components.md.Assumptions
stable-2026b(preview;hpoandviziership only instable-2026b) /stable-2026a(GA-locked withouthpo, soCreateExperiment(STUDY)fails closed withFailedPrecondition; XMC Components are always-on in1.0.0+with noenable_componentsflag).hpo/vizierComponent state scope: Whenhpoisenabled=true(acknowledged withvizierandclickhouseReady=Trueinstable-2026b),xmc-study-controllerrenewsxmc:study-reconcile-lock:{xid}every5sup to60sand cancels in-flight reconciles on lost leases; ifvizierorclickhouselater goes unready, the running controller remains in place (/readyzchecks Redis only in this slice) and retries Redis lock/renewal errors; whenhpoisenabled=falseor absent, noxmc-study-controllerpod runs, zero lock keys are written, andCreateExperiment(STUDY)returnsFailedPrecondition(bothenabled=trueandenabled=falsestates verified live on GKE).database_backend: Bothspannerandpostgresare supported (this change is Redis-only with zero SQL/DDL changes;spannerwas verified live on GKE).clickhouse(v1,Ready=Trueon live GKE) /bigquery(Unimplementedfor HPO queries; not queried in this slice).TTL/3) — Bounds eachExtendRPC to5s(ContextTimeoutEnabled: true) so a hung Redis aborts the reconcile att <= 10s(>= 5sbeforet = 15slock expiry) — Extreme host pause/GC stalls exceeding the remaining5s–15slease window still rely on downstream ESS/Vizier idempotency (client_idand principal/state checks).noopReconcile) — ESS study re-read, VizierCreateStudy, and trial dispatch land in follow-up slices — Automated unit and real-Redis container tests exercise multi-renewal and lost-lease cancellation directly via injectedReconcileFuncs until remote RPCs are wired in.Verification & Testing
Automated Tests
Layer 2):bazel test --@io_bazel_rules_go//go/config:race --test_env=TESTCONTAINERS_RYUK_DISABLED=true //internal/redistest/... //internal/redislock/... //cmd/study_controller/...(8/8targets passed, includingredistest_test,redislock_redis_test, andworker_redis_test) and./.githooks/pre-commitpassed.Layer 2):bazel coverage --combined_report=lcov --test_env=TESTCONTAINERS_RYUK_DISABLED=true //internal/redistest/... //internal/redislock/... //cmd/study_controller/...— 100.0% line coverage oninternal/redislock/redislock.go(54/54),cmd/study_controller/worker/handler.go(140/140), andcmd/study_controller/worker/consumer.go(49/49);93.75%(30/32) oninternal/redistest/redistest.go../tools/analyze_diff_size.py --base-ref feat/hpo-reconcile-lock-core-b583731472->228prod /574test (802total,threshold_exceeded: false).Live Deployment & Manual Testing
tools/dev_env/dev_env.py gke-upgrade --profile stable-2026b --allow-preview-profilewith the dev-only Vizier fixture (test/e2e/testdata/vizier/,vizier-service:7077).stable-2026b(preview;hpoandviziership only instable-2026b).hpo/vizierstates:enabled+Ready(Ready=True,DependenciesReady=True),enabled+unacknowledged(rejected at CRD admission via server-side dry-run--negative-gate), anddisabled(hpo.enabled: falseuninstallsxmc-cmp-hpoandCreateExperiment(STUDY)returnsFailedPrecondition).database_backend:spannerlive-verified on GKE (ESS schema v19, XID schema v1, zero pending migrations; Study Controller introduces no SQL/DDL changes sospannerandpostgresremain at full parity).clickhouse(Ready=True).test/e2e/; automated coverage is provided by hermetic unit tests and real Redis (redis:6-alpineviatestcontainers-go) multi-consumer integration tests (bazel test //internal/redistest/... //internal/redislock/... //cmd/study_controller/...), complemented by the live GKE scenarios below.env_pool.py verify: All 5 sections✅ PASS(18/18Ready pods, all 5ComponentCRs verified).GET /v1/health/configs): HTTP401.700onctk-out,600onxmc_app/terraform.tfstate.check_components_live.py --require clickhouse --require vizier --require hpo:Ready,DependenciesReady --negative-gate): Exited0("ok": true).CreateExperiment(STUDY)Initial Tick,/readyz&/metrics::8080/healthz,:8080/readyz, and:8080/metricsonxmc-study-controller, then invokedExperimentStateService/CreateExperiment(experimentType: "STUDY",algorithm: "RANDOM_SEARCH") viagrpcurlusing an impersonated service account OIDC token./healthzand/readyzreturned200 OK;/metricspre-initialized all fourxmc_study_controller_reconcile_ticks_totaloutcomelabel values (reconciled,contended,retry_error,invalid_payload) at0. Within ~215 ms ofCreateExperiment,xmc-study-controllerloggedINFO "Processed study reconcile tick",outcome="reconciled"incremented0 -> 1(+1),EXISTS xmc:study-reconcile-lock:<study-1>returned0(lease cleanly released), andXPENDING xmc-study-reconcile-stream study-controller-groupreturned0.deployment/xmc-study-controllertoreplicas=2(verifyingxmc-bridgepreservedreplicas=2and both pods reached1/1 Ready). Held a 10s lease (SET xmc:study-reconcile-lock:<study-1> demo-holder NX PX 10000) and pipelined 2 duplicate ticks for<study-1>; after lease expiry, pipelined a 7-tick burst (6 ticks for<study-1>+ 1 tick for<study-2>) across both active replicas; then enqueued 1 malformed tick ({"experiment":"not-an-experiment"}); finally scaled back toreplicas=1.pod-aandpod-b) loggedINFO "Skipped contended study reconcile tick"with distinctworker_ids; cluster-summedoutcome="contended"incremented0 -> 2(+2) whileoutcome="reconciled"remained unchanged (+0), andXPENDINGremained0.pod-a+5,pod-b+2) withretry_error=0,XPENDING=0,dlq_xlen=0, and both per-study locks released (EXISTS=0).WARN "Dropping invalid study reconcile tick"and"Task moving to DLQ"(reason="non_retryable_error"),outcome="invalid_payload"incremented0 -> 1(+1),XPENDING=0, andXLEN xmc-study-reconcile-dlq-streamincremented0 -> 1.hpo.enabled: false) & Clean Restore:components.hpo.enabled: falseviagke-upgrade --only app, checked Helm releases, pods, andCreateExperiment(STUDY), then restoredhpo.enabled: true(acknowledgeExperimental: true) and re-rancheck_components_live.py.hpo.enabled: false,xmc-bridgeuninstalledxmc-cmp-hpo, terminated allxmc-study-controllerpods (0pods), updatedComponent/hpotoCOMPONENT_STATE_DISABLED(Synced=True), andCreateExperiment(STUDY)failed closed with gRPCFailedPrecondition(component "hpo" is in state COMPONENT_STATE_DISABLED, required ENABLED). Restoringhpo.enabled: truerecreatedxmc-study-controller(1/1 Ready) andcheck_components_live.pyexited0.Live Execution Output / Logs (Sanitized)
Related Issues / Bugs
Fixes bug 583731472
BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d