Repository navigation
feat(study-controller): guard study reconcile ticks with per-study Redis lock - #1707
VLukiantsev wants to merge 2 commits into
Conversation
…dis lock
Introduce distributed per-study lease locking in the Study Controller so
multi-replica deployments never reconcile the same study concurrently when
consuming at-least-once ticks from xmc-study-reconcile-stream.
- Add internal/redislock to acquire non-blocking leases with SET ... NX PX
and random 16-byte hex tokens, releasing keys via a token-checked Lua DEL
script so an expired lease never deletes a newer holder's lock.
- Replace the raw consumer task callback in cmd/study_controller/worker with
Handler and NewRedisHandler, sharing the consumer's single *redis.Client
connection pool across stream consumption, per-study locking, and /readyz.
- Acknowledge contended ticks (ErrNotAcquired) immediately without retrying,
dead-letter malformed payloads before lock acquisition, and retry transient
lock or reconcile failures under a 60s reconcile timeout (< 2m stream
process timeout).
- Detach lease release from parent shutdown cancellation with a 2s bound and
extend consumer drain timeout to 10s so in-flight ticks release their lock
and settle cleanly on SIGTERM.
- Expose xmc_study_controller_reconcile_ticks_total{outcome} on /metrics.
BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
There was a problem hiding this comment.
Code Review
This pull request introduces a distributed locking mechanism using Redis to prevent concurrent study reconciliations in the study controller, along with Prometheus metrics to track processing outcomes. The feedback highlights two key issues: first, the default lease TTL is shorter than the default reconcile timeout, which could lead to premature lock expiration and concurrent processing; second, the isNilClient check only handles typed nil pointers for *redis.Client, which could cause nil pointer panics for other types implementing the Client interface. Using reflection is recommended to make the nil check robust.
| defaultLeaseTTL = 15 * time.Second | ||
| defaultReconcileTimeout = 60 * time.Second |
There was a problem hiding this comment.
The defaultLeaseTTL (15s) is shorter than the defaultReconcileTimeout (60s). Since there is no active lease renewal (heartbeat) mechanism, any reconciliation taking longer than 15 seconds will lose its lock. This allows another replica to acquire the lock for the same study and run a concurrent reconciliation, violating the mutual exclusion guarantee.
To ensure mutual exclusion throughout the entire reconciliation process, the lease TTL should be at least as long as the maximum reconciliation timeout (plus a small buffer for safety, e.g., 65s).
| defaultLeaseTTL = 15 * time.Second | |
| defaultReconcileTimeout = 60 * time.Second | |
| defaultLeaseTTL = 65 * time.Second | |
| defaultReconcileTimeout = 60 * time.Second |
There was a problem hiding this comment.
Keeping defaultLeaseTTL = 15s intentionally: in Layer 1 (#1707), Handler.reconcile is noopReconcile (zero I/O, completes in sub-microseconds well within 15s), and stacked Layer 2 (#1708) immediately adds the 5s (TTL/3) token-checked Lua PEXPIRE heartbeat renewal loop with context.WithCancelCause lost-lease fencing before any remote RPCs exist. Raising defaultLeaseTTL to 65s would leave a 65s orphaned lock on hard pod crash (SIGKILL/OOM), stalling promoted 10s ticks until the 5-minute stream PEL reclamation window.
| import ( | ||
| "context" | ||
| "crypto/rand" | ||
| "encoding/hex" | ||
| "errors" | ||
| "fmt" | ||
| "time" | ||
|
|
||
| "github.com/redis/go-redis/v9" | ||
| ) |
There was a problem hiding this comment.
To support robust nil checking for any concrete type implementing the Client interface (including custom test doubles or future client implementations like *redis.ClusterClient), we should use reflection in isNilClient. Please add "reflect" to the import block.
| import ( | |
| "context" | |
| "crypto/rand" | |
| "encoding/hex" | |
| "errors" | |
| "fmt" | |
| "time" | |
| "github.com/redis/go-redis/v9" | |
| ) | |
| import ( | |
| "context" | |
| "crypto/rand" | |
| "encoding/hex" | |
| "errors" | |
| "fmt" | |
| "reflect" | |
| "time" | |
| "github.com/redis/go-redis/v9" | |
| ) |
There was a problem hiding this comment.
Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).
| func isNilClient(client Client) bool { | ||
| if client == nil { | ||
| return true | ||
| } | ||
| c, ok := client.(*redis.Client) | ||
| return ok && c == nil | ||
| } |
There was a problem hiding this comment.
The current isNilClient implementation only checks for typed nil pointers of type *redis.Client. If a caller passes a typed nil pointer of another type implementing Client (such as *redis.ClusterClient or a custom test double like *stubLockClient), isNilClient will return false, leading to a nil pointer panic when SetArgs is called.
Using reflection allows us to safely detect typed nil pointers for any concrete type implementing the interface.
| func isNilClient(client Client) bool { | |
| if client == nil { | |
| return true | |
| } | |
| c, ok := client.(*redis.Client) | |
| return ok && c == nil | |
| } | |
| func isNilClient(client Client) bool { | |
| if client == nil { | |
| return true | |
| } | |
| v := reflect.ValueOf(client) | |
| return v.Kind() == reflect.Ptr && v.IsNil() | |
| } |
There was a problem hiding this comment.
Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a distributed Redis lease lock mechanism (internal/redislock) and integrates it into the study_controller worker to process study reconcile ticks under a per-study lock. It also adds Prometheus metrics to track processing outcomes and updates relevant documentation and tests. The review feedback suggests improving the robustness of the isNilClient helper by using reflection to detect any typed nil pointer, and swapping the defer order in reconcileWithLease to ensure context resources are freed immediately.
| func isNilClient(client Client) bool { | ||
| if client == nil { | ||
| return true | ||
| } | ||
| c, ok := client.(*redis.Client) | ||
| return ok && c == nil | ||
| } |
There was a problem hiding this comment.
The current isNilClient helper only checks if the interface is a typed nil for *redis.Client. If a caller passes a typed nil of another type that implements Client (such as *redis.ClusterClient or a custom mock/wrapper), isNilClient will return false. This will subsequently cause a nil pointer dereference panic when SetArgs or Eval is called on the client.
Using reflection allows us to safely check if any pointer type implementing the interface is nil. Note that you will need to import "reflect" at the top of the file.
| func isNilClient(client Client) bool { | |
| if client == nil { | |
| return true | |
| } | |
| c, ok := client.(*redis.Client) | |
| return ok && c == nil | |
| } | |
| func isNilClient(client Client) bool { | |
| if client == nil { | |
| return true | |
| } | |
| val := reflect.ValueOf(client) | |
| return val.Kind() == reflect.Ptr && val.IsNil() | |
| } |
There was a problem hiding this comment.
Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).
Swap LIFO defer ordering in reconcileWithLease so the reconcile context timer is cancelled immediately upon reconcile return before the detached Redis lease release RPC executes. Comment link: #1707 (comment) TAG=agy CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
Summary
Introduces
internal/redislock(SET ... NX PXwith a 16-bytecrypto/randtoken and token-checked LuaDELrelease) and wires per-study mutual exclusion (xmc:study-reconcile-lock:{xid}) intoxmc-study-controller(Sub-slice S1.3: trials run and the study finishes — random search withmax_trial_budget=5runs 5 JobSets with injected parameters and transitions toFINISHED). Shares a single*redis.Clientpool across stream consumption,/readyz, and locking, acknowledges contended duplicate ticks without reconciling, and exposesxmc_study_controller_reconcile_ticks_total{outcome}onGET /metrics.Motivation & Context
xmc-study-controlleris stateless and consumes unsigned, at-least-once reconcile hints fromxmc-study-reconcile-stream. Before trial suggestion andBatchCreateWorkUnitsdispatch land (and beforehposcales beyondreplicas: 1), concurrent ticks for the same study across replicas or redeliveries must never reconcile simultaneously.This is Layer 1 of a 2-PR
gh-stackdelivering the core non-blocking per-study lease lock, contention routing, single-pool wiring, and observability (344prod /480test lines); Layer 2 (#1708,feat/hpo-reconcile-lock-heartbeat-b583731472) adds periodic LuaPEXPIREheartbeat renewal, lost-leasecontext.WithCancelCausefencing, and thetestcontainers-goreal-Redis integration suite.Key Changes
internal/redislock: Added generic non-blocking Redis lease lock (New,Locker.Acquire,Lease.Release,ErrNotAcquired,ErrLeaseLost) usingSET key token NX PXwith a 128-bitcrypto/randhex token and an atomic Lua compare-and-delete release script.cmd/study_controller&cmd/study_controller/worker:*redis.Clientconnection pool (ContextTimeoutEnabled: true) acrossConsumerClient,/readyz, andNewRedisHandler.xmc:study-reconcile-lock:{xid}(15sTTL,60sreconcile timeout strictly below the2mstream consumer timeout, and2sdetachedcontext.WithoutCancelrelease timeout; consumer drain timeout raised from8sto10s).ErrNotAcquired->WasRunSuccessful: true,outcome="contended") without reconciling, while returning retryable failures (ShouldBeRetried: true,outcome="retry_error") on Redis transport or reconcile errors.xmc_study_controller_reconcile_ticks_total{outcome="reconciled|contended|retry_error|invalid_payload"}pre-initialized on/metricsand logged reconciled/contended ticks atINFO.deploy/profiles/stable-2026b/components/hpo/README.md,docs/flows/experiment_lifecycle.md,docs/flows/metrics_and_observability.md, anddocs/architecture/components.md.Assumptions
stable-2026b(preview;hpoandviziership only instable-2026b) /stable-2026a(GA-locked withouthpo, soCreateExperiment(STUDY)fails closed withFailedPrecondition; XMC Components are always-on in1.0.0+with noenable_componentsflag).hpo/vizierComponent state scope: Whenhpoisenabled=true(acknowledged withvizierandclickhouseReady=Trueinstable-2026b),xmc-study-controlleracquiresxmc:study-reconcile-lock:{xid}around each tick and exposesxmc_study_controller_reconcile_ticks_total; ifvizierorclickhouselater goes unready, the running controller remains in place (/readyzchecks Redis only in this slice) and retries Redis lock errors without dead-lettering; whenhpoisenabled=falseor absent, noxmc-study-controllerpod runs, zero lock keys are written, andCreateExperiment(STUDY)returnsFailedPrecondition(bothenabled=trueandenabled=falsestates verified live on GKE).database_backend: Bothspannerandpostgresare supported (this change is Redis-only with zero SQL/DDL changes;spannerwas verified live on GKE).clickhouse(v1,Ready=Trueon live GKE) /bigquery(Unimplementedfor HPO queries; not queried in this slice).noopReconcile) — ESS study re-read, VizierCreateStudy, and trial dispatch land in follow-up slices — In Layer 1, valid lock-holding ticks execute thenoopReconcilestub well within the15sinitial TTL, while backgroundPEXPIREheartbeat renewal for long-running reconciles lands immediately upstack in Layer 2 before remote RPCs are added.CreateExperiment(STUDY)before 10s delayed polling is wired — Once 10s periodic study polling lands, steady-state tick logs should be downgraded fromINFOtoDEBUGso multi-study clusters rely onxmc_study_controller_reconcile_ticks_totalwithout high log volume.Verification & Testing
Automated Tests
Layer 1):bazel test --@io_bazel_rules_go//go/config:race //internal/redislock/... //cmd/study_controller/...(5/5targets passed) and./.githooks/pre-commitpassed.Layer 1):bazel coverage --combined_report=lcov //internal/redislock/... //cmd/study_controller/...— 100.0% line coverage oninternal/redislock/redislock.go(45/45),cmd/study_controller/worker/handler.go(97/97),cmd/study_controller/worker/consumer.go(49/49),payload.go(13/13),health.go(21/21), andconfig.go(38/38)../tools/analyze_diff_size.py --base-ref origin/main->344prod /480test (824total,threshold_exceeded: false).Live Deployment & Manual Testing (Verified on Stack Tip)
spannerbackend) upgraded viatools/dev_env/dev_env.py gke-upgrade --profile stable-2026b --allow-preview-profilewith the dev-only Vizier fixture (test/e2e/testdata/vizier/,vizier-service:7077).env_pool.py verify: All 5 sections✅ PASS(18/18Ready pods); anonymousGET /v1/health/configs-> HTTP401.check_components_live.py --require clickhouse --require vizier --require hpo:Ready,DependenciesReady --negative-gate: Exited0("ok": true;clickhouse,vizier,hpoReady=True,DependenciesReady=True; unacknowledged EXPERIMENTAL dry-run rejected at CRD admission).CreateExperiment(STUDY)Initial Tick,/readyz&/metrics:/healthzand/readyzreturned200 OK;CreateExperiment(STUDY)enqueued the initial tick,xmc-study-controllerloggedINFO "Processed study reconcile tick",outcome="reconciled"incremented0 -> 1, lock keyEXISTS == 0, andXPENDING == 0.xmc-study-controllertoreplicas=2; with a 10s demo lease held in Redis, 2 duplicate ticks loggedINFO "Skipped contended study reconcile tick"across both pods (outcome="contended"0 -> 2,reconciled+0,XPENDING == 0). After expiry, a 7-tick burst across 2 studies reconciled cleanly (reconciled2 -> 9,retry_error == 0,dlq_xlen == 0), and a malformed payload incrementedoutcome="invalid_payload"0 -> 1intoxmc-study-reconcile-dlq-stream. Restoredreplicas=1.hpo.enabled: false) & Restore:hpo.enabled: falseuninstalledxmc-cmp-hpo(0controller pods) and rejectedCreateExperiment(STUDY)withFailedPrecondition; restoringhpo.enabled: truereturned1/1 Readyandcheck_components_live.pyexited0.Related Issues / Bugs
Part 1 of 2 for bug 583731472
BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d