Skip to content

feat(study-controller): guard study reconcile ticks with per-study Redis lock - #1707

Draft
VLukiantsev wants to merge 2 commits into
mainfrom
feat/hpo-reconcile-lock-core-b583731472
Draft

VLukiantsev wants to merge 2 commits into
mainfrom
feat/hpo-reconcile-lock-core-b583731472

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Introduces internal/redislock (SET ... NX PX with a 16-byte crypto/rand token and token-checked Lua DEL release) and wires per-study mutual exclusion (xmc:study-reconcile-lock:{xid}) into xmc-study-controller (Sub-slice S1.3: trials run and the study finishes — random search with max_trial_budget=5 runs 5 JobSets with injected parameters and transitions to FINISHED). Shares a single *redis.Client pool across stream consumption, /readyz, and locking, acknowledges contended duplicate ticks without reconciling, and exposes xmc_study_controller_reconcile_ticks_total{outcome} on GET /metrics.

Motivation & Context

xmc-study-controller is stateless and consumes unsigned, at-least-once reconcile hints from xmc-study-reconcile-stream. Before trial suggestion and BatchCreateWorkUnits dispatch land (and before hpo scales beyond replicas: 1), concurrent ticks for the same study across replicas or redeliveries must never reconcile simultaneously.

This is Layer 1 of a 2-PR gh-stack delivering the core non-blocking per-study lease lock, contention routing, single-pool wiring, and observability (344 prod / 480 test lines); Layer 2 (#1708, feat/hpo-reconcile-lock-heartbeat-b583731472) adds periodic Lua PEXPIRE heartbeat renewal, lost-lease context.WithCancelCause fencing, and the testcontainers-go real-Redis integration suite.

Key Changes

  • internal/redislock: Added generic non-blocking Redis lease lock (New, Locker.Acquire, Lease.Release, ErrNotAcquired, ErrLeaseLost) using SET key token NX PX with a 128-bit crypto/rand hex token and an atomic Lua compare-and-delete release script.
  • cmd/study_controller & cmd/study_controller/worker:
    • Shared a single *redis.Client connection pool (ContextTimeoutEnabled: true) across ConsumerClient, /readyz, and NewRedisHandler.
    • Wrapped each study tick in xmc:study-reconcile-lock:{xid} (15s TTL, 60s reconcile timeout strictly below the 2m stream consumer timeout, and 2s detached context.WithoutCancel release timeout; consumer drain timeout raised from 8s to 10s).
    • Acknowledged contended ticks (ErrNotAcquired -> WasRunSuccessful: true, outcome="contended") without reconciling, while returning retryable failures (ShouldBeRetried: true, outcome="retry_error") on Redis transport or reconcile errors.
    • Exported xmc_study_controller_reconcile_ticks_total{outcome="reconciled|contended|retry_error|invalid_payload"} pre-initialized on /metrics and logged reconciled/contended ticks at INFO.
  • Documentation: Updated deploy/profiles/stable-2026b/components/hpo/README.md, docs/flows/experiment_lifecycle.md, docs/flows/metrics_and_observability.md, and docs/architecture/components.md.

Assumptions

  • Gate matrix:
    • Components profile: stable-2026b (preview; hpo and vizier ship only in stable-2026b) / stable-2026a (GA-locked without hpo, so CreateExperiment(STUDY) fails closed with FailedPrecondition; XMC Components are always-on in 1.0.0+ with no enable_components flag).
    • hpo / vizier Component state scope: When hpo is enabled=true (acknowledged with vizier and clickhouse Ready=True in stable-2026b), xmc-study-controller acquires xmc:study-reconcile-lock:{xid} around each tick and exposes xmc_study_controller_reconcile_ticks_total; if vizier or clickhouse later goes unready, the running controller remains in place (/readyz checks Redis only in this slice) and retries Redis lock errors without dead-lettering; when hpo is enabled=false or absent, no xmc-study-controller pod runs, zero lock keys are written, and CreateExperiment(STUDY) returns FailedPrecondition (both enabled=true and enabled=false states verified live on GKE).
    • database_backend: Both spanner and postgres are supported (this change is Redis-only with zero SQL/DDL changes; spanner was verified live on GKE).
    • Metrics store: clickhouse (v1, Ready=True on live GKE) / bigquery (Unimplemented for HPO queries; not queried in this slice).
  • No-op reconcile scaffold (noopReconcile) — ESS study re-read, Vizier CreateStudy, and trial dispatch land in follow-up slices — In Layer 1, valid lock-holding ticks execute the noopReconcile stub well within the 15s initial TTL, while background PEXPIRE heartbeat renewal for long-running reconciles lands immediately upstack in Layer 2 before remote RPCs are added.
  • Info-level tick logging — Steady-state ticks are currently emitted once per CreateExperiment(STUDY) before 10s delayed polling is wired — Once 10s periodic study polling lands, steady-state tick logs should be downgraded from INFO to DEBUG so multi-study clusters rely on xmc_study_controller_reconcile_ticks_total without high log volume.

Verification & Testing

Automated Tests

  • Hermetic Unit & Race Suite (Layer 1): bazel test --@io_bazel_rules_go//go/config:race //internal/redislock/... //cmd/study_controller/... (5/5 targets passed) and ./.githooks/pre-commit passed.
  • Code Coverage (Layer 1): bazel coverage --combined_report=lcov //internal/redislock/... //cmd/study_controller/... — 100.0% line coverage on internal/redislock/redislock.go (45/45), cmd/study_controller/worker/handler.go (97/97), cmd/study_controller/worker/consumer.go (49/49), payload.go (13/13), health.go (21/21), and config.go (38/38).
  • Diff Sizing: ./tools/analyze_diff_size.py --base-ref origin/main -> 344 prod / 480 test (824 total, threshold_exceeded: false).

Live Deployment & Manual Testing (Verified on Stack Tip)

  • Deployment Target: GKE dev cluster (env-pool slot, spanner backend) upgraded via tools/dev_env/dev_env.py gke-upgrade --profile stable-2026b --allow-preview-profile with the dev-only Vizier fixture (test/e2e/testdata/vizier/, vizier-service:7077).
  • Cluster Health & Components Gate:
    • env_pool.py verify: All 5 sections ✅ PASS (18/18 Ready pods); anonymous GET /v1/health/configs -> HTTP 401.
    • check_components_live.py --require clickhouse --require vizier --require hpo:Ready,DependenciesReady --negative-gate: Exited 0 ("ok": true; clickhouse, vizier, hpo Ready=True, DependenciesReady=True; unacknowledged EXPERIMENTAL dry-run rejected at CRD admission).
  • Manual Test Scenarios Executed:
    1. End-to-End CreateExperiment(STUDY) Initial Tick, /readyz & /metrics: /healthz and /readyz returned 200 OK; CreateExperiment(STUDY) enqueued the initial tick, xmc-study-controller logged INFO "Processed study reconcile tick", outcome="reconciled" incremented 0 -> 1, lock key EXISTS == 0, and XPENDING == 0.
    2. Two-Replica Contention, 7-Tick Multi-Study Burst & Malformed-Payload DLQ: Scaled xmc-study-controller to replicas=2; with a 10s demo lease held in Redis, 2 duplicate ticks logged INFO "Skipped contended study reconcile tick" across both pods (outcome="contended" 0 -> 2, reconciled +0, XPENDING == 0). After expiry, a 7-tick burst across 2 studies reconciled cleanly (reconciled 2 -> 9, retry_error == 0, dlq_xlen == 0), and a malformed payload incremented outcome="invalid_payload" 0 -> 1 into xmc-study-reconcile-dlq-stream. Restored replicas=1.
    3. Component Disabled State (hpo.enabled: false) & Restore: hpo.enabled: false uninstalled xmc-cmp-hpo (0 controller pods) and rejected CreateExperiment(STUDY) with FailedPrecondition; restoring hpo.enabled: true returned 1/1 Ready and check_components_live.py exited 0.

Related Issues / Bugs

Part 1 of 2 for bug 583731472

BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d

…dis lock

Introduce distributed per-study lease locking in the Study Controller so
multi-replica deployments never reconcile the same study concurrently when
consuming at-least-once ticks from xmc-study-reconcile-stream.

- Add internal/redislock to acquire non-blocking leases with SET ... NX PX
  and random 16-byte hex tokens, releasing keys via a token-checked Lua DEL
  script so an expired lease never deletes a newer holder's lock.
- Replace the raw consumer task callback in cmd/study_controller/worker with
  Handler and NewRedisHandler, sharing the consumer's single *redis.Client
  connection pool across stream consumption, per-study locking, and /readyz.
- Acknowledge contended ticks (ErrNotAcquired) immediately without retrying,
  dead-letter malformed payloads before lock acquisition, and retry transient
  lock or reconcile failures under a 60s reconcile timeout (< 2m stream
  process timeout).
- Detach lease release from parent shutdown cancellation with a 2s bound and
  extend consumer drain timeout to 10s so in-flight ticks release their lock
  and settle cleanly on SIGTERM.
- Expose xmc_study_controller_reconcile_ticks_total{outcome} on /metrics.

BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a distributed locking mechanism using Redis to prevent concurrent study reconciliations in the study controller, along with Prometheus metrics to track processing outcomes. The feedback highlights two key issues: first, the default lease TTL is shorter than the default reconcile timeout, which could lead to premature lock expiration and concurrent processing; second, the isNilClient check only handles typed nil pointers for *redis.Client, which could cause nil pointer panics for other types implementing the Client interface. Using reflection is recommended to make the nil check robust.

Comment on lines +34 to +35
defaultLeaseTTL = 15 * time.Second
defaultReconcileTimeout = 60 * time.Second

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The defaultLeaseTTL (15s) is shorter than the defaultReconcileTimeout (60s). Since there is no active lease renewal (heartbeat) mechanism, any reconciliation taking longer than 15 seconds will lose its lock. This allows another replica to acquire the lock for the same study and run a concurrent reconciliation, violating the mutual exclusion guarantee.

To ensure mutual exclusion throughout the entire reconciliation process, the lease TTL should be at least as long as the maximum reconciliation timeout (plus a small buffer for safety, e.g., 65s).

Suggested change
defaultLeaseTTL = 15 * time.Second
defaultReconcileTimeout = 60 * time.Second
defaultLeaseTTL = 65 * time.Second
defaultReconcileTimeout = 60 * time.Second

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping defaultLeaseTTL = 15s intentionally: in Layer 1 (#1707), Handler.reconcile is noopReconcile (zero I/O, completes in sub-microseconds well within 15s), and stacked Layer 2 (#1708) immediately adds the 5s (TTL/3) token-checked Lua PEXPIRE heartbeat renewal loop with context.WithCancelCause lost-lease fencing before any remote RPCs exist. Raising defaultLeaseTTL to 65s would leave a 65s orphaned lock on hard pod crash (SIGKILL/OOM), stalling promoted 10s ticks until the 5-minute stream PEL reclamation window.

Comment on lines +19 to +28
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"time"

"github.com/redis/go-redis/v9"
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To support robust nil checking for any concrete type implementing the Client interface (including custom test doubles or future client implementations like *redis.ClusterClient), we should use reflection in isNilClient. Please add "reflect" to the import block.

Suggested change
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"time"
"github.com/redis/go-redis/v9"
)
import (
"context"
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"reflect"
"time"
"github.com/redis/go-redis/v9"
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).

Comment on lines +67 to +73
func isNilClient(client Client) bool {
if client == nil {
return true
}
c, ok := client.(*redis.Client)
return ok && c == nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current isNilClient implementation only checks for typed nil pointers of type *redis.Client. If a caller passes a typed nil pointer of another type implementing Client (such as *redis.ClusterClient or a custom test double like *stubLockClient), isNilClient will return false, leading to a nil pointer panic when SetArgs is called.

Using reflection allows us to safely detect typed nil pointers for any concrete type implementing the interface.

Suggested change
func isNilClient(client Client) bool {
if client == nil {
return true
}
c, ok := client.(*redis.Client)
return ok && c == nil
}
func isNilClient(client Client) bool {
if client == nil {
return true
}
v := reflect.ValueOf(client)
return v.Kind() == reflect.Ptr && v.IsNil()
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).

@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a distributed Redis lease lock mechanism (internal/redislock) and integrates it into the study_controller worker to process study reconcile ticks under a per-study lock. It also adds Prometheus metrics to track processing outcomes and updates relevant documentation and tests. The review feedback suggests improving the robustness of the isNilClient helper by using reflection to detect any typed nil pointer, and swapping the defer order in reconcileWithLease to ensure context resources are freed immediately.

Comment on lines +67 to +73
func isNilClient(client Client) bool {
if client == nil {
return true
}
c, ok := client.(*redis.Client)
return ok && c == nil
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The current isNilClient helper only checks if the interface is a typed nil for *redis.Client. If a caller passes a typed nil of another type that implements Client (such as *redis.ClusterClient or a custom mock/wrapper), isNilClient will return false. This will subsequently cause a nil pointer dereference panic when SetArgs or Eval is called on the client.

Using reflection allows us to safely check if any pointer type implementing the interface is nil. Note that you will need to import "reflect" at the top of the file.

Suggested change
func isNilClient(client Client) bool {
if client == nil {
return true
}
c, ok := client.(*redis.Client)
return ok && c == nil
}
func isNilClient(client Client) bool {
if client == nil {
return true
}
val := reflect.ValueOf(client)
return val.Kind() == reflect.Ptr && val.IsNil()
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushing back per .agents/rules/languages/golang.md §1 ("Avoid unnecessary reflection"): *redis.Client is the sole concrete single-instance Redis client in the repository (enforced by var _ Client = (*redis.Client)(nil)), whereas reflect.ValueOf adds runtime reflection overhead (and Kind() == reflect.Ptr still misses other nilable Go kinds like funcs/maps/slices).

Comment thread cmd/study_controller/worker/handler.go Resolved
Swap LIFO defer ordering in reconcileWithLease so the reconcile context
timer is cancelled immediately upon reconcile return before the detached
Redis lease release RPC executes.

Comment link: #1707 (comment)

TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant