Skip to content

feat(study-controller): renew per-study reconcile leases with lost-lease fencing - #1708

Draft
VLukiantsev wants to merge 2 commits into
feat/hpo-reconcile-lock-core-b583731472from
feat/hpo-reconcile-lock-heartbeat-b583731472
Draft

VLukiantsev wants to merge 2 commits into
feat/hpo-reconcile-lock-core-b583731472from
feat/hpo-reconcile-lock-heartbeat-b583731472

Conversation

@VLukiantsev

@VLukiantsev VLukiantsev commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds token-checked Lua PEXPIRE lease renewal ((*redislock.Lease).Extend), a 5s (TTL/3) background heartbeat ticker with context.WithCancelCause lost-lease fencing in xmc-study-controller, and a testcontainers-go real-Redis integration suite (Sub-slice S1.3: trials run and the study finishes — random search with max_trial_budget=5 runs 5 JobSets with injected parameters and transitions to FINISHED).

Motivation & Context

Stacked on #1707 (feat/hpo-reconcile-lock-core-b583731472, Layer 1 of 2) so the core non-blocking per-study lock and metrics land within repository PR sizing limits (344 prod lines in Layer 1; 228 prod lines in Layer 2) with zero dead code in either layer.

Keeping leaseTTL = 15s ensures a hard-crashed controller pod's orphaned lock expires quickly (<= 15s rather than stalling promoted 10s ticks until the 5-minute stream PEL reclamation window), while reconciles that invoke remote ESS, Vizier, and ClickHouse RPCs can run up to reconcileTimeout = 60s. Renewing the lease every 5s extends long reconciles safely and immediately cancels in-flight work (context.WithCancelCause) if the lease is lost or Redis becomes unresponsive (5s per-renewal timeout gives a 5s safety cushion before the 15s TTL expires).

Key Changes

  • internal/redislock: Added (*Lease).Extend(ctx) backed by an atomic token-checked Lua GET/PEXPIRE script, returning ErrLeaseLost when the key expired or changed owner.
  • cmd/study_controller/worker:
    • Added HandlerParams.RenewInterval (default 5s = TTL/3) and renewLoop, bounding each Extend call by renewInterval, cancelling reconcileCtx via context.WithCancelCause on any renewal failure, suppressing expected outer renewCtx cancellations/timeouts (renewCtx.Err() != nil && errors.Is(err, renewCtx.Err())) so graceful shutdown and reconcile deadline expiry log zero false renewal warnings while genuine ErrLeaseLost or hung-Redis extendCtx timeouts still fence and retry, awaiting wg.Wait() before releaseLease so Extend and Release never race, and skipping redundant Release when ErrLeaseLost was already observed.
  • internal/redistest & Real-Redis Integration Tests (tags = ["local"]):
    • Added testonly internal/redistest (RunMain) launching redis:6-alpine via testcontainers-go and explicitly calling testcontainers.TerminateContainer on a fresh 15s cleanup context (TESTCONTAINERS_RYUK_DISABLED=true CI safe).
    • Added internal/redislock:redislock_redis_test (Lua Acquire/Extend/Release, stale-holder token fencing after expiry + re-acquisition, 8-goroutine mutual exclusion) and cmd/study_controller/worker:worker_redis_test (2-replica stream contention ACK without overlap, cross-study xid non-blocking concurrency, and live lease extension past initial TTL).
  • Documentation: Updated deploy/profiles/stable-2026b/components/hpo/README.md and docs/architecture/components.md.

Assumptions

  • Gate matrix:
    • Components profile: stable-2026b (preview; hpo and vizier ship only in stable-2026b) / stable-2026a (GA-locked without hpo, so CreateExperiment(STUDY) fails closed with FailedPrecondition; XMC Components are always-on in 1.0.0+ with no enable_components flag).
    • hpo / vizier Component state scope: When hpo is enabled=true (acknowledged with vizier and clickhouse Ready=True in stable-2026b), xmc-study-controller renews xmc:study-reconcile-lock:{xid} every 5s up to 60s and cancels in-flight reconciles on lost leases; if vizier or clickhouse later goes unready, the running controller remains in place (/readyz checks Redis only in this slice) and retries Redis lock/renewal errors; when hpo is enabled=false or absent, no xmc-study-controller pod runs, zero lock keys are written, and CreateExperiment(STUDY) returns FailedPrecondition (both enabled=true and enabled=false states verified live on GKE).
    • database_backend: Both spanner and postgres are supported (this change is Redis-only with zero SQL/DDL changes; spanner was verified live on GKE).
    • Metrics store: clickhouse (v1, Ready=True on live GKE) / bigquery (Unimplemented for HPO queries; not queried in this slice).
  • 5s renewal interval on a 15s TTL (TTL/3) — Bounds each Extend RPC to 5s (ContextTimeoutEnabled: true) so a hung Redis aborts the reconcile at t <= 10s (>= 5s before t = 15s lock expiry) — Extreme host pause/GC stalls exceeding the remaining 5s–15s lease window still rely on downstream ESS/Vizier idempotency (client_id and principal/state checks).
  • No-op reconcile scaffold (noopReconcile) — ESS study re-read, Vizier CreateStudy, and trial dispatch land in follow-up slices — Automated unit and real-Redis container tests exercise multi-renewal and lost-lease cancellation directly via injected ReconcileFuncs until remote RPCs are wired in.

Verification & Testing

Automated Tests

  • Unit, Race & Real-Redis Testcontainers Suite (Layer 2): bazel test --@io_bazel_rules_go//go/config:race --test_env=TESTCONTAINERS_RYUK_DISABLED=true //internal/redistest/... //internal/redislock/... //cmd/study_controller/... (8/8 targets passed, including redistest_test, redislock_redis_test, and worker_redis_test) and ./.githooks/pre-commit passed.
  • Code Coverage (Layer 2): bazel coverage --combined_report=lcov --test_env=TESTCONTAINERS_RYUK_DISABLED=true //internal/redistest/... //internal/redislock/... //cmd/study_controller/... — 100.0% line coverage on internal/redislock/redislock.go (54/54), cmd/study_controller/worker/handler.go (140/140), and cmd/study_controller/worker/consumer.go (49/49); 93.75% (30/32) on internal/redistest/redistest.go.
  • Diff Sizing: ./tools/analyze_diff_size.py --base-ref feat/hpo-reconcile-lock-core-b583731472 -> 228 prod / 574 test (802 total, threshold_exceeded: false).

Live Deployment & Manual Testing

  • Deployment Target: GKE dev cluster (env-pool slot) upgraded via tools/dev_env/dev_env.py gke-upgrade --profile stable-2026b --allow-preview-profile with the dev-only Vizier fixture (test/e2e/testdata/vizier/, vizier-service:7077).
  • Gate Matrix Verified Live:
    • Components profile: stable-2026b (preview; hpo and vizier ship only in stable-2026b).
    • hpo / vizier states: enabled+Ready (Ready=True, DependenciesReady=True), enabled+unacknowledged (rejected at CRD admission via server-side dry-run --negative-gate), and disabled (hpo.enabled: false uninstalls xmc-cmp-hpo and CreateExperiment(STUDY) returns FailedPrecondition).
    • database_backend: spanner live-verified on GKE (ESS schema v19, XID schema v1, zero pending migrations; Study Controller introduces no SQL/DDL changes so spanner and postgres remain at full parity).
    • Metrics store: clickhouse (Ready=True).
    • KinD E2E status: The HPO KinD E2E harness is not yet present under test/e2e/; automated coverage is provided by hermetic unit tests and real Redis (redis:6-alpine via testcontainers-go) multi-consumer integration tests (bazel test //internal/redistest/... //internal/redislock/... //cmd/study_controller/...), complemented by the live GKE scenarios below.
  • Cluster Health & Security Self-Check:
    • env_pool.py verify: All 5 sections ✅ PASS (18/18 Ready pods, all 5 Component CRs verified).
    • Anonymous API rejection (GET /v1/health/configs): HTTP 401.
    • Local Terraform state/artifact permissions: 700 on ctk-out, 600 on xmc_app/terraform.tfstate.
    • Components Health & Negative Admission Gate (check_components_live.py --require clickhouse --require vizier --require hpo:Ready,DependenciesReady --negative-gate): Exited 0 ("ok": true).
  • Manual Test Scenarios Executed:
    1. Scenario A — End-to-End CreateExperiment(STUDY) Initial Tick, /readyz & /metrics:
      • Action: Probed :8080/healthz, :8080/readyz, and :8080/metrics on xmc-study-controller, then invoked ExperimentStateService/CreateExperiment (experimentType: "STUDY", algorithm: "RANDOM_SEARCH") via grpcurl using an impersonated service account OIDC token.
      • Result: /healthz and /readyz returned 200 OK; /metrics pre-initialized all four xmc_study_controller_reconcile_ticks_total outcome label values (reconciled, contended, retry_error, invalid_payload) at 0. Within ~215 ms of CreateExperiment, xmc-study-controller logged INFO "Processed study reconcile tick", outcome="reconciled" incremented 0 -> 1 (+1), EXISTS xmc:study-reconcile-lock:<study-1> returned 0 (lease cleanly released), and XPENDING xmc-study-reconcile-stream study-controller-group returned 0.
    2. Scenario B — Two-Replica Distributed Lock Contention, Pipelined Burst & Malformed-Payload DLQ:
      • Action: Scaled deployment/xmc-study-controller to replicas=2 (verifying xmc-bridge preserved replicas=2 and both pods reached 1/1 Ready). Held a 10s lease (SET xmc:study-reconcile-lock:<study-1> demo-holder NX PX 10000) and pipelined 2 duplicate ticks for <study-1>; after lease expiry, pipelined a 7-tick burst (6 ticks for <study-1> + 1 tick for <study-2>) across both active replicas; then enqueued 1 malformed tick ({"experiment":"not-an-experiment"}); finally scaled back to replicas=1.
      • Result:
        • Contention: Both replicas (pod-a and pod-b) logged INFO "Skipped contended study reconcile tick" with distinct worker_ids; cluster-summed outcome="contended" incremented 0 -> 2 (+2) while outcome="reconciled" remained unchanged (+0), and XPENDING remained 0.
        • Burst: All 7 ticks after lease expiry processed cleanly across both replicas (pod-a +5, pod-b +2) with retry_error=0, XPENDING=0, dlq_xlen=0, and both per-study locks released (EXISTS=0).
        • Invalid payload: Logged WARN "Dropping invalid study reconcile tick" and "Task moving to DLQ" (reason="non_retryable_error"), outcome="invalid_payload" incremented 0 -> 1 (+1), XPENDING=0, and XLEN xmc-study-reconcile-dlq-stream incremented 0 -> 1.
    3. Scenario C — Component Disabled State (hpo.enabled: false) & Clean Restore:
      • Action: Deployed profile overlay with components.hpo.enabled: false via gke-upgrade --only app, checked Helm releases, pods, and CreateExperiment(STUDY), then restored hpo.enabled: true (acknowledgeExperimental: true) and re-ran check_components_live.py.
      • Result: With hpo.enabled: false, xmc-bridge uninstalled xmc-cmp-hpo, terminated all xmc-study-controller pods (0 pods), updated Component/hpo to COMPONENT_STATE_DISABLED (Synced=True), and CreateExperiment(STUDY) failed closed with gRPC FailedPrecondition (component "hpo" is in state COMPONENT_STATE_DISABLED, required ENABLED). Restoring hpo.enabled: true recreated xmc-study-controller (1/1 Ready) and check_components_live.py exited 0.
Live Execution Output / Logs (Sanitized)
=== Structured Verification Summary ===
✅ PASS | 1. Kubernetes Pods         | 18/18 Ready
✅ PASS | 2. Kubernetes Services     | Core ClusterIP services present
✅ PASS | 3. Gateway & HTTPRoutes    | api-route, grpc-route, ui-route bound
✅ PASS | 4. TLS Certificates        | validator-ca Ready=True
✅ PASS | 5. Log Monitor (GKE)       | Runs as GKE Deployment; Cloud Run check skipped
=======================================

=== HPO Components Health & Negative Admission Gate ===
{
  "ok": true,
  "results": [
    { "component": "clickhouse", "ok": true, "found": true, "conditions": { "Ready": "True" } },
    { "component": "vizier", "ok": true, "found": true, "conditions": { "Ready": "True" } },
    { "component": "hpo", "ok": true, "found": true, "conditions": { "Ready": "True", "DependenciesReady": "True" } },
    {
      "check": "negative_gate",
      "ok": true,
      "verdict": "rejected",
      "detail": "The Component \"gate-probe-bb0bbfdd\" is invalid: spec: Invalid value: spec.acknowledgeExperimental must be set to true when enabling an EXPERIMENTAL component (spec.stability == 'EXPERIMENTAL')."
    }
  ]
}

=== Scenario A: Initial Study Tick ===
{"severity":"INFO","message":"Processed study reconcile tick","worker_id":"study-controller-worker-xmc-study-controller-<pod-a>","logging.googleapis.com/labels":{"xid":"<study-1>"}}
sum_reconcile_ticks_total: {"contended": 0.0, "invalid_payload": 0.0, "reconciled": 1.0, "retry_error": 0.0}
lock_exists["xmc:study-reconcile-lock:<study-1>"]: 0, xpending_count: 0

=== Scenario B: Two-Replica Contention, 7-Tick Burst & Malformed Payload DLQ ===
{"severity":"INFO","message":"Skipped contended study reconcile tick","worker_id":"study-controller-worker-xmc-study-controller-<pod-a>","logging.googleapis.com/labels":{"xid":"<study-1>"}}
{"severity":"INFO","message":"Skipped contended study reconcile tick","worker_id":"study-controller-worker-xmc-study-controller-<pod-b>","logging.googleapis.com/labels":{"xid":"<study-1>"}}
post-contention sum_reconcile_ticks_total: {"contended": 2.0, "invalid_payload": 0.0, "reconciled": 1.0, "retry_error": 0.0}, xpending_count: 0
post-burst sum_reconcile_ticks_total:      {"contended": 2.0, "invalid_payload": 0.0, "reconciled": 9.0, "retry_error": 0.0}, xpending_count: 0, dlq_xlen: 0
{"severity":"WARN","message":"Dropping invalid study reconcile tick: invalid experiment in study reconcile payload: resource name must be in the format 'experiments/{xid}': rpc error: code = InvalidArgument desc = invalid argument","worker_id":"study-controller-worker-xmc-study-controller-<pod-a>","payload_bytes":35}
{"severity":"WARN","message":"Task moving to DLQ","taskID":"1794925150391-0","reason":"non_retryable_error"}
post-invalid sum_reconcile_ticks_total:    {"contended": 2.0, "invalid_payload": 1.0, "reconciled": 9.0, "retry_error": 0.0}, xpending_count: 0, dlq_xlen: 1

=== Scenario C: Component Disabled (hpo.enabled=false) ===
hpo   COMPONENT_STATE_DISABLED   False   True   hyperparameter-optimization
ERROR:
  Code: FailedPrecondition
  Message: failed to check component requirements: component "hpo" is in state COMPONENT_STATE_DISABLED, required ENABLED: rpc error: code = FailedPrecondition desc = component requirement not met

Related Issues / Bugs

Fixes bug 583731472

BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces distributed lease renewal for the study controller's reconcile lock, allowing long-running reconciliations to periodically extend their lease. It adds an Extend method to redislock.Lease backed by a Lua script, implements a background renewLoop in the worker handler, and introduces a new internal/redistest package to run integration tests against real Redis containers using testcontainers-go. Feedback on the changes suggests refining the error handling in the renewLoop to avoid logging false-positive warnings when the context is cancelled or times out during normal worker shutdown or timeout boundaries.

Comment thread cmd/study_controller/worker/handler.go Resolved
@VLukiantsev

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements distributed Redis lease lock renewal to prevent locks from expiring during long-running study reconciliations. It adds an Extend method to the redislock package, implements a background renewal loop in the worker handler, and introduces a redistest package using testcontainers-go for integration testing. The reviewer identified an issue where misleading warnings could be logged during normal worker shutdown or context timeouts in the renewal loop, and suggested checking renewCtx.Err() directly to distinguish expected cancellations from genuine failures.

Comment thread cmd/study_controller/worker/handler.go Resolved
…ase fencing

Extend per-study Redis reconcile leases periodically while reconciliation is
in progress so ticks running up to the 60s reconcile timeout hold their 15s
lease without risking premature expiration, while immediately fencing work
if ownership is lost.

- Add (*redislock.Lease).Extend using a token-checked Lua PEXPIRE script that
  returns ErrLeaseLost if the key expired or changed owner.
- Run a 5s (TTL/3) heartbeat renewal loop inside worker.Handler using
  context.WithCancelCause to cancel in-flight reconciliation immediately on
  renewal failure, skipping unlock when ErrLeaseLost proves another worker may
  own the key and ignoring benign context.Canceled races during normal
  reconcile teardown.
- Add internal/redistest to manage disposable redis:6-alpine containers with
  explicit Ryuk-free termination (TESTCONTAINERS_RYUK_DISABLED=true), plus
  real-Redis integration suites verifying token-fenced lease semantics and
  multi-replica contention, cross-study isolation, and multi-renewal ticks.

BUG=b/583731472
TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
…cellation

Distinguish outer renewCtx cancellation and reconcile timeouts from
genuine Redis lease renewal failures in renewLoop using renewCtx.Err(),
avoiding misleading renewal warnings during graceful shutdown or
reconcile deadline expiry while preserving hung-Redis and ErrLeaseLost
fencing.

Comment link: #1708 (comment)
Comment link: #1708 (comment)

TAG=agy
CONV=dafb63f4-0f3c-404e-b8fc-511e6baf7d1d
@VLukiantsev
VLukiantsev force-pushed the feat/hpo-reconcile-lock-heartbeat-b583731472 branch from 8f59529 to df331d4 Compare November 17, 2026 14:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant