Skip to content

Add transport URL secret rotation with consumer finalizer - #537

Open
lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:finalize-secret-rotation
Open

Add transport URL secret rotation with consumer finalizer#537
lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:finalize-secret-rotation

Conversation

@lmiccini

@lmiccini lmiccini commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What this does

When infra-operator rotates a RabbitMQ transport URL (creating a new secret and user), consumer operators must hold a consumer finalizer on the old secret until all their pods have rolled out with the new credentials. Without this, infra-operator cleans up the old RabbitMQ user while pods are still connected with the old credentials, causing message-bus outages.

Approach (updated)

Note: this PR was reworked from the original grace-period design to an event-driven "applied input hash" design. The old approach released the finalizer after a fixed grace period based on the parent Ready condition, which was racy against the informer cache and could revoke credentials prematurely. The new approach releases the finalizer only once every child has provably rolled out.

  1. Add a consumer finalizer to the current transport URL secret early in reconcile. Status.TransportURLSecret is set for first-time setup only (empty or unchanged); during rotation the status is updated solely by FinalizeSecretRotation at the end of reconcile.

  2. Pass transportURL.Status.SecretName directly to sub-CR creation and config generation as a parameter — never read Status.TransportURLSecret for sub-CR specs.

  3. Each child controller records an AppliedInputSecretHash in its status, set only after statefulset.IsReadyForInput / deployment.IsReadyForInput confirms — via an uncached API read — that the workload is fully rolled out with the expected CONFIG_HASH.

  4. The parent mirrors a child's Ready condition only when its Generation == ObservedGeneration and AppliedInputSecretHash matches the current input hash; otherwise it sets the condition to Unknown.

  5. Guard: FinalizeSecretRotation removes the consumer finalizer from the old secret only when every child reports the expected hash and is ready. The guard is computed from Conditions.AllSubConditionIsTrue() (not IsReady()), because Conditions.Init() resets the Ready condition to Unknown on every reconcile.

The same pattern applies to notification transport URL secrets and application-credential secrets where applicable.

Dependency

Depends on the lib-common IsReadyForInput / FinalizeSecretRotation helpers (currently pinned via a replace to the fork commit while the lib-common PR is in review).

@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmiccini
Once this PR has been reviewed and has the lgtm label, please assign abays for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch 2 times, most recently from a34d9e1 to e8bb15f Compare August 13, 2026 09:39
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/e4bf123d038c45b7a0926e5e24aaf44f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 52m 50s
✔️ swift-operator-kuttl SUCCESS in 51m 32s
swift-operator-tempest FAILURE in 1h 53m 06s

@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch from 42e43fe to e8bb15f Compare August 14, 2026 09:49
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/576126539a6b4822a293bb152a34153b

openstack-k8s-operators-content-provider FAILURE in 8m 45s
⚠️ swift-operator-kuttl SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ swift-operator-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch from e8bb15f to 51ae872 Compare August 17, 2026 08:42
@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch from 51ae872 to 562e93c Compare August 17, 2026 09:13
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/00c9da1a705645d9a99671149b7de770

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 41m 29s
✔️ swift-operator-kuttl SUCCESS in 50m 57s
swift-operator-tempest FAILURE in 39m 03s

@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch 2 times, most recently from c5e2713 to 27c018b Compare August 20, 2026 09:02
@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 25bc4574-bac4-4b92-8b3c-ce87693c6b27


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/7bb05911bac945a189b8c40c8106ab20

✔️ openstack-k8s-operators-content-provider SUCCESS in 25m 59s
swift-operator-kuttl NODE_FAILURE Node(set) request 099-0000176041 failed in 0s
swift-operator-tempest NODE_FAILURE Node(set) request 099-0000176042 failed in 0s

When infra-operator rotates a RabbitMQ transport URL (creating a new
secret and user), consumer operators must hold a consumer finalizer on
the old secret until all their pods have rolled out with the new
credentials. Without this, infra-operator cleans up the old RabbitMQ
user while pods are still connected with old credentials, causing
message bus outages.

Design:

1. Add a consumer finalizer to the current transport URL secret early
   in reconcile. Set instance.Status.TransportURLSecret for first-time
   setup only (empty or unchanged); during rotation the status is
   updated solely by FinalizeSecretRotation at the end of reconcile.

2. Pass transportURL.Status.SecretName directly to sub-CR creation
   functions and config generation as a parameter — never read from
   instance.Status.TransportURLSecret for sub-CR specs.

3. Each child (sub-CR) controller records an AppliedInputSecretHash in
   its status, set only after statefulset.IsReadyForInput /
   deployment.IsReadyForInput confirms — via an uncached API read —
   that the workload is fully rolled out with the expected CONFIG_HASH.

4. The parent mirrors a child's Ready condition only when its
   Generation == ObservedGeneration and AppliedInputSecretHash matches
   the current input hash; otherwise it sets the condition to Unknown.

5. Guard: FinalizeSecretRotation removes the consumer finalizer from
   the old secret only when every child reports the expected hash and
   is ready.

The same pattern applies to notification transport URL secrets and
application credential secrets where applicable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lmiccini
lmiccini force-pushed the finalize-secret-rotation branch from 27c018b to bdc78ae Compare August 24, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant