[FR-126] Rollback by eniko-dif · Pull Request #274 · temporalio/temporal-worker-controller

eniko-dif · 2026-04-14T15:35:18Z

What was changed

Add a separate rollback config to the deployment specification that can be used to have different rollback behavior from the rollout one. It supports AllAtOnce or Progressive and does not have a gate. Uses the LastCurrentTime of a deployment to decide if the target version supposed to be rolled out or rolled back.

Why?

Open feature issue: #126

Checklist

Closes [Feature Request] Rollback mode #126
How was this tested:

Unit and integration tests (ongoing actual test with local setup).

Any docs updates needed?

Yes, but needs to be decided where.

CLAassistant · 2026-04-14T15:35:28Z

All committers have signed the CLA.

jlegrone · 2026-05-18T17:50:14Z

+	}
+
+	if config.RollbackStrategy.MaxVersionAge != nil && time.Since(*targetVersionInfo.LastCurrentTime) > config.RollbackStrategy.MaxVersionAge.Duration {
+		l.Info("Skipping rollback: the version's last current time exceeds MaxVersionAge",


Suggested change

l.Info("Skipping rollback: the version's last current time exceeds MaxVersionAge",

l.Debug("Skipping rollback: the version's last current time exceeds MaxVersionAge",

The logger has no Debug level, only Info and Error. (If I remember correctly, it was related to K8s events?)

jlegrone · 2026-05-18T18:07:06Z

 	}

 	allErrs = append(allErrs, validateRolloutStrategy(new.Spec.RolloutStrategy)...)
+	allErrs = append(allErrs, validateRollbackStrategy(*new.Spec.RollbackStrategy)...)


Could new.Spec.RollbackStrategy ever be nil? Should probably add a nil check here or update the type signature for validateRollbackStrategy instead of dereferencing.

The rollback strategy is a pointer, because it's not mandatory in the spec. At the point of the validation it should not be nil, because of the default we set, but just in case, I added a nil check.

jlegrone · 2026-05-18T18:20:05Z

+	if s.RollbackStrategy == nil {
+		s.RollbackStrategy = &RollbackStrategy{Strategy: RollbackAllAtOnce}
+	} else if s.RollbackStrategy.Strategy == "" {
+		s.RollbackStrategy.Strategy = RollbackAllAtOnce
+	}


Nit: this is entirely a matter of taste but I think two if statements is easier to read than the if/else flow (more clear that the strategy is always all at once if left unset).

Suggested change

if s.RollbackStrategy == nil {

s.RollbackStrategy = &RollbackStrategy{Strategy: RollbackAllAtOnce}

} else if s.RollbackStrategy.Strategy == "" {

s.RollbackStrategy.Strategy = RollbackAllAtOnce

}

if s.RollbackStrategy == nil {

s.RollbackStrategy = &RollbackStrategy{}

}

if s.RollbackStrategy.Strategy == "" {

s.RollbackStrategy.Strategy = RollbackAllAtOnce

}

I've done the change, but I remember the times, where I had to optimize even the number of if calls... :P

jlegrone · 2026-05-18T18:23:33Z

+	default:
+		strategy = temporaliov1alpha1.UpdateAllAtOnce


🤔 I think this ought to be an error condition; if the controller sees an unknown strategy it should fall back to manual mode and stop attempting updates to the default version.

Before this function is used, there is a check to see if the rollout strategy is set to manual, and if it is, rollback is disabled. In any case, I updated the default to be manual.

jlegrone · 2026-05-18T18:55:02Z

+		},
+	}
+
+	testCases := []struct {


I think there should be a test cases for

1.RollbackStrategy == nil (even though the webhook should set a default, it's still possible to run the controller without the webhook enabled). When nil I would assert that the behavior matches the default rollback strategy configured by the webhook.
2. RollbackStrategy.MaxVersionAge == nil (assert default max version age applied, again to match the default webhook behavior)
3. RollbackStrategy.MaxVersionAge == 0 (assert false, even when the rest of the current status would normally have resulted in a rollback)

I changed the Config to have rollback strategy at all times, so the first two are no longer needed and the third existed already.

jlegrone · 2026-05-18T19:28:13Z

@@ -23,8 +24,9 @@ func TestTemporalWorkerDeployment_ValidateCreate(t *testing.T) {
 	tests := map[string]struct {
 		obj      runtime.Object
 		errorMsg string
+		warnMsg  string


Should there be a warning emitted for a combination of Manual rollout + Progressive rollback? That seems in the same vein as rollout steps being faster than rollback steps. Please add a test case for that if so.

When rollback is Manual, rollback is disabled in the planner, so I don't think we need a warning.

jaypipes

@eniko-dif thanks very much for this! I think the UX can be simplified quite a bit by just making the rollback strategy match the rollout strategy. Also, I recommend removing the MaxVersionAge configuration option. Instead, simply roll back to the previous WDV's build ID. I really don't see a use case for configuring a rollback target version of anything other than the previous version...

jaypipes · 2026-06-15T16:56:05Z

+	// not affected by the rollback. Only new workflow executions will be routed to the rollback
+	// target version.


Suggested change

// not affected by the rollback. Only new workflow executions will be routed to the rollback

// target version.

// not affected by the rollback. Only new workflow executions that are not pinned

// to a specific build ID will be routed to the rollback target version.

jaypipes · 2026-06-15T16:57:09Z

+	// not affected by the rollback. Only new workflow executions will be routed to the rollback
+	// target version.
+	//
+	// Rollback is suppressed when the rollout strategy is Manual.


I'm just curious, but why is rollback suppressed when the rollout stategy is Manual?

jaypipes · 2026-06-15T17:00:21Z

+	// +optional
+	Steps []RolloutStep `json:"steps,omitempty"`
+
+	// MaxVersionAge limits which versions are eligible as rollback targets.


I have a suspicion this is overly complicated configuration for most users. Would a simpler "just roll back to the previous working (i.e. passes gates) version" behaviour be easier for users to grok? Where the controller simply would attempt rolling back to N-1, then N-2, etc... until landing on a working version?

jaypipes · 2026-06-15T17:07:10Z

+type RollbackStrategy struct {
+	// Strategy for rollback. Valid values are "AllAtOnce" or "Progressive".
+	// Defaults to "AllAtOnce" for fast recovery.
+	Strategy VersionRollbackStrategy `json:"strategy"`


Is there really a valid reason or use case for having a rollback strategy not match the rollout strategy? Why not just have the rollback strategy always be the same as rollout?

jaypipes · 2026-06-15T17:08:16Z

+func warnRollbackSlowerThanRollout(rollout RolloutStrategy, rollback RollbackStrategy) admission.Warnings {
+	switch rollout.Strategy {
+	case UpdateAllAtOnce:
+		if rollback.Strategy != RollbackAllAtOnce {
+			return admission.Warnings{"rollback strategy is slower than rollout: rollout is AllAtOnce, but rollback is Progressive — is that intended?"}
+		}
+	case UpdateProgressive:
+		if rollback.Strategy == RollbackProgressive {
+			var rolloutTotal, rollbackTotal time.Duration
+			for _, s := range rollout.Steps {
+				rolloutTotal += s.PauseDuration.Duration
+			}
+			for _, s := range rollback.Steps {
+				rollbackTotal += s.PauseDuration.Duration
+			}
+			if rollbackTotal > rolloutTotal {
+				return admission.Warnings{fmt.Sprintf(
+					"rollback strategy is slower than rollout: progressive rollback total duration (%s) exceeds progressive rollout total duration (%s) — is that intended?",
+					rollbackTotal, rolloutTotal,
+				)}
+			}
+		}
+	}
+	return nil
+}


see my comment above... I suspect there's not really a valid use case for having a rollback strategy not match the rollout strategy... if we can just say the rollback == rollout, we can simplify things quite a bit and remove the need for these warnings.

eniko-dif requested review from a team and jlegrone as code owners April 14, 2026 15:35

jlegrone reviewed Apr 14, 2026

View reviewed changes

jlegrone reviewed May 18, 2026

View reviewed changes

eniko-dif force-pushed the eniko/feature-rollback branch from 371578c to a5d1e33 Compare May 27, 2026 15:52

eniko-dif added 7 commits June 10, 2026 11:41

Implement Rollback

3071510

Implement max version age

981a132

Add warning message if rollback is set slower than rollout

52dc172

Update the docs

be0a69d

Address Jacob's PR comments

93a03ef

Rebase fixes

1130d7e

Jacob's comments

4b0bd82

eniko-dif force-pushed the eniko/feature-rollback branch from a5d1e33 to 4b0bd82 Compare June 10, 2026 15:41

eniko-dif requested a review from jlegrone June 10, 2026 18:38

jaypipes reviewed Jun 15, 2026

View reviewed changes

	l.Info("Skipping rollback: the version's last current time exceeds MaxVersionAge",
	l.Debug("Skipping rollback: the version's last current time exceeds MaxVersionAge",

		// not affected by the rollback. Only new workflow executions will be routed to the rollback
		// target version.

Conversation

eniko-dif commented Apr 14, 2026 • edited by jlegrone Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

CLAassistant commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jaypipes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eniko-dif commented Apr 14, 2026 •

edited by jlegrone

Loading

CLAassistant commented Apr 14, 2026 •

edited

Loading