Bug OCPBUGS-112464: Add x509/TLS errors to IsApplyErrorRetriable

[redhat-chai-bot]

Summary

During cluster upgrades, the MCO briefly goes Degraded with MachineConfigServerFailed because IsApplyErrorRetriable does not consider x509/TLS certificate errors as retriable. When the kube-apiserver briefly serves a cert without the kubernetes service ClusterIP (172.30.0.1) as an IP SAN during cert rotation, the error propagates immediately instead of being retried.

This is a ~1% CI flake observed across upgrade jobs on both 5.0 and 5.1.

Changes

• lib/resourceapply/apps.go: Added x509 and TLS certificate error checks to IsApplyErrorRetriable. Uses strings.Contains matching for "x509:" and "tls: failed to verify certificate", consistent with the existing "rpc error" check in the same function.
• lib/resourceapply/apps_test.go: Added 7 test cases covering rpc errors, x509 certificate errors, TLS verification failures, wrapped errors, and negative cases.

Root Cause

applyManifests() in pkg/operator/sync.go already retries with retry.DefaultRetry (5 retries, exponential backoff), but the predicate IsApplyErrorRetriable only retries on:

• "rpc error" (string match)
• apierrors.IsConflict
• apierrors.IsTimeout

x509/TLS errors fall through and are not retried, causing the MCO to go Degraded for ~15 seconds until the next sync cycle succeeds after the kube-apiserver cert rotation completes.

Testing

• Unit tests: go test ./lib/resourceapply/... -run TestIsApplyErrorRetriable — all 7 cases pass
• The existing retry.DefaultRetry backoff (5 retries starting at ~200ms) comfortably covers the ~15-second cert rotation window

Fixes: OCPBUGS-112464

---

AI-generated. Review for accuracy.

@sdodson requested in Slack thread

Summary by CodeRabbit

• Bug Fixes
  • Improved retry handling for certificate validation and TLS verification failures during resource application.
  • Wrapped certificate-related errors are now retried when appropriate.
• Tests
  • Added coverage for retryable RPC, certificate, and TLS errors, along with non-retryable generic and connection-refused errors.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@redhat-chai-bot: This pull request references Jira Issue OCPBUGS-112464, which is invalid:

• expected the bug to target the "5.1.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

During cluster upgrades, the MCO briefly goes Degraded with MachineConfigServerFailed because IsApplyErrorRetriable does not consider x509/TLS certificate errors as retriable. When the kube-apiserver briefly serves a cert without the kubernetes service ClusterIP (172.30.0.1) as an IP SAN during cert rotation, the error propagates immediately instead of being retried.

This is a ~1% CI flake observed across upgrade jobs on both 5.0 and 5.1.

Changes

• lib/resourceapply/apps.go: Added x509 and TLS certificate error checks to IsApplyErrorRetriable. Uses strings.Contains matching for "x509:" and "tls: failed to verify certificate", consistent with the existing "rpc error" check in the same function.
• lib/resourceapply/apps_test.go: Added 7 test cases covering rpc errors, x509 certificate errors, TLS verification failures, wrapped errors, and negative cases.

Root Cause

applyManifests() in pkg/operator/sync.go already retries with retry.DefaultRetry (5 retries, exponential backoff), but the predicate IsApplyErrorRetriable only retries on:

• "rpc error" (string match)
• apierrors.IsConflict
• apierrors.IsTimeout

x509/TLS errors fall through and are not retried, causing the MCO to go Degraded for ~15 seconds until the next sync cycle succeeds after the kube-apiserver cert rotation completes.

Testing

• Unit tests: go test ./lib/resourceapply/... -run TestIsApplyErrorRetriable — all 7 cases pass
• The existing retry.DefaultRetry backoff (5 retries starting at ~200ms) comfortably covers the ~15-second cert rotation window

Fixes: OCPBUGS-112464

---

AI-generated. Review for accuracy.

@sdodson requested in Slack thread

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: df8256dc-894f-4850-8ba2-b70b95871ead

📥 Commits

Reviewing files that changed from the base of the PR and between 7ff337a and 689ef8b.

📒 Files selected for processing (2)

• lib/resourceapply/apps.go
• lib/resourceapply/apps_test.go

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

---

Walkthrough

IsApplyErrorRetriable now retries x509 and TLS certificate verification errors. Tests cover direct, wrapped, retriable, and non-retriable errors.

Changes

Apply Error Retry Handling

Layer / File(s)	Summary
Retry classification and coverage lib/resourceapply/apps.go, lib/resourceapply/apps_test.go	IsApplyErrorRetriable now recognizes x509 and TLS certificate verification errors. Table-driven tests cover direct and wrapped certificate errors, RPC errors, generic errors, and connection refusals.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 689ef

The change retries transient x509/TLS certificate errors and adds focused coverage; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding x509 and TLS errors to IsApplyErrorRetriable.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR adds one standard Go Test with static t.Run names; it adds no Ginkgo It, Describe, Context, or When titles and no test name contains dynamic values.
Test Structure And Quality	✅ Passed	The added test is a standard Go table-driven unit test, not Ginkgo; it creates no cluster resources or waits, and its t.Errorf includes diagnostic context.
Microshift Test Compatibility	✅ Passed	The PR adds a standard Go table-driven unit test, not Ginkgo e2e tests. It uses no MicroShift-unavailable APIs, namespaces, or features.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The PR adds a standard Go unit test, not a Ginkgo e2e test; the diff contains no It, Describe, Context, or When markers and makes no SNO-relevant assumptions.
Topology-Aware Scheduling Compatibility	✅ Passed	The diff only changes retry error matching and adds unit tests; it adds no deployment manifests, controllers, replicas, affinity, selectors, tolerations, topology spread, or PDB constraints.
Ote Binary Stdout Contract	✅ Passed	The PR changes only lib/resourceapply logic and unit tests; no main/init/suite setup or stdout write was added. The existing klog call is unchanged and not process-level code.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The added test is a standard Go unit test, not a Ginkgo e2e test. Its IPv4 values occur only in synthetic error strings, with no network or external connectivity.
No-Weak-Crypto	✅ Passed	The changed diff only adds x509/TLS error-string matching and tests; it introduces no MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB, custom crypto, or secret comparison.
Container-Privileges	✅ Passed	The PR changes only Go code and tests. The complete patch adds no privileged, host namespace, SYS_ADMIN, or allowPrivilegeEscalation settings.
No-Sensitive-Data-In-Logs	✅ Passed	The commit adds no production logging. It only changes retry matching; the existing klog error log remains for non-retriable errors, and test diagnostics use fixed non-sensitive values.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: redhat-chai-bot
Once this PR has been reviewed and has the lgtm label, please assign dkhater-redhat for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[redhat-chai-bot]

/retest ci/prow/bootstrap-unit

---

AI-generated. Review for accuracy.

[sdodson]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-ocl-part1
/test e2e-gcp-op-ocl-part2
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift
/test tls-pqc-readiness

[redhat-chai-bot]

/test bootstrap-unit

---

AI-generated. Review for accuracy.

[redhat-chai-bot]

/retest-required

---

AI-generated. Review for accuracy.

[redhat-chai-bot]

/retest-required

The two remaining failures are unrelated to this change:

• e2e-gcp-op-ocl-part2: TestImagePrunerErrors/Docker.io/Delete/Existing_image — Docker Hub connectivity/rate-limit issue
• perfscale-control-plane-6nodes: Orion benchmarking noise (3/5 workloads flagged as regressions — this PR adds 9 lines of string matching in an error predicate and cannot affect control plane latency)

---

AI-generated. Review for accuracy.

[openshift-ci]

@redhat-chai-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/perfscale-control-plane-6nodes	689ef8b	link	false	/test perfscale-control-plane-6nodes
ci/prow/e2e-gcp-op-ocl-part2	689ef8b	link	true	/test e2e-gcp-op-ocl-part2

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.