Keep the ACK finalizer when a create partially succeeds - #268
Keep the ACK finalizer when a create partially succeeds#268gustavodiaz7722 wants to merge 1 commit into
Conversation
A resource manager's Create may issue several AWS API calls: creating the resource, then applying tags, attributes or associated sub-resources. When a later call fails, the resource exists in AWS but Create still returns an error, and createResource cannot tell that apart from a create that never happened. It keys off the error type alone, so any AWS API error removes the finalizer. The next reconciliation then finds an existing resource with no finalizer and terminally conditions it as not managed by ACK. Reconciliation stops there and the resource is orphaned in AWS. Add PostCreateError, which a resource manager uses to signal that the resource was created before the error occurred, and honour it in createResource ahead of the existing AWS error check. WrapPostCreateError only wraps AWS API errors, since those are the only ones that unmanage a resource; everything else passes through so sentinels such as NotFound and the ackrequeue signals stay comparable by identity. Removing the finalizer on a plain AWS error is unchanged, which is what lets the adoption logic run on subsequent reconciliations (aws-controllers-k8s#185). Issue: aws-controllers-k8s/community#2849
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: gustavodiaz7722 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/retest
This looks unrelated to the change.
|
|
Root-caused:
Verified by building mockery from source the way the job does:
I also reproduced the failure on unmodified Fix is in #269, where |
|
/retest |
Issue #2849
Description of changes:
A resource manager's
Createmay issue several AWS API calls: creating theresource, then applying tags, attributes or associated sub-resources. When a
later call fails, the resource exists in AWS but
Createstill returns anerror, and
createResourcecannot tell that apart from a create that neverhappened. It keys off the error type alone, so any AWS API error removes the
finalizer:
The next reconciliation then finds an existing resource with no finalizer and
failOnResourceUnmanagedterminally conditions it as not managed by ACK.HandleReconcileErrorswallowsackerr.Terminal, so there is no requeue:reconciliation stops and the resource is orphaned in AWS.
This is a recurring class rather than a single-service bug. #2670
(
publicAccessBlockdenied by an SCP) and #2730(bucket policy fails) are the same root cause; both were closed by moving the
extra work out of the create path in s3-controller, leaving the runtime
heuristic in place. Scanning every generated
sdkCreateacross the controllerfleet, 41 of 277 resources can fail after a successful create.
These changes add
PostCreateError, which a resource manager uses to signalthat the resource was created before the error occurred, and honour it in
createResourceahead of the existing AWS error check.WrapPostCreateErroronly wraps AWS API errors, returning anything elseunchanged. Those are the only errors that cause the reconciler to unmanage a
resource, so wrapping anything else would change no behaviour while hiding
sentinels such as
ackerr.NotFoundand theackrequeuesignals from theidentity comparisons callers still perform on them. Because the check unwraps,
an AWS error carried inside another error is still wrapped, which is intended:
such an error already triggers the unmanage path today.
Removing the finalizer on a plain AWS API error is unchanged, so the adoption
behaviour from #185 is preserved.
The companion code-generator change is
aws-controllers-k8s/code-generator#740, which makes generated
sdkCreateproduce these errors. It depends on a runtime release containingWrapPostCreateError, so this should merge and release first.Testing
Unit tests cover the wrapper (AWS error wrapped,
ackrequeueand sentinelpassthrough, requeue detection surviving a wrapped AWS error,
Unwrapchain)and the reconciler in both directions: the finalizer is retained on a
post-create error, and still removed on a plain AWS API error. I confirmed the
new reconciler test fails without the
createResourcechange, reportingMarkUnmanagedcalled.Verified end to end on an EKS cluster with ec2-controller, whose
SecurityGroupsdkCreatecallsRevokeSecurityGroupEgressafter the create. The failure wasinjected two ways: an IAM deny of
ec2:RevokeSecurityGroupEgress(the reportedscenario) and an ingress rule referencing a non-existent security group, which
fails in
syncSGRuleswith no IAM involved.Before, both scenarios lost the finalizer and reported the terminal
NotManagedReason, and both security groups remained in AWS after the CRs weredeleted:
After, the finalizer is retained and the condition is
ACK.Recoverablecarrying the
UnauthorizedOperation, which is the outcome the issue asks for:Reconciliation continued through the update path rather than dead-ending, no
duplicate resources were created, and once the deny was removed the resource
reached
Synced=Trueon its own and the originally failedRevokeSecurityGroupEgresscompleted. Deleting the CRs removed both securitygroups from AWS, so nothing was orphaned.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.