atmos terraform plan-diff fails on no-op github_repository plans (etag refresh between plan and apply) #125
Unanswered
Marat Bakeev (mtb-xt)
asked this question in
Essential Support
Replies: 1 comment
|
Marat Bakeev (@mtb-xt) we'll be working on planfile storage next week in atmos native CI, and can maybe take a look at this around that time as part of that work. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Context
We use
cloudposse/github-action-atmos-terraform-apply@v7(which callsatmos terraform plan-diffunder the hood) to gate apply on the saved PR plan matching a fresh re-plan at apply time. We're using:1.213.01.11.5integrations/githubprovider6.12.1cloudposse/repository/githubmodule1.6.0(vendored locally with two pending fixes — separate upstream PR coming)cloudposse/github-action-atmos-terraform-apply@v7cloudposse/github-action-atmos-terraform-plan-storage@v1For 31 of our
github-repositoryworkspaces, the apply job fails with:After we ship the upstream-module fixes (
security_and_analysis.advanced_security,template,pages.sourcedrift), 23 of those failures resolve — there was real config drift the saved plan and fresh plan agreed on. But 8 of the 31 jobs failed even though both plans reportNo changes. Your infrastructure matches the configuration.Those 8 are what this discussion is about.Reproduction (no-op case)
For a workspace with zero drift:
The PR-time saved planfile and the apply-time fresh planfile both say "no changes", yet
plan-diffstill reports a difference and fails the apply.Confirmed: etag refresh + plan timestamp
I downloaded the saved planfile from our plan-storage S3 bucket and regenerated a fresh planfile against the same source tree (the squash-merge SHA had identical tree to the PR HEAD that produced the saved plan), then
tofu show -json'd both and diffed.Result: 7346 of 7352 JSON lines identical. The only differences:
tfplanSHA-256:46bd6e7d5b48dbe52cd211f21444f971af40fc238b0fbe9e43898a8c4292ca06aff3aef0e9623c23e100c14ca5bfd31e374253bca7f16270c3e2ed03ccfa2dd7So semantically the two plans are identical — both correctly say "no changes" — but the
integrations/githubprovider repopulatesetagon every refresh from the GitHub API'sETagresponse header, and OpenTofu embeds the plan-creation timestamp in the binary. Two reads ⇒ two binary planfiles.plan-diffthen exits 1 even though there's nothing to apply.This explains why we see it on exactly the workspaces with no real drift to mask it. The "with-changes" workspaces also embed mismatched etag values, but plan-diff's existing thresholds presumably tolerate that when there's a legitimate change to surface.
What we've considered
lifecycle.ignore_changes = [etag]— no-op.etagisComputed-only, so Terraform never proposes to change it;ignore_changesonly affects diff calculation, not what's persisted to state.etagat provider level — the right long-term fix, but requires an upstream change tointegrations/terraform-provider-githubto marketagas transient or drop it from state.atmos terraform plan-diffto a content-aware comparison — difftofu show -jsonoutput rather than bytes, with an allow-list for inherently-volatile keys (etag,timestamp). The semantic content already matches in our case; only the byte-level prior-state differs.cloudposse/github-action-atmos-terraform-applyto skipplan-diff(trust the saved plan, apply it). For workspaces where we don't expect drift between PR merge and apply, the diff check is currently a false-positive generator.apply -refresh=falseduring apply so the saved prior_state is consumed verbatim. Loses the (small) ability to detect drift introduced between PR and merge, but eliminates the etag refresh entirely.Question
What's the recommended path here? Specifically:
plan-diffmode or input we've missed that does content-based comparison rather than binary byte-diff?--mode=content(or similar) flag toatmos terraform plan-diffplus a passthrough input oncloudposse/github-action-atmos-terraform-applybe welcomed?integrations/terraform-provider-githubto dropetagfrom state)?plan-diffper-workspace or per-component while we land the longer-term fix?We have a vendored module fix and the
advanced_security/template/pages.sourceupstream PR ready to go — that resolves 23/31 of our failures. The remaining 8 are blocked on whatever the right answer to the above turns out to be.Happy to share evidence dumps, full action logs, and the saved/fresh planfile pair (privately) if useful.
Thanks!
— Marat (9spokes platform team)
All reactions