Skip to content

Fix reconcile_config_map_values.yml: read ConfigMap state from the managed host, not the control node - #930

Open
tod-uma wants to merge 1 commit into
splunk:developfrom
tod-uma:fix/reconcile-config-map-remote-read
Open

Fix reconcile_config_map_values.yml: read ConfigMap state from the managed host, not the control node#930
tod-uma wants to merge 1 commit into
splunk:developfrom
tod-uma:fix/reconcile-config-map-remote-read

Conversation

@tod-uma

@tod-uma tod-uma commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Summary

reconcile_config_map_values.yml (added in #921 / CSPL-5085) always fails once a host's
ConfigMap-owned state file exists, in any topology where the Ansible control node and the
managed host are different machines.

Root cause

The "Load previous ConfigMap-owned values" task uses include_vars, which is a
control-node-local action (_requires_connection = False in
ansible/plugins/action/include_vars.py) — it reads its file: argument off the control
node's
filesystem, never the managed host. This task passes it an absolute path that only
exists on the target, e.g.
{{ conf_directory }}/.{{ conf_file }}.splunk-ansible-managed.yml.

  • First convergence: the preceding stat task (which correctly runs on the remote host via
    become_user: "{{ splunk.user }}") finds nothing, so include_vars is skipped by its when:
    guard — this is why the bug doesn't show up on a fresh host.
  • Second+ convergence: the state file now exists on the remote host (the copy task that
    writes it runs remotely, correctly), the guard flips true, include_vars looks for that same
    path on the control node, fails, and the failure is swallowed by
    no_log: "{{ hide_password }}".

#921's own testing was done entirely inside Splunk Operator for Kubernetes pods
(IndexerCluster + ClusterManager + LicenseManager), where the control node and the managed
host share one filesystem — the one topology where this bug is invisible. Any bare-metal or
AWX-style deployment (control node and target on different machines, connecting over SSH) hits
it on the second run.

Fix

Swap include_vars for slurp + b64decode | from_yaml, which actually reads the file from
the managed host, using the same become_user the stat task already uses.

Testing

  • Live-verified against a real bare-metal host (Ansible connecting over SSH): first convergence
    succeeds, second convergence — previously a hard failure — now completes and applies the
    reconciliation correctly.
  • ansible-lint on the changed file: no new findings (only pre-existing stylistic ones shared by
    the rest of the file — short module names, non-role-prefixed var names — left as-is to match
    surrounding style).
  • tests/small/test_hec_receiver.py::test_config_map_reconciliation_* (3 tests): unaffected,
    still pass. Worth noting these only exercise the stanza-removal logic in pure Python and never
    exercised the load path, which is part of why this went uncaught.

Note for reviewers

While verifying this fix I also noticed the second convergence isn't fully idempotent (a small
changed count even when config content is unchanged) — this looks like a separate, pre-existing
property of the reconcile feature (it appears to unconditionally remove-then-reapply every
ConfigMap-owned key rather than diffing first) rather than something introduced or fixed here.
Not addressed in this PR; happy to file separately if useful.

include_vars is a control-node-local action (_requires_connection =
False) that reads its file argument off the Ansible controller's own
filesystem, never the managed host. reconcile_config_map_values.yml
passed it an absolute remote-host path
(<conf_directory>/.<conf_file>.splunk-ansible-managed.yml), so the
"Load previous ConfigMap-owned values" task always fails once that
state file exists on a real target — masked on a host's first
convergence only because the preceding when: guard skips it.

Upstream's own PR splunk#921 (CSPL-5085) testing was done entirely inside
Splunk Operator for Kubernetes pods, where the control node and the
managed host share one filesystem, so the bug never surfaced there.
It reproduces on any topology where they don't (AWX, or a manual
ansible-playbook run against a remote host over SSH) - caught here on
indexer06's second splunk_universal_forwarder.yml convergence.

Swap include_vars for slurp + b64decode | from_yaml, which actually
reads the file from the managed host, matching the become_user the
preceding stat task already uses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant