Skip to content

Add SPLUNK_KVSTORE_DISABLED to opt out of the KV Store - #928

Open
tod-uma wants to merge 2 commits into
splunk:developfrom
tod-uma:feature/disable-kvstore
Open

Add SPLUNK_KVSTORE_DISABLED to opt out of the KV Store#928
tod-uma wants to merge 2 commits into
splunk:developfrom
tod-uma:feature/disable-kvstore

Conversation

@tod-uma

@tod-uma tod-uma commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Why

Splunk indexers and forwarders are documented as safe to run without the App Key Value Store (About the app key value store), but the collection has no way to express that. The only existing route is the generic splunk.conf escape hatch, and the one branch that tried this directly (CSPL-4011-disable-kvstore) is ingestor-scoped, unmerged, and puts kvstore: directly under value: — omitting the required content: level, so it never actually writes anything.

What

Mirrors the newest kvstore task's presence-gated pattern (set_default_kvstore_type.yml):

  • roles/splunk_common/tasks/set_kvstore_disabled.yml (new) — single ini_file task writing [kvstore] disabled to server.conf, following the register-and-restart pattern used elsewhere in this role (set_server_name.yml, set_postgres_sidecar.yml).
  • Wired into roles/splunk_common/tasks/main.yml right after set_default_kvstore_type.yml, gated on presence ('disabled' in splunk.kvstore) rather than a false default — same reasoning upstream already applied to default_kvstore_type, and it means roles/splunk_indexer/molecule/default/tests/test_default.py's mongod.lock assertion is untouched (this is opt-in, not a default-behavior change).
  • SPLUNK_KVSTORE_DISABLED env override in inventory/environ.py, mirroring getDefaultKVStoreType's shape.
  • Docs in docs/ADVANCED.md and docs/advanced/default.yml.spec.md.
  • Unit test in tests/small/test_environ.py.

Testing

  • tests/small/ (pytest): all new cases pass, no regressions.
  • Functional dry-run: ran the actual main.yml gate condition and set_kvstore_disabled.yml task via ansible-playbook against a scratch splunk.home, confirmed the gate only fires when kvstore.disabled is explicitly set, the first write is changed, a repeat is idempotent, and the resulting server.conf reads exactly [kvstore]\ndisabled = <bool>.
  • Not yet run: this hasn't been exercised on a live Splunk instance. We're building it as the KV-store-disable pilot for a new production indexer joining our cluster and expect to run it there within the next few days — will report back with the result.

Indexers and forwarders can run without the App Key Value Store, but
nothing in the collection lets a deployment express that -- the only
existing route is the generic splunk.conf escape hatch. Mirror the
newest kvstore task's presence-gated pattern (set_default_kvstore_type.yml)
to add a first-class splunk.kvstore.disabled setting, an env var
override, and docs.
@tod-uma
tod-uma requested a review from a team as a code owner September 3, 2026 01:26
The play (`splunkCluster.yml`) leaves `become: true` commented out at
the play level, so every task in `splunk_common` that writes into
`$SPLUNK_HOME` must opt into privilege escalation itself. Every other
task that does so -- `set_default_kvstore_type.yml` (this feature's
own structural sibling), `set_general_symmkey_password.yml` (the
closest analogue: single `ini_file` write to `server.conf`, same
register-and-restart shape) -- carries `become: yes` /
`become_user: "{{ splunk.user }}"`. `set_kvstore_disabled.yml` never
got it; it copied `set_server_name.yml`'s register-and-restart
*shape* but not the privilege-escalation `set_default_kvstore_type.yml`
actually needed for a kvstore-section write.

Invisible on indexer03/04/05, where the KV store was already disabled
by hand and this `ini_file` call is a no-op against existing content
-- it never attempts an actual write, so the missing become never
gets exercised. It surfaced on `indexer06`'s first build, a genuinely
new host where every value this play sets is a first-time write:

    OSError: [Errno 18] Invalid cross-device link: ... -> '/opt/splunk/etc/system/local/server.conf'
    fatal: [lv-o-splunk-indexer06...]: FAILED! =>
        msg: 'The destination directory (/opt/splunk/etc/system/local)
              is not writable by the current user.'

Ansible's atomic_move hits EXDEV against the remote_tmp mount (as
expected -- $SPLUNK_HOME is its own LV) and falls back to writing the
temp file inside the destination directory itself; that fallback then
hit Permission Denied because the task was running as the raw
connection user, not `splunk.user`.

Fix: add the same `become: yes` / `become_user: "{{ splunk.user }}"`
`set_default_kvstore_type.yml` and `set_general_symmkey_password.yml`
already carry.
@tod-uma

tod-uma commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a fix (9a662cb): the new set_kvstore_disabled task was missing become/become_user, which every other task in this file that writes into $SPLUNK_HOME sets explicitly (e.g. set_default_kvstore_type.yml, set_general_symmkey_password.yml) since this role's play-level become isn't assumed. Found building a real new indexer against this branch — invisible in any environment where the KV store is already disabled, since the ini_file write becomes a no-op there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant