Draft: SPL-Noah-spike: eliminate SHC captain bootstrap restart (zero restarts in Noah mode) - #923
Draft
vivekr-splunk wants to merge 4 commits into
Draft
vivekr-splunk wants to merge 4 commits into
vivekr-splunk wants to merge 4 commits into
Conversation
…password compat The declarative admin password path runs `splunk cmd splunkd rest --noauth` to apply the admin password on non-first-run starts. This brief splunkd invocation initializes the Noah C++ client (NoahConfiguration::loadNoahServiceFromConfFilesReloadable) which crashes in two ways when [noahService] is present: 1. pass4SymmKey error: [general] pass4SymmKey is written as plaintext by set_general_symmkey_password.yml; if [noahService] has an encrypted pass4SymmKey from the previous run, encrypt_fields processing aborts. 2. heartbeatPeriod assertion: when disabled=false, the Noah client crashes asserting parsedCorrectly==true because heartbeatPeriod is absent or fails as an unsigned integer. Fix: add set_noah_symmkey_password.yml which runs between set_general_symmkey_password.yml and enable_admin_auth.yml to write the full [noahService] stanza with disabled=true (so Noah client stays dormant during the brief splunkd start), heartbeatPeriod=30 (persists for the full start), and pass4SymmKey=plaintext. set_config_file.yml overwrites disabled with the real value after enable_admin_auth.yml completes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ng] writes splunk init shcluster-config always returns rc=0, and search_head_clustering.yml used changed_when: task_result.rc == 0 — so every pod restart unconditionally fired the Restart the splunkd service handler, adding ~537s. Fix: write all [shcclustering] fields (pass4SymmKey, label, replication_port, replication_factor, conf_deploy_fetch_url, mgmt_uri, disabled=false) to server.conf in configure_noah.yml before splunk start. This makes splunk init shcluster-config a no-op; changed_when: false prevents the restart. Works with Noah: configure_noah.yml already runs before enable_admin_auth.yml (before splunk start), preserving the disabled=true / pass4SymmKey sequencing for [noahService]. The init-etc init container (operator side) will be extended to also strip [shcclustering].pass4SymmKey so ini_file can overwrite the splunkd-encrypted value on each restart. Expected improvement: SHC pod roll time ~18 min → ~8 min. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…r SHC pods) configure_noah.yml already writes all [shcclustering] fields to server.conf before splunkd starts, making bootstrap shcluster-captain a no-op API call. Add a set_fact (noah_shc_prestart_configured) after those writes, then gate the captain bootstrap task's changed_when on that fact so the restart handler never fires in Noah mode. Non-Noah SHC flows are unaffected (fact unset → default(false) → restart fires as before). This completes zero-restart SHC pod rolls: v11 removed the init shcluster-config restart (537s); v12 removes the captain bootstrap restart (~537s additional). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminates the last remaining Splunk restart during SHC pod rolls when Noah pre-configures all
[shcclustering]fields beforesplunkdstarts. Produces thenoah-splunk-v12image.Context:
configure_noah.ymlalready writes 7[shcclustering]fields toserver.confbefore Splunk starts. This makesbootstrap shcluster-captaina no-op API call — no restart needed. The restart was still firing becausechanged_when: task_result.rc == 0was left on the Bootstrap task.Changes
roles/splunk_common/tasks/configure_noah.yml: Addedset_fact: noah_shc_prestart_configured: trueafter the last[shcclustering]pre-start write. Only set whensplunk.role == splunk_search_headandsplunk.shcis defined (Noah SHC mode only).roles/splunk_search_head/tasks/search_head_clustering.yml: ChangedBootstrap SHC captainchanged_whenfromtask_result.rc == 0to a two-condition block:In Noah mode this is always false →
Restart the splunkd servicehandler never fires.Non-Noah SHC flows are unaffected: when
configure_noah.ymldoes not run, the fact is unset,default(false)applies, and the restart fires as before.Restart History
init shcluster-config→changed_when: falsebootstrap shcluster-captain→ suppressed vianoah_shc_prestart_configuredValidation
noah-splunk-v12built and pushed to ECRvivek-noah-c3-devin progressfeature/shc-kubernetes-reliabilityPOC (theirprestart_configflag), adapted for our Noahconfigure_noah.ymlflowAI Assistance
Claude Code assisted with analysis and implementation.