fix(shc): wait for all SHC members before captain bootstrap to eliminate pod restart

[vivekr-splunk]

Problem

On first boot, all SHC pods start simultaneously. The designated captain pod's ansible role runs bootstrap shcluster-captain, which internally contacts all other SHC members. However, other pods take 8–10 minutes to start Splunk on first boot (measured: ~499s on EKS with warm images).

The default retry window for this step is:

shc_sync_retry_num (60) × retry_delay (6s) = 360 seconds

360s < 499s → bootstrap times out before other members are reachable → ansible exits with code 2 → K8s restarts the captain pod.

The second ansible run succeeds because all members now have Splunk running, but this adds ~16 minutes to SHC first-boot readiness time and causes unnecessary pod restarts in Kubernetes-managed deployments.

Fix

1. Add a pre-bootstrap wait on the captain pod (search_head_clustering.yml)

Before calling bootstrap shcluster-captain, the captain now polls each non-captain member's REST endpoint until reachable, using shc_sync_retry_num retries:

- name: Wait for all SHC members to be reachable before captain bootstrap
  uri:
    url: "{{ cert_prefix }}://{{ item }}:{{ splunk.svc_port }}"
    ...
  retries: "{{ shc_sync_retry_num }}"
  loop: "{{ groups['splunk_search_head'] | difference([inventory_hostname]) }}"
  when: splunk_search_head_captain | bool

2. Raise shc_sync_retry_num for Linux (splunk_defaults_linux.yml)

Changed from 60 to 100 (100 × 6s = 600s), covering the observed first-boot Splunk start time with margin. Windows already uses 300; this brings Linux inline with the same order of magnitude.

Effect

• Captain pod waits for other members before bootstrap → bootstrap succeeds on first attempt → no pod restart needed
• SHC first-boot time reduced by ~16 minutes in Kubernetes environments
• Behavior is identical for environments where pods start sequentially (the wait resolves immediately)
• Fully backward-compatible: the wait task is gated on splunk_search_head_captain | bool

Validation

Tested on EKS with 3-member SHC (noah-mode, operator-managed):

• Before fix: all 3 SH pods restart once at ~16 min, Ready at ~32 min
• After fix (expected): no restart, Ready at ~16 min

AI Assistance

Implemented with Claude Code assistance.