Regression in 10.2.5 and other images with custom configuration

[dv-dev-lr]

Hi,

newest package version has a regression with custom configuration. The problematic pr added a "remove stale configuration" to ansible: https://github.com/splunk/splunk-ansible/pull/904/changes#diff-0065d4d1159a6faa60f01a2722c4d39d9571e630d1487c3b14e151e75dfbea1fR10

The bug is, that custom configuration from /tmp/defaults overwrites other configurations, if it is conflicting with other configuration settings applied through environment variables.

My example disables the web port by setting the env var SPLUNK_HTTP_PORT=0. This results in a web.conf entry in etc/system/local/web.conf:

[settings]
startwebserver = 0

If we wanted to apply other configuration for web.conf, for example a special login_content, then this was posibble through a defauly.yml - see example.

Since 10.2.5 (but also in 9.4.13 and 10.4.1) this overwrites our web-port setting. the resulting file etc/system/local/web.conf doesn't contain startwebserver = 0 anymore.

This is just an example with web.conf - but applies to other configfiles in the same way.

Steps to reproduce:

default.yml:

splunk:
  conf:
    - key: web
      value:
        directory: /opt/splunk/etc/system/local
        content:
          settings:
            login_content : "<h1>HelloWorld</h1>"

start.sh:

#!/bin/bash
set -x

CONTAINER_NAME="splunk"
SPLUNK_IMAGE="docker.io/splunk/splunk:10.2.5"
SPLUNK_PASSWORD="${SPLUNK_PASSWORD:-changeme123}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

podman run -d \
  --name "${CONTAINER_NAME}" \
  --hostname splunk \
  -p 8000:8000 \
  -p 8089:8089 \
  -e SPLUNK_START_ARGS="--accept-license" \
  -e SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com \
  -e SPLUNK_PASSWORD="${SPLUNK_PASSWORD}" \
  -e SPLUNK_DEBUG=true \
  -e ANSIBLE_EXTRA_FLAGS="-v" \
  -e SPLUNK_HTTP_PORT="0" \
-v "${SCRIPT_DIR}/default.yml:/tmp/defaults/default.yml:Z" \
  "${SPLUNK_IMAGE}"

echo "Splunk container '${CONTAINER_NAME}' started."
echo "Web UI: http://localhost:8000"
echo "Management: https://localhost:8089"
echo "Login: admin / ${SPLUNK_PASSWORD}"

podman logs -f "${CONTAINER_NAME}"

run start.sh, wait for startup to complete, log into the container and check content of web.conf.

With 10.2.4:

podman exec -it splunk bash
[ansible@splunk splunk]$ sudo bash
[root@splunk splunk]# cat etc/system/local/web.conf 

[settings]
startwebserver = 0
mgmtHostPort = 0.0.0.0:8089
login_content = <h1>HelloWorld</h1>

[expose:tlPackage-agent-management-all-endpoints]
methods = GET,POST,DELETE,PATCH
[...]

With 10.2.5:

podman exec -it splunk bash
[ansible@splunk splunk]$ sudo bash
[root@splunk splunk]# cat etc/system/local/web.conf 

[settings]
login_content = <h1>HalloWelt</h1>

[expose:tlPackage-postgres_api_spec]
methods = get
pattern = /service/info/specs/v1/openapi.json
[...]

=> startwebserver = 0 is missing there.

This is kind of important, as it breaks existing configurations/installations. The problem ist, that we need to upgrade the latest version because of security fixes for splunk itself.