PMM-14931 Control built-in ClickHouse config via PMM_CLICKHOUSE_CONFIG env var

[4nte]

PMM-14931

Feature Build

Previously, switching the built-in ClickHouse required manually running switch-config.sh inside the container.
PMM_CLICKHOUSE_CONFIG is a new env var that instructs supervisord to start ClickHouse with specified config.

If not set, ClickHouse will start with default config.

switch-config.sh has been deprecated in favor of configuring via env var. When ran, it will print out a deprecation message with link to docs.

[codecov]

Codecov Report

❌ Patch coverage is 69.69697% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.25%. Comparing base (3c9dfc5) to head (af3ebcc).

Files with missing lines	Patch %	Lines
managed/cmd/pmm-managed-init/main.go	0.00%	5 Missing ⚠️
managed/services/clickhouse/config.go	81.48%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v3    #5414      +/-   ##
==========================================
+ Coverage   43.20%   43.25%   +0.05%     
==========================================
  Files         413      414       +1     
  Lines       42279    42309      +30     
==========================================
+ Hits        18265    18301      +36     
+ Misses      22146    22138       -8     
- Partials     1868     1870       +2     

Flag	Coverage Δ
admin	34.94% <ø> (ø)
agent	49.16% <ø> (+0.11%)	⬆️
managed	42.38% <69.69%> (+0.04%)	⬆️
vmproxy	72.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[maxkondr]

I have an alternative idea.

1. Create several configs for CH (default, low-memory, etc)
2. Keep switch-config.sh script that will do the following:
   2.1 create symlink /etc/clickhouse-server/config.xml -> /etc/clickhouse-server/config-default.xml
   2.2. run CH with clickhouse --config-file=/etc/clickhouse-server/config.xml

So that in general we may point:

/etc/clickhouse-server/config.xml to one of the available files:
 ->      /etc/clickhouse-server/config-default.xml  (by default)
         /etc/clickhouse-server/config-low-memory.xml
         /etc/clickhouse-server/config-high-memory.xml

So that no need to touch supervisord configuration (it always uses config.xml).
The only action required - switch symlink to the needed config and restart/reload CH.
The target config may be passed via ENV var as an option during container startup, but also allows to switch in already running container.

This approach allows switching configuration without passing ENV var with container restart.

[4nte]

The target config may be passed via ENV var as an option during container startup, but also allows to switch in already running container.

This approach allows switching configuration without passing ENV var with container restart.

@maxkondr I agree it's a nice solution to use symlinks the way it was previously implemented by @JiriCtvrtka. My issue is that we're now introducing a new way to control ClickHouse config through an env var, but still allow users to change the configuration in runtime meaning that PMM_CLICKHOUSE_CONFIG would not be single source of truth and not immutable.

For example it's possible for user to deploy PMM with PMM_CLICKHOUSE_CONFIG=A, then in runtime mutate the configuration with ./switch-config.sh B, while PMM_CLICKHOUSE_CONFIG was set to A. Having two ways to achieve same thing, and one overshadowing the other.

On top of that, configuration applied through switch-config.sh could be wiped on container restart. This is the reason why we're introducing an env var for this in the first place.
For example, every pod restart in k8s (PMM upgrade, node drain, reschedule, etc.) would reset the change applied through switch-config.sh, and the configuration would silently fallback to "default" or whatever was set with PMM_CLICKHOUSE_CONFIG.

For these reasons I think switch-config.sh is better off deprecated since PMM_CLICKHOUSE_CONFIG achieves the same goal.

[JiriCtvrtka]

@4nte could you check why test for config is failing? If this could be bug.

[4nte]

@4nte could you check why test for config is failing? If this could be bug.

@JiriCtvrtka Thanks for noticing. Had to update the test to match the actual error message which was changed just recently (8a2479b)