From ef0e89836767673e5be5739d5dbc60bd0cfe2fc3 Mon Sep 17 00:00:00 2001 From: Tomas Klapka Date: Mon, 31 Aug 2026 15:30:37 +0200 Subject: [PATCH 1/3] fix(influxdb3-ent): widen the startup probe window Nodes replay from object storage on boot, so startup scales with how much data a node reads back; startups past 12 minutes have been reported on clusters with a large history. The 160s window kills them mid-replay, and the restart begins the replay again. Raises failureThreshold alone, as #807 did, leaving the probe cadence untouched. The window is initialDelay + (threshold - 1) x period, so the previous comments were one period optimistic. Documents the probes, which the README did not cover at all: the helm --wait timeout that is shorter than the startup it waits for, the liveness window that takes over once the startup probe succeeds, and the two ways to shorten startup instead of tolerating it. Exit code 137 is also the OOMKilled signature, so troubleshooting says how to tell the two apart. --- charts/influxdb3-enterprise/Chart.yaml | 2 +- charts/influxdb3-enterprise/README.md | 52 +++++++++++++++++++++++++ charts/influxdb3-enterprise/values.yaml | 6 +-- 3 files changed, 56 insertions(+), 4 deletions(-) diff --git a/charts/influxdb3-enterprise/Chart.yaml b/charts/influxdb3-enterprise/Chart.yaml index 7705cbd6..e152bc24 100644 --- a/charts/influxdb3-enterprise/Chart.yaml +++ b/charts/influxdb3-enterprise/Chart.yaml @@ -2,7 +2,7 @@ apiVersion: v2 name: influxdb3-enterprise description: A Helm chart for deploying InfluxDB 3 Enterprise on Kubernetes type: application -version: 0.10.0 +version: 0.10.1 appVersion: "3.11.2" keywords: - influxdb diff --git a/charts/influxdb3-enterprise/README.md b/charts/influxdb3-enterprise/README.md index a875636b..297dd323 100644 --- a/charts/influxdb3-enterprise/README.md +++ b/charts/influxdb3-enterprise/README.md @@ -282,6 +282,39 @@ ingester: numThreads: 20 ``` +#### Health Probes + +All components share one probe configuration. The startup probe guards the +initialization window; liveness and readiness start only once it succeeds. + +```yaml +probes: + startup: + initialDelaySeconds: 10 + periodSeconds: 5 + timeoutSeconds: 5 + failureThreshold: 184 # 10s + (184 - 1) × 5s = 925s +``` + +Nodes replay from object storage on boot, so startup scales with how much data a +node reads back, and clusters with a large history have been reported to need +more than 12 minutes. Two things follow from a window this wide. + +`helm install` and `helm upgrade` with `--wait` default to a five-minute +timeout, which is shorter than the startup they are waiting for. Pass +`--timeout 20m` to match, or the release fails while the pods are still booting +normally. + +Once the startup probe succeeds, the liveness probe takes over with a much +narrower window (`3 × 10s`). A node that answers `/health` and then blocks +during later startup work can still be restarted; raise +`probes.liveness.failureThreshold` if that happens. + +Two alternatives shorten the startup itself rather than tolerating it: +[Compacted-Data Startup](#compacted-data-startup) bounds the file-index load, +and `ingester.persistence` keeps the WAL on a local volume instead of replaying +it from object storage. + #### TLS Enable TLS with inline cert/key or an existing secret: @@ -606,6 +639,19 @@ Check events: kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 ``` +A pod that restarts during startup while its logs show normal activity was +either killed by the startup probe or ran out of memory. Both exit with code +137, so check which before changing anything: + +```bash +kubectl get pod -n influxdb3 influxdb3-enterprise-ingester-0 \ + -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' +``` + +`OOMKilled` means raise the memory limit. `Error` together with a +`Startup probe failed` event means the probe window is too short; see +[Health Probes](#health-probes). + #### License Issues Verify license configuration: @@ -680,6 +726,12 @@ logs: | `security.auth.adminToken.recovery.httpBind` | Bind address for admin token recovery endpoint (`INFLUXDB3_ADMIN_TOKEN_RECOVERY_HTTP_BIND_ADDR`) | `""` | | `serviceAccount.automountServiceAccountToken` | Configure automatic mounting of the Kubernetes service account token in component pods and the created ServiceAccount | `not set` | | `extraEnv` | Extra environment variables applied to all components | `[]` | +| `probes.enabled` | Enable liveness, readiness, and startup probes on all components | `true` | +| `probes.startup.initialDelaySeconds` | Delay before the first startup check | `10` | +| `probes.startup.periodSeconds` | Interval between startup checks | `5` | +| `probes.startup.timeoutSeconds` | Timeout of a single startup check | `5` | +| `probes.startup.failureThreshold` | Failed startup checks before the pod is killed | `184` | +| `probes.liveness.*` / `probes.readiness.*` | Liveness and readiness timings; see `values.yaml` | see `values.yaml` | ### Object Storage Parameters diff --git a/charts/influxdb3-enterprise/values.yaml b/charts/influxdb3-enterprise/values.yaml index 672597d0..6b23c94a 100644 --- a/charts/influxdb3-enterprise/values.yaml +++ b/charts/influxdb3-enterprise/values.yaml @@ -731,13 +731,13 @@ probes: # Startup probe settings # Protects container during initialization. Once successful, never runs again. - # Allows slower startup without delaying successful readiness detection. - # 10s initial + (30 × 5s) = 160s total. + # 10s initial + (184 - 1) × 5s = 925s before the container is restarted. + # Nodes replay from object storage on boot, so startup scales with data size. startup: initialDelaySeconds: 10 periodSeconds: 5 # Check every 5 seconds timeoutSeconds: 5 - failureThreshold: 30 + failureThreshold: 184 # Liveness probe settings # Only starts checking AFTER startup probe succeeds From 86e8e45ea9f125d8b2eec2231f8738b9a4991e0a Mon Sep 17 00:00:00 2001 From: Tomas Klapka Date: Wed, 2 Sep 2026 08:11:06 +0200 Subject: [PATCH 2/3] docs(influxdb3-ent): drop unavailable remedies, soften the exit-137 claim The alternatives named mechanisms this chart version does not have: the compacted-data section lands in a separate PR, and ingester.persistence is a deprecated value no template consumes. Exit 137 is one possible signature, not proof. Kubelet asks the runtime to terminate first and honours terminationGracePeriodSeconds, so a process that exits during that window reports a different code, and 137 is also the OOM signature. The termination reason and the probe event are the discriminator. --- charts/influxdb3-enterprise/README.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/charts/influxdb3-enterprise/README.md b/charts/influxdb3-enterprise/README.md index 297dd323..d92d6cbd 100644 --- a/charts/influxdb3-enterprise/README.md +++ b/charts/influxdb3-enterprise/README.md @@ -310,10 +310,8 @@ narrower window (`3 × 10s`). A node that answers `/health` and then blocks during later startup work can still be restarted; raise `probes.liveness.failureThreshold` if that happens. -Two alternatives shorten the startup itself rather than tolerating it: -[Compacted-Data Startup](#compacted-data-startup) bounds the file-index load, -and `ingester.persistence` keeps the WAL on a local volume instead of replaying -it from object storage. +This widens the window rather than shortening the startup. If nodes routinely +need most of it, the boot work itself is worth investigating. #### TLS @@ -640,17 +638,21 @@ kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 ``` A pod that restarts during startup while its logs show normal activity was -either killed by the startup probe or ran out of memory. Both exit with code -137, so check which before changing anything: +either killed by the startup probe or ran out of memory. Read the termination +reason and the events rather than the exit code: ```bash kubectl get pod -n influxdb3 influxdb3-enterprise-ingester-0 \ -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' +kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 | grep -i probe ``` `OOMKilled` means raise the memory limit. `Error` together with a `Startup probe failed` event means the probe window is too short; see -[Health Probes](#health-probes). +[Health Probes](#health-probes). Exit code 137 is one possible signature of a +probe kill, not proof of one: kubelet asks the runtime to terminate first and +honours `terminationGracePeriodSeconds`, so a process that exits during that +window reports a different code, and 137 is also what an OOM kill produces. #### License Issues From 0217b0a3df2264cc1f12a0c0d8ccfb83681751d3 Mon Sep 17 00:00:00 2001 From: Tomas Klapka Date: Wed, 2 Sep 2026 12:04:49 +0200 Subject: [PATCH 3/3] docs(influxdb3-ent): use the Killing event as the probe discriminator Kubelet records 'Startup probe failed' for every failed attempt, including ones below failureThreshold, so it does not establish that the probe caused a given restart. The Killing event that reads 'failed startup probe, will be restarted' does. Also softens the either/or, since an externally killed container has other possible causes. The window figure now says it is the point termination is triggered, not when the container has restarted - that follows terminationGracePeriodSeconds. --- charts/influxdb3-enterprise/README.md | 25 ++++++++++++++----------- charts/influxdb3-enterprise/values.yaml | 3 ++- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/charts/influxdb3-enterprise/README.md b/charts/influxdb3-enterprise/README.md index d92d6cbd..cf5f0d7e 100644 --- a/charts/influxdb3-enterprise/README.md +++ b/charts/influxdb3-enterprise/README.md @@ -293,7 +293,7 @@ probes: initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 5 - failureThreshold: 184 # 10s + (184 - 1) × 5s = 925s + failureThreshold: 184 # 10s + (184 - 1) × 5s = 925s to termination ``` Nodes replay from object storage on boot, so startup scales with how much data a @@ -637,22 +637,25 @@ Check events: kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 ``` -A pod that restarts during startup while its logs show normal activity was -either killed by the startup probe or ran out of memory. Read the termination -reason and the events rather than the exit code: +A pod that restarts during startup while its logs show normal activity is most +often killed by the startup probe or out of memory, though an externally killed +container can have other causes. Read the termination reason and the kill event +rather than the exit code: ```bash kubectl get pod -n influxdb3 influxdb3-enterprise-ingester-0 \ -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' -kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 | grep -i probe +kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 | grep -i killing ``` -`OOMKilled` means raise the memory limit. `Error` together with a -`Startup probe failed` event means the probe window is too short; see -[Health Probes](#health-probes). Exit code 137 is one possible signature of a -probe kill, not proof of one: kubelet asks the runtime to terminate first and -honours `terminationGracePeriodSeconds`, so a process that exits during that -window reports a different code, and 137 is also what an OOM kill produces. +`OOMKilled` means raise the memory limit. A `Killing` event reading +`failed startup probe, will be restarted` means the probe window is too short; +see [Health Probes](#health-probes). Use that event rather than +`Startup probe failed`, which kubelet records for every failed attempt including +those below `failureThreshold`. Exit code 137 is one possible signature, not +proof: kubelet asks the runtime to terminate first and honours +`terminationGracePeriodSeconds`, so a process that exits during that window +reports a different code, and 137 is also what an OOM kill produces. #### License Issues diff --git a/charts/influxdb3-enterprise/values.yaml b/charts/influxdb3-enterprise/values.yaml index 6b23c94a..ac9338e5 100644 --- a/charts/influxdb3-enterprise/values.yaml +++ b/charts/influxdb3-enterprise/values.yaml @@ -731,7 +731,8 @@ probes: # Startup probe settings # Protects container during initialization. Once successful, never runs again. - # 10s initial + (184 - 1) × 5s = 925s before the container is restarted. + # 10s initial + (184 - 1) × 5s = 925s before the startup probe triggers + # termination; the restart itself follows terminationGracePeriodSeconds. # Nodes replay from object storage on boot, so startup scales with data size. startup: initialDelaySeconds: 10