fix(influxdb3-ent): widen startup probe window

[tomklapka]

Widens the startup probe window and documents the probe settings, which the README did not cover at all.

Why

Nodes replay from object storage on boot, so startup scales with how much data a node reads back. Support cases report startups past 12 minutes, while the window was 155s in practice. A node killed mid-replay restarts and begins the replay again.

What changes

failureThreshold goes from 30 to 184, leaving initialDelaySeconds: 10 and periodSeconds: 5 untouched, as #807 did when it last raised the window. That gives 10 + (184 - 1) × 5 = 925s before the probe triggers termination; the restart itself then follows terminationGracePeriodSeconds.

The arithmetic in the old comments was one period optimistic - the window is initialDelay + (threshold - 1) × period, so the documented 10s + (30 × 5s) = 160s was really 155s. Comments and docs now use the correct form.

Documentation

A Health Probes section covering three things that were not written down anywhere: helm --wait defaults to a five-minute timeout, shorter than the startup it waits for; the liveness probe keeps a narrow 3 × 10s window once the startup probe succeeds; and the troubleshooting note now uses the Killing event as the discriminator rather than exit code 137, which is also the OOMKilled signature.

Testing

helm lint, a render of all four CI values files and stock values.yaml, and a kind cluster on the CI node image running the full ct lint and ct install chain. The probe mechanism itself was verified separately: two otherwise identical pods taking 90s to open /health, one with a window shorter than the startup and one longer - the first is killed and restarts, the second reaches Ready.

Supersedes #801, which proposed 230s before the longer startups were reported.

Closes #800

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[tomklapka]

Verified locally on a kind cluster using the same node image as CI (v1.31.14), running ct lint, ct install and the workflow's log pattern check. All four CI values files install, the chart's helm test passes, and the pattern check finds nothing.

On the point raised in #800, that probe config cannot affect this much: both readings hold, they just answer different questions. The window does not shorten the replay, and the pod is not Ready until the replay finishes. What the window decides is whether the pod is allowed to get that far.

Two otherwise identical pods, each taking 90s before opening /health:

startup window	outcome	restarts
40s, shorter than the start	killed, exitCode: 137	1, then repeating
310s, longer than the start	Ready	0

For the first one the kubelet logs Container app failed startup probe, will be restarted, while the application log shows only its normal startup work and no error. That is the signature #800 describes: exit 137 with clean application logs. Each restart then begins the replay from the start, which is the loop being reported.

So a wider window does not make a slow start faster, and this PR does not claim that it does. It stops a slow start from turning into a restart loop.

One tradeoff worth stating: a wider window also delays the point at which a pod that will never start gets killed. In CI that is hidden behind the helm install --wait timeout, which fires first.

[tomklapka]

Updated after a review pass. Four things changed.

The window now moves through failureThreshold alone (12 -> 184), leaving initialDelaySeconds and periodSeconds as they were. Raising those delayed every normal startup by 20-30s without widening anything, and #807 set the precedent of touching only the threshold.

The arithmetic was one period optimistic. The window is initialDelay + (threshold - 1) x period, so the old 10s + (30 x 5s) = 160s comment described 155s. Comments and docs now use the correct form.

Exit code 137 is 128 + SIGKILL, which is also the OOMKilled signature, so the troubleshooting note no longer treats it as proof of a probe kill. It now shows how to read lastState.terminated.reason and distinguish the two before touching any threshold.

Three gaps are documented rather than silently left open. helm --wait defaults to a five-minute timeout, shorter than the startup it is waiting for, so the section says to pass --timeout 20m. The liveness probe keeps its narrow 3 x 10s window once the startup probe succeeds, which can still restart a node that answers /health and then blocks. And both compactedData (#828) and ingester.persistence shorten the startup itself instead of tolerating it, which is the better fix where it applies.

Not addressed here, worth separate issues if wanted: the probe block is shared by all four components, so a misconfigured querier now takes 15 minutes to surface instead of 160s, and influxdb3-core still carries the original 70s window with a comment describing a 15-second startup.

[bednar]

Two documentation issues need correction before merge; both comments are inline.

[bednar]

Please also update the PR description before merge. It still describes the obsolete configuration as a 930-second window made from a 30-second initial delay and 90 checks every 10 seconds, while this revision keeps a 10-second initial delay and 5-second period and changes failureThreshold from 30 to 184. The description should match the values and timing documented by the final diff. Two additional documentation issues are called out inline.

[bednar]

Two Kubernetes object-lifecycle terminology corrections are needed; both comments are inline.

[bednar]

A Pod itself does not restart when a startup probe fails; kubelet restarts the affected container within the existing Pod. Please change this to A container that restarts during startup... or A pod whose application container restarts during startup... so the troubleshooting guidance describes the lifecycle accurately.

[bednar]

The startup probe does not kill the Pod. After failureThreshold consecutive failures, kubelet triggers termination and restart of the affected container while the Pod remains. Please describe this as, for example, Failed startup checks before container termination is triggered; that also remains consistent with the terminationGracePeriodSeconds explanation above.