Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/influxdb3-enterprise/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: influxdb3-enterprise
description: A Helm chart for deploying InfluxDB 3 Enterprise on Kubernetes
type: application
version: 0.10.0
version: 0.10.1
appVersion: "3.11.2"
keywords:
- influxdb
Expand Down
57 changes: 57 additions & 0 deletions charts/influxdb3-enterprise/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,37 @@ ingester:
numThreads: 20
```

#### Health Probes

All components share one probe configuration. The startup probe guards the
initialization window; liveness and readiness start only once it succeeds.

```yaml
probes:
startup:
initialDelaySeconds: 10
periodSeconds: 5
timeoutSeconds: 5
failureThreshold: 184 # 10s + (184 - 1) × 5s = 925s to termination
```

Nodes replay from object storage on boot, so startup scales with how much data a
node reads back, and clusters with a large history have been reported to need
more than 12 minutes. Two things follow from a window this wide.

`helm install` and `helm upgrade` with `--wait` default to a five-minute
timeout, which is shorter than the startup they are waiting for. Pass
`--timeout 20m` to match, or the release fails while the pods are still booting
normally.

Once the startup probe succeeds, the liveness probe takes over with a much
narrower window (`3 × 10s`). A node that answers `/health` and then blocks
during later startup work can still be restarted; raise
`probes.liveness.failureThreshold` if that happens.

This widens the window rather than shortening the startup. If nodes routinely
need most of it, the boot work itself is worth investigating.

#### TLS

Enable TLS with inline cert/key or an existing secret:
Expand Down Expand Up @@ -606,6 +637,26 @@ Check events:
kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0
```

A pod that restarts during startup while its logs show normal activity is most

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Pod itself does not restart when a startup probe fails; kubelet restarts the affected container within the existing Pod. Please change this to A container that restarts during startup... or A pod whose application container restarts during startup... so the troubleshooting guidance describes the lifecycle accurately.

often killed by the startup probe or out of memory, though an externally killed
container can have other causes. Read the termination reason and the kill event
rather than the exit code:

```bash
kubectl get pod -n influxdb3 influxdb3-enterprise-ingester-0 \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
kubectl describe pod -n influxdb3 influxdb3-enterprise-ingester-0 | grep -i killing
```

`OOMKilled` means raise the memory limit. A `Killing` event reading
`failed startup probe, will be restarted` means the probe window is too short;
see [Health Probes](#health-probes). Use that event rather than
`Startup probe failed`, which kubelet records for every failed attempt including
those below `failureThreshold`. Exit code 137 is one possible signature, not
proof: kubelet asks the runtime to terminate first and honours
`terminationGracePeriodSeconds`, so a process that exits during that window
reports a different code, and 137 is also what an OOM kill produces.

#### License Issues

Verify license configuration:
Expand Down Expand Up @@ -680,6 +731,12 @@ logs:
| `security.auth.adminToken.recovery.httpBind` | Bind address for admin token recovery endpoint (`INFLUXDB3_ADMIN_TOKEN_RECOVERY_HTTP_BIND_ADDR`) | `""` |
| `serviceAccount.automountServiceAccountToken` | Configure automatic mounting of the Kubernetes service account token in component pods and the created ServiceAccount | `not set` |
| `extraEnv` | Extra environment variables applied to all components | `[]` |
| `probes.enabled` | Enable liveness, readiness, and startup probes on all components | `true` |
| `probes.startup.initialDelaySeconds` | Delay before the first startup check | `10` |
| `probes.startup.periodSeconds` | Interval between startup checks | `5` |
| `probes.startup.timeoutSeconds` | Timeout of a single startup check | `5` |
| `probes.startup.failureThreshold` | Failed startup checks before the pod is killed | `184` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The startup probe does not kill the Pod. After failureThreshold consecutive failures, kubelet triggers termination and restart of the affected container while the Pod remains. Please describe this as, for example, Failed startup checks before container termination is triggered; that also remains consistent with the terminationGracePeriodSeconds explanation above.

| `probes.liveness.*` / `probes.readiness.*` | Liveness and readiness timings; see `values.yaml` | see `values.yaml` |

### Object Storage Parameters

Expand Down
7 changes: 4 additions & 3 deletions charts/influxdb3-enterprise/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -731,13 +731,14 @@ probes:

# Startup probe settings
# Protects container during initialization. Once successful, never runs again.
# Allows slower startup without delaying successful readiness detection.
# 10s initial + (30 × 5s) = 160s total.
# 10s initial + (184 - 1) × 5s = 925s before the startup probe triggers
# termination; the restart itself follows terminationGracePeriodSeconds.
# Nodes replay from object storage on boot, so startup scales with data size.
startup:
initialDelaySeconds: 10
periodSeconds: 5 # Check every 5 seconds
timeoutSeconds: 5
failureThreshold: 30
failureThreshold: 184

# Liveness probe settings
# Only starts checking AFTER startup probe succeeds
Expand Down
Loading