Skip to content

feat(influxdb3-enterprise): Update chart for InfluxDB 3.11 with PachaTree support - #818

Open
dburton-influxdata wants to merge 4 commits into
influxdata:masterfrom
dburton-influxdata:feature/influxdb-3.11-support
Open

feat(influxdb3-enterprise): Update chart for InfluxDB 3.11 with PachaTree support#818
dburton-influxdata wants to merge 4 commits into
influxdata:masterfrom
dburton-influxdata:feature/influxdb-3.11-support

Conversation

@dburton-influxdata

Copy link
Copy Markdown
Contributor

Pull Request: InfluxDB 3.11 Helm Chart Update

Summary

Updates the InfluxDB 3 Enterprise Helm chart to support version 3.11.0, introducing PachaTree storage engine, graceful shutdown, and Processing Engine enhancements.

Chart Version: 0.8.0 → 0.9.0
App Version: 3.9.3 → 3.11.0
Closes: #817


⚠️ Live Infrastructure Testing Required

Static validation completed:

  • ✅ Helm lint passed (0 errors)
  • ✅ Template rendering validated (all scenarios)
  • ✅ PachaTree configuration verified
  • ✅ Graceful shutdown verified
  • ✅ Processing Engine updates verified

Live infrastructure testing needed:

  • ⏳ Fresh install on K8s cluster (verify PachaTree default behavior)
  • ⏳ Upgrade from 0.8.0 to 0.9.0 on existing cluster (verify Parquet preserved)
  • ⏳ PachaTree migration with real data (verify one-time migration)
  • ⏳ Runtime behavior validation with InfluxDB 3.11 binary

Test guidance: See charts/influxdb3-enterprise/MIGRATION-3.11.md for detailed test scenarios.


Changes

Breaking Changes (Fully Backward Compatible)

Environment Variables (12 renames):

  • INFLUXDB3_ENTERPRISE_*INFLUXDB3_* prefix
  • Old names still work with deprecation warnings in InfluxDB 3.11

WAL Configuration (3 renames):

  • WAL_SNAPSHOT_SIZEWAL_FILES_PER_SNAPSHOT
  • WAL_MAX_WRITE_BUFFER_SIZEWAL_MAX_BUFFERED_WRITES
  • NUM_WAL_FILES_TO_KEEPSNAPSHOTTED_WAL_FILES_TO_KEEP
  • Old names still work with deprecation warnings

Memory Configuration (1 rename + format change):

  • EXEC_MEM_POOL_BYTESEXEC_MEM_POOL_SIZE
  • Format change: Now requires unit suffix (e.g., "2GB")
  • Bare numbers are rejected by InfluxDB 3.11

Removed Options (2 deprecated flags):

  • objectStorage.cacheEndpoint (now automatic)
  • dataLifecycle.hardDeleteDefaultDuration (cluster-level only)

New Features

PachaTree Storage Engine Support

  • New clusters automatically use PachaTree (no configuration needed)
  • Existing Parquet clusters can opt-in to migration via engine.upgradePachaTree: true
  • Configuration options: shardCount, maxTotalColumns, snapshotSize, gen0MaxFileSize, walReplicaQueueLength

Configuration example:

engine:
  upgradePachaTree: true    # Set ONCE to start migration, then remove
  pachaTree:
    shardCount: 2
    maxTotalColumns: 10000
    snapshotSize: 200000

Graceful Shutdown

  • Configurable connection drain timeout (default: 30s)
  • shutdown.timeout configuration option
  • Improves rolling update experience

Configuration example:

shutdown:
  timeout: "60s"    # Drain connections for 60s before shutdown

Processing Engine Updates

  • disablePackageManagement: Lock down package installations
  • asyncTriggerConcurrencyLimit: Control async trigger concurrency

Configuration example:

processingEngine:
  disablePackageManagement: true          # Reject package install API calls
  asyncTriggerConcurrencyLimit: 10        # Max concurrent async triggers

Documentation

MIGRATION-3.11.md (25KB)

Comprehensive upgrade guide with 4 scenarios:

  1. Fresh install - PachaTree automatic
  2. Existing cluster - Stay on Parquet
  3. PachaTree migration - Detailed migration procedure
  4. Removed options - Handling deprecated configurations

Includes:

  • Breaking changes summary
  • Post-upgrade validation checklist
  • Rollback procedures
  • Troubleshooting guide (9 common issues)
  • Container image references

CHANGELOG.md (10KB)

Detailed v0.9.0 release notes following Keep a Changelog format:

  • Added: PachaTree, graceful shutdown, Processing Engine updates
  • Changed: 12 env vars, 3 WAL vars, 1 memory var
  • Removed: 2 deprecated options
  • Container images section (3 sources, 8 direct download links)

README.md Updates

  • "What's New in 0.9.0" section
  • "Storage Engine (PachaTree)" configuration section
  • "Graceful Shutdown" configuration section
  • Enhanced "Processing Engine" section
  • Enhanced "Upgrading" section with migration guide link
  • Container image references

Container Images

Quay.io (Recommended)

Enterprise:

image:
  registry: quay.io
  repository: influxdb/influxdb3-enterprise
  tag: "e5242f505d23039a340d21693a994b1a053b0f15"

Core:

image:
  registry: quay.io
  repository: influxdb/influxdb3-core
  tag: "139bab4c54b54db01d67539b6dc9f1e1a81dd1b7"

Docker Hub

  • Enterprise: docker.io/influxdb/influxdb3-enterprise:3.11.0-enterprise
  • Core: docker.io/influxdb/influxdb3-core:3.11.0-core

Direct Downloads (8 packages)

  • Enterprise/Core RPM (x86_64, aarch64)
  • Enterprise/Core DEB (amd64, arm64)

See CHANGELOG.md for full download links.


Backward Compatibility

✅ Fully Backward Compatible

Existing clusters upgrade seamlessly:

  • No data migration required (unless opting into PachaTree)
  • Old environment variable names still work (with deprecation warnings)
  • Existing Parquet clusters stay on Parquet (migration is opt-in)
  • No configuration changes required (unless using removed options)

Breaking Only For Users With:

  1. objectStorage.cacheEndpoint in values → Remove (now automatic)
  2. dataLifecycle.hardDeleteDefaultDuration in values → Remove (cluster-level only)
  3. Manual memory.execMemPoolBytes override → Update to size format with units

Mitigation: Clear error messages + detailed troubleshooting in MIGRATION-3.11.md


Testing Performed

Static Validation ✅

  • ✅ Helm lint: 0 errors, 0 warnings
  • ✅ Template rendering: All scenarios successful
  • ✅ PachaTree configuration: Args + env vars verified
  • ✅ Graceful shutdown: ConfigMap env var verified
  • ✅ Processing Engine: New env vars verified
  • ✅ Image version: 3.11.0-enterprise correct

Live Infrastructure Testing ⏳ (Maintainer Action Required)

  • ⏳ Fresh install test (verify PachaTree default)
  • ⏳ Upgrade test 0.8.0 → 0.9.0 (verify Parquet preserved)
  • ⏳ PachaTree migration test (verify migration flow)
  • ⏳ Runtime validation (InfluxDB 3.11 binary behavior)

Test Guidance: See MIGRATION-3.11.md for step-by-step test procedures.


Files Changed

Implementation (8 files)

File Lines Changed Purpose
Chart.yaml 2 Version bump
values.yaml +45, -2 New config sections
templates/configmap.yaml +18, -2 Env var renames
templates/_helpers.tpl +30 PachaTree helper
templates/ingester-statefulset.yaml +5 PachaTree support
templates/querier-statefulset.yaml +5 PachaTree support
templates/compactor-statefulset.yaml +5 PachaTree support
templates/processor-statefulset.yaml +8 Processing Engine

Documentation (3 files)

File Size Lines Purpose
MIGRATION-3.11.md 25 KB ~1,100 Upgrade guide
CHANGELOG.md 10 KB ~400 Release notes
README.md +3 KB +80 Feature docs

Total: 11 files, +1,393 insertions, -5 deletions


Commit History

ccddf97 - Phase 3: Documentation - 3.11 migration guide, changelog, README updates
f61d82b - Phase 2: InfluxDB 3.11 new features - PachaTree, graceful shutdown, Processing Engine
36d3e49 - Phase 1: InfluxDB 3.11 breaking changes - environment variables and configuration

Review Checklist

  • Code changes completed
  • Documentation created (migration guide, changelog, README)
  • Helm lint passed
  • Template rendering validated
  • Container images documented
  • Backward compatibility maintained
  • Breaking changes documented
  • Live infrastructure testing (maintainer validation needed)
  • Integration testing (maintainer validation needed)

References

dburton-influxdata and others added 4 commits July 31, 2026 12:32
…onfiguration

- Update Chart version 0.8.0 → 0.9.0, appVersion 3.9.3 → 3.11.0
- Rename 12 environment variables (INFLUXDB3_ENTERPRISE_* → INFLUXDB3_*)
- Rename WAL environment variables (WAL_SNAPSHOT_SIZE → WAL_FILES_PER_SNAPSHOT, etc.)
- Rename memory environment variables (EXEC_MEM_POOL_BYTES → EXEC_MEM_POOL_SIZE)
- Remove deleted configuration options (hardDeleteDefaultDuration, cacheEndpoint)
- Add breaking change warnings for memory flag validation
- Add cache configuration aliases (file* names alongside parquet* names)

Breaking changes:
- INFLUXDB3_ENTERPRISE_CLUSTER_ID → INFLUXDB3_CLUSTER_ID
- INFLUXDB3_START_WITHOUT_AUTH → INFLUXDB3_WITHOUT_AUTH
- INFLUXDB3_DB_DIR → INFLUXDB3_DATA_DIR
- INFLUXDB3_TELEMETRY_DISABLE_UPLOAD → INFLUXDB3_DISABLE_TELEMETRY_UPLOAD
- All resource limit variables lose ENTERPRISE prefix
- Memory flags now require unit suffixes (bare numbers rejected)

Ref: InfluxDB 3.11 Release Notes
…rocessing Engine

PachaTree Support:
- Add engine configuration section to values.yaml
- Add upgradePachaTree migration flag for existing Parquet clusters
- Add PachaTree-specific configuration (shardCount, maxTotalColumns, snapshotSize, etc.)
- Add --upgrade-pacha-tree arg to all StatefulSets when migration flag set
- Create pachaTreeEnv helper template for PachaTree environment variables
- Apply PachaTree env vars to ingester, querier, compactor StatefulSets

Graceful Shutdown (3.11+):
- Add shutdown.timeout configuration to values.yaml
- Add INFLUXDB3_SHUTDOWN_TIMEOUT to configmap.yaml
- Default 30s connection drain before forcibly closing

Processing Engine Updates (3.11+):
- Add disablePackageManagement flag (replaces packageManager: disabled)
- Add asyncTriggerConcurrencyLimit configuration
- Add INFLUXDB3_DISABLE_PACKAGE_MANAGEMENT env var to processor StatefulSet
- Add INFLUXDB3_ASYNC_TRIGGER_CONCURRENCY_LIMIT env var to processor StatefulSet

New configuration sections:
- engine.upgradePachaTree: Migration flag for Parquet→PachaTree
- engine.pachaTree.*: PachaTree-specific tuning options
- shutdown.timeout: Graceful shutdown connection drain timeout
- processingEngine.disablePackageManagement: Reject package installs
- processingEngine.asyncTriggerConcurrencyLimit: Async trigger concurrency

Ref: InfluxDB 3.11 Release Notes
Added comprehensive documentation for InfluxDB 3.11 upgrade:

MIGRATION-3.11.md (25KB):
- Complete upgrade guide with 4 scenarios (fresh install, existing cluster, PachaTree migration, removed options)
- Breaking changes summary (12 env var renames, 3 WAL renames, 1 memory format change)
- Step-by-step migration procedures with validation steps
- PachaTree migration guide (one-way, one-time process)
- Post-upgrade validation checklist
- Rollback procedures and troubleshooting
- Container image references (Quay.io, Docker Hub, direct downloads)

CHANGELOG.md (10KB):
- Detailed v0.9.0 changelog following Keep a Changelog format
- Added: PachaTree support, graceful shutdown, Processing Engine updates
- Changed: 12 env vars, 3 WAL vars, 1 memory var (all backward compatible)
- Removed: 2 deprecated options (objectStorage.cacheEndpoint, dataLifecycle.hardDeleteDefaultDuration)
- Container image links (Quay.io commit SHAs, Docker Hub tags, RPM/DEB downloads)
- Files changed summary (7 templates, 2 docs, 97 insertions)

README.md updates:
- Added 'What's New in 0.9.0' section highlighting major changes
- Added 'Storage Engine (PachaTree)' configuration section:
  - New clusters (automatic PachaTree)
  - Existing Parquet clusters (stay on Parquet or opt-in migration)
  - PachaTree benefits and tuning options
- Added 'Graceful Shutdown' configuration section
- Updated 'Processing Engine' section with 3.11 security/performance options
- Enhanced 'Upgrading' section with migration guide link and quick upgrade steps
- Added container image references in header (Quay.io, Docker Hub, RPM/DEB)

Container Images Documented:
- Quay.io Enterprise: quay.io/influxdb/influxdb3-enterprise:e5242f505d23039a340d21693a994b1a053b0f15
- Quay.io Core: quay.io/influxdb/influxdb3-core:139bab4c54b54db01d67539b6dc9f1e1a81dd1b7
- Docker Hub Enterprise: docker.io/influxdb/influxdb3-enterprise:3.11.0-enterprise
- Direct downloads: 8 links for Enterprise and Core (RPM/DEB, x86_64/aarch64/amd64/arm64)

Documentation highlights backward compatibility and seamless upgrade path for existing clusters.

Ref: InfluxDB 3.11 Release Notes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant