feat(kafka): add local NVMe reference with repair DaemonSet - #80
Merged
Conversation
animmo-cw
approved these changes
Aug 6, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new kafka-local-nvme reference architecture package for running Strimzi Kafka on CoreWeave node-local NVMe (/mnt/local/kafka), including an opt-in “fail-closed” repair DaemonSet plus supporting scripts/tests and documentation.
Changes:
- Introduces a local-path provisioner Kustomize package and two Strimzi profiles (
two-node,five-node) for Kafka on local PVs. - Adds an optional repair-agent (Go) and a renderer script to generate the DaemonSet + RBAC from an immutable config allowlist.
- Adds validation/smoke-test scripts and customer-facing documentation plus licensing/third-party notices.
Reviewed changes
Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Adds top-level index entry for the new Kafka local NVMe reference. |
| kafka-local-nvme/THIRD_PARTY_NOTICES.md | Documents third-party provenance for provisioner manifests. |
| kafka-local-nvme/tests/render-kafka-local-pv-repair-manifest-test.sh | Tests the renderer output for required resources and safety constraints. |
| kafka-local-nvme/scripts/validate.sh | Validates manifests render and runs Go + renderer tests. |
| kafka-local-nvme/scripts/smoke-test.sh | Provides a basic produce/consume readback smoke test using kubectl. |
| kafka-local-nvme/scripts/render-kafka-local-pv-repair-manifest.sh | Renders the repair DaemonSet/RBAC/ConfigMap from inputs + allowlist. |
| kafka-local-nvme/README.md | Package entry-point documentation and operator guidance. |
| kafka-local-nvme/provisioner/local-path/storage-class.yaml | Defines kafka-local StorageClass with local PV behavior. |
| kafka-local-nvme/provisioner/local-path/repair/service-account.yaml | ServiceAccount for the optional repair DaemonSet. |
| kafka-local-nvme/provisioner/local-path/repair/rbac.yaml | Namespaced Role/RoleBinding for repair PVC reads. |
| kafka-local-nvme/provisioner/local-path/repair/daemonset.template.yaml | Template for repair ConfigMap + ClusterRole/Binding + DaemonSet. |
| kafka-local-nvme/provisioner/local-path/rbac.yaml | Provisioner RBAC (derived from Rancher local-path-provisioner). |
| kafka-local-nvme/provisioner/local-path/namespace.yaml | Namespace for the local-path provisioner components. |
| kafka-local-nvme/provisioner/local-path/kustomization.yaml | Kustomize entry for provisioning stack resources. |
| kafka-local-nvme/provisioner/local-path/deployment.yaml | Deploys pinned local-path-provisioner v0.0.36. |
| kafka-local-nvme/provisioner/local-path/config-map.yaml | Provisioner config + helper pod setup/teardown scripts. |
| kafka-local-nvme/profiles/two-node/topic.yaml | Two-node profile KafkaTopic definition. |
| kafka-local-nvme/profiles/two-node/kustomization.yaml | Two-node profile kustomization entry. |
| kafka-local-nvme/profiles/two-node/kafka.yaml | Two-node Kafka cluster CR with listeners/config. |
| kafka-local-nvme/profiles/two-node/controllers.yaml | Two-node controllers KafkaNodePool manifest. |
| kafka-local-nvme/profiles/two-node/brokers.yaml | Two-node brokers KafkaNodePool manifest. |
| kafka-local-nvme/profiles/five-node/topic.yaml | Five-node profile KafkaTopic definition. |
| kafka-local-nvme/profiles/five-node/kustomization.yaml | Five-node profile kustomization entry. |
| kafka-local-nvme/profiles/five-node/kafka.yaml | Five-node Kafka cluster CR (enables Cruise Control). |
| kafka-local-nvme/profiles/five-node/controllers.yaml | Five-node controllers KafkaNodePool manifest. |
| kafka-local-nvme/profiles/five-node/brokers.yaml | Five-node brokers KafkaNodePool manifest. |
| kafka-local-nvme/LICENSE | Adds package-level Apache-2.0 license text. |
| kafka-local-nvme/internal/repair/types.go | Defines core repair types/config structures and interfaces. |
| kafka-local-nvme/internal/repair/select.go | Implements candidate selection and expected path validation. |
| kafka-local-nvme/internal/repair/repair_test.go | Linux-only tests for selection/mount verification/hierarchy behavior. |
| kafka-local-nvme/internal/repair/mount_linux.go | Verifies host mount signature via /proc/1/mountinfo. |
| kafka-local-nvme/internal/repair/identity.go | Verifies live node identity matches allowlist. |
| kafka-local-nvme/internal/repair/fs_linux.go | Safely recreates missing directory hierarchy under /mnt/local. |
| kafka-local-nvme/internal/repair/engine_linux.go | Orchestrates read-only verification + safe hierarchy creation loop. |
| kafka-local-nvme/internal/repair/api.go | Implements in-cluster Kubernetes API reader and config loading. |
| kafka-local-nvme/go.mod | Declares the Go module for the repair agent. |
| kafka-local-nvme/docs/customer-reference-architecture.md | Customer-facing architecture scope/limits and operational boundaries. |
| kafka-local-nvme/Dockerfile | Builds the repair agent container image (distroless runtime). |
| kafka-local-nvme/cmd/kafka-local-pv-repair/main.go | Repair agent entrypoint and reconciliation loop. |
| kafka-local-nvme/.dockerignore | Docker build context exclusions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+24
to
+40
| token, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/token") | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| ca, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| pool := x509.NewCertPool() | ||
| if !pool.AppendCertsFromPEM(ca) { | ||
| return nil, errors.New("invalid service account CA") | ||
| } | ||
| host, port := os.Getenv("KUBERNETES_SERVICE_HOST"), os.Getenv("KUBERNETES_SERVICE_PORT") | ||
| if host == "" || port == "" { | ||
| return nil, ErrUnavailable | ||
| } | ||
| return &APIReader{&http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}}}, "https://" + host + ":" + port, string(token)}, nil |
Comment on lines
+10
to
+13
| command -v kubectl >/dev/null 2>&1 || { | ||
| printf 'kubectl is required for Kustomize rendering.\n' >&2 | ||
| exit 1 | ||
| } |
| --nodes nodes.tsv | kubectl apply -f - | ||
| ``` | ||
|
|
||
| 5. Obtain the required Pod Security approval for `hostPID` and a read-write `/mnt/local` hostPath. The container runs as root only to create missing root-owned hierarchy components, with `CHOWN` and `DAC_OVERRIDE` as its only added capabilities. Check the DaemonSet rollout and logs, and serialize all related operator maintenance. A newly rendered authorization changes the Pod template and rolls the DaemonSet automatically. |
|
|
||
| The repository does not install the Repair DaemonSet through its normal Kustomizations. When explicitly rendered and applied, it is a privileged host-storage component: it uses `hostPID` to inspect `/proc/1/mountinfo` and a read-write hostPath for `/mnt/local`. Its trust boundary requires a Pod Security exception, deliberate operator ownership, and serialized maintenance. | ||
|
|
||
| The agent may recreate only missing components of the exact managed directory hierarchy for a Bound local PV with an authorized same Node identity and a verified host mount signature. It does not restore data, delete directories, or write Kubernetes PV/PVC objects. Kafka rebuilds only from healthy replicas; this is not a claim of automatic reboot or replacement recovery, cleanup, or provider validation. It runs as root solely to create missing root-owned directories, with only `CHOWN` and `DAC_OVERRIDE` added after dropping all capabilities. |
| @@ -0,0 +1,3 @@ | |||
| module github.com/coreweave/reference-architecture/kafka-local-nvme | |||
|
|
|||
| go 1.24.0 | |||
| @@ -0,0 +1,12 @@ | |||
| FROM golang:1.24-alpine AS build | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification