openshift · tchap · Jun 26, 2026 · Jul 20, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -0,0 +1,92 @@
+# Cluster Kube Controller Manager Operator
+
+A static pod operator that manages the lifecycle of `kube-controller-manager` on OpenShift control plane nodes. Built on the [library-go](https://github.com/openshift/library-go) static pod operator framework, it observes cluster configuration, rotates CSR signing certificates, manages the cluster-policy-controller sidecar, and reconciles the target kube-controller-manager config into static pod manifests. Installed by the [Cluster Version Operator](https://github.com/openshift/cluster-version-operator) (CVO).
+
+See [ARCHITECTURE.md](ARCHITECTURE.md) for the full design and data flow.
+
+## Build and Test
+
+```bash
+make build                       # Build all binaries (operator + OTE test runner)
+make test-unit                   # Unit tests (./pkg/... ./cmd/...)
+make verify                      # Formatting, vetting, golang version checks
+make test-e2e                    # E2E operator tests (30m timeout)
+make test-e2e-preferred-host     # Preferred host e2e tests (1h timeout)
+```
+
+Go version: see `go.mod`.
+
+## Project Structure
+
+| Directory | Purpose |
+|-----------|---------|
+| `cmd/cluster-kube-controller-manager-operator/` | Operator binary entry point (operator, render, installer, pruner, resource-graph, cert-sync, recovery-controller) |
+| `cmd/cluster-kube-controller-manager-operator-tests-ext/` | OTE test runner entry point |
+| `pkg/operator/starter.go` | Operator initialization — creates clients, informers, and starts all controllers |
+| `pkg/operator/targetconfigcontroller/` | Renders observed config + defaults into kube-controller-manager ConfigMaps/Secrets |
+| `pkg/operator/configobservation/` | Configuration observers — each subdirectory watches a cluster resource type |
+| `pkg/operator/certrotationcontroller/` | CSR signer certificate rotation and SA token signer controller |
+| `pkg/operator/resourcesynccontroller/` | Syncs ConfigMaps/Secrets between namespaces |
+| `pkg/operator/operatorclient/` | Namespace constants and operator client interfaces |
+| `pkg/operator/gcwatchercontroller/` | Monitors garbage collector metrics via Prometheus |
+| `pkg/cmd/operator/` | Operator subcommand — wires `RunOperator()` into the binary's command tree |
+| `pkg/cmd/render/` | Bootstrap manifest renderer for cluster installation |
+| `pkg/cmd/recoverycontroller/` | Certificate recovery controller (CSR signer + CSR approval) |
+| `pkg/cmd/resourcegraph/` | Resource dependency chain visualization |
+| `bindata/` | Embedded assets: default config, static pod template, RBAC, bootstrap manifests, vSphere resources |
+| `manifests/` | CVO deployment manifests (namespace, deployment, RBAC, ServiceMonitors, alerts) |
+| `test/e2e/` | E2E test suite (operator, network policy, SA token signer) |
+| `test/e2e-preferred-host/` | Preferred host communication tests |
+| `test/library/` | Shared test utilities |
+
+## Controller Pattern
+
+Controllers use the library-go `factory.Controller` base. Each controller has a `sync(ctx, syncContext)` method called by the framework on informer events or periodic resyncs. The operator wires them in `pkg/operator/starter.go` via `RunOperator()`.
+
+Config observers follow a specific pattern: each observer function receives the existing config and returns `(observedConfig, errors)`. Observers are registered in `pkg/operator/configobservation/configobservercontroller/observe_config_controller.go`.
+
+## Key Conventions
+
+- **Namespaces:** `openshift-kube-controller-manager-operator` (operator), `openshift-kube-controller-manager` (operand), `openshift-config` (user config), `openshift-config-managed` (platform config). Constants in `pkg/operator/operatorclient/interfaces.go`.
+- **Logging:** `k8s.io/klog/v2` with verbosity levels
+- **Error handling:** wrap with `fmt.Errorf("context: %w", err)`
+- **Feature gates:** controllers that depend on feature gates use `FeatureGateAccessor` from library-go; wait for gates before starting
+- **Platform conditionals:** vSphere legacy cloud provider resources are only deployed when `Infrastructure.Status.PlatformStatus.Type == VSpherePlatformType`
+- **Upstream changes:** controllers that wrap library-go functionality should have fixes made upstream in [library-go](https://github.com/openshift/library-go), not here
+
+## Critical Rules
+
+1. **Never edit `vendor/` directly.** Change `go.mod`, then `go mod tidy && go mod vendor`. Always commit vendor changes separately from code changes for reviewable diffs.
+
+2. **Static pod template changes affect all control plane nodes.** `bindata/assets/kube-controller-manager/pod.yaml` defines four containers (kube-controller-manager, cluster-policy-controller, cert-syncer, recovery-controller). Changes here trigger rolling restarts across control plane.
+
+3. **CVO manifest ordering matters.** Files in `manifests/` are prefixed `0000_25_` for run-level 25. This operator must upgrade after kube-apiserver (run-level 20). Don't change the prefix.
+
+4. **Cert rotation has safety gates.** The SA token signer waits for bootstrap node departure and uses a 5-minute promotion delay. Don't bypass these — they prevent token validation failures cluster-wide.
+
+5. **This operator does not run in HyperShift.** The operator Deployment is excluded from hosted control plane topologies. HyperShift's control-plane-operator manages KCM directly.
+
+6. **Feature gate changes cause operator exit.** The operator process calls `os.Exit(0)` when the resolved feature gate set changes (by design, via library-go). This is a restart, not a crash.
+
+## What NOT to Do
+
+- **Don't read cluster config directly in TargetConfigController.** Use the ObservedConfig pattern — ConfigObserver writes to the CR status, TargetConfigController reads from there.
+- **Don't add HyperShift logic.** This operator is standalone-only. HyperShift has its own KCM management.
+- **Don't modify `pkg/operator/configobservation/network/` without networking team review.** It has separate OWNERS.
+- **Don't skip `make verify` before submitting.** CI runs gofmt, govet, and Go version checks.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines. Key rules:
+
+- Do not modify files under `vendor/`. Use `go mod tidy && go mod vendor`.
+- `bindata/assets.go` uses Go's `embed` directive to embed asset files — update the embedded files, not this file.
+- Write unit tests for behavior and implementation changes. E2E tests for significant features. Documentation-only changes need `make verify` to pass.
+- Backwards compatibility matters — deprecate before removing.
+- Before modifying the operator API, ensure there is a corresponding enhancement proposal in [openshift/enhancements](https://github.com/openshift/enhancements). API changes require design review and approval.
+
+## Testing
+
+- **Unit tests:** co-located `*_test.go` files, table-driven, `go test ./pkg/... ./cmd/...`
+- **E2E tests:** suites under `test/e2e/` and `test/e2e-preferred-host/`, using Ginkgo v2.
+- **OTE framework:** `cluster-kube-controller-manager-operator-tests-ext` binary. See [CONTRIBUTING.md](CONTRIBUTING.md#openshift-tests-extension-ote) for usage.
diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md
@@ -0,0 +1,168 @@
+# Architecture
+
+## Overview
+
+The cluster-kube-controller-manager-operator is a static pod operator that manages the `kube-controller-manager` on OpenShift control plane nodes. It is deployed by the Cluster Version Operator (CVO) and uses the [library-go](https://github.com/openshift/library-go) static pod operator framework.
+
+The operator's primary responsibilities:
+- Observe cluster configuration from multiple sources and synthesize kube-controller-manager config
+- Manage kube-controller-manager static pods across control plane nodes (install, revision, prune)
+- Rotate CSR signing certificates
+- Manage service account token signing keys
+- Run the cluster-policy-controller as a sidecar in the same static pod
+- Report status via the `ClusterOperator/kube-controller-manager` resource
+
+## Data Flow
+
+```text
+ config.openshift.io resources          Secrets/ConfigMaps
+ (Infrastructure, Network, Node,        (service-account-private-key,
+  FeatureGate, Proxy, ...)               service-ca, cloud-config, ...)
+              |                                    |
+              v                                    v
+   +--------------------------------------------------+
+   |              Config Observer Controllers           |
+   |  (observe external state, produce observedConfig)  |
+   +------------------------+--------------------------+
+                            | observedConfig (sparse JSON)
+                            v
+   +--------------------------------------------------+
+   |           Target Config Controller                 |
+   |  (merge defaults + observedConfig + overrides      |
+   |   -> render ConfigMaps/Secrets in target ns)       |
+   +------------------------+--------------------------+
+                            | ConfigMaps, Secrets
+                            v
+   +--------------------------------------------------+
+   |          Static Pod Controllers (library-go)       |
+   |  Installer -> Revision Controller -> Pruner        |
+   |  (roll out new revisions to each control plane     |
+   |   node as static pod manifests)                    |
+   +------------------------+--------------------------+
+                            |
+                            v
+              kube-controller-manager static pods
+              (one per control plane node)
+```
+
+## Operator Startup
+
+Entry point: `cmd/cluster-kube-controller-manager-operator/main.go` -> `pkg/cmd/operator/cmd.go` -> `pkg/operator/starter.go:RunOperator()`.
+
+Startup sequence:
+1. Create clients (Kubernetes, config, operator)
+2. Create informers for watched namespaces (see [Namespaces](#namespaces))
+3. Initialize feature gates via `FeatureGateAccessor` and wait for observation (1-minute timeout)
+4. Create and start all controllers concurrently
+5. Block until context cancellation
+
+## Namespaces
+
+| Namespace | Constant | Purpose |
+|-----------|----------|---------|
+| `openshift-config` | `GlobalUserSpecifiedConfigNamespace` | User-provided configuration (certs, CAs, cloud config) |
+| `openshift-config-managed` | `GlobalMachineSpecifiedConfigNamespace` | Platform-managed configuration (generated CAs, signing certs) |
+| `openshift-kube-controller-manager-operator` | `OperatorNamespace` | Operator deployment and its resources (CSR signer secrets) |
+| `openshift-kube-controller-manager` | `TargetNamespace` | Operand: kube-controller-manager pods, config, certs |
+| `kube-system` | — | Additional watched namespace |
+| `openshift-infra` | — | Created by static resource controller |
+
+The `ResourceSyncController` copies ConfigMaps and Secrets between these namespaces as needed.
+
+## Static Pod Management
+
+The operator uses library-go's `staticpod.NewBuilder()` to manage kube-controller-manager static pods. This framework provides:
+
+- **Installer controller** — creates new static pod revisions on each control plane node. Uses a custom installer command (`cluster-kube-controller-manager-operator installer`).
+- **Revision controller** — tracks revisions of ConfigMaps and Secrets. When any revisioned resource changes, a new revision is created. The first ConfigMap in the list (`kube-controller-manager-pod`) contains the static pod manifest template.
+- **Pruner** — removes old static pod revisions to free disk space.
+- **PDB guard** — ensures availability during upgrades (only on multi-node clusters; disabled for single-node).
+
+Resources are split into two categories (see the `deploymentConfigMaps`, `deploymentSecrets`, `CertConfigMaps`, and `CertSecrets` variables in `pkg/operator/starter.go` for the authoritative list):
+- **Revisioned** — ConfigMaps and Secrets passed to `WithRevisionedResources`. A change to any of these triggers a new static pod revision.
+- **Unrevisioned certs** — ConfigMaps and Secrets passed to `WithUnrevisionedCerts`. These are updated in-place without triggering a revision.
+
+## Configuration Observers
+
+Configuration observers watch external cluster resources and produce a sparse JSON config (`observedConfig`) that gets merged into the kube-controller-manager configuration. Each observer function receives the existing config and returns `(observedConfig, errors)`.
+
+Observers are registered in `pkg/operator/configobservation/configobservercontroller/observe_config_controller.go`:
+
+| Observer | Watches | Config paths set |
+|----------|---------|-----------------|
+| `CloudProviderObserver` | `Infrastructure` CR | Cloud provider config for the target namespace |
+| `FeatureGatesObserver` (KCM) | `FeatureGate` CR | `extendedArguments.feature-gates` (excludes OpenShift-only gates) |
+| `FeatureGatesObserver` (cluster-policy-controller) | `FeatureGate` CR | `featureGates` |
+| `ObserveClusterCIDRs` | `Network` CR | `extendedArguments.cluster-cidr` |
+| `ObserveServiceClusterIPRanges` | `Network` CR | `extendedArguments.service-cluster-ip-range` |
+| `LatencyProfileObserver` | `Node` CR | `extendedArguments.node-monitor-grace-period` (Default: 40s, Medium: 2m, Low: 5m) |
+| `ProxyObserver` | `Proxy` CR | `targetconfigcontroller.proxy` |
+| `ObserveServiceCA` | Service CA ConfigMap | `serviceServingCert.certFile` |
+| `ObserveInfraID` | `Infrastructure` CR | `extendedArguments.cluster-name` |
+| `ObserveTLSSecurityProfile` | `APIServer` CR | TLS cipher suites and min version |
+
+Several observers include latency profile suppression logic to prevent config updates during extreme profile transitions.
+
+## Target Config Controller
+
+`pkg/operator/targetconfigcontroller/` takes the merged configuration (defaults + observedConfig + unsupportedConfigOverrides) and renders it into concrete resources in the target namespace:
+
+- `config` ConfigMap — the main kube-controller-manager configuration
+- `kube-controller-manager-pod` ConfigMap — the static pod manifest template (kube-controller-manager + cluster-policy-controller containers)
+- `cluster-policy-controller-config` ConfigMap — configuration for the cluster-policy-controller sidecar
+- `controller-manager-kubeconfig` and `kube-controller-cert-syncer-kubeconfig` ConfigMaps
+- `recycler-config` ConfigMap — persistent volume recycler configuration
+- `serviceaccount-ca` ConfigMap — CA bundle for service account token verification
+
+The default kube-controller-manager configuration lives in `bindata/assets/config/defaultconfig.yaml`. It enables leader election, dynamic provisioning, and all controllers except `ttl`, `bootstrapsigner`, and `tokencleaner`.
+
+## Certificate Rotation
+
+`pkg/operator/certrotationcontroller/` manages CSR signing certificate rotation:
+
+- **CSR signer signer** (`csr-signer-signer`) — the CA that signs the CSR signer, stored in the operator namespace. Validity: 2x refresh period, refresh: 30 days (or 2 hours with `ShortCertRotation` feature gate).
+- **CSR controller signer CA** (`csr-controller-signer-ca`) — CA bundle ConfigMap in the operator namespace.
+- **CSR signer** (`csr-signer`) — the active CSR signing certificate, stored in the operator namespace and synced to the target namespace. Validity: 1x refresh period, refresh: half the refresh period. Signs kubelet CSRs.
+
+## SA Token Signer
+
+`pkg/operator/certrotationcontroller/satokensigner_controller.go` manages the `next-service-account-private-key` secret. When the current `service-account-private-key` is about to expire (or is missing), it generates a new key pair and stores it as the "next" key for rotation.
+
+## Recovery Controller
+
+`pkg/cmd/recoverycontroller/` provides a certificate recovery mechanism for when CSR signing certificates have expired:
+
+- Runs the cert rotation controller in `RefreshOnlyWhenExpired` mode to regenerate expired certificates
+- Includes a CSR approval controller (`pkg/cmd/recoverycontroller/csrcontroller.go`) that auto-approves kubelet CSRs signed by the recovered signer
+- Invoked via the `cert-recovery-controller` subcommand
+
+## Render Command
+
+`pkg/cmd/render/` is a bootstrap manifest renderer used during cluster installation. It takes installer-provided inputs (cloud provider config, feature gates, cluster CIDRs, images) and renders the initial set of manifests needed to bootstrap kube-controller-manager before the operator is running. The templates live in `bindata/bootkube/`.
+
+## Other Controllers
+
+| Controller | Purpose |
+|-----------|---------|
+| `StaticResourceController` | Applies static manifests from `bindata/` (namespace, service, RBAC, network policies). Conditionally deploys vSphere legacy cloud provider resources. |
+| `ClusterOperatorStatusController` | Reports operator status, versions, and related objects to `ClusterOperator/kube-controller-manager` |
+| `GarbageCollectorWatcherController` | Monitors garbage collector sync failures via Prometheus metrics on the kube-controller-manager |
+| `LatencyProfileController` | Manages latency profile configuration, coordinates with the installer to reject extreme profiles during transitions |
+
+## Design Decisions
+
+1. **Static pod pattern over Deployment:** The kube-controller-manager runs as a static pod managed by kubelet, not as a Deployment. This avoids a circular dependency — KCM manages controllers that Deployments depend on (e.g., service account token controller). The operator writes pod manifests that kubelet picks up directly.
+
+2. **Revision-based rollouts:** Configuration changes create new revisions (numbered copies of ConfigMaps/Secrets). The installer controller rolls out one node at a time by writing a new static pod manifest referencing the latest revision. This provides rollback capability and audit trail.
+
+3. **Cluster-policy-controller as a sidecar:** The cluster-policy-controller ([openshift/cluster-policy-controller](https://github.com/openshift/cluster-policy-controller)) runs as a container in the KCM static pod rather than as a separate Deployment. This was a deliberate decision made for OpenShift 4.3 (PR #297, October 2019). The primary driver is a **bootstrap chicken-and-egg problem**: CPC's controllers (namespace SCC allocation, quota reconciliation, PSA label syncing) must be running before any Deployments can be scheduled — pods cannot be created without UID range and SELinux label allocation. Placing these controllers in a static pod breaks the circular dependency. The KCM pod was chosen because CPC shares the same service account (`system:kube-controller-manager`), RBAC, certificates, kubeconfig, and leader election namespace, avoiding infrastructure duplication.
+
+4. **5-minute SA key promotion delay:** New service account signing keys are staged in `next-service-account-private-key` for 5 minutes before promotion. This gives the kube-apiserver time to observe the new public key via the `sa-token-signing-certs` bundle, preventing token validation failures during rotation.
+
+5. **Bootstrap node departure gate:** SA token key rotation is blocked until the bootstrap node has left the cluster. The bootstrap node uses a different signing key; rotating before it departs could cause token validation failures for workloads it started.
+
+6. **ObservedConfig indirection:** Rather than reading cluster config directly in the target config controller, a separate ConfigObserver writes a merged JSON blob to `.status.observedConfig` on the CR. This decouples config sources from config consumers and makes the effective config inspectable via `oc get kubecontrollermanager cluster -o jsonpath='{.status.observedConfig}'`.
+
+7. **24-hour bootstrap signer with operator takeover:** The installer deliberately creates a short-lived CSR signer (24h) to minimize trust window during bootstrap. This operator's cert rotation controller takes ownership and issues a longer-lived replacement, ensuring the cluster transitions from minimal-trust bootstrap to managed cert lifecycle.
+
+8. **UseMoreSecureServiceCA bypasses ObservedConfig (tech debt):** The TargetConfigController reads `.spec.useMoreSecureServiceCA` directly from the operator spec rather than going through the ObservedConfig pattern. This is acknowledged in code as needing migration to a config observer, but requires changes to the observedConfig format.