Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# RustFS Kubernetes Operator

A Kubernetes operator for [RustFS](https://rustfs.com/) object storage, written in Rust with [kube-rs](https://github.com/kube-rs/kube). It reconciles a **`Tenant` custom resource** (`rustfs.com/v1alpha1`), validates referenced credential and KMS Secrets, and applies RBAC, Services, and StatefulSets so RustFS runs inside your cluster, from single-node single-disk development tenants to erasure-coded distributed clusters.
A Kubernetes operator for [RustFS](https://rustfs.com/) object storage, written in Rust with [kube-rs](https://github.com/kube-rs/kube). It reconciles a **`Tenant` custom resource** (`rustfs.com/v1alpha1`), validates referenced credential and KMS Secrets, and applies ServiceAccounts, Services, and StatefulSets so RustFS runs inside your cluster, from single-node single-disk development tenants to erasure-coded distributed clusters.

**Status:** v0.1.0 pre-release — under active development.

Expand Down
2 changes: 2 additions & 0 deletions deploy/k8s-dev/operator-rbac.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch"]
# Legacy Tenant workload RBAC watch/cleanup. Write verbs remain during the
# security migration so an older binary can coexist during a rolling upgrade.
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Expand Down
10 changes: 10 additions & 0 deletions deploy/rustfs-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,16 @@ spec:
image: "registry.example.com/rustfs/rustfs@sha256:<digest>"
```

The Operator also removes legacy Tenant workload Roles and RoleBindings and
disables automatic Kubernetes API token mounting for generated Tenant
ServiceAccounts. Existing default-ServiceAccount Tenants roll once to apply the
Pod template change. A custom image that calls the Kubernetes API must use a
user-owned ServiceAccount with the required token projection and a
least-privilege Role/RoleBinding under names other than the legacy
`{tenant}-role` and `{tenant}-role-binding`, then set `spec.serviceAccountName`.
Do not downgrade after reconciliation: an older Operator recreates the legacy
broad workload RBAC.

The annotation must change when the image reference changes and cannot override
a known-incompatible official alpha or beta.1 through beta.8 reference that is
not digest-qualified. For `tag@digest`, Kubernetes pulls by digest; after
Expand Down
3 changes: 3 additions & 0 deletions deploy/rustfs-operator/crds/tenant-crd.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,9 @@ spec:
type: object
type: object
createServiceAccountRbac:
description: |-
Deprecated compatibility field. The operator never grants Kubernetes API permissions to
Tenant workloads. Configure a custom ServiceAccount and manage any required RBAC explicitly.
nullable: true
type: boolean
credsSecret:
Expand Down
3 changes: 3 additions & 0 deletions deploy/rustfs-operator/crds/tenant.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,9 @@ spec:
type: object
type: object
createServiceAccountRbac:
description: |-
Deprecated compatibility field. The operator never grants Kubernetes API permissions to
Tenant workloads. Configure a custom ServiceAccount and manage any required RBAC explicitly.
nullable: true
type: boolean
credsSecret:
Expand Down
3 changes: 2 additions & 1 deletion deploy/rustfs-operator/templates/clusterrole.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ rules:
resources: ["nodes"]
verbs: ["get", "list", "watch"]

# RBAC resources created for tenants
# Legacy Tenant workload RBAC watch/cleanup. Write verbs remain during the
# security migration so an older binary can coexist during a rolling upgrade.
- apiGroups: ["rbac.authorization.k8s.io"]
resources: ["roles", "rolebindings"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Expand Down
25 changes: 19 additions & 6 deletions docs/operator-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ RustFS Operator manages RustFS object storage clusters on Kubernetes. Users desc
The operator provides:

- `Tenant` CRD (`rustfs.com/v1alpha1`) for declaring RustFS pools, persistence, scheduling, credentials, TLS, logging, encryption, and bootstrap provisioning.
- Controller reconciliation for Tenant-owned RBAC, Services, StatefulSets, PVC templates, status conditions, and Kubernetes Events.
- Controller reconciliation for Tenant ServiceAccounts, Services, StatefulSets, PVC templates, status conditions, and Kubernetes Events.
- Helm chart under `deploy/rustfs-operator/` for production-style installation.
- Operator Console API and UI for management workflows.
- Optional operator STS endpoint for workload identity based temporary RustFS credentials.
Expand All @@ -50,7 +50,8 @@ A `Tenant` is one RustFS cluster. A Tenant can contain one or more pools, but al

When a Tenant is applied, the operator creates and owns:

- one ServiceAccount, Role, and RoleBinding when Tenant RBAC is enabled;
- one ServiceAccount when `serviceAccountName` is omitted; Tenant workloads receive no
operator-managed Kubernetes API permissions;
- one headless Service named `{tenant}-hl` for StatefulSet peer DNS;
- one S3 Service named `{tenant}-io` on port `9000`;
- one Tenant Console Service named `{tenant}-console` on port `9001`;
Expand Down Expand Up @@ -123,6 +124,17 @@ image-bound acknowledgement. Before upgrade, either pin a verified RustFS
beta.9-or-later release tag, or verify the effective image and set
`operator.rustfs.com/runtime-default-image-ack` to that exact image reference.

The Operator also removes legacy Tenant workload Roles and RoleBindings and
disables automatic Kubernetes API token mounting for its generated
ServiceAccounts. Existing default-ServiceAccount Tenants roll once to apply the
Pod template change. A custom image that calls the Kubernetes API must migrate
before upgrade: create a user-owned ServiceAccount with the required token
projection, bind a least-privilege Role under names other than the legacy
`{tenant}-role` and `{tenant}-role-binding`, and set `spec.serviceAccountName`.
Granting RBAC to the generated ServiceAccount is insufficient because its Pod
template disables token mounting. `createServiceAccountRbac` is retained only
as an ignored compatibility field.

The built-in RustFS image fallback also changes from the mutable `latest` tag to
`rustfs/rustfs:1.0.0-beta.10`. A Tenant that omits `spec.image` and has no
`TENANT_RUSTFS_IMAGE` Operator environment override will therefore roll to that
Expand All @@ -132,9 +144,10 @@ control future RustFS upgrades independently of the Operator default.
Treat this as a one-way workload security migration. After the new Operator has
reconciled a Tenant, do not downgrade directly to an Operator version that
predates these restricted defaults. An older controller omits the new seccomp
and container security fields: restricted admission rejects that update, while
a cluster without restricted admission can roll the workload back to weaker
settings. Recover by rolling forward to this version or a newer fixed version.
and container security fields and recreates the legacy broad workload RBAC:
restricted admission may reject the Pod update, while a cluster without it can
roll the workload back to weaker settings. Recover by rolling forward to this
version or a newer fixed version.

Uninstall:

Expand Down Expand Up @@ -413,7 +426,7 @@ Useful Tenant-level fields:
| `scheduler` | Custom scheduler name. |
| `env` | Additional RustFS container environment variables. Do not override operator-managed variables. |
| `serviceAccountName` | Custom ServiceAccount for RustFS pods. |
| `createServiceAccountRbac` | Whether the operator should create Role/RoleBinding for the Tenant ServiceAccount. |
| `createServiceAccountRbac` | Deprecated compatibility field; ignored. Manage any custom ServiceAccount RBAC explicitly. |
| `priorityClassName` | Tenant-level priority class. |
| `lifecycle` | Kubernetes container lifecycle hooks. |
| `podManagementPolicy` | StatefulSet pod management policy. |
Expand Down
22 changes: 16 additions & 6 deletions docs/operator-user-guide.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ RustFS Operator 用于在 Kubernetes 中管理 RustFS 对象存储集群。用
Operator 提供以下能力:

- `Tenant` CRD(`rustfs.com/v1alpha1`):声明 RustFS pool、持久化、调度、凭据、TLS、日志、加密和初始化 provisioning。
- 控制器 reconciliation:维护 Tenant 相关的 RBAC、Service、StatefulSet、PVC 模板、状态条件和 Kubernetes Event。
- 控制器 reconciliation:维护 Tenant ServiceAccount、Service、StatefulSet、PVC 模板、状态条件和 Kubernetes Event。
- Helm Chart:位于 `deploy/rustfs-operator/`,用于安装 Operator。
- Operator Console API 和 UI:用于 Operator 管理场景。
- 可选 Operator STS:基于 Kubernetes 工作负载身份签发临时 RustFS 凭据。
Expand All @@ -52,7 +52,8 @@ Operator 提供以下能力:

创建 Tenant 后,Operator 会创建并维护:

- Tenant RBAC 启用时的 ServiceAccount、Role、RoleBinding;
- 未配置 `serviceAccountName` 时创建一个 ServiceAccount;Operator 不会向 Tenant
workload 授予 Kubernetes API 权限;
- headless Service:`{tenant}-hl`,用于 StatefulSet peer DNS;
- S3 Service:`{tenant}-io`,端口 `9000`;
- Tenant Console Service:`{tenant}-console`,端口 `9001`;
Expand Down Expand Up @@ -121,16 +122,25 @@ StatefulSet template 如果尚未包含这些值,会在下一次 reconcile 时
升级前应固定到已验证的 RustFS beta.9 或更高 release tag;也可先验证实际生效的镜像,
再将 `operator.rustfs.com/runtime-default-image-ack` 设置为完全相同的镜像引用。

Operator 还会删除旧版本为 Tenant workload 创建的 Role 和 RoleBinding,并禁止其
自动挂载 Kubernetes API token。使用默认 ServiceAccount 的已有 Tenant 会因 Pod
template 变化发生一次滚动更新。若自定义镜像需要调用 Kubernetes API,必须在升级前
创建用户自管的 ServiceAccount 并配置所需 token projection,以非旧版
`{tenant}-role`、`{tenant}-role-binding` 的名称绑定最小权限 Role,再设置
`spec.serviceAccountName`。只给默认生成的 ServiceAccount 增加 RBAC 不足以恢复访问,
因为其 Pod template 会禁用 token 挂载。`createServiceAccountRbac` 仅作为已忽略的
兼容字段保留。

内置 RustFS 镜像 fallback 也会从可变的 `latest` 改为
`rustfs/rustfs:1.0.0-beta.10`。未设置 `spec.image`,且 Operator 没有配置
`TENANT_RUSTFS_IMAGE` 环境变量覆盖的 Tenant,会在 reconcile 时滚动到该固定版本。
若希望 RustFS 升级节奏独立于 Operator 默认值,应显式设置 `spec.image`。

应将此次变更视为单向 workload security migration。新版本 Operator 完成 Tenant
reconcile 后,不要直接降级到尚未提供这些 restricted 默认值的旧版本。旧 Controller
会省略新增的 seccomp 和容器安全字段:restricted 准入会拒绝该更新;未启用
restricted 准入的集群则可能把 workload 滚动回较弱配置。故障恢复应向前升级到当前
版本或更新的修复版本
会省略新增的 seccomp 和容器安全字段,并重新创建旧版宽权限 workload RBAC:
restricted 准入可能拒绝 Pod 更新;未启用 restricted 准入的集群则可能把 workload
滚动回较弱配置。故障恢复应向前升级到当前版本或更新的修复版本

卸载:

Expand Down Expand Up @@ -406,7 +416,7 @@ spec:
| `scheduler` | 自定义 scheduler 名称。 |
| `env` | 额外 RustFS 容器环境变量。不要覆盖 Operator 自动管理的变量。 |
| `serviceAccountName` | RustFS Pod 使用的自定义 ServiceAccount。 |
| `createServiceAccountRbac` | 是否由 Operator 为 Tenant ServiceAccount 创建 Role/RoleBinding。 |
| `createServiceAccountRbac` | 已废弃的兼容字段,不再生效;自定义 ServiceAccount 所需 RBAC 必须显式管理。 |
| `priorityClassName` | Tenant 级 PriorityClass。 |
| `lifecycle` | Kubernetes 容器 lifecycle hook。 |
| `podManagementPolicy` | StatefulSet pod management policy。 |
Expand Down
20 changes: 11 additions & 9 deletions examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,15 +332,14 @@ kubectl apply -f examples/spot-instance-tenant.yaml

**Features demonstrated:**
- Custom ServiceAccount usage (2 configurations)
- Manual RBAC management (Role + RoleBinding)
- Operator-managed RBAC with custom SA
- Resource-scoped, user-managed RBAC
- Cloud workload identity integration (AWS/GCP/Azure)
- Additional permissions beyond defaults
- Explicit permissions for custom images
- Security best practices

**Two configurations:**
1. Custom SA without operator RBAC (you manage everything)
2. Custom SA with operator RBAC (operator creates Role/RoleBinding)
1. Custom SA with a user-managed, resource-scoped Role
2. Custom SA for cloud workload identity without Kubernetes API RBAC

**Use case:**
- Existing RBAC policies
Expand All @@ -349,6 +348,10 @@ kubectl apply -f examples/spot-instance-tenant.yaml
- Additional permissions needed

**Deployment:**

Replace `registry.example.com/rustfs-custom:1.0.0` and its matching acknowledgement
annotation with a verified custom image before applying the complete file.

```bash
kubectl apply -f examples/custom-rbac-tenant.yaml
```
Expand Down Expand Up @@ -456,10 +459,9 @@ spec:

When you apply a Tenant, the operator creates:

1. **RBAC Resources** (conditional based on configuration)
- Role
- ServiceAccount
- RoleBinding
1. **ServiceAccount** (only when `serviceAccountName` is omitted)
- Kubernetes API token automount is disabled
- No workload Role or RoleBinding is created

2. **Services**
- IO Service: `rustfs` (S3 API, port **9000**)
Expand Down
Loading
Loading