A self-service platform for provisioning and managing Kubernetes clusters on Proxmox VE. Combines Packer, Terraform, Ansible, and a FastAPI REST API with a web dashboard.
┌─────────────────────────────────────────────────┐
│ Web Dashboard │
│ (Single-page HTML/JS) │
└──────────────────┬──────────────────────────────┘
│ HTTP
┌──────────────────▼──────────────────────────────┐
│ FastAPI REST API │
│ Auth · Clusters · Environments · Namespaces │
│ Users · Permissions · Jobs │
└──┬────────────────┬─────────────────┬───────────┘
│ │ │
▼ ▼ ▼
Terraform Ansible Paramiko/SSH
(VM lifecycle) (K8s setup) (kubectl)
│ │ │
└────────┬───────┘ │
▼ ▼
Proxmox VE API K8s Control Plane
| Tool | Version | Purpose |
|---|---|---|
| Python | 3.12+ | API server |
| Terraform | >= 1.0 | VM provisioning |
| Ansible | >= 2.15 | K8s cluster configuration |
| Packer | >= 1.9 | VM template creation |
| Proxmox VE | 8.x | Hypervisor |
cd packer
packer init ubuntu-server.pkr.hcl
packer build ubuntu-server.pkr.hclThis creates an Ubuntu 22.04 template (VM ID 990) on Proxmox with cloud-init support.
cd terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your Proxmox credentials and desired cluster sizeKey variables in terraform.tfvars:
proxmox_api_url = "https://<proxmox-host>:8006/api2/json"
proxmox_api_token_id = "user@pam!token-name"
proxmox_api_token_secret = "<secret>"
target_node = "pve1"
control_plane_count = 1
worker_count = 2conda create -n k8s-api python=3.12 -y
conda activate k8s-api
pip install fastapi uvicorn[standard] paramikoset -a && source .env && set +a && uvicorn api.main:app --host 0.0.0.0 --port 8000On first start, a default admin user is created:
| Username | admin |
| Password | admin |
The admin API key is also written to data/admin_key.txt for programmatic access.
Navigate to http://<host>:8000 and sign in.
├── api/ # FastAPI application
│ ├── main.py # App entrypoint, admin bootstrap
│ ├── auth.py # Auth: sessions, API keys, RBAC
│ ├── config.py # Paths, network defaults, tfvars parser
│ ├── database.py # SQLite schema & connection factory
│ ├── errors.py # APIError exception + handler
│ ├── models.py # Pydantic request/response models
│ ├── routers/
│ │ ├── auth.py # POST /auth/login, /auth/logout
│ │ ├── clusters.py # CRUD clusters + PATCH env assignment
│ │ ├── environments.py # CRUD environments
│ │ ├── namespaces.py # CRUD namespaces (kubectl over SSH)
│ │ └── users.py # User management + /users/me
│ ├── services/
│ │ ├── provisioner.py # Async Terraform + Ansible orchestration
│ │ └── kubectl.py # Remote kubectl via Paramiko SSH
│ └── static/
│ └── index.html # Single-page web dashboard
├── terraform/ # Infrastructure as Code
│ ├── main.tf # VM resources + inventory generation
│ ├── variables.tf # Input variables
│ ├── outputs.tf # VM IPs, names, IDs
│ ├── providers.tf # Proxmox provider config
│ ├── versions.tf # Required providers & versions
│ ├── terraform.tfvars # Live configuration (git-ignored)
│ └── terraform.tfvars.example
├── ansible/ # Configuration management
│ ├── site.yml # Main playbook (4 plays)
│ ├── ansible.cfg # Ansible settings
│ ├── inventory.ini # Auto-generated by Terraform
│ └── roles/
│ ├── common/ # All nodes: hostname, swap, containerd, kubeadm
│ ├── control_plane/ # kubeadm init, Flannel CNI, join token
│ └── worker/ # kubeadm join, node labeling
├── packer/ # VM template builder
│ ├── ubuntu-server.pkr.hcl # Proxmox ISO builder config
│ └── http/
│ ├── user-data # Cloud-init autoinstall
│ └── meta-data
├── data/ # Runtime data (git-ignored)
│ ├── api.db # SQLite database
│ └── admin_key.txt # Default admin API key
└── workspaces/ # Per-cluster Terraform workspaces (git-ignored)
Base URL: /api/v1
All endpoints require a Authorization: Bearer <token> header. Tokens can be:
- Session tokens (
ses_...) — obtained viaPOST /auth/login - API keys (
ak_...) — assigned to users at creation time
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/auth/login |
POST | None | Sign in with username + password |
/auth/logout |
POST | Any | End session |
Login request:
{ "username": "admin", "password": "admin" }| Endpoint | Method | Auth | Description |
|---|---|---|---|
/clusters |
POST | Admin | Create cluster (triggers async provisioning) |
/clusters |
GET | Viewer+ | List all clusters |
/clusters/{id} |
GET | Viewer+ | Get cluster detail |
/clusters/{id} |
PATCH | Admin | Update cluster (e.g. assign environment) |
/clusters/{id} |
DELETE | Admin | Delete cluster (triggers async destruction) |
Create request:
{ "name": "my-cluster", "node_count": 3 }node_count is split into 1 control plane + (n-1) workers.
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/environments |
POST | Admin | Create environment |
/environments |
GET | Viewer+ | List environments |
/environments/{id} |
GET | Viewer+ | Get environment detail |
/environments/{id} |
DELETE | Admin | Delete (must have no active clusters) |
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/clusters/{id}/namespaces |
POST | Team Lead+ | Create namespace (runs kubectl) |
/clusters/{id}/namespaces |
GET | Viewer+ | List namespaces |
/clusters/{id}/namespaces/{ns_id} |
DELETE | Team Lead+ | Delete namespace |
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/jobs/{id} |
GET | Viewer+ | Get job status and error details |
| Endpoint | Method | Auth | Description |
|---|---|---|---|
/users/me |
GET | Any | Current user info |
/users |
POST | Admin | Create user (returns API key once) |
/users |
GET | Admin | List users |
/users/{id} |
DELETE | Admin | Deactivate user |
/users/{id}/permissions |
GET | Admin | List user permissions |
/permissions |
POST | Admin | Grant permission |
/permissions/{id} |
DELETE | Admin | Revoke permission |
| Role | Level | Capabilities |
|---|---|---|
admin |
4 | Full access. Create/delete clusters, users, environments. |
team_lead |
3 | Manage namespaces, write access to assigned resources. |
developer |
2 | Read access to assigned resources. |
viewer |
1 | Read-only access to assigned resources. |
Admins bypass all resource-level permission checks. Other roles require explicit permission grants per resource.
When a cluster is created via the API:
- IP allocation — reserves a contiguous block of IPs from the pool (starting at
.201) - Workspace setup — creates
workspaces/<cluster_id>/terraform/with symlinked.tffiles and a per-clusterterraform.tfvars - Terraform apply — clones the VM template, configures static IPs via cloud-init
- Ansible playbook — runs against the auto-generated inventory:
commonrole: sets hostnames, disables swap, installs containerd + kubeadmcontrol_planerole:kubeadm init, installs Flannel CNI, generates join tokenworkerrole: joins workers to the cluster, labels them
- Status update — cluster status moves from
creating→running(orfailed)
All provisioning runs in background threads. The UI polls every 5 seconds while clusters are in transitional states.
| Component | Version |
|---|---|
| Kubernetes | 1.31.4 |
| Container runtime | containerd |
| CNI | Flannel (10.244.0.0/16) |
| Init tool | kubeadm |
| Setting | Value |
|---|---|
| VM IP range | 10.40.19.201 onwards |
| Gateway | 10.40.19.254 |
| DNS | 10.40.2.1 |
| Network bridge | vmbr0 |
| Pod CIDR | 10.244.0.0/16 |
Each cluster gets a contiguous block of IPs. The first IP is always the control plane.
SQLite at data/api.db. Tables:
clusters— cluster metadata, status, IP allocationenvironments— logical groupings for clustersjobs— async provisioning/destruction job trackingip_allocations— tracks which IP ranges are in usenamespaces— cluster namespace recordsusers— credentials (password hash + API key hash)sessions— active login sessionspermissions— per-resource access grants
conda activate k8s-api
set -a && source .env && set +a && uvicorn api.main:app --host 0.0.0.0 --port 8000 --reloadcd terraform
terraform init
terraform apply
cd ../ansible
ansible-playbook site.ymlcd packer
packer build -force ubuntu-server.pkr.hclrm data/api.db
# Restart the API server — it will recreate the DB and default admin| Problem | Solution |
|---|---|
Cluster stuck in creating |
Check job status via Jobs page. Look at API server logs for Terraform/Ansible errors. |
| SSH "Permission denied" to VMs | Default VM credentials are ubuntu/ubuntu. Ensure Terraform ssh_user/ssh_password match. |
| Stale SSH host keys after VM recreation | Run ssh-keygen -R 10.40.19.<ip> for affected IPs. |
| Namespace creation fails | Cluster must be in running status. API uses Paramiko SSH to the control plane — verify connectivity. |
| Apt lock errors during provisioning | The common role waits for apt locks and disables unattended-upgrades. If it persists, SSH into the VM and check fuser /var/lib/dpkg/lock-frontend. |
| Terraform clone timeout | Proxmox storage I/O may be slow. VMs may get stuck in lock: clone state — use the Proxmox UI to unlock/destroy them. |
| Port 8000 already in use | fuser -k 8000/tcp to free the port before restarting. |