Skip to content
IsharaEkanayakaPublic

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Latest commit

 

History

15 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

K8s Cluster Manager

A self-service platform for provisioning and managing Kubernetes clusters on Proxmox VE. Combines Packer, Terraform, Ansible, and a FastAPI REST API with a web dashboard.

Architecture

┌─────────────────────────────────────────────────┐
│                  Web Dashboard                  │
│             (Single-page HTML/JS)               │
└──────────────────┬──────────────────────────────┘
                   │ HTTP
┌──────────────────▼──────────────────────────────┐
│              FastAPI REST API                    │
│  Auth · Clusters · Environments · Namespaces    │
│  Users · Permissions · Jobs                     │
└──┬────────────────┬─────────────────┬───────────┘
   │                │                 │
   ▼                ▼                 ▼
Terraform       Ansible           Paramiko/SSH
(VM lifecycle)  (K8s setup)       (kubectl)
   │                │                 │
   └────────┬───────┘                 │
            ▼                         ▼
     Proxmox VE API            K8s Control Plane

Prerequisites

Tool Version Purpose
Python 3.12+ API server
Terraform >= 1.0 VM provisioning
Ansible >= 2.15 K8s cluster configuration
Packer >= 1.9 VM template creation
Proxmox VE 8.x Hypervisor

Quick Start

1. Create the VM Template (one-time)

cd packer
packer init ubuntu-server.pkr.hcl
packer build ubuntu-server.pkr.hcl

This creates an Ubuntu 22.04 template (VM ID 990) on Proxmox with cloud-init support.

2. Configure Terraform

cd terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your Proxmox credentials and desired cluster size

Key variables in terraform.tfvars:

proxmox_api_url          = "https://<proxmox-host>:8006/api2/json"
proxmox_api_token_id     = "user@pam!token-name"
proxmox_api_token_secret = "<secret>"
target_node              = "pve1"
control_plane_count      = 1
worker_count             = 2

3. Install Python Dependencies

conda create -n k8s-api python=3.12 -y
conda activate k8s-api
pip install fastapi uvicorn[standard] paramiko

4. Start the API Server

set -a && source .env && set +a && uvicorn api.main:app --host 0.0.0.0 --port 8000

On first start, a default admin user is created:

Username admin
Password admin

The admin API key is also written to data/admin_key.txt for programmatic access.

5. Open the Dashboard

Navigate to http://<host>:8000 and sign in.

Project Structure

├── api/                        # FastAPI application
│   ├── main.py                 # App entrypoint, admin bootstrap
│   ├── auth.py                 # Auth: sessions, API keys, RBAC
│   ├── config.py               # Paths, network defaults, tfvars parser
│   ├── database.py             # SQLite schema & connection factory
│   ├── errors.py               # APIError exception + handler
│   ├── models.py               # Pydantic request/response models
│   ├── routers/
│   │   ├── auth.py             # POST /auth/login, /auth/logout
│   │   ├── clusters.py         # CRUD clusters + PATCH env assignment
│   │   ├── environments.py     # CRUD environments
│   │   ├── namespaces.py       # CRUD namespaces (kubectl over SSH)
│   │   └── users.py            # User management + /users/me
│   ├── services/
│   │   ├── provisioner.py      # Async Terraform + Ansible orchestration
│   │   └── kubectl.py          # Remote kubectl via Paramiko SSH
│   └── static/
│       └── index.html          # Single-page web dashboard
├── terraform/                  # Infrastructure as Code
│   ├── main.tf                 # VM resources + inventory generation
│   ├── variables.tf            # Input variables
│   ├── outputs.tf              # VM IPs, names, IDs
│   ├── providers.tf            # Proxmox provider config
│   ├── versions.tf             # Required providers & versions
│   ├── terraform.tfvars        # Live configuration (git-ignored)
│   └── terraform.tfvars.example
├── ansible/                    # Configuration management
│   ├── site.yml                # Main playbook (4 plays)
│   ├── ansible.cfg             # Ansible settings
│   ├── inventory.ini           # Auto-generated by Terraform
│   └── roles/
│       ├── common/             # All nodes: hostname, swap, containerd, kubeadm
│       ├── control_plane/      # kubeadm init, Flannel CNI, join token
│       └── worker/             # kubeadm join, node labeling
├── packer/                     # VM template builder
│   ├── ubuntu-server.pkr.hcl   # Proxmox ISO builder config
│   └── http/
│       ├── user-data           # Cloud-init autoinstall
│       └── meta-data
├── data/                       # Runtime data (git-ignored)
│   ├── api.db                  # SQLite database
│   └── admin_key.txt           # Default admin API key
└── workspaces/                 # Per-cluster Terraform workspaces (git-ignored)

API Reference

Base URL: /api/v1

Authentication

All endpoints require a Authorization: Bearer <token> header. Tokens can be:

  • Session tokens (ses_...) — obtained via POST /auth/login
  • API keys (ak_...) — assigned to users at creation time
Endpoint Method Auth Description
/auth/login POST None Sign in with username + password
/auth/logout POST Any End session

Login request:

{ "username": "admin", "password": "admin" }

Clusters

Endpoint Method Auth Description
/clusters POST Admin Create cluster (triggers async provisioning)
/clusters GET Viewer+ List all clusters
/clusters/{id} GET Viewer+ Get cluster detail
/clusters/{id} PATCH Admin Update cluster (e.g. assign environment)
/clusters/{id} DELETE Admin Delete cluster (triggers async destruction)

Create request:

{ "name": "my-cluster", "node_count": 3 }

node_count is split into 1 control plane + (n-1) workers.

Environments

Endpoint Method Auth Description
/environments POST Admin Create environment
/environments GET Viewer+ List environments
/environments/{id} GET Viewer+ Get environment detail
/environments/{id} DELETE Admin Delete (must have no active clusters)

Namespaces

Endpoint Method Auth Description
/clusters/{id}/namespaces POST Team Lead+ Create namespace (runs kubectl)
/clusters/{id}/namespaces GET Viewer+ List namespaces
/clusters/{id}/namespaces/{ns_id} DELETE Team Lead+ Delete namespace

Jobs

Endpoint Method Auth Description
/jobs/{id} GET Viewer+ Get job status and error details

Users & Permissions

Endpoint Method Auth Description
/users/me GET Any Current user info
/users POST Admin Create user (returns API key once)
/users GET Admin List users
/users/{id} DELETE Admin Deactivate user
/users/{id}/permissions GET Admin List user permissions
/permissions POST Admin Grant permission
/permissions/{id} DELETE Admin Revoke permission

RBAC Roles

Role Level Capabilities
admin 4 Full access. Create/delete clusters, users, environments.
team_lead 3 Manage namespaces, write access to assigned resources.
developer 2 Read access to assigned resources.
viewer 1 Read-only access to assigned resources.

Admins bypass all resource-level permission checks. Other roles require explicit permission grants per resource.

Provisioning Flow

When a cluster is created via the API:

  1. IP allocation — reserves a contiguous block of IPs from the pool (starting at .201)
  2. Workspace setup — creates workspaces/<cluster_id>/terraform/ with symlinked .tf files and a per-cluster terraform.tfvars
  3. Terraform apply — clones the VM template, configures static IPs via cloud-init
  4. Ansible playbook — runs against the auto-generated inventory:
    • common role: sets hostnames, disables swap, installs containerd + kubeadm
    • control_plane role: kubeadm init, installs Flannel CNI, generates join token
    • worker role: joins workers to the cluster, labels them
  5. Status update — cluster status moves from creating → running (or failed)

All provisioning runs in background threads. The UI polls every 5 seconds while clusters are in transitional states.

Kubernetes Details

Component Version
Kubernetes 1.31.4
Container runtime containerd
CNI Flannel (10.244.0.0/16)
Init tool kubeadm

Network Configuration

Setting Value
VM IP range 10.40.19.201 onwards
Gateway 10.40.19.254
DNS 10.40.2.1
Network bridge vmbr0
Pod CIDR 10.244.0.0/16

Each cluster gets a contiguous block of IPs. The first IP is always the control plane.

Database

SQLite at data/api.db. Tables:

  • clusters — cluster metadata, status, IP allocation
  • environments — logical groupings for clusters
  • jobs — async provisioning/destruction job tracking
  • ip_allocations — tracks which IP ranges are in use
  • namespaces — cluster namespace records
  • users — credentials (password hash + API key hash)
  • sessions — active login sessions
  • permissions — per-resource access grants

Development

Running in Development

conda activate k8s-api
set -a && source .env && set +a && uvicorn api.main:app --host 0.0.0.0 --port 8000 --reload

Manual Terraform/Ansible (standalone)

cd terraform
terraform init
terraform apply

cd ../ansible
ansible-playbook site.yml

Rebuilding the VM Template

cd packer
packer build -force ubuntu-server.pkr.hcl

Resetting the Database

rm data/api.db
# Restart the API server — it will recreate the DB and default admin

Troubleshooting

Problem Solution
Cluster stuck in creating Check job status via Jobs page. Look at API server logs for Terraform/Ansible errors.
SSH "Permission denied" to VMs Default VM credentials are ubuntu/ubuntu. Ensure Terraform ssh_user/ssh_password match.
Stale SSH host keys after VM recreation Run ssh-keygen -R 10.40.19.<ip> for affected IPs.
Namespace creation fails Cluster must be in running status. API uses Paramiko SSH to the control plane — verify connectivity.
Apt lock errors during provisioning The common role waits for apt locks and disables unattended-upgrades. If it persists, SSH into the VM and check fuser /var/lib/dpkg/lock-frontend.
Terraform clone timeout Proxmox storage I/O may be slow. VMs may get stuck in lock: clone state — use the Proxmox UI to unlock/destroy them.
Port 8000 already in use fuser -k 8000/tcp to free the port before restarting.

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages