HPC + Kubernetes Course: SLURM ↔ k3s Interlink Bridge

Objective: Learn how to bridge HPC (SLURM) and Kubernetes (k3s) using Interlink.

This repository contains step-by-step instructions to deploy a complete Interlink setup bridging two systems:

1. Machine 122 (SLURM): 192.168.2.122 - HPC backend with SLURM + Interlink
2. Machine 78 (k3s): 192.168.2.78 - Kubernetes cluster with VirtualKubelet

Kubernetes pods scheduled to the virtual node will be automatically offloaded to SLURM jobs on the HPC backend.

Documentation Structure

hpc-course/
├── README.md                              # This file (start here)
├── docs/
│   ├── PHASES/                           # Phase-by-phase learning path
│   │   ├── phase1-slurm-setup.md         # Detailed SLURM installation
│   │   ├── phase2-k3s-setup.md           # Detailed k3s installation
│   │   ├── phase3-interlink-setup.md     # Detailed Interlink setup
│   │   └── phase4-test-offload.md        # Detailed testing procedures

Architecture Overview

k3s (Machine 2)                 Interlink Bridge              SLURM (Machine 1)
───────────────────             ────────────────              ──────────────────
Pod submitted
   │
   ▼
Virtual Node "interlink-node"
   │ (watches for pods)
   │
   ├─► VirtualKubelet Pod (Helm)
   │       │
   │       │ REST/HTTP
   │       │
   │       └──────────────────────────────► Interlink API (port 3000)
   │                                              │
   │                                              │ REST/HTTP
   │                                              │
   │                                              ▼
   │                                        SLURM Plugin
   │                                              │
   │                                              │ sbatch/squeue
   │                                              │
   │                                              ▼
   │                                        SLURM Daemon
   │                                              │
   │ ◄──────────────────────────────────────────┘ (status)
   │
Pod completes on SLURM backend

What Gets Deployed

Component	Machine	Type	Port	Role
VirtualKubelet	2	Helm Pod	-	Watches k8s pods, translates to Interlink
Interlink API	1	Binary	3000	Translates pods to SLURM jobs
SLURM Plugin	1	Binary	4000	Submits jobs to SLURM
SLURM	1	Service	-	Schedules and executes jobs
k3s	2	Service	6443	Kubernetes control plane

Network Requirements

• Connectivity: Machine 78 must reach Machine 122 on port 3000 (Interlink API)
• SSH Keys: Key-based SSH between machines (for remote commands)
• Subnet: 192.168.2.0/24

Test connectivity before starting:

ping 192.168.2.122          # From Machine 78
ping 192.168.2.78           # From Machine 122
curl http://192.168.2.122:3000/  # From Machine 78 (after setup)

Current Status: ✅ All connectivity verified

Expected Workflow

1. Submit pod to k3s pointing to interlink-node
2. Pod status changes to Running
3. VirtualKubelet detects pod, sends spec to Interlink API
4. Interlink translates to SLURM job and submits via sbatch
5. Job appears in SLURM queue (squeue)
6. Pod status reflects job execution
7. Job completes, pod marked as Completed
8. Logs available via kubectl logs

Key Concepts

VirtualKubelet

• Kubernetes node implementation that schedules pods to external systems
• Watches k3s API for pods on "interlink-node"
• Translates pod specs and submits to Interlink

Interlink API

• Bridge service translating Kubernetes concepts to HPC concepts
• Pod spec → SLURM job script
• Pod status → SLURM job status
• Uses REST/HTTP for communication between components

SLURM Plugin

• Submits jobs to SLURM via sbatch
• Monitors job status via squeue
• Reports back to Interlink API

Verify Services are Running

Machine 122 - SLURM:

ssh rocky@192.168.2.122
/home/rocky/slurm-demo/bin/sinfo
# Expected: demo partition with localhost in idle state

Machine 122 - Interlink:

ssh rocky@192.168.2.122
ps aux | grep -E 'interlink-api|slurm-plugin'
curl -I http://localhost:3000/
# Expected: HTTP/1.1 404 (API is responding)

Machine 78 - VirtualKubelet:

ssh rocky@192.168.2.78
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl get pods -n virtual-kubelet
# Expected: virtual-kubelet pod in Running state

Pod stuck in Pending

1. Verify interlink-node exists: kubectl get nodes
2. Check VirtualKubelet pod: kubectl get pods -n virtual-kubelet
3. Check logs: kubectl logs -n virtual-kubelet -l app=virtual-kubelet

Interlink API not responding

1. Check if running: ssh rocky@192.168.2.122 'ps aux | grep interlink-api'
2. Check port: ssh rocky@192.168.2.122 'netstat -tlnp | grep 3000'
3. Check logs: ssh rocky@192.168.2.122 'tail -50 ~/interlink/interlink-api.log'

Network issues between machines

# From Machine 78
ssh rocky@192.168.2.78 "ping 192.168.2.122"
ssh rocky@192.168.2.78 "curl -I http://192.168.2.122:3000/"

For complete troubleshooting, see docs/FINAL-SETUP-GUIDE.md

Learning Objectives

By completing this course, you'll understand:

✓ How HPC systems (SLURM) differ from container orchestration (Kubernetes)
✓ How to implement a translation layer (Interlink) between systems
✓ How pod abstractions map to job abstractions
✓ How to debug cross-system workflows
✓ How to monitor and verify Kubernetes workloads on HPC infrastructure
✓ Real-world patterns for hybrid HPC+cloud deployments

References

• Interlink: https://github.com/interlink-hq/interlink
• SLURM: https://slurm.schedmd.com/
• k3s: https://docs.k3s.io/
• Kubernetes: https://kubernetes.io/docs/
• Virtual Kubelet: https://github.com/virtual-kubelet/virtual-kubelet