Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 55 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,28 +16,37 @@ English version | [中文版](README_cn.md) | [日本語版](README_ja.md)

# HAMi

**Kubernetes GPU virtualization and heterogeneous accelerator scheduling for AI infrastructure.**
**Kubernetes GPU sharing and heterogeneous accelerator scheduling for AI infrastructure.**

![HAMi Architecture](imgs/hami-architecture.png)
HAMi (Heterogeneous AI Computing Virtualization Middleware) lets you share GPUs across Kubernetes workloads with memory and compute isolation, without changing application code. Formerly k8s-vGPU-scheduler.

[Docs](https://docs.project-hami.io) | [Quick Start](#quick-start) | [Discord](https://discord.gg/hami) | [Issues](https://github.com/project-hami/hami/issues) | [[Contributing](CONTRIBUTING.md)

HAMi stands for **Heterogeneous AI Computing Virtualization Middleware**. Formerly known as `k8s-vGPU-scheduler`, HAMi helps platform teams share expensive GPUs and other AI accelerators across Kubernetes workloads, isolate device memory and compute, and schedule pods with device-aware policies without changing application code.
![HAMi Architecture](imgs/hami-architecture.png)

HAMi is a [CNCF Sandbox](https://www.cncf.io/sandbox-projects/) and [CNCF Landscape](https://landscape.cncf.io/?item=orchestration-management--scheduling-orchestration--hami) project. It is also listed in the [CNAI Landscape](https://landscape.cncf.io/?group=cnai&item=cnai--general-orchestration--hami).

![CNCF logo](imgs/cncf-logo.png)

## Why HAMi?
## The Problem

AI infrastructure teams often run into the same Kubernetes accelerator problems: whole GPUs are allocated to small jobs, teams compete for scarce devices, different accelerator vendors expose different operational models, and schedulers lack enough device context to place workloads efficiently.
Running GPU workloads on Kubernetes creates resource contention:

HAMi provides a Kubernetes-native layer for:
- **Single workload per GPU**: A 40GB GPU running a 2GB workload uses 5% of capacity
- **No fair allocation**: First-come-first-served; no quota or priority mechanism
- **Vendor fragmentation**: NVIDIA, Ascend, Cambricon each require different integration code
- **Scheduler blindness**: Standard Kubernetes scheduler can't optimize for GPU topology or memory fragmentation

## What HAMi Does

HAMi solves this by enabling:

- **Device sharing**: allocate a fraction of a physical accelerator by memory, core, or device count.
- **Resource isolation**: enforce per-workload accelerator memory and compute limits where the device backend supports it.
- **Device-aware scheduling**: place pods with topology-aware, binpack, spread, and device-specific scheduling policies.
- **Heterogeneous AI clusters**: manage NVIDIA GPUs, NPUs, DCUs, MLUs, and other accelerator types through one scheduling and allocation workflow.
- **Zero application changes**: keep using standard Kubernetes resource requests and limits.
- **Production operations**: expose metrics, dashboards, WebUI, Helm installation, and community-supported deployment guidance.
- **No code changes**: keep using standard Kubernetes resource requests and limits.
- **Production operations**: metrics, dashboards, WebUI, Helm installation, community support.

## Use Cases

Expand All @@ -49,37 +58,39 @@ HAMi provides a Kubernetes-native layer for:

## How It Works

HAMi is composed of a mutating webhook, scheduler extender, device plugins, and device-specific in-container virtualization components.

HAMi intercepts pod creation and GPU allocation through these stages:
```text
Pod submission
-> HAMi mutating webhook
-> HAMi scheduler filter / score / bind
-> device allocation written to pod annotations
-> device plugin Allocate()
-> container runtime environment
-> HAMi monitor and metrics
-> Mutating webhook (inject device request)
-> Scheduler (select GPU and node)
-> Device plugin (allocate GPU memory/cores)
-> Container runtime (inject HAMi driver)
-> HAMi core (enforce isolation)
-> Workload (sees isolated GPU)
```

## Device Virtualization

HAMi lets workloads request only the accelerator resources they need. For example, the following pod asks for one physical NVIDIA GPU with 3 GiB of GPU memory:

Request a fraction of a GPU by memory or compute:
```yaml
resources:
limits:
nvidia.com/gpu: 1
nvidia.com/gpumem: 3000
nvidia.com/gpumem: 3000 # 3GB of 40GB
```

The workload sees the allocated device resources inside the container, while HAMi coordinates scheduling, allocation, and isolation.
Inside the container, CUDA sees exactly 3GB - not 40GB. HAMi enforces this at the driver level.

**Memory isolation**: Hard limit. A pod requesting 3GB cannot exceed 3GB, even if memory is available.

**No code changes**: Existing PyTorch, TensorFlow, CUDA code works as-is.

![HAMi Example](imgs/example.png)

> Notes:
>
> 1. After installing HAMi, the value of `nvidia.com/gpu` registered on the node defaults to the number of vGPUs.
> 2. When requesting resources in a pod, `nvidia.com/gpu` refers to the number of physical GPUs required by the current pod.
Notes:

- After installing HAMi, the value of `nvidia.com/gpu` registered on the node defaults to the number of vGPUs.
- When requesting resources in a pod, `nvidia.com/gpu` refers to the number of physical GPUs required by the current pod.

## Supported Devices

Expand All @@ -104,35 +115,46 @@ For the NVIDIA device plugin path, prepare:
### Install With Helm

Label GPU nodes so HAMi can manage them:

```bash
kubectl label nodes <node-name> gpu=on
```

Add the HAMi Helm repository:

```bash
helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update
```

Install HAMi:

```bash
helm install hami hami-charts/hami -n kube-system
```

Verify that the scheduler and device plugin are running:

```bash
kubectl get pods -n kube-system
kubectl get pods -n kube-system | grep hami
# Output should show:
# hami-device-plugin-xxxx Running
# hami-scheduler-xxxx Running
```

When `hami-device-plugin` and `hami-scheduler` are both `Running`, submit an example workload:
### Run Your First Shared Workload

When both pods are `Running`, submit an example workload:
```bash
kubectl apply -f examples/nvidia/default_use.yaml
```
Verify the workload is running and check GPU allocation:
```bash
kubectl get pods
kubectl describe pod <pod-name>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The example pod defined in examples/nvidia/default_use.yaml is explicitly named gpu-pod. Using this specific name instead of the <pod-name> placeholder makes the quick start guide more user-friendly and directly executable for users following the steps.

Suggested change
kubectl describe pod <pod-name>
kubectl describe pod gpu-pod

# Output should show:
# nvidia.com/gpu: 1
# nvidia.com/gpumem: 3000
```

Check what the workload sees inside the container:
```bash
kubectl exec <pod-name> -- nvidia-smi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using the explicit pod name gpu-pod from the provided example allows users to copy and paste the command directly without needing to manually substitute a placeholder.

Suggested change
kubectl exec <pod-name> -- nvidia-smi
kubectl exec gpu-pod -- nvidia-smi

# Output shows only 3GB of GPU memory (not the full 40GB)
```

For the complete installation guide and configuration options, see the [HAMi documentation](https://project-hami.io/docs/get-started/deploy-with-helm/).

Expand Down