Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 76 additions & 14 deletions docs/benchmark.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
## Benchmarks

Three instances from ai-benchmark have been used to evaluate vGPU-device-plugin performance as follows:
> **⚠️ Note**: The benchmark data below is from an older version of the project (when it was called vGPU-device-plugin) and uses outdated test environments. While the results are kept for historical reference, they may not reflect the current performance of HAMi.
>
> For up-to-date performance testing, please refer to the [Running Benchmarks](#running-benchmarks) section below.

| Test Environment | description |
### Historical Benchmark Results (Legacy)

Three instances from ai-benchmark were used to evaluate vGPU-device-plugin performance:

| Test Environment | Description |
| ---------------- | :------------------------------------------------------: |
| Kubernetes version | v1.12.9 |
| Docker version | 18.09.1 |
| Docker version | 18.09.1 |
| GPU Type | Tesla V100 |
| GPU Num | 2 |

| Test instance | description |
| Test Instance | Description |
| ------------- | :---------------------------------------------------------: |
| nvidia-device-plugin | k8s + nvidia k8s-device-plugin |
| vGPU-device-plugin | k8s + VGPU k8s-device-pluginwithout virtual device memory |
| vGPU-device-plugin(virtual device memory) | k8s + VGPU k8s-device-pluginwith virtual device memory |
| vGPU-device-plugin | k8s + vGPU k8s-device-plugin, without virtual device memory |
| vGPU-device-plugin (virtual device memory) | k8s + vGPU k8s-device-plugin, with virtual device memory |

Test Cases:

| test id | case | type | params |
| Test ID | Case | Type | Params |
| ------- | :-----------: | :-------: | :---------------------: |
| 1.1 | Resnet-V2-50 | inference | batch=50,size=346*346 |
| 1.2 | Resnet-V2-50 | training | batch=20,size=346*346 |
Expand All @@ -30,20 +36,76 @@ Test Cases:
| 5.1 | LSTM | inference | batch=100,size=1024*300 |
| 5.2 | LSTM | training | batch=10,size=1024*300 |

Test Result: ![img](../imgs/benchmark_inf.png)
Historical Test Results:

![img](../imgs/benchmark_inf.png)

![img](../imgs/benchmark_train.png)

To reproduce:
---

## Running Benchmarks

To benchmark HAMi performance in your environment, follow these steps:

### Prerequisites

- HAMi installed and configured in your Kubernetes cluster (see [Quick Start](../README.md#quick-start))
- GPU nodes labeled with `gpu=on`
- Kubernetes version >= 1.18
- Docker or containerd runtime with NVIDIA support

### Build Benchmark Image (Optional)

If you want to build the benchmark image yourself:

```bash
cd benchmarks/ai-benchmark
sh build.sh
```

### Run Benchmark Jobs

1. install k8s-vGPU-scheduler, and configure properly
2. run benchmark job
HAMi provides two benchmark job configurations to compare performance:

**1. Run benchmark on HAMi:**

```bash
cd benchmarks/deployments
kubectl apply -f job-on-hami.yml
```
$ kubectl apply -f benchmarks/ai-benchmark/ai-benchmark.yml

This will deploy a job that uses HAMi's GPU sharing and memory isolation features (requesting 50% of GPU memory).

**2. Run benchmark on NVIDIA device plugin (for comparison):**

```bash
kubectl apply -f job-on-nvidia-device-plugin.yml
```

3. View the result by using kubctl logs
For installing the official NVIDIA device plugin, refer to the [NVIDIA k8s-device-plugin Quick Start](https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#quick-start).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Yonsun-w I think it is better to replace For installing with To install


### View Results

After the jobs complete, view the benchmark results:

```bash
# Check job status
kubectl get jobs

# View HAMi benchmark results
kubectl logs job/ai-benchmark-on-hami

# View NVIDIA device plugin benchmark results
kubectl logs job/ai-benchmark-on-official
```
$ kubectl logs [pod id]

### Customizing Benchmarks

You can modify the benchmark jobs in `benchmarks/deployments/` to test different configurations:

- Adjust GPU memory allocation (e.g., `nvidia.com/gpumem-percentage: 50`)
- Test with different GPU counts
- Compare with and without HAMi's memory isolation features

For more details, see the [benchmarks README](../benchmarks/README.md).
91 changes: 76 additions & 15 deletions docs/benchmark_cn.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
## 性能测试

在测试报告中,我们一共在下面五种场景都执行了ai-benchmark 测试脚本,并汇总最终结果:
> **⚠️ 注意**:以下基准测试数据来自项目的早期版本(当时称为 vGPU-device-plugin),使用的测试环境已经过时。虽然这些结果被保留作为历史参考,但可能无法反映 HAMi 当前的性能表现。
>
> 如需进行最新的性能测试,请参考下方的[运行基准测试](#运行基准测试)部分。

### 历史基准测试结果(旧版)

在测试报告中,我们在以下场景中执行了 ai-benchmark 测试脚本,并汇总了最终结果:

| 测试环境 | 环境描述 |
| ---------------- | :------------------------------------------------------: |
| Kubernetes version | v1.12.9 |
| Docker version | 18.09.1 |
| Docker version | 18.09.1 |
| GPU Type | Tesla V100 |
| GPU Num | 2 |

| 测试名称 | 测试用例 |
| -------- | :------------------------------------------------: |
| Nvidia-device-plugin | k8s + nvidia官方k8s-device-plugin |
| vGPU-device-plugin | k8s + VGPU k8s-device-plugin,无虚拟显存 |
| vGPU-device-plugin(virtual device memory) | k8s + VGPU k8s-device-plugin,高负载,开启虚拟显存 |
| Nvidia-device-plugin | k8s + nvidia 官方 k8s-device-plugin |
| vGPU-device-plugin | k8s + vGPU k8s-device-plugin,无虚拟显存 |
| vGPU-device-plugin (virtual device memory) | k8s + vGPU k8s-device-plugin,高负载,开启虚拟显存 |

测试内容
测试内容

| test id | 名称 | 类型 | 参数 |
| Test ID | 名称 | 类型 | 参数 |
| ------- | :-----------: | :-------: | :---------------------: |
| 1.1 | Resnet-V2-50 | inference | batch=50,size=346*346 |
| 1.2 | Resnet-V2-50 | training | batch=20,size=346*346 |
Expand All @@ -30,21 +36,76 @@
| 5.1 | LSTM | inference | batch=100,size=1024*300 |
| 5.2 | LSTM | training | batch=10,size=1024*300 |

测试结果: ![img](../imgs/benchmark_inf.png)
历史测试结果:

![img](../imgs/benchmark_inf.png)

![img](../imgs/benchmark_train.png)

测试步骤:
---

## 运行基准测试

要在您的环境中测试 HAMi 的性能,请按照以下步骤操作:

### 前置条件

- HAMi 已安装并在 Kubernetes 集群中配置完成(参见[快速开始](../README_cn.md#快速开始))
- GPU 节点已标记 `gpu=on` 标签
- Kubernetes 版本 >= 1.18
- Docker 或 containerd 运行时,支持 NVIDIA

### 构建基准测试镜像(可选)

如果您想自己构建基准测试镜像:

```bash
cd benchmarks/ai-benchmark
sh build.sh
```

### 运行基准测试任务

1. 安装nvidia-device-plugin,并配置相应的参数
2. 运行benchmark任务
HAMi 提供了两个基准测试任务配置来比较性能:

**1. 在 HAMi 上运行基准测试:**

```bash
cd benchmarks/deployments
kubectl apply -f job-on-hami.yml
```
$ kubectl apply -f benchmarks/ai-benchmark/ai-benchmark.yml

这将部署一个使用 HAMi GPU 共享和显存隔离功能的任务(请求 50% 的 GPU 显存)。

**2. 在 NVIDIA device plugin 上运行基准测试(用于对比):**

```bash
kubectl apply -f job-on-nvidia-device-plugin.yml
```

3. 通过kubctl logs 查看结果
要安装官方 NVIDIA device plugin,请参考 [NVIDIA k8s-device-plugin 快速开始](https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#quick-start)。

### 查看结果

任务完成后,查看基准测试结果:

```bash
# 检查任务状态
kubectl get jobs

# 查看 HAMi 基准测试结果
kubectl logs job/ai-benchmark-on-hami

# 查看 NVIDIA device plugin 基准测试结果
kubectl logs job/ai-benchmark-on-official
```
$ kubectl logs [pod id]
```

### 自定义基准测试

您可以修改 `benchmarks/deployments/` 中的基准测试任务来测试不同的配置:

- 调整 GPU 显存分配(例如:`nvidia.com/gpumem-percentage: 50`)
- 测试不同的 GPU 数量
- 比较启用和不启用 HAMi 显存隔离功能的性能差异

更多详细信息,请参阅[基准测试 README](../benchmarks/README.md)。