diff --git a/docs/benchmark.md b/docs/benchmark.md index a288e0eb5a..623b30771b 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,23 +1,29 @@ ## Benchmarks -Three instances from ai-benchmark have been used to evaluate vGPU-device-plugin performance as follows: +> **⚠️ Note**: The benchmark data below is from an older version of the project (when it was called vGPU-device-plugin) and uses outdated test environments. While the results are kept for historical reference, they may not reflect the current performance of HAMi. +> +> For up-to-date performance testing, please refer to the [Running Benchmarks](#running-benchmarks) section below. -| Test Environment | description | +### Historical Benchmark Results (Legacy) + +Three instances from ai-benchmark were used to evaluate vGPU-device-plugin performance: + +| Test Environment | Description | | ---------------- | :------------------------------------------------------: | | Kubernetes version | v1.12.9 | -| Docker version | 18.09.1 | +| Docker version | 18.09.1 | | GPU Type | Tesla V100 | | GPU Num | 2 | -| Test instance | description | +| Test Instance | Description | | ------------- | :---------------------------------------------------------: | | nvidia-device-plugin | k8s + nvidia k8s-device-plugin | -| vGPU-device-plugin | k8s + VGPU k8s-device-plugin，without virtual device memory | -| vGPU-device-plugin(virtual device memory) | k8s + VGPU k8s-device-plugin，with virtual device memory | +| vGPU-device-plugin | k8s + vGPU k8s-device-plugin, without virtual device memory | +| vGPU-device-plugin (virtual device memory) | k8s + vGPU k8s-device-plugin, with virtual device memory | Test Cases: -| test id | case | type | params | +| Test ID | Case | Type | Params | | ------- | :-----------: | :-------: | :---------------------: | | 1.1 | Resnet-V2-50 | inference | batch=50,size=346*346 | | 1.2 | Resnet-V2-50 | training | batch=20,size=346*346 | @@ -30,20 +36,76 @@ Test Cases: | 5.1 | LSTM | inference | batch=100,size=1024*300 | | 5.2 | LSTM | training | batch=10,size=1024*300 | -Test Result: ![img](../imgs/benchmark_inf.png) +Historical Test Results: + +![img](../imgs/benchmark_inf.png) ![img](../imgs/benchmark_train.png) -To reproduce: +--- + +## Running Benchmarks + +To benchmark HAMi performance in your environment, follow these steps: + +### Prerequisites + +- HAMi installed and configured in your Kubernetes cluster (see [Quick Start](../README.md#quick-start)) +- GPU nodes labeled with `gpu=on` +- Kubernetes version >= 1.18 +- Docker or containerd runtime with NVIDIA support + +### Build Benchmark Image (Optional) + +If you want to build the benchmark image yourself: + +```bash +cd benchmarks/ai-benchmark +sh build.sh +``` + +### Run Benchmark Jobs -1. install k8s-vGPU-scheduler, and configure properly -2. run benchmark job +HAMi provides two benchmark job configurations to compare performance: +**1. Run benchmark on HAMi:** + +```bash +cd benchmarks/deployments +kubectl apply -f job-on-hami.yml ``` -$ kubectl apply -f benchmarks/ai-benchmark/ai-benchmark.yml + +This will deploy a job that uses HAMi's GPU sharing and memory isolation features (requesting 50% of GPU memory). + +**2. Run benchmark on NVIDIA device plugin (for comparison):** + +```bash +kubectl apply -f job-on-nvidia-device-plugin.yml ``` -3. View the result by using kubctl logs +For installing the official NVIDIA device plugin, refer to the [NVIDIA k8s-device-plugin Quick Start](https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#quick-start). + +### View Results + +After the jobs complete, view the benchmark results: +```bash +# Check job status +kubectl get jobs + +# View HAMi benchmark results +kubectl logs job/ai-benchmark-on-hami + +# View NVIDIA device plugin benchmark results +kubectl logs job/ai-benchmark-on-official ``` -$ kubectl logs [pod id] \ No newline at end of file + +### Customizing Benchmarks + +You can modify the benchmark jobs in `benchmarks/deployments/` to test different configurations: + +- Adjust GPU memory allocation (e.g., `nvidia.com/gpumem-percentage: 50`) +- Test with different GPU counts +- Compare with and without HAMi's memory isolation features + +For more details, see the [benchmarks README](../benchmarks/README.md). diff --git a/docs/benchmark_cn.md b/docs/benchmark_cn.md index c1f5f1fa81..08925c2a82 100644 --- a/docs/benchmark_cn.md +++ b/docs/benchmark_cn.md @@ -1,23 +1,29 @@ ## 性能测试 -在测试报告中，我们一共在下面五种场景都执行了ai-benchmark 测试脚本，并汇总最终结果： +> **⚠️ 注意**：以下基准测试数据来自项目的早期版本（当时称为 vGPU-device-plugin），使用的测试环境已经过时。虽然这些结果被保留作为历史参考，但可能无法反映 HAMi 当前的性能表现。 +> +> 如需进行最新的性能测试,请参考下方的[运行基准测试](#运行基准测试)部分。 + +### 历史基准测试结果（旧版） + +在测试报告中，我们在以下场景中执行了 ai-benchmark 测试脚本，并汇总了最终结果： | 测试环境 | 环境描述 | | ---------------- | :------------------------------------------------------: | | Kubernetes version | v1.12.9 | -| Docker version | 18.09.1 | +| Docker version | 18.09.1 | | GPU Type | Tesla V100 | | GPU Num | 2 | | 测试名称 | 测试用例 | | -------- | :------------------------------------------------: | -| Nvidia-device-plugin | k8s + nvidia官方k8s-device-plugin | -| vGPU-device-plugin | k8s + VGPU k8s-device-plugin，无虚拟显存 | -| vGPU-device-plugin(virtual device memory) | k8s + VGPU k8s-device-plugin，高负载，开启虚拟显存 | +| Nvidia-device-plugin | k8s + nvidia 官方 k8s-device-plugin | +| vGPU-device-plugin | k8s + vGPU k8s-device-plugin，无虚拟显存 | +| vGPU-device-plugin (virtual device memory) | k8s + vGPU k8s-device-plugin，高负载，开启虚拟显存 | -测试内容 +测试内容： -| test id | 名称 | 类型 | 参数 | +| Test ID | 名称 | 类型 | 参数 | | ------- | :-----------: | :-------: | :---------------------: | | 1.1 | Resnet-V2-50 | inference | batch=50,size=346*346 | | 1.2 | Resnet-V2-50 | training | batch=20,size=346*346 | @@ -30,21 +36,76 @@ | 5.1 | LSTM | inference | batch=100,size=1024*300 | | 5.2 | LSTM | training | batch=10,size=1024*300 | -测试结果： ![img](../imgs/benchmark_inf.png) +历史测试结果： + +![img](../imgs/benchmark_inf.png) ![img](../imgs/benchmark_train.png) -测试步骤： +--- + +## 运行基准测试 + +要在您的环境中测试 HAMi 的性能，请按照以下步骤操作： + +### 前置条件 + +- HAMi 已安装并在 Kubernetes 集群中配置完成（参见[快速开始](../README_cn.md#快速开始)） +- GPU 节点已标记 `gpu=on` 标签 +- Kubernetes 版本 >= 1.18 +- Docker 或 containerd 运行时，支持 NVIDIA + +### 构建基准测试镜像（可选） + +如果您想自己构建基准测试镜像： + +```bash +cd benchmarks/ai-benchmark +sh build.sh +``` + +### 运行基准测试任务 -1. 安装nvidia-device-plugin，并配置相应的参数 -2. 运行benchmark任务 +HAMi 提供了两个基准测试任务配置来比较性能： +**1. 在 HAMi 上运行基准测试：** + +```bash +cd benchmarks/deployments +kubectl apply -f job-on-hami.yml ``` -$ kubectl apply -f benchmarks/ai-benchmark/ai-benchmark.yml + +这将部署一个使用 HAMi GPU 共享和显存隔离功能的任务（请求 50% 的 GPU 显存）。 + +**2. 在 NVIDIA device plugin 上运行基准测试（用于对比）：** + +```bash +kubectl apply -f job-on-nvidia-device-plugin.yml ``` -3. 通过kubctl logs 查看结果 +要安装官方 NVIDIA device plugin，请参考 [NVIDIA k8s-device-plugin 快速开始](https://github.com/NVIDIA/k8s-device-plugin?tab=readme-ov-file#quick-start)。 + +### 查看结果 + +任务完成后，查看基准测试结果： +```bash +# 检查任务状态 +kubectl get jobs + +# 查看 HAMi 基准测试结果 +kubectl logs job/ai-benchmark-on-hami + +# 查看 NVIDIA device plugin 基准测试结果 +kubectl logs job/ai-benchmark-on-official ``` -$ kubectl logs [pod id] -``` \ No newline at end of file + +### 自定义基准测试 + +您可以修改 `benchmarks/deployments/` 中的基准测试任务来测试不同的配置： + +- 调整 GPU 显存分配（例如：`nvidia.com/gpumem-percentage: 50`） +- 测试不同的 GPU 数量 +- 比较启用和不启用 HAMi 显存隔离功能的性能差异 + +更多详细信息，请参阅[基准测试 README](../benchmarks/README.md)。