chore(gpu): bump aks-gpu-grid to 570.237-20260817204535 - #9229

Open

Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into

mainfrom

ganesh/gpu-grid-570.237

Azure Pipelines / Agentbaker GPU E2E failed Aug 17, 2026 in 1h 3m 12s

Build #20260817.13 had test failures

1 errors / 5 warnings

Details

Failed: 6 (10.91%)
Passed: 49 (89.09%)
Other: 0 (0.00%)
Total: 55

Annotations

Check failure on line 2702 in Build log

azure-pipelines / Agentbaker GPU E2E

Build log #L2702

Script failed with exit code: 1

Check failure on line 1 in Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU

azure-pipelines / Agentbaker GPU E2E

Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU

Failed

Raw output


            === RUN   Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU
=== PAUSE Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU
=== CONT  Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU
    test_helpers.go:412: [8.446s] TAGS {Name:Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU ImageName:2404gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false Kata:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false RCV1PCertMode:false VMSeriesCoverageTest:false}
    test_helpers.go:222: [8.447s] → running scenario...
    cluster.go:79: [8.447s] → preparing cluster...
    cluster.go:319: [14.224s] → get or create cluster abe2e-kubenet-v5-6f4ab...
    cluster.go:327: [17.150s] ✓ get or create cluster abe2e-kubenet-v5-6f4ab done (2.9s)
    cluster.go:725: [17.213s] using shared bastion abe2e-shared-bastion in abe2e-westus3
    aks_model.go:316: [17.213s] → adding firewall rules...
    aks_model.go:362: [17.774s] Adding route "vnet-local" to AKS route table "aks-agentpool-19541795-routetable"
    cluster.go:894: [18.132s] → setting up private DNS for API server...
    cluster.go:933: [18.641s] private DNS zone "abe2e-kubenet-v5-wj74ke1x.hcp.westus3.azmk8s.io" already up to date
    cluster.go:934: [18.641s] ✓ setting up private DNS for API server done (0.5s)
    aks_model.go:362: [21.312s] Adding route "default-route-to-firewall" to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:373: [21.763s] Successfully added firewall routes to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:374: [21.763s] ✓ adding firewall rules done (4.5s)
    cluster.go:749: [21.763s] → collecting garbage VMSS...
    cluster.go:821: [21.866s] → collecting garbage K8s nodes...
    cluster.go:862: [21.953s] ✓ collecting garbage K8s nodes done (0.1s)
    cluster.go:813: [21.953s] ✓ collecting garbage VMSS done (0.2s)
    kube.go:397: [21.954s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:397: [22.155s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:574: [22.230s] Creating proxy daemonset e2e-proxy with image mcr.microsoft.com/cbl-mariner/base/python:3
    cluster.go:202: [22.443s] ✓ preparing cluster done (14.0s)
    test_helpers.go:239: [22.443s] using cluster abe2e-kubenet-v5-6f4ab in rg=abe2e-westus3 sub=8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8
    test_helpers.go:240: [22.443s] portal: https://portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v5-6f4ab/overview
    test_helpers.go:272: [22.445s] → preparing AKS node...
    vmss.go:369: [32.666s] → creating VMSS yxkq-2026-08-17-ubuntu2404nvidiadevicepluginrunningmigmul...
    vmss.go:263: [33.829s] VMSS portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v5-6f4ab_westus3/providers/Microsoft.Compute/virtualMachineScaleSets/yxkq-2026-08-17-ubuntu2404nvidiadevicepluginrunningmigmul/overview
    vmss.go:269: [33.829s] Managed cluster portal link: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/MC_abe2e-westus3_abe2e-kubenet-v5-6f4ab_westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v5-6f4ab/overview
2026/08/17 21:31:58 Using VM extension version 1.492 for extension type Compute.AKS.Linux.AKSNode in region westus3
    vmss.go:401: [53.972s] VM will be automatically deleted after the test
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

Check failure on line 1 in Test_Ubuntu2404_GPU_H100

azure-pipelines / Agentbaker GPU E2E

Test_Ubuntu2404_GPU_H100

Failed

Raw output


            === RUN   Test_Ubuntu2404_GPU_H100
=== PAUSE Test_Ubuntu2404_GPU_H100
=== CONT  Test_Ubuntu2404_GPU_H100
    azure.go:573: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/overview
    azure.go:662: [7.108s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2404gen2containerd/versions/1.1786954227.14622 is already in region uaenorth
    vhd.go:361: [7.108s] got version by tag branch=refs/heads/main: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-24.04-gen2/versions/1.1786954227.14622/overview
    test_helpers.go:412: [7.108s] TAGS {Name:Test_Ubuntu2404_GPU_H100 ImageName:2404gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false Kata:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false RCV1PCertMode:false VMSeriesCoverageTest:false}
    test_helpers.go:222: [16.736s] → running scenario...
    cluster.go:79: [16.736s] → preparing cluster...
    shared_infra.go:51: [16.736s] → ensuring shared infrastructure...
    shared_infra.go:80: [20.975s] ✓ ensuring shared infrastructure done (4.2s)
    cluster.go:319: [21.262s] → get or create cluster abe2e-kubenet-v5-8d077...
    cluster.go:327: [23.148s] ✓ get or create cluster abe2e-kubenet-v5-8d077 done (1.9s)
    cluster.go:725: [23.442s] using shared bastion abe2e-shared-bastion in abe2e-uaenorth
    aks_model.go:316: [23.442s] → adding firewall rules...
    aks_model.go:362: [23.984s] Adding route "vnet-local" to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:362: [24.958s] Adding route "default-route-to-firewall" to AKS route table "aks-agentpool-19541795-routetable"
    cluster.go:894: [25.786s] → setting up private DNS for API server...
    cluster.go:933: [26.260s] private DNS zone "abe2e-kubenet-v5-9oh1pilx.hcp.uaenorth.azmk8s.io" already up to date
    cluster.go:934: [26.260s] ✓ setting up private DNS for API server done (0.5s)
    aks_model.go:373: [27.202s] Successfully added firewall routes to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:374: [27.202s] ✓ adding firewall rules done (3.8s)
    cluster.go:749: [27.202s] → collecting garbage VMSS...
    cluster.go:821: [27.938s] → collecting garbage K8s nodes...
    cluster.go:862: [28.835s] ✓ collecting garbage K8s nodes done (0.9s)
    cluster.go:813: [28.835s] ✓ collecting garbage VMSS done (1.6s)
    kube.go:397: [28.835s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:397: [30.377s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:574: [31.135s] Creating proxy daemonset e2e-proxy with image mcr.microsoft.com/cbl-mariner/base/python:3
    cluster.go:202: [32.821s] ✓ preparing cluster done (16.1s)
    test_helpers.go:239: [32.821s] using cluster abe2e-kubenet-v5-8d077 in rg=abe2e-uaenorth sub=8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8
    test_helpers.go:240: [32.821s] portal: https://portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/abe2e-uaenorth/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v5-8d077/overview
    test_helpers.go:272: [32.826s] → preparing A
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

Check failure on line 1 in Test_Ubuntu2204_GPUA10

azure-pipelines / Agentbaker GPU E2E

Test_Ubuntu2204_GPUA10

Failed

Raw output


            === RUN   Test_Ubuntu2204_GPUA10
=== PAUSE Test_Ubuntu2204_GPUA10
=== CONT  Test_Ubuntu2204_GPUA10
    azure.go:573: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-22.04-gen2/overview
    azure.go:662: [6.565s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1786954193.18194 is already in region westus3
    vhd.go:361: [6.565s] got version by tag branch=refs/heads/main: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-22.04-gen2/versions/1.1786954193.18194/overview
    test_helpers.go:412: [6.565s] TAGS {Name:Test_Ubuntu2204_GPUA10 ImageName:2204gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false Kata:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false RCV1PCertMode:false VMSeriesCoverageTest:false}
    test_helpers.go:222: [6.565s] → running scenario...
    cluster.go:79: [6.565s] → preparing cluster...
    shared_infra.go:51: [6.565s] → ensuring shared infrastructure...
    shared_infra.go:80: [14.027s] ✓ ensuring shared infrastructure done (7.5s)
    cluster.go:319: [14.240s] → get or create cluster abe2e-kubenet-v5-150ee...
    cluster.go:327: [17.181s] ✓ get or create cluster abe2e-kubenet-v5-150ee done (2.9s)
    cluster.go:725: [17.243s] using shared bastion abe2e-shared-bastion in abe2e-westus3
    aks_model.go:316: [17.243s] → adding firewall rules...
    cluster.go:894: [17.614s] → setting up private DNS for API server...
    aks_model.go:362: [17.768s] Adding route "vnet-local" to AKS route table "aks-agentpool-19541795-routetable"
    cluster.go:933: [18.673s] private DNS zone "abe2e-kubenet-v5-doodzgxi.hcp.westus3.azmk8s.io" already up to date
    cluster.go:934: [18.673s] ✓ setting up private DNS for API server done (1.1s)
    aks_model.go:362: [20.242s] Adding route "default-route-to-firewall" to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:373: [20.625s] Successfully added firewall routes to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:374: [20.625s] ✓ adding firewall rules done (3.4s)
    cluster.go:749: [20.625s] → collecting garbage VMSS...
    cluster.go:821: [20.742s] → collecting garbage K8s nodes...
    cluster.go:862: [20.859s] ✓ collecting garbage K8s nodes done (0.1s)
    cluster.go:813: [20.859s] ✓ collecting garbage VMSS done (0.2s)
    kube.go:397: [20.859s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:397: [21.008s] Creating daemonset debugnonhost-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base/core:2.0
    kube.go:574: [21.077s] Creating proxy daemonset e2e-proxy with image mcr.microsoft.com/cbl-mariner/base/python:3
    cluster.go:202: [21.232s] ✓ preparing cluster done (14.7s)
    test_helpers.go:239: [21.232s] using cluster abe2e-kubenet-v5-150ee in rg=abe2e-westus3 sub=8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8
    test_helpers.go:240: [21.232s] portal: https://portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/resourceGroups/abe2e-westus3/providers/Microsoft.ContainerService/managedClusters/abe2e-kubenet-v5-150ee/overview
    test_helpers.go:272: [21.236s] → preparing AKS node...
    v
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

Check failure on line 1 in Test_Ubuntu2204_GPUA10

azure-pipelines / Agentbaker GPU E2E

Test_Ubuntu2204_GPUA10

Failed

Raw output


            === RUN   Test_Ubuntu2204_GPUA10
=== PAUSE Test_Ubuntu2204_GPUA10
=== CONT  Test_Ubuntu2204_GPUA10
    azure.go:573: [0.000s] Looking up images in https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-22.04-gen2/overview
    azure.go:662: [2.125s] Image version /subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/2204gen2containerd/versions/1.1786954193.18194 is already in region westus3
    vhd.go:361: [2.125s] got version by tag branch=refs/heads/main: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/c4c3550e-a965-4993-a50c-628fd38cd3e1/resourceGroups/aksvhdtestbuildrg/providers/Microsoft.Compute/galleries/PackerSigGalleryEastUS/images/aks-ubuntu-containerd-22.04-gen2/versions/1.1786954193.18194/overview
    test_helpers.go:412: [2.125s] TAGS {Name:Test_Ubuntu2204_GPUA10 ImageName:2204gen2containerd OS:ubuntu Arch:amd64 NetworkIsolated:false NonAnonymousACR:false GPU:true WASM:false Kata:false BootstrapTokenFallback:false KubeletCustomConfig:false Scriptless:false VHDCaching:false MockAzureChinaCloud:false RCV1PCertMode:false VMSeriesCoverageTest:false}
    test_helpers.go:222: [10.475s] → running scenario...
    cluster.go:79: [10.475s] → preparing cluster...
    shared_infra.go:51: [10.475s] → ensuring shared infrastructure...
    shared_infra.go:80: [13.719s] ✓ ensuring shared infrastructure done (3.2s)
    cluster.go:319: [13.907s] → get or create cluster abe2e-kubenet-v5-150ee...
    cluster.go:327: [15.434s] ✓ get or create cluster abe2e-kubenet-v5-150ee done (1.5s)
    cluster.go:725: [15.493s] using shared bastion abe2e-shared-bastion in abe2e-westus3
    aks_model.go:316: [15.493s] → adding firewall rules...
    cluster.go:894: [15.608s] → setting up private DNS for API server...
    aks_model.go:362: [15.811s] Adding route "vnet-local" to AKS route table "aks-agentpool-19541795-routetable"
    cluster.go:933: [16.771s] private DNS zone "abe2e-kubenet-v5-doodzgxi.hcp.westus3.azmk8s.io" already up to date
    cluster.go:934: [16.771s] ✓ setting up private DNS for API server done (1.2s)
    aks_model.go:362: [17.559s] Adding route "default-route-to-firewall" to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:373: [17.921s] Successfully added firewall routes to AKS route table "aks-agentpool-19541795-routetable"
    aks_model.go:374: [17.921s] ✓ adding firewall rules done (2.4s)
    cluster.go:749: [17.921s] → collecting garbage VMSS...
    cluster.go:821: [18.018s] → collecting garbage K8s nodes...
    cluster.go:855: [18.175s] deleted stale K8s node "0hnv-2026-08-17-ubuntu2204artifactstreamingarm64000000" (VMSS "0hnv-2026-08-17-ubuntu2204artifactstreamingarm64" not found)
    cluster.go:855: [18.244s] deleted stale K8s node "4mug-2026-08-17-ubuntu2204arm64000000" (VMSS "4mug-2026-08-17-ubuntu2204arm64" not found)
    cluster.go:855: [18.271s] deleted stale K8s node "dq79-2026-08-17-ubuntu2204arm64kubeletcustomconfig000000" (VMSS "dq79-2026-08-17-ubuntu2204arm64kubeletcustomconfig" not found)
    cluster.go:855: [18.299s] deleted stale K8s node "rcmc-2026-08-17-aclgen2fipstl000000" (VMSS "rcmc-2026-08-17-aclgen2fipstl" not found)
    cluster.go:855: [18.328s] deleted stale K8s node "xngk-2026-08-17-ubuntu2404gen1000000" (VMSS "xngk-2026-08-17-ubuntu2404gen1" not found)
    cluster.go:862: [18.328s] ✓ collecting garbage K8s nodes done (0.3s)
    cluster.go:813: [18.328s] ✓ collecting garbage VMSS done (0.4s)
    kube.go:397: [18.328s] Creating daemonset debug-mariner-tolerated with image mcr.microsoft.com/cbl-mariner/base
... [The stack trace has been truncated as it exceeded the maximum allowed size. Please refer to the complete log available in the Test Run attachments for full details.]

View more details on Azure Pipelines