Skip to content

chore(gpu): bump aks-gpu-grid to 570.237-20260817204535 - #9229

Open
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
mainfrom
ganesh/gpu-grid-570.237
Open

chore(gpu): bump aks-gpu-grid to 570.237-20260817204535#9229
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
mainfrom
ganesh/gpu-grid-570.237

Conversation

@ganeshkumarashok

Copy link
Copy Markdown
Contributor

What

Bumps the aks-gpu-grid container image from 570.211.01-20260522192315 to 570.237-20260817204535.

This picks up the NVIDIA GRID driver bump merged in Azure/aks-gpu#179, which moved the GRID runfile from 570.211.01 to 570.237 — the vGPU18.8 build for NVadsA10_v5, per the Azure N-series driver docs.

⚠️ The new version has only TWO components, and that broke things

570.237 is not a truncated 570.237.xx. It is genuinely two-component. Verified from the runfile itself:

  • runfile header: version_string=570.237, targetdir=NVIDIA-Linux-x86_64-570.237-grid-azure
  • the built image ships libnvidia-ml.so.570.237

Two places in this repo assumed a three-component version and had to be fixed:

1. .github/renovate.json — this is why Renovate never proposed it

The aks/aks-gpu-grid rule pinned versioning to a regex requiring exactly three numeric components:

regex:^(?<major>\d+)\.(?<minor>\d+)\.(?<patch>\d+)-(?<prerelease>\d{14})$

570.237-20260817204535 does not match, so Renovate could not parse the new tag and would have silently skipped this bump indefinitely — no error, no PR. Retriggering Renovate would never have worked. The patch group is now optional:

regex:^(?<major>\d+)\.(?<minor>\d+)(\.(?<patch>\d+))?-(?<prerelease>\d{14})$

Renovate treats omitted groups as 0, so ordering still works correctly — 570.237 sorts above 570.211.01 on the minor component. Empirically verified the new regex:

input result
570.211.01-20260522192315 matches, patch=01
570.237-20260817204535 matches, patch undefined
bogus-tag no match
13-digit timestamp no match (still anchored)

2. pkg/agent/datamodel/gpu_components_test.go

TestLoadConfig asserted ^\d+\.\d+\.\d+$ and failed on the new version:

gpu_components_test.go:48: NvidiaGridDriverVersion '570.237' does not match expected format

Relaxed to ^\d+(\.\d+)+$ (two or more components), which mirrors the version guard used in the upstream Azure/aks-gpu repo.

No production code change was needed. gpu_components.go splits latestVersion on - into version + suffix and makes no assumption about component count.

Validation

  • ✅ Confirmed the new tag is already mirrored to MCR (mcr.microsoft.com/aks/aks-gpu-grid:570.237-20260817204535)
  • go test ./pkg/... — all green (pkg/agent, pkg/agent/datamodel, pkg/agent/toggles, pkg/vhdbuilder/datamodel)
  • go test ./pkg/gpu/... ./parser/... in aks-node-controller — green
  • make generate produces no diff for this change — the GRID driver version is not embedded in any testdata, so no regeneration is required
  • components.json validated as well-formed JSON
Note on make generate / validate-shell in this repo

make generate currently fails at the validate-shell step, and make generate-testdata produces testdata churn — but both reproduce identically on a pristine clone of main with zero changes applied. They are pre-existing drift, unrelated to this PR, and are deliberately excluded from this diff to keep it reviewable.

Follow-up (deliberately out of scope)

The three sibling GPU Renovate rules carry the identical strict 3-component regex and have the same latent silent-failure risk if NVIDIA ships a two-component build on those branches:

  • aks/aks-gpu-cuda-lts
  • aks/aks-gpu-cuda
  • aks/aks-gpu-grid-v20

I left them alone to keep this PR scoped to the driver actually being bumped, but they should probably be relaxed the same way.

Also: existing Renovate PR #8306 proposes 570.211.01-20260629214405 — an older driver with a newer rebuild timestamp. It is superseded by this PR and should be closed.

Picks up the NVIDIA GRID driver bump merged in Azure/aks-gpu#179, which
moved the GRID runfile from 570.211.01 to 570.237 (vGPU18.8 build for
NVadsA10_v5). The published image tag is already mirrored to MCR.

Note that 570.237 has only TWO version components, not the usual three.
This is genuine, not a truncation: the runfile header reports
version_string=570.237 and the image ships libnvidia-ml.so.570.237.
That breaks two places that assumed a three-component version:

1. .github/renovate.json - the aks/aks-gpu-grid versioning regex required
   exactly three numeric components, so Renovate could never parse the
   new tag and would have silently skipped this bump forever. The patch
   group is now optional. Renovate treats omitted groups as 0, so
   ordering still works (570.237 > 570.211.01 on the minor component).

2. pkg/agent/datamodel/gpu_components_test.go - TestLoadConfig asserted
   ^\d+\.\d+\.\d+$ and failed on the new version. Relaxed to
   ^\d+(\.\d+)+$, mirroring the guard in the upstream aks-gpu repo.

The Go parser in gpu_components.go needed no change: it splits on "-"
into version and suffix and makes no assumption about component count.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e525ff6d-7072-4f71-92b9-4d5a87808b20
@github-actions

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   12 suites   52s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit 8cb4cb3.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the NVIDIA GRID GPU driver container image tag used by AgentBaker’s GPU component configuration, and adjusts Renovate/test guards to support NVIDIA’s newly introduced two-component driver version format (e.g., 570.237).

Changes:

  • Bump aks/aks-gpu-grid image tag in parts/common/components.json to 570.237-20260817204535.
  • Relax Renovate regex versioning for aks/aks-gpu-grid to make the patch component optional (so Renovate can detect two-component tags).
  • Relax the datamodel test’s driver-version regex to accept 2+ dot-separated numeric components.

Package Update Analysis: aks/aks-gpu-grid

Version change: 570.211.01-20260522192315570.237-20260817204535 (minor update on the driver line: 570.211570.237, with patch omitted in the new tag)
OS variants affected: Linux GPU flows that consume this image tag (Ubuntu and Azure Linux/Mariner paths that rely on GPUContainerImages)
OS variants NOT updated: None (single shared image tag entry)

Upstream changelog between versions: Upstream, point-to-point release notes for exactly 570.211.01570.237 were not found publicly in a way that allows a precise enumerated diff; NVIDIA vGPU documentation typically provides release notes PDFs and compatibility/known-issues guidance, but the exact 18.8/570.237 delta appears to be gated or not easily discoverable. Manual workload validation is recommended for this driver jump. (See NVIDIA vGPU documentation portal and generic Linux KVM release notes PDFs for the closest official references.)

Overall Risk: 🟡 Medium → 🔴 High (environment-dependent)

Justification: Although the code/config changes are small and well-scoped, this updates a production GPU driver payload. Driver changes can cause workload regressions, kernel/guest compatibility issues, or performance/behavior changes that aren’t detectable via unit tests alone.
Recommendation: Flag for manual testing on representative NVadsA10_v5 workloads before merge.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pkg/agent/datamodel/gpu_components_test.go Relax test guard to accept 2+ component NVIDIA driver versions.
parts/common/components.json Bump aks-gpu-grid image tag to 570.237-20260817204535.
.github/renovate.json Relax aks/aks-gpu-grid Renovate regex versioning to allow optional patch component.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

components This pull request updates cached components on Linux or Windows VHDs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants