fix: pin 2404 kernel to 6.8.0-1062-azure due to perf regression - #9186
Open
Nishchay (awesomenix) wants to merge 1 commit into
Open
fix: pin 2404 kernel to 6.8.0-1062-azure due to perf regression#9186Nishchay (awesomenix) wants to merge 1 commit into
Nishchay (awesomenix) wants to merge 1 commit into
Conversation
Nishchay (awesomenix)
requested review from
Chou Hu (AbelHu),
Devinwong,
Sri Harsha (SriHarsha001),
Calvin S. (calvin197),
Cameron Meissner (cameronmeissner),
Sylvain Boily (djsly),
Ganeshkumar Ashokavardhanan (ganeshkumarashok),
Karen Chen (karenychen),
lilypan26,
Mark Ibrahim (mxj220),
Peter Damianov (pdamianov-dev),
Patrick W. Healy (phealy),
r2k1,
Runzhen (runzhen),
sulixu,
Tim Wright (timmy-wright),
Thibault Cohen (titilambert),
Xu Xue (xuexu6666) and
Zachary (zachary-bailey)
as code owners
August 12, 2026 14:14
Contributor
Windows Unit Test Results 3 files 12 suites 51s ⏱️ Results for commit 8b201db. |
Contributor
There was a problem hiding this comment.
Pull request overview
Pins the Ubuntu 24.04 (noble) Azure kernel used during VHD build/tests to avoid a reported performance regression specific to 6.8.0-1063-azure, aligning the VHD build output with the intended rollback target (6.8.0-1062-azure).
Changes:
- Pin Ubuntu 24.04 non-CVM kernel install in
pre-install-dependencies.shto6.8.0-1062-azure(including matching tools/headers/modules-extra packages). - Update the Linux VHD content test to enforce an exact kernel match on Ubuntu 24.04 non-CVM builds, while skipping the exact pin check for CVM builds.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| vhdbuilder/packer/test/linux-vhd-content-test.sh | Adds Ubuntu 24.04 non-CVM exact kernel version validation (and skips exact pin validation for CVM builds). |
| vhdbuilder/packer/pre-install-dependencies.sh | Installs a pinned 6.8.0-1062-azure kernel package set for Ubuntu 24.04 non-CVM builds and avoids pulling the LTS metapackage/modules-extra metapackage that could reintroduce 1063. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| echo "OS is Ubuntu, FIPS is not enabled, check LTS kernel version" | ||
| # CVM builds use linux-image-azure-fde-lts-* (different flavor), skip exact pin check | ||
| local is_cvm=false | ||
| if grep -q "cvm" <<< "$FEATURE_FLAGS"; then |
lilypan26
approved these changes
Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kernel 6.8.0-1063-azure (Ubuntu 24.04 / noble) introduces a regression affecting networking dataplane and CPU-bound workloads. 6.8.0-1062-azure and 6.8.0-1059-azure are unaffected. The regression is in the kernel only — userspace (iptables 1.8.10-3ubuntu2 , etc.) is unchanged across the affected/unaffected builds, so package versions do not explain the symptoms.
Root cause — two independent 1063-only changes
• Reverts a perf optimization (added to fix a bridge hairpinning bug). The conntrack input hook now runs on every promisc-mode packet.
• Impact: nf_tables ruleset programming instability and elevated per-packet conntrack CPU on bridge/promisc dataplanes. Observed as a NAT ruleset that reports a successful sync but does not persist ( nft list ruleset empty despite hundreds of programmed rules), causing loss of service DNAT → TCP retransmit stalls (65/134/285/365s) and 5xx.
• Enables ITS mitigation (aligned branch/return thunks, CVE-2024-28956) on affected Intel SKUs.
• Impact: extra indirect-branch overhead on JIT/dispatch-heavy workloads (observed ~2x CPU/request on a managed-runtime service). Confirmed active: indirect_target_selection: Mitigation: Aligned branch/return thunks .
Observed impact (all on kernel 6.8.0-1063-azure)
• Dataplane: nf_tables NAT rules fail to persist on affected nodes → service routing/DNAT loss → connection stalls and 5xx.
• CPU: ~2x CPU/request + intermittent 5xx on an indirect-branch-heavy workload, immediately after upgrade from a 1062-based image.
• Mitigation confirmed in the field: rolling the affected nodes back to a pre-1063 image cleared the issue in every case.
Repro status
Attribution is from the 6.8.0-1059/1062 → 1063 changelog diff plus field rollback evidence. A clean same-SKU 1062-vs-1063 A/B is still pending (test nodes landed on differing CPU SKUs).