Skip to content

fix: pin 2404 kernel to 6.8.0-1062-azure due to perf regression - #9186

Open
Nishchay (awesomenix) wants to merge 1 commit into
mainfrom
nishp/revert-erofs/main
Open

fix: pin 2404 kernel to 6.8.0-1062-azure due to perf regression#9186
Nishchay (awesomenix) wants to merge 1 commit into
mainfrom
nishp/revert-erofs/main

Conversation

@awesomenix

@awesomenix Nishchay (awesomenix) commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Holding at 6.8.0-1062 forgoes 507 CVE fixes from the 1063 security rebase + 1064 roll-up. ~35 are critical-class (UAF/OOB/double-free/race) in cloud-reachable paths — core net/netfilter/TCP-IP, KVM, NVMe-oF/RDMA, the Azure mana NIC driver (CVE-2026-43056), and SMB/NFS. Rollback should therefore be a temporary, targeted mitigation for affected workloads, with a fixed 1064-base kernel (regressions reverted) prioritized.

Kernel 6.8.0-1063-azure (Ubuntu 24.04 / noble) introduces a regression affecting networking dataplane and CPU-bound workloads. 6.8.0-1062-azure and 6.8.0-1059-azure are unaffected. The regression is in the kernel only — userspace (iptables  1.8.10-3ubuntu2 , etc.) is unchanged across the affected/unaffected builds, so package versions do not explain the symptoms.

Root cause — two independent 1063-only changes

  1. LP #2070072 —  SAUCE: Revert "netfilter: br_netfilter: skip conntrack input hook for promisc packets" 
    • Reverts a perf optimization (added to fix a bridge hairpinning bug). The conntrack input hook now runs on every promisc-mode packet.
    • Impact: nf_tables ruleset programming instability and elevated per-packet conntrack CPU on bridge/promisc dataplanes. Observed as a NAT ruleset that reports a successful sync but does not persist ( nft list ruleset  empty despite hundreds of programmed rules), causing loss of service DNAT → TCP retransmit stalls (65/134/285/365s) and 5xx.
  2. LP #2144730 — "ITS mitigation is not enabled on affected CPUs"
    • Enables ITS mitigation (aligned branch/return thunks, CVE-2024-28956) on affected Intel SKUs.
    • Impact: extra indirect-branch overhead on JIT/dispatch-heavy workloads (observed ~2x CPU/request on a managed-runtime service). Confirmed active:  indirect_target_selection: Mitigation: Aligned branch/return thunks .

Observed impact (all on kernel 6.8.0-1063-azure)

• Dataplane: nf_tables NAT rules fail to persist on affected nodes → service routing/DNAT loss → connection stalls and 5xx.
• CPU: ~2x CPU/request + intermittent 5xx on an indirect-branch-heavy workload, immediately after upgrade from a 1062-based image.
• Mitigation confirmed in the field: rolling the affected nodes back to a pre-1063 image cleared the issue in every case.

Repro status

Attribution is from the  6.8.0-1059/1062 → 1063  changelog diff plus field rollback evidence. A clean same-SKU 1062-vs-1063 A/B is still pending (test nodes landed on differing CPU SKUs).

@github-actions

Copy link
Copy Markdown
Contributor

Windows Unit Test Results

  3 files   12 suites   51s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit 8b201db.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins the Ubuntu 24.04 (noble) Azure kernel used during VHD build/tests to avoid a reported performance regression specific to 6.8.0-1063-azure, aligning the VHD build output with the intended rollback target (6.8.0-1062-azure).

Changes:

  • Pin Ubuntu 24.04 non-CVM kernel install in pre-install-dependencies.sh to 6.8.0-1062-azure (including matching tools/headers/modules-extra packages).
  • Update the Linux VHD content test to enforce an exact kernel match on Ubuntu 24.04 non-CVM builds, while skipping the exact pin check for CVM builds.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
vhdbuilder/packer/test/linux-vhd-content-test.sh Adds Ubuntu 24.04 non-CVM exact kernel version validation (and skips exact pin validation for CVM builds).
vhdbuilder/packer/pre-install-dependencies.sh Installs a pinned 6.8.0-1062-azure kernel package set for Ubuntu 24.04 non-CVM builds and avoids pulling the LTS metapackage/modules-extra metapackage that could reintroduce 1063.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

echo "OS is Ubuntu, FIPS is not enabled, check LTS kernel version"
# CVM builds use linux-image-azure-fde-lts-* (different flavor), skip exact pin check
local is_cvm=false
if grep -q "cvm" <<< "$FEATURE_FLAGS"; then
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants