Fix Cloud Hypervisor TSC synchronization causing clock skew in VMs

[josharian]

Author: skybrian2
Channel: #feedback
Link: https://discord.com/channels/1405685085923049482/1405686161791516873/1530767831233204245

---

Issue

Playwright tests are failing randomly on skybrian-links VM due to Chromium's network process crashing. Root cause analysis using Opus 5 determined that clock_gettime(CLOCK_MONOTONIC, &ts) is returning non-monotonic values.

Root Cause

The VM is using the TSC (Time Stamp Counter) as its time source. Cloud Hypervisor advertised CLOCKSOURCE_STABLE_BIT and invariant TSC, but did not actually synchronize the guest TSCs across vCPUs. When processes switch vCPUs under load, the clock jumps backwards by up to ~1.197 ms.

The kernel has a check to detect unstable TSC, but it appears to be disabled.

Workaround

Switching to kvm-clock resolves the issue, but only until the next reboot.

Additional Details

• Affected VM: skybrian-links in pdx
• VM uptime: 132 days at time of report
• The clock skew may have developed over time as TSCs drifted apart
• A minidump from the Chromium crash was obtained

Context

Confirmed by @josharian as a Cloud Hypervisor issue that should be fixed on our end.

[josharian]

FWIW, an LLM suggestion as a workaround in the interim:

Make kvm-clock survive reboot (the real gap in his current fix). A one-line systemd unit or tmpfiles rule; I tested both on this VM and both apply cleanly at boot:

# /etc/systemd/system/kvmclock.service
[Unit]
Description=Use kvm-clock (unsynced guest TSCs make CLOCK_MONOTONIC jump backwards)
DefaultDependencies=no
After=sysinit.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'echo kvm-clock > /sys/devices/system/clocksource/clocksource0/current_clocksource'
[Install]
WantedBy=multi-user.target
sudo systemctl enable --now kvmclock

Or the two-line version: echo 'w /sys/devices/system/clocksource/clocksource0/current_clocksource - - - - kvm-clock' | sudo tee /etc/tmpfiles.d/clocksource.conf. Both verified working. The cost is ~120 ns extra per clock_gettime; for Playwright/Chromium that's noise.

If you try this out, please let me know how it goes.

cc @skybrian

[skybrian]

Thanks, I'm good at this point. BTW, here's a test written in TypeScript (for Deno, but probably easy to port) that will fail if clock skew shows up again:

https://gist.github.com/skybrian/221ffb8eef2ffc4100771bb0a7ad2816