Repository navigation
state_unit: record unit contracts and re-create hot-plugged PCIe devices - #5015
Closed
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Closed
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Pedro Henrique Penna (ppenna) wants to merge 3 commits into
Conversation
A VM's saved state holds only the state units that have state to save, keyed by name. A restore fails for a saved unit that the destination VM lacks, but a destination with additional units restores without complaint, those units in their initial state, and nothing records what each unit was to the guest: whether the VM's configuration created it or it was hot-plugged, or that a destination's device differs from the saved one behind the same name. Give each unit a contract, which it declares when it is added: a stable, guest-visible identity, such as the PCIe port of its device; its kind; its lifetime, static, dynamic, or derived; and descriptors of what the guest sees of it. vmotherboard derives a chipset device's descriptors from its static regions, interrupt lines, PCI placement, and the identity that its PCI configuration space reports, and records a dynamically added device as dynamic. A PCIe device takes its port as its stable identity, whether configured or hot-plugged. StateUnits::save_with_inventory records the contracts of the units that it saved from the same snapshot of the unit set as their saved state, so a unit removed while the save runs appears in both or in neither. StateUnits::check_inventory matches a saved inventory against the units by name, whatever their registration order, reports missing, unexpected, and changed units, duplicate records, and saved state without a record in a typed InventoryMismatch, and returns the recorded dynamic units that the destination lacks, which the VM worker refuses until it can re-create them. Move the VM worker's saved-state envelope from openvmm_core to openvmm_defs, keeping its protobuf name, and record the contracts in a new field. A restore checks them when the saved state has them; saved state from before this change has none, and restores as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A device hot-added to a PCIe port records a dynamic state-unit contract whose stable identity is its port, but a restore into a new process had no such device, so it could only fail. Re-create such devices. On --restore-snapshot, the controller reads the ports that the saved state records as hot-plugged and hands the devices that the command line places on those ports to the VM worker instead of building them with the machine. Before it applies saved state, the worker re-creates each recorded hot-plugged device on its port from the supplied device, without a hotplug notification, since the root complex's saved state restores what the guest saw of the port, and then requires the inventory, which now includes the re-created devices' PCI identities, to match exactly. A recorded device without a supplied device, a supplied device of another kind, and a supplied device on a port that records none fail the restore. The runtime hot-add and the re-creation build the device with the same code. A worker restart supplies no devices, so it still cannot restore a VM with a hot-plugged device, now with an error that names the device. Add the add-pcie-device and rm-pcie-device console commands, so that a command-line VM can hot-plug a virtio-rng device and remove it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hot-add a virtio-rng device to a PCIe root port, pulse save/restore, and check that the guest still enumerates the device, that it still produces entropy, and that it can still be hot-removed. The save records the device's dynamic state-unit contract for its port, and the restore checks it. The test is unstable for the reason that pcie_save_restore is: pulse save/restore of PCIe on Linux direct is flaky in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Nov 10, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Restore can attach a device to a port with hotplug disabled, changing guest-visible capabilities and preventing removal, while the existing test never exercises fresh-worker device recreation.
2 open issues
What changed in this PR
This PR adds state-unit contracts to OpenVMM snapshots, strengthening restore compatibility checks and enabling recreation of hot-plugged PCIe devices in a new worker. Snapshots without an inventory retain the previous restore behavior.
Changes:
- Record unit identities, lifetimes, and device descriptors alongside saved state, then validate them before restore.
- Recreate saved hot-plugged PCIe devices from supplied resources without guest notifications.
- Add console commands, contract tests, a hotplug save/restore test, and documentation.
| File | Description |
|---|---|
vmm_tests/vmm_tests/tests/tests/multiarch/pcie.rs |
Tests hotplug save/restore, entropy, and removal. |
vmm_core/vmotherboard/src/lib.rs |
Exports dynamic contract helpers. |
vmm_core/vmotherboard/src/chipset/mod.rs |
Registers and exports the contract module. |
vmm_core/vmotherboard/src/chipset/line_sets.rs |
Marks interrupt line units as derived. |
vmm_core/vmotherboard/src/chipset/contract.rs |
Defines PCI identity and region descriptors. |
vmm_core/vmotherboard/src/chipset/builder/mod.rs |
Records contracts for dynamically added devices. |
vmm_core/vmotherboard/src/chipset/backing/arc_mutex/state_unit.rs |
Tests static device registration contracts. |
vmm_core/vmotherboard/src/chipset/backing/arc_mutex/services.rs |
Collects device registration descriptors. |
vmm_core/vmotherboard/src/chipset/backing/arc_mutex/device.rs |
Adds device contract builder methods. |
vmm_core/state_unit/src/tests/inventory.rs |
Tests inventory validation and save consistency. |
vmm_core/state_unit/src/lib.rs |
Tracks contract metadata through unit operations. |
vmm_core/state_unit/src/inventory.rs |
Implements contracts, validation, and recreation plans. |
vmm_core/src/device_builder.rs |
Declares PCIe and virtual PCI device identities. |
vm/devices/pci/vpci_relay/src/lib.rs |
Declares relayed device contracts. |
petri/src/worker.rs |
Initializes the restore-device parameter. |
openvmm/openvmm_entry/src/ttrpc/mod.rs |
Initializes the restore-device parameter. |
openvmm/openvmm_entry/src/repl.rs |
Adds PCIe hot-add and removal commands. |
openvmm/openvmm_entry/src/lib.rs |
Separates hot-plugged resources during snapshot restore. |
openvmm/openvmm_entry/Cargo.toml |
Adds the contract test dependency. |
openvmm/openvmm_defs/src/worker.rs |
Defines the shared snapshot envelope and restore resources. |
openvmm/openvmm_defs/Cargo.toml |
Adds saved-state and test dependencies. |
openvmm/openvmm_core/src/worker/dispatch/pcie_hotplug.rs |
Shares hotplug construction and implements recreation. |
openvmm/openvmm_core/src/worker/dispatch.rs |
Integrates inventory saving and restore validation. |
Guide/src/user_guide/openvmm/snapshots.md |
Explains contract-based restore compatibility. |
Guide/src/reference/openvmm/management/interactive_console.md |
Documents PCIe console commands. |
Guide/src/reference/emulated/pcie/overview.md |
Documents hotplug restore and removal behavior. |
Guide/src/dev_guide/snapshot_format.md |
Describes inventory encoding and matching rules. |
Cargo.lock |
Records updated workspace dependencies. |
Review effort: Balanced
Comment on lines
+222
to
+224
| let attached = root_complex | ||
| .lock() | ||
| .add_pcie_device(devfn, "hotplug-device", bus_device); |
|
|
||
| // Pulse save/restore — drop agent first (vsock won't survive) | ||
| drop(agent); | ||
| vm.backend().verify_save_restore().await?; |
Member
|
I think we need a distinction between which state_units are guest visible, and which are not. We still need the distinction between what is static or part of the VM when it was created, and what was hot-added, but even then some of the devices could be considered hot-added (or hot addable?) even though they were there at creation time. And some of the things that are added by VMM infrastructure (like vmtime) must be saved because they are guest visible. One way to think about guest visibility is "is there saved state for this state unit?" where if we didn't save state and restore it, did restoring the VM put it in the wrong state? |
Contributor
Author
|
Closing as not planned, together with #5014 and microsoft/nvx#1450. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A VM's saved state holds only the state units that have state to save, keyed by name. A restore fails for a saved unit that the destination lacks. A destination with extra units, however, restores without complaint, and those units stay in their initial state. Nothing in the saved state says what a unit was to the guest. #4624 records the names of all the units. This pull request is an alternative to it: each unit records its contract, the part of the VM's contract that the unit owns. A contract also catches a different device behind the same name, and lets a restore re-create the devices that were hot-plugged onto PCIe ports.
Changes
state_unit). A unit declares its contract when it is added, throughUnitBuilder::stable_id,kind,lifetime, anddescriptorordescriptors. The contract holds:Static(the configuration created it),Dynamic(added while the VM ran, such as a hot-plugged device), orDerived(VMM infrastructure, such asvmtime);key=valuedescriptors of what the guest sees of it.vmotherboard,vmm_core,vpci_relay).vmotherboardderives a chipset device's descriptors from what the device registers, without device-specific code: each static MMIO and PIO region, its interrupt lines and their targets, its PCI placement, and the vendor and device, class, revision, and subsystem IDs that its configuration space reports. The chipset unit and line sets are derived.add_dyn_devicetakes aDynamicDeviceContractand records the device as dynamic. A PCIe device's identity is its port,pcie-port:<port>, whether configured or hot-plugged.StateUnits::save_with_inventorysaves the units and returns the contracts of exactly the units that answered, from the same snapshot of the unit set. The VM worker's saved-state envelope moves fromopenvmm_coretoopenvmm_defs, keeping its protobuf package, and records the contracts as field 2.StateUnits::check_inventorymatches the recorded contracts against the destination's units by name, whatever order either VM registered them in. It reports missing, unexpected, and changed units, duplicate records, and saved state without a record in a typedInventoryMismatch, or returns the recorded dynamic units that the destination lacks. Saved state without field 2, written before this change, restores as before, without the check.openvmm_core,openvmm_entry). On--restore-snapshot, the devices that the command line places on the ports that the snapshot records as hot-plugged go to the VM worker, instead of being built with the machine.add-pcie-device <PORT> virtio-rngandrm-pcie-device <PORT>.pcie_hotplug_save_restorehot-adds a virtio-rng device, pulses save and restore, and checks that the guest still enumerates the device, that it still produces entropy, and that it can be hot-removed once the guest releases it. It is unstable for the reason thatpcie_save_restoreis.Compared with #4624
Validation
At
19d627c80, on Windows x86-64 with Rust 1.97.1:cargo clippy --all-targets -- -D warningsforstate_unit,vmotherboard,vmm_core,vpci_relay,openvmm_defs,openvmm_core,openvmm_entry,petri, andvmm_tests: clean.cargo doc --no-depswithRUSTDOCFLAGS=-D warningsfor the same packages exceptvmm_tests: clean.cargo nextest run --profile agentfor the same packages exceptvmm_tests: 308 passed.cargo xtask fmt --fix: clean.On Linux/KVM (Azure Linux 3.0) with Rust 1.97.1, at
19d627c80:cargo xflowey vmm-tests-run --filter 'test(/openvmm_linux_x64_/)': 38 passed, includingpcie_hotplug_save_restore,pcie_hotplug, andpcie_save_restore.A cross-process check with this branch's
openvmm, driven through the console, with the OpenVMM test kernel on two hot-pluggable root ports:save-snapshot, and restore in a new process with--virtio-rng --virtio-rng-pcie-port: the device produces entropy, and can be removed and added again.All 9 cases passed.
The same change on the NVX fork's microVM branch, #5014, passed the cross-process check on KVM, MSHV, and WHP.
On Windows,
save-snapshotfails with "Access is denied" before and after this change:vm_controlleropens the RAM backing file read-only, then callssync_allon it. The cross-process check on WHP opened the file writable to pass that point; this pull request does not change it.