[Pedro Henrique Penna (ppenna)]

Records in saved state, for each state unit, the part of the VM's contract that the unit owns, in place of the flat list of state-unit names that position 017 (#4624) records, so that a restore can tell what each unit is to the guest and can re-create the devices that were hot-plugged onto PCIe ports. This is an alternative to #4624, and #5015 proposes the same change to main.

Three commits on 6ddd6ca50, the head of feature/microvm/dev. microsoft/nvx#1450 pins this head, as roadmap positions 243-245.

Record a contract for each state unit (021d269b8)

• Cause. A name says nothing of what a unit is to the guest: whether the VM's configuration created it or it was hot-plugged, or that the destination's device behind the same name differs, such as in its BARs or PCI IDs. Position 017 also takes the names in a second pass over the unit set after the save, so a unit removed while the save runs can have saved state but no name.
• Change.
  • state_unit: each unit declares a contract when it is added (UnitBuilder::stable_id, kind, lifetime, descriptor, descriptors): its name; a stable, guest-visible identity, the name unless declared; its kind; its lifetime, Static, Dynamic, or Derived; and sorted, deduplicated key=value descriptors. StateUnits::inventory returns the contracts by name. save_with_inventory saves the units and returns the contracts of exactly the units that answered, from the same snapshot of the unit set (run_op_tracked), and, as save does since 78e752de1, fails without sending a request while a unit is in a terminal state. check_inventory matches a saved inventory against the units by name, whatever their registration order, and either reports missing, unexpected, and changed units, duplicate records, and saved state without a record in a typed InventoryMismatch, or returns an InventoryPlan whose recreate lists the recorded dynamic units that the VM lacks.
  • vmotherboard: derives a chipset device's descriptors from what it registers, without device-specific code: mmio and pio for each static region (<label>@<base>+<length>), line and line.target, pci.bdf, pcie.port, or pcie.rciep, and the pci.id, pci.class, pci.revision, and pci.subsystem that its configuration space reports. Its chipset unit and line sets are derived, and add_dyn_device takes a DynamicDeviceContract and records the device as dynamic. A PCIe device's stable identity is its port, pcie-port:<port>, whether vmm_core::device_builder configures it or it is hot-plugged; VPCI and VPCI relay devices record their kinds.
  • openvmm_core, openvmm_defs, openvmm_helpers: the worker's SavedState records the contracts in field 3. Field 2, position 017's names, is retired, and SAVED_STATE_SCHEMA_VERSION moves to 2, so a restore refuses version 1 snapshots by their manifest. vmtime and input are derived. A restore refuses saved state whose check plans to re-create units, until the next commit. The microVM manifest takes its state-unit names from the contracts.
• Tests. state_unit/src/tests/inventory.rs: the contracts' order, each kind of mismatch and its message, the dynamic plan, a save, re-creation, and restore, and a unit removed while the save runs, answered or not. tests/terminal.rs: save_with_inventory fails like save on a terminal unit. vmotherboard: chipset/contract.rs and a static device's registrations. openvmm_core's time ABI tests read the contracts.

Re-create hot-plugged PCIe devices on restore (07b4281df)

• Cause. A device hot-added to a PCIe port has no configuration in a new process, so a restore of its snapshot could only fail.
• Change. On --restore-snapshot, the controller reads the ports that the saved state records as hot-plugged (SavedState::hotplugged_pcie_ports) and hands the devices that the command line places on those ports to the VM worker (VmWorkerParameters::restore_pcie_devices) instead of building them with the machine. Before it applies saved state, the worker re-creates each on its port (dispatch/pcie_hotplug.rs) with the code that the runtime hot-add uses, attaches it without a hotplug notification, since the root complex's saved state restores what the guest saw of the port, and then requires the inventory to match exactly. A recorded device without a supplied device, a supplied device of another kind, and a supplied device on a port that records none fail the restore. Adds the add-pcie-device <PORT> virtio-rng and rm-pcie-device <PORT> console commands, and documents the restore rules and that a guest should release a device before its hot-remove.
• Scope. Only devices on PCIe ports are re-created; another recorded dynamic unit, such as a VPCI device, still fails the restore. A worker restart supplies no devices.
• Tests. openvmm_defs (hotplugged_pcie_ports) and openvmm_entry (take_hotplugged_pcie_devices).

Test a hot-plugged device across save and restore (3e7809ba4)

pcie_hotplug_save_restore hot-adds a virtio-rng device to a root port, pulses save and restore, and checks that the guest still enumerates it, that it still produces entropy, and that it can be hot-removed once the guest releases it. It is unstable for the reason that pcie_save_restore is.

Validation

At this head, 3e7809ba4, on Windows with Rust 1.97.1: cargo check and cargo clippy --all-targets for state_unit, vmotherboard, vmm_core, openvmm_defs, openvmm_helpers, openvmm_core, openvmm_entry, petri, vpci_relay, and vmm_tests, and cargo doc --no-deps for all but vmm_tests, without warnings; cargo nextest run --profile agent for all but vmm_tests, 699 passed; cargo xtask fmt --fix, clean.

At this head, on NVX's KVM, MSHV, and WHP test VMs, through microsoft/nvx#1450's head, 6905c055a, with OpenVMM built from this head:

• KVM: cargo xflowey vmm-tests-run --filter 'test(/openvmm_linux_x64_/)', 40 passed, including pcie_hotplug_save_restore, pcie_hotplug, and pcie_save_restore.

• All three: NVX's default microVM suite, and the runtime adapter's 161 scenarios.

• A cross-process check through the console: hot-add a virtio-rng device to a root port, save-snapshot, and restore in a new process with --virtio-rng --virtio-rng-pcie-port.

  • The device works, and can be removed and added again.
  • A restore without the device, with it on another port, or with a device on the port of one removed before the save fails with an inventory error that names the contract.
  • A device removed before the save, and a cold-plugged one, restore.

  All 9 cases passed on KVM and MSHV. On WHP they passed once save-snapshot opened its RAM file writable: vm_controller's File::open(...).sync_all() fails there with "Access is denied", a separate, existing bug that this pull request does not change.

Before the rebase onto 6ddd6ca50, the same patches, as ad47c5688 on 46bfe02ed, had passed the same runs. The rebase added only save_with_inventory's terminal-unit check.

Land by fast-forwarding feature/microvm/dev to this head, 3e7809ba4343ba630f10841052f35f86b521d231, once microsoft/nvx#1450 is approved, since this repository's merge button squashes and would leave NVX's pin off the branch.

[Pedro Henrique Penna (ppenna)]

Closing as not planned, together with microsoft/nvx#1450, which pinned this head.