openvmm: carry a clone's activation record in the restore packet by ppenna · Pull Request #1419 · microsoft/nvx

Pedro Henrique Penna (ppenna) · 2026-11-09T16:44:59Z

Part of #544

What and why

#544 asks to deliver a platform clone's activation record before the first
restored vCPU runs, in the restore packet, or in a configuration region outside
captured RAM only if measurements call for one, and then to retire the
activation RPC: the Activate in which nvx::host::template::clone sends a
clone its record on its control session, once the host has connected after the
restore. Its scope change asks to measure the byte channel first, against a
budget for the slowest backend, and to keep the packet if the expected record
fits. This pull request is the first part: the restore packet's record in
OpenVMM, its pin, the guest's decoding of it, and the measurements, which keep
the packet. The libraries still deliver the record in Activate; part 2, below,
moves them to the packet and retires Activate.

microsoft/openvmm#4996, one commit on 6cb6ba87c, the head of
feature/microvm/dev:

Bit 5 of the restore packet's flags, ACTIVATION, which was reserved, says
that the packet ends, after the DNS server, with the record's length, a
little-endian u32 from 1 to 1,024, then the record's bytes, which OpenVMM
passes to the guest without interpreting them.
--restore-activation-record PATH, and the management RPC's
MicrovmSnapshotConfig.restore_activation_record_path (15), which the
server converts to that option as OpenVMM: support sandbox blocks, networking, and supplied RAM in the microVM management RPC #760's conversion does every restore
field, read the record before any vCPU runs, reading at most one byte past
the bound. Only a platform snapshot admits it, and such a restore gates host
input until the guest acknowledges it, so the guest has read the record
before host input reaches it.

In NVX, the time ABI's decoder (nvx::guest::time) takes the flag: the header
adds the record's length to the body, and the packet takes the record's bytes,
bounded by nvx::activation::MAX_RECORD_BYTES, in the packet's port
transaction, before the time sample; a length of 0 or above the bound fails
the repair, as every invalid field does. Restore::activation_record returns
the bytes, undecoded, which part 2's activation decodes, and nvx-time's test
mode prints them. An ignored test of tests/openvmm_vectors.rs checks that
OpenVMM's bound is MAX_RECORD_BYTES. The guide's restore packet (5.5.5)
describes the flag, the record's layout and cost, and why the packet keeps its
version.

Decisions:

The packet stays version 1. Every guest that decodes version 1 refuses a
packet that sets a reserved flag bit, so a guest that predates the flag fails
a restore that hands it a record rather than ignore the record, and a packet
without the flag keeps its layout and meaning (both checked on KVM, below).
No contract version changes, so existing snapshots and templates stay
restorable, template keys, SOURCE-MANIFEST.json, and the release that
bundled pins stay valid, and the crate's publication isn't blocked.
nvx-rs: replace activation RPC delivery with an uncaptured VMM configuration region #544's scope change proposed a new version; the network settings' precedent
bumped the version only for the packets that carried the new field, which
the reserved bit makes unnecessary here.
A u32 length. It keeps the record on a four-byte boundary of the packet
(24-byte header, 16-byte ranges, 64 bytes of entropy, 16-byte DNS server,
4-byte length): the guest reads the packet four bytes per port access and
keeps only the bytes that it asked for of the last one, so a read that
spanned a 2-byte length and the record would drop the record's first bytes.
The bound is 1,024 bytes, MAX_RECORD_BYTES, whose largest record holds
602 bytes. OpenVMM's bound is the same, and the vector test keeps them equal.
Platform snapshots only, and gated. A snapshot of a later tier continues
a tenant's workload with the settings that it was captured with, and one
without a tier has no activation.

Measurements

Each four bytes of the record cost the guest one port access. Measured on the
three test hosts (Azure VMs: nested virtualization, shared with other jobs, so
their p95 is noisy, KVM's above all), with the platform snapshot of
nvx.py benchmark --suite platform-snapshot-restore-vcpu (capacity 8,
boot-online 1, sandbox blocks), restored at target 1, a gated restore whose
guest reads the packet through nvx-time, with records of 0, 64, 256, and
1,024 bytes, interleaved, forward and backward in turn, 2 warmups and 30 runs
per size and backend. A measurement-only span in OpenVMM's portb device, not
in this pull request, timed the packet's read from its selection to its last
byte. 4,096 bytes exceeds the bound, so its row is extrapolated from the cost
per read.

Packet read, p50 / p95 (ms):

Record	Packet	Reads	KVM	MSHV	WHP
none	104 B	26	0.51 / 3.05	0.71 / 0.79	2.44 / 3.54
64 B	172 B	43	0.70 / 2.91	1.15 / 1.30	4.26 / 6.28
256 B	364 B	91	1.22 / 3.67	2.52 / 2.62	6.91 / 10.84
1,024 B	1,132 B	283	3.02 / 19.51	7.74 / 7.92	16.65 / 22.65

The cost of one four-byte read, by least squares over the per-size medians:
KVM 9.7 µs, MSHV 27.4 µs, WHP 53.5 µs. What a record adds, at that cost:

Record	KVM	MSHV	WHP
131 B, the smallest clone record of the activation vectors	0.33 ms	0.93 ms	1.82 ms
265 B, a GPT clone with an export and a network	0.66 ms	1.86 ms	3.64 ms
602 B, the largest record	1.48 ms	4.16 ms	8.13 ms
1,024 B, the bound	2.50 ms	7.04 ms	13.74 ms
4,096 B, extrapolated	9.96 ms	28.08 ms	54.79 ms

The restore to the guest's readiness marker, p50 / p95, from no record to
1,024 bytes: MSHV 50.03 / 55.16 to 57.81 / 61.95 ms, WHP 218.30 / 235.12 to
231.53 / 267.25 ms; on KVM the host's noise exceeds the difference
(114.71 / 152.41 to 117.29 / 146.29 ms).

Budget and delivery. #544 leaves the budget to agree; this pull request
proposes 10 ms at p50 on the slowest backend for the largest record that
nvx::activation produces, and 5 ms for the expected records, 131 to 265
bytes. WHP, the slowest, takes 8.1 ms and 1.8 to 3.6 ms, and MSHV and KVM
less, so the packet fits on every backend, and no configuration region, nor
its change of the machine contract, is needed. Records near the 4 KiB that the
issue lists would not fit WHP's budget, but none needs more than 602 bytes.

Part 2

Left for #544's next pull request:

The host passes each clone's record in its restore packet: the clone's
launch hands OpenVMM the record that it writes to the sandbox, with
--restore-activation-record.
The guest activates from the packet: the repaired clone carries the
record, and nvx::guest::clone::Activation validates and applies it right
after the repair, before the host connects, and sends Ready unprompted once
the session's reset came, in that session and in each later one.
Activate is removed from nvx::guest::clone, from
nvx::host::template::clone (CloneControl::activate becomes a wait for
Ready, and reattach checks a later session's Ready), and from the runtime
adapter's framing and agents.
Template keys refuse templates whose guests predate the flag, whose
clones would otherwise fail their repair after their launch.
Restore-to-Ready is measured, source-matched and interleaved, against the
Activate path, on KVM, MSHV, and WHP.

Commits

openvmm: pin the restore packet's activation record moves the gitlink from
6cb6ba87c to d501e50660136000ec93412aea78fdafb9cf391c, the head of
[DO NOT REVIEW] microvm: carry a clone's activation record in the restore packet openvmm#4996, and changes the guest's decoder, its tests, the
public API snapshot, and the vector test with it. The protocol vectors and
CPU profiles that nvx-rs copies are unchanged.
doc: describe the restore packet's activation record updates the guide's
restore packet and guest repair (5.5.5, 5.5.7), the compatibility model
(1.5), and the CI guide's account of the vector test.
doc: map the restore packet's activation record in the OpenVMM roadmap
maps position 233, unassigned, 102 commits past the mapped tip, with the
mapped work that it extends: the restore preparation of PR17's position 079,
the platform tier of PR19's position 093, the gate of PR20's position 095,
and the management RPC's conversion of position 228. It conflicts with
upstream main in all nine of its files, so it follows the time ABI series,
PR15, PR17, PR19, PR20, and PR25 upstream.

Validation

Local gates, local_gates.py --base origin/dev --run --keep-going at this
head:

Windows: test-changes, markdown-links, test-layout,
openvmm-vectors, openvmm-cpu-profiles, and the nvx-rs gates (format,
lint, test, bundling, doc, MSRV, and the runtime adapter's protocol and host)
passed. nvx-cli ran 2,188 tests, 88 of them skipped, and failed only in the
three unreached cases of test_host_loopback_allow_script_checks_publication_and_traffic,
which time out in Git for Windows' sh on this machine and pass in CI and
on Linux, and test_a_shallow_clone_cannot_count_the_chain, which passes
alone; CI's NVX CLI / Windows and NVX CLI / Linux passed.
Linux (nvx-mshv-test, Azure Linux 3.0): test-changes, markdown-links,
test-layout, openvmm-vectors (with
the_activation_bound_is_the_submodules), openvmm-cpu-profiles, and every
nvx-rs gate, the guest's binaries and the runtime adapter's protocol, host,
and guest included, passed. nvx-cli failed only because the host's Python
has no pip, and openvmm-ci waits for [DO NOT REVIEW] microvm: carry a clone's activation record in the restore packet openvmm#4996's CI, whose
unit tests and Linux VMM tests have passed.

OpenVMM, at d501e5066: cargo clippy --all-targets, cargo doc --no-deps,
and cargo nextest run --profile agent of chipset_resources, chipset,
openvmm_entry, and openvmm_helpers passed on Windows (522 tests) and on
Linux (554 tests), and cargo xtask fmt --fix is clean.

Remote runs on the test hosts, with OpenVMM built from the pin and the guest
rebuilt (the guest's artifacts of this head are byte-identical to those of
the first round). The restore scenarios are snapshot-core,
snapshot-tiers, restore-processors, restore-memory,
untiered-network-policy, platform-restore-processors,
platform-network-policy, and time-abi-conformance, of
nvx.py test-microvm.

Profile	Backend	Host type	Before: `dev`	First round: `2a061ea61` on `a386c22e4`	This head: `6a7eec4be`, pin `d501e5066`
`nvx-kvm-test`	`kvm`	`virtual-machine`	Release `v0.1.0-dev.8ebadbd3dae0` (OpenVMM `98e5b3ca5`): OpenVMM refuses `--restore-activation-record` (`unexpected argument`, exit 2). Its guest, with this pin's OpenVMM and a platform snapshot: a restore without a record reaches its marker; one with a 64-byte record fails closed, `G_REPAIR_PACKET`, `restore packet flags 0x24 set reserved bits`	8 restore scenarios passed; measurements	8 restore scenarios passed; this guest's restores with and without a 64-byte record reach their markers
`nvx-mshv-test`	`mshv`	`virtual-machine`	Not run; the guest's refusal doesn't depend on the backend	8 restore scenarios passed; measurements	8 restore scenarios passed
`nvx-whp-test`	`whp`	`virtual-machine`	Not run	8 restore scenarios passed; measurements	8 restore scenarios passed

Left to CI: the full matrix (the runtime adapter's VM suites, the Ubuntu
guest, every microVM scenario, the guest's determinism, and the benchmarks),
nvx-cli on Linux, and OpenVMM's CI of microsoft/openvmm#4996, which the
openvmm-ci gate checks. CI's run of this head,
https://github.com/microsoft/nvx/actions/runs/42657899579, passed, its
OpenVMM CI of the pinned revision job included. OpenVMM's run of the pin,
https://github.com/microsoft/openvmm/actions/runs/42650574781, passed every
job but the non-required run vmm-tests [x64-windows-amd-snp], in which two
Hyper-V OpenHCL TPM tests timed out (tpm138::..._snp_ak_cache and
tpm::..._snp_ak_pub_refresh), as other TPM tests of that job did on
microsoft/openvmm#4993 and microsoft/openvmm#4994; the change touches neither
OpenHCL nor the TPM.

Changes to existing tests

tests-modified: check_test_changes.py --base origin/dev lists 8 changes
to existing tests, all mechanical, and 11 added tests:

guest::time::decoders::tests: packets_decode_their_header_then_their_body,
packet_errors_name_the_invalid_field,
network_packets_carry_their_network_settings, and
packets_refuse_malformed_network_settings pass RestorePacket::parse its
new reader of the record, which reads nothing for packets without one.
guest::time::api::tests: the Restore::to helper,
requests_and_restores_carry_the_abi_values, and
a_restore_hands_over_its_seed_once_and_formats_none_of_it build a
Restore with the new activation field, None, and
requests_and_restores_carry_the_abi_values also checks
Restore::activation_record; restore_input becomes a wrapper of
restore_input_with, which can append a record, and returns the same input.

OpenVMM pin

Pull request: [DO NOT REVIEW] microvm: carry a clone's activation record in the restore packet openvmm#4996, pinned at
d501e50660136000ec93412aea78fdafb9cf391c, one commit on 6cb6ba87c, the
head of feature/microvm/dev, which NVX pinned before; no other
prerequisite.
Equivalence: d501e5066 is patch-identical (git patch-id --stable) to
2a061ea61, the same change with its fixups on ed8fc0713, which the
measurements and the first round above used.
Roadmap: position 233, after position 232 (6cb6ba87c); positions 132-233
stay unassigned.

Landing order

Land microsoft/openvmm#4996 first, by fast-forwarding feature/microvm/dev to
d501e50660136000ec93412aea78fdafb9cf391c (OpenVMM's merge button squashes,
which would leave the pin off the branch); then merge this pull request.

Pin microsoft/openvmm#4996's head, d501e50660136000ec93412aea78fdafb9cf391c, "microvm: carry a clone's activation record in the restore packet", one commit on feature/microvm/dev 6cb6ba87c, and decode what it adds (#544). A platform restore with --restore-activation-record now ends the restore packet with the activation record of its clone, which the launcher supplies: the packet's flag bit 5, ACTIVATION, which was reserved, then, after the DNS server, the record's length, a little-endian u32 from 1 to 1024, and its bytes. The packet stays version 1, since every guest that decodes version 1 refuses a packet that sets a reserved bit, so a packet without the flag keeps its layout and meaning, and a guest that predates the flag fails a packet with a record rather than ignore it. No contract version, and no template's compatibility key, changes. The guest's decoder takes the flag: a header with it adds the record's length to the body, and the packet takes the record's bytes, bounded by nvx::activation::MAX_RECORD_BYTES, in four-byte reads that start on a four-byte boundary, before the time sample, in the same port transaction. A length of 0 or above the bound fails repair, as every invalid field does. Restore::activation_record returns the bytes, undecoded, and nvx-time's test mode prints them. An ignored test of tests/openvmm_vectors.rs compares the bound with OpenVMM's MAX_RESTORE_ACTIVATION_LEN. Tests: decoder tests of records of 1, 3, 602, and 1024 bytes, of lengths of 0, 1025, and u32::MAX, of a body that ends within the length, and of the flags that stay reserved; a test of the port reads of a packet whose record ends with a whole and a partial read, after which the time sample still decodes; and test-mode tests of the decoded record and of its refused lengths. Existing tests pass the decoder's new reader of the record, and build restores with the new field. The vector and CPU-profile tests pass against the pin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The guide's restore packet (5.5.5) gains the ACTIVATION flag that the pinned OpenVMM defines: a platform restore that hands its clone an activation record with --restore-activation-record ends the packet with the record's length, a u32 from 1 to 1,024, and its bytes, which OpenVMM passes through without interpreting them, so the record carries no secret, and which cost the guest one port access per four bytes. It says why the packet stays version 1, gives the size that a record adds, and keeps ACK_REQUIRED on such a restore. Guest repair (5.5.7) validates the record's length, the compatibility model (1.5) names the record among the packet's contents, and the CI guide says that the ignored vector test also compares the record's bound with OpenVMM's (#544). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Maps NVX's new pin, d501e5066, the head of microsoft/openvmm#4996, at position 233, after position 232, 6cb6ba87c, the head of feature/microvm/dev, which microsoft/openvmm#4995 merged. The pin is 102 commits past the mapped tip, and positions 132-233 stay unassigned. Position 233 lets a platform restore end the restore packet with its clone's activation record: bit 5 of the packet's flags, which position 193 reserved when it restarted the packet at version 1, and --restore-activation-record, which the restore preparation of PR17's position 079 reads before any vCPU runs, only for the platform tier of PR19's position 093 and with the post-restore gate of PR20's position 095, and which the management RPC's conversion of position 228 takes too. It changes the restore packet of the time ABI series, which positions 132, 160, 181, and 193 add and change, and tests the portb device of PR15's position 060. Replaying it on upstream main conflicts in all nine of its files, since upstream main lacks the time ABI series, the microVM CLI and restore of PR15 and PR17, and the conversion of position 228, so it follows them upstream. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

🔵 Needs a closer look

Landing depends on unmerged OpenVMM #4996, verification of signed exact-pin provenance and exact-head CI, and maintainer confirmation of restore-ABI compatibility and reported platform validation.

No open issues 🎉

What changed in this PR

Adds guest-side support for carrying a platform clone’s activation record in the OpenVMM restore packet, advancing #544 while leaving Activate RPC replacement to a follow-up.

Changes:

Decodes bounded activation records within the packet’s port transaction and exposes their bytes through Restore::activation_record.
Adds decoder, port-read, guest-tool, and OpenVMM bound-consistency tests.
Documents the packet extension, version-1 compatibility, and new OpenVMM pin’s roadmap position.

File	Description
nvx-rs/tests/vectors/README.md	Documents the activation-bound consistency check.
nvx-rs/tests/openvmm_vectors.rs	Checks that NVX and OpenVMM share the activation-record bound.
nvx-rs/tests/guest_tools.rs	Tests activation-record output and invalid lengths.
nvx-rs/src/guest/time/test_mode.rs	Reads and prints activation bytes in test mode.
nvx-rs/src/guest/time/decoders/tests.rs	Tests record decoding, bounds, and reserved flags.
nvx-rs/src/guest/time/decoders.rs	Adds the activation flag, length validation, and record decoding.
nvx-rs/src/guest/time/api/tests.rs	Tests the accessor and record reads before time sampling.
nvx-rs/src/guest/time/api.rs	Reads activation bytes and exposes them on `Restore`.
nvx-rs/src/contract.rs	Documents the extension’s version-1 compatibility.
nvx-rs/README.md	Describes the expanded OpenVMM consistency checks.
nvx-rs/public-api/linux.txt	Records the new public accessor.
doc/openvmm-upstream-roadmap.md	Maps the new pin at position 233 and its dependencies.
doc/design/5.5-restore-protocol.md	Specifies activation layout, bounds, gating, and compatibility.
doc/design/1.5-compatibility-and-versioning-model.md	Includes activation records in the restore-packet contract description.
doc/ci.md	Documents CI verification of the shared activation bound.

_{Review effort: Balanced}

Pedro Henrique Penna (ppenna) and others added 3 commits November 9, 2026 08:24

Copilot AI balanced review requested due to automatic review settings November 9, 2026 16:45

Pedro Henrique Penna (ppenna) added the tests-modified Changes existing tests, test data, or test wiring label Nov 9, 2026

Pedro Henrique Penna (ppenna) mentioned this pull request Nov 9, 2026

[DO NOT REVIEW] microvm: carry a clone's activation record in the restore packet microsoft/openvmm#4996

Merged

Copilot started reviewing on behalf of Pedro Henrique Penna (ppenna) November 9, 2026 16:45

View session

Pedro Henrique Penna (ppenna) mentioned this pull request Nov 9, 2026

nvx-rs: replace activation RPC delivery with an uncaptured VMM configuration region #544

Open

4 tasks

Copilot AI reviewed Nov 9, 2026

View reviewed changes

Pedro Henrique Penna (ppenna) added this pull request to the merge queue Nov 9, 2026

Merged via the queue into dev with commit 8ada1a9 Nov 9, 2026
50 checks passed

Pedro Henrique Penna (ppenna) deleted the agents/issue-544-activation-packet branch November 9, 2026 17:43

Pedro Henrique Penna (ppenna) mentioned this pull request Nov 9, 2026

nvx-rs: activate a clone from its restore packet's record #1427

Merged

Navigation Menu

Repository navigation

openvmm: carry a clone's activation record in the restore packet - #1419

openvmm: carry a clone's activation record in the restore packet#1419
Pedro Henrique Penna (ppenna) merged 3 commits into
devfrom
agents/issue-544-activation-packet

Pedro Henrique Penna (ppenna) commented Nov 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Pedro Henrique Penna (ppenna) commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why

Measurements

Part 2

Commits

Validation

Changes to existing tests

OpenVMM pin

Landing order

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pedro Henrique Penna (ppenna) commented Nov 9, 2026 •

edited

Loading