Skip to content

Commit d6a8aec

Browse files
authored
Merge pull request #1248 from microsoft/agents/issue-549-openvmm-pin
openvmm: pin the microVM byte channel without host input
2 parents fa79978 + 09f03d0 commit d6a8aec

6 files changed

Lines changed: 73 additions & 52 deletions

File tree

‎doc/design/3.1-platform-control-device.md‎

Lines changed: 27 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ platform uses.
1010

1111
| Port | Function | Architectural purpose |
1212
| ---: | --- | --- |
13-
| `0xe9..0xeb` | Host/guest byte channel | Early console data, bounded host input, restore metadata, generation identity, and host time samples |
13+
| `0xe9..0xeb` | Host/guest byte channel | Early console data, restore metadata, generation identity, and host time samples |
1414
| `0x604` | Shutdown | Guest-requested process termination with a status byte |
1515
| `0x605` | Snapshot request | Guest-initiated capture and post-restore acknowledgement |
1616

@@ -26,8 +26,10 @@ machine omits, and carries:
2626

2727
- The guest's console. The kernel writes its boot messages to the channel
2828
from early in its initialization, before it probes any driver, and the
29-
channel is also the kernel console, with host input, unless a
30-
[boot console](3.3-consoles.md#331-boot-console) is configured.
29+
channel is also the kernel console unless a
30+
[boot console](3.3-consoles.md#331-boot-console) is configured. The channel
31+
takes no host input, so host input reaches the guest only through its
32+
[consoles](3.3-consoles.md).
3133
- Failure reports from the guest agents, including failures of the control
3234
console.
3335
- The one-time [restore packet](5.5-restore-protocol.md#555-restore-packet),
@@ -54,10 +56,10 @@ silently lost; see
5456

5557
The channel's protocol is also minimal: it needs no feature negotiation,
5658
queues in guest memory, or interrupt line. Each output byte is one port write,
57-
and the guest polls a status port for input. Every access to the channel's
58-
ports is a synchronous exit that the VMM completes before the guest continues,
59-
so a host time that the VMM records during an access lies between two guest
60-
TSC reads taken around it. The guest checks each
59+
and the guest reads records and status with port reads. Every access to the
60+
channel's ports is a synchronous exit that the VMM completes before the guest
61+
continues, so a host time that the VMM records during an access lies between
62+
two guest TSC reads taken around it. The guest checks each
6163
[time sample](3.2-clock-and-time.md#329-time-samples)'s pairing of host UTC
6264
with its TSC against them.
6365

@@ -68,7 +70,7 @@ port, a status and record-selector port, and a time-sample window.
6870

6971
| Port | Access | Behavior |
7072
| ---: | --- | --- |
71-
| `0xe9` | Read and write | Raw byte input and output. A console read consumes one pending byte and zero-fills the rest of the access. With a record selected, a 1-, 2-, or 4-byte read returns that many bytes of the record, little-endian and zero-filled past its end. |
73+
| `0xe9` | Read and write | Raw byte output. Without a selected record, a read returns zeros. With a record selected, a 1-, 2-, or 4-byte read returns that many bytes of the record, little-endian and zero-filled past its end. |
7274
| `0xea` | Read | Status bits, listed in Table 3.3 |
7375
| `0xea` | Write | Record selector: `0xa5` selects the one-time [restore packet](5.5-restore-protocol.md#555-restore-packet), `0xa6` selects the 16-byte generation ID, and `0xa7` latches a fresh [time sample](3.2-clock-and-time.md#329-time-samples) without changing the `0xe9` selection |
7476
| `0xeb` | Read | The next bytes of the latched time sample, zero-filled past its end or when no sample is latched |
@@ -79,7 +81,7 @@ Table 3.3 lists the status bits.
7981

8082
| Status bit | Meaning |
8183
| ---: | --- |
82-
| 0 | Host input is pending |
84+
| 0 | Zero: the channel takes no host input |
8385
| 1 | A fresh restore packet is available |
8486
| 2 | The restore carries a processor target |
8587
| 3 | The restore carries a memory target |
@@ -99,22 +101,22 @@ how a restored guest checks the ID, and
99101
[Entropy Refresh](5.5-restore-protocol.md#558-entropy-refresh) how it obtains
100102
fresh random numbers.
101103

102-
The time-sample window is separate from the data port because the guest
103-
console driver polls status and reads data from a timer context, so a sample
104-
on the console stream could interleave with input bytes. Guest users of the
105-
selectors serialize their transactions. Every byte the guest reads is a port
106-
exit, so records are small and readable four bytes per exit.
107-
108-
The receive and transmit buffers are each bounded at 1 MiB. Output overflow
109-
drops the newest bytes and emits a rate-limited warning; input applies
110-
backpressure by not reading the host endpoint while its buffer is full.
111-
Pending bytes are saved, so capture does not silently lose VMM-owned I/O. The
112-
time-sample window is not saved state.
113-
114-
While host input is gated for a snapshot boundary or post-restore repair, the
115-
channel stops reading its host endpoint, returns zero for console reads, and
116-
clears the input-pending bit. Guest output, the generation ID, any restore
117-
packet, and the time-sample window remain available.
104+
The time-sample window is separate from the data port, so that the guest can
105+
latch and read a sample without changing the data port's record selection.
106+
Guest users of the selectors serialize their transactions. Every byte the
107+
guest reads is a port exit, so records are small and readable four bytes per
108+
exit.
109+
110+
The transmit buffer is bounded at 1 MiB. Overflow drops the newest bytes and
111+
emits a rate-limited warning. Pending output is saved, so capture does not
112+
silently lose VMM-owned output, and a restore refuses saved state that holds
113+
host input, as a snapshot of an earlier VMM can. The time-sample window is not
114+
saved state.
115+
116+
Since the channel takes no host input, gating host input for a snapshot
117+
boundary or post-restore repair leaves it unchanged: guest output, the
118+
generation ID, any restore packet, and the time-sample window remain
119+
available.
118120

119121
## 3.1.2 Shutdown
120122

‎doc/design/3.3-consoles.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,11 @@ endpoints, saved state, and reconnect policies.
1515
## 3.3.1 Boot Console
1616

1717
The optional boot console carries boot messages and an ordinary byte stream.
18-
The kernel console uses it once its driver is ready, and the byte channel
19-
remains the early console. Host RX accepted by the device and partial guest TX
20-
progress are device-private saved state, so restore never replays a
18+
It is the only console for interactive host input, since the
19+
[byte channel](3.1-platform-control-device.md#311-hostguest-byte-channel)
20+
takes none. The kernel console uses it once its driver is ready, and the byte
21+
channel remains the early console. Host RX accepted by the device and partial
22+
guest TX progress are device-private saved state, so restore never replays a
2123
descriptor from byte zero or silently loses VMM-owned data.
2224

2325
The host attachment is one of:

‎doc/design/5.1-boot-protocol.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,8 @@ earlycon=xe9 console=hvc0 reboot=t panic=-1
5555
```
5656

5757
When the boot virtio-console is present, the kernel console becomes `hvc1`,
58-
and the byte channel remains the early console. User arguments follow the
58+
and the byte channel remains the early console. Without it, the kernel console
59+
stays on the byte channel, which takes no host input. User arguments follow the
5960
base tokens. A fresh boot then appends host-owned tokens in this order:
6061

6162
1. `nr_cpus=<capacity>`, so Linux sizes its processor state to the validated

‎doc/openvmm-upstream-roadmap.md‎

Lines changed: 32 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,20 @@
77
`b2676222b383bf50c7022f6a5983920421469a47`; see [Upstream status](#upstream-status).
88
- Mapped fork tip: `3cdf4fe3834cfb90b9ee685875493bc9bff1b7a4`, position 135. The graph, PR
99
summary, and commit mapping below cover positions 001-135 and the PR20 fixup at position 152.
10-
- NVX's pinned OpenVMM revision: `5aa8b028503237757b95f5629ebdbb4d56a67d51`, position 211.
11-
Position 211, which microsoft/openvmm#4894 proposes for `feature/microvm/dev`, is one commit on
12-
position 210, `eda13d80a`, the head of the `feature/microvm/release/v4.0` branch of
13-
microsoft/openvmm, to which `feature/microvm/dev` also points. The v4.0 reorganization moved the
14-
205 commits of `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto
15-
`c6d3da0ff`. It dropped the eight commits whose changes upstream had landed or superseded,
16-
retired the fallible device start that microsoft/nvx#1232 replaces with awaited device
17-
restoration, folded that restoration into the commits that it changes, and added 14 commits that
18-
give each ABI that the chain introduces a single version; see
19-
[The v4.0 reorganization](#the-v40-reorganization). The pin is 76 commits past the mapped tip on
20-
the same first-parent chain. Positions 136-211, except the PR20 fixup at position 152, are not
21-
yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip).
10+
- NVX's pinned OpenVMM revision: `4713c539ac989c5760415f8441bdb85ae9be3e82`, position 213.
11+
Position 213, which microsoft/openvmm#4887 proposes for `feature/microvm/dev`, is one commit on
12+
position 212, `de29849a4`, which microsoft/openvmm#4886 proposes as one commit on position 211,
13+
`5aa8b0285`, which microsoft/openvmm#4894 proposes as one commit on position 210, `eda13d80a`,
14+
the head of the `feature/microvm/release/v4.0` branch of microsoft/openvmm, to which
15+
`feature/microvm/dev` also points. The v4.0 reorganization moved the 205 commits of
16+
`feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto `c6d3da0ff`. It
17+
dropped the eight commits whose changes upstream had landed or superseded, retired the fallible
18+
device start that microsoft/nvx#1232 replaces with awaited device restoration, folded that
19+
restoration into the commits that it changes, and added 14 commits that give each ABI that the
20+
chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The
21+
pin is 78 commits past the mapped tip on the same first-parent chain. Positions 136-213, except
22+
the PR20 fixup at position 152, are not yet assigned to PRs; see
23+
[Commits after the mapped tip](#commits-after-the-mapped-tip).
2224
- Upstream main: `ca6b2b177da5e33ae1bcac72986b31d6b039811c` (2026-11-02), one commit past the
2325
fork base; see [Upstream main past the fork base](#upstream-main-past-the-fork-base).
2426

@@ -644,8 +646,8 @@ c6d3da0ff
644646
-> command lines [189] -> KVP test [190-191]
645647
-> sandbox block roles [192] -> virtio feature masks [193-194]
646648
-> IOAPIC interrupt lines [195-196] -> ABI restart [197-210]
647-
-> boot console exit drain [211]
648-
-> 5aa8b0285 (NVX pin)
649+
-> boot console exit drain [211] -> portb without host input [212-213]
650+
-> 4713c539a (NVX pin)
649651
```
650652

651653
For upstream review, PR01-PR07 and PR14 can be rebased independently onto current upstream main,
@@ -661,12 +663,12 @@ see [The v3.0 reorganization](#the-v30-reorganization).
661663

662664
## Commits after the mapped tip
663665

664-
The NVX pin extends the chain past position 135 without merges. Of the 76 later commits, position
666+
The NVX pin extends the chain past position 135 without merges. Of the 78 later commits, position
665667
152 is a PR20 fixup, and the others are not yet classified or assigned to PRs. Each of positions
666668
136-191 combines the v2.4 commits that the table lists, as v3.0 positions 145-200 did, which
667669
developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 192-196
668670
followed the v3.0 reorganization, positions 197-210 are the ABI restart of the v4.0
669-
reorganization, and position 211 followed it, so they combine none.
671+
reorganization, and positions 211-213 followed it, so they combine none.
670672

671673
| Position | Commit | v3.0 position | v2.4 positions | Topic |
672674
| ---: | --- | ---: | --- | --- |
@@ -746,6 +748,8 @@ reorganization, and position 211 followed it, so they combine none.
746748
| 209 | `31d684ae0` | - | - | microVM: drop the history from the layout and snapshot helpers |
747749
| 210 | `eda13d80a` | - | - | openvmm_helpers: number the snapshot's fields without gaps |
748750
| 211 | `5aa8b0285` | - | - | microVM: drain the boot console's terminal output at guest exit (microsoft/openvmm#4894) |
751+
| 212 | `de29849a4` | - | - | vmm_tests: drive the microVM guests without portb input (microsoft/openvmm#4886) |
752+
| 213 | `4713c539a` | - | - | microVM: remove portb's host input (microsoft/openvmm#4887) |
749753

750754
The remarks that say whether commits apply to upstream main record what the v3.0 reorganization
751755
found on its base, `b2676222b`, at the positions that the commits have now.
@@ -774,7 +778,15 @@ position 087 selects, to the IOAPIC input of its window, which changes the inter
774778
discovery tokens that PR17's position 078 records in the snapshot contract and that the slots of
775779
PR18, PR19, and PR23 use. Position 211 extends the portb exit drain of PR24's position 123 to the
776780
relay that copies the boot console of PR16's position 071 to the terminal, and gives the
777-
management RPC's drain of PR25's position 128 no such relay. Assigning the later
781+
management RPC's drain of PR25's position 128 no such relay. Position 212 makes the guests of the
782+
microVM cold-boot test and portb harness of PR15's position 069, of the check of PR17's position
783+
084 that an unconfigured snapshot request continues, and of the management RPC's microVM test of
784+
PR25's position 128 run without portb's host input, and removes the harness's input. Position 213
785+
then removes that input: the receive path of the portb console of PR15's position 064, which
786+
PR20's position 098 shares with the restore records and PR20's position 099 gates, and the
787+
console input that PR15's position 068 gives the device on the CLI. Since those reads also
788+
noticed a departed peer of a listening endpoint, position 213 has the device poll the backend for
789+
it instead. Assigning the later
778790
commits therefore means revisiting those PRs, not only appending new ones.
779791

780792
Position 182, by contrast, changes only `disk_blob`, which upstream main and the fork base
@@ -819,6 +831,9 @@ introduce those ABIs upstream.
819831
Position 211 changes only the microVM CLI's output drain and host consoles, the anonymous serial
820832
pair that they use, and the CLI guide: replaying it on upstream main, `ca6b2b177`, conflicts in
821833
five of its seven files, four of which upstream main lacks, so it follows those PRs upstream.
834+
Positions 212 and 213 change only the microVM portb device, its CLI and test harness, the VMM tests
835+
of PR15, PR17, and PR25, and their Guide pages: replaying them on upstream main conflicts in all
836+
nine of their files, six of which upstream main lacks, so they follow those PRs upstream.
822837

823838
## PR summary
824839

‎doc/run.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -377,11 +377,12 @@ Most `nvx-rs run` options pass through unchanged: `--machine`, `--processors`,
377377

378378
Always include `--single-process`. Include `--virtio-console console` to type
379379
into the guest's shell on a fresh boot: without a boot console, the guest's
380-
console is the byte channel, `hvc0`. A restore keeps the console of its
381-
snapshot. OpenVMM requires `--virtio-console console` again to restore a
382-
snapshot of a boot that passed it, and refuses `--virtio-console` for a
383-
snapshot without a boot console. `nvx-rs run` passes the option to a fresh boot
384-
only, so it cannot yet restore a snapshot of a boot with it
380+
console is the byte channel, `hvc0`, which takes no input, so OpenVMM discards
381+
the terminal's keystrokes, though Ctrl-Q still opens its REPL. A restore keeps
382+
the console of its snapshot. OpenVMM requires `--virtio-console console` again
383+
to restore a snapshot of a boot that passed it, and refuses `--virtio-console`
384+
for a snapshot without a boot console. `nvx-rs run` passes the option to a
385+
fresh boot only, so it cannot yet restore a snapshot of a boot with it
385386
([#1254](https://github.com/microsoft/nvx/issues/1254)). When restoring, omit
386387
`--memory`, `--kernel`, `--initrd`, and `--net`, whose addresses the snapshot
387388
keeps; see [Networked snapshots](#networked-snapshots). Every restore gives the

0 commit comments

Comments
 (0)