Merge pull request #1255 from microsoft/agents/issue-549-boot-console-exit-drain

openvmm: pin the boot console's exit drain

Author: ppenna

doc/ci.md

@@ -414,8 +414,10 @@ pair instead.
 The `console-exit` scenario delays host console reads for two seconds after
 snapshot restore to exercise output backpressure. For each requested processor
 count it requires byte-exact delivery of a 64 KiB payload and the final marker,
-and preserves guest exit statuses 0 and 37. This checks both device and host-relay
-draining without adding sleeps to the measured benchmark workloads.
+which the guest writes to the boot console before it exits with status 0 and
+to the byte channel before it exits with status 37, and preserves both
+statuses. This checks the draining of both consoles' devices and host relays
+without adding sleeps to the measured benchmark workloads.
 The harness waits for the output reader's EOF notification even after the
 process exits, so delayed final output chunks cannot create a false failure.
 

doc/design/3.3-consoles.md

@@ -25,7 +25,9 @@ The host attachment is one of:
 - A listener on a Unix socket, a named pipe, or loopback TCP with a fixed
   nonzero port, which retains pending guest output until a client connects.
 - A client connection.
-- The inherited terminal.
+- The inherited terminal, whose output a guest-requested exit or capture
+  writes before the process ends; see
+  [Final Output and Exit Status](5.6-shutdown-and-failure-semantics.md#561-final-output-and-exit-status).
 - An explicitly disconnected endpoint that discards guest output.
 
 The snapshot records the attachment's stable ID, canonical endpoint identity,

doc/design/5.6-shutdown-and-failure-semantics.md

@@ -11,14 +11,16 @@ after teardown reaches a known state.
 ## 5.6.1 Final Output and Exit Status
 
 Guest-requested exit drains the byte channel and, when present, its host
-output relay before reporting completion, within one combined five-second
-deadline. Success preserves the guest's exit status; an endpoint, relay, or
-timeout failure becomes a process-exit error instead of silently discarding
-final output. This guarantee covers the byte channel and its relay, not every
-console endpoint. Under the management RPC, a zero guest status completes a
-pending wait and stops the server cleanly, while a nonzero status or a failed
-drain fails the wait and ends the server with an error. After teardown, the
-VMM can write the bounded
+output relay before reporting completion, and writes the output that a
+[boot console](3.3-consoles.md#331-boot-console) on the host's standard
+streams accepted, within one combined five-second deadline. Success preserves
+the guest's exit status; an endpoint, relay, or timeout failure becomes a
+process-exit error instead of silently discarding final output. This guarantee
+covers the byte channel, its relay, and the boot console on the standard
+streams, not every console endpoint. Under the management RPC, a zero guest
+status completes a pending wait and stops the server cleanly, while a nonzero
+status or a failed drain fails the wait and ends the server with an error.
+After teardown, the VMM can write the bounded
 [outcome report](1.4-trust-boundaries.md#145-host-facing-outputs).
 
 As Table 5.9 shows, the guest reports its own fail-fast conditions through

doc/openvmm-upstream-roadmap.md

@@ -7,17 +7,18 @@
   `b2676222b383bf50c7022f6a5983920421469a47`; see [Upstream status](#upstream-status).
 - Mapped fork tip: `3cdf4fe3834cfb90b9ee685875493bc9bff1b7a4`, position 135. The graph, PR
   summary, and commit mapping below cover positions 001-135 and the PR20 fixup at position 152.
-- NVX's pinned OpenVMM revision: `eda13d80a85e49c16ed0500a9d4ae762ca810304`, position 210, the
-  head of the `feature/microvm/release/v4.0` branch of microsoft/openvmm, to which
-  `feature/microvm/dev` also points. The v4.0 reorganization moved the 205 commits of
-  `feature/microvm/release/v3.0` and NVX's previous pin, `35e76265f`, onto `c6d3da0ff`. It
-  dropped the eight commits whose changes upstream had landed or superseded, retired the fallible
-  device start that microsoft/nvx#1232 replaces with awaited device restoration, folded that
-  restoration into the commits that it changes, and added 14 commits that give each ABI that the
-  chain introduces a single version; see [The v4.0 reorganization](#the-v40-reorganization). The
-  pin is 75 commits past the mapped tip on the same first-parent chain. Positions 136-210, except
-  the PR20 fixup at position 152, are not yet assigned to PRs; see
-  [Commits after the mapped tip](#commits-after-the-mapped-tip).
+- NVX's pinned OpenVMM revision: `5aa8b028503237757b95f5629ebdbb4d56a67d51`, position 211.
+  Position 211, which microsoft/openvmm#4894 proposes for `feature/microvm/dev`, is one commit on
+  position 210, `eda13d80a`, the head of the `feature/microvm/release/v4.0` branch of
+  microsoft/openvmm, to which `feature/microvm/dev` also points. The v4.0 reorganization moved the
+  205 commits of `feature/microvm/release/v3.0` and the NVX pin before it, `35e76265f`, onto
+  `c6d3da0ff`. It dropped the eight commits whose changes upstream had landed or superseded,
+  retired the fallible device start that microsoft/nvx#1232 replaces with awaited device
+  restoration, folded that restoration into the commits that it changes, and added 14 commits that
+  give each ABI that the chain introduces a single version; see
+  [The v4.0 reorganization](#the-v40-reorganization). The pin is 76 commits past the mapped tip on
+  the same first-parent chain. Positions 136-211, except the PR20 fixup at position 152, are not
+  yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip).
 - Upstream main: `ca6b2b177da5e33ae1bcac72986b31d6b039811c` (2026-11-02), one commit past the
   fork base; see [Upstream main past the fork base](#upstream-main-past-the-fork-base).
 
@@ -643,7 +644,8 @@ c6d3da0ff
   -> command lines [189] -> KVP test [190-191]
   -> sandbox block roles [192] -> virtio feature masks [193-194]
   -> IOAPIC interrupt lines [195-196] -> ABI restart [197-210]
-  -> eda13d80a (NVX pin)
+  -> boot console exit drain [211]
+  -> 5aa8b0285 (NVX pin)
 ```
 
 For upstream review, PR01-PR07 and PR14 can be rebased independently onto current upstream main,
@@ -659,12 +661,12 @@ see [The v3.0 reorganization](#the-v30-reorganization).
 
 ## Commits after the mapped tip
 
-The NVX pin extends the chain past position 135 without merges. Of the 75 later commits, position
+The NVX pin extends the chain past position 135 without merges. Of the 76 later commits, position
 152 is a PR20 fixup, and the others are not yet classified or assigned to PRs. Each of positions
 136-191 combines the v2.4 commits that the table lists, as v3.0 positions 145-200 did, which
 developed one concept; "(part)" marks a v2.4 fixup that split across commits. Positions 192-196
-followed the v3.0 reorganization, and positions 197-210 are the ABI restart of the v4.0
-reorganization, so they combine none.
+followed the v3.0 reorganization, positions 197-210 are the ABI restart of the v4.0
+reorganization, and position 211 followed it, so they combine none.
 
 | Position | Commit | v3.0 position | v2.4 positions | Topic |
 | ---: | --- | ---: | --- | --- |
@@ -743,6 +745,7 @@ reorganization, so they combine none.
 | 208 | `bec51b3ac` | - | - | vmservice: give the microVM fields upstream-compatible numbers |
 | 209 | `31d684ae0` | - | - | microVM: drop the history from the layout and snapshot helpers |
 | 210 | `eda13d80a` | - | - | openvmm_helpers: number the snapshot's fields without gaps |
+| 211 | `5aa8b0285` | - | - | microVM: drain the boot console's terminal output at guest exit (microsoft/openvmm#4894) |
 
 The remarks that say whether commits apply to upstream main record what the v3.0 reorganization
 found on its base, `b2676222b`, at the positions that the commits have now.
@@ -769,7 +772,9 @@ inputs from the loader of PR15's position 066. Position 196 moves every virtio s
 IRQ that PR16's position 071 gives it, and the NIC from the backend-specific IRQ that PR18's
 position 087 selects, to the IOAPIC input of its window, which changes the interrupts and
 discovery tokens that PR17's position 078 records in the snapshot contract and that the slots of
-PR18, PR19, and PR23 use. Assigning the later
+PR18, PR19, and PR23 use. Position 211 extends the portb exit drain of PR24's position 123 to the
+relay that copies the boot console of PR16's position 071 to the terminal, and gives the
+management RPC's drain of PR25's position 128 no such relay. Assigning the later
 commits therefore means revisiting those PRs, not only appending new ones.
 
 Position 182, by contrast, changes only `disk_blob`, which upstream main and the fork base
@@ -811,6 +816,10 @@ saved states of virtio-console, virtio-fs, and the RTC, the egress policy digest
 identity, the virtio-fs slot, and the management RPC's microVM fields, so they follow the PRs that
 introduce those ABIs upstream.
 
+Position 211 changes only the microVM CLI's output drain and host consoles, the anonymous serial
+pair that they use, and the CLI guide: replaying it on upstream main, `ca6b2b177`, conflicts in
+five of its seven files, four of which upstream main lacks, so it follows those PRs upstream.
+
 ## PR summary
 
 Each count is of v4.0 commits, which include the fixups that the v3.0 reorganization folded into

openvmm

@@ -1,5 +1,6 @@
 #!/bin/sh
 set -eu
+exec >@CONSOLE@
 head -c @PAYLOAD_BYTES@ /dev/zero | tr '\000' x
 printf '\n@COMPLETION_MARKER@\n'
 /sbin/nvx-exit @EXIT_CODE@

scripts/nvx_tools/microvm_test_scripts/console-exit.sh.in

@@ -348,6 +348,10 @@ class MicrovmVirtioDevice(NamedTuple):
 CONSOLE_EXIT_COMPLETION_MARKER = b"NVX-CONSOLE-EXIT-OK"
 CONSOLE_EXIT_PAYLOAD_BYTES = 64 * 1024
 CONSOLE_EXIT_READ_DELAY_SECONDS = 2.0
+# Each console-exit run's guest exit status and the console that carries its
+# payload: the boot console, on OpenVMM's standard output, or the byte channel,
+# on its standard error.
+CONSOLE_EXIT_RUNS = ((0, "hvc1"), (37, "hvc0"))
 ENDPOINT_POLICY = ("10.0.0.9:8443", "192.0.2.7:443", "10.0.0.9:443")
 ENDPOINT_POLICY_BEFORE_MARKER = b"NVX-ENDPOINT-POLICY-BEFORE"
 ENDPOINT_POLICY_AFTER_MARKER = b"NVX-ENDPOINT-POLICY-AFTER"
@@ -1606,15 +1610,22 @@ def run_console_exit(
     timeout: float,
     output_dir: Path,
 ) -> None:
+    """Check that a guest-requested exit keeps the guest's last output and its
+    exit status.
+
+    Each run restores a guest that writes a payload and a marker to one of its
+    consoles and then exits, while the host delays its reads, so that OpenVMM's
+    relay of that console lags behind the guest. The exit must write the whole
+    output of the boot console and of the byte channel alike."""
     expected = (
         b"x" * CONSOLE_EXIT_PAYLOAD_BYTES
         + b"\n"
         + CONSOLE_EXIT_COMPLETION_MARKER
         + b"\n"
     )
     with tempfile.TemporaryDirectory(prefix="nvx-console-exit-") as temporary:
-        for exit_code in (0, 37):
-            name = f"console-exit-{processors}vcpu-status-{exit_code}"
+        for exit_code, console in CONSOLE_EXIT_RUNS:
+            name = f"console-exit-{processors}vcpu-{console}-status-{exit_code}"
             snapshot_path = Path(temporary) / f"snapshot-{exit_code}"
             boot_command = workload_boot_command(
                 executable,
@@ -1625,23 +1636,27 @@ def run_console_exit(
                 "quiet loglevel=0",
                 processors=processors,
             )
+            boot_command.extend(("--virtio-console", "console"))
             capture_snapshot(
                 [*boot_command, "--snapshot-destination", str(snapshot_path)],
                 snapshot_path,
                 timeout=timeout,
                 processors=processors,
                 post_restore_script=_render_script(
                     "console-exit.sh.in",
+                    CONSOLE=f"/dev/{console}",
                     PAYLOAD_BYTES=str(CONSOLE_EXIT_PAYLOAD_BYTES),
                     COMPLETION_MARKER=CONSOLE_EXIT_COMPLETION_MARKER.decode(),
                     EXIT_CODE=str(exit_code),
                 ),
                 log_path=output_dir / f"{name}-capture.log",
             )
+            restore_command = snapshot_restore_command(
+                executable, backend, snapshot_path, processors=processors
+            )
+            restore_command.extend(("--virtio-console", "console"))
             with OpenvmmProcess(
-                snapshot_restore_command(
-                    executable, backend, snapshot_path, processors=processors
-                ),
+                restore_command,
                 output_dir / f"{name}-restore.log",
                 output_read_delay=CONSOLE_EXIT_READ_DELAY_SECONDS,
             ) as process:

scripts/nvx_tools/microvm_tests.py

@@ -5770,8 +5770,15 @@ def test_console_exit_preserves_full_output_and_guest_status(self):
 
         self.assertEqual(capture.call_count, 2)
         self.assertEqual(process.call_count, 2)
-        for call, code in zip(capture.call_args_list, (0, 37), strict=True):
+        # The boot console carries the first run's output, and the byte channel
+        # the second's.
+        for call, (code, console) in zip(
+            capture.call_args_list, ((0, "hvc1"), (37, "hvc0")), strict=True
+        ):
             self.assertEqual(call.kwargs["processors"], 2)
+            self.assertIn(
+                f"\nexec >/dev/{console}\n", call.kwargs["post_restore_script"]
+            )
             self.assertIn("head -c 65536", call.kwargs["post_restore_script"])
             self.assertIn(
                 f"/sbin/nvx-exit {code}\n", call.kwargs["post_restore_script"]
@@ -5780,6 +5787,10 @@ def test_console_exit_preserves_full_output_and_guest_status(self):
             self.assertEqual(call.kwargs["output_read_delay"], 2.0)
             self.assertIn("--restore-snapshot", call.args[0])
             self.assertEqual(call.args[0][call.args[0].index("--processors") + 1], "2")
+        for call in (*capture.call_args_list, *process.call_args_list):
+            command = call.args[0]
+            self.assertEqual(command.count("--virtio-console"), 1)
+            self.assertEqual(command[command.index("--virtio-console") + 1], "console")
 
     def test_console_exit_rejects_truncation_even_if_marker_survives(self):
         marker = b"\n" + microvm_tests.CONSOLE_EXIT_COMPLETION_MARKER + b"\n"