microVM: drain the boot console's terminal output at guest exit

A guest-requested exit drained portb's output and its host relay, and
then OpenVMM terminated the process. When --virtio-console console
attaches the boot console to the terminal, its relay copies the
anonymous pair to stdout on a thread of its own, which nothing awaited,
so the process could end before the relay wrote the output that the
guest had printed last. The guest's virtio-console driver waits until
the device completes each of its writes, and the device writes each
descriptor to the pair before completing it, so the output was already
in the pair; only the relay, waiting for the reactor to report it, was
behind. microsoft/nvx#549 moves NVX's interactive and test consoles to
the boot console, where a benchmark lost its completion marker at
nvx-exit.

The boot console's relay now takes drain requests. It holds a
non-blocking duplicate of its end of the pair, from which it reads what
the pair holds without waiting for the reactor, and writes and flushes
it before it answers. MicrovmOutputDrain asks it at a guest-requested
exit, after portb, within the same five-second deadline, and before a
guest-requested capture, whose source process ends once the snapshot
commits. Without a boot console on the terminal, nothing changes.

Tests: the exit drain writes the output that the reactor has yet to
report, which it lost before this change; the capture flush writes the
pending output and the relay keeps copying; a failed write fails the
drain; and on both Linux sockets and Windows pipes, the duplicate reads
a pair until it is empty without waiting for more.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: ppenna

Guide/src/reference/openvmm/management/cli.md

@@ -1023,7 +1023,8 @@ Serial devices can be configured to appear as different devices inside the guest
     restore command must explicitly resupply the matching client attachment.
   * `console`: require the restore caller to supply the same inherited terminal
     attachment. Portb recovery output moves to stderr while the terminal is
-    attached to `hvc1`.
+    attached to `hvc1`. A guest-requested exit writes the boot console output
+    that the device accepted before OpenVMM exits.
   * `none`: keep the device present and explicitly discard guest TX while
     disconnected.
 
@@ -1144,13 +1145,17 @@ a supervisor can tell the exit reasons apart.
 For microVMs, a process-status shutdown through port `0x604`, or a power event
 configured to exit, drains previously accepted portb console output before the
 process exits. This includes the host stdout/stderr relay, not just the device's
-transmit buffer. Draining is bounded to five seconds; an output error or timeout
+transmit buffer. When `--virtio-console console` attaches the boot console to
+the terminal, the drain also writes the boot console output that the device
+accepted before the shutdown, which the host relay has yet to write to stdout.
+Draining is bounded to five seconds; an output error or timeout
 is reported as a failure rather than a successful exit with truncated output.
 In RPC-server mode, a drain failure also fails pending `WaitVm` requests and
 terminates the server with an error instead of leaving it uninitialized.
 A guest-requested snapshot capture instead flushes accepted portb output to its
-endpoint before quiescing, without closing it because the capture can still roll
-back; see `--snapshot-destination`.
+endpoint, and the boot console's terminal output, before quiescing, without
+closing them because the capture can still roll back; see
+`--snapshot-destination`.
 
 * `--crash-dump-path <PATH>`: when the guest triple-faults, write a
   WinDbg-compatible `.vmrs` dump of the VM's processor state and guest memory to

openvmm/openvmm_entry/Cargo.toml

@@ -126,6 +126,7 @@ workspace = true
 features = [
   "Win32_Foundation",
   "Win32_Storage_FileSystem",
+  "Win32_System_Pipes",
 ]
 
 [build-dependencies]

openvmm/openvmm_entry/src/microvm/config.rs

@@ -22,6 +22,8 @@ use super::network::microvm_network_endpoint;
 use super::network::microvm_restore_network;
 use super::output::MicrovmOutputDrain;
 use super::output::OutputCompletion;
+use super::output::RelayDrain;
+use super::output::spawn_drainable_output;
 use super::output::spawn_output;
 use super::restore::fresh_microvm_generation_id;
 use crate::ConsoleState;
@@ -78,6 +80,8 @@ pub(crate) struct MicrovmConfigBuilder<'a> {
     control_broker_config: Option<VirtioControlConsoleBrokerConfig>,
     control_console_backend: Option<Resource<SerialBackendHandle>>,
     portb: Option<(Resource<SerialBackendHandle>, OutputCompletion)>,
+    /// The drain of the boot console's relay to the terminal.
+    console_relay: Option<RelayDrain>,
     resources: MicrovmResources,
 }
 
@@ -224,6 +228,7 @@ impl<'a> MicrovmConfigBuilder<'a> {
             control_broker_config,
             control_console_backend: None,
             portb: None,
+            console_relay: None,
             resources,
         })
     }
@@ -264,13 +269,14 @@ impl<'a> MicrovmConfigBuilder<'a> {
         } else {
             SerialConfigCli::Console
         };
-        self.portb = Some(setup_host_console(
+        let (backend, output, _) = setup_host_console(
             "microvm-portb",
             backend,
             "hvc0",
             console_state,
             serial_driver,
-        )?);
+        )?;
+        self.portb = Some((backend, output));
         Ok(())
     }
 
@@ -320,13 +326,14 @@ impl<'a> MicrovmConfigBuilder<'a> {
                     )?)
                 }
                 SerialConfigCli::Console => {
-                    let (backend, _) = setup_host_console(
+                    let (backend, _, relay) = setup_host_console(
                         "virtio-console",
                         SerialConfigCli::Console,
                         "hvc1",
                         console_state,
                         serial_driver,
                     )?;
+                    self.console_relay = relay;
                     Some(backend)
                 }
                 SerialConfigCli::None => Some(DisconnectedSerialBackendHandle.into_resource()),
@@ -407,7 +414,8 @@ impl<'a> MicrovmConfigBuilder<'a> {
             return Ok(());
         };
         let opt = self.opt;
-        let (drain, output_drain) = MicrovmOutputDrain::new(Some(output_completion));
+        let (drain, output_drain) =
+            MicrovmOutputDrain::new(Some(output_completion), self.console_relay.take());
         self.resources.output_drain = Some(drain);
         let (generation_id, time_abi_restore) = if self.restore.machine_contract.is_some() {
             let network = if self.restore.network_settings {
@@ -856,32 +864,39 @@ fn aggregate_child_kind(
 }
 
 /// Connects a portb or boot virtio-console endpoint to the host console or
-/// stderr, returning its backend and output-relay completion.
+/// stderr, returning its backend, its output-relay completion, and, for the
+/// console, the drain of its relay.
 fn setup_host_console(
     name: &str,
     backend: SerialConfigCli,
     device: &'static str,
     console_state: &RefCell<Option<ConsoleState<'static>>>,
     serial_driver: &DefaultDriver,
-) -> anyhow::Result<(Resource<SerialBackendHandle>, OutputCompletion)> {
+) -> anyhow::Result<(
+    Resource<SerialBackendHandle>,
+    OutputCompletion,
+    Option<RelayDrain>,
+)> {
     Ok(match backend {
         SerialConfigCli::Console => {
             if let Some(console_state) = console_state.borrow().as_ref() {
                 bail!("console already set by {}", console_state.device);
             }
-            let (config, serial) = serial_io::anonymous_serial_pair(serial_driver)?;
+            let (config, serial, pending) =
+                serial_io::anonymous_serial_pair_with_pending(serial_driver)?;
             let (serial_read, serial_write) = AsyncReadExt::split(serial);
             *console_state.borrow_mut() = Some(ConsoleState {
                 device,
                 input: Box::new(serial_write),
             });
-            let completed = spawn_output(name, serial_read, term::raw_stdout())?;
-            (config, completed)
+            let (completed, relay) =
+                spawn_drainable_output(name, serial_read, pending, term::raw_stdout())?;
+            (config, completed, Some(relay))
         }
         SerialConfigCli::Stderr => {
             let (config, serial) = serial_io::anonymous_serial_pair(serial_driver)?;
             let completed = spawn_output(name, serial, term::raw_stderr())?;
-            (config, completed)
+            (config, completed, None)
         }
         _ => unreachable!("microVM host consoles use the console or stderr"),
     })