feat(skillet): deploy verified binaries before remote apply

Author: gbagnoli

ublue/design/host-application.md

@@ -18,8 +18,16 @@ it. Caddy staging still performs scoped-token and DNS provider operations.
 Workstation SSH uses the operator's usual known-hosts file and OpenSSH identity
 selection unless overridden, while retaining strict host verification and
 noninteractive authentication. VM keys remain explicit run-owned artifacts.
-The workflow assumes installed guest binaries and temporary provisioning sudo;
-it does not deploy binaries, migrate data, finalize access or isolate controllers.
+Remote apply deploys current generic and caller binaries before vault access or
+service activation; `--no-deploy` explicitly reuses installed binaries. Standalone
+`deploy` has no vault or service side effects. Both use Cargo-reported artifacts
+and the VM delivery owner without manufacturing a VM identity. VM callers retain
+run locks and ownership guards. Delivery verifies identity, bytes, root metadata
+and version, and uses unique staging plus atomic replacement per executable.
+A failed partial pair stops apply and can be retried; pair-wide atomicity and
+concurrent independent deployment are not guaranteed. Temporary provisioning
+sudo is required. Deployment does not migrate data, finalize access or isolate
+controllers.
 
 
 Caller-selected native networking prepares boot profiles before monitoring,

ublue/plan/HOST-NETWORKING.md

@@ -31,3 +31,7 @@ apply prepares networking without switching the active connection. See
 Offline parser validation and injected tests do not establish physical wireless
 association or actual reboot reachability. Record evidence separately in
 `plan/archive/ACCEPTANCE.md`. Live switching/checkpoint rollback remains future work.
+
+Physical binary updates now use `skillet deploy --host HOST --target USER@HOST`;
+remote `apply` performs that deployment by default. Network-only delivery still
+requires running the guest base phase once to install the preparation unit.

ublue/plan/archive/ACCEPTANCE.md

@@ -2743,3 +2743,22 @@ update / rollback matrix cases. A header backup is not a service-data backup.
 - Formatting, strict workspace Clippy and 437 workspace/all-target tests passed
   (five routine ignores). Fedora joey base repeat apply passed with no repeated
   service start/restart. Physical binaries were not deployed by this change.
+
+## 2026-10-13 — production deployment before apply
+
+- Added `deploy --host HOST --target USER@HOST`; remote apply deploys first by
+  default, with `--no-deploy` restricted to remote use. Both paths print versions;
+  standalone deployment has no vault or service activation dependency.
+- Physical and VM updates share Cargo-reported artifact selection and verified
+  per-executable atomic delivery. VM run guards/capture evidence are preserved.
+  Uploaded bytes/root ownership/mode are checked before promotion, SELinux labels
+  are restored, and both installed hashes and versions are checked. A failed
+  partial pair prevents apply and retry converges the remaining executable.
+- Local checks passed: formatting, strict workspace Clippy, 445 workspace/all-target
+  tests (five routine ignores), deploy/apply CLI help, and Fedora joey base repeat
+  apply with no repeated service start/restart. Stateful fake transport tests
+  cover both files, corrupt staging, partial replacement/retry, no-op delivery,
+  profile mismatch before build/upload, and retained VM disposal guards.
+- Live deployment to physical clamps and native/Flatpak VM update were not run.
+  The last read-only SSH probe to the physical host timed out; mock delivery and
+  container baseline results do not establish real SSH/SELinux deployment acceptance.

ublue/skillet/AGENTS.md

@@ -232,3 +232,8 @@ skillet/
 - Keep release and source identity shared across generic and host binaries.
   Version metadata must refresh after Git or tracked-source changes without
   requiring a clean build; report unavailable provenance explicitly.
+
+- Remote application deploys its matching artifacts before consumer activation
+  unless reuse is explicit. Share artifact selection and verified replacement
+  across physical and disposable targets while retaining target-specific trust
+  guards; stop application after incomplete deployment.

ublue/skillet/README.md

@@ -70,8 +70,8 @@ root with `bash ublue/butane/tests/bootstrap.sh`.
    [secrets checklist](../../SECRETS.md). The
    [host/module index](crates/host-profiles/README.md) links each service guide.
    Stop conflicting production identities and DNS writers before migration.
-7. **Apply the host.** Installed guest binaries and units must match the selected
-   profile. From the workstation:
+7. **Apply the host.** Remote apply builds and deploys the current
+   binaries before configuring services. From the workstation:
 
    ```sh
    cargo run --release -p skillet -- secrets unlock

ublue/skillet/crates/cli/src/apply.rs

@@ -1,10 +1,7 @@
 use super::{SecretDeliverArgs, UiEnvironmentName};
 use anyhow::{anyhow, Context, Result};
 use skillet_cli_common::hosts::ApplyPhase;
-use skillet_vm::{
-    credential::ActivationPolicy,
-    transport::{GuestCommand, GuestTransport, SshTransport},
-};
+use skillet_vm::{credential::ActivationPolicy, transport::SshTransport};
 use skillet_workstation::{host_apply, vault::Vault};
 use std::path::PathBuf;
 
@@ -22,6 +19,9 @@ pub(super) struct Args {
     /// Provision all declared services over verified SSH to USER@HOST
     #[arg(long, requires = "host")]
     target: Option<String>,
+    /// Reuse installed guest binaries instead of deploying current source
+    #[arg(long, requires = "target")]
+    no_deploy: bool,
     #[arg(long, default_value_t = 22)]
     port: u16,
     /// SSH private key; omitted uses OpenSSH configuration, defaults and agent
@@ -88,17 +88,17 @@ fn remote(args: &Args) -> Result<()> {
         args.identity.as_deref(),
         args.known_hosts.as_deref(),
     )?;
-    let identity = transport.execute(
-        &GuestCommand {
-            program: "cat",
-            arguments: &["/etc/skillet/host"],
-        },
-        None,
-    )?;
-    if !identity.status.success() || std::str::from_utf8(&identity.stdout)?.trim() != host {
-        return Err(anyhow!(
-            "SSH host identity differs from selected host; no credentials delivered"
-        ));
+    skillet_workstation::deployment::verify_host(host, &transport)?;
+    if args.no_deploy {
+        println!(
+            "Workstation Skillet {}; deployment skipped (--no-deploy)",
+            skillet_cli_common::version::version()
+        );
+        let installed = skillet_workstation::deployment::installed_versions(host, &transport)?;
+        println!("Installed: {}", installed.generic_version);
+        println!("Installed: {}", installed.host_version);
+    } else {
+        super::deploy::deploy(host, &transport)?;
     }
     let path = args.database.clone().map_or_else(Vault::default_path, Ok)?;
     let mut vault = Vault::open(&path, args.key_file.as_deref())?;

ublue/skillet/crates/cli/src/apply/tests.rs

@@ -22,6 +22,7 @@ fn remote_apply_has_simple_defaults_and_production_policy() {
     assert_eq!(args.phase, ApplyPhase::Full);
     assert!(matches!(args.environment, UiEnvironmentName::Production));
     assert_eq!(args.port, 22);
+    assert!(!args.no_deploy);
     assert!(args.identity.is_none());
     assert!(args.known_hosts.is_none());
 }
@@ -98,3 +99,42 @@ fn invalid_remote_profile_or_policy_fails_before_connection_and_unlock() {
             .contains("all production services"));
     }
 }
+
+#[test]
+fn deployment_can_only_be_skipped_for_remote_apply() {
+    let args = parse(&[
+        "skillet",
+        "apply",
+        "--host",
+        "clamps",
+        "--target",
+        "admin@example.invalid",
+        "--no-deploy",
+    ]);
+    assert!(args.no_deploy);
+    assert!(
+        crate::Args::try_parse_from(["skillet", "apply", "--host", "clamps", "--no-deploy"])
+            .is_err()
+    );
+}
+
+#[test]
+fn standalone_deploy_requires_profile_and_target_and_has_ssh_defaults() {
+    assert!(crate::Args::try_parse_from(["skillet", "deploy", "--host", "clamps"]).is_err());
+    assert!(crate::Args::try_parse_from([
+        "skillet",
+        "deploy",
+        "--target",
+        "admin@example.invalid"
+    ])
+    .is_err());
+    assert!(crate::Args::try_parse_from([
+        "skillet",
+        "deploy",
+        "--host",
+        "clamps",
+        "--target",
+        "admin@example.invalid"
+    ])
+    .is_ok());
+}

ublue/skillet/crates/cli/src/deploy.rs

@@ -0,0 +1,44 @@
+use anyhow::{Context, Result};
+use skillet_vm::transport::GuestTransport;
+use std::path::PathBuf;
+
+#[derive(clap::Args, Debug)]
+pub(super) struct Args {
+    #[arg(long)]
+    host: String,
+    #[arg(long)]
+    target: String,
+    #[arg(long, default_value_t = 22)]
+    port: u16,
+    #[arg(long)]
+    identity: Option<PathBuf>,
+    #[arg(long)]
+    known_hosts: Option<PathBuf>,
+}
+
+pub(super) fn dispatch(args: &Args) -> Result<()> {
+    skillet_hosts::profile_for_name(&args.host)
+        .ok_or_else(|| anyhow::anyhow!("unknown host profile: {}", args.host))?;
+    let transport = super::ssh::transport(
+        &args.target,
+        args.port,
+        args.identity.as_deref(),
+        args.known_hosts.as_deref(),
+    )?;
+    deploy(&args.host, &transport)
+}
+
+pub(super) fn deploy(host: &str, guest: &impl GuestTransport) -> Result<()> {
+    println!(
+        "Workstation Skillet {}",
+        skillet_cli_common::version::version()
+    );
+    println!("Building and deploying skillet and skillet-{host}");
+    let report = skillet_workstation::deployment::deploy(host, &super::workspace_root()?, guest)
+        .context(
+        "deploying Skillet; apply has not started, retry deployment after resolving the failure",
+    )?;
+    println!("Deployed and verified: {}", report.generic_version);
+    println!("Deployed and verified: {}", report.host_version);
+    Ok(())
+}

ublue/skillet/crates/cli/src/main.rs

@@ -6,6 +6,7 @@ use tracing::{info, Level};
 use tracing_subscriber::FmtSubscriber;
 
 mod apply;
+mod deploy;
 mod installation;
 mod root_recovery;
 mod secret_delivery;
@@ -58,6 +59,8 @@ enum Commands {
     },
     /// Apply locally, or deliver credentials and apply all declared services over SSH
     Apply(apply::Args),
+    /// Build and deploy current generic and host binaries over verified SSH
+    Deploy(deploy::Args),
     /// Run the real-runtime smoke scenario against a disposable VM
     Test {
         #[command(subcommand)]
@@ -384,6 +387,7 @@ fn main() -> Result<()> {
             command: SecretCommands::Lock(args),
         } => secret_delivery::lock_vault(args.database.as_deref())?,
         Commands::Apply(args) => apply::dispatch(&args)?,
+        Commands::Deploy(args) => deploy::dispatch(&args)?,
         Commands::Test {
             command: TestCommands::Run(args),
         } => run_container_test(&args)?,

ublue/skillet/crates/vm/src/artifacts.rs

@@ -54,6 +54,35 @@ pub fn build(request: &BuildRequest<'_>) -> Result<BTreeMap<String, PathBuf>> {
         .collect()
 }
 
+/// Inspect a selected executable with a bounded read-only command.
+pub fn binary_version(path: &Path) -> Result<String> {
+    let mut command = Command::new(path);
+    command.arg("--version");
+    let output = crate::process::capture(command, std::time::Duration::from_secs(10))?;
+    if !output.status.success() {
+        return Err(Error::Invalid(
+            "selected binary could not report its version".into(),
+        ));
+    }
+    version_text(&output.stdout)
+}
+
+/// Version output is a single bounded printable line, safe for terminal display.
+pub fn version_text(bytes: &[u8]) -> Result<String> {
+    let text = std::str::from_utf8(bytes)
+        .map_err(|_| Error::Invalid("invalid binary version output".into()))?
+        .trim();
+    if text.is_empty()
+        || text.len() > 200
+        || !text
+            .bytes()
+            .all(|byte| byte.is_ascii_graphic() || byte == b' ')
+    {
+        return Err(Error::Invalid("invalid binary version output".into()));
+    }
+    Ok(text.into())
+}
+
 #[derive(Deserialize)]
 struct CargoMessage {
     reason: String,