fix(builtins): markkit/fetchkit respect the exec deadline off the async worker (#2870)

The markkit and fetchkit CLIs run on the blocking pool, and their timeouts are capped at the remaining exec budget, so a large document can no longer outlive the exec timeout or stall co-tenants on the worker (TM-DOS-143).

Author: chaliy

crates/bashkit/docs/threat-model.md

@@ -35,6 +35,7 @@ through configurable limits.
 | Pagination amplification (TM-DOS-137) | Long `pr -h` headers repeated on many short pages | Check the 16 MiB cap before every append; lease input, rows and output capacity before growth; borrow cells/page slices and retain date timestamps; charge input/work and poll cancellation/deadlines, including skipped pages | MITIGATED |
 | Embedded CLI limit escalation (TM-DOS-139) | `markkit extract x --max-input-size 99999999999` | `markkit` limits can only be lowered below their defaults; `fetchkit` bodies capped by HTTP limits; `--ocr` refused | MITIGATED |
 | Tree-walk memory (TM-DOS-140) | `find`, `du` or `rg` over a tree with millions of files | `FileSystem::walk` answers are bounded batches (10k entries) refilled as `find`/`du` descend; `rg` searches each file as it reads it without holding the text, charges it to the aggregate-input budget, stops at the first refusal and streams its walk; `ls -R` stops at the stdout limit | MITIGATED |
+| Extraction outlives deadline (TM-DOS-143) | `markkit extract` on a huge document under a short exec timeout | markkit/fetchkit run off the async worker; their timeouts are capped at the remaining exec budget | MITIGATED |
 | Root real mount drops limits (TM-DOS-144) | Writing past `filesystem_limits` with a host dir mounted at `/` | The root overlay keeps the configured `FsLimits` | MITIGATED |
 | Variable explosion (TM-DOS-003) | `x=$(cat /dev/urandom)` | /dev/urandom returns bounded 8KB | MITIGATED |
 | Array growth (TM-DOS-004) | `arr+=(element)` in loop | Command limit | MITIGATED |

crates/bashkit/src/builtins/fetchkit.rs

@@ -20,6 +20,10 @@
 //!   (bashkit signs in `HttpClient`; a key in argv would double-sign and land in
 //!   history), `--allow-remote-conversion` (uploads documents to a third
 //!   party), and the `mcp` subcommand. All still parse and show in `--help`.
+//! - **Off the async worker (TM-DOS-143).** HTML-to-Markdown conversion is
+//!   synchronous CPU work; the run happens on tokio's blocking pool so the
+//!   `exec` timeout can cancel the caller and co-tenants keep the worker. Per
+//!   request, the transport timeout is capped at the remaining `exec` budget.
 //! - **Stateless.** No globals; every run builds its own fetchkit `Tool`.
 
 use std::io;
@@ -48,9 +52,15 @@ impl Builtin for Fetchkit {
             stderr: Mutex::new(String::new()),
             env: ctx.env.clone(),
             http: ctx.http_client.cloned(),
+            deadline: ctx
+                .remaining_deadline()
+                .map(|left| crate::time_compat::Instant::now() + left),
         };
-        let args = std::iter::once("fetchkit".to_string()).chain(ctx.args.iter().cloned());
-        let exit_code = fetchkit_cli::run(args, &host).await;
+        let args: Vec<String> = std::iter::once("fetchkit".to_string())
+            .chain(ctx.args.iter().cloned())
+            .collect();
+        let host = Arc::new(host);
+        let exit_code = run_off_worker(args, host.clone()).await;
         Ok(ExecResult {
             stdout: take(&host.stdout).into(),
             stderr: take(&host.stderr).into(),
@@ -78,6 +88,19 @@ struct Host {
     stderr: Mutex<String>,
     env: std::collections::HashMap<String, String>,
     http: Option<Arc<HttpClient>>,
+    /// End of the `exec` wall-clock budget, if limited.
+    deadline: Option<crate::time_compat::Instant>,
+}
+
+/// Runs the CLI on tokio's blocking pool (TM-DOS-143). `Handle::block_on`
+/// drives the transport there; on a current-thread runtime the caller, parked
+/// awaiting the join handle, keeps driving IO and timers.
+async fn run_off_worker(args: Vec<String>, host: Arc<Host>) -> i32 {
+    let Ok(handle) = tokio::runtime::Handle::try_current() else {
+        return fetchkit_cli::run(args, &*host).await;
+    };
+    let job = tokio::task::spawn_blocking(move || handle.block_on(fetchkit_cli::run(args, &*host)));
+    job.await.unwrap_or(1)
 }
 
 #[async_trait]
@@ -106,6 +129,7 @@ impl CliHost for Host {
         builder
             .transport(Arc::new(SandboxTransport {
                 http: self.http.clone(),
+                deadline: self.deadline,
             }))
             .block_private_ips(false)
             .use_env_proxy(false)
@@ -121,6 +145,7 @@ impl CliHost for Host {
 /// fetchkit follows redirects itself, so every hop is re-checked here.
 struct SandboxTransport {
     http: Option<Arc<HttpClient>>,
+    deadline: Option<crate::time_compat::Instant>,
 }
 
 #[async_trait]
@@ -139,7 +164,15 @@ impl HttpTransport for SandboxTransport {
             TransportMethod::Head => Method::Head,
         };
         // bashkit clamps the deadline to its own bounds; round sub-second up.
-        let timeout_secs = req.timeout.map(|t| t.as_secs().max(1));
+        let left = self.deadline.map(|end| {
+            end.checked_duration_since(crate::time_compat::Instant::now())
+                .unwrap_or_default()
+        });
+        let timeout = match (req.timeout, left) {
+            (Some(t), Some(left)) => Some(t.min(left)),
+            (t, left) => t.or(left),
+        };
+        let timeout_secs = timeout.map(|t| t.as_secs().max(1));
         let response = http
             .request_with_timeout(method, req.url.as_str(), None, &req.headers, timeout_secs)
             .await

crates/bashkit/src/builtins/markkit.rs

@@ -19,6 +19,12 @@
 //! - **Limits only go down.** `--max-*`/`--timeout` flags may lower markkit's
 //!   default limits but never raise them (`Host::limits`), so one tenant's
 //!   script cannot buy a larger CPU/memory budget from a shared process.
+//! - **Deadline (TM-DOS-143).** Extraction is synchronous CPU work. Inline
+//!   it held the tenant's async worker (starving co-tenants scheduled there)
+//!   and the `exec` timeout could not interrupt it. The run now happens on
+//!   tokio's blocking pool, so the caller's future stays cancellable and the
+//!   worker stays free, and markkit's cooperative `timeout` is capped at the
+//!   remaining `exec` deadline so abandoned work stops soon after.
 //! - **Refused in the sandbox:** `--ocr` (runs a host Tesseract binary, reads
 //!   host model files) and the `mcp` subcommand (long-running stdio server).
 //!   Both still parse and show in `--help`; they fail with exit 1.
@@ -51,9 +57,15 @@ impl Builtin for Markkit {
             cwd: ctx.cwd.clone(),
             #[cfg(feature = "http_client")]
             http: ctx.http_client.cloned(),
+            deadline: ctx
+                .remaining_deadline()
+                .map(|left| crate::time_compat::Instant::now() + left),
         };
-        let args = std::iter::once("markkit".to_string()).chain(ctx.args.iter().cloned());
-        let exit_code = markkit_cli::run(args, &host).await;
+        let args: Vec<String> = std::iter::once("markkit".to_string())
+            .chain(ctx.args.iter().cloned())
+            .collect();
+        let host = Arc::new(host);
+        let exit_code = run_off_worker(args, host.clone()).await;
         Ok(ExecResult {
             stdout: host.take_stdout().into(),
             stderr: host.take_stderr().into(),
@@ -71,6 +83,18 @@ impl Builtin for Markkit {
     }
 }
 
+/// Runs the CLI on tokio's blocking pool (TM-DOS-143). `Handle::block_on`
+/// drives the host's async VFS/HTTP calls there; on a current-thread runtime
+/// the caller, parked awaiting the join handle, keeps driving IO and timers.
+async fn run_off_worker(args: Vec<String>, host: Arc<Host>) -> i32 {
+    let Ok(handle) = tokio::runtime::Handle::try_current() else {
+        return markkit_cli::run(args, &*host).await;
+    };
+    let job = tokio::task::spawn_blocking(move || handle.block_on(markkit_cli::run(args, &*host)));
+    // A markkit panic that unwinds reports as a plain failure.
+    job.await.unwrap_or(1)
+}
+
 /// [`CliHost`] over one builtin invocation.
 struct Host {
     stdout: Mutex<String>,
@@ -80,6 +104,9 @@ struct Host {
     cwd: PathBuf,
     #[cfg(feature = "http_client")]
     http: Option<Arc<crate::network::HttpClient>>,
+    /// End of the `exec` wall-clock budget, if limited. An instant, not a
+    /// duration, so time queued behind a busy blocking pool still counts.
+    deadline: Option<crate::time_compat::Instant>,
 }
 
 impl Host {
@@ -148,7 +175,11 @@ impl CliHost for Host {
     }
 
     fn limits(&self, requested: Limits) -> Limits {
-        clamp_limits(requested, &Limits::default())
+        let left = self.deadline.map(|end| {
+            end.checked_duration_since(crate::time_compat::Instant::now())
+                .unwrap_or_default()
+        });
+        clamp_limits(requested, &deadline_cap(left))
     }
 
     /// Fail closed: every host-only feature (today `--ocr`) is refused.
@@ -172,6 +203,15 @@ fn to_io_error(err: Error) -> io::Error {
     }
 }
 
+/// markkit's defaults, with `timeout` capped at the remaining `exec` budget.
+fn deadline_cap(deadline: Option<std::time::Duration>) -> Limits {
+    let mut cap = Limits::default();
+    if let Some(remaining) = deadline {
+        cap.timeout = cap.timeout.min(remaining);
+    }
+    cap
+}
+
 /// Each limit is the smaller of what the flags asked for and the cap.
 fn clamp_limits(requested: Limits, cap: &Limits) -> Limits {
     let mut l = requested;
@@ -298,6 +338,16 @@ mod tests {
         assert_eq!(got, cap);
     }
 
+    #[test]
+    fn timeout_capped_at_remaining_deadline() {
+        let remaining = std::time::Duration::from_millis(1500);
+        let got = clamp_limits(Limits::default(), &deadline_cap(Some(remaining)));
+        assert_eq!(got.timeout, remaining);
+        // No deadline: markkit's default stands.
+        let got = clamp_limits(Limits::default(), &deadline_cap(None));
+        assert_eq!(got.timeout, Limits::default().timeout);
+    }
+
     #[test]
     fn clamp_keeps_lowered_limits() {
         let cap = Limits::default();

crates/bashkit/tests/integration/markkit_builtin_tests.rs

@@ -251,3 +251,35 @@ mod network {
         assert_eq!(site.calls.lock().unwrap().len(), 6);
     }
 }
+
+// THREAT[TM-DOS-143]: extraction runs synchronously where the exec timeout
+// cannot cancel it, so markkit's own timeout is capped at the remaining
+// exec budget. Without the cap this ran to markkit's 60 s default.
+#[tokio::test]
+async fn extraction_stops_at_exec_deadline() {
+    use bashkit::ExecutionLimits;
+    use std::time::{Duration, Instant};
+
+    let mut bash = Bash::builder()
+        .limits(ExecutionLimits::new().timeout(Duration::from_millis(300)))
+        .build();
+    let page = "<div><p>word <b>bold</b> <a href=\"/x\">link</a></p></div>".repeat(170_000);
+    bash.fs()
+        .write_file(std::path::Path::new("/big.html"), page.as_bytes())
+        .await
+        .unwrap();
+    let started = Instant::now();
+    let r = bash
+        .exec("markkit extract --file /big.html > /dev/null")
+        .await;
+    let elapsed = started.elapsed();
+    assert!(
+        elapsed < Duration::from_secs(3),
+        "markkit ran {elapsed:?} past a 300 ms exec timeout ({r:?})" // debug-ok: test diagnostics
+    ); // The timeout itself must have fired, not a fast success.
+    let timed_out = match &r {
+        Ok(r) => r.exit_code == 124,
+        Err(e) => e.to_string().contains("timed out") || e.to_string().contains("timeout"),
+    };
+    assert!(timed_out, "exec did not time out: {r:?}"); // debug-ok: test diagnostics
+}

knowledge/log.md

@@ -2,6 +2,7 @@
 
 ## 2026-11-06
 
+* **markkit/fetchkit honor the exec deadline**: a security audit found `markkit extract` on a large document ran synchronously on the tenant's async worker, so a 300 ms `exec` timeout returned after 7+ s and co-tenants on that worker stalled. Both builtins now run on tokio's blocking pool, and their timeouts are capped at the remaining `exec` budget (302 ms in the regression test). See [markkit/fetchkit builtins](runtimes/markkit-fetchkit-builtins.md), TM-DOS-143.
 * **`json.dumps` indent preflight**: a security audit found `json.dumps([1], indent=10**12)` (and huge string indents or separators on deep or long data) allocated past the Python memory limit unchecked; the metered allocator cannot refuse, so one tenant aborted the process. The Monty fork now preflights the indent repeat and every separator/indent write, raising `MemoryError`. See [Threat Model](security/threat-model.md), TM-PY-038.
 * **Re-entry depth and on-demand stack growth**: a security audit found three ways one tenant could overflow the native stack and abort the shared process: `f(){ eval f; }` and `compgen -C` (plus `fc` and PROMPT_COMMAND) re-entered the interpreter with no depth count; `./s.sh` running itself skipped the `sh s.sh` child-shell guard; and the product of capped nesting kinds (compounds x functions x eval) still exceeded a 2 MiB release worker stack. Re-entry now counts against `max_function_depth` and `MAX_REENTRY_DEPTH` (8), path-invoked scripts take the child-shell guard, and `execute_command` grows the stack on the heap through `stacker` (already in the graph via Monty; not on wasm32). See [Threat Model](security/threat-model.md), TM-DOS-125, TM-DOS-141, TM-DOS-142.
 * **Real mount hardening**: a security audit found `chmod 4755` on a read-write real mount set setuid on the host file, and a host dir mounted at `/` lost the configured `filesystem_limits` (the root overlay used defaults). `RealFs::chmod` now passes only `mode & 0o777`, and the root overlay keeps the base limits. See [Threat Model](security/threat-model.md), TM-ESC-010, TM-DOS-144.

knowledge/runtimes/markkit-fetchkit-builtins.md

@@ -48,7 +48,8 @@ The `CliHost` trait is the only boundary:
 | file reads (`--file`, `--rules`) | VFS, resolved against `cwd` | n/a |
 | env | n/a | shell env (`ctx.env`), never host env |
 | URL egress | `HttpClient` per hop, redirects followed here | fetchkit `HttpTransport` over `HttpClient` |
-| limits | `--max-*`/`--timeout` may lower, never raise, markkit defaults | bashkit `HttpLimits` cap every body |
+| limits | `--max-*`/`--timeout` may lower, never raise, markkit defaults; `timeout` also capped at the remaining `exec` deadline | bashkit `HttpLimits` cap every body; each request's timeout capped at the remaining `exec` deadline |
+| scheduling | runs on tokio's blocking pool (`Handle::block_on`), so the `exec` timeout cancels the caller and co-tenants keep the async worker (TM-DOS-143) | same |
 | `mcp` subcommand | refused | refused |
 
 ### Network

knowledge/security/threat-model.md

@@ -314,6 +314,7 @@ on default 2 MiB test threads (TM-DOS-044).
 | TM-DOS-138 | Join matching-row output growth | Equal-key row groups multiply generated output and synchronous work before dispatcher capture | Parsed rows, fields, keys, decoded input and generated output own shared-budget capacity before growth. VFS reads charge aggregate input and admit metadata size; output fields append directly without temporary row storage, and the byte-stream copy is admitted. Parsing, grouping (including suppressed matches), comparisons and emissions charge work and poll cancellation/deadlines. Errors abort the request. Ordinary two-row fixtures cover memory/work rejection, cancellation/deadlines, session recovery, and complete pipeline/redirect output | **MITIGATED** |
 | TM-DOS-139 | Embedded CLI limit escalation | `markkit extract x --max-input-size 99999999999 --timeout 1e9` buys a larger CPU/memory budget in a shared process | Host clamps every markkit limit to its default after flags (`CliHost::limits`); fetchkit bodies capped by `HttpLimits`; OCR (host binaries) refused | **MITIGATED** |
 | TM-DOS-140 | Tree-walk memory on huge trees | A tenant mounts or creates a tree with millions of files and runs `find`, `du` or `rg` over it; a bulk prefetch of every listing, or rg collecting every file's text before charging the input budget, holds the whole tree in memory | `FileSystem::walk` answers are bounded batches (`WalkOptions::max_entries`, default 10k entries; a listing is never split); `find`/`du` refill through `builtins/walk_cache.rs` one batch at a time, consume listings as they descend, and fall back to `read_dir` if an answer does not hold the requested directory; rg searches each file as it is read and drops its text, charges it to the aggregate-input budget and stops at the first refusal, and in path order its walk reads files as it finds them, one root after another (no candidate list); `ls` (recursive or one wide directory) stops appending lines at `max_stdout_bytes`. Heap peaks over a 200k/2M-file procedural tree are pinned by `large_tree_stress_tests` (walkers under 8 MiB). Residuals: a non-path `--sort`, `--files`, or nested roots still build the full candidate list (paths + metadata per file, no budget charge) before reading; one wide directory is held whole by `read_dir`. Regressions: `walk_cache::tests::batches_stay_bounded_and_cover_every_directory`, `remote_fs_round_trip_tests::walk_batches_are_bounded_*`, `rg_stops_reading_at_input_budget` | **MITIGATED** |
+| TM-DOS-143 | Document extraction outlives the exec deadline | `markkit extract` on a large document (or `fetchkit` converting a large page) ran synchronously on the tenant's async worker: the `exec` timeout could not interrupt it and co-tenants scheduled on that worker stalled, up to markkit's 60 s default | Both builtins run their CLI on tokio's blocking pool, so the caller's future stays cancellable and the worker free; markkit's cooperative `timeout` and fetchkit's per-request timeout are capped at the remaining `exec` budget. Residual: abandoned extraction keeps a blocking-pool thread until markkit's own (coarse) deadline check fires. fetchkit request timeouts have the HTTP client's 1 s floor, so an abandoned request can outlive a sub-second remaining budget by up to 1 s. Regression: `extraction_stops_at_exec_deadline` | **MITIGATED** |
 | TM-DOS-141 | Re-entry recursion via eval/compgen/fc | `f(){ eval f; }`, `compgen -C f`, `fc` re-exec, or PROMPT_COMMAND re-enter the interpreter without a function frame, so recursion was bounded only by the native stack and aborted the shared process | Each re-entry counts against `max_function_depth` and a separate `MAX_REENTRY_DEPTH` (8); over the cap the command fails `maximum nesting depth exceeded` (exit 2). Regressions: `recursive_eval_is_bounded`, `recursive_compgen_command_is_bounded` | **MITIGATED** |
 | TM-DOS-142 | Product of nesting caps overflows worker stack | Each nesting kind has its own cap, but their product (compound nesting x functions x eval x child shells) can exceed a 2 MiB tokio worker stack in release builds | `execute_command` polls through `interpreter/grow_stack.rs`, which uses `stacker::maybe_grow` (512 KiB red zone, 4 MiB segments) so deep but capped nesting extends the stack on the heap instead of aborting. Not on wasm32 (single-threaded, host stack). Regression: `nested_compounds_inside_recursion_do_not_overflow` | **MITIGATED** |
 | TM-DOS-144 | Root real mount drops FsLimits | `mount_real_readonly(dir)` / `mount_real_readwrite(dir)` at `/` wrapped the host dir in `OverlayFs::new`, whose upper layer used default limits, so a tenant could write past the configured `filesystem_limits` | The root overlay is built with `OverlayFs::with_limits` and the base filesystem's limits. Regression: `root_real_mount_keeps_filesystem_limits` | **MITIGATED** |