Skip to content

Commit abb626a

Browse files
author
hack-cli-tests
committed
fix(native): preserve live source recovery boundaries
2 parents 0114933 + fe06bfc commit abb626a

77 files changed

Lines changed: 12728 additions & 352 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/reference/native-authored-commands.md‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ routing, endpoints and host effects remain outside this bounded frontend. Unsupp
3333
refuse before managed value resolution and provider work. Early typed input
3434
capability refusals retain `E_NATIVE_PROJECT_UNSUPPORTED` without exposing
3535
compiler diagnostics or creating native source/start/run authority. `--detach`,
36-
`--json`, service subsets, recovery and other lifecycle operations refuse before input
36+
`--json`, service subsets and other lifecycle operations refuse before input
3737
acquisition. `--branch` supplies an explicit native namespace; its omission uses
3838
the canonical project namespace without inferring a Git branch. Profile and env
3939
selection keep their compiler contracts. `--env base` bypasses inherited overlays
@@ -47,11 +47,41 @@ Ctrl-C requests that shutdown. Exact durable Removed evidence retires the tagged
4747
native run and startup intent. Unknown, changed or live cleanup retains the attempt
4848
and refuses replay; a missing ready mapping does not authorize a fresh start.
4949

50+
`down --recover` explicitly retires a complete stored native generation whose
51+
foreground publisher and frontend admission owner are both dead on the same host
52+
boot. It requires the current version 3 native publication, the exact stored Ready,
53+
startup intent and private source envelope, and the original guest incarnation.
54+
Live owners, older version 2 dead publications, partial startup, pending writes,
55+
changed files and rebooted guests refuse. Env/profile changes, service subsets and
56+
`--json` are unsupported for this operation. It neither recompiles authored input
57+
nor acquires managed values or starts a provider.
58+
Omit `--profile` to recover the stored generation; explicit empty or named profile
59+
overrides both refuse.
60+
61+
Recovery commits a distinct private intent before calling the Rust cleanup owner.
62+
The original raw selectors and resource inventory remain fixed through retries;
63+
current cleanup phases may advance. Before retiring each frontend file, the owner
64+
inspects the exact native receipt and requires all resources Removed with null
65+
observations. A committed retirement phase alone permits the corresponding file's
66+
expected absence. Pending writes and replacements remain retained refusals. The
67+
recovery lease blocks ordinary startup throughout this work. Completed frontend
68+
history is archived under fresh startup admission before a later generation can
69+
start; incomplete history never permits automatic takeover. Persistent data stays
70+
retained, and no recovery operation reboots the guest or allocates new resources.
71+
72+
Dead recovery lease takeover retains its private directory and commits the exact
73+
next owner file before replacing the old owner. A recorded candidate may be pending
74+
or already published after interruption; both forms must match the saved inode and
75+
bytes before a fresh lease is issued. Separate committed release flags authorize
76+
only the corresponding owner/directory absence. An unrecorded pending candidate
77+
and an interrupted completed-history hardlink archive remain retained refusals;
78+
this operation does not repair arbitrary partial lock or file publications.
79+
5080
The native receipt and source paths are distinct from strict Compose v1 artifacts.
5181
Image-only graph receipts remain v2; source-bearing graph receipts use a closed v3
5282
binding, distinct from the existing foreground publication-owner v3. Ready/control
5383
envelopes remain v2. Dead-owner recovery of source-bearing receipts is not admitted
5484
by the separately qualified image-only recovery path. No native hash substitutes
5585
for a normalized Compose hash. Source and fake-driver
5686
checks do not qualify an installed frontend, a live provider, the full authored
57-
corpus, interrupted recovery or resource overhead; those remain separate gates.
87+
corpus, actual version 3 dead-owner recovery or resource overhead; those remain separate gates.

‎docs/reference/native-compose-adoption.md‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,8 +117,44 @@ the engine atomically. Public output remains field/count/status metadata;
117117
capabilities and authored probe arguments are private. Prior versions 1–4 keep
118118
their existing execution behavior, and older upgraded owners refuse version 5
119119
before engine/key reads. Unmodified older launchers still require the previously
120-
documented upgrade boundary. Completed-job dependencies and container recreation
121-
remain unsupported required follow-up work.
120+
documented upgrade boundary. Version 5 does not authorize completed jobs;
121+
container recreation remains unsupported.
122+
123+
The distinct version 7 receipt and manifest support explicitly mapped completion
124+
jobs in a static authored pair with the original default bridge and binding version
125+
1. Names, full container IDs, source/configuration hashes and volume witnesses stay
126+
unchanged. Generated/managed inputs, typed locals, profiles, custom networks, routes
127+
and host hooks remain refused for this family before resource effects or value
128+
acquisition. Jobs require no health check and a disabled observed restart policy
129+
(`no` or the daemon's empty spelling), with zero retries; authored restart omission
130+
stays omitted. Version 6 is reserved for its
131+
separate static-network work and grants no job authority.
132+
133+
Start and restart capture each job's prior daemon `StartedAt` immediately before
134+
an admitted exact-ID start. The successful child and a fresh nonzero UTC timestamp
135+
at nanosecond precision, followed by exited status and exit zero, are required
136+
before starting dependents. Fast jobs need not be observed running. Old exit zero,
137+
equivalent timestamp spellings, unknown facts or a fresh nonzero exit cannot satisfy
138+
completion. The real ordered adapter alone can issue a private one-use completion
139+
bound to this invocation's candidate, resource binding, source capability and
140+
deadline. Injected schedulers, numeric zero, cloned or replayed completions cannot
141+
clear the pending journal. Final publication reacquires all exact workload facts
142+
and rechecks source, receipt, original ownership and the same deadline.
143+
144+
Stop and `down --recover` stop only the original IDs in reverse dependency order;
145+
they never replay jobs. A failed start, cancellation, changed inputs or uncertain
146+
postcondition preserves pending. Restart is an explicit new forward attempt on the
147+
same IDs. Stopped rollback retains data and restores the held legacy pair. These
148+
cooperative daemon observations do not prove hostile-engine resistance, clock
149+
authentication or exactly-once business effects. The finite offline job model and
150+
synthetic owner tests are separate from the required maintained two-worktree SQL,
151+
interruption and cleanup acceptance.
152+
153+
An ordered observer or scheduler refusal can add a fixed
154+
`legacy_adoption_refusal: {stage, reason}` to the JSON error detail. The category
155+
identifies the rejecting boundary without exposing daemon output, resource IDs,
156+
SQL or environment values. It does not assert which effect occurred; other
157+
ownership failures keep the existing fixed generic error.
122158

123159
The maintained `native-compose-adoption-dependency-worktrees` selector exercises
124160
an explicit exec-healthy edge and a short started edge whose target has no health

‎docs/reference/native-compose-ownership.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,15 @@ the in-flight probe's owned subprocess group once, including descendants retaini
7878
its pipes. A completed child with closed streams is never signalled by cleanup;
7979
its former process-group ID is no longer an owned target.
8080

81+
The probe owner also issues private, closed failure metadata for admission or
82+
spawn, nonzero child exit, timeout, cancellation, I/O budget, capture and UTF-8
83+
decoding. Only the exact issued error carries this observation; copied errors,
84+
prototypes and arbitrary error properties cannot supply it. Defined probe error
85+
codes, budgets and cleanup obligations remain unchanged; internal capture and
86+
UTF-8 errors now normalize to the fixed `E_NATIVE_COMPOSE_PROBE` refusal. Ordered retained-job
87+
commands translate this metadata and their separate JSON/object boundaries into
88+
fixed redacted reasons; no argv, daemon output, IDs or exit values are retained.
89+
8190
The closest regression suite is `tests/native-compose-ownership.test.ts`. Its
8291
isolated executable checks accepted resources, collisions, stale generations,
8392
inventory changes, redaction, and actual subprocess overflow/timeout/cancellation.

‎docs/reference/native-config-import.md‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -88,16 +88,15 @@ one-shot labels, job health checks and non-`no` job restart policies refuse. Omi
8888
job restart stays omitted; no native default is invented. Supported profile fields
8989
remain authored selection, and unsupported fields in inactive jobs still refuse.
9090

91-
This is pure conversion, not retained-job adoption. Existing retained adoption
92-
owners still refuse job candidates: their receipt versions do not qualify job
93-
ordering, completion, recovery or replay. A job import preview does not upgrade a
91+
This is pure conversion, not retained-job adoption. Versions 1–5 do not qualify job
92+
ordering, completion, recovery or replay. The distinct [version-7 retained owner](native-compose-adoption.md)
93+
qualifies only its closed static/default-network family. A job import preview does not upgrade a
9494
receipt, launch or recreate a container, transfer ownership, or qualify application
95-
migration. Job-aware retained lifecycle and two-worktree data acceptance remain
96-
separate work.
95+
migration. Two-worktree data acceptance remains a separate live gate.
9796

98-
Retained resource planning uses its own closed adoption baseline and refuses jobs
99-
before acquiring existing engine resource bindings, even when pure conversion
100-
can preserve a job's named mounts. Symbolic conversion is not an ownership grant.
97+
The service-only adoption baseline continues to refuse jobs. Only the version-7
98+
owner selects its distinct completed-job baseline after verifying the closed
99+
static family. Symbolic conversion is not an ownership grant.
101100

102101
Pure preview can preserve a completed job's explicit attachment and aliases on
103102
these owned bridges. Retained storage/adoption still refuses custom-network
@@ -111,7 +110,7 @@ that fit the compiler's positive u32 milliseconds. Missing or zero timings,
111110
and the native contract cannot express all of those options. Explicit `disable:
112111
false` is the default enabled setting. Optional edges and restart propagation
113112
refuse. Completed-job conversion is supported as described above; retained-job
114-
execution remains a separate slice. Unknown HTTP/TCP fields also refuse.
113+
execution has its own version and family boundary. Unknown HTTP/TCP fields also refuse.
115114
The compiler rejects missing, cyclic or inactive dependency targets and ready
116115
edges whose target has no explicit readiness. Refusals include inactive profiles.
117116

‎packages/runtime-core/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,50 @@ it. Native runtime receipts are v2 for image-only graphs and v3 for source-beari
174174
graphs; the authenticated control/ready wire remains v2;
175175
Compose receipt and owner formats remain unchanged. This provenance alone grants
176176
no dead-owner recovery authority.
177+
The inactive read-only recovery selector admits only a complete Ready journal and
178+
a dead version3 publication on the same native host boot. It captures the current
179+
private owner bytes and inode; its raw SHA is a selector for later independent
180+
admission, not an external identity anchor from before that capture. After a
181+
durable intent, the saved inode and digest refuse replacement. Cleanup progress
182+
retains the original Ready selectors and exact inventory. Pending writes refuse;
183+
retired paths require the intent's preceding phase and unchanged archived inode.
184+
`graph native recovery-selection --run-id ID --json` exposes only the closed
185+
selection: run, original value-free Ready receipt, raw receipt/owner hashes and
186+
native host boot time. The receipt contains review hashes, workload readiness,
187+
resource IDs/images/networks and terminal observations; it contains no environment
188+
keys/values, command argv, source bytes or publication process/path metadata. The
189+
selector creates no intent and connects no provider. It grants no cleanup or
190+
frontend recovery authority.
191+
192+
The separate explicit library recovery entrypoint re-admits both raw selectors
193+
and commits a bounded 64 KiB recovery intent before cleanup. It retains the
194+
publication gate, original operation lock and cleanup provider lease through
195+
matching Removed resources, private-environment absence and publication
196+
retirement. A metadata-only current slot inventory is bound into the intent;
197+
changed or missing records refuse. Bounded read-only guest slot enumeration
198+
cross-checks the validated native and Compose host records before cleanup and
199+
again before publication retirement. Unknown guest slots refuse even when host
200+
records are empty; no payload contents are read or inferred from missing records.
201+
Durable phases precede each exclusive archive move; original Ready
202+
hashes and inventory remain fixed while cleanup phases advance. Completed retry
203+
checks absence rather than repeating retirement. Pending writes, unexpected
204+
replacements or missing original and archive paths retain a refusal. Public
205+
frontend recovery and actual dead-owner acceptance remain separate gates.
206+
The distinct macOS command `graph native recover-live-owner --run-id ID
207+
--expect-receipt SHA --expect-owner SHA --json` exposes only this explicit
208+
cleanup entrypoint. Both hashes select the original read-only recovery envelope;
209+
source, private stdin, review, action and timeout options refuse. Existing live
210+
control and direct-cleanup ownership gates remain strict. This command does not
211+
retire frontend start/ready/source artifacts or enable ordinary `up` takeover.
212+
213+
Guarded native cleanup checks its retained authority before and after every
214+
engine observation, removal request and admitted stop batch, including failed
215+
calls. The existing stop batch retains its concurrent HTTP-stop semantics. Cleanup
216+
checks authority before each journal publication. It also
217+
checks each private-environment absence and retirement call. A lost guard refuses
218+
before the next request or journal transition; an already attempted effect remains
219+
durable evidence for an explicitly admitted retry. This prerequisite introduces
220+
no dead-publisher cleanup entrypoint.
177221
Cancellation fences startup before admission and subsequent effects. Cleanup
178222
retains stop failures by admitted workload name, and retires the publication only
179223
after exact cleanup succeeds. Replies bind their requested action and immutable

‎packages/runtime-core/src/graph_cli/native.rs‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ use std::{
99
fn refused() -> CandidateError {
1010
CandidateError::new(
1111
"native_graph_arguments",
12-
"Use graph native plan --source-file FILE, run|serve --source-file FILE --expect-review SHA [--environment-stdin] [--timeout-seconds 1..300], inspect|cleanup --run-id ID, or control --run-id ID --action status|cleanup; optional --json. Foreground serve/control require macOS. Native source and private envelopes require their exact kind/version; values omitted.",
12+
"Use graph native plan --source-file FILE, run|serve --source-file FILE --expect-review SHA [--environment-stdin] [--timeout-seconds 1..300], inspect|cleanup|recovery-selection --run-id ID, recover-live-owner --run-id ID --expect-receipt SHA --expect-owner SHA, or control --run-id ID --action status|cleanup; optional --json. Foreground ownership and its recovery require macOS. Native source and private envelopes require their exact kind/version; values omitted.",
1313
)
1414
}
1515
fn hex(value: &str, len: usize) -> bool {
@@ -35,7 +35,7 @@ pub(super) fn command(candidate: &Candidate, args: &[&str]) -> Result<Value, Can
3535
index += 1;
3636
}
3737
key @ ("--source-file" | "--expect-review" | "--timeout-seconds" | "--run-id"
38-
| "--action") => {
38+
| "--action" | "--expect-receipt" | "--expect-owner") => {
3939
let value = *args
4040
.get(index + 1)
4141
.filter(|value| !value.starts_with("--"))
@@ -48,6 +48,59 @@ pub(super) fn command(candidate: &Candidate, args: &[&str]) -> Result<Value, Can
4848
_ => return Err(refused()),
4949
}
5050
}
51+
if *action == "recover-live-owner" {
52+
if private || singles.len() != 3 {
53+
return Err(refused());
54+
}
55+
let run = *singles
56+
.get("--run-id")
57+
.filter(|run| hex(run, 32))
58+
.ok_or_else(refused)?;
59+
let receipt = *singles
60+
.get("--expect-receipt")
61+
.filter(|receipt| hex(receipt, 64))
62+
.ok_or_else(refused)?;
63+
let owner = *singles
64+
.get("--expect-owner")
65+
.filter(|owner| hex(owner, 64))
66+
.ok_or_else(refused)?;
67+
#[cfg(target_os = "macos")]
68+
{
69+
return serde_json::to_value(native::foreground::recovery::recover(
70+
candidate,
71+
native::foreground::recovery::Options {
72+
run,
73+
expect_receipt: receipt,
74+
expect_owner: owner,
75+
},
76+
)?)
77+
.map_err(|_| refused());
78+
}
79+
#[cfg(not(target_os = "macos"))]
80+
{
81+
let _ = (run, receipt, owner);
82+
return Err(foreground_unavailable());
83+
}
84+
}
85+
if *action == "recovery-selection" {
86+
if private || singles.len() != 1 {
87+
return Err(refused());
88+
}
89+
let run = *singles
90+
.get("--run-id")
91+
.filter(|run| hex(run, 32))
92+
.ok_or_else(refused)?;
93+
#[cfg(target_os = "macos")]
94+
{
95+
return serde_json::to_value(native::foreground::recovery::select(candidate, run)?)
96+
.map_err(|_| refused());
97+
}
98+
#[cfg(not(target_os = "macos"))]
99+
{
100+
let _ = run;
101+
return Err(foreground_unavailable());
102+
}
103+
}
51104
if *action == "control" {
52105
if private || singles.len() != 2 {
53106
return Err(refused());
@@ -99,6 +152,8 @@ pub(super) fn command(candidate: &Candidate, args: &[&str]) -> Result<Value, Can
99152
if !["plan", "run", "serve"].contains(action)
100153
|| singles.contains_key("--run-id")
101154
|| singles.contains_key("--action")
155+
|| singles.contains_key("--expect-receipt")
156+
|| singles.contains_key("--expect-owner")
102157
{
103158
return Err(refused());
104159
}

‎packages/runtime-core/src/provider/environment_recovery.rs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,28 @@ pub fn recorded_slots(candidate: &Candidate) -> Result<Vec<String>, CandidateErr
259259
}
260260
Ok(slots.into_iter().collect())
261261
}
262+
/// Read-only current-guest metadata for native recovery completeness. Pending
263+
/// records refuse rather than being promoted or used as missing-slot authority.
264+
#[cfg(all(target_os = "macos", feature = "native-config-plan"))]
265+
pub(super) fn known_current_slots(
266+
candidate: &Candidate,
267+
incarnation: &str,
268+
boot: &str,
269+
) -> Result<std::collections::BTreeSet<String>, CandidateError> {
270+
let mut known = std::collections::BTreeSet::new();
271+
for slot in recorded_slots(candidate)? {
272+
let path = root(candidate).join(format!("{slot}.pending"));
273+
match fs::symlink_metadata(path) {
274+
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
275+
_ => return Err(error()),
276+
}
277+
let intent = read_mode(candidate, &slot, incarnation, None, false)?;
278+
if intent.boot == boot {
279+
known.insert(slot);
280+
}
281+
}
282+
Ok(known)
283+
}
262284
/// Validates immutable records before associating them with graph cleanup.
263285
pub(super) fn graph_slots(
264286
candidate: &Candidate,

‎packages/runtime-core/src/provider/graph/foreground/transport.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ fn metadata(path: &Path) -> Result<Option<fs::Metadata>, CandidateError> {
191191

192192
/// A dead PID alone cannot prove that another process did not inherit the
193193
/// listener. A single nonblocking connect must report a refused socket.
194-
fn no_listener(path: &Path) -> Result<(), CandidateError> {
194+
pub(in crate::provider::graph) fn no_listener(path: &Path) -> Result<(), CandidateError> {
195195
let bytes = path.as_os_str().as_encoded_bytes();
196196
// SAFETY: sockaddr_un is plain C storage, populated before connect.
197197
let mut address: libc::sockaddr_un = unsafe { std::mem::zeroed() };

‎packages/runtime-core/src/provider/graph/native/foreground.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ use crate::provider::graph::foreground::{signals, transport};
44
use serde::{Deserialize, Serialize};
55
use std::{cell::Cell, io::Write};
66
mod owner;
7+
pub mod recovery;
78
pub(super) use owner::DirectGuard;
89
#[cfg(test)]
910
mod tests;

0 commit comments

Comments
 (0)