diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index 6ea9103d8..3618c8358 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -347,11 +347,25 @@ v4 state is enabled. Runtime startup is a further explicit operation subject to host resource admission. Read `--help` for the candidate command surface. The development profile reserves the declared 32 GiB storage disk and 10 GiB -overlay ceiling plus 16 GiB of host disk space before startup. It reports host -load but does not reject an interactive start solely because unrelated jobs raise -the one-minute load average; memory pressure, thermal state and swapouts remain -admission checks. The research profile keeps its separate 100 GiB disk and host -load qualification envelope. +overlay ceiling plus 16 GiB of host disk space before startup. Its unchanged memory +budget is a 10 GiB free-plus-file-cache estimate: 6 GiB guest, 2 GiB provisional +provider overhead and 2 GiB host reserve. Normal and warning macOS pressure can +qualify only with that headroom, normal thermal/performance state and stable +swapouts. Development samples before the startup lease and again under the lease +before allocation, keeping the same swapout baseline across both windows. +Critical, unknown or incomplete observations refuse; there is no bypass option. +It reports host load without refusing solely because unrelated jobs raise the +one-minute average. The research profile keeps its normal-only pressure policy, +16 GiB raw-free-memory floor, 100 GiB disk floor and host-load qualification +envelope. + +For live Development effects, the host headroom floor is 2 GiB under normal +pressure and 4 GiB under warning, plus measured provider footprint above the +6 GiB guest and 2 GiB provisional overhead estimate. Changed swapouts and thermal +warnings still refuse effects. This does not cap provider footprint or qualify +runtime efficiency. Admission JSON records the numeric pressure level and typed +state; the legacy `memory_pressure_normal` boolean remains false for warning. +Saved reports do not grant resource authority. The normal `hack-runtime-candidate` / `hack-local` development build remains bound to its source checkout, including when all Cargo features are enabled. Only the diff --git a/packages/runtime-core/README.md b/packages/runtime-core/README.md index c01642c93..53d9f7091 100644 --- a/packages/runtime-core/README.md +++ b/packages/runtime-core/README.md @@ -907,12 +907,33 @@ alias or cross-VM network. Internal Docker networks remain internal. The development profile's guest allocation is 6 GiB; guest allocation plus 2 GiB is a provisional provider-footprint estimate, not a hard operating cap. -Before new effects, the candidate requires normal macOS memory pressure, -unchanged swapouts, and at least 2 GiB of estimated host headroom plus any -provider footprint above that estimate. A larger healthy application graph can -therefore continue operating, while its excess footprint consumes an equal -amount of the allowed headroom. `runtime status` reports the actual provider -footprint separately; this admission rule does not qualify its efficiency. +Startup requires the unchanged 10 GiB free-plus-file-cache estimate: 6 GiB guest, +2 GiB provisional provider overhead and 2 GiB host reserve. Development admits +observed normal or warning macOS memory pressure only with that headroom, normal +thermal/performance status, and unchanged swapouts. It samples before acquiring +the startup lease, then samples a fresh three-observation window under the lease +before creating ownership, aliases, disks or provider capacity. Swapouts must +remain at the pre-lease baseline; a stale or incomplete observation refuses. +The final pre-boot check remains in place. Research keeps its normal-only pressure +policy, 16 GiB raw-free-memory floor and three samples 15 seconds apart. + +Before new live Development effects, normal pressure requires at least 2 GiB of +estimated host headroom; warning requires 4 GiB, retaining both provisional +overhead and host reserve after the guest allocation has been charged. Either +floor increases by the measured provider footprint above the guest-plus-overhead +estimate. Critical or unknown pressure, changed swapouts, or thermal/performance +warnings refuse new effects. Cleanup retains its separate ownership checks. A +larger healthy application graph can continue operating when its excess footprint +has matching host headroom. `runtime status` reports that footprint separately; +this experimental policy does not qualify efficiency or host capacity. + +Admission reports add `memory_pressure_level` (the observed sysctl value or null) +and `memory_pressure_state` (`normal`, `warning`, `critical`, or `unknown`). Apple's +exported dispatch masks are 1, 2 and 4; other or missing values refuse. The retained +`memory_pressure_normal` field stays true only for normal pressure and remains +false in an admitted warning report. Older reports containing only that boolean are +historical observations and cannot authorize the warning policy; every operation +uses a fresh typed observation. An existing owned pool changes policy only while stopped through `runtime network internet --json`, then an explicit up with `--internet`. diff --git a/packages/runtime-core/src/provider/admission.rs b/packages/runtime-core/src/provider/admission.rs index be451deb6..53d918756 100644 --- a/packages/runtime-core/src/provider/admission.rs +++ b/packages/runtime-core/src/provider/admission.rs @@ -9,6 +9,55 @@ pub const FREE_MEMORY_FLOOR: u64 = 16 * 1024 * 1024 * 1024; const RESEARCH_DISK_FLOOR_GIB: u64 = 100; const DEVELOPMENT_HOST_DISK_RESERVE_GIB: u64 = 16; +/// The sysctl exports dispatch pressure masks, not XNU's internal pressure enum. +/// Unknown/missing observations never authorize allocation or live effects. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "kebab-case")] +pub enum MemoryPressureState { + Normal, + Warning, + Critical, + Unknown, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +pub struct MemoryPressure { + pub memory_pressure_level: Option, + pub memory_pressure_state: MemoryPressureState, +} + +impl MemoryPressure { + pub fn parse(text: &str) -> Self { + Self::from_level(text.trim().parse().ok()) + } + + pub fn from_level(level: Option) -> Self { + Self { + memory_pressure_level: level, + memory_pressure_state: match level { + Some(1) => MemoryPressureState::Normal, + Some(2) => MemoryPressureState::Warning, + Some(4) => MemoryPressureState::Critical, + _ => MemoryPressureState::Unknown, + }, + } + } + + fn admitted(self, profile: Profile) -> bool { + self == Self::from_level(self.memory_pressure_level) + && matches!( + (profile, self.memory_pressure_state), + (_, MemoryPressureState::Normal) + | (Profile::Development, MemoryPressureState::Warning) + ) + } +} + +fn thermal_normal(text: &str) -> bool { + text.contains("No thermal warning level has been recorded") + && text.contains("No performance warning level has been recorded") +} + fn disk_floor_bytes(profile: Profile) -> u64 { let gib = match profile { Profile::Research => RESEARCH_DISK_FLOOR_GIB, @@ -39,6 +88,9 @@ pub struct Admission { pub minimum_free_memory_bytes: u64, pub free_plus_file_cache_estimate_bytes: Option, pub memory_budget_basis: &'static str, + #[serde(flatten)] + pub memory_pressure: MemoryPressure, + /// Compatibility only: true means literally normal, never an admitted warning. pub memory_pressure_normal: bool, pub disk_free_bytes: Option, pub minimum_disk_free_bytes: u64, @@ -107,7 +159,9 @@ pub fn parse_cache_headroom(text: &str) -> Result { pub struct OperatingSample { pub free_memory_bytes: u64, pub free_plus_file_cache_estimate_bytes: u64, - pub memory_pressure_normal: bool, + #[serde(flatten)] + pub memory_pressure: MemoryPressure, + pub thermal_normal: bool, pub swapouts: u64, } @@ -121,6 +175,10 @@ pub fn operating_sample() -> Result { .args(["-n", "kern.memorystatus_vm_pressure_level"]), Duration::from_secs(3), )?; + let thermal = run( + clean_command(Path::new("/usr/bin/pmset")).args(["-g", "therm"]), + Duration::from_secs(3), + )?; let swapouts = text .lines() .find_map(|line| line.strip_prefix("Swapouts:")) @@ -129,7 +187,8 @@ pub fn operating_sample() -> Result { Ok(OperatingSample { free_memory_bytes: parse_free_memory(&text)?, free_plus_file_cache_estimate_bytes: parse_cache_headroom(&text)?, - memory_pressure_normal: pressure.trim() == "1", + memory_pressure: MemoryPressure::parse(&pressure), + thermal_normal: thermal_normal(&thermal), swapouts, }) } @@ -140,23 +199,29 @@ pub fn validate_operating( footprint: u64, profile: Profile, ) -> Result<(), CandidateError> { - let budget = (u64::from(profile.memory_mib()) + 2048) * 1024 * 1024; + let budget = profile.provider_memory_budget_bytes(); // A provider can exceed the provisional guest-plus-overhead estimate while // the host remains healthy. Require matching additional host headroom for // that excess instead of permanently fencing an already-running graph. let excess_footprint = footprint.saturating_sub(budget); - let headroom_floor = (2_u64 * 1024 * 1024 * 1024).saturating_add(excess_footprint); + let headroom_floor = profile + .operating_host_reserve_bytes(sample.memory_pressure.memory_pressure_state) + .saturating_add(excess_footprint); let headroom_failed = sample.free_plus_file_cache_estimate_bytes < headroom_floor; - let pressure_failed = !sample.memory_pressure_normal; + let pressure_failed = !sample.memory_pressure.admitted(profile); let swap_failed = sample.swapouts != baseline_swapouts; - if headroom_failed || pressure_failed || swap_failed { + let thermal_failed = !sample.thermal_normal; + if headroom_failed || pressure_failed || swap_failed || thermal_failed { // Fixed labels and numeric observations only: callers can safely classify // the failed predicates without capturing provider output or environment. return Err(CandidateError::new( "runtime_pressure", format!( - "Development effects paused: runtime_pressure headroom_failed={headroom_failed} pressure_failed={pressure_failed} swap_failed={swap_failed} headroom_bytes={} headroom_floor_bytes={headroom_floor} swapouts_baseline={baseline_swapouts} swapouts_current={} provider_footprint_bytes={footprint} provider_budget_bytes={budget} provider_excess_bytes={excess_footprint}. Inspect and stop owned capacity if pressure persists.", - sample.free_plus_file_cache_estimate_bytes, sample.swapouts, + "{profile:?} effects paused: runtime_pressure headroom_failed={headroom_failed} pressure_failed={pressure_failed} swap_failed={swap_failed} thermal_failed={thermal_failed} pressure_state={:?} pressure_level={:?} headroom_bytes={} headroom_floor_bytes={headroom_floor} swapouts_baseline={baseline_swapouts} swapouts_current={} provider_footprint_bytes={footprint} provider_budget_bytes={budget} provider_excess_bytes={excess_footprint}. Inspect and stop owned capacity if pressure persists.", + sample.memory_pressure.memory_pressure_state, + sample.memory_pressure.memory_pressure_level, + sample.free_plus_file_cache_estimate_bytes, + sample.swapouts, ), )); } @@ -212,6 +277,7 @@ fn probe_with_disk_budget( "free-plus-file-cache-estimate" }, memory_pressure_normal: false, + memory_pressure: MemoryPressure::from_level(None), disk_free_bytes: None, minimum_disk_free_bytes: disk_budget(profile, owned_live).0, disk_budget_basis: disk_budget(profile, owned_live).1, @@ -238,7 +304,9 @@ fn probe_with_disk_budget( .args(["-n", "kern.memorystatus_vm_pressure_level"]), Duration::from_secs(3), )?; - result.memory_pressure_normal = pressure.trim() == "1"; + result.memory_pressure = MemoryPressure::parse(&pressure); + result.memory_pressure_normal = + result.memory_pressure.memory_pressure_state == MemoryPressureState::Normal; let budget_memory = if profile == Profile::Development { let estimate = parse_cache_headroom(&memory)?; result.free_plus_file_cache_estimate_bytes = Some(estimate); @@ -254,10 +322,10 @@ fn probe_with_disk_budget( Profile::Development => "Free-plus-file-cache estimate is below the 10 GiB experimental development budget (6 GiB guest plus 4 GiB overhead/reserve).", }.into()); } - if !result.memory_pressure_normal { + if !result.memory_pressure.admitted(profile) { result .reasons - .push("macOS memory pressure is not normal.".into()); + .push("macOS memory pressure is not admitted for this profile.".into()); } result.swapouts = memory .lines() @@ -319,8 +387,7 @@ fn probe_with_disk_budget( clean_command(Path::new("/usr/bin/pmset")).args(["-g", "therm"]), Duration::from_secs(3), )?; - result.thermal_normal = thermal.contains("No thermal warning level has been recorded") - && thermal.contains("No performance warning level has been recorded"); + result.thermal_normal = thermal_normal(&thermal); if !result.thermal_normal { result .reasons @@ -342,29 +409,85 @@ pub fn sample_for(path: &Path, profile: Profile) -> Result, Candi pub(super) fn sample_with( profile: Profile, + observe: impl FnMut() -> Result, +) -> Result, CandidateError> { + sample_checked(profile, None, observe, std::thread::sleep) +} + +/// The initial observation and a lease wait cannot hide a swapout counter change. +pub(super) fn sample_after( + profile: Profile, + baseline_swapouts: Option, + observe: impl FnMut() -> Result, +) -> Result, CandidateError> { + let baseline = baseline_swapouts.ok_or_else(|| { + CandidateError::new( + "admission_unavailable", + "Cannot observe the swapout baseline.", + ) + })?; + sample_checked(profile, Some(baseline), observe, std::thread::sleep) +} + +/// Development revalidates a full window after obtaining the startup lease, +/// before creating an owner, disks, provider capacity, or any aliases. +pub(super) fn recheck_after_lease( + profile: Profile, + samples: &[Admission], + observe: impl FnMut() -> Result, +) -> Result, CandidateError> { + if profile == Profile::Research { + return Ok(Vec::new()); + } + sample_after( + profile, + samples.last().and_then(|sample| sample.swapouts), + observe, + ) +} + +fn sample_checked( + profile: Profile, + mut baseline_swapouts: Option, mut observe: impl FnMut() -> Result, + mut pause: impl FnMut(Duration), ) -> Result, CandidateError> { let mut samples: Vec = Vec::new(); for index in 0..3 { let sample = observe()?; - if !sample.admitted || sample.profile != profile { + let headroom = match profile { + Profile::Research => sample.free_memory_bytes, + Profile::Development => sample.free_plus_file_cache_estimate_bytes, + }; + // `admitted` remains a report field, not authority to reinterpret a + // legacy normal-only boolean or ignore an incomplete typed observation. + if !sample.admitted + || !sample.host_supported + || sample.profile != profile + || !sample.memory_pressure.admitted(profile) + || !sample.thermal_normal + || sample.swapouts.is_none() + || headroom.is_none_or(|bytes| bytes < profile.minimum_free_memory_bytes()) + { return Err(CandidateError::new( "admission_rejected", - sample.reasons.join(" "), + if sample.reasons.is_empty() { + "A required startup resource observation is not admitted.".into() + } else { + sample.reasons.join(" ") + }, )); } - if samples - .first() - .is_some_and(|first| first.swapouts != sample.swapouts) - { + if baseline_swapouts.is_some() && baseline_swapouts != sample.swapouts { return Err(CandidateError::new( "admission_rejected", - "Swapouts increased during admission.", + "Swapouts changed during admission.", )); } + baseline_swapouts = sample.swapouts; samples.push(sample); if index < 2 { - std::thread::sleep(Duration::from_secs(if profile == Profile::Research { + pause(Duration::from_secs(if profile == Profile::Research { 15 } else { 1 @@ -374,6 +497,9 @@ pub(super) fn sample_with( Ok(samples) } +#[cfg(test)] +mod pressure_tests; + #[cfg(test)] mod tests { use super::*; @@ -441,7 +567,8 @@ mod tests { let sample = OperatingSample { free_memory_bytes: 0, free_plus_file_cache_estimate_bytes: headroom, - memory_pressure_normal: normal, + memory_pressure: MemoryPressure::from_level(Some(if normal { 1 } else { 4 })), + thermal_normal: true, swapouts, }; let error = @@ -452,7 +579,9 @@ mod tests { assert_eq!( error.message, format!( - "Development effects paused: runtime_pressure {flags} headroom_bytes={headroom} headroom_floor_bytes={expected_floor} swapouts_baseline=42 swapouts_current={swapouts} provider_footprint_bytes={footprint} provider_budget_bytes=8589934592 provider_excess_bytes={expected_excess}. Inspect and stop owned capacity if pressure persists." + "Development effects paused: runtime_pressure {flags} thermal_failed=false pressure_state={:?} pressure_level={:?} headroom_bytes={headroom} headroom_floor_bytes={expected_floor} swapouts_baseline=42 swapouts_current={swapouts} provider_footprint_bytes={footprint} provider_budget_bytes=8589934592 provider_excess_bytes={expected_excess}. Inspect and stop owned capacity if pressure persists.", + sample.memory_pressure.memory_pressure_state, + sample.memory_pressure.memory_pressure_level, ) ); } @@ -465,7 +594,8 @@ mod tests { let mut sample = OperatingSample { free_memory_bytes: 0, free_plus_file_cache_estimate_bytes: 2 * 1024 * 1024 * 1024, - memory_pressure_normal: true, + memory_pressure: MemoryPressure::from_level(Some(1)), + thermal_normal: true, swapouts: u64::MAX, }; assert!(validate_operating(&sample, u64::MAX, budget, profile).is_ok()); @@ -486,7 +616,8 @@ mod tests { let mut sample = OperatingSample { free_memory_bytes: 0, free_plus_file_cache_estimate_bytes: 6 * 1024 * 1024 * 1024, - memory_pressure_normal: true, + memory_pressure: MemoryPressure::from_level(Some(1)), + thermal_normal: true, swapouts: 42, }; // The guest can consume its admitted allocation without needing a second full reservation. @@ -507,7 +638,7 @@ mod tests { sample.free_plus_file_cache_estimate_bytes = 1024 * 1024 * 1024; assert!(validate_operating(&sample, 42, 0, Profile::Development).is_err()); sample.free_plus_file_cache_estimate_bytes = 6 * 1024 * 1024 * 1024; - sample.memory_pressure_normal = false; + sample.memory_pressure = MemoryPressure::from_level(Some(4)); assert!(validate_operating(&sample, 42, 0, Profile::Development).is_err()); } #[test] diff --git a/packages/runtime-core/src/provider/admission/pressure_tests.rs b/packages/runtime-core/src/provider/admission/pressure_tests.rs new file mode 100644 index 000000000..35878422a --- /dev/null +++ b/packages/runtime-core/src/provider/admission/pressure_tests.rs @@ -0,0 +1,275 @@ +use super::*; +use std::cell::Cell; + +const GIB: u64 = 1024 * 1024 * 1024; + +fn observation(profile: Profile, level: Option, headroom: u64) -> Admission { + let pressure = MemoryPressure::from_level(level); + Admission { + profile, + host_supported: true, + free_memory_bytes: Some(headroom), + minimum_free_memory_bytes: profile.minimum_free_memory_bytes(), + free_plus_file_cache_estimate_bytes: Some(headroom), + memory_budget_basis: "fixture", + memory_pressure: pressure, + memory_pressure_normal: pressure.memory_pressure_state == MemoryPressureState::Normal, + disk_free_bytes: Some(disk_floor_bytes(profile)), + minimum_disk_free_bytes: disk_floor_bytes(profile), + disk_budget_basis: "fixture", + one_minute_load: Some(0.0), + load_ceiling: load_ceiling(profile), + thermal_normal: true, + swapouts: Some(42), + admitted: true, + reasons: Vec::new(), + } +} + +fn operating(level: Option, headroom: u64) -> OperatingSample { + OperatingSample { + free_memory_bytes: 0, + free_plus_file_cache_estimate_bytes: headroom, + memory_pressure: MemoryPressure::from_level(level), + thermal_normal: true, + swapouts: 42, + } +} + +#[test] +fn exported_dispatch_masks_are_distinct_from_internal_xnu_enum_and_fail_closed() { + for (text, level, state) in [ + ("1\n", Some(1), MemoryPressureState::Normal), + ("2", Some(2), MemoryPressureState::Warning), + ("4", Some(4), MemoryPressureState::Critical), + ("0", Some(0), MemoryPressureState::Unknown), + ("3", Some(3), MemoryPressureState::Unknown), + ("8", Some(8), MemoryPressureState::Unknown), + ("4294967295", Some(u32::MAX), MemoryPressureState::Unknown), + ("4294967296", None, MemoryPressureState::Unknown), + ("-1", None, MemoryPressureState::Unknown), + ("", None, MemoryPressureState::Unknown), + ("normal", None, MemoryPressureState::Unknown), + ("1 2", None, MemoryPressureState::Unknown), + ] { + let pressure = MemoryPressure::parse(text); + assert_eq!(pressure.memory_pressure_level, level); + assert_eq!(pressure.memory_pressure_state, state); + } + let inconsistent = MemoryPressure { + memory_pressure_level: Some(4), + memory_pressure_state: MemoryPressureState::Normal, + }; + assert!(!inconsistent.admitted(Profile::Development)); +} + +#[test] +fn startup_windows_keep_profile_floors_and_distinguish_warning_from_normal() { + assert_eq!(Profile::Development.minimum_free_memory_bytes(), 10 * GIB); + assert_eq!(Profile::Research.minimum_free_memory_bytes(), 16 * GIB); + for (profile, level, headroom, accepted) in [ + (Profile::Development, Some(1), 10 * GIB, true), + (Profile::Development, Some(2), 10 * GIB, true), + (Profile::Development, Some(2), 10 * GIB - 1, false), + (Profile::Development, Some(4), 128 * GIB, false), + (Profile::Development, None, 128 * GIB, false), + (Profile::Development, Some(3), 128 * GIB, false), + (Profile::Research, Some(1), 16 * GIB, true), + (Profile::Research, Some(1), 16 * GIB - 1, false), + (Profile::Research, Some(2), 128 * GIB, false), + (Profile::Research, Some(4), 128 * GIB, false), + ] { + let pauses = Cell::new(0); + let result = sample_checked( + profile, + Some(42), + || Ok(observation(profile, level, headroom)), + |duration| { + assert_eq!( + duration, + Duration::from_secs(if profile == Profile::Research { 15 } else { 1 }) + ); + pauses.set(pauses.get() + 1); + }, + ); + assert_eq!( + result.is_ok(), + accepted, + "{profile:?}, {level:?}, {headroom}" + ); + assert_eq!(pauses.get(), if accepted { 2 } else { 0 }); + } +} + +#[test] +fn incomplete_or_unsafe_samples_cannot_be_authorized_by_admitted_or_legacy_booleans() { + for defect in 0..6 { + let mut sample = observation(Profile::Development, Some(2), 10 * GIB); + match defect { + 0 => sample.thermal_normal = false, + 1 => sample.swapouts = None, + 2 => sample.free_plus_file_cache_estimate_bytes = None, + 3 => sample.profile = Profile::Research, + 4 => sample.host_supported = false, + _ => { + // Old reports have no typed observation. A normal-only boolean + // cannot be interpreted as authority for the new warning policy. + sample.memory_pressure = MemoryPressure::from_level(None); + sample.memory_pressure_normal = true; + } + } + let mut sample = Some(sample); + assert!( + sample_checked( + Profile::Development, + Some(42), + || Ok(sample.take().unwrap()), + |_| { panic!("unsafe initial observation must stop before another sample") } + ) + .is_err() + ); + } +} + +#[test] +fn swapouts_must_be_observed_and_unchanged_across_the_full_window() { + for swaps in [Some(43), Some(41), None] { + let mut calls = 0; + let result = sample_checked( + Profile::Development, + Some(42), + || { + calls += 1; + let mut sample = observation(Profile::Development, Some(2), 10 * GIB); + if calls == 2 { + sample.swapouts = swaps; + } + Ok(sample) + }, + |_| {}, + ); + assert!(matches!(result, Err(error) if error.code == "admission_rejected")); + assert_eq!(calls, 2); + } +} + +#[test] +fn post_lease_window_blocks_allocation_when_prelease_admission_goes_stale() { + let prelease = [observation(Profile::Development, Some(2), 10 * GIB)]; + for defect in 0..7 { + let allocated = Cell::new(false); + let calls = Cell::new(0); + let checked = recheck_after_lease(Profile::Development, &prelease, || { + calls.set(calls.get() + 1); + let mut sample = observation(Profile::Development, Some(2), 10 * GIB); + match defect { + 0 => sample.free_plus_file_cache_estimate_bytes = Some(10 * GIB - 1), + 1 => sample.memory_pressure = MemoryPressure::from_level(Some(4)), + 2 => sample.memory_pressure = MemoryPressure::from_level(None), + 3 => sample.swapouts = Some(43), + 4 => sample.swapouts = Some(41), + 5 => sample.thermal_normal = false, + _ => { + sample.admitted = false; + sample.reasons.push("disk budget".into()); + } + } + Ok(sample) + }) + .map(|_| allocated.set(true)); + assert!(matches!(checked, Err(error) if error.code == "admission_rejected")); + assert!(!allocated.get()); + assert_eq!(calls.get(), 1); + } + assert!(recheck_after_lease(Profile::Development, &[], || panic!("missing baseline")).is_err()); + assert!( + recheck_after_lease(Profile::Research, &[], || panic!( + "research timing unchanged" + )) + .unwrap() + .is_empty() + ); +} + +#[test] +fn healthy_post_lease_warning_is_resampled_before_allocation() { + let prelease = [observation(Profile::Development, Some(1), 10 * GIB)]; + let calls = Cell::new(0); + let samples = recheck_after_lease(Profile::Development, &prelease, || { + calls.set(calls.get() + 1); + Ok(observation(Profile::Development, Some(2), 10 * GIB)) + }) + .unwrap(); + assert_eq!(calls.get(), 3); + assert_eq!(samples.len(), 3); + assert!( + samples + .iter() + .all(|sample| !sample.memory_pressure_normal && sample.swapouts == Some(42)) + ); +} + +#[test] +fn warning_operating_reserve_adds_measured_excess_without_a_provider_cap() { + let budget = Profile::Development.provider_memory_budget_bytes(); + for footprint in [0, budget, budget + 1, 20 * GIB, 100 * GIB] { + let floor = 4 * GIB + footprint.saturating_sub(budget); + assert!( + validate_operating( + &operating(Some(2), floor), + 42, + footprint, + Profile::Development + ) + .is_ok() + ); + assert!( + validate_operating( + &operating(Some(2), floor - 1), + 42, + footprint, + Profile::Development + ) + .is_err() + ); + } + assert!(validate_operating(&operating(Some(2), 128 * GIB), 42, 0, Profile::Research).is_err()); + let mut hot = operating(Some(2), 128 * GIB); + hot.thermal_normal = false; + let error = validate_operating(&hot, 42, 0, Profile::Development).unwrap_err(); + assert!(error.message.contains("thermal_failed=true")); +} + +#[test] +fn additive_reports_keep_legacy_normal_boolean_literal_and_numeric_state_explicit() { + for (level, state, normal) in [ + (Some(1), "normal", true), + (Some(2), "warning", false), + (Some(4), "critical", false), + (None, "unknown", false), + ] { + let report = + serde_json::to_value(observation(Profile::Development, level, 10 * GIB)).unwrap(); + assert_eq!( + report["memory_pressure_level"], + serde_json::to_value(level).unwrap() + ); + assert_eq!(report["memory_pressure_state"], state); + assert_eq!(report["memory_pressure_normal"], normal); + assert!(report.get("memory_pressure").is_none()); + } +} + +#[test] +fn thermal_observation_requires_both_unchanged_normal_markers() { + let normal = "No thermal warning level has been recorded\nNo performance warning level has been recorded\n"; + assert!(thermal_normal(normal)); + for text in [ + "", + "Thermal Warning Level = 1", + "No thermal warning level has been recorded", + "No performance warning level has been recorded", + ] { + assert!(!thermal_normal(text)); + } +} diff --git a/packages/runtime-core/src/provider/lifecycle.rs b/packages/runtime-core/src/provider/lifecycle.rs index 9dcceb193..ff0796b9b 100644 --- a/packages/runtime-core/src/provider/lifecycle.rs +++ b/packages/runtime-core/src/provider/lifecycle.rs @@ -1,7 +1,10 @@ mod admission_pool; pub mod host_filesystem; mod interrupted; +mod operating_guard; mod prepared_boot; +#[cfg(test)] +mod pressure_tests; #[cfg(any(target_os = "macos", test))] mod private_child; mod relay_process; @@ -282,6 +285,20 @@ pub(super) fn startup_lease(root: &Path, wait: Duration) -> Result Result, +) -> Result<(state::Lock, Vec), CandidateError> { + let lock = startup_lease(root, wait)?; + let fresh = admission::recheck_after_lease(profile, samples, observe)?; + Ok((lock, fresh)) +} + /// Read-only observation takes no mutation lease and detects lifecycle changes around each read. pub(super) struct ObservedGuest<'a> { candidate: &'a Candidate, @@ -342,7 +359,15 @@ pub(super) struct OwnedGuest<'a> { owner: Owner, _lock: state::Lock, allocation_allowed: bool, - guard: Option<(u64, std::cell::Cell)>, + guard: Option, +} + +fn operating_observation( + owner: &Owner, +) -> Result<(admission::OperatingSample, u64), CandidateError> { + let sample = admission::operating_sample()?; + let usage = identity::memory_usage(owner.process.as_ref().expect("verified process").pid)?; + Ok((sample, usage.physical_footprint_bytes)) } impl<'a> OwnedGuest<'a> { @@ -373,20 +398,8 @@ impl<'a> OwnedGuest<'a> { self.owner.project_share.as_ref() } pub(super) fn before_effect(&self) -> Result<(), CandidateError> { - if let Some((swapouts, last)) = &self.guard { - if last.get().elapsed() >= Duration::from_secs(2) { - let sample = admission::operating_sample()?; - let usage = identity::memory_usage( - self.owner.process.as_ref().expect("verified process").pid, - )?; - admission::validate_operating( - &sample, - *swapouts, - usage.physical_footprint_bytes, - self.owner.profile, - )?; - last.set(Instant::now()); - } + if let Some(guard) = &self.guard { + guard.before_effect(self.owner.profile, || operating_observation(&self.owner))?; } Ok(()) } @@ -448,16 +461,10 @@ impl<'a> OwnedGuest<'a> { Ok(current) })?; let guard = if enforce_budget && current.profile == super::Profile::Development { - let sample = admission::operating_sample()?; - let usage = - identity::memory_usage(current.process.as_ref().expect("verified process").pid)?; - admission::validate_operating( - &sample, - sample.swapouts, - usage.physical_footprint_bytes, + Some(operating_guard::OperatingGuard::connect( current.profile, - )?; - Some((sample.swapouts, std::cell::Cell::new(Instant::now()))) + || operating_observation(¤t), + )?) } else { None }; @@ -1060,16 +1067,33 @@ fn start_pool( admission.reasons.join(" "), )); } - let samples = admission::sample_with(profile, || { - admission_pool::probe(candidate, profile, admission_owner.as_ref()) - })?; + let mut samples = if profile == super::Profile::Development { + let baseline_swapouts = admission.swapouts; + let mut samples = vec![admission]; + samples.extend(admission::sample_after(profile, baseline_swapouts, || { + admission_pool::probe(candidate, profile, admission_owner.as_ref()) + })?); + samples + } else { + // Retain the frozen Research window and baseline semantics. + admission::sample_with(profile, || { + admission_pool::probe(candidate, profile, admission_owner.as_ref()) + })? + }; artifact::verify(candidate)?; artifact::verify_engine(candidate)?; super::network_tools::verify(candidate)?; - let lock = startup_lease(&root(candidate), STARTUP_LEASE_WAIT)?; + let (lock, fresh_samples) = startup_admission_lease( + &root(candidate), + STARTUP_LEASE_WAIT, + profile, + &samples, + || admission_pool::probe(candidate, profile, admission_owner.as_ref()), + )?; if let Some(selected) = &admission_owner { selected.reverify(candidate)?; } + samples.extend(fresh_samples); #[cfg(target_os = "macos")] if let Some(guard) = &retained_guard { guard.verify(candidate)?; @@ -2489,9 +2513,9 @@ printf 'verified-cache\n' .stage_with_guest(&guest) .unwrap(); // Force a baseline mismatch without creating real host pressure or changing global state. - guest.guard = Some(( + guest.guard = Some(operating_guard::OperatingGuard::fixture( u64::MAX, - std::cell::Cell::new(Instant::now() - Duration::from_secs(3)), + Instant::now() - Duration::from_secs(3), )); assert_eq!( lease diff --git a/packages/runtime-core/src/provider/lifecycle/operating_guard.rs b/packages/runtime-core/src/provider/lifecycle/operating_guard.rs new file mode 100644 index 000000000..003b16dda --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/operating_guard.rs @@ -0,0 +1,156 @@ +use super::super::{Profile, admission}; +use crate::CandidateError; +use std::cell::Cell; +use std::time::{Duration, Instant}; + +/// A live Development connection carries its observed swapout baseline across +/// effects. Failed observations never refresh the last successful check. +pub(super) struct OperatingGuard { + swapouts: u64, + last: Cell, +} + +impl OperatingGuard { + pub(super) fn connect( + profile: Profile, + observe: impl FnOnce() -> Result<(admission::OperatingSample, u64), CandidateError>, + ) -> Result { + let (sample, footprint) = observe()?; + admission::validate_operating(&sample, sample.swapouts, footprint, profile)?; + Ok(Self { + swapouts: sample.swapouts, + last: Cell::new(Instant::now()), + }) + } + + pub(super) fn before_effect( + &self, + profile: Profile, + observe: impl FnOnce() -> Result<(admission::OperatingSample, u64), CandidateError>, + ) -> Result<(), CandidateError> { + self.check_at(Instant::now(), profile, observe) + } + + fn check_at( + &self, + now: Instant, + profile: Profile, + observe: impl FnOnce() -> Result<(admission::OperatingSample, u64), CandidateError>, + ) -> Result<(), CandidateError> { + if now.duration_since(self.last.get()) >= Duration::from_secs(2) { + let (sample, footprint) = observe()?; + admission::validate_operating(&sample, self.swapouts, footprint, profile)?; + self.last.set(Instant::now()); + } + Ok(()) + } + + #[cfg(test)] + pub(super) fn fixture(swapouts: u64, last: Instant) -> Self { + Self { + swapouts, + last: Cell::new(last), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use admission::{MemoryPressure, OperatingSample}; + + const GIB: u64 = 1024 * 1024 * 1024; + + fn observation( + level: Option, + headroom: u64, + swapouts: u64, + thermal: bool, + ) -> OperatingSample { + OperatingSample { + free_memory_bytes: 0, + free_plus_file_cache_estimate_bytes: headroom, + memory_pressure: MemoryPressure::from_level(level), + thermal_normal: thermal, + swapouts, + } + } + + #[test] + fn connection_checks_warning_reserve_critical_unknown_and_thermal_before_authority() { + for (level, headroom, thermal, admitted) in [ + (Some(1), 2 * GIB, true, true), + (Some(2), 4 * GIB, true, true), + (Some(2), 4 * GIB - 1, true, false), + (Some(4), 100 * GIB, true, false), + (None, 100 * GIB, true, false), + (Some(2), 100 * GIB, false, false), + ] { + let connected = OperatingGuard::connect(Profile::Development, || { + Ok((observation(level, headroom, 42, thermal), 8 * GIB)) + }); + assert_eq!( + connected.is_ok(), + admitted, + "{level:?}, {headroom}, {thermal}" + ); + } + } + + #[test] + fn effect_checks_block_mutation_and_keep_the_failed_guard_due() { + for (level, headroom, swapouts, thermal, footprint) in [ + (Some(2), 4 * GIB - 1, 42, true, 8 * GIB), + (Some(2), 4 * GIB, 42, true, 8 * GIB + 1), + (Some(4), 100 * GIB, 42, true, 8 * GIB), + (None, 100 * GIB, 42, true, 8 * GIB), + (Some(1), 100 * GIB, 43, true, 8 * GIB), + (Some(1), 100 * GIB, 41, true, 8 * GIB), + (Some(1), 100 * GIB, 42, false, 8 * GIB), + ] { + let last = Instant::now() - Duration::from_secs(3); + let guard = OperatingGuard::fixture(42, last); + let mutation = Cell::new(false); + let refused = guard + .before_effect(Profile::Development, || { + Ok((observation(level, headroom, swapouts, thermal), footprint)) + }) + .map(|()| mutation.set(true)); + assert!(matches!(refused, Err(error) if error.code == "runtime_pressure")); + assert!(!mutation.get()); + assert_eq!(guard.last.get(), last); + let rechecks = Cell::new(0); + guard + .before_effect(Profile::Development, || { + rechecks.set(rechecks.get() + 1); + Ok((observation(Some(2), 5 * GIB, 42, true), 9 * GIB)) + }) + .unwrap(); + assert_eq!(rechecks.get(), 1); + assert!(guard.last.get() > last); + } + } + + #[test] + fn healthy_effect_uses_the_bounded_check_interval_and_retains_the_baseline() { + let guard = OperatingGuard::connect(Profile::Development, || { + Ok((observation(Some(1), 2 * GIB, 42, true), 8 * GIB)) + }) + .unwrap(); + guard + .check_at( + guard.last.get() + Duration::from_secs(1), + Profile::Development, + || panic!("fresh guard must not repeat a host probe"), + ) + .unwrap(); + guard + .check_at( + guard.last.get() + Duration::from_secs(2), + Profile::Development, + || Ok((observation(Some(2), 4 * GIB, 42, true), 8 * GIB)), + ) + .unwrap(); + assert_eq!(guard.swapouts, 42); + } +} diff --git a/packages/runtime-core/src/provider/lifecycle/pressure_tests.rs b/packages/runtime-core/src/provider/lifecycle/pressure_tests.rs new file mode 100644 index 000000000..1c1dbd337 --- /dev/null +++ b/packages/runtime-core/src/provider/lifecycle/pressure_tests.rs @@ -0,0 +1,80 @@ +use super::super::Profile; +use super::*; +use admission::{Admission, MemoryPressure}; +use std::sync::{ + Arc, + atomic::{AtomicU32, Ordering}, +}; + +fn report(level: u32) -> Admission { + const GIB: u64 = 1024 * 1024 * 1024; + Admission { + profile: Profile::Development, + host_supported: true, + free_memory_bytes: Some(10 * GIB), + minimum_free_memory_bytes: 10 * GIB, + free_plus_file_cache_estimate_bytes: Some(10 * GIB), + memory_budget_basis: "fixture", + memory_pressure: MemoryPressure::from_level(Some(level)), + memory_pressure_normal: level == 1, + disk_free_bytes: Some(58 * GIB), + minimum_disk_free_bytes: 58 * GIB, + disk_budget_basis: "fixture", + one_minute_load: Some(0.0), + load_ceiling: None, + thermal_normal: true, + swapouts: Some(42), + admitted: true, + reasons: Vec::new(), + } +} + +#[test] +fn lease_wait_pressure_change_refuses_owner_commit_and_releases_the_mutation_lease() { + let path = std::fs::canonicalize(std::env::temp_dir()) + .unwrap() + .join(format!( + "hkl-pressure-lease-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + struct Remove(std::path::PathBuf); + impl Drop for Remove { + fn drop(&mut self) { + let _ = fs::remove_dir_all(&self.0); + } + } + let _remove = Remove(path.clone()); + state::private_directory(&path).unwrap(); + let held = state::Lock::acquire(&path).unwrap(); + let pressure = Arc::new(AtomicU32::new(2)); + let changed = Arc::clone(&pressure); + let worker = std::thread::spawn(move || { + std::thread::sleep(Duration::from_millis(100)); + changed.store(4, Ordering::SeqCst); + drop(held); + }); + let prelease = [report(pressure.load(Ordering::SeqCst))]; + let owner = path.join("owner.json"); + let checked = startup_admission_lease( + &path, + Duration::from_secs(2), + Profile::Development, + &prelease, + || { + assert!( + matches!(state::Lock::acquire(&path), Err(error) if error.code == "provider_busy") + ); + Ok(report(pressure.load(Ordering::SeqCst))) + }, + ) + .map(|(_lease, _samples)| fs::write(&owner, b"must-not-create-owner").unwrap()); + worker.join().unwrap(); + assert!(matches!(checked, Err(error) if error.code == "admission_rejected")); + assert!(!owner.exists()); + // A failed fresh sample does not leave a lease or grant allocation authority. + assert!(state::Lock::acquire(&path).is_ok()); +} diff --git a/packages/runtime-core/src/provider/profile.rs b/packages/runtime-core/src/provider/profile.rs index b9d26ca4e..7e5d446a8 100644 --- a/packages/runtime-core/src/provider/profile.rs +++ b/packages/runtime-core/src/provider/profile.rs @@ -43,6 +43,21 @@ impl Profile { Self::Development => (u64::from(self.memory_mib()) + 4096) * 1024 * 1024, } } + pub(super) fn provider_memory_budget_bytes(self) -> u64 { + (u64::from(self.memory_mib()) + 2048) * 1024 * 1024 + } + pub(super) fn operating_host_reserve_bytes( + self, + pressure: super::admission::MemoryPressureState, + ) -> u64 { + if self == Self::Development && pressure == super::admission::MemoryPressureState::Warning { + // Keep the full provisional overhead + interactive-host reserve + // when macOS warns; the guest's allocation has already been charged. + self.minimum_free_memory_bytes() - u64::from(self.memory_mib()) * 1024 * 1024 + } else { + 2 * 1024 * 1024 * 1024 + } + } pub fn qualification(self) -> &'static str { match self { Self::Research => "WU02-live-qualification-pending",