De-confound CSI: drop structureless null frames (occupancy-blind fix)#940
Closed
williammalone wants to merge 2 commits into
Closed
De-confound CSI: drop structureless null frames (occupancy-blind fix)#940williammalone wants to merge 2 commits into
williammalone wants to merge 2 commits into
Conversation
…rect CSI header offsets After external IPEX antennas were added to the ESP32-S3 mesh nodes, a confirmed-empty room read "present" indefinitely. Two root-cause bugs: 1. motion_score saturation. `variance_motion` and `mbp_motion` used fixed divisors (/10, /25) calibrated for the antenna-less regime. Antennas raised amplitudes ~5x and these amplitude^2 energies ~30x, pinning both terms at the 1.0 clamp — so raw_motion could not fall near the presence floor and the adaptive baseline subtraction in smooth_and_classify was defeated. Normalize both by signal power (mean_amp^2) — the same dimensionless sqrt-of-power-ratio form already used by temporal_motion_score — making motion_score amplitude-scale-invariant. This fixes the single shared extract_features_from_frame used by BOTH the aggregate and the per-node paths, so room-level presence benefits too. (csi.rs carries the identical change in its dead mirror copy to keep the two in sync.) 2. parse_esp32_frame header offsets were 2 bytes early vs the firmware layout (csi_collector.c csi_serialize_frame: seq @ [12..16), rssi @ [16], noise_floor @ [17]). rssi was decoded from sequence-counter byte 2 — which stays 0 for the first 65,536 frames — yielding an impossible rssi=0 dBm that zeroed the RSSI fusion weights and the SNR-based signal_quality. The I/Q payload at byte 20 was already correct (CSI_HEADER_SIZE == 20), so amplitude-derived features were unaffected. Adds regression tests asserting motion_score is amplitude-scale-invariant and that a quiet high-amplitude signal does not saturate. Full binary suite green (103 tests). Validated live on the 2-node mesh: RSSI now reports real values (-28..-74 dBm, was 0) and an empty room now produces genuine low-motion frames. A residual over-read remains (real multi-subcarrier CSI reads elevated even when empty) — that intrinsic empty-vs- still ambiguity needs a learned reference (adaptive classifier retrain), tracked separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…line Diagnosed against real 2-node mesh data: the occupancy-blind bimodality is NOT a subcarrier-width artifact (frames are a fixed width at any instant) nor a transmitter/RSSI split (RSSI is identical across modes). It is a *frame-structure* split — promiscuous capture interleaves ~50% structureless "null" frames (near-constant amplitude vectors, spatial CoV² ≈ 0) with real CSI frames (CoV² ~0.4) at the SAME width and RSSI. Diffing a null frame against a real one in temporal_motion_score fabricates large fake "motion", which is what makes presence bimodal and occupancy-blind. Fix: a per-node adaptive gate (`admit_frame_structure`) admits a frame into the feature pipeline only if its spatial CoV² is >= DECONFOUND_FRACTION (0.1) of the node's rolling-P95 CoV². No brittle absolute threshold — it self-calibrates per node and is scale-invariant. Gates BOTH the global and per-node frame_history so the person-count fallback stays consistent too. Excluded frames are counted (`off_structure_frames`). Enabled by default; disable with SENSING_DECONFOUND=0. Supersedes the static-MAC `filter_mac` approach, which starved CSI on the mesh, and the modal-width gate, which is a no-op here (width is fixed at any instant). Tests (3): CoV² is zero for flat / large for structured / scale-invariant; the gate excludes null frames after warmup and counts them; and gating collapses the fabricated temporal motion of an alternating null/rich stream (>10x reduction). Full binary suite green (127 passed). Open: real-occupancy validation (does de-confounding make presence track people?) needs raw-frame capture through the running server — the recordings only preserve a processed amplitude, not raw frames, so it could not be confirmed offline yet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Server-side fix for the occupancy-blind CSI bimodality, diagnosed against real 2-node mesh data.
Root cause (corrected): the bimodal, occupancy-blind presence signal is not a subcarrier-width artifact (frames are a fixed width at any instant) nor a transmitter/RSSI split (RSSI is identical across the two modes). It's a frame-structure split — promiscuous capture interleaves ~50% structureless null frames (near-constant amplitude vectors, spatial CoV² ≈ 0) with real CSI frames (CoV² ~0.4) at the same width and RSSI. Diffing a null frame against a real one in
temporal_motion_scorefabricates large fake "motion" → bimodal, occupancy-blind features.This supersedes two earlier dead-ends: the static-MAC
filter_macapproach (starved CSI on the WiFi-7 mesh) and a modal-width gate (a no-op here, since width is fixed at any instant).The fix
A per-node adaptive structure gate (
admit_frame_structure): admit a frame into the feature pipeline only if its scale-invariant spatial dispersion (CoV² = var/mean²) is ≥DECONFOUND_FRACTION(0.1) of the node's rolling-P95 CoV². No brittle absolute threshold — self-calibrates per node. Gates both the global and per-nodeframe_history(so the person-count fallback stays consistent). Excluded frames counted inoff_structure_frames. Default on;SENSING_DECONFOUND=0to disable.Tests
3 new (
deconfound_tests): CoV² is zero-for-flat / large-for-structured / scale-invariant; the gate excludes null frames after warmup and counts them; and gating collapses the fabricated temporal motion of an alternating null/rich stream (>10× reduction). Full binary suite green: 127 passed.Open / not yet validated
Real-occupancy validation (does de-confounding make presence actually track people?) needs raw-frame capture through the running server — the recordings only preserve a processed amplitude vector, not raw frames, so it couldn't be confirmed offline. That's the remaining step (needs a node streaming to a server running this build).
🤖 Generated with Claude Code