Eager connect watchdog: mitigate InPlay/iPhone-16 connection wedges - #133
Open
ps2 wants to merge 19 commits into
Open
Eager connect watchdog: mitigate InPlay/iPhone-16 connection wedges#133ps2 wants to merge 19 commits into
ps2 wants to merge 19 commits into
Conversation
InPlay-firmware DASH pods (peripheral name "InPlay BLE") silently ignore the LL CONNECTION_PARAM_REQ control PDU; on iPhone 16-class controllers iOS often issues that early in a connect, deadlocking the LL procedure queue so the connect wedges in .connecting with no callback (~7s until the pod terminates the dead link, then iOS silently auto-retries — chains of invisible ~20s stalls, and the wedged pod stops advertising so it's invisible to discovery scans). Add an eager cancel/retry strategy, gated to affected phones (hw.machine iPhone17,x / iPhone18,x) and InPlay/unknown pods (shouldUseEagerConnect), default on and fully UserDefaults-tunable: - Connect watchdog: if a connect hasn't reached .connected within ~3s (3-5x the measured healthy <1s population), presume the wedge — cancelPeripheralConnection (tears it down on-air, freeing the pod to advertise + re-arming iOS's fast connection scan), wait ~200ms, re-connect; repeat until a bounded budget. Runs under the existing runCommand .connect wait, which clears only on a real didConnect. - On-demand command connects to affected pods go direct (retrievePeripherals + plain connect, no fresh-discovery scan) + watchdog. - Pairing/discovery connects (timedConnect, incl. didDiscover new-pod) arm the watchdog — fixing the existing bug where a timed-out pairing connect was abandoned WITHOUT cancelling, leaving iOS re-wedging the pod and blinding rediscovery. - didDisconnect / didFailToConnect defer to the watchdog while it owns a connect, so their reconnect paths don't race it. - Telemetry: a distinct device-log event when the watchdog fires with state==connecting (pathognomonic) tagged with the peripheral name, to measure prevalence and confirm the 3s threshold never clips healthy connects. Non-affected phones / known-non-InPlay pods are unchanged (zero regression).
The affected controllers are the iPhone 16 family (all variants incl. 16e = iPhone17,x) and the iPhone 17e (iPhone18,5). The rest of the iPhone 17 family (iPhone18,1-18,4) is not affected — drop the overbroad iPhone18, prefix match.
…ness When an InPlay-variant DASH pod (usingInPlayPod) is paired on an affected iPhone model (iPhoneWithPossibleInPlayIssues: iPhone 16 family / iPhone 17e), show a standing notice row in pod settings — 'Slower Connections Expected' — with a NavigationLink to a detail view explaining the firmware bug, what to expect (automatic stall detection/retry, occasional ~30s connects, extra pairing attempts), and that insulin delivery is unaffected.
… racing connects - Watchdog 3s -> 2s: healthy connects are sub-second (ATT ~90-280ms after capture on-air; 0-1s app-level) and a cancel/retry cycle is ~1.3s, so 2s keeps ~2x margin while wasting ~1s less per wedge. A false trip costs ~1s. - Budget 18s -> 28s and connectOnDemand runCommand timeout 20s -> 30s: 13/35 locked- phone attempts exhausted 18s (~6 cycles) on 2026-08-19; 28s at the ~2.4s cycle gives ~11 attempts. - beginCommandConnect dedupe: racing sessions produced doubled command connects seconds apart; when the watchdog already owns an in-flight .connecting attempt, refresh its budget instead of issuing a duplicate connect.
…e pods Field failure (2026-08-19): after a pairing attempt was abandoned mid-watchdog-cycle, the last re-issued connect was left pending. Because the pairing scan hears a pod once per scan (no allowDuplicates), every subsequent discoverPods logged 'heard pod ... pairable=true state=1' (.connecting) and declined to connect — pairing could never succeed until the zombie cleared. In the didDiscover pairing branch: a pairable pod heard while .connecting with NO active watchdog cannot be in a live connection (we just heard it advertise) — cancel the stale connect and reconnect fresh (re-arming the watchdog) after teardown.
…pods Field data shows per-attempt wedge probability is per-pod: ~55% on one InPlay pod, ~84% on another (sniffer-confirmed: every watchdog retry recaptures on-air within ~0.7-1.6s and real wedges follow — the 2s watchdog is not clipping reacquisition). At 84%, the 28s budget (~12 cycles) measured ~10% command failure, all budget exhaustions at wd=12. 40s (~17 cycles) predicts ~5%; failures self-heal next loop cycle. Pairing budget raised to match.
Collaborator
|
There already is an existing UIDevice extension (see Common/UIDevice.swift) that returns a user friendly string for the iPhone model instead of the non-obvious Apple model number (e.g., "iPhone 17e" instead of "iPhone18,5") that should probably be used instead of BluetoothManager new deviceModelIdentifier. Also OmniPumpManager's iPhoneWithPossibleInPlayIssues var does similar testing as BluetoothManager's new isEagerConnectDeviceModel let var and OmniPumpManager's usingInPlayPod var does similar testing as the BluetoothManager's new shouldUseEagerConnect. It would be good to rework things for more sharing for these things. |
Review feedback (itsmojo): the eager-connect gate duplicated existing detection. - UIDevice.hasPossibleInPlayBLEIssues (Common/UIDevice.swift) is now the single affected-model predicate (iPhone 16 family + iPhone 17e), built on the existing UIDevice.modelName mapping; BluetoothManager's parallel hw.machine-based deviceModelIdentifier/isEagerConnectDeviceModel are removed and both the eager gate and OmniPumpManager.iPhoneWithPossibleInPlayIssues use the shared predicate. - OmniPumpManager.usingInPlayPod now matches BluetoothManager.inPlayPeripheralName instead of a second hardcoded string. Placed in Common so the Bluetooth layer doesn't depend on the pump manager.
Collaborator
|
I like the consolidation changes, but we could go a bit further. With the new UIDevice.hasPossibleInPlayBLEIssues var, there isn't any need for an OmniPumpManager.iPhoneWithPossibleInPlayIssues var. I'd suggest just removing this var's definition from OmniPumpManager and replacing its two uses with |
…nects Pass EnableAutoReconnect (iOS 17+, tunable OmnipodKit.eagerAutoReconnectEnabled, default on) on eager direct connects and watchdog re-issues, to probe whether it changes the stack's reacquisition behavior on wedge-prone InPlay pods. Adopt centralManager(_:didDisconnectPeripheral:timestamp:isReconnecting:error:) (called instead of the classic callback when implemented): both delegate methods route into a shared handleDisconnect. isReconnecting=true logs distinct [autoReconnect] telemetry (with drop age) and defers entirely to the system's re-establishment — no app-side reconnect or probe re-arm; didConnect completes it. The eager watchdog stays armed as a bounded supervisor; its cancel also cancels a pending system auto-reconnect before re-issuing a supervised connect. Explicit cancels (idle-disconnect) cancel auto-reconnect, preserving the normally- disconnected model.
Use UIDevice.hasPossibleInPlayBLEIssues directly at both former call sites (pod keep-alive defaulting during pairing, settings advisory view model) and update the predicate's comment accordingly.
…ds, 60s) Field data: with the 4s idle-disconnect, one loop cycle's status→compute→dose burst (sessions ~10-25s apart) paid 2-3 wedge storms — the idle window discarded a working connection seconds before the next session needed it (observed: instant connect, temp basal, idle-disconnect at +4s, then a 19s storm 2s later for the next session). On eager-gated pods a reconnect risks a median ~10s / worst ~30s storm, so use a 60s idle window (tunable OmnipodKit.eagerIdleDisconnectSeconds) so the whole cycle shares one connection. The cycle still ends disconnected, so the background heartbeat probe re-arms normally.
The 60s window hung up ~1 min before the next ~3-min loop cycle every time, paying a wedge storm per cycle anyway. 240s holds the connection across cycles while looping; if cycles stop, the pod still disconnects and the heartbeat probe re-arms.
Cycle cadence varies 3-5 min; 60s and 240s windows both repeatedly hung up under a minute before the next cycle, paying a wedge storm each time (pcap: every on-air terminate is iPhone-initiated — the pod never disconnects us). Hold the link as long as any command lands within the hour; a true idle hour still releases the pod.
Track peripherals with isReconnecting=true pending and log '[autoReconnect] link re-established by system after Xs' when didConnect completes it — the key observable for the EnableAutoReconnect experiment (pod-side inactivity disconnects with Pod Keep Alive disabled).
Previously only isReconnecting==true reached the device log, so an Issue Report showing no [autoReconnect] events was ambiguous between 'iOS never called the new signature' and 'called with isReconnecting=false'. Log both cases.
The user may want to bolus the moment the app opens, so foreground connects get: - A shorter watchdog interval (eagerConnectForegroundWatchdogSeconds, 1.5s) — a wedge can't be waited out and a retry cycle costs ~1.3s, so retrying harder strictly reduces time-to-connect. - Action on a .connecting peripheral, not just .disconnected: previously foregrounding did nothing while a slow reacquisition or wedge was in flight (the exact 'user opens app and waits' case). Now re-arm the watchdog on the foreground interval, or cancel an unsupervised in-flight connect and restart the eager cycle. - Telemetry '[foreground] connected Xs after foregrounding' — the user-visible metric for how long the app couldn't talk to the pod after being opened.
… in foreground Field data showed we tore down a working connection on EVERY backgrounding (69 of 73 transitions in one 6h report) and then paid a wedge storm to get it back — on a pod that releases the link itself after ~180s anyway. Split the strategy by app state for eager-gated pods: - BACKGROUND: never tear down (shouldHoldConnection now true for these pods), and reconnect drops with a standing connect carrying EnableAutoReconnect, so the system restores the link (~27s median measured) without app CPU while suspended. - FOREGROUND: no auto-reconnect — the user may be waiting to bolus and the system's silent reacquisition is far slower than the eager cancel/retry cycle (~1.3s). Any drop while foregrounded reconnects eagerly with the fast watchdog. Non-eager pairings are unchanged.
The eager mitigation holds the pod connected, while the heartbeat's StartDelay probe requires a DISCONNECTED pod — the two connection modalities are mutually exclusive, so these combos cannot provide pump-driven background wake-ups. Rather than silently dropping a capability the host asked for, surface it: - BluetoothManager.isBLEHeartbeatRequested / BlePodComms passthrough / OmniPumpManager.bleHeartbeatUnsupportedForThisPod (InPlay + affected iPhone + heartbeat requested). - Pod settings shows a 'Pump Heartbeat Unavailable' notice explaining that looping relies on the CGM, linking to a new 'Pump Heartbeat' section in the info view.
When a host needs pump-provided background wakes on a wedge-prone combo, the usual StartDelay probe can't be used (it needs a disconnected pod). Rather than leaving those setups with no heartbeat, use link drops as the wake source: - In this mode do NOT pass EnableAutoReconnect — the system silently re-establishing the link robs us of the wake (and produced an irregular ~9min cadence in the field). - Let the pod hang up on its own ~180s inactivity timer; CoreBluetooth delivers didDisconnect even to a suspended app (State Restoration), which is the wake. - Fire the heartbeat on that drop, then eagerly reconnect — the fresh connection re-arms the pod's timer, so the next hangup is the next wake: a self-sustaining ~3min cadence. Throttled by eagerHeartbeatMinIntervalSeconds (150s) so a wedge storm's own watchdog cancels can't each count as a wake; optional CGM-staleness gate (eagerHeartbeatStaleReadingSeconds, default 0 = always fire). Settings advisory reworded from 'Pump Heartbeat Unavailable' to 'Reduced Background Wake-Ups' to match the new behavior.
… title #134) Two conflicts, both straight combinations: - shouldHoldConnection: keep the eager-pod hold, use #125's podKeepAliveKeepsConnectedInBackground accessor for the PKA case. - Pairing keep-alive auto-enable: keep UIDevice.hasPossibleInPlayBLEIssues (the consolidated predicate) with #125's state.podKeepAlive storage. #125 still referenced the iPhoneWithPossibleInPlayIssues var removed on this branch, so this resolution is required for the merged tree to compile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
InPlay-firmware DASH pods (CoreBluetooth peripheral name
"InPlay BLE") silently ignore theLL_CONNECTION_PARAM_REQlink-layer control PDU instead of replyingLL_UNKNOWN_RSP. On iPhone 16-class controllers iOS often issues that procedure early in a connect; when it does, the LL procedure queue deadlocks and the app gets no callback at all — nodidConnect, no error,CBPeripheral.statestuck at.connecting. The pod terminates the dead link after ~7 s and iOS silently auto-retries, producing chains of invisible ~20 s wedges. A wedged pod also stops advertising, so it's invisible to any concurrent discovery scan (→ "no pods found" when pairing).Root cause is pod firmware — not fixable app-side. This caps its cost.
Sniffer-confirmed (nRF captures, 2026-08-19/20): every wedge is preceded by an ignored
LL_CONNECTION_PARAM_REQ, and per-attempt wedge probability is per-pod — ~55% and ~84% measured on two different InPlay pods.Results (field, iPhone 16 Pro + InPlay pod, phone locked)
The metric that matters is CGM-driven loops that reach the pump:
Local CGM (Libre, CGM-driven loops):
11.3 h in range, 137 CGM readings, zero missed loops, fresh InPlay pod. Foreground opens: 50% find the link already connected; foreground loops 13/13 at 1 s median.
Network CGM (no local CGM wakes — pump heartbeat required), 18 h:
Design: two connection modalities
Affected combos only (
shouldUseEagerConnect: feature on, iPhone 16 family or 17e, pod is InPlay or not-yet-known). Everything else is unchanged.Foreground — app-driven, fast. The user may be waiting to bolus, so we never wait out a wedge: direct connect + a 1.5 s watchdog that cancels and retries.
cancelPeripheralConnectiontears a wedge down on-air immediately (capture-verified:LL_TERMINATE_IND, pod re-advertises ~10 ms later) and re-arms iOS's fast connection scan. Auto-reconnect is deliberately not used here — the system's silent reacquisition (~27 s median measured) is ~20× slower than our retry cycle (~1.3 s). Foregrounding also acts on a.connectingperipheral (previously it only handled.disconnected, so a wedge in flight meant the user just waited).Background — system-driven, suspend-proof. Never tear down (previously we cancelled a working link on every backgrounding — 69 times in one 6 h report — then paid a wedge storm to get it back). Drops are answered with a standing
connect()carryingCBConnectPeripheralOptionEnableAutoReconnect, so the system restores the link with no app CPU — the only mechanism that works while suspended. This matters because the pod hangs up on its own ~180 s inactivity timer (median measured hold: 182 s) and a suspended app cannot send the keep-alive traffic that would prevent it.Background, when the host needs pump-provided wakes — disconnect-driven heartbeat. The usual heartbeat probe uses
CBConnectPeripheralOptionStartDelayKey, which requires a disconnected pod, so it is mutually exclusive with holding the link. Rather than leaving these setups with no background wake at all, this mode inverts the problem: auto-reconnect is deliberately not used (the system silently re-establishing the link steals the wake — measured cadence was an unusable ~9 min, irregular), the pod is allowed to hang up on its own ~180 s inactivity timer, CoreBluetooth deliversdidDisconnecteven to a suspended app (State Restoration), and that drop is the wake: fire the heartbeat, then eagerly reconnect, which re-arms the pod's timer so the next hangup becomes the next wake. Self-sustaining, measured 3.2 min median cadence over 18 h. Throttled (eagerHeartbeatMinIntervalSeconds) so a wedge storm's own watchdog cancels can't each count as a wake.Pairing/discovery connects are watchdog-supervised too, fixing an existing bug where a timed-out pairing connect was abandoned without cancelling — leaving iOS re-wedging the pod and blinding rediscovery (observed in the field as pairing that could never succeed until the app restarted).
User-visible
Two persistent notices in pod settings (shown only on affected combos), with a shared detail view:
Tunables (all
UserDefaults, settable via scheme launch arguments)eagerConnectEnabled(true) ·eagerConnectForceAllDevices(false, bench A/B) ·eagerConnectWatchdogSeconds(2) ·eagerConnectForegroundWatchdogSeconds(1.5) ·eagerConnectTeardownSeconds(0.2) ·eagerConnectBudgetSeconds(40) ·eagerPairingBudgetSeconds(40) ·eagerIdleDisconnectSeconds(3600) ·eagerAutoReconnectEnabled(true) ·eagerHeartbeatMinIntervalSeconds(150) ·eagerHeartbeatStaleReadingSeconds(0 = always fire)Thresholds are field-derived: healthy connects complete sub-second (ATT ~90–280 ms after capture on-air), a cancel/retry cycle is ~1.3 s, and at an 84% wedge rate a 28 s budget measured ~10% command failure while 40 s (~17 cycles) predicts ~5%.
Telemetry
Distinct device-log events for prevalence measurement and threshold validation:
[eager] watchdog fired state=N name=…(state 1 =.connectingis pathognomonic for the wedge),[eager] drop while background/foreground …,[foreground] connected Xs after foregrounding,[heartbeat] firing on link drop …,[autoReconnect] ….Note on
EnableAutoReconnect: iOS never invokeddidDisconnectPeripheral(timestamp:isReconnecting:error:)in 22 h of testing (verified by logging every invocation), soisReconnectingwas never observable — yet pod hangups were followed by reconnects with no app-side connect issued, so the option does appear to be doing the work silently.Notes
UIDevice.hasPossibleInPlayBLEIssuesis the single affected-model predicate andBluetoothManager.inPlayPeripheralNamethe single pod-name constant.OmniTestsbuilds clean; all field results above are from device builds of this branch.