Repository navigation
Ambient ASR garbage — background audio (TV/podcast) feeds Whisper as fake user input #105
Description
Context
Found during the #44 LED bench sweep on 2026-05-23, looking at xiaozhi-server logs.
A continuous stream of Whisper ASR transcriptions are being treated as user messages and forwarded to the LLM, even though no human was speaking to Dotty. Source was background audio (TV / podcast playing in the room).
Evidence
xiaozhi-server log fragment, single ~12 minute window:
```
20:27:07 大模型收到用户消息: {"content": "When somebody focuses on an external enemy, immigrants, other races, trans people..."}
20:28:24 大模型收到用户消息: {"content": "which I think are really old fashioned social democrats still are not yet practical..."}
20:28:27 大模型收到用户消息: {"content": "Let's do whatever it is."}
20:28:32 大模型收到用户消息: {"content": "possible. But, uh, uh, there is..."}
20:29:01 大模型收到用户消息: {"content": "I intended to call Trump a very specific kind of party..."}
20:29:13 大模型收到用户消息: {"content": "I like to cut the action in another way. You know Trump likes to attack the left..."}
20:29:32 大模型收到用户消息: {"content": "Sorry but if there ever was a historicist relativist, it's Trump..."}
20:31:03 大模型收到用户消息: {"content": "So in 2016, you said that a top victory might be for the best in America..."}
20:38:00 ASR result: "one of the ways of how to talk a lot and pretending to change things without really changing society…" (4-paragraph political monologue)
```
Each of these triggered an LLM chat turn — pumping background podcast content into Dotty's conversation history.
Why this is bad
- TTS race: every spurious chat turn overwrites `conn.sentence_id`, which (per say bypass races chat turns under continuous ASR — sentence_id arbitration #104) silently drops server-pushed greetings (`face_greeter` say bypass, ProactiveGreeter, etc.).
- Persona drift: the LLM's "Speaking with Brett" turn history gets polluted with disconnected paragraphs about fascism / political theory / podcast snippets, which can degrade later genuine responses.
- Persona PII leakage risk: if the LLM context includes any household member's personal info, Dotty might respond to a podcast's tangential content with personalised greetings or memory writes (e.g. `remember` tool calls on bogus user input).
- Kid-safety surface: ambient audio is unfiltered by the kid-mode content rules (those apply to LLM output, not ASR input).
- Battery + cost: continuous WS turns + LLM calls + (when VLM_API_KEY is set) photo captures cost real money and shorten battery life.
Suspected mechanisms
- VAD threshold too low / too sensitive — VAD is firing on any voice-band audio above the noise floor. SileroVAD's threshold should probably be tuned up, OR
- No directional gating — the firmware emits `sound_event` with energy + direction; xiaozhi-server could refuse to start ASR on sounds that look like distant/diffuse audio (low single-direction confidence, low peak energy) vs. close-talk speech.
- No wake-word gating on turn continuation — once a turn opens, subsequent ASR turns auto-fire without re-requiring a wake. Brief silence after a podcast pause looks like turn boundary.
- No speaker recognition / voiceprint gate — `声纹识别URL未配置,声纹识别将被禁用` (voiceprint URL not configured) — speakers other than household members aren't filtered out.
Possible mitigations (cheapest first)
- Bump SileroVAD threshold (config-only). Test if fewer ambient triggers happen with a higher threshold; measure false-negative rate on real human speech.
- Energy floor on ASR — require ASR audio's RMS energy to exceed a threshold before forwarding to LLM. Background TV is loud-but-distant; close-talk speech has higher concentrated energy.
- Wake-word required between turns — after each LLM turn, require a fresh wake-word before allowing the next ASR turn. Loses conversational flow but reliably scopes Dotty to direct address.
- Wire up voiceprint provider — only forward ASR to LLM when the speaker matches a household voice_print_id.
Severity
P1 — actively burning cloud LLM tokens + producing garbage in conversation context whenever anyone has a TV / podcast / radio on near Dotty. Also blocks #104 from being user-visible: even if we fix the say race, Dotty's TTS will still get drowned out by ambient-triggered chat turns.
Related
- say bypass races chat turns under continuous ASR — sentence_id arbitration #104 (say bypass races chat turns) — same root cause symptom-amplified
- CLAUDE.md notes that `声纹识别URL未配置` ships as a warning at startup but is treated as non-blocking
Activity
BrettKinny commentedon May 29, 2026
2026-05-29 — instrumentation deployed, awaiting calibration data run
Took the "instrument first, then gate" approach. Added non-behavioral
ASR-METRICSlogging tocustom-providers/asr/whisper_local.pyspeech_to_text— logs per ASR turn: RMS + peak energy, faster-whisperno_speech_prob+avg_logprob(mean across segments), language probability, segment count, and the transcribed text.Deployed to xiaozhi-server (
docker compose restart, healthcheck clean, device reconnected, 15 MCP tools). Per-file rollback atwhisper_local.py.bak-105instr-pre.Why: the cheap mitigations (VAD threshold bump, energy floor, whisper-confidence reject) can only be tuned against measured distributions — guessing a threshold risks rejecting real quiet speech. And there's a known ceiling: clearly-articulated TV speech produces genuine, confident transcriptions that energy/confidence gates can't separate from real address (only wake-word gating or voiceprint can).
Next step (needs device time): capture a labelled data run — ~5–8 real close-talk turns (the "keep" set) + a few minutes of TV/podcast with no direct address (the "reject" set). Then
grep ASR-METRICSthe logs, compare the two clusters, and either set a calibrated reject gate or conclude the cheap signals are insufficient and escalate to wake-word/voiceprint.The instrumentation is logging-only and harmless if left in place; strip it once gate thresholds are chosen.
#105: instrument ASR with per-utterance metrics (logging only)
Vision-alignment pass: pay down post-#36 / #115 drift (#129)
BrettKinny commentedon Oct 5, 2026
Two data points and two changes from a bench session on 2026-10-05:
whisper_local.pythat drops transcripts aboveno_speech_threshold(default 0.6). Deployed; not yet seen rejecting one live.close_connection_no_voice_timewas 86400 on the deployment, so after any conversation the mic stayed open and Dotty answered room speech indefinitely. Withend_prompt.enable: falsethe old farewell problem no longer applies, so the deployment is back to 120 s: verified the session closes silently at 120 s and the device returns to idle with the mic closed.Neither addresses real background speech (TV/podcast) while a session is open; the ASR-METRICS distributions above may help choose an rms/logprob gate.
AI-assisted: drafted by Claude (Opus 5.5) from a supervised bench session on the physical robot, 2026-10-05.
BrettKinny commentedon Oct 6, 2026
Maintainer decision, 2026-10-06 (recorded by Claude Opus 5.5 at the maintainer's direction): for the mic staying open in a busy room, the preferred direction is to close the session rather than keep listening, with touch to wake — not an always-open mic and not a wake word per conversation.
Evidence from that morning: with the mic left open after a test, the robot produced 18 recognitions and 19 device aborts in three minutes from room/TV audio (no_speech 0.07–0.59, rms mostly 0.002–0.014, overlapping a quiet child's real speech at 0.007–0.017, so a level gate cannot separate them). With #175's turn lock those turns queue and are answered late, in order.
Not implemented yet. Open sub-questions: how many noise-like turns before closing, and whether a queued turn should be skipped when something newer was said ("latest utterance wins").
BrettKinny commentedon Oct 6, 2026
Follow-up to the note above, 2026-10-06 (recorded by Claude Opus 5.5 at the maintainer's direction): the maintainer has also said yes to "latest utterance wins" — a queued turn should be skipped when something newer was said while it waited behind the #175 turn lock. Not implemented yet. Still open: how many noise-like turns before the session closes.