Skip to content

Ambient ASR garbage — background audio (TV/podcast) feeds Whisper as fake user input #105

Open

Description

@BrettKinny

Context

Found during the #44 LED bench sweep on 2026-05-23, looking at xiaozhi-server logs.

A continuous stream of Whisper ASR transcriptions are being treated as user messages and forwarded to the LLM, even though no human was speaking to Dotty. Source was background audio (TV / podcast playing in the room).

Evidence

xiaozhi-server log fragment, single ~12 minute window:

```
20:27:07 大模型收到用户消息: {"content": "When somebody focuses on an external enemy, immigrants, other races, trans people..."}
20:28:24 大模型收到用户消息: {"content": "which I think are really old fashioned social democrats still are not yet practical..."}
20:28:27 大模型收到用户消息: {"content": "Let's do whatever it is."}
20:28:32 大模型收到用户消息: {"content": "possible. But, uh, uh, there is..."}
20:29:01 大模型收到用户消息: {"content": "I intended to call Trump a very specific kind of party..."}
20:29:13 大模型收到用户消息: {"content": "I like to cut the action in another way. You know Trump likes to attack the left..."}
20:29:32 大模型收到用户消息: {"content": "Sorry but if there ever was a historicist relativist, it's Trump..."}
20:31:03 大模型收到用户消息: {"content": "So in 2016, you said that a top victory might be for the best in America..."}
20:38:00 ASR result: "one of the ways of how to talk a lot and pretending to change things without really changing society…" (4-paragraph political monologue)
```

Each of these triggered an LLM chat turn — pumping background podcast content into Dotty's conversation history.

Why this is bad

  1. TTS race: every spurious chat turn overwrites `conn.sentence_id`, which (per say bypass races chat turns under continuous ASR — sentence_id arbitration #104) silently drops server-pushed greetings (`face_greeter` say bypass, ProactiveGreeter, etc.).
  2. Persona drift: the LLM's "Speaking with Brett" turn history gets polluted with disconnected paragraphs about fascism / political theory / podcast snippets, which can degrade later genuine responses.
  3. Persona PII leakage risk: if the LLM context includes any household member's personal info, Dotty might respond to a podcast's tangential content with personalised greetings or memory writes (e.g. `remember` tool calls on bogus user input).
  4. Kid-safety surface: ambient audio is unfiltered by the kid-mode content rules (those apply to LLM output, not ASR input).
  5. Battery + cost: continuous WS turns + LLM calls + (when VLM_API_KEY is set) photo captures cost real money and shorten battery life.

Suspected mechanisms

  • VAD threshold too low / too sensitive — VAD is firing on any voice-band audio above the noise floor. SileroVAD's threshold should probably be tuned up, OR
  • No directional gating — the firmware emits `sound_event` with energy + direction; xiaozhi-server could refuse to start ASR on sounds that look like distant/diffuse audio (low single-direction confidence, low peak energy) vs. close-talk speech.
  • No wake-word gating on turn continuation — once a turn opens, subsequent ASR turns auto-fire without re-requiring a wake. Brief silence after a podcast pause looks like turn boundary.
  • No speaker recognition / voiceprint gate — `声纹识别URL未配置,声纹识别将被禁用` (voiceprint URL not configured) — speakers other than household members aren't filtered out.

Possible mitigations (cheapest first)

  1. Bump SileroVAD threshold (config-only). Test if fewer ambient triggers happen with a higher threshold; measure false-negative rate on real human speech.
  2. Energy floor on ASR — require ASR audio's RMS energy to exceed a threshold before forwarding to LLM. Background TV is loud-but-distant; close-talk speech has higher concentrated energy.
  3. Wake-word required between turns — after each LLM turn, require a fresh wake-word before allowing the next ASR turn. Loses conversational flow but reliably scopes Dotty to direct address.
  4. Wire up voiceprint provider — only forward ASR to LLM when the speaker matches a household voice_print_id.

Severity

P1 — actively burning cloud LLM tokens + producing garbage in conversation context whenever anyone has a TV / podcast / radio on near Dotty. Also blocks #104 from being user-visible: even if we fix the say race, Dotty's TTS will still get drowned out by ambient-triggered chat turns.

Related

Activity

  1. added
    bugSomething isn't working
    status:activeReady to start, not blocked
    area:xiaozhiUnraid xiaozhi-server container + custom providers
    on May 23, 2026
  2. BrettKinny commented on May 29, 2026

    @BrettKinny
    OwnerAuthor

    2026-05-29 — instrumentation deployed, awaiting calibration data run

    Took the "instrument first, then gate" approach. Added non-behavioral ASR-METRICS logging to custom-providers/asr/whisper_local.py speech_to_text — logs per ASR turn: RMS + peak energy, faster-whisper no_speech_prob + avg_logprob (mean across segments), language probability, segment count, and the transcribed text.

    Deployed to xiaozhi-server (docker compose restart, healthcheck clean, device reconnected, 15 MCP tools). Per-file rollback at whisper_local.py.bak-105instr-pre.

    Why: the cheap mitigations (VAD threshold bump, energy floor, whisper-confidence reject) can only be tuned against measured distributions — guessing a threshold risks rejecting real quiet speech. And there's a known ceiling: clearly-articulated TV speech produces genuine, confident transcriptions that energy/confidence gates can't separate from real address (only wake-word gating or voiceprint can).

    Next step (needs device time): capture a labelled data run — ~5–8 real close-talk turns (the "keep" set) + a few minutes of TV/podcast with no direct address (the "reject" set). Then grep ASR-METRICS the logs, compare the two clusters, and either set a calibrated reject gate or conclude the cheap signals are insufficient and escalate to wake-word/voiceprint.

    The instrumentation is logging-only and harmless if left in place; strip it once gate thresholds are chosen.

  3. added 2 commits that reference this issue on May 30, 2026
  4. BrettKinny commented on Oct 5, 2026

    @BrettKinny
    OwnerAuthor

    Two data points and two changes from a bench session on 2026-10-05:

    • Silence hallucination. 1.3 s of near-silence right after Dotty finished speaking was transcribed as "Thank you." (no_speech_prob 0.81, rms 0.004) and answered. About 45 real utterances that day all scored ≤ 0.32. f0e6cda adds a gate in whisper_local.py that drops transcripts above no_speech_threshold (default 0.6). Deployed; not yet seen rejecting one live.
    • Mic never closed. close_connection_no_voice_time was 86400 on the deployment, so after any conversation the mic stayed open and Dotty answered room speech indefinitely. With end_prompt.enable: false the old farewell problem no longer applies, so the deployment is back to 120 s: verified the session closes silently at 120 s and the device returns to idle with the mic closed.

    Neither addresses real background speech (TV/podcast) while a session is open; the ASR-METRICS distributions above may help choose an rms/logprob gate.

    AI-assisted: drafted by Claude (Opus 5.5) from a supervised bench session on the physical robot, 2026-10-05.

  5. BrettKinny commented on Oct 6, 2026

    @BrettKinny
    OwnerAuthor

    Maintainer decision, 2026-10-06 (recorded by Claude Opus 5.5 at the maintainer's direction): for the mic staying open in a busy room, the preferred direction is to close the session rather than keep listening, with touch to wake — not an always-open mic and not a wake word per conversation.

    Evidence from that morning: with the mic left open after a test, the robot produced 18 recognitions and 19 device aborts in three minutes from room/TV audio (no_speech 0.07–0.59, rms mostly 0.002–0.014, overlapping a quiet child's real speech at 0.007–0.017, so a level gate cannot separate them). With #175's turn lock those turns queue and are answered late, in order.

    Not implemented yet. Open sub-questions: how many noise-like turns before closing, and whether a queued turn should be skipped when something newer was said ("latest utterance wins").

  6. BrettKinny commented on Oct 6, 2026

    @BrettKinny
    OwnerAuthor

    Follow-up to the note above, 2026-10-06 (recorded by Claude Opus 5.5 at the maintainer's direction): the maintainer has also said yes to "latest utterance wins" — a queued turn should be skipped when something newer was said while it waited behind the #175 turn lock. Not implemented yet. Still open: how many noise-like turns before the session closes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:xiaozhiUnraid xiaozhi-server container + custom providersbugSomething isn't workingstatus:activeReady to start, not blocked

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions