@@ -113,7 +113,7 @@ bare `ipcRenderer.invoke`, `fetchWithRetry` has no per-request deadline, and
113113whisper-server's 30 s readiness budget only starts once the download resolved.
114114
115115The word aligner (§ Word-level alignment, step 5) is not part of that phase: it
116- is fetched in the background the first time a transcription detects English or
116+ is fetched in the background the first time a transcription on a GPU detects English or
117117French (109 or 348 MB), and no chunk ever waits for it.
118118
119119Three edges make that promise hold, and each is load-bearing:
@@ -261,7 +261,7 @@ it verbatim in the response.
261261 the same thing as whisper.cpp's ` t_dtw ` matching faster-whisper's word
262262 ** end** (` tools/stt-eval/whispercpp-dtw-poc/REPORT.md ` ).
2632635 . ** Re-time the words on a CTC aligner** (issue #948 , phase 3) — when the
264- language whisper detected has one , a wav2vec2 model fine-tuned for CTC scores
264+ chunk ran on a GPU and the language whisper detected has an aligner , a wav2vec2 model fine-tuned for CTC scores
265265 every 20 ms frame of the speech against every letter, and
266266 [ ` electron/stt/ctcAlign.ts ` ] ( ../../electron/stt/ctcAlign.ts ) forces
267267 whisper's own words through those scores:
@@ -287,12 +287,17 @@ it verbatim in the response.
287287 midpoint. Next to a pause (a gap of 100 ms or more between letters) the
288288 edges move further out, 60 ms before and 150 ms after, both capped at
289289 mid-pause: early there is silence, late is an audible attack.
290- - ** Fallback.** No aligner for the language, a download not landed yet or
290+ - ** GPU only.** On the CPU the forward pass would add 29% (English) to 58%
291+ (French) to a transcription, against 9 to 14% on Vulkan, for a gain over
292+ step 2's character-level DTW that is not worth that there. A chunk whisper
293+ ran on the CPU skips this step, and a CPU-only machine never downloads the
294+ model.
295+ - ** Fallback.** CPU, no aligner for the language, a download not landed yet or
291296 failed, a helper without ` /emissions ` , a stretch whose letters do not fit in
292297 its frames: the words keep the times of step 4. Step 6 runs either way.
2932986 . ** Anchor phrase edges on the speech** —
294299 [ ` electron/stt/snapWordBoundaries.ts ` ] ( ../../electron/stt/snapWordBoundaries.ts )
295- leaves boundaries inside a phrase alone: they are within ~ 16 ms of the audio
300+ leaves boundaries inside a phrase alone: they are within ~ 15 ms of the audio
296301 already, and an energy snap on top of them drags correct boundaries early.
297302 The edges of a phrase are different. The token before a phrase's first word
298303 is the previous phrase's last one, so that word starts in the pause, as
@@ -329,21 +334,29 @@ themselves about 10 to 20 ms from a human's.
329334| ---| ---| ---| ---| ---|
330335| First-token start + 150 ms RMS snap + VAD edges (before #948 ), TTS | 105 / 275 ms / 32% | 83% | 5% | 84% |
331336| Previous-token start + two-way VAD edges (phase 1), TTS | 31 / 125 ms / 64% | 89% | 19% | 88% |
332- | + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 96% |
333- | + CTC aligner (phase 3), TTS noisy | 15 / 39 ms / 95% | 93% | 48% | 93% |
334- | Phase 1, LibriSpeech | 40 / 145 ms / 55% | 67% | 14% | 71% |
335- | + CTC aligner (phase 3), LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
336-
337- With the aligner, deleting one word leaves on average 17 ms of it audible and
338- cuts 15 ms into its neighbours (TTS clean; 20 and 13 ms on LibriSpeech), against
339- 40 and 44 ms before. The misses left on phrase-initial starts are mostly the
340- reference's: the French TTS voices start a word on its silent stop closure,
341- which no aligner hears. On a real French take (25 s, no reference) checked on
342- its spectrogram, the aligner moved six boundaries by more than 40 ms: three
343- onto a visible boundary that phase 1 had missed by 65 to 250 ms (e.g.
344- "montrer | ça" from inside ` montrer ` to the onset of the /s/), one the right
345- way but still about 150 ms early (a drawn-out "Bah… voilà"), and two between
346- vowels, where the spectrogram cannot tell.
337+ | Same, character-level DTW (phase 2), TTS clean | 17 / 60 ms / 86% | 89% | 39% | 94% |
338+ | + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 95% |
339+ | Phase 2, TTS noisy | 16 / 60 ms / 86% | 88% | 40% | 92% |
340+ | + CTC aligner, TTS noisy | 15 / 39 ms / 95% | 94% | 48% | 93% |
341+ | Phase 2, LibriSpeech | 20 / 65 ms / 82% | 63% | 35% | 81% |
342+ | + CTC aligner, LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
343+
344+ Per language, character-level DTW gives French 19 ms median and 83% within
345+ 50 ms (was 26 ms, 70%) and English 15 ms and 89% (was 40 ms, 57%). The aligner
346+ takes French to 14 ms and 95% (P90 66 → 36 ms, clean cuts 33% → 47%) and
347+ English to 15 ms and 96% (P90 50 → 35 ms, clean cuts 45% → 52%). Deleting one
348+ word then leaves 17 ms of it audible and cuts 15 ms into its neighbours (TTS
349+ clean; 20 and 12 ms on LibriSpeech), against 19 and 24 ms with phase 2 alone.
350+ The misses left on phrase-initial starts are mostly the reference's: the French
351+ TTS voices start a word on its silent stop closure, which no aligner hears.
352+
353+ On a real French take (25 s, no reference) read on its spectrogram, the
354+ aligner is not uniformly better than phase 2: it moved "c'est" onto the onset
355+ of its /s/ (phase 2 was 90 ms early), but put "quoi" after "c'est" 70 to 80 ms
356+ early twice, inside the vowel of * c'est* , where phase 2 had it on the /k/
357+ closure. French silent final letters are the likely cause: the model was
358+ trained on spelling, and the ` t ` of * c'est* has no sound to sit on (phase 2
359+ drops such letters before aligning; this pass does not yet).
347360
348361A +15 ms calibration offset on every boundary gained 4 points of inner
349362boundaries within 50 ms but dropped noisy phrase deletes to 82%, so it is not
@@ -896,12 +909,15 @@ it deletes data
896909 in ` electron/native/whisper-stt/src/main.cpp ` are exercised only at
897910 runtime.
898911- ** Word timing without an aligner.** Phrase edges sit on the VAD (step 6),
899- and English and French words are re-timed on a CTC aligner (step 5). Every
900- other language keeps whisper-small's DTW: about 30 ms median and 125 ms P90
901- inside a phrase, so one single-word delete in five is clean. A multilingual
902- CTC model (` facebook/omniASR-CTC-300M ` ) would cover them, at a cost per
912+ and on a GPU English and French words are re-timed on a CTC aligner (step 5).
913+ Every other language, and every language on the CPU, keeps whisper-small's
914+ character-level DTW: about 16 ms median and 60 ms P90 on synthetic speech, so
915+ about two single-word deletes in five are clean. A multilingual CTC model
916+ (` facebook/omniASR-CTC-300M ` ) would cover the other languages, at a cost per
903917 language family nobody has measured yet (issue #948 ).
904- - ** The aligner on the CPU.** On the GPU it adds about 10% to a transcription.
905- On the CPU fallback (16 threads, Ryzen 7 5800X) it adds 28% in English and
906- 42 to 61% in French: the French model is a 24-layer wav2vec2 large, and its
907- forward pass costs about half of whisper-small's.
918+ - ** The aligner is GPU-only.** On the CPU fallback (16 threads, Ryzen 7 5800X)
919+ it would add 29% in English and 58% in French: the French model is a 24-layer
920+ wav2vec2 large, about half of whisper-small's cost on its own.
921+ - ** French silent letters.** The aligner spells * c'est* with its ` t ` , which
922+ has no sound; on a real take that put the next word 70 to 80 ms early.
923+ Dropping silent final consonants before aligning, as step 2 does, is untried.
0 commit comments