Skip to content

Commit 4c36990

Browse files
committed
feat(stt): run the word aligner on the GPU only, measured against phase 2
On top of character-level DTW, the CTC aligner still moves TTS inner starts from 17/60 ms (median/P90) to 14/35 ms and LibriSpeech from 20/65 to 15/45 ms, for +9% (English) to +14% (French) on Vulkan. On the CPU it would cost +29% to +58%, so a chunk whisper ran on the CPU keeps the DTW times, and a CPU-only machine never downloads the model.
1 parent e29deac commit 4c36990

5 files changed

Lines changed: 68 additions & 34 deletions

File tree

‎electron/stt/whisperServer.test.ts‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -422,11 +422,11 @@ describe("WhisperServerManager", () => {
422422
],
423423
};
424424

425-
async function run(emissionsReply: Response) {
425+
async function run(emissionsReply: Response, reply: object = inference) {
426426
const fetchMock = vi.fn(async (url: string, _init?: RequestInit) =>
427427
url.endsWith("/emissions")
428428
? emissionsReply
429-
: new Response(JSON.stringify(inference), { status: 200 }),
429+
: new Response(JSON.stringify(reply), { status: 200 }),
430430
);
431431
vi.stubGlobal("fetch", fetchMock);
432432
try {
@@ -462,6 +462,19 @@ describe("WhisperServerManager", () => {
462462
expect(result.timing?.rtf).toBeCloseTo(0.3);
463463
});
464464

465+
it("leaves the aligner out on the CPU, without even asking for it", async () => {
466+
const { result, alignerFor, fetchMock } = await run(
467+
new Response(JSON.stringify(emissions), { status: 200 }),
468+
{ ...inference, backend: "whispercpp-cpu" },
469+
);
470+
expect(alignerFor).not.toHaveBeenCalled();
471+
expect(fetchMock).toHaveBeenCalledOnce();
472+
expect(result.wordSegments.map((w) => [w.word, w.startSec, w.endSec])).toEqual([
473+
["ab", 0.2, 0.5],
474+
["ba", 0.5, 1],
475+
]);
476+
});
477+
465478
it("keeps whisper's times when the aligner fails", async () => {
466479
const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
467480
try {

‎electron/stt/whisperServer.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -646,7 +646,13 @@ export class WhisperServerManager {
646646
)
647647
.filter((w) => w.word.length > 0);
648648
let alignSec = 0;
649-
if (speech?.length && words.length && opts.alignerFor) {
649+
const backend = this.toBackend(json.backend);
650+
// GPU only. On the CPU the aligner's forward pass costs +29% (English) to
651+
// +58% (French) of a transcription, against +9% to +14% on Vulkan, and
652+
// character-level DTW alone is already within 16 ms median there
653+
// (tools/stt-eval/word-timing, issue #948). Not asking also means a
654+
// CPU-only machine never downloads the model.
655+
if (speech?.length && words.length && opts.alignerFor && backend !== "whispercpp-cpu") {
650656
try {
651657
const modelPath = await opts.alignerFor(detectedLanguage);
652658
if (modelPath) {
@@ -670,7 +676,6 @@ export class WhisperServerManager {
670676
// VAD is the better judge of where speech starts after a pause (see
671677
// snapWordBoundaries.ts).
672678
const wordSegments: SttWordSegment[] = anchorWordsOnSpeech(words, speech);
673-
const backend = this.toBackend(json.backend);
674679
const timing = this.toTiming(json.timing);
675680
// The aligner is part of what this chunk cost.
676681
if (timing && alignSec > 0) {

‎technical-documentation/architecture/transcription-and-captions.md‎

Lines changed: 43 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ bare `ipcRenderer.invoke`, `fetchWithRetry` has no per-request deadline, and
113113
whisper-server's 30 s readiness budget only starts once the download resolved.
114114

115115
The word aligner (§ Word-level alignment, step 5) is not part of that phase: it
116-
is fetched in the background the first time a transcription detects English or
116+
is fetched in the background the first time a transcription on a GPU detects English or
117117
French (109 or 348 MB), and no chunk ever waits for it.
118118

119119
Three edges make that promise hold, and each is load-bearing:
@@ -261,7 +261,7 @@ it verbatim in the response.
261261
the same thing as whisper.cpp's `t_dtw` matching faster-whisper's word
262262
**end** (`tools/stt-eval/whispercpp-dtw-poc/REPORT.md`).
263263
5. **Re-time the words on a CTC aligner** (issue #948, phase 3) — when the
264-
language whisper detected has one, a wav2vec2 model fine-tuned for CTC scores
264+
chunk ran on a GPU and the language whisper detected has an aligner, a wav2vec2 model fine-tuned for CTC scores
265265
every 20 ms frame of the speech against every letter, and
266266
[`electron/stt/ctcAlign.ts`](../../electron/stt/ctcAlign.ts) forces
267267
whisper's own words through those scores:
@@ -287,12 +287,17 @@ it verbatim in the response.
287287
midpoint. Next to a pause (a gap of 100 ms or more between letters) the
288288
edges move further out, 60 ms before and 150 ms after, both capped at
289289
mid-pause: early there is silence, late is an audible attack.
290-
- **Fallback.** No aligner for the language, a download not landed yet or
290+
- **GPU only.** On the CPU the forward pass would add 29% (English) to 58%
291+
(French) to a transcription, against 9 to 14% on Vulkan, for a gain over
292+
step 2's character-level DTW that is not worth that there. A chunk whisper
293+
ran on the CPU skips this step, and a CPU-only machine never downloads the
294+
model.
295+
- **Fallback.** CPU, no aligner for the language, a download not landed yet or
291296
failed, a helper without `/emissions`, a stretch whose letters do not fit in
292297
its frames: the words keep the times of step 4. Step 6 runs either way.
293298
6. **Anchor phrase edges on the speech** —
294299
[`electron/stt/snapWordBoundaries.ts`](../../electron/stt/snapWordBoundaries.ts)
295-
leaves boundaries inside a phrase alone: they are within ~16 ms of the audio
300+
leaves boundaries inside a phrase alone: they are within ~15 ms of the audio
296301
already, and an energy snap on top of them drags correct boundaries early.
297302
The edges of a phrase are different. The token before a phrase's first word
298303
is the previous phrase's last one, so that word starts in the pause, as
@@ -329,21 +334,29 @@ themselves about 10 to 20 ms from a human's.
329334
|---|---|---|---|---|
330335
| First-token start + 150 ms RMS snap + VAD edges (before #948), TTS | 105 / 275 ms / 32% | 83% | 5% | 84% |
331336
| Previous-token start + two-way VAD edges (phase 1), TTS | 31 / 125 ms / 64% | 89% | 19% | 88% |
332-
| + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 96% |
333-
| + CTC aligner (phase 3), TTS noisy | 15 / 39 ms / 95% | 93% | 48% | 93% |
334-
| Phase 1, LibriSpeech | 40 / 145 ms / 55% | 67% | 14% | 71% |
335-
| + CTC aligner (phase 3), LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
336-
337-
With the aligner, deleting one word leaves on average 17 ms of it audible and
338-
cuts 15 ms into its neighbours (TTS clean; 20 and 13 ms on LibriSpeech), against
339-
40 and 44 ms before. The misses left on phrase-initial starts are mostly the
340-
reference's: the French TTS voices start a word on its silent stop closure,
341-
which no aligner hears. On a real French take (25 s, no reference) checked on
342-
its spectrogram, the aligner moved six boundaries by more than 40 ms: three
343-
onto a visible boundary that phase 1 had missed by 65 to 250 ms (e.g.
344-
"montrer | ça" from inside `montrer` to the onset of the /s/), one the right
345-
way but still about 150 ms early (a drawn-out "Bah… voilà"), and two between
346-
vowels, where the spectrogram cannot tell.
337+
| Same, character-level DTW (phase 2), TTS clean | 17 / 60 ms / 86% | 89% | 39% | 94% |
338+
| + CTC aligner (phase 3), TTS clean | 14 / 35 ms / 96% | 95% | 50% | 95% |
339+
| Phase 2, TTS noisy | 16 / 60 ms / 86% | 88% | 40% | 92% |
340+
| + CTC aligner, TTS noisy | 15 / 39 ms / 95% | 94% | 48% | 93% |
341+
| Phase 2, LibriSpeech | 20 / 65 ms / 82% | 63% | 35% | 81% |
342+
| + CTC aligner, LibriSpeech | 15 / 45 ms / 92% | 73% | 44% | 82% |
343+
344+
Per language, character-level DTW gives French 19 ms median and 83% within
345+
50 ms (was 26 ms, 70%) and English 15 ms and 89% (was 40 ms, 57%). The aligner
346+
takes French to 14 ms and 95% (P90 66 → 36 ms, clean cuts 33% → 47%) and
347+
English to 15 ms and 96% (P90 50 → 35 ms, clean cuts 45% → 52%). Deleting one
348+
word then leaves 17 ms of it audible and cuts 15 ms into its neighbours (TTS
349+
clean; 20 and 12 ms on LibriSpeech), against 19 and 24 ms with phase 2 alone.
350+
The misses left on phrase-initial starts are mostly the reference's: the French
351+
TTS voices start a word on its silent stop closure, which no aligner hears.
352+
353+
On a real French take (25 s, no reference) read on its spectrogram, the
354+
aligner is not uniformly better than phase 2: it moved "c'est" onto the onset
355+
of its /s/ (phase 2 was 90 ms early), but put "quoi" after "c'est" 70 to 80 ms
356+
early twice, inside the vowel of *c'est*, where phase 2 had it on the /k/
357+
closure. French silent final letters are the likely cause: the model was
358+
trained on spelling, and the `t` of *c'est* has no sound to sit on (phase 2
359+
drops such letters before aligning; this pass does not yet).
347360

348361
A +15 ms calibration offset on every boundary gained 4 points of inner
349362
boundaries within 50 ms but dropped noisy phrase deletes to 82%, so it is not
@@ -896,12 +909,15 @@ it deletes data
896909
in `electron/native/whisper-stt/src/main.cpp` are exercised only at
897910
runtime.
898911
- **Word timing without an aligner.** Phrase edges sit on the VAD (step 6),
899-
and English and French words are re-timed on a CTC aligner (step 5). Every
900-
other language keeps whisper-small's DTW: about 30 ms median and 125 ms P90
901-
inside a phrase, so one single-word delete in five is clean. A multilingual
902-
CTC model (`facebook/omniASR-CTC-300M`) would cover them, at a cost per
912+
and on a GPU English and French words are re-timed on a CTC aligner (step 5).
913+
Every other language, and every language on the CPU, keeps whisper-small's
914+
character-level DTW: about 16 ms median and 60 ms P90 on synthetic speech, so
915+
about two single-word deletes in five are clean. A multilingual CTC model
916+
(`facebook/omniASR-CTC-300M`) would cover the other languages, at a cost per
903917
language family nobody has measured yet (issue #948).
904-
- **The aligner on the CPU.** On the GPU it adds about 10% to a transcription.
905-
On the CPU fallback (16 threads, Ryzen 7 5800X) it adds 28% in English and
906-
42 to 61% in French: the French model is a 24-layer wav2vec2 large, and its
907-
forward pass costs about half of whisper-small's.
918+
- **The aligner is GPU-only.** On the CPU fallback (16 threads, Ryzen 7 5800X)
919+
it would add 29% in English and 58% in French: the French model is a 24-layer
920+
wav2vec2 large, about half of whisper-small's cost on its own.
921+
- **French silent letters.** The aligner spells *c'est* with its `t`, which
922+
has no sound; on a real take that put the next word 70 to 80 ms early.
923+
Dropping silent final consonants before aligning, as step 2 does, is untried.

‎tools/stt-eval/word-timing/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ node real-check.mjs <whisper-stt-server.exe> take.wav [--align <aligner.gguf>]
8282

8383
It prints how far each word's first-token time lies after its start, and where
8484
each phrase's first word lands against the VAD onset, raw and after the
85-
post-pass. With `--align`, every word is listed with its phase-1 and aligner
85+
post-pass. With `--align`, every word is listed with its DTW and aligner
8686
times, so the boundaries that moved can be checked by ear or on a spectrogram.
8787

8888
## Reading the numbers

‎tools/stt-eval/word-timing/real-check.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -99,7 +99,7 @@ for (const [i, s] of speech.entries()) {
9999
}
100100
if (aligned) {
101101
const moved = [];
102-
console.log("\nword phase 1 (s) aligner (s) start / end moved");
102+
console.log("\nword DTW (s) aligner (s) start / end moved");
103103
raw.forEach((w, j) => {
104104
const a = post[j];
105105
const b = aligned[j];
@@ -111,6 +111,6 @@ if (aligned) {
111111
moved.sort((x, y) => x - y);
112112
const m = (p) => (moved[Math.floor(moved.length * p)] * 1000).toFixed(0);
113113
console.log(
114-
`\naligner vs phase 1, |moved| per boundary: median ${m(0.5)} ms, p90 ${m(0.9)} ms (${emissions.elapsed_s.toFixed(2)} s on ${emissions.device})`,
114+
`\naligner vs DTW, |moved| per boundary: median ${m(0.5)} ms, p90 ${m(0.9)} ms (${emissions.elapsed_s.toFixed(2)} s on ${emissions.device})`,
115115
);
116116
}

0 commit comments

Comments
 (0)