Add SpeechRecognitionResult Timestamps

[alan33d]

Description

The Web Speech API currently does not expose the start and end timestamps of the source audio corresponding to a transcription result (SpeechRecognitionResult).

While SpeechRecognitionEvent.timeStamp provides a single timestamp for the event itself, it does not convey the start time, end time, or duration of the acoustic speech segment within the audio stream.

Without acoustic bounds for each result, web applications cannot associate transcribed text with media timelines:

• Automated Subtitling & Caption Tracks: Cannot construct frame-accurate subtitle cues (VTTCue(start, end, text)) for recorded video or audio.
• Interactive "Click-to-Seek" Transcripts: Cannot jump media playback (mediaElement.currentTime = result.speechStartTime) to the exact moment a phrase was spoken.
• Live Video Conferencing & Recording Sync: In applications like Google Meet or WebRTC calls, captions displayed live cannot be accurately aligned with the recorded video track for post-meeting review or synchronized with video playout delay buffers.
• Text-Based Media Editing: Tools that allow users to edit audio/video by cutting or rearranging transcript text require precise acoustic boundaries for each segment.

---

Specification

Expose speech segment bounds on the SpeechRecognitionResult interface in seconds as a double, measured on the timeline of the audio stream consumed by the recognizer ($t = 0.0\text{s}$, marked by audiostart).

Web IDL

partial interface SpeechRecognitionResult {
    // Start timestamp of the recognized speech segment in seconds relative to the audio stream (0.0s).
    readonly attribute double speechStartTime;

    // End timestamp of the recognized speech segment in seconds relative to the audio stream.
    readonly attribute double speechEndTime;
};

---

Usage Example: Subtitling & "Click-to-Seek" Navigation

const mediaElement = document.querySelector('video');
const track = mediaElement.addTextTrack('captions', 'English', 'en');
track.mode = 'showing';

const recognition = new SpeechRecognition();
recognition.continuous = true;
recognition.interimResults = false;

recognition.onresult = (event) => {
  for (let i = event.resultIndex; i < event.results.length; ++i) {
    const result = event.results[i];
    if (!result.isFinal) continue;

    const transcriptText = result[0].transcript;

    // 1. Create frame-accurate WebVTT cue directly from seconds
    const cue = new VTTCue(result.speechStartTime, result.speechEndTime, transcriptText);
    track.addCue(cue);

    // 2. Click-to-seek transcript element
    const item = document.createElement('p');
    item.textContent = `[${result.speechStartTime.toFixed(1)}s] ${transcriptText}`;
    item.onclick = () => {
      mediaElement.currentTime = result.speechStartTime;
      mediaElement.play();
    };
    document.getElementById('transcript').appendChild(item);
  }
};

recognition.start(mediaElement.captureStream().getAudioTracks()[0]);

---

Security & Privacy Considerations

• Fingerprinting Mitigation: To mitigate micro-architectural timing attacks and hardware fingerprinting, user agents may reduce the resolution of speechStartTime and speechEndTime or introduce jitter, in accordance with user agent security and privacy policies (consistent with [[HR-TIME-3]] and [[HTML]]).

---

Alternatives Considered

• Existing SpeechRecognitionEvent.timeStamp: Represents a single point in time (the event occurrence in document time). It does not provide the duration or start/end acoustic boundaries of the utterance within the stream, making subtitle cue generation (VTTCue) and click-to-seek impossible.
• Separate Events (speechstart / speechend): Fire only once at session voice-activity boundaries, cannot be correlated to individual stream results in continuous recognition, and do not provide segment start/end bounds per result.
• DOMHighResTimeStamp (Milliseconds): Semantically represents time relative to document origin rather than a media stream timeline, and causes friction with existing W3C media APIs (HTMLMediaElement.currentTime, BaseAudioContext.currentTime, and VTTCue) which universally operate in seconds as a double.

---

Links & References

• Explainer: speech-recognition-result-timestamps.md (PR #206)
• Spec PR: PR #192
• Interactive Demos:
  • Live Meeting Caption Sync & Recording Demo
  • WebRTC Caption Playout Delay Test
• ChromeStatus: Web Speech API: Add SpeechRecognitionResult Timestamps

[padenot]

https://webaudio.github.io/web-speech-api/#speechreco-result:~:text=For%20all%20these,endpointer%29%2E

In Firefox, this accounts for input latency (for direct microphone access), various buffering stages e.g. in Media playback, IPC boundaries, and the ASR model inherent latency. Output latency is available on the web platform to have super tight synchronization for e.g. auto-captionning.

This is workable, and we have built neat demos that show that alignment works well (e.g. putting words below the wave form). It does however require doing a little bit of arithmetic, but you get high flexibility in return.

@alan33d, @evanbliu, what does Chrome do?

[michaelwasserman]

Can you point to the demos? I'm not too experienced here and more of a visual learner, so I vibe coded a demo (source) to try learning about the status quo.

I tried testing the same in FF with about:config media.webspeech.* flags all set true, but it wasn't working; lmk if you have tips.

I'm hoping we can continue to explore the viability of using the existing API surface, but new timestamps may prove useful if not necessary.

[padenot]

Our implementation isn't merged.

[alan33d]

https://webaudio.github.io/web-speech-api/#speechreco-result:~:text=For%20all%20these,endpointer%29%2E

In Firefox, this accounts for input latency (for direct microphone access), various buffering stages e.g. in Media playback, IPC boundaries, and the ASR model inherent latency. Output latency is available on the web platform to have super tight synchronization for e.g. auto-captionning.

This is workable, and we have built neat demos that show that alignment works well (e.g. putting words below the wave form). It does however require doing a little bit of arithmetic, but you get high flexibility in return.

@alan33d, @evanbliu, what does Chrome do?

Thanks for the suggestion. The current proposal addresses a developer request to detect speech recognition latency degradation mid-audiostream. From @michaelwasserman 's demo using the existing API surface, we noticed the existing event updates will only allow websites to accurately detect the latency of speech recognition at the beginning of the stream which is not enough granularity for the developer's use case. Thus we are proposing this change to get us the latency updates on an individual SpeechRecognitionResult call basis.

[michaelwasserman]

I published an improved demo and Chrome seems to fire speechstart, speechend, etc. just once over the course of the recognition session. While the API might permit firing those events around each recognition result, clients would have a hard time correlating separate events and results. It seems like including the timing information in the results is a vast ergonomic improvement.

[padenot]

The timestamp information on those events is supposed to be in the timeline and time domain of the media, is that not the case in Chrome? See my link in #191 (comment)

Thanks for providing a demo, I'll run it on our implementation.

[michaelwasserman]

The events do seem to be in the time domain of the media, but:

1. soundstart and speechstart only fire once per continuous recognition session, so they can't be used for continual latency checks of results emitted through the session.
2. Even if the spec permits multiple speechstart/end events per recognition session, and implementations were updated to support that, correlating those events with the respective recognition results is not entirely trivial for each API client interested in latency information.
3. Attempting to modify each recognition result DOM Event timeStamp to match the corresponding speechstart or most recent estimate of a speechend event, wouldn't enable clients to calculate latency information IIUC.

Adding start and end timestamps to each result would seem to clearly and ergonomically give solve the client pain points around timeline association and continual latency detection. Please let us know what you find in your implementation.

[alan33d]

Update on SpeechRecognitionResult Timestamps

Following recent discussions in the WebAudio WG meeting and review feedback from @padenot and @ReillyG, we have updated both the Specification and the Explainer with the following:

1. Unit of Measurement (double in Seconds):

   • Updated audioStartTime and audioEndTime from DOMHighResTimeStamp (milliseconds) to double representing seconds.
   • This aligns with adjacent media and audio APIs on the web platform (BaseAudioContext.currentTime and HTMLMediaElement.currentTime), avoiding unit conversions when piping audio streams between Web Audio and VTTCue.

2. Non-Nullable Attributes:

   • Removed nullability (?) to keep the specification clean, without introducing artificial sentinel values (e.g. 0 or -1).

3. Stream-Relative Timeline & Document Alignment:

   • The timestamps represent elapsed time starting at $0.0\text{s}$ within the audio stream actively consumed by the recognizer.
   • The Explainer includes patterns for converting stream offsets to the document timeline (performance.timeOrigin / event.timeStamp) to support live translation, caption alignment, and real-time processing latency tracking.

4. Privacy & Precision:

   • Aligned the spec text with W3C High Resolution Time Level 3 and HTMLMediaElement.currentTime precedent, allowing user agents to coarsen timer resolution or add jitter in accordance with their security and isolation policies without mandating a hardcoded quantization value in the normative text.

5. Focus on Timeline Association: Streamlined the proposal to focus on the core capability that currently cannot be done on the web: associating transcripts with media timelines (generating VTTCue subtitle tracks, "click-to-seek" transcript navigation, recorded meeting alignment, and text-based media editing).

6. Attribute Naming: Renamed audioStartTime / audioEndTime to speechStartTime / speechEndTime to mirror speechstart / speechend.

7. Spec & Explainer Links: Updated links to PR #192 (spec) and PR #206 (explainer), along with live interactive demos.