Update SpeechRecognitionResult with audio timing attributes - #192
Update SpeechRecognitionResult with audio timing attributes#192alan33d wants to merge 2 commits into
Conversation
Added audioStartTime and audioEndTime attributes to SpeechRecognitionResult interface with explanations as proposed in WebAudio#191
…+ missing root suite Source-review against the explainer (readonly attribute DOMHighResTimeStamp? audioStartTime/audioEndTime on SpeechRecognitionResult — milliseconds relative to time origin, nullable when the backend supplies no segment timing, ~2ms fuzzing to mirror HTMLMediaElement.currentTime; latency = event.timeStamp - result.audioEndTime), PR WebAudio/web-speech-api#192 (OPEN/unmerged as of 2026-07-24, updated 2026-07-15), Chromium main's speech_recognition_result.idl ([LegacyNoInterfaceObject]; currently only length/item/isFinal — the timestamp attributes are NOT in trunk and no dedicated runtime feature exists yet, so even the experimental-features catch-all may expose nothing), the direct chromestatus detail (updated 2026-07-16: v153 listing dev trial desktop_first 153; detail desktop None, 'Proposed'; ff/safari=5 no-signal), MDN (SpeechRecognitionResult 200, no timestamp coverage yet), and WPT (27 speech-api files, no timestamp test as of 2026-07-25). The family was already structurally honest: the LegacyNoInterfaceObject detection recipe (no constructor to probe — inspect a live result instance), absent-property vs present-but-null distinction (unsupported vs no-value), SAMPLE-tagged walkthroughs that never pose as live results, headless/no-mic errors reported as availability problems, and per-result values only ever read from real result objects. Hardened: - Index lede 'Chrome 153 adds' replaced by labelled-proposal framing; 'why it shipped' renamed 'why timestamps matter'; an explicit availability paragraph states the in-flux implementation state (dev trial at 153 per listing; detail 'Proposed'; PR unmerged; not in trunk IDL as of 2026-07-24; no dedicated flag yet). - All three concepts' enablement notes now carry the same in-flux precision instead of promising the catch-all flag works. - pagehide recognition.stop() cleanup added to latency-monitor and caption-timeline (capability-probe already stops on end/stop). Conformance: NEW immutable root suite (8 assertions = 8 rendered rows = 8 reported): the recognition constructor host, the LegacyNoInterfaceObject no-global trap, the explicit start instance-vs-static pair, the interim/continuous configuration members, the no-synthetic-result-construction contract (per-result values must come from a live object — never fabricated or inferred from event cadence), and the two clock contracts (Event.prototype.timeStamp as the latency subtraction operand; performance.timeOrigin as the shared origin the audio times are relative to). No microphone is requested anywhere; the timestamp ATTRIBUTES themselves are structurally untestable without a live recognition on an implementing engine, recorded honestly in the suite descriptions. Executes live, recounted from rendered rows: 8 total = 8 pass / 0 fail / 0 blocked on Chrome 150. Missing feature-level suites 3->2 (795/798 -> 796/798). Lifecycle: feature critique with guidanceConsulted (accessibility, privacy); responsive-support ok/ok (harness); check-routes PASS (798/798, touched=1); fmt/audit/check exit 0. MCP on HeadlessChrome/150 (no blind mic grants): surface probe (SpeechRecognition + webkit present, SpeechRecognitionResult global ABSENT as LegacyNoInterfaceObject requires), live inspect reports 'Recognition error: not-allowed' as an availability problem with the probe unaffected and duplicate suppression restoring the button, latency sample walkthrough (SAMPLE-tagged segments, verdict tracks threshold changes, gauge fill/marker live), caption timeline sample with 'SAMPLE — not a live recognition result' banner and positioned segments, zero console errors, zero failed requests. Two mobile defects found post-interaction and fixed in place: probe results table 543px -> 306px (overflow-wrap + fixed layout), and the index's unbreakable 23-char title word spilling text-level past its 342px box -> h1{overflow-wrap:anywhere} (433px -> 390px). All four pages re-verified contained at 390px. Co-Authored-By: Kimi K3 <noreply@moonshot.ai>
|
No objections from me, but I’ll defer the review part to @padenot to make sure we have browser alignment here. |
padenot
left a comment
padenot
left a comment
There was a problem hiding this comment.
Some questions below, thanks.
|
|
||
| <li>To mitigate the risk of fingerprinting, user agents MUST NOT personalize speech recognition when performing speech recognition on a {{MediaStreamTrack}}.</li> | ||
|
|
||
| <li>To mitigate fingerprinting vectors associated with high-precision timing, user agents MUST apply timestamp fuzzing and precision reduction to {{SpeechRecognitionResult/audioStartTime}} and {{SpeechRecognitionResult/audioEndTime}} before exposing these attributes to scripts (e.g. by rounding to 2ms precision).</li> |
There was a problem hiding this comment.
This isn't a useful mitigation is it? Think about how you'd implement the same ASR system in WASM, will it be limited? It's absolutely trivial to implement audioStartTime etc. in WASM, and it will be sample-accurate w.r.t the audio stream.
|
|
||
| <dl> | ||
| <dt><dfn attribute for=SpeechRecognitionResult>audioStartTime</dfn> attribute</dt> | ||
| <dd>A nullable {{DOMHighResTimeStamp}} representing the start of the audio segment corresponding to this recognition result, in milliseconds relative to the start of the audio stream. Returns null if the underlying recognition engine does not support audio segment start timestamps.</dd> |
There was a problem hiding this comment.
We use seconds throughout those APIs, or microseconds in codec APIs. 64-bits floating point numbers have enough resolution here at even high sample rates, and it means we're not going to convert to/from seconds.
I don't think those should be nullalble. If your backend doesn't support timestamping, it can be implemented, or you can ship something decent (in the sense that it's going to be accurate enough for a lot of use case) in front of it.
We tend (not 100%) to not specificy APIs because of certain technologies limitations. Instead, we improve those technologies and write a good spec.
As discussed in the other PR, "relative to the start of the audio stream" is very fuzzy. How does it work if you get a MediaStream with a certain currentTime (internally), then fed into the Web Audio API (which has a different currentTime), then fed into an HTMLMediaElement (different currentTime), then captureStream(), and piped into SpeechRecognition. Which it is? The last one? Probably, and in that clock domain as well, which is important to mention because crossing clock domain makes alignment hard (which is what we want to do here).
Why is this not speechStartTime, mirroring speechStart (that also has a timestamp). What are the constraints between the two? It should be strictly equal probably?
4 participants
Added audioStartTime and audioEndTime attributes to SpeechRecognitionResult interface with explanations as proposed in #191.
Fixes: #191
Preview | Diff