feat(cloud): fire estimated word boundaries progressively during streaming - #14
Merged
Merged
Conversation
…aming Estimated boundaries for engines without API timing data (OpenAI, Deepgram, Watson, …) previously fired in one batch after the response completed, so callers interleaving marks with playback (the VoiceGarden-SPD speech-dispatcher module) could only report them at the end — word highlighting was useless on long utterances. - EstimatePlan: pre-resolves the 150-wpm estimates (char offsets included, SSML stripped first) into firing order - EstimateFirer: tracks cumulative delivered PCM samples and fires estimate i once its start-time worth of audio has been emitted — anchoring text-based estimates onto the real audio clock (a voice slower than 150 wpm simply holds marks until their audio arrives; audio shorter than the estimate flushes the remainder at end) - stream_body_to_on_audio now takes a single StreamEvt callback (Audio | Boundary) — threading two &mut dyn FnMut callbacks hit trait-object lifetime invariance - IncrementalDecoder exposes the observed sample rate (needed to turn emitted bytes into seconds for MP3 bodies) - raw-PCM bodies (Azure, Cartesia) are pinned 24 kHz in their configs Tests: silent-MP3 dribble stream proves a Boundary event fires between Audio events (interleaved, not end-flushed); every estimate fires; EstimatePlan strips SSML before estimating.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
VoiceGarden-SPD (speech-dispatcher module) exposes this: for engines without API timing data (OpenAI, Deepgram, Watson, …),
on_boundaryestimates fired in one batch after the response completed. Since PR #10 streams audio as it downloads, callers interleaving index marks with playback could only emit them at the end — word highlighting was effectively useless on long utterances for ~15 of the 20 cloud engines.What
EstimatePlan— pre-resolves the 150-wpm estimates (char offsets included, SSML stripped first) into firing orderEstimateFirer— tracks cumulative delivered PCM samples and fires estimate i once ≥ its start-time worth of audio has been emitted. This anchors the text-based estimates onto the real audio clock: a voice slower than the 150-wpm baseline simply holds each mark until its audio arrives; audio shorter than the estimates flushes the remainder at end-of-streamStreamEvt—stream_body_to_on_audionow takes a singleAudio | Boundaryevent callback (threading two&mut dyn FnMutcallbacks hit trait-object lifetime invariance)IncrementalDecoder::sample_rate()— exposes the rate observed in the first decoded packet so MP3 bytes can be converted to secondsReal-timing engines (Azure/Edge WS, Google timepoints, ElevenLabs
with-timestamps) are unchanged.Testing
estimated_boundaries_interleave_with_audio_events: a Boundary event provably fires between Audio events on a dribbled silent-MP3 stream (not at the end flush)estimated_boundaries_fire_progressively_during_streaming: every estimate fires; flush covers short audioestimate_plan_strips_ssml_before_estimating-D warningsclean