Skip to content

feat(cloud): fire estimated word boundaries progressively during streaming - #14

Merged
willwade merged 1 commit into
mainfrom
feat/progressive-estimated-boundaries
Aug 16, 2026
Merged

feat(cloud): fire estimated word boundaries progressively during streaming#14
willwade merged 1 commit into
mainfrom
feat/progressive-estimated-boundaries

Conversation

@willwade

Copy link
Copy Markdown
Contributor

Why

VoiceGarden-SPD (speech-dispatcher module) exposes this: for engines without API timing data (OpenAI, Deepgram, Watson, …), on_boundary estimates fired in one batch after the response completed. Since PR #10 streams audio as it downloads, callers interleaving index marks with playback could only emit them at the end — word highlighting was effectively useless on long utterances for ~15 of the 20 cloud engines.

What

  • EstimatePlan — pre-resolves the 150-wpm estimates (char offsets included, SSML stripped first) into firing order
  • EstimateFirer — tracks cumulative delivered PCM samples and fires estimate i once ≥ its start-time worth of audio has been emitted. This anchors the text-based estimates onto the real audio clock: a voice slower than the 150-wpm baseline simply holds each mark until its audio arrives; audio shorter than the estimates flushes the remainder at end-of-stream
  • StreamEvtstream_body_to_on_audio now takes a single Audio | Boundary event callback (threading two &mut dyn FnMut callbacks hit trait-object lifetime invariance)
  • IncrementalDecoder::sample_rate() — exposes the rate observed in the first decoded packet so MP3 bytes can be converted to seconds
  • Raw-PCM bodies (Azure, Cartesia) use their pinned 24 kHz config rate

Real-timing engines (Azure/Edge WS, Google timepoints, ElevenLabs with-timestamps) are unchanged.

Testing

  • estimated_boundaries_interleave_with_audio_events: a Boundary event provably fires between Audio events on a dribbled silent-MP3 stream (not at the end flush)
  • estimated_boundaries_fire_progressively_during_streaming: every estimate fires; flush covers short audio
  • estimate_plan_strips_ssml_before_estimating
  • Full cloud suite green (115 tests); clippy -D warnings clean

…aming

Estimated boundaries for engines without API timing data (OpenAI,
Deepgram, Watson, …) previously fired in one batch after the response
completed, so callers interleaving marks with playback (the
VoiceGarden-SPD speech-dispatcher module) could only report them at
the end — word highlighting was useless on long utterances.

- EstimatePlan: pre-resolves the 150-wpm estimates (char offsets
  included, SSML stripped first) into firing order
- EstimateFirer: tracks cumulative delivered PCM samples and fires
  estimate i once its start-time worth of audio has been emitted —
  anchoring text-based estimates onto the real audio clock (a voice
  slower than 150 wpm simply holds marks until their audio arrives;
  audio shorter than the estimate flushes the remainder at end)
- stream_body_to_on_audio now takes a single StreamEvt callback
  (Audio | Boundary) — threading two &mut dyn FnMut callbacks hit
  trait-object lifetime invariance
- IncrementalDecoder exposes the observed sample rate (needed to turn
  emitted bytes into seconds for MP3 bodies)
- raw-PCM bodies (Azure, Cartesia) are pinned 24 kHz in their configs

Tests: silent-MP3 dribble stream proves a Boundary event fires between
Audio events (interleaved, not end-flushed); every estimate fires;
EstimatePlan strips SSML before estimating.
@willwade
willwade merged commit c6b04f0 into main Aug 16, 2026
9 checks passed
@willwade
willwade deleted the feat/progressive-estimated-boundaries branch August 16, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant