Skip to content

feat(sherpaonnx): stream audio per sentence batch via the generate callback - #15

Merged
willwade merged 1 commit into
mainfrom
feat/sherpa-streaming-delivery
Aug 16, 2026
Merged

feat(sherpaonnx): stream audio per sentence batch via the generate callback#15
willwade merged 1 commit into
mainfrom
feat/sherpa-streaming-delivery

Conversation

@willwade

Copy link
Copy Markdown
Contributor

Why

The README claimed sherpa-onnx "synthesises whole clips by design" — that was wrong. Verified in sherpa-onnx's C++ sources (offline-tts-vits-impl.h / kokoro / matcha / supertonic): the generate progress callback receives each sentence batch's newly generated samples as synthesis progresses (the Rust crate's "samples generated so far" doc comment was the error). We already set max_num_sentences=1, so batches are sentence-sized. js-tts-wrapper's synthToBytestream looked like streaming but enqueues one whole-clip buffer — this does the real thing.

What

  • Sentence-level streaming: speak() emits each batch through on_audio immediately; volume/pitch applied per batch (batches are sentence-aligned, so per-batch pitch resampling never seams mid-speech). Single-sentence utterances still complete before delivery (inherent).
  • Progressive estimated boundaries: shares the EstimateFirer machinery introduced in feat(cloud): fire estimated word boundaries progressively during streaming #14, with a 1/speed time-scale so estimates anchor to delivered audio. Reported times stay on the rate-1.0 baseline — existing rate-compensating callers (VoiceGarden-SPD, the SAPI adapter) are unaffected.
  • New boundaries module: EstimatePlan/EstimateFirer moved out of cloud_engine into an always-compiled module; the firer now owns its plan so it can move into 'static callbacks.
  • 'static callback plumbing: the generate callback can't capture method-lifetime borrows, so on_audio/on_boundary are stashed as lifetime-erased pointers in thread-locals for the duration of the synchronous call (same-thread invocation per the C++ docs; serialised by the tts_instance mutex — same pattern as VISEME_CB), and the firer is shared via Arc<Mutex<..>> so the outer scope flushes remaining estimates after generation.

Testing

  • New live test sherpa_streams_audio_per_sentence_batch (piper-nl-rdh-low): asserts a Boundary event fires before the final audio chunk and first-audio < last-audio — both hold with the real model
  • Full local suite green (294 tests incl. moved boundary tests); clippy -D warnings clean; live vits/kokoro families pass (matcha tests skipped locally — model not downloaded, CI covers)
  • README streaming notes corrected

…llback

The crate's README (and engine docs) claimed sherpa-onnx could not
stream — it can: the generate progress callback receives each batch's
NEWLY generated samples (verified in sherpa-onnx csrc: the vits/kokoro/
matcha/supertonic impls call the callback after every sentence batch;
the Rust crate's 'samples generated so far' doc comment was wrong).
With max_num_sentences=1 already set, batches are sentence-sized.

- speak() now emits each batch through on_audio immediately (volume/
  pitch applied per batch — batches are sentence-aligned, so per-batch
  resampling never seams mid-speech), giving sentence-level streaming
  for multi-sentence utterances
- estimated word boundaries fire progressively via the shared
  EstimateFirer (1/speed time-scale anchors estimates to delivered
  audio; reported times stay on the rate-1.0 baseline for existing
  rate-compensating callers)
- the generate callback must be 'static: on_audio/on_boundary are
  stashed as lifetime-erased pointers in thread-locals for the
  synchronous call (same-thread callback per the C++ docs; serialised
  by the tts_instance mutex), and the firer is shared via Arc<Mutex>
  so the outer scope flushes the remainder afterwards
- EstimateFirer/EstimatePlan moved to a new always-compiled
  boundaries module (shared by cloud + sherpa); EstimateFirer now owns
  its plan so it can move into 'static callbacks

Live-verified with piper-nl-rdh-low: a boundary event provably fires
before the final audio chunk and first audio precedes the last
(sherpa_streams_audio_per_sentence_batch). js-tts-wrapper parity note:
its synthToBytestream enqueues one whole-clip buffer — this goes
further.
@willwade
willwade merged commit 51d7d0b into main Aug 16, 2026
12 checks passed
@willwade
willwade deleted the feat/sherpa-streaming-delivery branch August 16, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant