feat(sherpaonnx): stream audio per sentence batch via the generate callback - #15
Merged
Merged
Conversation
…llback The crate's README (and engine docs) claimed sherpa-onnx could not stream — it can: the generate progress callback receives each batch's NEWLY generated samples (verified in sherpa-onnx csrc: the vits/kokoro/ matcha/supertonic impls call the callback after every sentence batch; the Rust crate's 'samples generated so far' doc comment was wrong). With max_num_sentences=1 already set, batches are sentence-sized. - speak() now emits each batch through on_audio immediately (volume/ pitch applied per batch — batches are sentence-aligned, so per-batch resampling never seams mid-speech), giving sentence-level streaming for multi-sentence utterances - estimated word boundaries fire progressively via the shared EstimateFirer (1/speed time-scale anchors estimates to delivered audio; reported times stay on the rate-1.0 baseline for existing rate-compensating callers) - the generate callback must be 'static: on_audio/on_boundary are stashed as lifetime-erased pointers in thread-locals for the synchronous call (same-thread callback per the C++ docs; serialised by the tts_instance mutex), and the firer is shared via Arc<Mutex> so the outer scope flushes the remainder afterwards - EstimateFirer/EstimatePlan moved to a new always-compiled boundaries module (shared by cloud + sherpa); EstimateFirer now owns its plan so it can move into 'static callbacks Live-verified with piper-nl-rdh-low: a boundary event provably fires before the final audio chunk and first audio precedes the last (sherpa_streams_audio_per_sentence_batch). js-tts-wrapper parity note: its synthToBytestream enqueues one whole-clip buffer — this goes further.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The README claimed sherpa-onnx "synthesises whole clips by design" — that was wrong. Verified in sherpa-onnx's C++ sources (offline-tts-vits-impl.h / kokoro / matcha / supertonic): the generate progress callback receives each sentence batch's newly generated samples as synthesis progresses (the Rust crate's "samples generated so far" doc comment was the error). We already set
max_num_sentences=1, so batches are sentence-sized. js-tts-wrapper'ssynthToBytestreamlooked like streaming but enqueues one whole-clip buffer — this does the real thing.What
speak()emits each batch throughon_audioimmediately; volume/pitch applied per batch (batches are sentence-aligned, so per-batch pitch resampling never seams mid-speech). Single-sentence utterances still complete before delivery (inherent).EstimateFirermachinery introduced in feat(cloud): fire estimated word boundaries progressively during streaming #14, with a1/speedtime-scale so estimates anchor to delivered audio. Reported times stay on the rate-1.0 baseline — existing rate-compensating callers (VoiceGarden-SPD, the SAPI adapter) are unaffected.boundariesmodule:EstimatePlan/EstimateFirermoved out of cloud_engine into an always-compiled module; the firer now owns its plan so it can move into'staticcallbacks.on_audio/on_boundaryare stashed as lifetime-erased pointers in thread-locals for the duration of the synchronous call (same-thread invocation per the C++ docs; serialised by thetts_instancemutex — same pattern asVISEME_CB), and the firer is shared viaArc<Mutex<..>>so the outer scope flushes remaining estimates after generation.Testing
sherpa_streams_audio_per_sentence_batch(piper-nl-rdh-low): asserts a Boundary event fires before the final audio chunk and first-audio < last-audio — both hold with the real model-D warningsclean; live vits/kokoro families pass (matcha tests skipped locally — model not downloaded, CI covers)