Add zipvoice + pocket zero-shot voice-cloning support - #21
Closed
willwade wants to merge 2 commits into
Closed
Conversation
…nx engine Both model types from registry v2026-08-18 now synthesise instead of erroring with 'Unsupported SherpaOnnx model_type': - build_zipvoice_config: encoder/decoder (int8 preferred), tokens, lexicon, espeak-ng-data, and the non-bundled vocos_24khz.onnx vocoder resolved from the model dir then the shared base dir (the matcha convention) — a missing vocoder errors with the download URL instead of failing inside sherpa-onnx's create(). - build_pocket_config: lm_flow/lm_main/decoder int8 preference; encoder + text_conditioner are never quantised upstream, so the plain .onnx files are picked; vocab.json + token_scores.json sidecars. - Zero-shot cloning needs a reference clip via GenerationConfig: referenceAudio/referenceText credentials override the bundled test_wavs clip (zipvoice additionally requires the clip's exact transcript; the bundled leijun-1.wav transcript is known). Adds a minimal RIFF reader for 16-bit PCM wavs (stereo downmix, clear errors for other formats). num_steps defaults from the sherpa-onnx docs (zipvoice 4, pocket 2). - Live tests + workflow steps for both models (linux-only, like supertonic) incl. the vocos vocoder download. Verified locally against the downloaded models: all three new live tests synthesise non-empty audio; 9 new unit tests cover the config builders, wav reader, and reference resolution.
Attributes on expressions are still experimental on the stable toolchain CI pins; the function-level allow covers the same downmix cast.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #20 (needs the v2026-08-18 registry for the new model ids). Adds full engine support for the two new zero-shot model families so they synthesise instead of erroring with
Unsupported SherpaOnnx model_type.What's added
Config builders (sherpa-onnx crate 1.13.5 already ships both config types):
build_zipvoice_config— encoder/decoder (int8 preferred), tokens/lexicon/espeak-ng-data, plus the non-bundledvocos_24khz.onnxvocoder resolved model-dir-first then the shared base dir (same convention as Matcha). Missing vocoder → clear error with the download URL, not a C++ create() failure.build_pocket_config— int8 preference forlm_flow/lm_main/decoder;encoder+text_conditionerare never quantised upstream so plain.onnxis picked; JSON sidecars.Zero-shot reference handling — both models clone a voice from a reference clip delivered via
GenerationConfig:referenceAudio(path to 16-bit PCM wav) andreferenceText(zipvoice requires the clip's exact transcript)test_wavs/clip; known transcript for sherpa-onnx'sleijun-1.wavis built innum_stepsfrom the sherpa-onnx docs (zipvoice 4, pocket 2)CI: 3 new live tests (linux-only, like supertonic) + workflow downloads incl. the vocos vocoder.
Validation
zipvoice_clones_bundled_reference_voice,zipvoice_reference_transcript_override,pocket_synthesises_with_bundled_referenceall synthesise non-empty audioAfter #20 merges this PR can be retargeted to main (GitHub does it automatically).