diff --git a/QUANTIZATION_SELECTION.md b/QUANTIZATION_SELECTION.md new file mode 100644 index 0000000..b8eaa4d --- /dev/null +++ b/QUANTIZATION_SELECTION.md @@ -0,0 +1,163 @@ +# Quantization Selection Guide + +This document describes all of the mechanisms provided by the +`qwen3-tts.cpp` repository for selecting which quantized model artifacts are +used during setup and runtime. The goal is to make it trivial to generate or +load *any* supported quantization type, and to understand how the system +chooses a file when multiple versions are present. + +--- + +## Supported Quantization Types + +- **TTS model** (`qwen3-tts-0.6b` stem): + - `f16` (no quantization) + - `f32` (float32; larger than f16) + - `q8_0` (8-bit GGUF quantization) + - `q4_k` (4-bit GGUF quantization; recommended for smaller size) + +- **Tokenizer/vocoder model** (`qwen3-tts-tokenizer` stem): + - `f16` + - `f32` + - `q8_0` (*q4_k is not supported by the converter*) + +These types correspond to the `--type` argument of the conversion scripts +`convert_tts_to_gguf.py` and `convert_tokenizer_to_gguf.py`. + + +## Setup Script (`scripts/setup_pipeline_models.py`) + +The one‑shot model setup script can now produce any of the above quantized +outputs. Two new CLI options were added: + +```bash +--tts-type {f16,f32,q8_0,q4_k} # default: q4_k +--tokenizer-type {f16,f32,q8_0} # default: q8_0 +``` + +When you run the script, it names the resulting GGUF files accordingly: + +- `models/qwen3-tts-0.6b-.gguf` +- `models/qwen3-tts-tokenizer-.gguf` + +This file naming is the convention that the runtime loader uses to pick models. + +Example: + +```bash +python scripts/setup_pipeline_models.py \ + --tts-type q4_k --tokenizer-type q8_0 +``` + +`--skip-download` and `--force` still work exactly as before. The readme +documentation has been updated with the new flags and examples. + + +## Runtime Model Discovery and Explicit Selection + +`src/qwen3_tts.cpp` contains the logic for locating GGUF files when the CLI is +called with `-m `. Earlier versions hard‑coded +`qwen3-tts-0.6b-f16.gguf` and +`qwen3-tts-tokenizer-f16.gguf`; this has been replaced with +`find_model_file()`: + +```cpp +static std::string find_model_file(const std::string &model_dir, + const std::string &stem) { + static const char *preferred[] = {"f16", "q8_0", "q4_k", "f32"}; + // look for stem-type.gguf in preferred order, otherwise return first match +} +``` + +This function returns the first existing file matching `stem-*.gguf` using a +preferred type ordering. By default `f16` wins, so if *only* a `q4_k` file is +present it will still be selected, but if both `f16` and `q4_k` exist the +script keeps picking `f16` unless instructed otherwise. + +To give callers complete control, the model‑loading API was extended: + +```cpp +bool Qwen3TTS::load_models(const std::string &model_dir, + const std::string &tts_model_path = {}, + const std::string &tokenizer_model_path = {}); +``` + +If either override argument is non‑empty it is used verbatim (absolute paths are +accepted; relative paths are joined with `model_dir`). These parameters are +exposed via two new CLI options: + +``` +--tts-model # path to a GGUF, overriding discovery +--tokenizer-model # same for tokenizer/vocoder +``` + +Examples: + +```bash +./build/qwen3-tts-cli -m models --tts-model qwen3-tts-0.6b-q4_k.gguf \ + --tokenizer-model qwen3-tts-tokenizer-q8_0.gguf -t "Hello" -o out.wav +``` + +The help text and README explain both the automatic and explicit modes. + + +## Discoverability Rules Summary + +1. If explicit paths are given on the CLI, they are used. +2. Otherwise, `find_model_file()` searches `model_dir` for + `stem-*.gguf`: + - Preferred order is `f16`, `q8_0`, `q4_k`, `f32`. + - If none of the preferred types exist but any file matches `stem-*.gguf`, + the lexicographically first match is returned. + - If no file is found an error is reported and loading fails. + +This allows mixed directories such as: + +``` +models/ + qwen3-tts-0.6b-f16.gguf + qwen3-tts-0.6b-q4_k.gguf + qwen3-tts-tokenizer-q8_0.gguf +``` + +…to behave predictably while still enabling explicit selection when desired. + + +## Practical Advice + +- **For size-conscious deployments**, run the setup script with `--tts-type + q4_k` and `--tokenizer-type q8_0`. The resulting files are about 1.8 GB and + 273 MB respectively (after mixed-precision fallbacks). +- **For development or debugging**, using `f16` models avoids quantization + variability and is the default when running `./build/qwen3-tts-cli` without + any tuning. +- **When multiple quantizations coexist** (e.g. you’re evaluating trade‑offs), + add the `--tts-model`/`--tokenizer-model` flags to pick the desired pair + explicitly. +- The `find_model_file()` preferred type order can be reversed or modified in + code if you decide another default preference makes more sense in the future. + + +## Notes on Conversion Failures + +During quantization the converter may warn that certain tensors could not be +quantized (e.g. "Can't quantize tensor with shape … to Q8_0, falling back to +F16"). These warnings are informational; the converter still writes a valid +GGUF file but the particular tensor stays in higher precision. This behavior +is orthogonal to model selection and is a property of the converter itself. + +If you need a stricter conversion policy, you could modify the converter to +error out on fallback or to print a summary of how many tensors were actually +quantized versus kept in F16. + + +## Repository Notes + +The symbolic rules and CLI options are now documented in this file so future +contributors understand the design. The details are also mentioned in +`/memories/repo/model-artifacts.md` to assist long-lived memory-based tools. + +--- + +Any changes to supported quantization types or file naming should be reflected +both here and in the C++ discovery logic. diff --git a/README.md b/README.md index 9e7c59b..639cfdf 100644 --- a/README.md +++ b/README.md @@ -1,35 +1,3 @@ -# Qwen3-tts.cpp fork with saving/loading Speaker Embeddings and interactive mode - -you can precompute speaker embeddings (and load them later) to speed up speech generations. You can use an interactive mode to test a voice with many prompts. - -``` -Usage: ./build/qwen3-tts-cli [options] -m - -Options: - -m, --model Model directory (required) - -t, --text Text to synthesize (required unless interactive or saving speaker) - -i, --interactive Run in interactive loop mode (load once, generate many) - -o, --output Output WAV file (default: output.wav) - -r, --reference Reference audio for voice cloning - -s, --speaker Load precomputed speaker embedding (.spk) - --save-speaker Extract embedding from -r and save to file - --temperature Sampling temperature (default: 0.9, 0=greedy) - --top-k Top-k sampling (default: 50, 0=disabled) - --top-p Top-p sampling (default: 1.0) - --max-tokens Maximum audio tokens (default: 4096) - --repetition-penalty Repetition penalty (default: 1.05) - -l, --language Language: en,ru,zh,ja,ko,de,fr,es (default: en) - -j, --threads Number of threads (default: 4) - -h, --help Show this help - -Example: - ./build/qwen3-tts-cli -m ./models -t "Hello, world!" -o hello.wav - ./build/qwen3-tts-cli -m ./models -i -r reference.wav -o output.wav - ./build/qwen3-tts-cli -m ./models -r ref.wav --save-speaker voice.spk - ./build/qwen3-tts-cli -m ./models -s voice.spk -t "Hello, world!" -o output.wav -``` - - # qwen3-tts.cpp ![PyTorch vs qwen3-tts.cpp benchmark](./docs/benchmark_pytorch_vs_cpp.png) @@ -102,10 +70,10 @@ python scripts/setup_pipeline_models.py -o examples/readme_example_clone.wav ``` -Expected model artifacts after step 5: +Expected model artifacts after step 5 depend on the selected quantization. By default: -- `models/qwen3-tts-0.6b-f16.gguf` -- `models/qwen3-tts-tokenizer-f16.gguf` +- `models/qwen3-tts-0.6b-q4_k.gguf` +- `models/qwen3-tts-tokenizer-q8_0.gguf` - `models/coreml/code_predictor.mlpackage` (on macOS) Expected audio outputs after steps 6-7: @@ -159,9 +127,13 @@ python scripts/setup_pipeline_models.py Useful flags: - `--force` re-downloads and re-generates all artifacts. +- `--tts-type {f16,f32,q8_0,q4_k}` selects the main TTS model quantization. +- `--tokenizer-type {f16,f32,q8_0}` selects the tokenizer/vocoder quantization. - `--coreml auto|on|off` controls CoreML export behavior. - `--skip-download` skips HF download and uses existing local model dirs. +The CLI auto-detects `qwen3-tts-0.6b-*.gguf` and `qwen3-tts-tokenizer-*.gguf` files in `models/`, so you do not need to rename quantized outputs. + ## Manual Model Conversion (Advanced) Convert HuggingFace models to GGUF format: @@ -190,7 +162,14 @@ Place both `.gguf` files in a `models/` directory. # Basic synthesis ./build/qwen3-tts-cli -m models -t "Hello, world!" -o hello.wav -# Voice cloning from reference audio +The setup script can produce different quantization variants (`--tts-type` and `--tokenizer-type`), +and the CLI will auto-select the "best" file in the models folder. +Alternatively you can bypass discovery by specifying explicit GGUF filenames: + +``` +./build/qwen3-tts-cli -m models --tts-model qwen3-tts-0.6b-q4_k.gguf \ + --tokenizer-model qwen3-tts-tokenizer-q8_0.gguf -t "Hello" -o out.wav +``` ./build/qwen3-tts-cli -m models -t "Hello! How are you?" -r reference.wav -o cloned.wav # Greedy decoding with max length diff --git a/scripts/setup_pipeline_models.py b/scripts/setup_pipeline_models.py index 2becf93..cd1371f 100755 --- a/scripts/setup_pipeline_models.py +++ b/scripts/setup_pipeline_models.py @@ -3,14 +3,17 @@ One-shot model setup for qwen3-tts.cpp. This script downloads required Hugging Face model assets and generates all model -artifacts needed by the final C++ pipeline: +artifacts needed by the final C++ pipeline. -- models/qwen3-tts-0.6b-f16.gguf -- models/qwen3-tts-tokenizer-f16.gguf +The GGUF output filenames include their selected quantization types, for +example: + +- models/qwen3-tts-0.6b-q4_k.gguf +- models/qwen3-tts-tokenizer-q8_0.gguf - models/coreml/code_predictor.mlpackage (optional, macOS) Example: - python scripts/setup_pipeline_models.py + python scripts/setup_pipeline_models.py --tts-type q4_k --tokenizer-type q8_0 Minimal usage for CI/offline conversion: python scripts/setup_pipeline_models.py --skip-download @@ -40,6 +43,12 @@ "Qwen/Qwen3-TTS-Tokenizer-12Hz", ] +TTS_GGUF_TYPES = ("f16", "f32", "q8_0", "q4_k") +TOKENIZER_GGUF_TYPES = ("f16", "f32", "q8_0") + +DEFAULT_TTS_GGUF_TYPE = "q4_k" +DEFAULT_TOKENIZER_GGUF_TYPE = "q8_0" + def eprint(msg: str) -> None: print(msg, file=sys.stderr) @@ -155,12 +164,18 @@ def ensure_tokenizer_assets( return tokenizer_dir +def gguf_output_path(models_dir: Path, stem: str, output_type: str) -> Path: + return models_dir / f"{stem}-{output_type}.gguf" + + def convert_gguf( python_exe: str, base_dir: Path, tokenizer_input_dir: Path, out_tts: Path, out_tok: Path, + tts_type: str, + tokenizer_type: str, force_convert: bool, ) -> None: require_modules( @@ -188,7 +203,7 @@ def convert_gguf( "--output", str(out_tts), "--type", - "f16", + tts_type, ], cwd=REPO_ROOT, ) @@ -205,7 +220,7 @@ def convert_gguf( "--output", str(out_tok), "--type", - "f16", + tokenizer_type, ], cwd=REPO_ROOT, ) @@ -258,6 +273,24 @@ def parse_args() -> argparse.Namespace: p.add_argument("--hf-token", default=os.environ.get("HF_TOKEN", ""), help="Hugging Face token (or set HF_TOKEN)") p.add_argument("--skip-download", action="store_true", help="Skip model downloads") p.add_argument("--skip-gguf", action="store_true", help="Skip GGUF conversion") + p.add_argument( + "--tts-type", + choices=TTS_GGUF_TYPES, + default=DEFAULT_TTS_GGUF_TYPE, + help=( + "Quantization for the main TTS GGUF " + f"(default: {DEFAULT_TTS_GGUF_TYPE})" + ), + ) + p.add_argument( + "--tokenizer-type", + choices=TOKENIZER_GGUF_TYPES, + default=DEFAULT_TOKENIZER_GGUF_TYPE, + help=( + "Quantization for the tokenizer/vocoder GGUF " + f"(default: {DEFAULT_TOKENIZER_GGUF_TYPE})" + ), + ) p.add_argument( "--coreml", choices=["auto", "on", "off"], @@ -274,8 +307,8 @@ def main() -> int: models_dir = Path(args.models_dir).resolve() base_dir = models_dir / "Qwen3-TTS-12Hz-0.6B-Base" tokenizer_dir = models_dir / "Qwen3-TTS-Tokenizer-12Hz" - out_tts = models_dir / "qwen3-tts-0.6b-f16.gguf" - out_tok = models_dir / "qwen3-tts-tokenizer-f16.gguf" + out_tts = gguf_output_path(models_dir, "qwen3-tts-0.6b", args.tts_type) + out_tok = gguf_output_path(models_dir, "qwen3-tts-tokenizer", args.tokenizer_type) out_coreml = models_dir / "coreml" / "code_predictor.mlpackage" hf_token = args.hf_token.strip() or None @@ -290,7 +323,16 @@ def main() -> int: tokenizer_input_dir = base_dir if (base_dir / "speech_tokenizer" / "model.safetensors").exists() else tokenizer_dir if not args.skip_gguf: - convert_gguf(sys.executable, base_dir, tokenizer_input_dir, out_tts, out_tok, args.force) + convert_gguf( + sys.executable, + base_dir, + tokenizer_input_dir, + out_tts, + out_tok, + args.tts_type, + args.tokenizer_type, + args.force, + ) wants_coreml = args.coreml == "on" or (args.coreml == "auto" and platform.system() == "Darwin") if wants_coreml: diff --git a/src/main.cpp b/src/main.cpp index b609099..73d5880 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -9,6 +9,8 @@ void print_usage(const char * program) { fprintf(stderr, "\n"); fprintf(stderr, "Options:\n"); fprintf(stderr, " -m, --model Model directory (required)\n"); + fprintf(stderr, " --tts-model Explicit TTS model GGUF (overrides discovery)\n"); + fprintf(stderr, " --tokenizer-model Explicit tokenizer/vocoder GGUF\n"); fprintf(stderr, " -t, --text Text to synthesize (required)\n"); fprintf(stderr, " -o, --output Output WAV file (default: output.wav)\n"); fprintf(stderr, " -r, --reference Reference audio for voice cloning\n"); @@ -31,6 +33,8 @@ int main(int argc, char ** argv) { std::string text; std::string output_file = "output.wav"; std::string reference_audio; + std::string explicit_tts_model; + std::string explicit_tokenizer_model; qwen3_tts::tts_params params; @@ -65,6 +69,18 @@ int main(int argc, char ** argv) { return 1; } reference_audio = argv[i]; + } else if (arg == "--tts-model") { + if (++i >= argc) { + fprintf(stderr, "Error: missing tts model path\n"); + return 1; + } + explicit_tts_model = argv[i]; + } else if (arg == "--tokenizer-model") { + if (++i >= argc) { + fprintf(stderr, "Error: missing tokenizer model path\n"); + return 1; + } + explicit_tokenizer_model = argv[i]; } else if (arg == "--temperature") { if (++i >= argc) { fprintf(stderr, "Error: missing temperature value\n"); @@ -145,7 +161,7 @@ int main(int argc, char ** argv) { qwen3_tts::Qwen3TTS tts; fprintf(stderr, "Loading models from: %s\n", model_dir.c_str()); - if (!tts.load_models(model_dir)) { + if (!tts.load_models(model_dir, explicit_tts_model, explicit_tokenizer_model)) { fprintf(stderr, "Error: %s\n", tts.get_error().c_str()); return 1; } diff --git a/src/qwen3_tts.cpp b/src/qwen3_tts.cpp index c2394b5..9234649 100644 --- a/src/qwen3_tts.cpp +++ b/src/qwen3_tts.cpp @@ -3,11 +3,13 @@ #include #include +#include #include #include -#include #include #include +#include +#include #ifdef __APPLE__ #include @@ -17,6 +19,8 @@ namespace qwen3_tts { +namespace fs = std::filesystem; + static int64_t get_time_ms() { return std::chrono::duration_cast( std::chrono::steady_clock::now().time_since_epoch()).count(); @@ -81,6 +85,46 @@ static void log_memory_usage(const char * label) { format_bytes(mem.phys_footprint_bytes).c_str()); } +static std::string find_model_file(const std::string & model_dir, const std::string & stem) { + const fs::path dir(model_dir); + std::error_code ec; + if (!fs::exists(dir, ec) || !fs::is_directory(dir, ec)) { + return ""; + } + + static const char * preferred_types[] = { "f16", "q8_0", "q4_k", "f32" }; + for (const char * type : preferred_types) { + const fs::path candidate = dir / (stem + "-" + type + ".gguf"); + ec.clear(); + if (fs::exists(candidate, ec) && fs::is_regular_file(candidate, ec)) { + return candidate.string(); + } + } + + std::vector matches; + for (fs::directory_iterator it(dir, ec), end; !ec && it != end; it.increment(ec)) { + const fs::directory_entry & entry = *it; + ec.clear(); + if (!entry.is_regular_file(ec)) { + ec.clear(); + continue; + } + + const fs::path path = entry.path(); + const std::string filename = path.filename().string(); + if (path.extension() == ".gguf" && filename.rfind(stem + "-", 0) == 0) { + matches.push_back(path.string()); + } + } + + if (matches.empty()) { + return ""; + } + + std::sort(matches.begin(), matches.end()); + return matches.front(); +} + static void resample_linear(const float * input, int input_len, int input_rate, std::vector & output, int output_rate) { double ratio = (double)input_rate / output_rate; @@ -105,7 +149,9 @@ Qwen3TTS::Qwen3TTS() = default; Qwen3TTS::~Qwen3TTS() = default; -bool Qwen3TTS::load_models(const std::string & model_dir) { +bool Qwen3TTS::load_models(const std::string & model_dir, + const std::string & tts_model_override, + const std::string & tokenizer_model_override) { int64_t t_start = get_time_ms(); log_memory_usage("load/start"); @@ -114,9 +160,37 @@ bool Qwen3TTS::load_models(const std::string & model_dir) { transformer_loaded_ = false; decoder_loaded_ = false; - // Construct model paths - std::string tts_model_path = model_dir + "/qwen3-tts-0.6b-f16.gguf"; - std::string tokenizer_model_path = model_dir + "/qwen3-tts-tokenizer-f16.gguf"; + // Determine which files to load. The caller may override by providing + // explicit paths, otherwise we auto-discover based on stem + preferred + // ordering. + std::string tts_model_path; + if (!tts_model_override.empty()) { + tts_model_path = tts_model_override; + if (!fs::path(tts_model_path).is_absolute()) { + tts_model_path = fs::path(model_dir) / tts_model_path; + } + } else { + tts_model_path = find_model_file(model_dir, "qwen3-tts-0.6b"); + if (tts_model_path.empty()) { + error_msg_ = "No TTS model found in " + model_dir + " (expected qwen3-tts-0.6b-*.gguf)"; + return false; + } + } + + std::string tokenizer_model_path; + if (!tokenizer_model_override.empty()) { + tokenizer_model_path = tokenizer_model_override; + if (!fs::path(tokenizer_model_path).is_absolute()) { + tokenizer_model_path = fs::path(model_dir) / tokenizer_model_path; + } + } else { + tokenizer_model_path = find_model_file(model_dir, "qwen3-tts-tokenizer"); + if (tokenizer_model_path.empty()) { + error_msg_ = "No tokenizer model found in " + model_dir + " (expected qwen3-tts-tokenizer-*.gguf)"; + return false; + } + } + tts_model_path_ = tts_model_path; decoder_model_path_ = tokenizer_model_path; encoder_loaded_ = false; diff --git a/src/qwen3_tts.h b/src/qwen3_tts.h index 80dcc3e..e94d46f 100644 --- a/src/qwen3_tts.h +++ b/src/qwen3_tts.h @@ -86,7 +86,12 @@ class Qwen3TTS { // Load all models from directory // model_dir should contain: transformer.gguf, tokenizer.gguf, vocoder.gguf - bool load_models(const std::string & model_dir); + // If `tts_model_path` or `tokenizer_model_path` are non-empty they override + // the automatic discovery logic and are used verbatim. The overrides can + // either be absolute paths or relative to `model_dir`. + bool load_models(const std::string & model_dir, + const std::string & tts_model_path = std::string(), + const std::string & tokenizer_model_path = std::string()); // Generate speech from text // text: input text to synthesize