Added selectable quantization levels

QUANTIZATION_SELECTION.md

@@ -0,0 +1,163 @@
+# Quantization Selection Guide
+
+This document describes all of the mechanisms provided by the
+`qwen3-tts.cpp` repository for selecting which quantized model artifacts are
+used during setup and runtime.  The goal is to make it trivial to generate or
+load *any* supported quantization type, and to understand how the system
+chooses a file when multiple versions are present.
+
+---
+
+## Supported Quantization Types
+
+- **TTS model** (`qwen3-tts-0.6b` stem):
+  - `f16`  (no quantization)
+  - `f32`  (float32; larger than f16)
+  - `q8_0` (8-bit GGUF quantization)
+  - `q4_k` (4-bit GGUF quantization; recommended for smaller size)
+
+- **Tokenizer/vocoder model** (`qwen3-tts-tokenizer` stem):
+  - `f16`
+  - `f32`
+  - `q8_0`  (*q4_k is not supported by the converter*)
+
+These types correspond to the `--type` argument of the conversion scripts
+`convert_tts_to_gguf.py` and `convert_tokenizer_to_gguf.py`.
+
+
+## Setup Script (`scripts/setup_pipeline_models.py`)
+
+The one‑shot model setup script can now produce any of the above quantized
+outputs.  Two new CLI options were added:
+
+```bash
+--tts-type {f16,f32,q8_0,q4_k}          # default: q4_k
+--tokenizer-type {f16,f32,q8_0}          # default: q8_0
+```
+
+When you run the script, it names the resulting GGUF files accordingly:
+
+- `models/qwen3-tts-0.6b-<tts-type>.gguf`
+- `models/qwen3-tts-tokenizer-<tokenizer-type>.gguf`
+
+This file naming is the convention that the runtime loader uses to pick models.
+
+Example:
+
+```bash
+python scripts/setup_pipeline_models.py \
+    --tts-type q4_k --tokenizer-type q8_0
+```
+
+`--skip-download` and `--force` still work exactly as before.  The readme
+documentation has been updated with the new flags and examples.
+
+
+## Runtime Model Discovery and Explicit Selection
+
+`src/qwen3_tts.cpp` contains the logic for locating GGUF files when the CLI is
+called with `-m <model_dir>`. Earlier versions hard‑coded
+`qwen3-tts-0.6b-f16.gguf` and
+`qwen3-tts-tokenizer-f16.gguf`; this has been replaced with
+`find_model_file()`:
+
+```cpp
+static std::string find_model_file(const std::string &model_dir,
+                                   const std::string &stem) {
+    static const char *preferred[] = {"f16", "q8_0", "q4_k", "f32"};
+    // look for stem-type.gguf in preferred order, otherwise return first match
+}
+```
+
+This function returns the first existing file matching `stem-*.gguf` using a
+preferred type ordering.  By default `f16` wins, so if *only* a `q4_k` file is
+present it will still be selected, but if both `f16` and `q4_k` exist the
+script keeps picking `f16` unless instructed otherwise.
+
+To give callers complete control, the model‑loading API was extended:
+
+```cpp
+bool Qwen3TTS::load_models(const std::string &model_dir,
+                            const std::string &tts_model_path = {},
+                            const std::string &tokenizer_model_path = {});
+```
+
+If either override argument is non‑empty it is used verbatim (absolute paths are
+accepted; relative paths are joined with `model_dir`).  These parameters are
+exposed via two new CLI options:
+
+```
+--tts-model <file>        # path to a GGUF, overriding discovery
+--tokenizer-model <file>  # same for tokenizer/vocoder
+```
+
+Examples:
+
+```bash
+./build/qwen3-tts-cli -m models --tts-model qwen3-tts-0.6b-q4_k.gguf \
+    --tokenizer-model qwen3-tts-tokenizer-q8_0.gguf -t "Hello" -o out.wav
+```
+
+The help text and README explain both the automatic and explicit modes.
+
+
+## Discoverability Rules Summary
+
+1. If explicit paths are given on the CLI, they are used.
+2. Otherwise, `find_model_file()` searches `model_dir` for
+   `stem-*.gguf`:
+   - Preferred order is `f16`, `q8_0`, `q4_k`, `f32`.
+   - If none of the preferred types exist but any file matches `stem-*.gguf`,
+     the lexicographically first match is returned.
+   - If no file is found an error is reported and loading fails.
+
+This allows mixed directories such as:
+
+```
+models/
+  qwen3-tts-0.6b-f16.gguf
+  qwen3-tts-0.6b-q4_k.gguf
+  qwen3-tts-tokenizer-q8_0.gguf
+```
+
+…to behave predictably while still enabling explicit selection when desired.
+
+
+## Practical Advice
+
+- **For size-conscious deployments**, run the setup script with `--tts-type
+  q4_k` and `--tokenizer-type q8_0`.  The resulting files are about 1.8 GB and
+  273 MB respectively (after mixed-precision fallbacks).
+- **For development or debugging**, using `f16` models avoids quantization
+  variability and is the default when running `./build/qwen3-tts-cli` without
+  any tuning.
+- **When multiple quantizations coexist** (e.g. you’re evaluating trade‑offs),
+  add the `--tts-model`/`--tokenizer-model` flags to pick the desired pair
+  explicitly.
+- The `find_model_file()` preferred type order can be reversed or modified in
+  code if you decide another default preference makes more sense in the future.
+
+
+## Notes on Conversion Failures
+
+During quantization the converter may warn that certain tensors could not be
+quantized (e.g. "Can't quantize tensor with shape … to Q8_0, falling back to
+F16").  These warnings are informational; the converter still writes a valid
+GGUF file but the particular tensor stays in higher precision.  This behavior
+is orthogonal to model selection and is a property of the converter itself.
+
+If you need a stricter conversion policy, you could modify the converter to
+error out on fallback or to print a summary of how many tensors were actually
+quantized versus kept in F16.
+
+
+## Repository Notes
+
+The symbolic rules and CLI options are now documented in this file so future
+contributors understand the design.  The details are also mentioned in
+`/memories/repo/model-artifacts.md` to assist long-lived memory-based tools.
+
+---
+
+Any changes to supported quantization types or file naming should be reflected
+both here and in the C++ discovery logic.

README.md

@@ -1,35 +1,3 @@
-# Qwen3-tts.cpp fork with saving/loading Speaker Embeddings and interactive mode
-
-you can precompute speaker embeddings (and load them later) to speed up speech generations. You can use an interactive mode to test a voice with many prompts.
-
-```
-Usage: ./build/qwen3-tts-cli [options] -m <model_dir>
-
-Options:
-  -m, --model <dir>      Model directory (required)
-  -t, --text <text>      Text to synthesize (required unless interactive or saving speaker)
-  -i, --interactive      Run in interactive loop mode (load once, generate many)
-  -o, --output <file>    Output WAV file (default: output.wav)
-  -r, --reference <file> Reference audio for voice cloning
-  -s, --speaker <file>   Load precomputed speaker embedding (.spk)
-  --save-speaker <file>  Extract embedding from -r and save to file
-  --temperature <val>    Sampling temperature (default: 0.9, 0=greedy)
-  --top-k <n>            Top-k sampling (default: 50, 0=disabled)
-  --top-p <val>          Top-p sampling (default: 1.0)
-  --max-tokens <n>       Maximum audio tokens (default: 4096)
-  --repetition-penalty <val> Repetition penalty (default: 1.05)
-  -l, --language <lang>  Language: en,ru,zh,ja,ko,de,fr,es (default: en)
-  -j, --threads <n>      Number of threads (default: 4)
-  -h, --help             Show this help
-
-Example:
-  ./build/qwen3-tts-cli -m ./models -t "Hello, world!" -o hello.wav
-  ./build/qwen3-tts-cli -m ./models -i -r reference.wav -o output.wav
-  ./build/qwen3-tts-cli -m ./models -r ref.wav --save-speaker voice.spk
-  ./build/qwen3-tts-cli -m ./models -s voice.spk -t "Hello, world!" -o output.wav
-```
-
-
 # qwen3-tts.cpp
 
 ![PyTorch vs qwen3-tts.cpp benchmark](./docs/benchmark_pytorch_vs_cpp.png)
@@ -102,10 +70,10 @@ python scripts/setup_pipeline_models.py
   -o examples/readme_example_clone.wav
 ```
 
-Expected model artifacts after step 5:
+Expected model artifacts after step 5 depend on the selected quantization. By default:
 
-- `models/qwen3-tts-0.6b-f16.gguf`
-- `models/qwen3-tts-tokenizer-f16.gguf`
+- `models/qwen3-tts-0.6b-q4_k.gguf`
+- `models/qwen3-tts-tokenizer-q8_0.gguf`
 - `models/coreml/code_predictor.mlpackage` (on macOS)
 
 Expected audio outputs after steps 6-7:
@@ -159,9 +127,13 @@ python scripts/setup_pipeline_models.py
 Useful flags:
 
 - `--force` re-downloads and re-generates all artifacts.
+- `--tts-type {f16,f32,q8_0,q4_k}` selects the main TTS model quantization.
+- `--tokenizer-type {f16,f32,q8_0}` selects the tokenizer/vocoder quantization.
 - `--coreml auto|on|off` controls CoreML export behavior.
 - `--skip-download` skips HF download and uses existing local model dirs.
 
+The CLI auto-detects `qwen3-tts-0.6b-*.gguf` and `qwen3-tts-tokenizer-*.gguf` files in `models/`, so you do not need to rename quantized outputs.
+
 ## Manual Model Conversion (Advanced)
 
 Convert HuggingFace models to GGUF format:
@@ -190,7 +162,14 @@ Place both `.gguf` files in a `models/` directory.
 # Basic synthesis
 ./build/qwen3-tts-cli -m models -t "Hello, world!" -o hello.wav
 
-# Voice cloning from reference audio
+The setup script can produce different quantization variants (`--tts-type` and `--tokenizer-type`),
+and the CLI will auto-select the "best" file in the models folder.
+Alternatively you can bypass discovery by specifying explicit GGUF filenames:
+
+```
+./build/qwen3-tts-cli -m models --tts-model qwen3-tts-0.6b-q4_k.gguf \
+    --tokenizer-model qwen3-tts-tokenizer-q8_0.gguf -t "Hello" -o out.wav
+```
 ./build/qwen3-tts-cli -m models -t "Hello! How are you?" -r reference.wav -o cloned.wav
 
 # Greedy decoding with max length

scripts/setup_pipeline_models.py

@@ -3,14 +3,17 @@
 One-shot model setup for qwen3-tts.cpp.
 
 This script downloads required Hugging Face model assets and generates all model
-artifacts needed by the final C++ pipeline:
+artifacts needed by the final C++ pipeline.
 
-- models/qwen3-tts-0.6b-f16.gguf
-- models/qwen3-tts-tokenizer-f16.gguf
+The GGUF output filenames include their selected quantization types, for
+example:
+
+- models/qwen3-tts-0.6b-q4_k.gguf
+- models/qwen3-tts-tokenizer-q8_0.gguf
 - models/coreml/code_predictor.mlpackage (optional, macOS)
 
 Example:
-  python scripts/setup_pipeline_models.py
+    python scripts/setup_pipeline_models.py --tts-type q4_k --tokenizer-type q8_0
 
 Minimal usage for CI/offline conversion:
   python scripts/setup_pipeline_models.py --skip-download
@@ -40,6 +43,12 @@
     "Qwen/Qwen3-TTS-Tokenizer-12Hz",
 ]
 
+TTS_GGUF_TYPES = ("f16", "f32", "q8_0", "q4_k")
+TOKENIZER_GGUF_TYPES = ("f16", "f32", "q8_0")
+
+DEFAULT_TTS_GGUF_TYPE = "q4_k"
+DEFAULT_TOKENIZER_GGUF_TYPE = "q8_0"
+
 
 def eprint(msg: str) -> None:
     print(msg, file=sys.stderr)
@@ -155,12 +164,18 @@ def ensure_tokenizer_assets(
     return tokenizer_dir
 
 
+def gguf_output_path(models_dir: Path, stem: str, output_type: str) -> Path:
+    return models_dir / f"{stem}-{output_type}.gguf"
+
+
 def convert_gguf(
     python_exe: str,
     base_dir: Path,
     tokenizer_input_dir: Path,
     out_tts: Path,
     out_tok: Path,
+    tts_type: str,
+    tokenizer_type: str,
     force_convert: bool,
 ) -> None:
     require_modules(
@@ -188,7 +203,7 @@ def convert_gguf(
                 "--output",
                 str(out_tts),
                 "--type",
-                "f16",
+                tts_type,
             ],
             cwd=REPO_ROOT,
         )
@@ -205,7 +220,7 @@ def convert_gguf(
                 "--output",
                 str(out_tok),
                 "--type",
-                "f16",
+                tokenizer_type,
             ],
             cwd=REPO_ROOT,
         )
@@ -258,6 +273,24 @@ def parse_args() -> argparse.Namespace:
     p.add_argument("--hf-token", default=os.environ.get("HF_TOKEN", ""), help="Hugging Face token (or set HF_TOKEN)")
     p.add_argument("--skip-download", action="store_true", help="Skip model downloads")
     p.add_argument("--skip-gguf", action="store_true", help="Skip GGUF conversion")
+    p.add_argument(
+        "--tts-type",
+        choices=TTS_GGUF_TYPES,
+        default=DEFAULT_TTS_GGUF_TYPE,
+        help=(
+            "Quantization for the main TTS GGUF "
+            f"(default: {DEFAULT_TTS_GGUF_TYPE})"
+        ),
+    )
+    p.add_argument(
+        "--tokenizer-type",
+        choices=TOKENIZER_GGUF_TYPES,
+        default=DEFAULT_TOKENIZER_GGUF_TYPE,
+        help=(
+            "Quantization for the tokenizer/vocoder GGUF "
+            f"(default: {DEFAULT_TOKENIZER_GGUF_TYPE})"
+        ),
+    )
     p.add_argument(
         "--coreml",
         choices=["auto", "on", "off"],
@@ -274,8 +307,8 @@ def main() -> int:
     models_dir = Path(args.models_dir).resolve()
     base_dir = models_dir / "Qwen3-TTS-12Hz-0.6B-Base"
     tokenizer_dir = models_dir / "Qwen3-TTS-Tokenizer-12Hz"
-    out_tts = models_dir / "qwen3-tts-0.6b-f16.gguf"
-    out_tok = models_dir / "qwen3-tts-tokenizer-f16.gguf"
+    out_tts = gguf_output_path(models_dir, "qwen3-tts-0.6b", args.tts_type)
+    out_tok = gguf_output_path(models_dir, "qwen3-tts-tokenizer", args.tokenizer_type)
     out_coreml = models_dir / "coreml" / "code_predictor.mlpackage"
 
     hf_token = args.hf_token.strip() or None
@@ -290,7 +323,16 @@ def main() -> int:
         tokenizer_input_dir = base_dir if (base_dir / "speech_tokenizer" / "model.safetensors").exists() else tokenizer_dir
 
     if not args.skip_gguf:
-        convert_gguf(sys.executable, base_dir, tokenizer_input_dir, out_tts, out_tok, args.force)
+        convert_gguf(
+            sys.executable,
+            base_dir,
+            tokenizer_input_dir,
+            out_tts,
+            out_tok,
+            args.tts_type,
+            args.tokenizer_type,
+            args.force,
+        )
 
     wants_coreml = args.coreml == "on" or (args.coreml == "auto" and platform.system() == "Darwin")
     if wants_coreml: