Skip to content

vibevoice asr #52

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cjpais wants to merge 4 commits into
base: main
Choose a base branch
from
Open

vibevoice asr #52

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d76d14a
stage 1
cjpais Jun 26, 2026
33ad944
stage 2
cjpais Jun 27, 2026
e292dcf
stage 4
cjpais Jun 27, 2026
56f55f2
partial vibevoice bu
cjpais Jun 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
151 changes: 151 additions & 0 deletions docs/porting/families/vibevoice.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# VibeVoice

Status: research

## Identity

- Family key: `vibevoice`
- Upstream architecture string: `VibeVoiceForASRTraining` (model_type `vibevoice`)
- Hugging Face repo: `microsoft/VibeVoice-ASR`
- Hugging Face revision: `d0c9efdb8d614685062c04425d91e01b6f37d944`
- License: MIT
- Variants: `vibevoice-asr` (~9B params, ~17.4 GB BF16)

## References

- Canonical reference: `github.com/microsoft/VibeVoice` — the `vibevoice` Python package
(`vibevoice.modular.modeling_vibevoice_asr.VibeVoiceASRForConditionalGeneration`,
`vibevoice.processor.vibevoice_asr_processor.VibeVoiceASRProcessor`). The HF repo
ships weights + config.json only; no modeling code, processor, tokenizer, or
generation_config.
- Instrumented reference: same package (`demo/vibevoice_asr_inference_from_file.py`).
- Cross-check references: tech report arXiv:2601.18184; `github.com/localai-org/vibevoice.cpp`
(community ggml port — targets VibeVoice **TTS** and small Qwen2.5-0.5B variants, useful
as a causal-conv / tokenizer reference only, not a drop-in for the 9B ASR model).

LM backbone is Qwen2.5-7B (decoder_config: hidden 3584, 28 layers, 28 heads / 4 KV,
intermediate 18944, vocab 152064, rope_theta 1e6, max_position 131072). The tokenizer
is the stock `Qwen/Qwen2.5-7B` byte-level BPE, loaded by the processor at runtime.

Frontend is **not** a spectrogram. Raw 24 kHz waveform feeds two parallel causal-conv
VAE encoders (acoustic vae_dim 64, semantic vae_dim 128), each with cumulative 3200x
downsampling → ~7.5 Hz token rate.

## Commands

Reference run:

```bash
# from a checkout of github.com/microsoft/VibeVoice
uv run --project scripts/envs/vibevoice \
python demo/vibevoice_asr_inference_from_file.py \
--model_path microsoft/VibeVoice-ASR --audio_files samples/jfk.wav
```

Reference dumps (Modal — the 9B model needs a >=40 GB GPU; canonical dumper is
`scripts/dump_reference_vibevoice_author.py`, hosted by `_vibevoice_modal_dump.py`):

```bash
uv run --project scripts/envs/qwen3_asr modal run scripts/_vibevoice_modal_dump.py
# -> build/validate/vibevoice/vibevoice-asr/jfk/ref/ (+ envelope.json, transcript.json)
uv run scripts/build_vibevoice_oracle.py # dump_coverage.json + provisional tolerances
```

Reference WER (Modal L40S; ~2 h for full test-clean):

```bash
uv run --project scripts/envs/qwen3_asr modal run \
scripts/wer/remote/modal_sweep.py::reference_sweep \
--variants vibevoice:vibevoice-asr --gpu L40S
uv run scripts/wer/score.py reports/wer/vibevoice-asr-REF.librispeech-test-clean.jsonl
```

Conversion:

```bash
TODO # porting-3-convert: scripts/convert-vibevoice.py
```

Validation:

```bash
uv run scripts/validate.py all --family vibevoice --variant vibevoice-asr
```

Benchmarks:

```bash
TODO # porting-6-bench
```

## Capability Validation

Proposed `Target` values below; non-forced rows require user sign-off at intake.

| Capability | Mode | Command / test | Expected observable | Target | Status |
|------------|------|----------------|---------------------|--------|--------|
| Transcribe | explicit language hint | `build/bin/transcribe-cli -m models/vibevoice-asr/vibevoice-asr-BF16.gguf --language en samples/jfk.wav` | non-empty plausible English transcript | MUST PASS | TODO |
| Transcribe | auto / no language hint | `build/bin/transcribe-cli -m models/vibevoice-asr/vibevoice-asr-BF16.gguf samples/jfk.wav` | non-empty plausible transcript on the auto-detect path | MUST PASS | TODO |
| Translate | only if exposed | n/a | — | OUT OF SCOPE — model transcribes (incl. code-switching) but does not translate; not advertised | TODO |
| Segment timestamps | structured Who/When/What output | `<TODO structured-output parse>` | segment-level timestamps present in parsed output | MUST PASS | TODO |
| Word timestamps | only if exposed | n/a | — | OUT OF SCOPE — model emits segment-level, not word-level timestamps | TODO |
| Speaker diarization | structured Who/When/What output | `<TODO diarization parse>` | speaker labels present in parsed output, DER/cpWER scored | MUST PASS | TODO |
| Hotwords / context bias | prompt-level biasing | `<TODO hotword prompt>` | biased transcript on domain terms | OUT OF SCOPE — first port uses the default prompt; hotword prompt path deferred | TODO |
| Streaming | non-streaming model | n/a | — | OUT OF SCOPE — model is non-streaming (60-min single pass); `capabilities.streaming: false` | TODO |
| Batch (offline) | run_batch vs serial | `uv run scripts/batch_parity.py --model models/vibevoice-asr/vibevoice-asr-BF16.gguf --samples-dir samples/wer/librispeech-test-clean --batch-sizes 2,4,8 --backend cpu` | byte-identical hypotheses + CPU tensor parity | MUST PASS | TODO |

## Notes

- Acceptance dataset: LibriSpeech test-clean (English supported; publisher does not report
LibriSpeech, so the gate is the measured Oracle reference baseline, not a publisher score).
**Measured Oracle reference WER: 2.11%** (2620 utts, 0 errors, CI [1.93%, 2.28%]; mean
acoustic path, BF16 LM, L40S). Publisher-reported English WER on other sets: 7.99%
MLC-Challenge, 18.81% AMI-IHM (harder/multi-speaker — not directly comparable).
- The model always emits structured Who/When/What output. Plain-WER scoring requires a
normalizer that strips speaker tags + timestamps from the hypothesis (Stage 2 / Stage 7).
- Segment timestamps and speaker diarization are **in scope (MUST PASS)** for this port:
Stage 2/Stage 4 must parse the structured output and Stage 7 must score it (timestamps +
DER/cpWER), in addition to plain transcript WER.
- Long-form (60-minute / 64K-token) inference is deferred; first port targets short utterances
(jfk / LibriSpeech segments). See intake `known_risks` for the full list, including the
learned-VAE frontend, dual-tokenizer fusion, the unused diffusion head, and KV-memory at
long context.

### Conversion (Stage 3)

Reference-dtype GGUF: `models/VibeVoice-ASR/VibeVoice-ASR-BF16.gguf` (slug-cased dir per the
`find_gguf` slug-driven convention, same as `Qwen3-ASR-0.6B`). Converter
`scripts/convert-vibevoice.py`; manifest `reports/convert/vibevoice-asr-BF16.json`. Streams
shards one tensor at a time (never instantiates the 9B model), so it runs on a 16 GB machine.

Tensor-mapping decisions (the contract Stage 4 `src/arch/vibevoice/weights.cpp` implements
against):

- **901 tensors kept, 276 dropped.** Dropped = `model.acoustic_tokenizer.decoder.*` (the
VibeVoice TTS acoustic-decoder path; ASR calls only `acoustic_tokenizer.encode(...).mean`).
The config's `diffusion_head_config` ships **no weights** in the ASR checkpoint. The
`semantic_tokenizer` is encoder-only.
- **Naming:** Qwen2.5 LM → `dec.token_embd` / `dec.output_norm` / `dec.output` (untied head;
`lm_head.weight` ships separately, `tie_word_embeddings=false`) + `dec.blocks.{i}.{norm_attn,
attn.{q,k,v,o}[.bias],norm_ffn,ffn.{gate,up,down}}` (Qwen2 carries q/k/v biases, no o bias).
Connectors → `conn.{acoustic,semantic}.{fc1,norm,fc2}`. VAE encoders → `enc.{acoustic,
semantic}.<source-suffix verbatim>` (suffix preserved, incl. the `conv.conv[.conv]` SConv1d
nesting).
- **Fusion is a Stage-4 runtime op, not baked in:** `enc.combined = acoustic_feat +
semantic_feat` (element-wise sum of the two connector outputs), then scattered into the LM
embedding stream at `speech_pad` positions. The converter only stores both encoders + both
connectors.
- **Per-tensor dtype (BF16 reference):** norms / biases / VAE layer-scale `gamma`+`ffn_gamma`
→ F32; all VAE causal-conv kernels → F16 (loader has no BF16 conv; F16 also out-precisions
BF16); linears / embed / head → BF16. Result: 306 BF16, 527 F32, 68 F16.
- **Tokenizer** pulled from stock `Qwen/Qwen2.5-7B` (152064 tokens, 151387 merges, gpt2/qwen2
byte-level BPE). Speech markers reuse existing Qwen specials: `speech_start=<|object_ref_start|>`
(151646), `speech_end=<|object_ref_end|>` (151647), `speech_pad=<|box_start|>` (151648);
`eos=<|endoftext|>` (151643). Chat template, `SYSTEM_PROMPT`, the three speech-token ids,
`im_start`/`im_end`, and `speech_tok_compress_ratio=3200` are stored as `stt.vibevoice.*` /
`tokenizer.*` KV so Stage 4 can rebuild the prompt (which is otherwise assembled dynamically
from the audio duration and a `ceil(n_samples/3200)`-long speech-pad run).
- **Frontend:** `stt.frontend.type=raw_waveform`, `sample_rate=24000`; no STFT/mel buffers.

Preflight Gate B: **PASS** (dtype, frontend, tokenizer, architecture, capabilities). Loader-open
smoke returns `unsupported architecture` (expected — `src/arch/vibevoice` is a Stage 4 artifact).
129 changes: 129 additions & 0 deletions reports/porting/vibevoice/forward-map.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Forward map - vibevoice

Reference: `models/_vendor/VibeVoice/vibevoice/` (author `vibevoice` package) @ HF rev
`d0c9efdb8d614685062c04425d91e01b6f37d944`. Canonical modules:
`modular/modeling_vibevoice_asr.py`, `modular/modular_vibevoice_tokenizer.py`,
`processor/vibevoice_asr_processor.py`.
Closest in-tree analog: `src/arch/qwen3_asr/` (audio-llm: audio frontend + Qwen causal LM
with audio-token injection). The VAE conv frontend has **no** in-tree analog.

Audio-LLM. Raw 24 kHz waveform -> two parallel causal-conv VAE encoders (acoustic vae_dim 64,
semantic vae_dim 128) -> SpeechConnector (-> 3584) each -> **element-wise SUM** -> scattered
into the Qwen2.5-7B LM embedding stream at `speech_pad` positions -> 28 Qwen2 layers -> lm_head
-> greedy decode of structured JSON. **Contract uses the acoustic VAE MEAN, never a sample.**

## Frontend

| Stage | Reference location | Output shape | Gate tensor | ggml / C++ pattern | In-tree analog |
|-------|--------------------|--------------|-------------|--------------------|----------------|
| Resample to 24 kHz mono | processor `__call__` (librosa load sr=24000) | [samples] | — | host-side resample (input PCM) | qwen3_asr frontend entry |
| AudioNormalizer | `processor/vibevoice_tokenizer_processor.py` AudioNormalizer (target_dB_FS=-25, eps 1e-6): `rms=sqrt(mean(x^2)); x*=10^(-25/20)/(rms+eps)`; then declip `if max(|x|)>1: x/=max(|x|)+eps` | [samples] | `enc.input_waveform` [264000] | host-side scalar pass over PCM (F32); **pin exact 0.0 drift** (env-inject for isolation) | none (new) |

## Encoder (acoustic + semantic VAE; same structure, different vae_dim / pad)

Reference: `modular_vibevoice_tokenizer.py` `TokenizerEncoder.forward` + `encode()`. Both encoders
share structure; acoustic vae_dim=64 (std_dist gaussian, fix_std 0.5 — irrelevant, we use mean),
semantic vae_dim=128 (std_dist none). `disable_last_norm=True` -> final norm is Identity.
encoder_ratios=[8,5,5,4,2,2], depths=[3,3,3,3,3,3,8], n_filters=32, layer_scale 1e-6.

| Stage | Reference location | Output shape | Gate tensor | ggml / C++ pattern | In-tree analog |
|-------|--------------------|--------------|-------------|--------------------|----------------|
| Stem conv (downsample_layers.0) | SConv1d 1->32, k=7, s=1, causal | [32, S] | — | `ggml_conv_1d` w/ causal left-pad `(k-1)-(s-1)=6`, + stride-align right pad; bias add | none (new) |
| Block stack stage[i] (depth 3 or 8) | `Block1D.forward` | [C, S] | — | per-block: see Block1D pattern below | none (new) |
| Downsample conv (layers.1..6) | SConv1d C->2C, k=2·ratio, s=ratio, causal | [2C, S/ratio] | — | `ggml_conv_1d` stride=ratio, causal left-pad `(k-1)-(s-1)`, declip via `get_extra_padding` (ceil align) | none (new) |
| Final norm | `nn.Identity` (disable_last_norm) | [2048, T] | — | no-op | — |
| Head conv | SConv1d 2048->vae_dim, k=7, causal | [vae_dim, T] | — | `ggml_conv_1d` causal | none (new) |
| Permute -> mean | `encode()` returns `latents.permute(0,2,1)` | [T, vae_dim] | `enc.acoustic.mean` [83,64], `enc.semantic.mean` [83,128] | `ggml_cont(ggml_transpose())` | none (new) |

**Block1D pattern** (`Block1D.forward`, map once): operates on [C, S].
1. `r=x; x=ConvRMSNorm(x); x=DepthwiseConv(x); x*=gamma[C]; x=r+x`
- ConvRMSNorm: RMS over channel axis per time step (transpose [C,S]->[S,C], RMSNorm dim=C, weight only, eps; transpose back). **eps: confirm 1e-5 vs 1e-6 in code before finalizing.**
- DepthwiseConv (`mixer`, groups=C, k=7, causal, bias): `ggml_conv_1d_dw` (or grouped conv), causal left-pad 6.
- `gamma` is per-channel layer-scale [C], broadcast over S.
2. `r=x; x=ConvRMSNorm_ffn(x); x=permute[C,S]->[S,C]; x=Linear2(GELU(Linear1(x))); permute back; x*=ffn_gamma[C]; x=r+x`
- FFN linear1/linear2 with bias; activation GELU (`ACT2FN["gelu"]`).

T for jfk = ceil(264000/3200) = 83, matching `vae_tok_len` (count of speech_pad tokens).

## Decoder (Qwen2.5-7B LM)

Reference: `modeling_vibevoice_asr.py` forward + the Qwen2 `language_model`. 28 layers, hidden 3584,
heads 28 / kv 4, head_dim 128, intermediate 18944, rope_theta 1e6, rms_norm_eps 1e-6, vocab 152064.

| Stage | Reference location | Output shape | Gate tensor | ggml / C++ pattern | In-tree analog |
|-------|--------------------|--------------|-------------|--------------------|----------------|
| Connector (×2) | `SpeechConnector`: fc1(vae_dim->3584)->RMSNorm(3584)->fc2(3584->3584) | [T, 3584] | `enc.acoustic.feat`, `enc.semantic.feat` [83,3584] | matmul+bias, RMSNorm, matmul+bias. **confirm no activation between norm and fc2** | qwen3_asr proj |
| Fusion | `combined = a_feat + s_feat` | [T, 3584] | `enc.combined` [83,3584] | `ggml_add` | none |
| Token embed | `embed_tokens(input_ids)` | [L, 3584] | `dec.token_emb` [143,3584]; pin exact 0.0 | `ggml_get_rows`; pure read | qwen3_asr embed |
| Speech scatter | `inputs_embeds[acoustic_input_mask] = combined` (replace speech_pad embeds) | [L, 3584] | `dec.inputs_embeds` [143,3584] | `ggml_set_rows` / masked copy at speech_pad positions | qwen3_asr audio inject |
| Qwen2 layer ×28 | standard Qwen2 block (pattern below) | [L, 3584] | `dec.block.{0,13,27}.out` | attn+rope+ffn (see pattern) | qwen3_asr decoder (adapt) |
| Final RMSNorm | `language_model.norm` | [L, 3584] | `dec.final_norm` [143,3584] | `ggml_rms_norm`×weight | qwen3_asr |
| lm_head | `lm_head` (untied, `dec.output.weight`) | [L, vocab] | `dec.logits_lastpos` [152064] (last pos) | matmul | qwen3_asr |

**Qwen2 layer pattern** (differs from qwen3_asr — call out in Deviations):
`r=x; x=RMSNorm_attn(x); q=Wq·x+bq; k=Wk·x+bk; v=Wv·x+bv` (GQA 28/4, head_dim 128, **q/k/v have
bias, no q/k-norm**); `rope(q,k, rotate_half, theta=1e6)`; causal SDPA; `x=r+Wo·attn` (Wo no bias);
`r=x; x=RMSNorm_ffn(x); x=r+Wdown(silu(Wgate·x)*Wup·x)`.

## Generation / KV Path

| Stage | Reference location | Output shape | Gate tensor | ggml / C++ pattern | In-tree analog |
|-------|--------------------|--------------|-------------|--------------------|----------------|
| Prompt build | processor `__call__` lines ~340-380 | token ids | — | host: `<\|im_start\|>system\n{SYSTEM_PROMPT}<\|im_end\|>\n<\|im_start\|>user\n` + speech_start + speech_pad×T + speech_end + `\nThis is a {dur:.2f} seconds audio, please transcribe it with these keys: Start time, End time, Speaker ID, Content<\|im_end\|>\n<\|im_start\|>assistant\n` | qwen3_asr prompt builder |
| Prefill | model forward, `use_cache` | [L, vocab] | `dec.logits_lastpos` | full causal prefill, KV fill | qwen3_asr prefill |
| Incremental decode | `generate` greedy; speech inputs only when cache_position[0]==0 | 1 tok/step | **`dec.logits_raw.gen<N>` (N≥8) — MUST ADD to dumper + C++** | KV-cache step, argmax, append | qwen3_asr decode |
| Stop | `eos_token_id` (151643) or `stop_strings=["}]"]`, max_new_tokens=448 | — | — | host: eos + JSON-end string match | qwen3_asr stop |

## Capabilities And Language Controls

| Capability | Reference behavior | C++ API behavior | Family-doc Capability Validation row |
|------------|--------------------|------------------|--------------------------------------|
| Transcribe (auto lang) | greedy JSON | non-empty plausible transcript | Transcribe — MUST PASS |
| Language detect | implicit (model self-detects) | advertised via `stt.capability.lang_detect` | (covered by transcribe) |
| Segment timestamps | structured Start/End in JSON | parse JSON -> segment times | Segment timestamps — MUST PASS |
| Speaker diarization | Speaker ID in JSON | parse JSON -> speaker labels | Speaker diarization — MUST PASS |
| Translate / word-ts / streaming / hotwords | n/a | n/a | OUT OF SCOPE (per family doc) |
| Batch (offline) | n/a | `run_batch()` parity | Batch (offline) — MUST PASS |

## Deviations From Closest Analog (qwen3_asr)

- **Frontend is a learned causal-conv VAE, not a mel/STFT spectrogram.** Entirely new ggml
subsystem: causal SConv1d (left-pad `(k-1)d-(s-1)` + ceil stride-align right pad), depthwise
conv mixer, ConvRMSNorm (norm over channel axis), per-channel layer-scale gamma. No in-tree code.
- **Two parallel encoders + connectors fused by element-wise SUM**, then scattered into the LM at
`speech_pad` positions. qwen3_asr injects a single audio stream.
- **LM is plain Qwen2.5, not Qwen3:** q/k/v carry biases; NO per-head q/k RMSNorm; **plain
rotate_half RoPE (theta 1e6), NOT MRoPE/interleaved** (qwen3_asr uses mrope_section). Untied
lm_head (`dec.output.weight`).
- **Determinism:** use the acoustic VAE mean (`mode`), never `sample` (fix_std=0.5 gaussian).
- **Structured output:** WER scoring needs a normalizer that strips Speaker/Start/End and keeps
Content (Stage 7); segment-ts + diarization are in scope (parse the JSON).

## Variant Notes

- `vibevoice-asr`: family baseline (only variant). ~8.3 B params kept, BF16 reference. Long-form
(60-min / 64K KV) deferred; first port targets short utterances (jfk / LibriSpeech segments).

## Resolved tensor/shape contract (from GGUF inspection)

- **Downsample strides come from REVERSED encoder_ratios.** Weight shapes (ggml ne
`[k,in,out]`): `downsample.0`=[7,1,32] stem (s1); `downsample.1`=[4,32,64] s2; … ;
`downsample.6`=[16,1024,2048] s8. **kernel = 2·stride for every strided conv**, so the port
derives `stride = kernel/2` from the loaded weight (ordering-agnostic). Stage channels
C[i]=32·2^i for i=0..6 (32,64,128,256,512,1024,2048); stage i runs AFTER downsample i.
- **VAE FFN = 4× expansion + GELU.** `ffn.linear1` C->4C, `ffn.linear2` 4C->C (e.g. 32->128,
2048->8192). Both carry bias. `ffn_dim = 4*C`.
- Mixer depthwise weight `[7,1,C]` (groups=C, k=7, s1, causal); head `[7,2048,vae_dim]` (k7,s1).
- Connector: `fc1[vae_dim->3584]` -> `RMSNorm[3584]` -> `fc2[3584->3584]`; fc1/fc2 carry bias.

## Open items (resolve during Step 2, before sign-off)

- ConvRMSNorm eps (1e-5 config vs 1e-6 Block1D kwarg default) — read the constructed value.
- SpeechConnector: confirm exact op order and whether any activation sits between RMSNorm and fc2.
- `get_extra_padding_for_conv1d` exact arithmetic (right-pad for ceil stride alignment) — must
match so encoder T == ceil(n_samples/3200).
- ggml causal conv: `conf::conv_1d_f32` takes symmetric padding only; left-pad manually
(`ggml_pad`/view) by `(k-1)d-(s-1)` + ceil right-pad, then conv with padding=0.
- Compute location: the 9B LM will not run on this 16 GB machine; `validate.py all` block/logits
gates and the Step 11 WER gate run on **Modal** (encoder-only gates may run locally). ← the
local→Modal boundary; notify the user here.
Loading
Loading