Add Parakeet CTC automatic speech recognition support - #471
Conversation
Implement the NVIDIA FastConformer encoder, feature-input CTC task, processor-aware golden infrastructure, and Hugging Face weight alignment for parakeet_ctc. Add synthetic and real-checkpoint parity coverage, deterministic audio goldens, pinned revision loading, CUDA fp16 validation, and explicit unsupported-runtime guards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
Format the new FastConformer component and CUDA integration test with the Ruff 0.16.1 version pinned by CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
There was a problem hiding this comment.
Pull request overview
Adds first-class Mobius support for the Hugging Face parakeet_ctc architecture (nvidia/parakeet-ctc-1.1b) by introducing a feature-input CTC ASR task (log-mel features + frame mask → frame logits), the Parakeet FastConformer encoder implementation, registration/config plumbing, and deterministic golden/integration coverage.
Changes:
- Implement Parakeet CTC model/config/components and register
model_type: parakeet_ctcwith a newfeature-ctc-asrtask. - Extend golden generation and e2e golden execution to support feature-input CTC and revision-pinned builds (plus deterministic frame-count enforcement for CTC L5).
- Add ORT GenAI export guardrails for this unsupported feature-input CTC contract, plus new unit/integration tests and goldens.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/parakeet_ctc_integration_test.py | New real-audio, real-weight CUDA parity + fp16 frame-ID validation for Parakeet CTC. |
| tests/e2e_golden_test.py | Thread revision into build; add feature-ctc-asr feed prep + CTC generation path and deterministic frame-count enforcement. |
| tests/build_graph_test.py | Include feature-ctc-asr in expected package-role mapping. |
| tests/_test_configs.py | Add a tiny parakeet_ctc test config entry for graph/unit coverage. |
| testdata/golden/audio/parakeet-ctc-1.1b.json | New L4 golden summary for Parakeet CTC. |
| testdata/golden/audio/parakeet-ctc-1.1b_generation.json | New L5 deterministic frame argmax IDs + transcript golden for Parakeet CTC. |
| testdata/cases/schema.json | Register feature-ctc-asr as a valid task_type. |
| testdata/cases/audio/parakeet-ctc-1.1b.yaml | New L4+L5 Parakeet CTC real-audio case pinned to a specific revision. |
| src/mobius/tasks/_ctc_asr.py | Add FeatureCTCAsrTask (feature-input CTC) alongside existing waveform CTC task. |
| src/mobius/tasks/init.py | Export/register FeatureCTCAsrTask in the task registry. |
| src/mobius/models/parakeet_ctc.py | New ParakeetForCTCModel implementing encoder + Conv1d CTC head and bf16 rejection. |
| src/mobius/models/parakeet_ctc_test.py | New unit tests for config extraction, bf16 rejection, weight-name alignment, parity, ORT GenAI export rejection. |
| src/mobius/models/init.py | Export ParakeetForCTCModel. |
| src/mobius/integrations/ort_genai/auto_export.py | Explicitly reject ORT GenAI export for parakeet_ctc models. |
| src/mobius/components/_whisper.py | Generalize Conv1d to support groups and optional bias. |
| src/mobius/components/_parakeet_audio.py | New offline FastConformer components used by Parakeet CTC encoder. |
| src/mobius/components/_conv.py | Add BatchNorm1d implemented via elementwise inference math (fp32 accumulation). |
| src/mobius/components/init.py | Export BatchNorm1d and ParakeetFastConformerEncoder. |
| src/mobius/_weight_loading.py | Add revision-aware parallel shard download and revision-aware _download_weights. |
| src/mobius/_registry.py | Register parakeet_ctc model/task/config and add a default model-id mapping. |
| src/mobius/_configs/_base.py | Add ParakeetCTCConfig with nested encoder extraction + safe dtype defaulting. |
| src/mobius/_configs/init.py | Export ParakeetCTCConfig. |
| src/mobius/_config_resolver.py | Make direct config.json fetch revision-aware. |
| src/mobius/_builder.py | Add revision parameter and thread it through config + weight loading. |
| scripts/generate_golden.py | Extend CTC golden generation to support revision + AutoModelForCTC (MMS-special-casing preserved). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
🏗️ Architecture Diff
whisper (speech-to-text) / encoder — 2 change(s)Op summary: 50 → 50 nodes No op-sequence changes. Modified attributes:
Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
CI baseline / unrelated-failure analysis (2026-08-10)The remaining failures in CI run 31418606770 are outside the Parakeet change surface:
The exact base SHA is Parakeet-specific L1 is green in job 93553773623. Local CUDA L4/L5 and real-checkpoint fp32/fp16 validations are documented in the PR body; GitHub L4/L5 are still running in this CI run and will be checked to completion. |
Remove the broad cache fallback from L4 and L5 so a changed golden model set cannot restore unrelated checkpoints and exhaust the hosted GPU runner disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Patch the runtime Hugging Face hub and Xet cache constants for each golden test, then remove that cache at teardown so all-model GPU runs do not accumulate checkpoints until the runner disk is full. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Expose the reusable affected-model input to manual L4 and L5 dispatches so model-specific hosted validation can run independently of unrelated all-model baseline failures. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
Hosted Parakeet golden follow-upThe Parakeet-specific validation is now complete on the hosted GPU runner at head
The earlier aggregate L4/L5 failures were infrastructure-only. The first two attempts exhausted the self-hosted runner disk because Commit |
justinchuby
left a comment
justinchuby
left a comment
There was a problem hiding this comment.
Correctness review: two findings.
src/mobius/_builder.py:539: the new revision argument is dropped on the diffusers fallback. If AutoConfig fails and this path is selected, the pipeline index/configs/weights come from the repository default revision rather than the requested commit. Please thread revision through build_diffusers_pipeline and its Hub downloads.
Use the configured Parakeet activation in feed-forward and convolution modules. Propagate build revisions through diffusers artifacts and pin all golden-test Hugging Face input artifacts to each case revision. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: justinchuby <justinchuby@users.noreply.github.com>
|
Also fixed the review-summary finding in 612f3eb: |
Update the Parakeet branch to current main while retaining all newly added model registrations. Emit frozen BatchNorm1d as standard BatchNormalization so ORT can fold it into neighboring convolutions, and assert the optimized Parakeet graph keeps fused Attention, Swish, BatchNormalization, and SkipLayerNormalization operators. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
|
Updated PR #471 to current Optimization/fusion evidence:
Post-merge validation:
|
Bring the Parakeet branch to main at ed0319d while retaining both speech-recognition architectures. Preserve Conv1d positional bias compatibility and add grouped convolution support required by Parakeet depthwise blocks, with regression coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchu@microsoft.com>
|
Final main refresh completed in The only new integration conflict was the shared Whisper/Moonshine Final post-merge evidence:
Earlier real-checkpoint CUDA FP32/FP16 and hosted Parakeet-only L4/L5 evidence remains applicable because the latest merge did not alter Parakeet math; its only shared-component resolution preserves the already-validated grouped Conv1d graph and is covered by the graph tests above. |
## Summary - distill post-single-shot lessons from the Parakeet CTC, Mage-VL, Nemotron Parse, LFM2.5, Moonshine, MiniCPM-V, Qwen Image Edit, and Muse Glimmer integrations - tighten source/config/revision fidelity, real processor and runtime contracts, semantic multi-dtype validation, exact golden testing, and weighted graph optimization evidence - add precise CI baseline triage, targeted GPU validation, linear rebase/worktree guidance, quantization schema checks, and executable diffusion pipeline requirements - shorten the duplicated multimodal float32 input section and resolve contradictory encoder/component import guidance ## Sources Direct agent retrospectives plus commit/review history for merged PRs #462-#465, #471-#473, and #475. ## Validation - documentation diff is whitespace-clean - initialized repository-pinned Ruff/lintrunner tooling - independent rubber-duck review found no blocking issues; follow-up consistency suggestions were applied No runtime code is changed. --------- Signed-off-by: Justin Chu <justinchu@microsoft.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Summary
Add first-class Mobius support for
nvidia/parakeet-ctc-1.1b, Hugging Facemodel_type: parakeet_ctc.This is an independent offline CTC ASR architecture and does not duplicate #411, which targets streaming Parakeet RNNT. The new path consumes processor-normalized log-mel features and produces frame-level CTC logits.
Architecture
bias_u/bias_vinput_features [B, T, 80]plus booleanattention_masknum_batches_trackedThe reusable component surface also gains grouped/bias-free
Conv1dand frozenBatchNorm1dinference with fp32 accumulation.Model and golden integration
parakeet_ctcconfig/model/task extractionFeatureCTCAsrTaskfeature-ctc-asrYAML/schema and golden generation support20e63a0fed6aedba145b74b826dbd41df0941730Committed real-audio goldens use nonzero LibriSpeech audio and contain all 114 frame argmax IDs plus the full decoded transcript:
Validation
GPU: NVIDIA RTX A1000 Laptop GPU, 8 GB; CUDA and
CUDAExecutionProvideravailable.python -m pytest tests/build_graph_test.py tests/cli_test.py src/ -q -k "not phi4mm and not apply_weights_unknown" --tb=short -n autopython -m pytest tests/arch_validation_test.py -k parakeet -q --tb=shortpython -m pytest tests/yaml_schema_test.py -q --tb=shortlintrunner -aCLI/runtime/quantization evidence
Evidence-based limitations and waivers
input_featuresin its decoder model schema. This offline feature-input CTC model cannot be represented by the current ORT GenAI decoder contract, so Mobius now fails explicitly rather than producing a misleading package. Foundry Local depends on ORT GenAI and is consequently not applicable until that runtime supports feature-input CTC pipelines.onnx_irfailed during safetensors CLI serialization; standard ONNX external-data export succeeded and was used for CUDA runtime and quantization validation.