docs: add self-hosted audio models guide with Voxtral Small/vLLM by mfournioux · Pull Request #898 · agentgateway/website

mfournioux · 2026-08-14T15:08:16Z

Summary

Add documentation for self-hosted audio models (like Voxtral Small) through agentgateway using Passthrough routing. Audio models expose endpoints like /v1/audio/transcriptions that are handled via Passthrough routing — agentgateway forwards the request and response without parsing or modifying the payload.

What's included

Complete self-hosted guide for deploying audio models via vLLM in Kubernetes
Model health verification step
Agentgateway Backend configuration with Passthrough routes
HTTPRoute setup for routing audio requests
curl examples for transcription requests
Supported endpoints table

Deployment example

Deploys Voxtral Small 24B using vllm/vllm-openai:latest with Mistral-specific configuration.

New documentation for configuring audio models (Whisper, Voxtral, etc.) through agentgateway using Passthrough routing. Covers: - Self-hosted audio model deployment with Kubernetes Deployment and Service - Backend configuration with ai.routes for /v1/audio/transcriptions, /v1/audio/speech, /v1/audio/translations - HTTPRoute setup for routing to the audio backend - Examples for transcription (multipart form data) and speech (JSON) requests - Table of supported audio endpoints - Info callout about Passthrough policy limitations - Cleanup instructions Includes source doc (assets) and Hugo wrappers for latest and main versions. Addresses documentation gap for audio model providers that expose OpenAI-compatible APIs but require Passthrough processing for non-LLM endpoints. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

…io docs Update all curl examples throughout the audio models provider documentation to use the explicit --request POST, --url, --header, and --form flags instead of the shorthand -X, -H, -F flags. Also update: - Authorization header to use $AUDIO_API_KEY environment variable - Model name to voxtral-small-2507 - File path to ./testdata/sample-audio.webm - Add explicit Content-Type: multipart/form-data header This aligns with the user's preferred curl format and provides a consistent format across all curl examples in the documentation. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

…ints Fix critical inaccuracies in audio models provider documentation: Deployment: - Replace placeholder image 'voxtral/audio-model:latest' with 'ghcr.io/virtuos/vllm-voxtral:latest' (official community image) - Add GPU resource requirements (nvidia.com/gpu: 1) for vLLM - Add VLLM_DISABLE_COMPILE_CACHE environment variable - Update container name, labels, and app selector to voxtral-vllm - Add warning callout about GPU requirement (nvidia-container-toolkit, 16GB VRAM minimum) - Recommend Whisper via Speaches for CPU-only deployments API endpoints: - Remove /v1/audio/speech (text-to-speech) — Voxtral is speech-to-text only, does not support TTS - Remove /v1/chat/completions route (not applicable to audio models) - Keep only audio endpoints: transcriptions, translations, models - Add Voxtral warning callout explaining speech-to-text limitation Naming: - Update all resource names: audio-model -> voxtral-vllm, audio-model-service -> voxtral-service, audio-model-route -> voxtral-audio-route, audio-model-secret -> voxtral-audio-secret, backend name: audio-model -> voxtral-audio - Update HTTPRoute backendRefs to reference voxtral-audio Added: - Supported audio formats callout (WAV, MP3, MKV, WEBM) - Link to Mistral Voxtral HF model page Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

Update audio models documentation to use the exact vLLM deployment command and model name from official Mistral/HuggingFace docs: Deployment: - Use official 'vllm/vllm-openai:latest' image instead of community image - Use 'vllm serve mistralai/Voxtral-Small-24B-2507' with exact flags: --tokenizer_mode mistral --config_format mistral --load_format mistral --tensor-parallel-size 2 --tool-call-parser mistral --enable-auto-tool-choice - Update GPU requirements to 2 GPUs (~55 GB VRAM) - Model name updated to 'voxtral-small-24b-2507' in backend and curl examples - Add link to Voxtral Mini for smaller GPU memory deployments This matches the exact vLLM serve command documented by Mistral for deploying Voxtral Small 24B on HuggingFace. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

- Remove hard requirement of 2 GPUs; default to 1 GPU - Clarify that 55 GB VRAM / tensor-parallel-size 2 is for bf16/fp16 full model - Recommend Voxtral Mini or quantized variants for single-GPU deployments - Keep --tensor-parallel-size 2 in args as per official Mistral docs - Update deployment explanation to reflect this flexibility The --tensor-parallel-size 2 flag comes from the official Mistral HF README but is optional depending on available GPU memory. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

- Remove --tensor-parallel-size 2 from default args (it's a performance recommendation, not an obligation) - Fix indentation of --tool-call-parser argument - Clarify GPU requirements: 24 GB for single-GPU quantized deployment, ~55 GB for bf16 full precision Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

- Simplify warning callout to only mention nvidia-container-toolkit - Remove ~55 GB VRAM note from deployment description - Keep only the vLLM deployment command and factual YAML details Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

/v1/audio/translations is an OpenAI Whisper-specific endpoint, not supported by Voxtral. Remove all references to this endpoint from the audio model guide. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

- Remove API key step (Step 1) - no auth for self-hosted models - Add Step 2: verify model is responding via /v1/models endpoint - Remove Authorization headers from curl examples - Remove Kubernetes secret creation and cleanup Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

vLLM/mistral-common uses libsndfile for audio loading, which supports WAV, FLAC, OGG, AU natively — not MP3, MKV, WEBM or M4A as previously stated. Fix the callout and curl examples to use .wav format. Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

artberger

Thank you for contributing this guide for audio models!

artberger · 2026-08-18T20:33:12Z

+**Supported audio formats:** WAV, FLAC, OGG and AU (via [libsndfile](https://github.com/libsndfile/libsndfile)). These formats are natively supported by vLLM's audio processing pipeline.
+{{< /callout >}}
+
+{{< /steps %}}


Suggested change

{{< /steps %}}

{{% /steps %}}

artberger · 2026-08-18T20:36:02Z

+      backendRefs:
+        - name: voxtral-audio
+          namespace: {{< reuse "agw-docs/snippets/namespace.md" >}}
+          group: agentgateway.dev


Suggested change

group: agentgateway.dev

group: {{< reuse "agw-docs/snippets/group.md" >}}

artberger · 2026-08-18T20:36:33Z

+{{< callout type="info" >}}
+**Supported audio formats:** WAV, FLAC, OGG and AU (via [libsndfile](https://github.com/libsndfile/libsndfile)). These formats are natively supported by vLLM's audio processing pipeline.
+{{< /callout >}}


newer note style

Suggested change

{{< callout type="info" >}}

**Supported audio formats:** WAV, FLAC, OGG and AU (via [libsndfile](https://github.com/libsndfile/libsndfile)). These formats are natively supported by vLLM's audio processing pipeline.

{{< /callout >}}

> [!INFO]

> **Supported audio formats:** WAV, FLAC, OGG and AU (via [libsndfile](https://github.com/libsndfile/libsndfile)). These formats are natively supported by vLLM's audio processing pipeline.

artberger · 2026-08-18T20:36:59Z

+{{< callout type="info" >}}
+When a route is set to `Passthrough`, agentgateway does not apply any LLM-specific policies (such as cost tracking, rate limiting, or prompt guards) to those requests. The requests are forwarded exactly as received.
+{{< /callout >}}


Suggested change

{{< callout type="info" >}}

When a route is set to `Passthrough`, agentgateway does not apply any LLM-specific policies (such as cost tracking, rate limiting, or prompt guards) to those requests. The requests are forwarded exactly as received.

{{< /callout >}}

> [!NOTE]

> When a route is set to `Passthrough`, agentgateway does not apply any LLM-specific policies (such as cost tracking, rate limiting, or prompt guards) to those requests. The requests are forwarded exactly as received.

artberger · 2026-08-18T20:37:24Z

+{{< callout type="warning" >}}
+**Voxtral Small requires a Docker environment with NVIDIA GPU support (`nvidia-container-toolkit`).**
+{{< /callout >}}


newer note style

Suggested change

{{< callout type="warning" >}}

**Voxtral Small requires a Docker environment with NVIDIA GPU support (`nvidia-container-toolkit`).**

{{< /callout >}}

> [!WARNING]

> **Voxtral Small requires a Docker environment with NVIDIA GPU support (`nvidia-container-toolkit`).**

artberger · 2026-08-18T20:49:09Z

+
+### Step 3: Create the LLM backend with Passthrough routes
+
+Create an {{< reuse "agw-docs/snippets/backend.md" >}} resource with `ai.routes` to forward audio endpoints via `Passthrough` processing.


Suggested change

Create an {{< reuse "agw-docs/snippets/backend.md" >}} resource with `ai.routes` to forward audio endpoints via `Passthrough` processing.

Create an {{< reuse "agw-docs/snippets/backend.md" >}} resource with `policies.ai.routes` to forward audio endpoints via `Passthrough` processing.

artberger · 2026-08-18T20:49:59Z

@@ -0,0 +1,7 @@
+---
+title: Audio models (Whisper, Voxtral)


Is Whisper available through Voxtral or something? I didn't notice it mentioned in the guide.

Suggested change

title: Audio models (Whisper, Voxtral)

title: Audio models

artberger · 2026-08-18T20:51:03Z

+---
+title: Audio models (Whisper, Voxtral)
+weight: 60
+description: Configure audio models with Passthrough routing for transcription and speech endpoints.


Was there a speech endpoint? maybe just transcription

Suggested change

description: Configure audio models with Passthrough routing for transcription and speech endpoints.

description: Configure self-hosted audio models such as Voxtral with passthrough routing for transcription endpoints.

artberger · 2026-08-18T20:51:21Z

@@ -0,0 +1,7 @@
+---
+title: Audio models (Whisper, Voxtral)


Suggested change

title: Audio models (Whisper, Voxtral)

title: Audio models

artberger · 2026-08-18T20:51:30Z

+---
+title: Audio models (Whisper, Voxtral)
+weight: 60
+description: Configure audio models with Passthrough routing for transcription and speech endpoints.


Suggested change

description: Configure audio models with Passthrough routing for transcription and speech endpoints.

description: Configure self-hosted audio models such as Voxtral with passthrough routing for transcription endpoints.

mfournioux added 11 commits August 14, 2026 16:12

fix: remove Voxtral TTS warning callout from audio docs

67633ef

Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

artberger reviewed Aug 18, 2026

View reviewed changes

Navigation Menu

Uh oh!

docs: add self-hosted audio models guide with Voxtral Small/vLLM - #898

docs: add self-hosted audio models guide with Voxtral Small/vLLM#898
mfournioux wants to merge 11 commits into
agentgateway:mainfrom
mfournioux:docs/audio-models-passthrough

mfournioux commented Aug 14, 2026

Uh oh!

artberger left a comment

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

artberger Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	group: agentgateway.dev
	group: {{< reuse "agw-docs/snippets/group.md" >}}


		### Step 3: Create the LLM backend with Passthrough routes

		Create an {{< reuse "agw-docs/snippets/backend.md" >}} resource with `ai.routes` to forward audio endpoints via `Passthrough` processing.

	description: Configure audio models with Passthrough routing for transcription and speech endpoints.
	description: Configure self-hosted audio models such as Voxtral with passthrough routing for transcription endpoints.

Uh oh!

Conversation

mfournioux commented Aug 14, 2026

Summary

What's included

Deployment example

Uh oh!

artberger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants