diff --git a/_posts/2026-08-20-speculative-decoding-amd-gpus.md b/_posts/2026-08-20-speculative-decoding-amd-gpus.md
new file mode 100644
index 00000000..b22456c5
--- /dev/null
+++ b/_posts/2026-08-20-speculative-decoding-amd-gpus.md
@@ -0,0 +1,11054 @@
+---
+layout: post
+title: "Exploring Speculative Decoding in vLLM on AMD GPUs"
+author: "AMD and Embedded LLM"
+summary: "A practical guide to speculative decoding in vLLM on AMD GPUs, covering draft-and-verify mechanics, MTP, EAGLE-3, DFlash, DSpark, configuration, tuning, and benchmark results."
+image: /assets/figures/2026-08-20-speculative-decoding-amd-gpus/figure-01.svg
+tags:
+ - speculative-decoding
+ - amd
+---
+
+
+
+**TL;DR:** Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior.
+
+---
+
+## Introduction
+
+Large language models support a wide range of applications, but serving them at scale requires careful optimization. Standard autoregressive decoding is the baseline used by most LLM serving systems: the model generates one token, appends it to the sequence, and then uses the updated sequence to generate the next token. This process is simple and reliable, but the serving loop still advances one committed token at a time because output tokens must be produced in strict left-to-right order.
+
+Speculative decoding [[1]](#ref-1) builds on this baseline through a draft-and-verify mechanism. A lightweight draft component proposes candidate future tokens, and the target model verifies those candidates before they are committed. When several draft tokens are accepted, the system can commit multiple output tokens from a single target-model verification step while preserving the target model's output behavior.
+
+This post explores how speculative decoding works in vLLM and shares measurements from our test environment. We first review the autoregressive decoding baseline and the draft-and-verify process. We then examine five speculative-drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. These methods differ in how the draft component receives information from the target model and whether candidate tokens are generated sequentially, autoregressively, in parallel, or through a hybrid approach. Finally, we show how to enable the methods tested in our environment, report measurements from our experiments on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ open software platform, and discuss practical tuning and observability considerations.
+
+---
+
+## The autoregressive decoding baseline
+
+In standard autoregressive decoding, each decode step produces and commits one new token. For example, generating four output tokens requires four sequential decode steps:
+
+
+
+ Step 1:
+ context
+ →
+ model
+ →
+ T1
+
+
+ Step 2:
+ context + T1
+ →
+ model
+ →
+ T2
+
+
+ Step 3:
+ context + T1 T2
+ →
+ model
+ →
+ T3
+
+
+ Step 4:
+ context + T1 T2 T3
+ →
+ model
+ →
+ T4
+
+
+
+After each step, the generated token is appended to the sequence and becomes part of the input for the next step. This makes the decoding loop straightforward, but it also requires one model decode step for every output token. During long generations, this token-by-token loop can dominate latency and limit serving throughput.
+
+The key question behind speculative decoding is therefore:
+
+
Can we preserve the output behavior of the original model while reducing how often generation advances by only one token at a time?
+
+Speculative decoding addresses this by separating proposal from verification. A draft component first proposes several candidate future tokens. The original model, acting as the target model, then verifies those candidates before they are committed.
+
+---
+
+## Core idea of speculative decoding
+
+Speculative decoding does not replace the original model. Instead, it keeps the original model as the target model, which remains responsible for the final output, and adds a faster proposal stage in front of it.
+
+The process has two parts:
+
+- Draft: propose several candidate future tokens.
+- Verify: use the target model to check those candidates.
+
+During each speculative decoding round, as illustrated in Figure 1, a lightweight draft component proposes one or more future tokens. These tokens are only candidates and are not committed immediately. The target model then evaluates the candidate token sequence in one verification pass.
+
+Verification proceeds from left to right. Each draft token is checked using the target model's result at the corresponding position. Accepted tokens are committed to the output sequence. When a draft token is rejected, later candidates from the same proposal are no longer accepted.
+
+If a draft token is rejected, the target model provides the next token. The remaining draft tokens are discarded, and generation continues from the updated sequence.
+
+Conceptually, standard autoregressive decoding advances like this:
+
+
+
+ target model
+ →T1
+
+
+
+
+
+ target model
+ →T2
+
+
+
+
+
+ target model
+ →T3
+
+
+
+
+
+ target model
+ →T4
+
+
+
+
+
+
+Speculative decoding instead allows several candidate positions to be evaluated together:
+
+
+
+ draft proposes
+ T1
+ T2
+ T3
+ T4
+
+
+ model verifies
+ ✓
+ ✓
+ ✗
+ stop
+
+
+ commit
+ T1
+ T2
+ replacement token
+ -
+
+
+
+This can reduce the number of target-model decoding rounds when multiple candidates are accepted. When the draft component produces tokens that the target model accepts, several output tokens can be committed from one target-model verification step. When a proposal is rejected, the target-side result determines how generation continues.
+
+
+
+
+
+
+
+
Figure 1. Speculative decoding flow: a draft component proposes candidate future tokens, and the target model verifies them before output tokens are committed.
+
+### A simple accept/reject example
+
+Figure 2 gives an example of one speculative decoding round. Green boxes are draft tokens that survive verification, the red box marks the first rejected draft token, and the gray box is a later draft token that is discarded. The blue token in the output comes from the target model, not from the draft proposal.
+
+
+
+
+
+
+
+
Figure 2. Left-to-right verification of a draft proposal. The first two draft tokens are accepted, the rejected position uses a target-model token, and the remaining candidate is discarded.
+
+Suppose the current prompt is:
+
+
+ The weather today is
+
+
+The draft component proposes several future tokens:
+
+
+ sunny
+ and
+ warm
+ outside
+
+
+The target model verifies the draft tokens from left to right:
+
+
+
+ draft proposes
+ sunny
+ and
+ warm
+ outside
+
+
+ model verifies
+ ✓
+ ✓
+ ✗
+ stop
+
+
+ commit
+ sunny
+ and
+ clear
+ -
+
+
+
+The first two draft tokens, sunny and and, are accepted. At the third position, the draft proposes warm, but the target model selects clear. The remaining candidate, outside, is discarded because it follows the first rejected position.
+
+The next decoding round therefore continues from:
+
+
+ The weather today is
+ sunny
+ and
+ clear
+
+
+---
+
+## How the drafting methods work
+
+Although all speculative decoding methods follow the same overall draft-and-verify process, they differ in how the draft component is designed and how it works with the target model.
+
+The main differences are:
+
+- The type of information received from the target model.
+- How this information is incorporated into the drafting process.
+- Whether candidate tokens are generated sequentially or in parallel.
+
+Based on these differences, the drafting methods discussed in this post can be grouped into three broad categories: native MTP modules, separate MTP drafters, and dedicated target-conditioned draft networks.
+
+- **Native MTP modules:** built directly into the target-model architecture; use a model-native auxiliary prediction path; generate candidate tokens sequentially.
+- **Separate MTP drafters:** use a separate checkpoint paired with a specific target model; use target-model activations and shared KV-cache information during inference; generate candidate tokens sequentially.
+- **Dedicated target-conditioned draft networks:** use separate speculator models trained for a specific target model, including EAGLE-3, DFlash, and DSpark. EAGLE-3 drafts autoregressively from target-model hidden states, DFlash drafts parallel blocks from target-model hidden states, and DSpark adds lightweight causal correction and confidence-based prefix selection.
+
+These categories describe the draft component architecture, not the target-model family. A target model may support native MTP while also having separately trained EAGLE-3, DFlash, or DSpark draft models.
+
+The draft component does not operate entirely on its own. Depending on the method, the draft component may receive:
+
+- A hidden representation from the target model.
+- Hidden states from several selected target layers.
+- The target model's KV cache.
+- Features produced by combining multiple target-model representations.
+
+The following sections explain how each method uses this information and how it generates candidate tokens.
+
+### Native MTP
+
+Multi-Token Prediction, or MTP, refers to a family of model-native mechanisms for predicting tokens beyond the immediate next token. In vLLM, native MTP is available when the target model includes a compatible auxiliary prediction component [[2]](#ref-2). The exact MTP architecture varies across model families, but each implementation provides an auxiliary path for proposing future tokens.
+
+At the first speculative step, the MTP component combines a hidden representation from the target model with information from the current token to predict the first draft token. At subsequent steps, the newly drafted token and the hidden state produced by the previous MTP step are used to predict the next candidate. After the configured number of candidates has been proposed, the target model evaluates them together in one verification pass.
+
+
+
+
+
First draft token
+
Target-model hidden representation
+
↓
+
MTP component
+
↓
+
Draft token 1 + updated hidden state
+
+
↓
+
+
Subsequent draft tokens
+
Previous MTP hidden state + latest draft token
+
↓
+
MTP component
+
↓
+
Next draft token + updated hidden state
+
+
↓
+
Configured draft sequence complete
+
↓
+
Target model evaluates all proposed tokens together in one verification pass
+
+
+
+Many native MTP implementations follow a similar pattern. A hidden representation from the target model or from the previous MTP prediction is combined with the embedding of a shifted input token or the latest drafted token:
+
+
+
+
+
Target-model or previous MTP hidden representation
+
+
+
Shifted input-token or latest draft-token embedding
+
+
↓
+
Model-specific fusion or projection
+
↓
+
Auxiliary prediction layer
+
↓
+
Draft-token logits
+
+
+
+The two inputs serve different purposes: (1) the hidden representation carries information about the preceding sequence; and (2) the token embedding identifies the latest token from which drafting continues. In common implementations, they are combined along the hidden dimension and transformed before entering the auxiliary prediction layer.
+
+The number of physical MTP layers and the configured speculative length are separate concepts. When `num_speculative_tokens` exceeds the prediction depth directly provided by the checkpoint, vLLM can reuse the MTP path through additional forward passes. A larger value therefore proposes more candidates before verification, but also introduces more sequential drafting work.
+
+
+
+Native MTP is closely tied to the target-model architecture. In many implementations, parts of the MTP path share components with the target model, which can keep the additional memory overhead relatively modest. However, generating multiple speculative tokens still requires sequential drafting before verification.
+
+### Gemma 4 MTP
+
+Gemma 4 uses a separately packaged MTP draft component paired with a specific target model [[3]](#ref-3). Although the draft component has its own checkpoint, it remains closely connected to the target model during inference.
+
+
+
+
Gemma 4 target model
+
↓
+
+
Target-model activations
+
Shared target KV cache
+
+
↓
+
Gemma 4 MTP draft component
+
↓
+
Candidate tokens
+
+
+
+The draft component uses activations produced by the target model and shares the target model's KV cache. This allows it to reuse contextual information that the target has already computed instead of processing the accepted prefix independently.
+
+As with native MTP, the number of layers in the draft component is separate from the configured speculative length. When several candidate tokens are requested, the draft component generates them sequentially:
+
+
+
+### EAGLE-3
+
+EAGLE-3 uses a dedicated draft network trained for a specific target model. The draft component has its own execution path, but it remains closely conditioned on information produced by the target model [[4]](#ref-4).
+
+During the target-model forward pass, EAGLE-3 records hidden states from three stages of the target Transformer: near the beginning, around the middle, and near the end. These are contextual representations of the same accepted sequence at different stages of target-model processing.
+
+
+
+
+
Early-layer hidden state
+
Middle-layer hidden state
+
Late-layer hidden state
+
+
↓
+
Concatenate + projection
+
↓
+
Fused target feature
+
+
+
+The three hidden states are concatenated and projected into a single fused target feature. This fused representation is then combined with the embedding of the sampled token before entering the EAGLE-3 draft decoder.
+
+
+
+
+
Fused target feature
+
+
+
Sampled-token embedding
+
+
↓
+
Concatenate + projection
+
↓
+
EAGLE-3 draft decoder
+
↓
+
Draft token
+
+
+
+The two inputs serve different purposes:
+
+- The fused target feature summarizes the accepted sequence using information from several stages of the target-model forward pass.
+- The sampled-token embedding identifies the token from which drafting continues.
+
+EAGLE-3 generates draft tokens autoregressively. For the first draft token, it uses the fused target feature computed from the accepted sequence together with the sampled-token embedding. After a draft token is produced, its embedding is fed into the next drafting stage.
+
+Because the target model has not yet processed the later speculative positions, target-model hidden states for those positions are not available. EAGLE-3 therefore uses the previous draft-component output when continuing the draft sequence.
+
+
+
+
+
First draft token
+
+
Fused target feature
+
+
+
Sampled-token embedding
+
+
↓
+
Draft token 1
+
+
↓
+
+
Subsequent draft tokens
+
+
Previous draft-component output
+
+
+
Newly sampled-token embedding
+
+
↓
+
+ Draft token 2
+ Draft token 3
+ ...
+
+
+
+
+
+This sequential feedback gives later draft tokens direct dependence on earlier drafted tokens along the proposed sequence. However, generating more speculative tokens also requires more sequential drafting work before verification.
+
+### DFlash
+
+DFlash uses a dedicated draft network trained for a specific target model. Unlike MTP and EAGLE-3, which generate candidate tokens sequentially, DFlash predicts a whole block of future positions in parallel [[5]](#ref-5).
+
+DFlash begins each draft block with an anchor token. The anchor is a known token produced or confirmed by the target model, so DFlash does not need to predict it. Instead, it provides a known starting point for the masked positions that follow. In later decoding rounds, this is typically the additional target token returned by the previous verification pass.
+
+The anchor occupies the first position of the block, while the remaining positions are masked and predicted in parallel:
+
+A draft block starts with a confirmed anchor token, followed by masked positions:
+
+
+
+Here, `anchor` is the known target-model token, while the masked positions are predicted by DFlash.
+
+A single DFlash forward pass predicts all masked positions together:
+
+
+
+Like EAGLE-3, DFlash first combines hidden states from several target-model layers into a fused representation.
+
+
+
+
Target hidden states from selected layers
+
↓
+
Concatenate + projection
+
↓
+
Fused target context
+
+
+
+The main difference is how this fused representation is used. EAGLE-3 combines it with the sampled-token embedding at the input of its autoregressive draft network. DFlash instead converts the fused target context into additional Key and Value representations that are available in every layer of the draft network.
+
+Queries from the masked draft positions can therefore attend to both:
+
+- Key and Value representations derived from the target model.
+- Key and Value representations produced from the draft block itself.
+
+
Masked draft-position queries attend to both target-derived and draft-block K/V
+
+
+
+The target-model context therefore remains available throughout the draft network, rather than being supplied only once at its input.
+
+After the draft block has been generated, the target model evaluates all proposed tokens in one verification pass. The acceptance decision is then applied from left to right: accepted tokens are committed until the first rejection, and the remaining candidates are discarded.
+
+
+
+Here, the target-model token replaces the first rejected draft token, while the remaining draft tokens are discarded.
+
+A defining characteristic of DFlash is that all masked positions are predicted together in one draft-network forward pass.
+
+
+
+Because all masked positions are predicted together, a later position is not conditioned on the sampled output of an earlier position during the same pass. This removes the token-by-token feedback used by autoregressive drafting. The effectiveness of later positions therefore depends on the trained checkpoint and workload, particularly when longer draft blocks are used.
+
+### DSpark
+
+DSpark extends parallel drafting with two additional mechanisms:
+
+- A lightweight sequential head that introduces dependence between tokens within the draft block.
+- Confidence-based selection of the prefix submitted for target-model verification.
+
+DSpark uses a modified DFlash model as its parallel backbone [[6]](#ref-6). The backbone performs the main draft computation for all positions in one forward pass, producing a hidden state and a set of base logits for each draft position. It therefore inherits the target-context conditioning described in the DFlash section.
+
+
+
+
Target-derived context
+
↓
+
DSpark parallel backbone
+
↓
+
+
Hidden states for all draft positions
+
Base logits for all draft positions
+
+
+
+
+A fully parallel draft component predicts every position without first seeing the tokens selected at earlier positions in the same block. When several continuations are plausible, this can produce inconsistent combinations. For example, both "of course" and "no problem" may be reasonable continuations, but independent position-wise predictions could produce "of problem."
+
+DSpark addresses this behavior by applying a lightweight sequential head after the parallel backbone. The backbone still computes the base logits for every position together. The sequential head then selects tokens from left to right, adjusting each position using information from the previously selected draft tokens.
+
+DSpark applies a lightweight Markov head that introduces dependence between the selected draft tokens. For each position, the Markov head uses the immediately preceding selected token to produce a small bias. This bias adjusts the base logits produced by the parallel backbone:
+
+
+
+
+
Base logits for position k
+
+
+
Bias from draft token k-1
+
+
↓
+
Adjusted distribution for position k
+
+
+
+The main draft network processes all candidate positions together in one forward pass. After that, only the lightweight Markov head runs from left to right to adjust each position using the previously selected draft token.
+
+
+
+This allows later draft tokens to depend on tokens already selected within the same block without running the full draft network again for every position.
+
+The DSpark design also includes a confidence head that can select a shorter draft prefix for target-model verification. This feature was not active in the vLLM path used for our experiments, so the benchmark results reflect only the parallel draft network and lightweight Markov correction.
+
+The target model evaluates the proposed sequence in one verification pass, and draft tokens are committed from left to right until the first rejection.
+
+### Summary of the drafting methods
+
+Figure 3 gives a visual side-by-side view of the five drafting methods: what the draft component looks like, which target-model information it uses, and whether candidate tokens are generated sequentially or in parallel. The table below the figure restates the same comparison in a compact form. In all five methods, the target model still evaluates the proposed sequence in one verification pass, and the acceptance decision is applied from left to right until the first rejected draft token.
+
+
+
+
+
+
+
+
Figure 3. Draft structure and token generation patterns for the five speculative decoding methods discussed in this post.
+
+| Method | Draft component | Target-model information used | How draft tokens are generated |
+| --- | --- | --- | --- |
+| Native MTP | Model-native auxiliary MTP path | A target-model or previous MTP hidden representation combined with current draft-token information | Sequentially through repeated use of the MTP path |
+| Gemma 4 MTP | Separate MTP draft component paired with the target model | Target-model activations and the shared target KV cache | Sequentially through the paired MTP component |
+| EAGLE-3 | Dedicated autoregressive draft network | Hidden states captured near the beginning, around the middle, and near the end of the target-model forward pass, fused into one representation | Sequentially, with each drafted token influencing the next |
+| DFlash | Dedicated parallel draft network | Fused target-model hidden states provided as additional Key and Value information in every draft layer | All candidate positions are predicted together in one parallel forward pass |
+| DSpark | DFlash-style parallel draft network with a lightweight Markov head | The same target-conditioned information used by the parallel draft network | One parallel forward pass followed by lightweight sequential adjustment of token selection |
+
+---
+
+## How to enable speculative decoding in vLLM
+
+In vLLM, speculative decoding is configured through `--speculative-config`. The main differences are the method name, whether a separate draft checkpoint is required, and the number of candidate tokens requested. Current vLLM supports mtp, eagle3, dflash, and dspark as method values.
+
+
+
+For native MTP, the draft component is included with the target model, so the model field is omitted:
+
+```bash
+vllm serve \
+ --speculative-config '{
+ "method": "mtp",
+ "num_speculative_tokens":
+ }'
+```
+
+For Gemma 4 MTP, EAGLE-3, DFlash, and DSpark, the model field normally points to a checkpoint trained for the target model:
+
+```bash
+vllm serve \
+ --speculative-config '{
+ "method": "",
+ "model": "",
+ "num_speculative_tokens":
+ }'
+```
+
+Gemma 4 assistant checkpoints use the MTP path even though they are supplied through the model field. vLLM connects the assistant component to the target model and allows it to share the target KV cache.
+
+Before enabling a method, check that:
+
+- The installed vLLM version supports the method and model architecture.
+- The draft checkpoint is compatible with the target model and method.
+- `num_speculative_tokens` is compatible with the checkpoint.
+- The model card supports the intended hardware and inference backend.
+
+### Memory considerations
+
+Native MTP does not load a separate draft checkpoint and may share components such as the embedding table or output head with the target model. Gemma 4 MTP, EAGLE-3, DFlash, and DSpark load additional draft weights, so sufficient GPU memory headroom should be reserved. The actual overhead depends on the draft-component size, numerical precision, tensor-parallel configuration, and runtime buffers.
+
+---
+
+## Where to find the pretrained draft models
+
+Several organizations now publish pretrained draft models on Hugging Face. Google provides MTP assistants for Gemma 4, while Z-Lab maintains a collection of DFlash checkpoints. Red Hat AI offers draft models across EAGLE-3, DFlash, and DSpark, and DeepSeek's DeepSpec collection provides matched checkpoints for all three methods. LightSeek focuses on EAGLE-based draft models for Kimi, while Inferact publishes draft models for MiniMax and Kimi.
+
+| Draft-model publisher | Methods | Representative models and targets |
+| --- | --- | --- |
+| Google | Gemma 4 MTP | Assistant checkpoints for Gemma 4 E2B, E4B, 12B, 26B-A4B, and 31B target models. [[7]](#ref-7) |
+| LightSeek Foundation | EAGLE-3 and EAGLE-3.1 | EAGLE-based draft models for Kimi-K2.5, Kimi-K2.6, and Kimi-K2.7-Coder, including standard and MLA variants. [[8]](#ref-8) |
+| Red Hat AI | EAGLE-3, DFlash, and DSpark | A collection covering target families such as Llama, Qwen, Gemma, GPT-OSS, GLM, Nemotron, and Mistral. Common suffixes include -speculator.eagle3, -speculator.dflash, and -speculator.dspark. [[9]](#ref-9) |
+| Z-Lab | DFlash | DFlash checkpoints for targets including Qwen3, Qwen3.5, Qwen3.6, Gemma 4, Kimi, MiniMax, GPT-OSS, and Llama. Checkpoint names generally follow the <target>-DFlash pattern. [[10]](#ref-10) |
+| DeepSeek AI | EAGLE-3, DFlash, and DSpark | The DeepSpec collection provides versions of all three methods for Qwen3-4B, Qwen3-8B, and Qwen3-14B, as well as Gemma 4 12B. Examples include eagle3_qwen3_8b_ttt7, dflash_qwen3_8b_block7, and dspark_qwen3_8b_block7. [[11]](#ref-11) |
+| Inferact | EAGLE-3 and DSpark | Draft models including Inferact/MiniMax-M3-EAGLE3, its GQA variants, and Inferact/Kimi-K3-DSpark. [[12]](#ref-12) |
+
+---
+
+## Experimental setup and measurements
+
+After enabling speculative decoding, the practical question is whether the additional drafting work improves end-to-end serving performance. Candidate tokens do not need to be correct at every position because the target model evaluates them before they are committed. Performance therefore depends on how many proposed tokens are accepted and whether the saved target-model decoding work outweighs the cost of drafting and verification.
+
+We evaluate model quality and serving performance using task-grounded benchmarks rather than random token sequences. Acceptance behavior depends on the structure and predictability of actual model outputs, so task-based prompts provide a more representative view of practical performance.
+
+The main performance indicators are:
+
+- Output-token throughput and speedup over the non-speculative baseline.
+- Mean accepted length and draft-token acceptance rates, where available.
+- Model quality relative to the non-speculative baseline.
+
+### Models and experiment coverage
+
+The experiments cover five speculative-drafting approaches across several target-model families. A check mark indicates that benchmark results are available for that target-method combination; a dash indicates that the combination was not included in the current experiments.
+
+
+
+
+
Target model
+
Native MTP
+
Gemma 4 MTP
+
EAGLE-3
+
DFlash
+
DSpark
+
+
+
+
+
google/gemma-4-26B-A4B-it
+
-
+
✓Google
+
✓Red Hat AI
+
✓Z-Lab
+
-
+
+
+
google/gemma-4-31B-it
+
-
+
✓Google
+
✓Red Hat AI
+
✓Z-Lab
+
✓Red Hat AI
+
+
+
Qwen/Qwen3-8B
+
-
+
-
+
✓Red Hat AI
+
✓Z-Lab
+
✓DeepSeek
+
+
+
Qwen/Qwen3.5-27B
+
✓Built-in
+
-
+
-
+
✓Z-Lab
+
-
+
+
+
Qwen/Qwen3.5-122B-A10B
+
✓Built-in
+
-
+
-
+
✓Z-Lab
+
-
+
+
+
Qwen/Qwen3.6-27B
+
✓Built-in
+
-
+
-
+
✓Z-Lab
+
-
+
+
+
Qwen/Qwen3.6-35B-A3B
+
✓Built-in
+
-
+
-
+
✓Z-Lab
+
-
+
+
+
moonshotai/Kimi-K2.5
+
-
+
-
+
✓LightSeek
+
✓Z-Lab
+
-
+
+
+
MiniMaxAI/MiniMax-M3-MXFP8
+
-
+
-
+
✓Inferact
+
-
+
-
+
+
+
+
+The table summarizes the target-method combinations included in the experiments and shows how speculative decoding behaves across different models, workloads, and proposal lengths. Each result should be interpreted within its test configuration, since model architecture, active parameter count, draft-component size, workload, and serving conditions can all affect performance.
+
+### Throughput measurements
+
+For throughput, we measure generated tokens per second against a standard autoregressive baseline and sweep the number of speculative tokens to study how speculation depth affects end-to-end serving throughput.
+
+
+
+
+
+
+
+
+
+
+
Figure 4. Measured output throughput by method and experiment, with the non-speculative baseline included as a reference. Use the selector to switch target models; hover over bars to see speedup and selected proposal length N.
+
+### Main observations
+
+The measurements varied by target model, drafting method, workload, and proposal length.
+
+For gemma-4-26B-A4B-it, the largest measured throughput ratios within the tested sweep were 2.74× and 2.62× for Gemma 4 MTP on GSM8K and MBPP, respectively, and 2.87× and 2.79× for DFlash on MATH500 and HumanEval. The EAGLE-3 measurements ranged from 2.11× to 2.27× across the four datasets.
+
+For gemma-4-31B-it, Gemma 4 MTP measurements reached 2.00× on GSM8K and 1.99× on MBPP, while DFlash reached 2.34× on MATH500 and 2.05× on HumanEval. The EAGLE-3 and DSpark measurements were also above baseline across the four evaluated datasets. The proposal length associated with the largest measured throughput varied by workload.
+
+For Qwen3-8B, the DSpark measurements ranged from 1.15× on MATH500 to 1.63× on GSM8K. DFlash measurements ranged from 1.08× to 1.27×. EAGLE-3 was above baseline on GSM8K, HumanEval, and MBPP, while its largest measured MATH500 value remained below the baseline.
+
+For Qwen3.5-27B, Qwen3.5-122B-A10B, and Qwen3.6-27B, the maximum measured native-MTP values within the tested sweeps were higher than the corresponding maximum DFlash values. The largest ratio in this group was 2.20× for Qwen3.5-122B-A10B on MATH500. The native-MTP proposal length associated with the largest measured throughput ranged from N=4 to N=7, depending on the model and dataset.
+
+For Qwen3.6-35B-A3B, the DFlash measurements ranged from 1.77× to 2.06×, with the largest value occurring at N=7 for each of the four datasets. Native-MTP measurements ranged from 1.28× to 1.49×, with the largest values occurring at N=6. The difference from the Qwen3.6-27B measurements shows that results can vary between models in the same family.
+
+For MiniMax-M3-MXFP8, the EAGLE-3 measurements reached 2.09× on HumanEval at N=4. For Kimi-K2.5, EAGLE-3 measurements reached up to 2.33× and DFlash measurements reached up to 2.68×. Within the tested sweeps, the largest EAGLE-3 values generally occurred at N=4, while the largest DFlash values occurred at N=7.
+
+Across the experiments, the proposal length associated with the largest measured throughput was not constant. For the sequential methods, throughput often increased over the first few values of N before reaching a plateau. For DFlash and DSpark, N=7 was frequently among the higher-throughput settings, while larger values did not consistently increase throughput.
+
+These observations reflect the hardware, software, target model, draft checkpoint, workload, and sweep settings used in this study.
+
+---
+
+## Tuning considerations
+
+Speculative decoding should be treated as a runtime optimization rather than a fixed setting that works equally well for every workload. The value of `num_speculative_tokens` associated with the highest throughput depends on how many proposed tokens are accepted and whether the avoided target-model decode work outweighs the cost of drafting and verification.
+
+Observability is therefore important. A model-card recommendation or example configuration provides a useful starting point, but the final setting should be selected using representative workloads and end-to-end measurements. Useful signals include throughput, mean accepted length, overall acceptance rate, and per-position acceptance rate.
+
+A larger proposal window gives the system more opportunities to commit several tokens in one verification pass. However, acceptance may decrease at later draft positions. When this happens, the additional candidates contribute little while still adding drafting and verification work, causing throughput to flatten or regress.
+
+### Start from a supported configuration
+
+For native MTP, N=1 is a conservative starting point because it introduces the least additional sequential drafting work:
+
+```json
+{"method": "mtp", "num_speculative_tokens": 1}
+```
+
+After confirming correctness and stability, sweep larger values such as 2, 3, 4, 5, 6, and 7.
+
+In our measurements, the native-MTP setting associated with the largest measured throughput varied by target model and workload. For Qwen3.5-27B, the largest measured throughput occurred at N=5 for GSM8K and MATH500, N=4 for HumanEval and MBPP, and N=3 for MT-Bench. For Qwen3.5-122B-A10B, the largest measured throughput across the four listed reasoning and code datasets occurred at N=7.
+
+The Qwen3.6 measurements also show that this setting can change between models in the same family. For Qwen3.6-27B, the largest measured values occurred at N=4 or N=5, while throughput for the tested Qwen3.6-35B-A3B configurations increased through N=6.
+
+For Gemma 4 MTP and EAGLE-3, increasing N also adds sequential drafting work. A short sweep is therefore useful even when the checkpoint provides a recommended configuration. In our Gemma 4 and EAGLE-3 experiments, measured throughput generally increased over the first few values of N before reaching a plateau.
+
+For DFlash, begin with the proposal lengths recommended or supported by the draft checkpoint. Many DFlash checkpoints are trained with a fixed block size. For example, when:
+
+```text
+block_size = 16
+```
+
+the maximum proposal length is normally:
+
+```text
+num_speculative_tokens = 15
+```
+
+because the first position is the confirmed anchor token and the remaining 15 positions are draft candidates.
+
+This is the maximum supported proposal length, not necessarily the highest-throughput setting. In practice, it is useful to test smaller values such as:
+
+```text
+N = 3, 7, 11, 15
+```
+
+Across our DFlash experiments, N=7 was frequently among the higher-throughput settings. For some workloads, the largest measured throughput occurred at N=11.
+
+For DSpark, `num_speculative_tokens` sets the number of candidate tokens generated in each speculative round. In our vLLM experiments, the full configured proposal was submitted for target-model verification, so values such as N=3 and N=7 should be compared using end-to-end throughput.
+
+### Monitor acceptance behavior
+
+Relevant signals to monitor include:
+
+| Signal | What it shows |
+| --- | --- |
+| Throughput | How end-to-end serving performance changes relative to the non-speculative baseline |
+| Mean accepted length | How many draft tokens are committed per speculative round on average |
+| Overall acceptance rate | What proportion of proposed draft tokens are accepted |
+| Per-position acceptance rate | Whether later positions in the proposal remain useful |
+
+Per-position acceptance is particularly helpful when tuning proposal length. If the first few positions are accepted frequently but later positions contribute very little, reducing `num_speculative_tokens` may improve throughput by avoiding unnecessary draft work.
+
+Acceptance metrics should be interpreted together with throughput. A method may show higher throughput relative to baseline even with a lower acceptance rate when draft generation is inexpensive. Conversely, a high acceptance rate does not necessarily correspond to higher throughput when the draft component adds additional overhead.
+
+### Match the sweep to the workload
+
+Different workloads can produce different acceptance patterns.
+
+In our GSM8K and MATH500 measurements, medium or deeper proposal lengths were often associated with higher measured throughput within the tested sweeps. For native MTP on Qwen3.5-122B-A10B, measured throughput increased through N=7. For DFlash, higher measured values frequently occurred at N=7 or N=11.
+
+For HumanEval and MBPP, moderate proposal lengths were often among the higher-throughput settings. Code contains predictable local structure, but formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge.
+
+### Example tuning workflow
+
+1. Begin with a configuration supported or recommended for the checkpoint.
+
+2. Benchmark using representative prompts and generation settings.
+
+3. Record throughput, mean accepted length, and acceptance rates.
+
+4. Sweep several smaller and larger proposal lengths.
+
+5. Select a setting based on the metric most relevant to the intended workload. In these experiments, end-to-end serving throughput was the primary selection metric.
+
+The selected configuration does not necessarily have the longest proposal, the highest acceptance rate, or the largest mean accepted length. Selection should consider the trade-off among drafting cost, verification cost, accepted tokens, and the metric most relevant to the intended workload.
+
+---
+
+## Training a speculator for a new target model
+
+This guide does not cover speculator training in depth. The following workflow summarizes practical considerations from the referenced vLLM Speculators and DeepSpec resources [[13]](#ref-13), [[14]](#ref-14), and [[15]](#ref-15).
+
+A typical workflow is:
+
+1. Prepare representative prompts.
+2. Generate responses with the target model.
+3. Choose a hidden-state generation mode.
+4. Collect the required target-model hidden states.
+5. Train the speculator.
+6. Test acceptance and serving throughput.
+
+### Prepare representative prompts
+
+Start with prompts that reflect the expected workload, such as chat, mathematics, code generation, tool use, or multilingual tasks. Keep a separate set of prompts for evaluation.
+
+The responses used for training should be generated by the exact target model that the speculator will support. The tokenizer, chat template, thinking mode, and generation configuration should also match the intended deployment. The vLLM documentation emphasizes that applying the target model's tokenizer or chat template to existing responses does not make the data target-specific; the responses themselves must come from the target model.
+
+### Choose how to obtain hidden states
+
+The speculator receives internal hidden states from the target model during training. The vLLM Speculators workflow supports three ways to provide them:
+
+| Training mode | How it works | Main consideration |
+| --- | --- | --- |
+| Online | Hidden states are generated by a running vLLM server when needed and discarded afterward | Avoids a large disk cache but requires resources for target inference and training at the same time |
+| Offline | Hidden states are generated and stored before training begins | Frees all GPUs for training afterward but requires substantial storage |
+| Hybrid | Hidden states are generated and cached during the first epoch, then reused | Pays the generation cost once without requiring a separate preprocessing stage |
+
+The selected mode changes where the hidden states come from; the remaining training workflow is largely the same.
+
+### Collect target-model information
+
+A vLLM server can run the target model and expose hidden states from the layers required by the selected drafting method. When custom target layers are chosen, the same layer selections must also be used in the speculator-training configuration.
+
+The information collected depends on the method:
+
+- EAGLE-3 uses hidden states from selected target-model layers for autoregressive drafting. [[4]](#ref-4)
+- DFlash uses target-model features to train a network that predicts a block of future positions in parallel. [[16]](#ref-16)
+- DSpark adds lightweight sequential and confidence heads to a DFlash-style draft network. [[6]](#ref-6)
+- MTP training fine-tunes the target model's own MTP component and therefore requires a target model that already contains compatible MTP layers. [[13]](#ref-13)
+
+### Train and test the speculator
+
+The speculator configuration must match the target model's hidden size, vocabulary, tokenizer, and selected target layers. Method-specific settings such as draft-network depth, block size, sequence length, and learning rate must also be selected.
+
+After training, inspect the checkpoint and serve it together with the target model in vLLM. Training loss alone is not enough to judge the result; the important measurements are accepted length, acceptance rate, draft latency, GPU memory use, and end-to-end serving throughput. The vLLM Speculators tutorial covers the complete path from data preparation and hidden-state extraction to checkpoint testing and serving.
+
+When acceptance is weak for a particular workload, the prompt mixture or training configuration can be adjusted and the process repeated. The main principle is to use the same target model, generation mode, and representative workload that the speculator is expected to support.
+
+---
+
+## Summary
+
+This blog explored speculative decoding in vLLM as a draft-and-verify approach for LLM serving. A draft component proposes candidate future tokens, and the target model evaluates the proposal before any tokens are committed.
+
+We examined five drafting approaches: native MTP, Gemma 4 MTP, EAGLE-3, DFlash, and DSpark. They differ mainly in how they use information from the target model and whether candidate tokens are generated sequentially, in parallel, or through a combination of parallel prediction and lightweight sequential correction.
+
+The experiments covered selected Gemma, Qwen, MiniMax, and Kimi models on AMD Instinct™ MI300X and MI355X GPUs using the ROCm™ software platform. Measured throughput varied across target models, draft checkpoints, workloads, proposal lengths, and serving configurations.
+
+Across the tested configurations, some settings produced smaller changes or throughput below the non-speculative baseline, while several model-workload combinations produced throughput ratios above 2×. Examples at the upper end of the observed range included 2.87× for DFlash on gemma-4-26B-A4B-it, 2.83× for Gemma 4 MTP on the same target, and 2.68× for DFlash on Kimi-K2.5.
+
+Proposal length was also an important experimental variable. Increasing `num_speculative_tokens` sometimes increased throughput over the first few settings, while larger values could lead to a plateau or lower throughput. Checkpoint recommendations can provide starting points, but representative workload measurements and acceptance metrics are needed when selecting a deployment configuration.
+
+## Future work
+
+Future benchmarking could include non-learned approaches such as n-gram speculation and suffix decoding, particularly for workloads with repeated token patterns such as code editing and agentic loops.
+
+Broader evaluation across concurrency levels, prompt and output lengths, batch sizes, and sampling settings would also help show how speculative decoding behaves under different serving conditions.
+
+Another useful direction is to study how speculator training data affects acceptance across code, mathematics, chat, multilingual prompts, tool use, and structured output. This could provide clearer guidance when choosing or training a draft checkpoint for a specific workload.
+
+Finally, deeper profiling of draft generation, target verification, KV-cache behavior, graph execution, and scheduling would help explain the performance differences observed across target models and workloads.
+
+---
+
+## References
+
+1. vLLM documentation, "Speculative Decoding" https://docs.vllm.ai/en/latest/features/speculative_decoding/
+2. vLLM documentation, "MTP Speculative Decoding" https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/
+3. Google Developers Blog, "Multi-token prediction in Gemma 4" https://blog.google/innovation-and-ai/technology/developers-tools/multi-token-prediction-gemma-4/
+4. EAGLE-3 paper, "Scaling up Inference Acceleration of Large Language Models via Training-Time Test" https://arxiv.org/pdf/2503.01840
+5. Z-Lab, "DFlash" GitHub repository https://github.com/z-lab/dflash
+6. DSpark paper, arXiv preprint https://arxiv.org/pdf/2607.05147
+7. Google, "Gemma 4" Hugging Face collection https://huggingface.co/collections/google/gemma-4
+8. LightSeek Foundation model collection on Hugging Face https://huggingface.co/lightseekorg/models
+9. Red Hat AI, "Speculator Models" Hugging Face collection https://huggingface.co/collections/RedHatAI/speculator-models
+10. Z-Lab, "DFlash" Hugging Face collection https://huggingface.co/collections/z-lab/dflash
+11. DeepSeek-AI, "DeepSpec" Hugging Face collection https://huggingface.co/collections/deepseek-ai/deepspec
+12. Inferact model collection on Hugging Face https://huggingface.co/Inferact/models
+13. vLLM Speculators documentation, "Training a Speculator" https://docs.vllm.ai/projects/speculators/en/latest/user_guide/tutorials/train/
+14. vLLM Project, "Speculators" GitHub repository https://github.com/vllm-project/speculators
+15. DeepSeek-AI, "DeepSpec" GitHub repository https://github.com/deepseek-ai/DeepSpec
+16. DFlash paper, arXiv preprint https://arxiv.org/pdf/2602.06036
+
+## Appendix
+
+The appendix focuses on acceptance behavior by draft position. Choose a target model, drafting method, and experiment to view one larger per-position acceptance heatmap. Rows are proposal lengths `N`; columns are draft positions; darker cells indicate higher acceptance. Each row also includes measured speedup and output throughput for context.
+
+
+
+
+