Skip to content
Kevin Davila mascotkevindavila.dev> SOFTWARE ENGINEERES
Angular•3 min read

Speculative decoding: cut your TPOT with MTP and EAGLE-3

The number that defines whether your agent feels fast isn't throughput: it's TPOT, the time between tokens. Speculative decoding cuts it without touching the hardware, with an old and elegant trick: a small model proposes several tokens, your big model verifies them all in a single forward pass. It accepts the matches, discards the rest. The output distribution doesn't change — it's mathematically lossless, validated with rejection sampler convergence tests. Source: speculative decoding docs, updated October 23, 2026.

The free method: n-gram

Before training or downloading anything, try the method that needs no draft model. n-gram looks for sequences that already appeared in your prompt and proposes them as speculation:

vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8 \
  --speculative-config '{
    "method": "ngram",
    "num_speculative_tokens": 4,
    "prompt_lookup_min": 2,
    "prompt_lookup_max": 5
  }'

On repetitive workloads — code that repeats imports, JSON with the same structure, edits that rewrite what the model just read — acceptance rates are high and you gain between 20% and 40% speed. On creative prose it barely wins anything. It's the 30-second experiment: turn it on, measure, and if it doesn't move your needle, turn it off.

MTP and EAGLE-3: when the draft is real

Methods with a draft model give the biggest gains, and in 2026 the ecosystem sorted itself into three:

  • MTP (multi-token prediction): the best when your model ships a native MTP head. DeepSeek V4, Qwen3.8 and Gemma 4 do. The draft is an extra layer of the model itself, not a separate model.
  • EAGLE-3: the general-purpose workhorse. A whole community training drafters (speculators); version 3.1 improved robustness with FC normalization. On AMD GPUs it measured up to 2.00x throughput serving Kimi-K2.5 with EAGLE3. Source: vLLM blog, July 2026.
  • P-EAGLE and DFlash: parallel drafting — they generate several draft tokens in a single draft-model forward, instead of sequentially. P-EAGLE beat EAGLE-3 on B200.

The format in vLLM is the same JSON:

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --speculative-config '{
    "method": "mtp",
    "num_speculative_tokens": 2
  }'

num_speculative_tokens is how many tokens the draft proposes per step. More is not better: each extra token proposes more work for the verifier, and if your acceptance rate drops, you're paying for compute that gets thrown away. Start at 2, go up to 4, and measure with your chapter 6 benchmark.

When NOT to enable it

The docs' method selection table is honest: draft-model methods give "high gain" at low QPS and "medium to high" at high QPS; n-gram and suffix give "low to medium" and "medium". The reason is physics: if your GPU is already saturated with a big batch, the draft model's forward steals compute from the main batch.

For that problem, adaptive verification landed in vLLM this year: instead of verifying every proposed token with the same budget, it sizes verification per request based on draft-model confidence. The measured result: a single speculator holds the throughput/latency frontier from batch 1 to batch 256, with no retuning per traffic level. Source: vLLM blog, August 2026. For now it's exclusive to DSpark drafters.

A compatibility gotcha that will save you an afternoon: pipeline parallelism does not compose with speculative decoding (through vLLM 0.15.0). If your deployment uses PP, pick one: PP or speculation, not both.

Checkpoint and the typical error

Checkpoint: your server runs with --speculative-config active and you compared TPOT with and without speculation on your real workload, not the tutorial's. The test: same benchmark, on and off — if the gain is under 10%, the method or the model isn't for your traffic.

The typical error: pushing num_speculative_tokens to 8 because "more speculation = faster". With a low acceptance rate that makes the server slower, not faster. The rate is measurable: vLLM exposes per-request acceptance metrics, and the per-position acceptance rate tells you where your ceiling is.

Next chapter

You sped up decode. Now the other bottleneck: memory. What to do when your agents' KV cache no longer fits the GPU.

> More posts