Gemma-4-26B on one A100: cutting latency per token.
Each optimization measured on the same model, GPU, and workload — single-stream, one request at a time. The race below replays the measured per-token cadence: vLLM's default against the fastest config we found.
the ladder
Every lever, measured
findings
What the measurements show
CUDA graphs are most of the default's speed
Eager mode launches thousands of small kernels per token for a 128-expert MoE; graphs replay the step as one — 9.4× the throughput, and on by default.
FP8 weights help modestly
+17%. On Ampere the MoE experts stay under-accelerated, and they are only 39% of the bytes moved per token.
A tuned MoE kernel doesn't help at batch 1
vLLM ships no E=128, N=704 config for A100; we autotuned one. Neutral single-stream — decode is bandwidth-bound, so tiling can't beat the HBM wall. It's a throughput lever, for higher concurrency.
Speculative decoding (MTP) is the main win
Gemma-4's MTP drafter is broken in released vLLM (issue #47794); we fixed it with a one-line patch (PR #47953). On coherent text it reaches ~87% draft acceptance and, stacked on FP8, runs 1.89× the default (1.48× at temperature 0.7; ~1.3–1.5× expected on varied traffic). An earlier random-token benchmark had wrongly shown it slower.