Draft, Verify, Accept: How Speculative Decoding Works

You run a language model on hardware with a spec sheet full of teraflops. You send a prompt, and the answer arrives one word at a time, as if someone were typing it. The processor is not slow. It is waiting.

This article explains what it is waiting for, how speculative decoding gets more output from each wait, and why that trick never changes the answer. You will also get a clear picture of when the trick loses.

Why Decoding Is Slow

A language model produces text as tokens - words or pieces of words - one at a time. Each new token depends on every token before it, so generation is a loop: run the model, pick a token, append it, run the model again. That loop is called decode.

Every pass through the loop reads the model's weights, the billions of learned numbers that make up the model, from memory into the processor. The arithmetic on those weights is fast. Moving them is not. At small batch sizes the speed limit is set by memory bandwidth, not compute:

tokens per second (ceiling) ≈ memory bandwidth (GB/s) ÷ bytes of weights read per token (GB)

A worked example makes the point. A dense 70-billion-parameter model stored at 16-bit precision is about 140 GB of weights. On a system with 273 GB/s of memory bandwidth, the ceiling is 273 ÷ 140, or roughly 2 tokens per second - no matter how many compute units sit idle. Decode is memory-bound.

The naive fix is faster hardware. More bandwidth does raise the ceiling, and smaller or quantized weights lower the bytes per token. Neither changes the rule underneath: one full read of the weights buys one token. To go faster on the same hardware, you need more than one token per read.

That idle time is the opportunity. Because the GPU spends most of each step waiting for weights to arrive, checking several tokens in one pass takes about as long as generating one. Speculative decoding is built on that fact.

One Cycle: Draft, Verify, Accept

Speculative decoding pairs two models. The target is the large model whose output you actually want. The drafter is a much cheaper model that shares the target's tokenizer and guesses ahead, a few tokens at a time - k of them per cycle. The target then checks every guess in a single pass, keeps the ones it agrees with, and adds one token of its own. The video Guess Cheaply, Check in Bulk walks through one cycle step by step.

Across many cycles, the fraction of drafted tokens that survive is the acceptance rate, written α (alpha). It is the single number that decides whether speculation pays.

Why the Answer Does Not Change

This is the property that makes speculative decoding safe to deploy: it is lossless. It changes how fast output arrives, not what the output is.

The greedy case is easy to see. With greedy decoding the target always picks its most likely token. The accept rule keeps a drafted token only if it is exactly the token the target would have picked at that position, and the bonus token is the target's own pick. Every token in the final output is therefore a token the target chose. Run the same prompt with the drafter on and off, and the text is identical, token for token.

Sampling needs a cleverer rule, because the target does not always pick the same token. Let p be the target's probability for the drafted token and q the drafter's:

• If the target likes the token at least as much as the drafter did (p ≥ q), accept it.
• Otherwise accept it with probability p / q.
• On rejection, sample a replacement from the leftover distribution - the positive part of p minus q, renormalized - which covers exactly the probability the drafter under-proposed.

The result is that every output token follows the target's distribution exactly, not approximately. The original research reported 2x to 3x speedups on large models with identical output quality.

Where the Drafter Comes From

A drafter is anything that can propose likely next tokens cheaply. Three families cover most deployments:

A separate small model is the easiest to understand and the easiest to try. Embedded heads usually win in production, because they see the target's own internal state and are trained to predict its choices. Prompt lookup is free, and on copy-heavy work it is hard to beat: vLLM measured up to 2.8x with n-gram lookup on a summarization dataset, against up to 1.5x for a draft model on chat data, both with Llama 3 70B on four H100 GPUs at low load.

Higher acceptance does not automatically mean the fastest drafter. A drafter with excellent acceptance but a heavy draft step can lose to a lighter one with lower acceptance. The figure of merit is accepted tokens per unit of drafter cost.

When It Helps and When It Loses

Three conditions decide the outcome.

Workload predictability. Acceptance depends on how predictable the next token is. Autocompleting a repetitive configuration file is nearly free money: a closing brace is a closing brace. A free-form short story has many good next words, so drafts miss more often. The same drafter on the same hardware can be a clear win on one and a loss on the other.

Load. Speculative decoding spends spare compute to save memory reads. When a server is busy, that spare compute is gone: batching many users already turns each weight read into many tokens, and decode starts to become compute-bound. vLLM reported the same draft-model setup that sped up low-load serving produced a 1.4x slowdown at high request rates, and n-gram lookup a 1.8x slowdown. The rule of thumb: speculation is a latency tool for low-concurrency serving, and at high throughput you shorten k or turn it off.

Drafter overhead. Every rejected draft is wasted work, and every draft step costs time even when it is accepted. If acceptance is low, the overhead wins.

Sidebar: one real deployment

These figures are from one system, measured by the author, and are included to show the shape of the trade-off, not as a benchmark. Hardware: NVIDIA DGX Spark (128 GB unified memory). Target: NVIDIA Nemotron 3.5 Lightning 30B-A3B in NVFP4. Drafter: NVIDIA's published DSpark checkpoint for this model, k = 3. Server: vLLM 0.28.0. Captured 2026-10-06, single request at a time, temperature 0, 400 output tokens, median of three runs, throughput counted from the server's reported token usage. Acceptance is the share of drafted tokens the target accepted, read from the server's metrics.

Same model, same hardware, same flags: JSON nearly doubles, prose barely moves. In an earlier session on the same system, a different drafter type went further, about +25% on code and about −17% on prose, an opposite sign decided purely by how predictable the text was. Vendor speedup charts are often code-generation numbers. Check what workload a published figure used before assuming it applies to yours.

One measurement trap is worth naming because it produced a wrong conclusion on this exact system. With speculation on, a streaming server can deliver several accepted tokens in one streamed chunk. An early benchmark counted chunks as tokens, understated drafter-on throughput by roughly the acceptance factor, and concluded speculation was a net loss everywhere. It was not. A 400-token response had arrived in 197 chunks. Count tokens from the server's usage figures, never from stream events.

The Takeaway

Decode speed is set by how often the model's weights must be read, not by how fast the processor computes. Speculative decoding lets a cheap drafter guess ahead and the target check several guesses in one read, so each read yields more than one token - and the accept rule guarantees the output is exactly what the target alone would produce. Whether it pays is set by one number, the acceptance rate, which depends on your workload and your load. Measure it before you ship it.

References

• Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192
• Chen, C. et al. Accelerating Large Language Model Decoding with Speculative Sampling. DeepMind, 2023. https://arxiv.org/abs/2302.01318
• Google Research. Looking back at speculative decoding. 2024. https://research.google/blog/looking-back-at-speculative-decoding/
• vLLM Project. How Speculative Decoding Boosts vLLM Performance by up to 2.8x. 2024. https://vllm-project.github.io/2024/10/17/spec-decode.html
• Cai, T. et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. 2024. https://arxiv.org/abs/2401.10774
• Li, Y. et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. 2024. https://arxiv.org/abs/2401.15077
• Ebert, D. Speculative Decoding, Visualized. Adaptive ML, 2026. https://www.adaptive-ml.com/post/speculative-decoding-visualized
• Cheng, X. et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. 2026. https://arxiv.org/abs/2607.05147
• NVIDIA. NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark model card. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark