After rearranging where requests ran, we tried reducing the serial work inside one request. A drafter proposes several tokens for the target model to verify together. On our one-user chat grid, trained drafters reduced token latency substantially; with 16 users, the result depended on the prompt type. The strongest coding result did not translate into an equally strong chat-throughput result.
A normal decode step produces one new token while reading the weights and state needed for that step. Sparse MoE models activate only a subset of their experts. Speculative decoding adds a cheaper proposal stage: a draft model, a built-in multi-token-prediction head or prompt lookup offers candidate tokens, and the target verifies them. Accept enough candidates per target pass and fewer serial target steps are needed; reject too many and the extra work can erase the saving.
That trade depends on how many users share the GPU. At low batch sizes, decoding is often limited by memory bandwidth. Checking several draft tokens together can use that bandwidth more efficiently, but verification and drafting still have a cost. With many users the batch already fills the GPU, and the extra guessed tokens compete with real users' tokens. Everything in this post is read along that axis.
A faster guess still has to pay for verification.
- 01 · DraftPropose several tokens
A cheaper draft path guesses what the target model will write next.
- 02 · VerifyCheck with the target
The target evaluates those guesses together. Rejected guesses cannot simply be emitted.
- 03 · AdvanceKeep useful work
Accepted tokens move generation forward. Drafting and verification overhead still count toward latency.
Measure tokens delivered per second and time per output token. A high acceptance rate alone does not establish a speedup.
What we ran
We used one spec_quick grid per setup: chat with one and 16 users, code-writing with 16 users, and a 126K-token document with one user, requesting 512-1,024 output tokens. The Qwen servers also ran JSON/tool-call checks. Needle-gate coverage was not uniform: the gpt-oss EAGLE-3 servers lacked their own check and were excluded under the inherited gate described below. Keep that exception separate from the measured Qwen and GLM comparisons. These runs used vLLM 0.31.0.
- Qwen3.8-27B (dense, one GPU): its built-in MTP head, the DFlash2 drafter [S72] (training recipe and 8× H200 table: [S42]), the DSpark drafter from RadixArk Hugging Face: RadixArk/Qwen3.8-27B-DSpark, and prompt-lookup n-gram.
- Qwen3.6-35B-A3B (MoE, one GPU): its built-in MTP head and the z-lab DFlash drafter Hugging Face: z-lab/Qwen3.6-35B-A3B-DFlash.
- GLM-5.3-Flash (MoE, TP4 on one quad): its MTP head [S121], with vLLM's default sparse-MLA attention backend and with
FLASH_ATTN_MLA_SPARSE. - Qwen3.8-Flash-Next (TEP4 on one quad): MTP only, no baseline (see below).
- gpt-oss-120b (one GPU): two NVIDIA EAGLE-3 drafters [S46], Eagle3-v3 [S73] and a long-context variant. Unpublishable; see below.
The full table
| Model | Drafter | Chat, 1 user: ms/token | Chat, 16 users: tok/s | Code, 16 users: tok/s | 126K doc, 1 user: ms/token | Accepted tokens/step (code) | Run |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B (dense, 1 GPU) | none (baseline) | 11.4 | 874 | 884 | 14.1 | - | 1011-022639 |
| DFlash2 k7 | 4.7 (2.42×) | 1,051 (1.20×) | 2,651 (3.00×) | 6.7 (2.09×) | 5.47 | 1011-022639 | |
| MTP k1 | 7.3 (1.56×) | 950 (1.09×) | 1,412 (1.60×) | 9.2 (1.53×) | 1.91 | 1011-022639 | |
| Qwen3.8-27B (dense, 1 GPU) | none (baseline) | 11.4 | 896 | 855 | 13.9 | - | 1010-033958 |
| DSpark k7 | 5.8 (1.98×) | 986 (1.10×) | 2,228 (2.60×) | 7.1 (1.96×) | 4.75 | 1010-034830 | |
| MTP k2 | 6.1 (1.86×) | 1,016 (1.13×) | 1,856 (2.17×) | 7.5 (1.85×) | 2.71 | 1010-033958 | |
| Qwen3.8-27B (dense, 1 GPU) | none (baseline) | 11.4 | 893 | 927 | 14.2 | - | 1011-023344 |
| n-gram k3 | 14.0 (0.82×) | 458 (0.51×) | 435 (0.47×) | 16.7 (0.86×) | 2.49 | 1011-023344 | |
| Qwen3.6-35B-A3B (MoE, 1 GPU) | none (baseline) | 4.4 | 1,480 | 1,506 | 5.9 | - | 1011-024319 |
| DFlash k7 | 2.6 (1.69×) | 1,323 (0.89×) | 3,545 (2.35×) | 4.1 (1.43×) | 5.02 | 1011-024319 | |
| MTP k1 | 2.9 (1.48×) | 1,340 (0.91×) | 2,263 (1.50×) | 4.1 (1.44×) | 1.91 | 1011-024319 | |
| MTP k2 | 2.7 (1.64×) | 1,314 (0.89×) | 2,714 (1.80×) | 3.6 (1.63×) | 2.69 | 1011-024319 | |
| GLM-5.3-Flash (MoE, 4 GPUs) | none (baseline) | 6.7 | 901 | 1,085 | 6.9 | - | 1011-043339 |
| MTP k3 | 3.7 (1.83×) | 1,062 (1.18×) | 1,835 (1.69×) | 3.5 (1.96×) | 3.01 | 1011-044429 | |
| GLM-5.3-Flash (MoE, 4 GPUs) | none (baseline) | 6.7 | 889 | 1,125 | 6.8 | - | 1011-045400 |
| MTP k3 · FA backend | 3.7 (1.83×) | 1,054 (1.18×) | 1,813 (1.61×) | 3.5 (1.96×) | 2.98 | 1011-050314 | |
| MTP k5 · FA backend | 4.0 (1.67×) | 976 (1.10×) | 1,644 (1.46×) | 3.5 (1.95×) | 3.58 | 1011-051215 |
Medians, one run per cell, our measurement. Speed-up vs the baseline row directly above it, from the same server batch. Accepted tokens per step counts the verified token too, so 1.0 means no guesses kept.
One user: the easy win
One chat user was the clearest favourable case. Qwen3.8-27B with DFlash2 reduced token latency from 11.4 to 4.7 ms (reported output rate 85→215 tokens/s); GLM-5.3-Flash MTP k3 moved from 6.71 to 3.66 ms (124→233 tokens/s). The 126K-document slice also retained gains. That establishes a useful candidate for this one long-document workload, not a guarantee that arbitrary long contexts preserve draft acceptance.
Sixteen users: it depends on what they write
The coding prompts gave the drafter more useful work in this grid. At 16 users, Qwen3.6 with DFlash increased aggregate output from 1,506 to 3,545 tokens/s (2.35×), accepting about 5.0 tokens per step. Qwen3.8-27B DFlash2 reached 2,651 tokens/s (3.0×), and GLM-5.3-Flash MTP k3 increased from 1,085 to 1,835. Repeated names and syntax are plausible reasons for good draft acceptance, but we did not measure prompt entropy; the evidence is the acceptance and throughput for these synthetic tasks.
Chat at 16 users made the cost visible. Qwen3.8-27B DFlash2 improved aggregate output by about 20% (874→1,051), and GLM-5.3-Flash by 18%. Every Qwen3.6 drafter reduced aggregate throughput: 1,480 tokens/s without one versus 1,323 with DFlash. Yet DFlash reduced per-token latency from 9.5 to 7.3 ms. These metrics can move in different directions because end-to-end server output includes the whole request workload. A small active MoE can make the baseline target step relatively cheap; without profiling we cannot assign the aggregate loss solely to draft overhead.
Prompt-lookup n-gram was slower than the no-speculation baseline at every point we ran. It proposes continuations by finding matching text in the prompt, so this is a negative result for our chat and code sets, not a rejection of lookup for workloads with substantial verbatim reuse. It would need a separate editing or copying workload before making that decision.
Deeper isn't better. On GLM-5.3-Flash, guessing 5 tokens instead of 3 raised acceptance but lowered code throughput with 16 users (1,644 vs 1,813 tokens/s on the same backend).
The Qwen JSON and tool-call checks passed 100% with and without drafters. Those checks support compatibility with the tested output contracts, not semantic equivalence or broad quality preservation. Exact speculative sampling can preserve the target distribution when its verification and acceptance procedure meets the algorithm’s assumptions; a generic statement that the target “checks every token” is insufficient to establish that for every implementation. We did not run a distributional validation here. [S46]
An H100 report found the auto-selected FlashInfer sparse-MLA backend 36-70% slower than FlashAttention for GLM-5.3-Flash. We repeated the backend choice on H200 and confirmed FLASH_ATTN_MLA_SPARSE in the log. One-user chat was 6.71 ms/token by default versus 6.67 ms with FlashAttention, with other cells within about 3%. Without repeats or error bars we cannot call that statistical noise; we can say the large reported slowdown did not reproduce in this grid. [S124]
- One run per cell, 16 prompts for the 1-user chat cell, no error bars. Different baselines (rows marked by run time) were run at different times on the same GPU.
- Qwen3.8-Flash-Next MTP k3 ran (5.70 ms per token for one chat user, 1,618 tokens/s for 16 code users, about 3.3 tokens accepted per step), but there's no no-MTP baseline on the same grid, so we can't say what MTP bought. A baseline was planned and dropped for time.
- gpt-oss-120b EAGLE-3 ran (
..1011-041636, baseline and both drafters), but these one-GPU servers didn't run a needle check of their own. Our rule then falls back to the most recent check for the same model and context length, which was the FlashInfer server from Part 3 at 2/3 = 0.667, just under our 0.67 bar. So none of those rows are published. That's a gap in our bookkeeping, not evidence that EAGLE-3 hurt quality. - DSpark for DeepSeek-V4-Flash-0731 on SGLang's DSpark image didn't boot (
AttributeError: 'Fp8MoEMethod' object has no attribute 'runner'). Not run. The DSpark row above is the Qwen3.8-27B drafter on vLLM, which did run. - Chat and code prompts are synthetic sets of a few hundred tokens. Real traffic with long system prompts or repeated text will accept more or fewer tokens.
- Speculation is a per-user latency tool first. With one user, every trained drafter gave 1.5-2.4× faster tokens on every model we could measure.
- Keep a matched baseline for each traffic class. At 16 coding users, Qwen3.6 DFlash gained 2.35× aggregate output and Qwen3.8-27B DFlash2 gained 3.0×. The chat improvements were much smaller, and Qwen3.6 chat lost throughput. Acceptance length alone did not predict which service metric improved.
- Trained drafters (DFlash, DFlash2, MTP) beat prompt lookup on chat and code. n-gram lost everywhere here.
- More guessed tokens isn't free: k5 accepted more and still delivered less than k3.
The server also has to start and remember
Speculation improved selected steady-state workloads, but it assumed an engine already running with room for its state. Part 10 moves to the operational boundary: startup, sleep/wake and returning conversations whose combined context no longer fits in HBM.
The failed original DeepSeek-V4-Flash-0731 DSpark launch above is distinct from the later completed DeepSeek native and DSpark comparisons. The afterword records their checkpoint, workload and validation limits.
Sources
Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.
- [S72] Hugging Face model card (publisher = repo owner), https://huggingface.co/incoai/Qwen3.8-27B-DFlash2
- [S42] SpecForge (SGLang project), Qwen3.8-27B DFlash2 disaggregated training recipe (8x H200 throughput table)
- [S121] canada-quant, https://huggingface.co/canada-quant/GLM-5.3-Flash-W4A16-MTP
- [S46] Peking U. / Microsoft / U. Waterloo / Vector (Yuhui Li et al.), EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (arXiv 2503.01840)
- [S73] Hugging Face model card (publisher = repo owner), https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-v3
- [S124] vLLM project, https://github.com/vllm-project/vllm/issues/56564
- https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark (Hugging Face: RadixArk/Qwen3.8-27B-DSpark)
- https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash (Hugging Face: z-lab/Qwen3.6-35B-A3B-DFlash)