Part 5: The giants: GLM, MiniMax and DeepSeek

The next models forced us to treat memory layout as part of the recipe. GLM-5.3’s 766 GB FP8 checkpoint exceeded one quad’s HBM before allocating KV state; MiniMax-M3 and DeepSeek-V4-Pro were also tested across all eight GPUs. Getting those servers ready was only the first milestone. We then had to ask whether a fast prefill survived the first quality probe, and whether it left a usable decoder behind.

FP8 weights · FP8 KV vLLM 0.31.0 TP8 + DCP8 across both quads

Why eight GPUs, and why that hurts

GLM-5.3 in FP8 is a 766 GB download. One quad has 564 GB of HBM, so the model has to span both quads, and every tensor-parallel all-reduce has to cross the link between the CPU sockets. In Part 1 we measured that link: 83 GB/s for an 8-GPU all-reduce at best, against 337 GB/s inside one quad. The vLLM recipe for GLM-5.3 runs tensor parallelism over 8 GPUs with FP8 KV [S89]. To reach 1M we added decode context parallelism (DCP), which splits the KV cache across GPUs along the token dimension instead of duplicating it on each tensor-parallel rank [S06]. Since this is a TP job that spans both sockets, we also set NCCL_P2P_LEVEL=SYS, the setting that helped all-reduce in our NCCL tests [S24] [S100].

First launch: a crash with a helpful message

The first GLM-5.3 launch got through weight loading and died while initialising attention:

$ grep -m1 NotImplementedError results/runs/glm53__tp8_dcp8__1024k__vllm__1010-045128/server.log
(Worker_TP2_DCP2 pid=1926) ERROR 10-10 05:07:25 [multiproc_executor.py:1058] NotImplementedError: DCP for FlashMLA sparse
  is only validated with the default 'ag_rs' DCP comm backend; got 'a2a'

The error names its own fix. A vLLM issue filed on 4× H200 NVL describes the same failure: a recent change made a2a the default DCP backend for this model family, and passing --dcp-comm-backend ag_rs gets past it vLLM issue #59306. The second launch used it:

$ cat results/runs/glm53__tp8_dcp8__1024k__vllm__1010-053939/cmd.sh      (docker wrapper trimmed)
-e NCCL_P2P_LEVEL=SYS vllm/vllm-openai:v0.31.0 serve /mnt/data/models/zai-org__GLM-5.3 \
  --max-model-len 1048576 --gpu-memory-utilization 0.95 --enable-chunked-prefill --kv-cache-dtype fp8 \
  --enable-prefix-caching --tensor-parallel-size 8 --decode-context-parallel-size 8 --max-num-seqs 8 \
  --dcp-comm-backend ag_rs
$ cat launch.out
ready_s=609.1
engine_kv_tokens=5659136
cross_p2p=ok

It was ready after 10.2 minutes, with a KV cache of 5,659,136 tokens, enough by the engine's estimate for 5.40 one-million-token sequences at once. Without DCP, each of the 8 tensor-parallel ranks would hold a full copy of the compressed MLA cache, and a user of GLM-5.2-FP8 on 8× H200 reports only about 1.5 sequences of 262K at TP8 vLLM issue #53134.

It reads a million tokens, correctly

The needle gate came first: a ~1.05M-token prompt with the code hidden at 10%, 50% and 90% depth. GLM-5.3 is a reasoning model, so each answer started with a search through the text. All three were right.

$ cut -c1-160 results/runs/glm53__tp8_dcp8__1024k__vllm__1010-053939/niah.log
{"key": 0, "depth": 0.1, "prompt_tokens": 1047888, "ok": true, "latency_s": 663.9, "answer": "gh the massive text, I found: \"The access code ...
{"key": 0, "depth": 0.5, "prompt_tokens": 1047886, "ok": true, "latency_s": 664.0, "answer": "hrough the text... I found it: ...
{"key": 0, "depth": 0.9, "prompt_tokens": 1047888, "ok": true, "latency_s": 663.3, "answer": "rough the massive text, I find: ...
GATE niah score= 1.0

With fresh prompts sent one request at a time, GLM-5.3 took 627.5 s to first token at 1M and then 28.9 ms per output token. At 512K and 250K, the first-token waits were 308 s and 148 s. Decode stayed around 29-31 ms/token in this slice, so most of the observed latency growth appeared before the first token. That tells us where to investigate without proving which component of prefill caused the cost.

GLM-5.3’s observed prefill rate stayed around 1,600-1,700 tokens/s from 32K to 1M. That made the TP8+DCP8 layout worth questioning: a nearly flat rate is compatible with a substantial layout or communication cost, but cannot identify one without profiling. We needed an alternative that changed how prefill was distributed, while measuring the effect on decode as well.

Upstream work gave us a concrete alternative. A vLLM pull request reported 39.7K tokens/s for TP4+DCP4 on B300 versus 144K for TP1×PCP4×DCP4. Those are the authors’ results on different hardware, not a diagnosis of our H200 run. A maintainer’s eight-H200 GLM-5.3 recipe combined PCP8, DCP8 and expert parallelism, supported in v0.31.0. Neither source supplied a 1M first-token time for our node, so we used the recipe as the next experiment rather than importing its expected speed-up. [S94] [S91] [S92] [S93]

Measured: PCP8 × DCP8 + EP at 1M

We ran it (fx_glm53_pcp8, recipe from the gist above, stock v0.31.0, finished 00:54 IST Sunday). One difference from the published recipe: we didn't use DeepEP's high-throughput kernels, because DeepEP needs NVLink between all ranks [S11] and our two islands don't have it. The needle test passed 3/3 again (about 331 s per answer, against 664 s on TP8 + DCP8).

The alternative improved the metric we had targeted and worsened another one: prefill roughly doubled, bringing the 1M first token below five minutes, while decode became about 7× slower. That trade prevented us from calling it a general fix. The alternative also disabled CUDA graphs, so parallelism alone was not the only changed variable. For a document-reading service the shorter wait could matter; for a long answer the slower stream could consume the saving.

MiniMax-M3: finished, with one open question

MXFP8 weights (W8A16 on Hopper) · FP8 KV vLLM nightly 1a001d58 TP8 + EP8

MiniMax-M3 introduced a different compatibility problem. We used the eight-GPU recipe for its roughly 444 GB MXFP8 checkpoint and long-context target; the checkpoint size alone does not establish that it could never fit on one 564 GB quad. Our first stable vLLM 0.31.0 launch failed in the attention path, which supported BF16 KV for that configuration rather than the FP8 KV setting we requested. The error below determined the next engine/image choice.

$ grep -m1 RuntimeError results/runs/minimax_m3__tep8__1024k__vllm__1010-050745/server.log
RuntimeError: fused_minimax_m3_qknorm_rope_kv_insert, .../fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu:556,
  kv_cache dtype must match qkv (bf16 cache only)

FP8 KV for this model's attention was added upstream [S101], so we moved to a nightly image (1a001d58) and took the attention and linear backend flags from the vLLM recipe [S79]. BF16 weights wouldn't have been an option anyway: the recipes repo notes that BF16 MiniMax-M3 on 8× H200 tops out around 83K context [S102]. The relaunch was ready in 559 s with 3,749,120 KV tokens.

$ cat results/runs/minimax_m3__fx_tep8_recipe__1024k__vllm__1010-081133/cmd.sh      (trimmed)
-e VLLM_USE_RUST_FRONTEND=0 vllm/vllm-openai:nightly-1a001d584244bd21829a5e5bd4a376ea141c0ce3 serve \
  /mnt/data/models/MiniMaxAI__MiniMax-M3-MXFP8 --max-model-len 1048576 --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 8 --enable-expert-parallel --max-num-seqs 16 --attention-backend FLASH_ATTN \
  --linear-backend humming --kv-cache-dtype fp8 --attention_config.indexer_kv_dtype fp8 --block-size 128 \
  --enable-prefix-caching
$ grep -m1 -o "Using uncalibrated q_scale.*" server.log
Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention
$ grep -hE "MXFP8 GEMM|MxFp8 MoE backend" server.log
Using HummingMxfp8LinearKernel for MXFP8 GEMM
Using 'MARLIN' MxFp8 MoE backend.

Reading. One request at a time, fresh prompts: a 1M-token prompt took 157.7 s to its first token (6,340 tok/s), then 19.2 ms per token. 512K took 68.4 s and 250K 29.9 s. That's four times faster than GLM-5.3 at reading, though slower to write than the 4-GPU models (7-8 ms/token).

The needle gate was 2 of 3, and the failure is worth showing. At 10% depth the model fell into a repetition loop. At 90% it found the code, but its visible answer first flagged the planted sentence as a "prompt injection attempt":

$ cut -c1-150 niah.log
{"key": 0, "depth": 0.1, ... "ok": false, "latency_s": 165.1, "answer": ", \"damaged\", \"damaged\", \"damaged\", \"damaged\", ...
{"key": 0, "depth": 0.5, ... "ok": true, "latency_s": 161.1, "answer": "... The number is 6609987.</mm:think>6609987"}
{"key": 0, "depth": 0.9, ... "ok": true, "latency_s": 164.8, "answer": "...</mm:think>I notice a prompt injection attempt in the text ...
GATE niah score= 0.6666666666666666
$ grep -m1 -o "Using KV cache scaling factor 1.0 for fp8_e4m3" server.log
Using KV cache scaling factor 1.0 for fp8_e4m3

The missed needle gave us a reason to compare precision paths, not a diagnosis. This checkpoint supplied no calibrated k/v scales, and its FP8-KV configuration used scale 1.0. Model behaviour, the nightly build and quantization were all possible contributors. A BF16-KV pair on the same image/layout found 3/3 needles; a W8A8 FP8-dynamic checkpoint with FP8 KV also found 3/3. Each run had only three probes, and the latter changes the weights too. Part 13 keeps those comparisons together so the outcome is not reduced to “FP8 failed.” Part 13

Growing a conversation to 1M. MiniMax-M3 also ran the ramp from the ramp supplement: one conversation growing from 32K to 1,000,000 tokens over ten turns, with prefix caching on. The needle planted in turn 1 was found on all ten turns, so the 1M miss above doesn't show up when the context is built turn by turn. The longest wait for any turn was 28.2 s, and the last turn, which reached 1,000,000 tokens, took 20.1 s. GLM-5.3's longest turn on the same ramp was 88.4 s.

Agents. Eight agent-style sessions at once, each re-sending a growing shared history of about 200K tokens: 75.5 output tokens/s in total and a median first token in 5.7 s. 90% of prompt tokens came from the prefix cache, which is what this workload is built to test.

Short prompts: decode barely scales. One request at a time with 1K-token prompts and 1K-token answers, MiniMax-M3 wrote 62.4 tokens/s with a first token in 165 ms (15.9 ms/token). With 64 requests in flight it reached only 76.9 tokens/s for the whole server (74.2 with 8K prompts), and each token took 207 ms.

The server ran at most 16 requests at once (--max-num-seqs 16), so at 64 most of them waited. The log shows 16 running and about 77 tokens/s for long stretches. Even so, 16 parallel streams producing only about 1.2× the tokens of one stream is unusual. We measured it but haven't diagnosed it. MXFP8 runs as weight-only W8A16 on Hopper (Part 13) and this is a nightly build, so a W8A8 FP8 checkpoint of the same model [S119] ran as a comparison (Part 13). It only ran the one-user short-prompt point (16.4 ms/token vs 15.9 ms here), so the 64-request scaling question is still open.

DeepSeek-V4-Pro: from the first boot to 1M

FP4 experts as published (W4A16 Marlin) + FP8 dense · FP8 KV vLLM 0.31.0 TEP8 at 800K context

DeepSeek-V4-Pro-0813 is the biggest model in the campaign. Its first layout, data-parallel attention with expert parallelism across 8 GPUs (DEP8), never started, at 800K or at 1M:

$ grep -hE "Model loading took|Available KV cache" results/runs/dsv4_pro__dep8__781k__vllm__1010-051328/server.log | head -2
Model loading took 122.92 GiB memory and 971.870094 seconds
Available KV cache memory: -5.34 GiB

The vLLM recipe page for this model recommends exactly that layout for 8× H200, capped at 800K context [S85]. On our H200 NVL cards, with this build, it didn't fit, and the recipes tracker reports the same squeeze, about 5 GiB of KV per rank [S104]. With data-parallel attention, every GPU keeps its own copy of the dense (non-expert) weights. The same recipe page's default single-node command uses TEP8 instead: tensor parallelism over all 8 GPUs for the dense layers and expert parallelism for the MoE layers. The page notes that TP must equal the GPU count "to avoid OOM from replicated dense layers" . NVIDIA's Dynamo H200 recipe also uses TEP8 [S103]. We relaunched with TEP8 at an 800K context:

$ cat results/runs/dsv4_pro__tep8__781k__vllm__1010-110029/cmd.sh      (docker wrapper trimmed)
vllm/vllm-openai:v0.31.0 serve /mnt/data/models/deepseek-ai__DeepSeek-V4-Pro-0813 --max-model-len 800000 \
  --gpu-memory-utilization 0.95 --kv-cache-dtype fp8 --block-size 256 --no-enable-flashinfer-autotune \
  --compilation-config '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --tensor-parallel-size 8 --enable-expert-parallel --max-num-seqs 16 --max-num-batched-tokens 8192
$ grep -hE "Model loading took|Available KV cache|GPU KV cache size|MoE backend" server.log | head -4
Model loading took 102.31 GiB memory and 70.385249 seconds
Using 'MARLIN' Mxfp4 MoE backend.
Available KV cache memory: 25.8 GiB
GPU KV cache size: 1,573,325 tokens, Maximum concurrency for 800,000 tokens per request: 1.97x

Each GPU now holds 102.31 GiB of weights instead of 122.92 GiB, leaving 25.8 GiB per GPU for KV: 1,573,325 tokens, enough for 1.97 sequences of 800K. The engine was ready after 775 s. The FP4 experts run through the Marlin weight-only kernel, the only FP4 MoE path that works on Hopper in this build (Part 13).

The needle gate at about 799K tokens passed 3 of 3:

$ cat results/runs/dsv4_pro__tep8__781k__vllm__1010-110029/niah.log      (first line trimmed)
{"key": 0, "depth": 0.1, "prompt_tokens": 799337, "ok": true, "latency_s": 177.0, "answer": "9242890"}
{"key": 0, "depth": 0.5, "prompt_tokens": 799345, "ok": true, "latency_s": 173.4, "answer": "9242890"}
{"key": 0, "depth": 0.9, "prompt_tokens": 799338, "ok": true, "latency_s": 173.3, "answer": "9242890"}
GATE niah score= 1.0

At about 5-6K tokens/s of prefill, DeepSeek-V4-Pro reads three to four times faster than GLM-5.3 on the same 8 GPUs, and writes at about 22 ms per token.

Two caveats from the logs:

• Some prefix-cache hits at 250K. In the 250K points, 20% (one at a time) and 11% (four at a time) of prompt tokens were served from the prefix cache, according to the engine's counters. In the one-at-a-time point that's one whole request that came back in 3.7 s. Our filter drops it, so the 250K row above uses the other three. The 512K points had no cache hits.
• Queueing at higher concurrency. With four requests at once, the median first token was 84 s at 250K and 112 s at 512K. Those numbers mostly measure waiting for other prompts' prefill, so they stay out of the one-at-a-time table above. They're plotted in How it scales with users below, along with the 16-at-a-time 250K point, which has since finished. Sixteen 250K requests need more KV than the pool holds (6 fit), so that point is marked as queue-dominated.

DeepSeek-V4-Pro at 1M

The same TEP8 layout, relaunched with --max-model-len 1048576 (run dsv4_pro__tep8__1024k__vllm__1010-142942), was ready after 508 s with 1,876,404 tokens of KV. The needle gate at about 1.05M tokens passed 3 of 3.

One 1M-token prompt at a time: first token after 254 s (3,939 tokens/s of prefill), then 23.2 ms per output token. That's 2.5 times faster to read than GLM-5.3 on the same 8 GPUs, and slower than MiniMax-M3 (158 s).

Two simultaneous 1M requests exposed another unresolved trade-off. Median first-token time was 291 s, while median output-token time rose to 445 ms. The server counters showed very little decoding during portions of the workload, consistent with interference while processing the long prompts. The configured prefill chunk was 8,192 tokens, but we did not collect a scheduler trace that would attribute each stall to a particular chunk. This is a warning about the deployed recipe, not a tuned concurrency limit.

We also grew a conversation toward 1M tokens turn by turn (the ramp). DeepSeek-V4-Pro found the needle on all 30 turns across its three sessions. On the last turn of the single-session ramp, about 1M tokens deep with 92% of the prompt already in the prefix cache, the first token came after 34.5 s and decoding ran at 23.2 ms per token.

How it scales with users

The giants were launched for one long prompt at a time, with small sequence caps (--max-num-seqs 8 for GLM-5.3 and 16 for MiniMax-M3 and DeepSeek-V4-Pro) and KV pools that hold only a few 1M-token sequences. So this section is mostly about where they stop scaling, and those points are marked rather than hidden. All MiniMax-M3 points here come from the server that scored 2/3 on the 1M needle check, so they are provisional (see the box above).

How to read these charts. Concurrent users is the number of requests the benchmark keeps in flight (vllm bench serve --max-concurrency). Each point is a separate benchmark run against the named server. Seeded reruns and the exclusions below address cache reuse in the original grids. We plot median and p90 time to first token, decode speed per user (1000 / median time per output token), total output tokens/s for the whole server, and requests completed out of requests sent. Workloads: 256K / 512K / 1M = one long document per request, 512 output tokens. Decode @32K/256K/1M = prompts of that length with 1,024 output tokens. Agent 200K = multi-turn sessions re-sending a ~190K shared history, 2,048 output tokens. Short 1K/1K and 8K/1K = chat-sized prompts. Requests answered from the prefix cache (our 60K tokens/s rule) are dropped from TTFT. A point that contained any is shown for TTFT only, and the rows in invalid_rows.csv are left out.

Read each curve against two capacity limits. The configured sequence cap, vLLM’s --max-num-seqs or SGLang’s --max-running-requests, can hold requests in a queue. The KV/state allocation can also constrain admission or trigger preemption and recomputation. Hybrid and compressed-attention models account for state differently, so the table uses each engine’s logged capacity rather than assuming identical bytes per token. First-token latency includes waiting as well as prompt processing. Hatched grey points mark the capacity-limited cases identified by the campaign; they are not direct scheduler traces. [S36] vLLM docs: Optimization and Tuning

Even below those limits, requests compete for execution time. vLLM’s documented chunked-prefill policy gives decode priority within a token budget and uses remaining capacity for prompt chunks. That is a useful explanation to investigate when another long request slows a stream, but our latency counters do not reconstruct the exact schedule of every request. The charts report the observed effect; assigning the whole effect to one scheduler mechanism would need traces.

How many sequences fit

Max concurrent sequences that fit in the KV pool at each context length, derived from our logged kv_tokens: floor(KV pool tokens per engine ÷ (prompt + output tokens)) × engines (data-parallel ranks). The KV pool is the engine's own number from its startup log (vLLM GPU KV cache size, SGLang max_total_num_tokens). The sequence cap is the other limit, and the lower of the two applies. "vLLM default" is 1,024 per engine, which we read from engine/arg_utils.py in the v0.31.0 image (the value for GPUs with at least 70 GB). "-" means longer than the server's maximum context.

GLM-5.3

MiniMax-M3

DeepSeek-V4-Pro-0813

What the second quad should buy

The giant-model runs justified eight GPUs through particular memory and context requirements, not an assumption that eight is always faster. Part 6 asks the complementary question: when a model fits on one GPU or one quad, should the other island join it or serve its own users?

Later GLM recipe rescues and DeepSeek follow-ups belong to a separate phase. The afterword records the completed follow-up and corrected quality harness; it does not turn the provisional or failed original arms above into validated results.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S89] vLLM project (recipes), https://recipes.vllm.ai/zai-org/GLM-5.3
2. [S06] vLLM project, Context Parallel Deployment (docs; DCP, interleaving credited to Chao Hong, Moonshot AI)
3. [S24] NVIDIA, NCCL environment variables
4. [S100] NVIDIA NCCL, https://github.com/NVIDIA/nccl/issues/1603
5. [S94] vLLM project, https://github.com/vllm-project/vllm/pull/59916
6. [S91] Lucas Wilkinson (vLLM), https://gist.github.com/LucasWilkinson/b39792adeea1cc2b1505fb6b7a2a2654
7. [S92] Red Hat PSAP, https://github.com/openshift-psap/inference-playbooks/pull/43
8. [S93] vLLM project, https://github.com/vllm-project/vllm/pull/57075
9. [S11] DeepSeek, DeepEP README (requirements)
10. [S101] vLLM project, https://github.com/vllm-project/vllm/pull/45744
11. [S79] vLLM project (recipes), https://recipes.vllm.ai/MiniMaxAI/MiniMax-M3
12. [S102] vLLM project (recipes), https://github.com/vllm-project/recipes/pull/1071
13. [S119] EmbeddedLLM, https://huggingface.co/EmbeddedLLM/MiniMax-M3-FP8-dynamic
14. [S85] vLLM project (recipes), https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro-0813
15. [S104] vLLM project (recipes), https://github.com/vllm-project/recipes/issues/381
16. [S103] NVIDIA Dynamo, https://github.com/ai-dynamo/dynamo/blob/main/recipes/deepseek-v4/deepseek-v4-pro/vllm/agg-h200-agentic/deploy.yaml
17. [S36] vLLM project, Engine arguments (docs)
18. drakosha, https://github.com/vllm-project/vllm/issues/59306 (vLLM issue #59306)
19. indrajeetkumar056, https://github.com/vllm-project/vllm/issues/53134 (vLLM issue #53134)
20. https://docs.vllm.ai/en/latest/configuration/optimization.html (vLLM docs: Optimization and Tuning)