Part 12: Nine models, one node, one million tokens

At the end of the original campaign, a single league table would have hidden the most useful result. Several models accepted a million-token prompt; fewer retained the planted fact as a conversation grew; and adding one more user could undo an impressive single-user decode rate. We put those measurements beside each other to choose the next deployment test, not to declare an overall best model.

Where we ended up

We started with eight H200 NVL GPUs arranged as two NVLink quads, 2 TB of host RAM and 12 TB of NVMe. The configurations below reached different milestones on that node. The fresh-document measurements follow the campaign filters; the ramp intentionally reuses prior context, and its MiniMax row remains provisional. Combining the evidence in one table does not make every row equally validated.

Read the first two quality columns as different experiments. The fresh-document check sends one large prompt and tests recall at sampled positions. The conversation ramp grows from 32K toward 1M over ten turns and asks for the original hidden fact again each time. The ramp report explains the requests. Neither is a broad reasoning evaluation, and completing one does not certify the other.

Sources for every cell: needle columns from results/runs/<run>/niah.log (three depths, 10/50/90%) and results/ramp1m.csv; speed and cost from results/ramp1m.csv at $3.50 per GPU-hour (Part 11 explains the price); GSM8K from results/quality/quality.jsonl, only where we ran it. Layouts and engines are in Parts 4 and 5. We didn't run the same accuracy test on every model, so this isn't a quality ranking.

Read the result with its test

Three different claims about a million tokens.

1. 01 · CapacityThe engine accepts the request

   A configured limit and a completed prefill establish that this workload fits.

2. 02 · RetrievalThe answer survives the context

   A hidden-fact check tests specific recall, at the sampled positions and prompts.

3. 03 · ConversationThe next turn still works

   A growing conversation exercises repeated turns, retained state and competing users.

Passing one does not establish the others. None of these checks alone measures broad reasoning or production reliability.

The last turn changed when another user arrived

User concurrency is the theme we kept coming back to, and at 1M it bites hardest. With one conversation, the fastest models answer a ~1M-token turn in 7-14 s and then write at well over 100 tokens per second. Add a second user growing their own conversation and two things happen:

• The second request waits. One of the two users waits roughly twice as long for the first token, while the other conversation also has new tokens to prefill. The timing is consistent with shared-resource interference; we did not collect a scheduler trace for every step.
• Decode stalls behind prefill. The user already receiving output slows while the other request prefills. The measurements show the interference, though they do not isolate a single scheduler mechanism. The worst case was DeepSeek-V4-Flash FP8 (sgl-project): 129 tokens/s alone, 7 with a second user, about 18× slower.

The slowdown made prefill/decode separation worth testing: give the two kinds of work separate GPU groups and pay the cost of transferring state between them. The key question was whether isolation would repay that cost on this two-quad node. Part 8 records the short-prompt tests that fitted into the campaign.

Only a short-prompt version of that test fitted in the time we had. Splitting GLM-5.3-Flash across the two quads gave one user 3.6 ms per output token, but 7.4 s to the first token even for a 1K-token prompt (provisional: no 256K needle check exists for that server). The Qwen3.8-27B split ran after a one-line fix: slightly faster decode than the same model aggregated on the same eight GPUs, but a slower first token and less output (Part 8). Neither ran at 1M, so whether a split fixes the stall above is still untested on this node.

Which result we would use for which decision

These are readings of our own numbers on this node and these engine versions, not general rankings.

• Fastest 1M conversations on four GPUs: Qwen3.8-Flash-Next and GLM-5.3-Flash. GLM-5.3-Flash is the fastest and cheapest, but it recalled the hidden fact on only 1 of 10 turns of the growing conversation (it passed the 1M one-shot test). Qwen3.8-Flash-Next recalled 9/10 alone and 20/20 with two users.
• Complete recall on these sampled 1M checks: GLM-5.3 and DeepSeek-V4-Pro, both 3/3 on the one-shot test and 10/10 and 20/20 on the growing conversation, both on eight GPUs. MiniMax-M3 also recalled 10/10 and 20/20 and was quicker and cheaper than both on the last turn, but its server scored 2/3 on the one-shot test, so its numbers are provisional.
• Many users, short prompts: small mixture-of-experts models as one-GPU copies. Qwen3.6-35B-A3B reached the lowest cost per token we measured (Part 11), and one copy per GPU beat splitting a model across GPUs (Part 6).
• A dense model you want on one GPU: Qwen3.8-27B FP8, one copy per GPU. It reaches 1M on one GPU only slowly (about 9 minutes to the first token of a fresh 1M prompt), so we'd keep it at 256K.
• DeepSeek-V4-Flash: the two community FP8 builds behaved almost identically on speed; the AtlasCloud FP8 cast improved measured prompt latency versus the original FP4 release, but the unequal GSM8K sample sizes do not establish equivalent quality (Part 13).

The original campaign still had open questions

What we would carry into the next run

1. Check the topology first. Two NVLink quads joined by PCIe change every layout decision (Part 1).
2. Set the IOMMU to passthrough before anything else; cross-socket GPU copies failed until we did (Part 1).
3. Start from the engine's or publisher's recipe, then read the issue tracker. Every model here started from a published recipe (Parts 2, 4 and 5), and most of our fixes came from issue trackers (Part 7).
4. Audit your benchmark for prefix-cache hits. A reused random seed made some of our "250K" prompts come back from cache in under a second (Part 7).
5. Grow conversations to 1M, don't just send one 1M prompt. GLM-5.3-Flash passed the one-shot test and failed the growing one (ramp episode).
6. Count users × context, not context. It decides when you need KV offload (Part 10) and what a token costs (Part 11).
7. Don't split one model across the islands if replicas will do (Part 6).
8. Speculative decoding paid most with a published drafter for the exact model: DFlash more than doubled 16-user coding output on Qwen3.6-35B, while n-gram drafting made Qwen3.8-27B slower (Part 9).
9. Keep the vLLM compile cache on a volume; it was the biggest cold-start saving we measured (Part 10).
10. Label precision honestly. FP8, MXFP8, MXFP4 and FP4 behave differently on Hopper (Part 13).

The tables and charts retain their workload conditions and run references. The original evidence is in the campaign’s results/ directory. Keeping the failures there mattered as much as saving the best result: otherwise the next person would repeat the same invalid comparisons.

One last question before choosing a checkpoint

The table labels still hid an assumption: that two files called FP8 would use comparable storage and execution paths. They did not. Part 13 opens the tensor headers and checks what the labels actually mean.

Later follow-up: the “failed” and “dropped” entries above describe the original campaign, not the final state of this project. The afterword adds the missing GLM replica baseline and the completed DeepSeek-V4.1 native, DSpark and two-replica tests. They use different workloads and context limits, so their new numbers stay outside this million-token comparison.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S86] vLLM project (recipes), https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4.1-Flash