Growing one conversation to a million tokens

The fresh-document tests left a practical question unanswered: what happens when a conversation grows instead of arriving all at once? We took GLM-5.3 through ten turns toward a million tokens. Its longest first-token wait was 88.4 seconds, against roughly ten and a half minutes for a fresh million-token prompt. We then added a second conversation and watched much of that improvement disappear for the user waiting behind another long prefill.

GLM-5.3 · FP8 weights · FP8 KV vLLM 0.31.0, TP8 + DCP8 prefix caching on

Why a ramp

A fresh million-token document is one valid workload. A growing coding session or document conversation is another. To test the latter, we ran a ramp: one conversation that starts at 32K tokens and grows over ten turns to 1,000,000 (32K, 64K, 128K, then +131K per turn up to 917K, then 1M). Each turn re-sends the whole history plus new text and asks for 128 output tokens. A needle is planted in the first turn and asked for on every turn.

With prefix caching enabled, the engine can reuse compatible KV blocks from earlier requests instead of prefilling the entire history again. We expected that reuse to reduce the work needed for later turns, and checked the engine counters alongside latency. vLLM exposes prefix-cache queries and hits on its metrics endpoint, which we sampled after each turn to get a hit rate [S35].

One conversation, ten turns

Each turn cost about what its new tokens cost: 131K new tokens took 79-88 s whether the conversation was at 262K or at 917K. The last turn added 82K tokens to reach 1,000,000 and took 57.6 s. Summed over all ten turns, time to first token was 631 s, about the same as the one-shot prompt (628 s), consistent with reuse of the earlier prompt blocks and a similar effective prefill rate. Summed first-token waits are a latency measure, not a direct measurement of total compute. What changes is the experience: the longest single wait was 88 s instead of 10.5 minutes.

Decode stayed at 34.7 tokens/s for this single conversation on every turn, matching the 28.9 ms/token of the one-shot run. At 1M the conversation used 17.7% of the KV cache, consistent with the engine's estimate that about five such conversations fit.

The hit rate tracks the share of each prompt that was already sent, 0% on turn 1 and 92% on turn 10. Those counters support the reuse explanation; they do not trace every kernel or prove that no work was repeated. The needle from turn 1 was found on all ten turns. As with every needle test, that shows retrieval at depth, not reasoning over the whole conversation.

Two conversations at once

Then we ran the same ramp with two conversations side by side. This exposed interference that the single-conversation run could not show.

The latency traces were asymmetric: one session waited much longer than the other. We did not collect a scheduler trace, so the measurements establish interference rather than an exact ordering of every prefill chunk. On every turn one session waited about as long as a single session, and the other about twice as long. At the last turn that was 114.8 s and 62.7 s. Decode suffered too. The session served first decoded at only 1.5 tokens/s (median), while the other conversation was still prefilling. That timing is consistent with decode/prefill interference. The second session, decoding alone afterwards, ran at 23.0 tokens/s. Mixing long prefills into decode steps is a known source of latency spikes in vLLM. A GLM-5.3-Flash user on GB200 traced every slow decode step to a step that also carried prefill tokens vLLM issue #56975. With two sessions the KV cache peaked at 35.4%.

A second model: MiniMax-M3

Publication boundary: the MiniMax FP8-KV server below passed only two of three fresh-prompt retrieval checks at 1M. Its ramp results remain provisional, even though the planted fact was recovered on every ramp turn. We include them to explain the experiment, not as a validated model ranking.

MXFP8 weights (W8A16 on Hopper) · FP8 KV vLLM nightly 1a001d58, TP8 + EP8 prefix caching on

MiniMax-M3 ran the same ramp on the same 8 GPUs (Part 5 has its launch story). Every turn was faster than GLM-5.3's, and the shape is the same: each turn costs roughly what its new tokens cost.

MiniMax-M3's longest turn took 28.2 s, and the last turn, which reached 1,000,000 tokens, took 20.1 s, against 157.7 s for the same million tokens sent in one shot. Summed over all ten turns, time to first token was 169 s. Prefill on the new tokens slowed as the context grew, from 9,587 tokens/s on turn 1 to 4,098 on turn 10, so for this model attention cost does grow with depth. Decode held at 56-62 tokens/s on every turn, the hit rate reached 92% on turn 10, and the needle from turn 1 was found on all ten turns. That does not erase its earlier miss on one of three fresh-prompt retrieval checks. This FP8-KV run remains provisional under the campaign's publication gate; Part 5 records the failed gate.

With two conversations at once, MiniMax-M3 showed the same pattern as GLM-5.3. One session waited for the other's prefill (last turn: 36.6 s and 21.1 s), and the session decoding during the other's prefill slowed to a median of 6.5 tokens/s, against 35.6 for the one decoding on its own. The needle was found on every turn in both sessions.

Every model on the same ramp (updated 11 Oct)

By the end of the campaign we had completed the ramp for the models below: one conversation, the same ten turns from 32K to 1,000,000 tokens, prefix caching on, 128 output tokens per turn. The 4-GPU models ran on one quad; the giants on all eight GPUs.

The 4-GPU models are much faster per turn than the giants, and the linear-attention GLM-5.3-Flash is the fastest of all. But the needle column matters as much as the time columns. GLM-5.3, MiniMax-M3 and DeepSeek-V4-Pro found the turn-1 needle on every turn. GLM-5.3-Flash found it on only one of ten turns of this ramp, even though it found the needle in all one-shot 1M tests in Part 4, and the two DeepSeek-V4-Flash builds and Qwen3.6 missed it on some turns. A needle is a retrieval check, not a reasoning test, so a miss here is a warning sign for multi-turn recall at depth rather than a full verdict. SGLang doesn't expose the prefix-cache hit counters we sampled from vLLM, so the DeepSeek-V4-Flash hit rates are not available.

How it scales with users

The ramp ran with one conversation, then with two at the same time on the same server. That is only two concurrency points. It tests interference between long sessions, not a complete capacity curve. By our derivation from the logged KV pool, GLM-5.3 fits 5 1M-token sequences (sequence cap 8) and MiniMax-M3 fits 3 (cap 16). Two sessions are below both estimated capacity limits. KV exhaustion therefore does not explain these particular waits, though the shared scheduler and GPU work can still delay either session.

The mechanism, according to the vLLM docs: decode steps are scheduled first, and prompt tokens fill the rest of each step's token budget in chunks (vLLM docs: Optimization and Tuning). When one session sends a 131K-token turn, the other session's decode shares every step with those prefill chunks. The numbers are our measurement.

Turn 10 (1,000,000-token context), 128 output tokens per session. Our measurement. A second session at 1M left GLM throughput roughly unchanged and increased MiniMax throughput only modestly. The small aggregate gain did not prevent much longer first-token waits for one session or much slower decode while another long prefill was in flight.

Status of the other ramps (updated Sun 11 Oct)

• Added to the table above: DeepSeek-V4-Pro (TEP8), GLM-5.3-Flash, both DeepSeek-V4-Flash builds, Qwen3.6-35B-A3B and Qwen3.8-Flash-Next, one conversation each.
• Qwen3.8-27B ramped on four replicas to turn 9 (917K tokens) before the run's time cap stopped it; it found the turn-1 needle on turns 1-7 and missed it on turns 8-9. It's not in the table above because it never reached the 1M turn. Part 2 has its numbers.
• Dropped for time when the deadline moved forward: a two-conversation Qwen3.8-27B ramp, a ramp of the DeepSeek-V4-Flash-0731 FP4 release on SGLang's W4A8 path, and a repeat of the MiniMax-M3 ramp. None of them ran, so there are no numbers for them.

• Prefill/decode split (prefill on one quad, decode on the other): the GLM-5.3-Flash split finished (run glm53_flash__pd_1p1d__256k__vllm__1011-063501, provisional because no 256K needle check exists for that model) and so did the Qwen3.8-27B split (run qwen38_27b__pd_1p1d__256k__vllm__1011-073653). Both used the short parallelism grid, not this ramp, so they can't answer the two-session interference question directly. Details in Part 8.

We left this experiment with two separate requirements for a useful long-context service: keep prior work reusable, and protect a user who is already decoding from another user’s large prefill. The ramp exposed both questions; it did not settle the best scheduler or layout. Part 8 follows the short-prompt split experiments, and the afterword adds the missing GLM replica baseline. Neither is a completed million-token split test.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S35] vLLM project, Production metrics (docs)
2. [S33] PKU / UCSD (Zhong et al.), DistServe: Disaggregating Prefill and Decoding (arXiv 2401.09670, OSDI 2024)
3. NolenLiang, https://github.com/vllm-project/vllm/issues/56975 (vLLM issue #56975)
4. https://docs.vllm.ai/en/latest/configuration/optimization.html (vLLM docs: Optimization and Tuning)