Part 4: Chasing one million tokens on four GPUs

With the basic serving path established, we moved the question from “does it fit?” to “what does a near-million-token request cost?” Five MoE checkpoint variants ran on one NVLink quad each. Their selected one-shot first-token times ranged from about 59 to 145 seconds. All passed the three-depth retrieval probe, but their rankings changed when we measured decoding under load.

The question

A context limit on a model card left three engineering questions unanswered: could the engine allocate the required state, could the model retrieve a fact at that length, and would the response arrive soon enough to be useful? We tested those separately. The table includes four approximately 1M-token prompts and Qwen3.6’s 963K-token prompt, so these are deployment observations, not a perfectly length-matched model leaderboard.

1. Does it start with --max-model-len 1048576, and how many 1M-token sequences fit in the KV cache?
2. Does it still find things? A random seven-digit code hidden at 10%, 50% and 90% depth of a ~1.05M-token prompt.
3. How long does it take to read a 1M-token prompt (time to first token, TTFT), and how fast does it write afterwards?

The two quads ran different models at the same time, so every number below comes from a 4-GPU job with its own NVLink island and CPU socket.

Two of these need a footnote before any number. Qwen3.6-35B-A3B was launched with --data-parallel-size 4 and no expert parallelism. For a mixture-of-experts model that isn't four replicas: vLLM runs the attention data-parallel and shards the experts tensor-parallel across all four GPUs [S98], and our log says so ("Detected DP deployment with no --enable-expert-parallel. Falling back to AllGather+ReduceScatter dispatch/combine."). We label it data-parallel attention + TP4 experts (no EP). Qwen3.6 also needs YaRN ×4 to go past its native 262K [S80], and its "1M" prompt is 963K tokens. The two DeepSeek-V4-Flash builds are third-party FP8 checkpoints served by SGLang. Where they came from is the subject of Part 13.

$ cat results/runs/glm53_flash__tp4_a__1024k__vllm__1010-000210/cmd.sh      (docker wrapper trimmed)
vllm/vllm-openai:v0.31.0 serve /mnt/data/models/zai-org__GLM-5.3-Flash --max-model-len 1048576 \
  --gpu-memory-utilization 0.92 --enable-chunked-prefill --enable-prefix-caching --language-model-only \
  --max-num-batched-tokens 8192 --tensor-parallel-size 4 --max-num-seqs 64
$ cat results/runs/dsv4_flash_0731_fp8__tp4_a__1024k__sglang__1010-040022/cmd.sh      (trimmed)
-e SGLANG_DSV4_FP4_EXPERTS=0 lmsysorg/sglang:v0.5.21 -m sglang.launch_server \
  --model-path /mnt/data/models/AtlasCloud__DeepSeek-V4-Flash-0731-FP8-DSpark --context-length 1048576 \
  --tp 4 --kv-cache-dtype fp8_e4m3 --page-size 256 --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 --swa-full-tokens-ratio 0.1 --max-running-requests 64

Step 1: does it start, and how much fits?

The "fits" column is the engine's own estimate printed at start-up (Maximum concurrency for 1,048,576 tokens per request). For Qwen3.6, each of the four data-parallel engine cores prints this line for its own rank (11,304,766 tokens each), so one request sees 10.8× and the quad as a whole holds about four times that. SGLang reports its pool as max_total_num_tokens and uses a separate sliding-window pool (--swa-full-tokens-ratio 0.1), so its number can't be compared directly.

Step 2: does it still find the needle?

$ for m in glm53_flash qwen38_flash_next dsv4_flash_0731_fp8 qwen36_35b; do grep '"depth": 0.9' results/runs/${m}__*1024k*/niah.log; done
                     (model name prefixed by us; long reasoning answers trimmed)
glm53_flash          {"depth": 0.9, "prompt_tokens": 1047888, "ok": true, "latency_s": 62.9, ...</think>6609987"}
qwen38_flash_next    {"depth": 0.9, "prompt_tokens": 1047894, "ok": true, "latency_s": 88.6, "answer": "6609987"}
dsv4_flash_0731_fp8  {"depth": 0.9, "prompt_tokens": 1047883, "ok": true, "latency_s": 132.6, "answer": "6609987"}
qwen36_35b           {"depth": 0.9, "prompt_tokens": 1047894, "ok": true, "latency_s": 234.0, "answer": "6609987"}

All five models found the code at all three depths: 15 of 15 needles. That's the minimum bar, not proof of long-context reasoning. A needle is one exact string, and community tests of long-context reasoning on similar models fall off far earlier r/LocalLLM: Qwen3.8-27B MRCRv2 test.

Step 3: how long does a 1M-token prompt take?

GLM-5.3-Flash reached the first token in 58.6 s at 1M input tokens (17,054 tokens/s), then decoded at 7.0 ms/token. Its mixture of KDA linear-attention and sparse full-attention layers is relevant architectural context, but this run does not isolate how much each component contributed. Qwen3.8-Flash-Next took about 85 s. The two DeepSeek-V4-Flash FP8 builds were close at 123 s and 126 s; shared architecture and engine make that unsurprising, without proving the cause. Qwen3.6-35B-A3B took 145 s for 963K tokens and decoded at 17.0 ms/token, compared with 8.1 ms at 250K. That last comparison was the first warning that reading a deep context and writing into it needed separate budgets. r/LocalLLaMA GLM-5.3-Flash megathread (architecture summary)

The clean reruns (11 Oct). With fresh prompts per point and nothing cached, Qwen3.8-Flash-Next read 250K tokens in 11.9 s (21.1K tokens/s) and wrote at 8.2 ms/token, against 85 s (11.8K tokens/s) for the one-shot 1M prompt. The two DeepSeek-V4-Flash FP8 builds again landed together at 250K: 14.1 s (0731, AtlasCloud) and 14.8 s (sgl-project), both about 7.0 ms/token. The sgl-project build's 512K point took 41.2 s (12.4K tokens/s). Run ids: qwen38_flash_next__tep4_b__1024k__vllm__1011-000332, dsv4_flash_0731_fp8__tp4_a__1024k__sglang__1011-012409, dsv4_flash_fp8__tp4_b__1024k__sglang__1011-003258 (point 0 of each 256K/512K grid, 3-4 requests, one at a time).

Writing, not just reading: decode at depth

Time to first token is half the story. The other half is how fast each user's tokens come out once the context is deep and several users share the quad. We ran a decode-depth grid: 16 concurrent requests at 32K context, 8 at 256K and 2 at 1M, each writing 1,024 tokens. From this grid we use only the per-token decode time. Its TTFTs include cache hits from the seed issue above.

At 32K context with 16 users, Qwen3.6 had the lowest token time in this group at 10.2 ms, followed by the DeepSeek builds at 11.9 and 12.2 ms. GLM-5.3-Flash, the fastest one-shot reader, was the slowest writer here at 30.0 ms. At 256K with eight users, the range was 40-73 ms/token. Both context depth and concurrency change between these two slices, so they do not isolate the cost of context alone. They do show why the model with the fastest first token need not provide the fastest stream under load.

There's a known Hopper-specific issue in this exact spot: on H100, vLLM auto-selects a FlashInfer sparse-MLA backend for GLM-5.3-Flash that a user measured as 36-70% slower than the FlashAttention one [S124]. Our GLM-5.3-Flash logs show the same auto-selection (Using FLASHINFER_MLA_SPARSE_SM90).

The rerun (Sun 11 Oct). We ran GLM-5.3-Flash on TP4 (quad A, vLLM 0.31.0) with the default backend (log: FLASHINFER_MLA_SPARSE_SM90) and with --attention-backend FLASH_ATTN_MLA_SPARSE (log confirms it), on a short grid of chat, code and one 126K document, plus MTP with 3 draft tokens on the default backend:

Medians, one run per cell, our measurement. Runs glm53_flash__tp4_a__1024k__vllm__spec-nospec__1011-043339, glm53_flash__tp4_a__1024k__vllm__spec-nospec_fasp__1011-045400, glm53_flash__tp4_a__1024k__vllm__spec-mtp_k3__1011-044429.

On H200 with this build the per-token times of the two attention backends were within about 3% of each other in every cell (the one-user output rate differed by about 9%, which we didn't investigate), so the H100 slowdown [S124] didn't show up here, and the decode numbers above stand. MTP, the model's own draft head [S121], is what moved decode: 6.7 → 3.7 ms per token for one user. Part 9 has the full speculative-decoding comparison, including how the gain shrinks with more users.

How it scales with users

The earlier one-request measurements establish prompt-processing baselines; the decode grids already show how sharing changes the response. Here we bring the full concurrency sweeps together, including dense Qwen3.8-27B at its 256K configuration. These grids stayed below the logged sequence and nominal KV limits, but that does not eliminate scheduler waiting. It only rules out those particular configured capacity caps as a sufficient explanation for the curves.

How to read these charts. Concurrent users is the number of requests the benchmark keeps in flight (vllm bench serve --max-concurrency). Each point is a separate benchmark run against the named server. Seeded reruns and the exclusions below address cache reuse in the original grids. We plot median and p90 time to first token, decode speed per user (1000 / median time per output token), total output tokens/s for the whole server, and requests completed out of requests sent. Workloads: 256K / 512K / 1M = one long document per request, 512 output tokens. Decode @32K/256K/1M = prompts of that length with 1,024 output tokens. Agent 200K = multi-turn sessions re-sending a ~190K shared history, 2,048 output tokens. Short 1K/1K and 8K/1K = chat-sized prompts. Requests answered from the prefix cache (our 60K tokens/s rule) are dropped from TTFT. A point that contained any is shown for TTFT only, and the rows in invalid_rows.csv are left out.

Read each curve against two capacity limits. The configured sequence cap, vLLM’s --max-num-seqs or SGLang’s --max-running-requests, can hold requests in a queue. The KV/state allocation can also constrain admission or trigger preemption and recomputation. Hybrid and compressed-attention models account for state differently, so the table uses each engine’s logged capacity rather than assuming identical bytes per token. First-token latency includes waiting as well as prompt processing. Hatched grey points mark the capacity-limited cases identified by the campaign; they are not direct scheduler traces. [S36] vLLM docs: Optimization and Tuning

Even below those limits, requests compete for execution time. vLLM’s documented chunked-prefill policy gives decode priority within a token budget and uses remaining capacity for prompt chunks. That is a useful explanation to investigate when another long request slows a stream, but our latency counters do not reconstruct the exact schedule of every request. The charts report the observed effect; assigning the whole effect to one scheduler mechanism would need traces.

How many sequences fit

Max concurrent sequences that fit in the KV pool at each context length, derived from our logged kv_tokens: floor(KV pool tokens per engine ÷ (prompt + output tokens)) × engines (data-parallel ranks). The KV pool is the engine's own number from its startup log (vLLM GPU KV cache size, SGLang max_total_num_tokens). The sequence cap is the other limit, and the lower of the two applies. "vLLM default" is 1,024 per engine, which we read from engine/arg_utils.py in the v0.31.0 image (the value for GPUs with at least 70 GB). "-" means longer than the server's maximum context.

GLM-5.3-Flash

Qwen3.8-Flash-Next

DeepSeek-V4-Flash-0731 FP8 (AtlasCloud)

DeepSeek-V4-Flash FP8 (sgl-project)

Qwen3.6-35B-A3B

Qwen3.8-27B

When one quad is no longer the baseline

The one-quad results established several feasible long-context candidates, but left larger checkpoints outside the comparison. Part 5 crosses the socket boundary for GLM-5.3, MiniMax-M3 and DeepSeek-V4-Pro, where memory placement, kernel compatibility and decoding cost become inseparable from the launch recipe.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S90] vLLM project (recipes), https://recipes.vllm.ai/zai-org/GLM-5.3-Flash
2. [S82] vLLM project (recipes), https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next
3. [S84] vLLM project (recipes), https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Flash-0731
4. [S80] vLLM project (recipes), https://recipes.vllm.ai/Qwen/Qwen3.6-35B-A3B
5. [S98] vLLM docs, https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html
6. [S124] vLLM project, https://github.com/vllm-project/vllm/issues/56564
7. [S121] canada-quant, https://huggingface.co/canada-quant/GLM-5.3-Flash-W4A16-MTP
8. [S36] vLLM project, Engine arguments (docs)
9. u/Melodic-Clue9555, https://www.reddit.com/r/LocalLLM/comments/1vp1l5p/ (r/LocalLLM: Qwen3.8-27B MRCRv2 test)
10. https://www.reddit.com/r/LocalLLaMA/comments/1vyzzxu/ (r/LocalLLaMA GLM-5.3-Flash megathread (architecture summary))
11. https://docs.vllm.ai/en/latest/configuration/optimization.html (vLLM docs: Optimization and Tuning)