Part 2: Qwen3.8-27B, from smoke test to the 1M wall

We chose Qwen3.8-27B as a baseline because its FP8 weights fit on one H200. That let us separate two questions early: how much parallelism helps a request, and how much concurrency helps the server. At native 256K context the three-depth retrieval gate passed. Extending to 1M then exposed a different class of problem: enough reported memory did not guarantee a working first request.

FP8 weights · FP8 KV vLLM 0.31.0 native 262K, YaRN ×4 for 1M

Why this model first

The first checkpoint was small enough to make iteration practical: Qwen3.8-27B-FP8 downloaded 30.9 GB across 66 shards in 181 s. One H200 could hold the weights and a substantial KV pool, so we could compare four ways of spending the same four GPUs without first solving an eight-GPU memory problem. The layouts below ask different product questions: a faster lone request, more independent requests, or a compromise between them. [S81] [S04]

$ cat results/runs/qwen38_27b__dp4_a__256k__vllm__1009-230337/cmd.sh     (wrapped, volumes trimmed)
docker run -d --gpus '"device=0,1,2,3"' --ipc=host --network=host --shm-size=64g \
  --cpuset-cpus 0-23,96-119,24-47,120-143 --cpuset-mems 0,1 \
  vllm/vllm-openai:v0.31.0 serve /mnt/data/models/Qwen__Qwen3.8-27B-FP8 \
  --served-model-name qwen38_27b --max-model-len 262144 --gpu-memory-utilization 0.9 \
  --enable-chunked-prefill --kv-cache-dtype fp8 --language-model-only --enable-prefix-caching \
  --reasoning-parser qwen3 --data-parallel-size 4
$ grep -E "Model loading|Graph capturing|KV cache size|init engine" server.log      (one DP rank; log prefixes trimmed)
Model loading took 27.64 GiB memory and 15.007332 seconds
Graph capturing finished in 87 secs, took 1.22 GiB
GPU KV cache size: 2,915,222 tokens, Maximum concurrency for 262,144 tokens per request: 11.12x
init engine (profile, create kv cache, warmup model) took 204.82 s (compilation: 29.57 s)

Quality before speed

We made correctness a launch gate before timing throughput. Each server received a synthetic long prompt with a random seven-digit access code planted at 10%, 50% and 90% depth. The request was to return that code. This deliberately small check catches some unusable configurations cheaply; it is not a substitute for a model evaluation. The campaign’s publication threshold was a score of at least 0.67 at the same configured context length.

$ tail -4 niah.log
{"key": 0, "depth": 0.1, "prompt_tokens": 261565, "ok": true, "latency_s": 51.5, "answer": "1078296"}
{"key": 0, "depth": 0.5, "prompt_tokens": 261565, "ok": true, "latency_s": 51.7, "answer": "1078296"}
{"key": 0, "depth": 0.9, "prompt_tokens": 261564, "ok": true, "latency_s": 51.7, "answer": "1078296"}
GATE niah score= 1.0

The numbers at 256K

With DP4, each request runs on a single GPU. One 250K-token prompt took 45.0 s to its first token, a prefill speed of about 5.3K tokens/s on one H200, then 16.7 ms per output token. That's the median of the requests that actually had to process the prompt. The benchmark's warm-up request is identical to the first measured one, so that one was served from the prefix cache in 0.5 s, and we drop it.

On short chat-sized traffic (1K tokens in, 1K out) the four layouts behave very differently. With a single user, TP4 spreads every token over four GPUs and decodes at 6.3 ms/token, against 11.5 ms for a one-GPU replica. With 256 concurrent users, TP4 delivers the most total throughput (7,854 tok/s for the quad) but each user waits 30.6 ms per token. Two TP2 replicas give 5,897 tok/s at 16.2 ms, and four TP1 replicas 5,314 tok/s at 17.6 ms. Which one is "best" depends on whether you sell throughput or latency, the trade-off NVIDIA describes as a Pareto frontier [S18].

The same TP4 configuration also gave us a reason to investigate speculative decoding: its built-in multi-token-prediction head, drafting three tokens per step, reduced single-user decode from 6.3 to 2.6 ms/token in that run. Part 9 tests drafting more systematically across chat, code and different user counts; this one point is the lead, not the whole recommendation.

Going to one million tokens

The recipe reaches 1M with YaRN, scaling the native 262,144-token rope window by 4 [S81]. In vLLM that is an --hf-overrides JSON. The engine accepted it and reported room for 2.85 one-million-token requests per GPU.

$ grep -oE -- "--max-model-len [0-9]+|--hf-overrides .*" cmd.sh
--max-model-len 1048576
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10],
  "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0,
  "original_max_position_embeddings": 262144}}}'
$ grep -m1 "KV cache size" server.log
GPU KV cache size: 2,993,490 tokens, Maximum concurrency for 1,048,576 tokens per request: 2.85x

Then the first 1,047,894-token prompt ran for 436 seconds and took the server down with a DeepGEMM error. We tried twice more with community-suggested changes. Each attempt failed differently.

# attempt 1: defaults
{"key": 0, "depth": 0.1, "prompt_tokens": 1047894, "ok": false, "latency_s": 436.1, "answer": "ERROR 'choices'"}
RuntimeError: Panic error (.../deep_jit/backend/cuda/driver.hpp:33): deep_jit::cuda::driver::lazy_cuTensorMapEncodeTiled( &tensor_map ...
torch.AcceleratorError: CUDA error: operation not supported on global/shared address space
# attempt 2: VLLM_USE_DEEP_GEMM=0, --max-num-batched-tokens 16384
RuntimeError: Triton Error [CUDA]: an illegal memory access was encountered      (during engine init)
# attempt 3: + --max-num-seqs 16 (ready in 156 s, prefilled the 1M prompt, then 0 tok/s for ~10 min)
{"key": 0, "depth": 0.1, "prompt_tokens": 1047894, "ok": false, "latency_s": 631.3, "answer": "ERROR ..."}
TimeoutError: RPC call to sample_tokens timed out.
Engine core process EngineCore_DP0 died unexpectedly.

What the community and the vLLM tracker told us afterwards:

• The sample_tokens timeout is vLLM’s execute-model deadline, 300 s by default. Our engine reported 0 tokens/s for about ten minutes. That was enough to mark the attempt unsuccessful, but not enough to distinguish a deadlock from an exceptionally slow or stalled kernel. Increasing the deadline alone was not a demonstrated fix; it would first need evidence that useful progress continued. [S106]
• A YaRN multi-axis rope bug affecting Qwen3.8 at long context was fixed in vLLM on 10 October, the same day as these runs [S107]. The fix isn't in any image we have.
• One user runs Qwen3.8-27B at 512K on vLLM and had to raise the batched-token budget to 32K "to make prompt processing bearable over 256K" r/LocalLLaMA: extending Qwen 3.8 27B to 1M (comment by u/rmhubbert). Nobody in that thread reports a working 1M run on vLLM.
• A community report described a related failure mode: without a fixed KV allocation, the pool could consume memory needed by the first real request. That made an explicit memory reserve worth testing. The report used Qwen3.8-27B NVFP4 on a 5090, so it was a lead for our H200 investigation rather than confirmation of the same cause. r/LocalLLM: Qwen3.8-27B NVFP4 on a 5090 (u/PersonalStorage)

The fourth attempt: 1M works (updated Sun 11 Oct)

The fourth attempt ran on Sunday morning with the KV pool pinned at 40 GiB (--kv-cache-memory-bytes 42949672960, 1,297,127 tokens), PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128, VLLM_USE_BREAKABLE_CUDAGRAPH=0, DeepGEMM off, --max-num-batched-tokens 8192 and --max-num-seqs 16, four replicas on quad A (dp4_a). It served a 1,047,894-token prompt and found the needle 3 times out of 3, about 11 minutes per probe. A second arm on quad B (dp4_b) used a 32K batched-token budget. We changed four things at once and didn't isolate which one fixed the failure. The YaRN fix mentioned above is still not in our image, and we can't rule out that the bug affects answers in ways a needle probe doesn't catch.

The second successful server, dp4_b, also found the needle 3/3. Raising the batched-token budget from 8K to 32K reduced the fresh 1M first-token time from 551 s to 527 s, about 4%. The arms ran on different quads, and several fixes had changed between the failed and successful launch generations. We could recommend the working configuration, but could not attribute the rescue to one flag or treat the small budget difference as an isolated causal result.

Medians, one prompt per user, our measurement. With 2 or 4 users each sending a prompt of that size at once, each user's times stayed close to one user's: the four replicas served them in parallel, one request per replica, instead of queueing.

We then grew one conversation toward 1M on dp4_a, following the ramp supplement. First-token latency rose from 2.9 s at turn 1 (32,752 tokens) to 127.5 s at turn 9 (917,490 tokens), while decode fell from 79 to 31 tokens/s. Reusing a prefix does not make newly added tokens independent of the existing context, and the counters do not prove that only new tokens required work. Turn 9 was still much faster to first token than the 551 s fresh 1M request, but the workloads differ. The planted fact was found on turns 1-7 and missed on turns 8 and 9, beyond roughly 650K tokens, despite the successful one-shot 1M probes. The last turn hit the run’s time cap and is excluded; the two-conversation ramp never started before the campaign cut-off. the ramp post

A repeat of the ramp on dp4_b (run ..1011-044415) reached turn 8 (786,418 tokens, first token after 105.0 s) before its own time cap. Times matched the first ramp within a few percent, and the turn-1 needle was found on all 8 turns, including turn 8, where the first ramp missed it. So the late-turn misses weren't repeatable across the two runs; with one probe per turn, we can't say more than that.

The next variable was the kernel

Qwen gave us a working baseline and a caution: a model’s context limit is not a completed workload. Part 3 turns to gpt-oss-120b, where memory was already sufficient and the intended variable was the MoE kernel. That test asks whether a local execution choice can beat adding parallelism.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S81] vLLM project (recipes), https://recipes.vllm.ai/Qwen/Qwen3.8-27B
2. [S04] vLLM project, Data Parallel Deployment (docs)
3. [S18] NVIDIA, Demystifying AI Inference Deployments for Trillion Parameter LLMs (blog)
4. [S106] vLLM project, https://github.com/vllm-project/vllm/pull/58779
5. [S107] vLLM project, https://github.com/vllm-project/vllm/issues/58870
6. u/Melodic-Clue9555, https://www.reddit.com/r/LocalLLM/comments/1vp1l5p/ (r/LocalLLM: Qwen3.8-27B MRCRv2 long-context test)
7. https://www.reddit.com/r/LocalLLaMA/comments/1vrncdt/extending_qwen_38_27b_to_1m/ (r/LocalLLaMA: extending Qwen 3.8 27B to 1M (comment by u/rmhubbert))
8. https://www.reddit.com/r/LocalLLM/comments/1vv5j3y/ (r/LocalLLM: Qwen3.8-27B NVFP4 on a 5090 (u/PersonalStorage))