Part 8: Tuning and splitting prefill from decode

Long prompts made the service problem visible: a request could have acceptable steady decode on its own and become unpleasant when another user started a large prefill. We tried changing the scheduler’s token budget and separating prefill from decode across the two quads. The first barely changed the million-token wait; the second produced a measurable short-prompt trade-off, while the long-context isolation question remained open.

The latency problem we wanted to solve

The ramp and final comparison showed why a single-user benchmark was insufficient. Adding a second growing conversation increased first-token waits and, for some overlapping turns, sharply reduced decode speed. Interference from prompt processing was a plausible explanation. We wanted to test whether spending resources differently could protect the stream a user was already reading, not merely raise peak tokens/s. final head-to-head ramp supplement vLLM docs: Optimization and Tuning

There are two standard knobs for this, and we tried both:

1. Change the per-step token budget. --max-num-batched-tokens controls how much work a scheduler step can admit. A larger budget may reduce the number of chunks needed for a long prefill, but can also lengthen a step that contains prefill work; a smaller budget may help inter-token latency while extending the wait for the prompt. The right direction is therefore a measurement question, not “larger is faster.” vLLM docs: Optimization and Tuning
2. Separate prefill and decode. In a PD deployment one instance processes prompts and transfers the resulting state to an instance responsible for generation. The vLLM documentation motivates separate latency tuning and tail-latency control; it does not promise higher raw throughput for this implementation. DistServe supplies the broader research context. Our experiment asks whether isolation pays for its transfer and resource-allocation cost on this two-quad node. vLLM docs: Disaggregated Prefilling [S33]

Everything below the setup is our own measurement on this node, vLLM 0.31.0.

Knob 1: a bigger prefill chunk (Qwen3.8-27B at 1M)

Qwen3.8-27B reached 1M on one GPU per copy only on its fourth attempt, with --max-num-batched-tokens 8192 among the fixes (Part 7). We then ran the same layout (four one-GPU copies) with a 4× bigger budget, 32,768, on the other quad. Same image, same prompts, same needle gate (3/3 on both).

The larger budget changed the outcome less than we had hoped: 551 s became 527 s for a fresh 1M prompt, with 2-5% improvements at the other measured long lengths. Four times the token budget was not four times the prompt-processing rate. The arms used different quads, so even this small difference includes a hardware-placement confound. It supports a modest recipe adjustment, not a claim that we identified the remaining bottleneck.

Two honest limits. The two arms ran on different quads (A and B), so a quad difference is folded in, though the ramp's early turns match within a few percent. And the 32K arm's conversation stopped after turn 8: on turn 9 a first token came back after 81 s but no decode rate was recorded and the needle check failed, and the server then refused connections, so turns 9-10 and its two-user ramp have no data. We didn't find out why in the time we had.

For GLM-5.3 on eight GPUs the same knob was planned (budget 32,768) and then swapped for a different layout, because a vLLM pull request pointed at the layout rather than the chunk as the prefill bottleneck [S94]: prefill context parallelism halved the 1M wait but made decode about 7× slower (Part 5).

Knob 2: prefill on one quad, decode on the other

The node has two NVLink quads joined by PCIe through the CPU sockets (Part 1), so the split is natural: prefill on GPUs 0-3, decode on GPUs 4-7, each as one TP4 engine, with vLLM's NixlConnector moving each request's KV cache across the socket link after prefill. A small proxy in front sends every request to the prefill engine first, then to the decode engine. The connector and its roles are from vLLM's docs vLLM docs: Disaggregated Prefilling. The splits were queued on our short par_quick grid at 256K max context: 1K-token prompts with 1K-token answers at 1, 64 and 256 users, and 8K-token prompts at 64 users. That's the grid we had time for on Sunday; it's not the 1M ramp, so it can't directly reproduce the two-conversation stall above.

$ grep -A3 "pd_prefill_a\|pd_decode_b" configs/models.yaml      (Qwen3.8-27B, trimmed)
pd_prefill_a: {gpus: $GROUP_A, tp: 4, env: {VLLM_KV_CACHE_LAYOUT: HND, UCX_NET_DEVICES: all},   # + VLLM_SSM_CONV_STATE_LAYOUT: DS for the rerun
               args: [--tensor-parallel-size 4, --kv-transfer-config '{"kv_connector":"NixlConnector","kv_role":"kv_producer"}']}
pd_decode_b:  {gpus: $GROUP_B, tp: 4, ...  '{"kv_connector":"NixlConnector","kv_role":"kv_consumer"}'}
# GLM-5.3-Flash adds MTP (3 draft tokens) on both sides, --enforce-eager on prefill, --max-num-seqs 512 on decode,
# VLLM_SSM_CONV_STATE_LAYOUT=DS so both sides keep the same state layout

GLM-5.3-Flash: steady decode, a slow first token

With prefill moved to its own quad, one user got 3.6 ms per output token (p99 8.1 ms), and 64 users got 13.1 ms (p99 37.9 ms) with 2,374 output tokens/s for the whole server. The decode engine's log showed MTP accepting about 2.4-2.5 tokens per step late in the run.

The first-token result stopped us from treating fast decode as a deployment win: a single 1K-prompt user waited a median 7.4 s (p90 7.5 s), rising to 9.4 s at 64 users. Transfer and handoff overhead were candidates to investigate, but the run did not instrument them separately from scheduling and the engine configuration. The decode side also used eager mode, another difference to carry into a controlled follow-up. We could describe the trade-off, not assign those seconds entirely to PCIe KV transfer.

At 256 users and with 8K-token prompts our proxy held at most about 100 and 50 requests in flight, so those two points are left out (same rule as Part 6).

Qwen3.8-27B: the split against the same model aggregated

The first Qwen3.8-27B split didn't start (below). The rerun with the fix (Sunday 13:00 IST, run qwen38_27b__pd_1p1d__256k__vllm__1011-073653) did, and this model has the baseline GLM-5.3-Flash lacks: the same grid aggregated as TP4 × DP2 on the same eight GPUs (one TP4 engine per quad), and as one TP4 engine on one quad.

• Decode per user was about the same or slightly better. At 64 users the split gave 10.0 ms per token (p99 10.3), against 10.7 ms (p99 12.4) for TP4 × DP2 and 13.8 ms for TP4 on one quad. One user: 5.8 against 6.3 ms.
• The first token got much slower. One user waited 250 ms against 80 ms aggregated; 64 users waited 1,131 ms against 100 ms.
• Throughput went down, as vLLM's docs warn. 4,195 output tokens/s at 64 users against 5,117 for TP4 × DP2 on the same GPUs (0.82×), and about the same as one TP4 engine on half the GPUs (4,131).

At 256 users and with 8K-token prompts our proxy held at most about 100 and 50 requests in flight (the 8K point also had prefix-cache hits), so those points are left out. Like GLM-5.3-Flash, the extra first-token time is unexplained: we didn't log the KV transfer or the proxy, so it's to measure ourselves. Its needle gate is inherited from the last Qwen3.8-27B check at 256K (3/3), as for the other layout runs.

For the measured Qwen short-prompt workload, the same-eight-GPU comparison was decisive enough to reject this split as the default: about 18% less output and longer first-token waits bought a small improvement in token latency. That does not settle the original 1M interference question. We had not run the split under that workload, where isolation might be worth a cost that looks unattractive on short prompts.

The first try: a missing state layout

$ grep -m1 AssertionError results/runs/qwen38_27b__pd_prefill_a__256k__vllm__pd__1011-064806/server.log
(Worker_TP0 pid=1007) ERROR 10-11 06:51:45 [multiproc_executor.py:1058] AssertionError: 3-read Mamba conv transfer
  requires DS conv state layout. Set VLLM_SSM_CONV_STATE_LAYOUT=DS

Qwen3.8-27B is a hybrid model (linear-attention layers with a convolution state, plus full attention), so the split has to move that state as well as the KV cache, and vLLM's transfer needs the DS state layout on both sides. Our GLM-5.3-Flash config set that variable; the Qwen3.8-27B one didn't. The error named the fix; we added VLLM_SSM_CONV_STATE_LAYOUT=DS to both engines and the rerun above started cleanly.

Reduce serial work inside the request

Moving work between quads did not give us a universal latency improvement, and the million-token PD comparison remained unrun. Part 9 changes the unit of work instead: propose several tokens, verify them together, and measure whether accepted drafts repay the added computation at the intended user count.

Later work, including the completed DeepSeek configurations and quality-harness corrections, is separated in the afterword. It adds deployment evidence without supplying the missing matched 1M PD test described here.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S33] PKU / UCSD (Zhong et al.), DistServe: Disaggregating Prefill and Decoding (arXiv 2401.09670, OSDI 2024)
2. [S94] vLLM project, https://github.com/vllm-project/vllm/pull/59916
3. vLLM, https://docs.vllm.ai/en/latest/configuration/optimization.html (vLLM docs: Optimization and Tuning)
4. vLLM, https://docs.vllm.ai/en/latest/features/disagg_prefill.html (vLLM docs: Disaggregated Prefilling)