The original campaign finished with two problems still worth investigating: a GLM prefill/decode split without a matching replica baseline, and a DeepSeek-V4.1 FP8 checkpoint that would not load. We returned for focused rescue tests. This unnumbered afterword records those completed follow-ups separately from the original 13-part campaign. The extended work continues in Parts 14-15 of the series.
These measurements did not pass through the original collector’s full publication gate. The correctness checks, configuration differences and context limits below travel with the results.
Splitting prefill and decode needed a better baseline
Prefill and decode do different work. Giving them separate GPU groups is an appealing way to schedule them independently. On this node, though, the saved GLM-5.3-Flash disaggregated run had a glaring problem: at concurrency 256, median time to first token was 32.36 seconds for a nominal 1K-token input.
The campaign ended without the matching aggregated baseline. We came back and ran two independent TP4 replicas, one in each NVLink quad, behind the replica proxy. At that same load, aggregate output throughput rose from 3,809 to 7,979 tokens/s and median time to first token fell to 278 milliseconds.
The improvement came with a tradeoff: median time per output token increased from 16.9 to 24.1 milliseconds. Once output started, tokens arrived more slowly under load. That distinction is exactly why a single tokens-per-second headline is inadequate.
One faster headline, three different outcomes.
Nominal 1K input / 1K output · concurrency 256
- Output throughputtokens/s · higher is better
- Saved PD3,809Two TP4 replicas7,979
- First-token latencyp50 milliseconds · lower is better
- Saved PD32,365Two TP4 replicas278
- Time per output tokenp50 milliseconds · lower is better
- Saved PD16.9Two TP4 replicas24.1
More aggregate output. Earlier first tokens. Slower token delivery once output starts.
Compare 1,024 input / 1,024 output · concurrency 1
- Output throughputtokens/s · higher is better
- Saved PD129Two TP4 replicas226
- First-token latencyp50 milliseconds · lower is better
- Saved PD7,444Two TP4 replicas139
- Time per output tokenp50 milliseconds · lower is better
- Saved PD3.6Two TP4 replicas3.5
Compare 1,024 input / 1,024 output · concurrency 64
- Output throughputtokens/s · higher is better
- Saved PD2,374Two TP4 replicas2,982
- First-token latencyp50 milliseconds · lower is better
- Saved PD9,383Two TP4 replicas196
- Time per output tokenp50 milliseconds · lower is better
- Saved PD13.1Two TP4 replicas14.1
Compare 8,192 input / 1,024 output · concurrency 64
- Output throughputtokens/s · higher is better
- Saved PD1,659Two TP4 replicas2,351
- First-token latencyp50 milliseconds · lower is better
- Saved PD15,595Two TP4 replicas619
- Time per output tokenp50 milliseconds · lower is better
- Saved PD14.3Two TP4 replicas21.7
Both configurations used the same GLM checkpoint, vLLM 0.31.0, eight GPUs, BF16 KV cache, MTP depth 3, and an 8,192-token batching budget. They were not otherwise identical: the saved prefill engine ran in eager mode, while the new replicas used CUDA graphs. The new replicas each allowed 512 sequences. We did not run a fresh interleaved A/B test, so this is a historical configuration comparison, not proof that topology alone caused the gain.
All 1,552 measured requests completed across the four new benchmark points, with zero recorded prefix-cache hits. At concurrency 256, the new p99 first-token latency was still 4.66 seconds. A good median can hide a queue that is uncomfortable for some users.
These rescue measurements remain separate from the original campaign’s publication gate. Their correctness checks covered strict JSON, tool calls, and retrieval around 16K input tokens. They do not certify the configured 256K context limit. No energy telemetry was collected for the rescue runs.
The quality harness was part of the bug
Before trusting those GLM numbers, we had to understand a quality failure. The inherited harness sent enable_thinking=false. The local checkpoint’s chat template still opened a thinking block. The request setting and template behavior disagreed, and reasoning text could appear in the final content field.
There was a second problem: the harness searched the response for something that looked like JSON. That could accept an object surrounded by prose, even when an application expected the entire response to be valid JSON.
We kept the checkpoint’s default thinking behavior and changed the check to parse the complete content field with json.loads. The corrected run passed 24 of 24 JSON tasks and eight of eight tool calls. It also passed nine exact retrieval checks near 16K input tokens.
Check the harness before grading the model.
Respect the chat template
The request tried to disable thinking, but the checkpoint still opened a thinking block.
Retest with the checkpoint’s default behavior.Parse the complete response
Finding a JSON-shaped substring can accept an answer that the application will reject.
Validate the whole content withjson.loads.Those are useful smoke tests, with a narrow scope. A separate Fibonacci prompt exhausted a 512-token output budget without returning a final answer. Passing structured-output checks did not establish broad reasoning quality, and an application using thinking mode still needs a suitable output budget.
Our lesson was to save the complete requests and responses. A percentage score was too small an explanation for this failure.
“FP8” was not enough to describe the checkpoint
The other unfinished item was the community DeepSeek-V4.1-Flash FP8 checkpoint. It failed before we could benchmark it. The checkpoint combined FP8 experts with other stored representations, and the engine’s dispatch and scale-loading assumptions did not cover that combination correctly.
The first repair uncovered another loader failure. The final isolated patch addressed linear-format dispatch, dense scale names, and expert scale conversion. One regression check made the risk concrete: a scale that should evaluate to 0.0078125 could instead become 120.0 when encoded exponent bits were treated as a numerical scale.
We patched a pinned vLLM nightly in a separate image and left the downloaded checkpoint intact. Focused regression checks covered the scale handling before the server was brought back up. The working configuration used TP8, FP8 KV cache, eager mode, Triton kernels, and CPU offload for the Engram tables, with a configured 32K context.
It passed 24 JSON tasks, eight tool calls, nine exact retrieval checks near 31K input tokens, and five executions of a generated Fibonacci function. At nominal 1K input / 256 output, it delivered 10.8 output tokens/s at concurrency one and 130.2 aggregate tokens/s at concurrency 16. These request lengths differ from the GLM tests above.
Getting DeepSeek to load was the beginning
At that point, the community FP8 checkpoint worked, but its eager-mode speed was disappointing. We had fixed compatibility without establishing a useful serving configuration. The next tests separated three questions: whether CUDA graphs helped, whether the draft head was healthy, and whether the publisher’s original checkpoint could stay inside one NVLink quad.
Enabling CUDA graphs on the community FP8 TP8 configuration made the nominal 1K-input/256-output, single-request test much faster: 10.8 to 69.1 output tokens/s. At concurrency 16, however, aggregate throughput moved from 130.2 to 119.5 tokens/s. A large single-request gain had not carried over to the loaded server. That was our reason to stop treating one concurrency point as the result.
A second loader had the same scale problem
The first DSpark run booted and passed the small output checks, yet its acceptance length was only about 1.02. Inspecting the draft head found another copy of the scale conversion problem: its expert loader interpreted exponent bytes as numerical scales even when the destination parameter expected floating-point values.
The fix selected the conversion from the destination parameter type: retain the encoded byte representation for a byte parameter, and convert to the numerical scale for a floating-point parameter. A focused regression check covered both destinations. We built another pinned image and excluded the pre-fix draft run from the comparison.
On the community FP8 checkpoint, the repaired DSpark path reached 368 aggregate output tokens/s at concurrency 16, against 119.4 without speculation on the same real-chat prompt set. At concurrency one it was slightly slower, 64.1 versus 68.9 tokens/s. Acceptance had improved, but the benefit still depended on load.
The native checkpoint changed the layout we could test
We then loaded the publisher’s original checkpoint, deepseek-ai/DeepSeek-V4.1-Flash, at revision 2cba9e42aa026125f3ed06c6d98c1db82f7ca027. Its mixture of MXFP4 experts and other tensor formats is described in the vLLM recipe. We used Engram CPU offload and kept tensor parallelism inside GPUs 0-3. The rest of the node was available for a second independent replica.
The native baseline and its DSpark variant used the same four GPUs, 32K configured context, FP8 KV, 32 maximum sequences, an 8,192-token batching budget and disabled prefix caching. The two-replica run used those limits per replica, with each process pinned to its quad’s CPU and memory nodes. All used the same patched vLLM image. DSpark used five draft tokens with adaptive verification disabled; this is one tested setting, not a search over draft lengths or verification policies.
The draft head helps most at concurrency one.
On these real-chat prompts, native TP4 with DSpark delivered 268.8 output tokens/s at concurrency one, versus 115.6 without it: a 2.33× gain. At concurrency 16, the gain was smaller, 901.1 to 1,030.5 aggregate tokens/s. Median time per output token improved from 15.4 to 11.6 ms at that load.
Two native TP4 replicas, without speculation, delivered 1,160.5 aggregate tokens/s at concurrency 16. They used eight GPUs, so the four-GPU DSpark result is especially interesting for the next capacity test. At this stage, we had not run two replicas with DSpark. The matched 256K follow-up below tests that combination separately.
Adding a second quad helped most when the queue grew
The separate capacity sweep used synthetic requests with nominal 1,024 input and 256 output tokens. At concurrency 64, one native TP4 server had only 32 sequence slots. Two replicas provided 64 slots in total as well as twice the GPUs. We wanted that practical deployment comparison, but it cannot isolate the effect of topology from the added compute and serving capacity.
A second replica cuts the wait to start.
Nominal 1K input / 256 output · concurrency 64 · 128 requests · no speculation
- Output throughputtokens/s · higher is better
- One TP4 · 4 GPUs1,427.9Two TP4 · 8 GPUs2,788.0
- First-token latencyp50 ms · lower is better
- One TP4 · 4 GPUs6,317.6Two TP4 · 8 GPUs857.4
- Time per output tokenp50 ms · lower is better
- One TP4 · 4 GPUs20.0Two TP4 · 8 GPUs19.1
Throughput rose from 1,427.9 to 2,788.0 output tokens/s. Median first-token latency fell from 6.32 seconds to 857 ms, while median time per output token was similar: 20.0 versus 19.1 ms. The second replica mostly changed how much work could proceed at once and how long requests waited to start. This is a better account of the experience than the throughput gain alone.
What the first 32K follow-up established
All four configurations in the first 32K queue passed 24 strict JSON tasks, eight tool calls and nine exact retrieval checks near 31K input tokens. All measured benchmark requests completed. These are smoke checks, not evidence that the native checkpoint and the FP8 conversion have equal reasoning quality.
The chat sweep used 40 held-out ShareGPT first-user prompts, split into eight at concurrency one and 32 at concurrency 16. Both comparison configurations saw the same prompts within each concurrency point. Thinking was disabled, temperature was zero, output was capped at 256 tokens and end-of-sequence was honored. Concurrency-one and concurrency-16 prompts were different sets. In that first stage we ran each point once; generated lengths could differ, and we had no confidence intervals or sustained-load results.
That left a concrete next experiment: hold the two-replica allocation fixed, switch DSpark on and off, and extend the context checks. The completed round below answers that question at a 256K configured ceiling. It is separate from the earlier 32K results and from the original million-token campaign.
The practical lesson is narrower than “FP4 wins” or “speculation wins.” The checkpoint representation changed which layout was feasible; a loader defect distorted the draft path; and the workload decided whether a working draft head saved time. Keeping those three questions separate gave us results we could act on.
Two replicas with DSpark: more output, a longer tail
The next run closed the main gap from the 32K experiments: two native TP4 replicas with DSpark against the same two replicas without it. Both used all eight H200 NVL GPUs, one TP4 group per quad, with a 262,144-token configured ceiling. Holding the allocation and sequence limits fixed made this a closer comparison than adding a second quad to a four-GPU baseline.
We kept the native checkpoint revision and patched image above, Engram CPU offload, NUMA pinning, FP8 KV, CUDA graphs, 32 sequence slots per replica and an 8,192-token batching budget. Prefix caching stayed disabled. The speculative arm used DSpark with five draft tokens, probabilistic draft sampling and block rejection, with adaptive verification disabled.
| Workload | Without DSpark | With DSpark | Ratio |
|---|---|---|---|
| Chat, concurrency 1 | 119.1 | 278.3 | 2.34× |
| Chat, concurrency 16 | 1,184.2 | 1,531.4 | 1.29× |
| Chat, concurrency 64 | 2,369.1 | 2,918.2 | 1.23× |
| Sustained chat, concurrency 64 | 2,260.7 | 3,123.3 | 1.38× |
The finite chat rows are medians of three runs per configuration. Each run used eight prompts at concurrency one, 32 at concurrency 16 or 256 at concurrency 64, with identical files and seeds between configurations at each point. Prompt sets differ across concurrency levels. Temperature was zero, thinking was disabled, output was capped at 256 tokens and EOS was honored. These are whole-server output rates, not per-user decode speeds.
For sustained load, each configuration completed three 120-second request-issuing windows at concurrency 64, draining all in-flight requests between windows. The table pools completed output tokens over total elapsed time including drain. Both configurations cycled the same shuffled 256-prompt ShareGPT pool; the faster configuration completed more cycles, so exact prompt frequencies need not match. Across the round, all 2,952 requests in 38 finite benchmark points and all 8,376 sustained requests completed without recorded failures.
More throughput did not improve the slowest whole-request responses. Across the three sustained windows, P99 first-token latency was slightly lower with DSpark: 0.76-0.83 s versus 0.79-0.88 s. But P99 whole-request latency increased to 8.04-8.25 s, from 7.68-7.88 s without it. Those ranges describe each window’s P99, not a pooled percentile. The finite concurrency-64 chat runs did improve median first-token latency, from 466.5 to 235.1 ms, and median time per output token, from 23.90 to 18.11 ms. A throughput target and a whole-request tail-latency target could therefore lead to different deployment choices.
A 256K ceiling, with retrieval checked up to 250K
Both configurations passed 24/24 strict JSON tasks and 8/8 tool calls before and after load. Each also passed all 27 exact retrieval checks: nine near 31K, nine near 126K and nine near 250K input tokens, using three keys at 10%, 50% and 90% depth. Input lengths were measured with the serving model’s chat tokenizer. These targeted checks support the tested output contracts and retrieval cases; they do not certify broad reasoning quality, distribution equivalence or retrieval at the full configured ceiling.
The separate synthetic stress test also exposed a limit to the chat gain. At 250K input tokens and concurrency one, DSpark was slightly slower: 18.7 versus 19.5 output tokens/s, with median first-token latency of 11.52 versus 11.16 s. The concurrency-two result had the same direction, 37.5 versus 39.0 tokens/s. These random-prompt tests forced exactly 256 output tokens with EOS ignored, unlike the chat tests, and had only one pass with two or four requests per point. Their acceptance and throughput should not be generalized to real long-document conversations.
This round gives us a stronger capacity candidate, not a finished reliability claim. The order was fixed, baseline first and DSpark second, rather than a randomized crossover. Three short sustained windows establish repeatability over those windows; they do not establish multi-hour reliability or confidence intervals. At this stage, higher sequence limits, draft-length and adaptive-verification sweeps, mixed long-prefill/short-decode traffic, and 512K/1M contexts were still untested in this matched recipe. The original million-token campaign results still belong to their own configurations.
The 256K follow-up summary JSON includes correctness summaries and sustained-window measurements; the finite-results CSV retains all 38 benchmark points. Run IDs are dsv41_flash__followup_2rep_base__256k__vllm__matched__1011-164014 and dsv41_flash__followup_2rep_dspark__256k__vllm__matched__1011-170401. These downloads contain measurement records, not the prompt corpus.
12 October: the wider tests changed the handoff
Part 14 follows the 1M ceiling, higher-concurrency and draft-policy sweeps, mixed-JSON failures, recovery and engine comparison. Part 15 records the invalid adaptive soak and corrected two-hour tests. The faster K3 configuration completed its load but failed one post-load JSON request; the baseline passed its complete before/load/after sequence. Those later findings qualify the next deployment choice without changing the earlier measurements above.
Returning to GLM on SGLang
The engine comparison later left another unfinished result: GLM-5.3-Flash crashed in SGLang at C64. We reopened it on 12 October. The original configuration failed again, this time during C1, but enabling prefix caching or changing EP1 to EP4 each passed three complete sweeps and all structured-output checks. Part 14 follows the retries, including the unsuccessful overlap and state-pool changes.
The practical EP4 recipe also passed with built-in five-step MTP. It improved single-request output from 107.0 to 208.8 tokens/s, while reducing reported KV capacity from 1,598,848 to 241,344 tokens. That gave us a latency option and a capacity tradeoff, rather than one setting that won everywhere. These 32K-configured short-chat results belong to a separate round from the GLM replica and DeepSeek loader rescues above.
Measurements and reproduction
The DeepSeek follow-up CSV and JSON with correctness summaries retain the completed queue results, configuration identities, run IDs and workload fields. Its full evidence remains in h200-deepseek-experiments-20261011/results/runs/ on the node. The patched image is h200/dsv41-dense-scale-rescue:20261011-v3; its draft-loader regression check and patch are in that experiment folder’s dspark-patch/ directory.
The curated chart-data JSON retains the selected rows, run IDs, units and comparison notes. The GLM comparison CSV contains all four rescue comparison points. These are evidence extracts, not a replacement for raw logs and per-request measurements.
The rescue transcripts and benchmark JSON remain in h200-rescue-20261011/results/ on the campaign node. The isolated DeepSeek loader patch and regression checks remain in h200-rescue-20261011/patch/. The original campaign’s final.csv, invalid_rows.csv and FINAL_SUMMARY.md remain separate.
Upstream references: the vLLM GLM-5.3-Flash recipe and the community DeepSeek-V4.1-Flash FP8 checkpoint. These describe recipes and model files; the measurements above came from this node.