Part 14: Pushing the serving envelope

The 256K follow-up left us with a useful DeepSeek deployment candidate: two TP4 replicas, each contained within an NVLink quad, with DSpark improving selected chat workloads. It also left a longer tail for whole-request latency. The next question was whether that recipe remained useful when we raised context and concurrency, changed draft length, and mixed long documents into short chat traffic.

We extended the configured ceiling to 1,048,576 tokens and swept concurrency 1, 16, 64, 128 and 256. The server still used the native DeepSeek-V4.1-Flash checkpoint, Engram CPU offload, FP8 KV, CUDA graphs and an 8,192-token batching budget, with prefix caching disabled. Two replicas used all eight GPUs. We varied 32 versus 64 sequence slots per replica and DSpark draft lengths K3, K5 and K8, including an adaptive K5 arm. These were configuration comparisons, not independent estimates of each flag’s effect.

A million-token request and a busy chat server are different tests

The original extension’s baseline with 32 slots and K5 with 32 slots each found all nine exact retrieval answers near 510K tokens and all nine near 1M. That is 36 successful checks across two configurations and two lengths. Actual prompt lengths were about 509,993-509,995 and 999,994-999,996 tokens through the serving tokenizer.

Those checks establish a narrow capability: these configurations retrieved the planted facts in the tested requests. They do not prove general long-document reasoning or 128 simultaneous million-token conversations. The high-concurrency sweeps used short chat prompts. Separate synthetic stress requests measured serving behaviour under another input distribution; we did not fold their output rates into the real-chat comparison.

Three separate questions

A larger ceiling does not certify a larger service.

1. 01 · Context36/36 retrieval checks

   Baseline32 and K5/32, near 510K and 1M. Targeted facts in long documents.

2. 02 · Finite traffic83 benchmark points

   27,504 completed requests, zero request failures in that finite-results ledger.

3. 03 · Whole recipeNo initial qualifier

   Mixed-output or engine failures still disqualified the original extension recipes.

Why a zero-failure benchmark ledger still ended in failure

The finite results contained 83 points and 27,504 completed requests, with no recorded request failures. The queue nevertheless failed its wider acceptance checks. Some mixed long-document responses wrapped JSON in Markdown fences; another left a string unfinished. A transport-successful response was not necessarily an object the application could parse.

We retained strict whole-response parsing. Stripping fences to turn those outputs into passes would have changed the advertised contract. The diagnostic question was whether we had configured that contract explicitly enough, and whether the output budget could contain it.

There were also distinct engine failures. At the original 95% GPU-memory allocation, K5/64 hit OOM in the retrieval path. K8/64 failed in the dispatcher; we did not establish that it had the same memory cause. Readiness and an attractive nominal KV allocation did not guarantee enough working memory for execution. We retained the failed attempts rather than using only configurations that happened to reach the throughput grid.

Recovery changed both the memory budget and the request contract

For the 64-slot recovery recipes, we reduced the GPU-memory utilization target to 90%. That is an engine allocation setting, not a measured utilization percentage or a promise of exactly 10% free HBM throughout a request. We also requested constrained JSON for the mixed long-document tasks and raised their output cap to 512 tokens, leaving short chat at 256.

The validator checked exact codes, JSON keys and types, and complete parseable objects. Although the prompt asked for a summary within 60 words, the validator did not enforce that word limit or evaluate summary faithfulness. The passing label below refers to the checks we implemented, not all properties described in the prompt.

Each finite cell is the median of three runs. C means concurrent requests; slots are per replica. The 64-slot recovery arms used the reduced 90% allocation target. The table preserves failed recipes because short-chat performance alone would otherwise make them look eligible. All seven recipes completed their finite grids: 105 points, 38,472 requests, zero request failures in that ledger. Only four passed the complete short-test sequence: baseline32, K3/64, K8/64 and adaptive K5/64.

The remaining failures were not fenced JSON. Baseline64, K5/32 and K5/64 produced verbose, incomplete JSON under the 512-token cap in a mixed long-context request. An unfinished string is not evidence that a retrieved code was wrong; it is evidence that the response contract did not complete. These three configurations remained failed.

Adaptive K5/64 had the highest median C128 throughput among the four eligible candidates, 4,172.6 tokens/s. That selected it for the next reliability attempt, not for a production recommendation. K8/64 passed the short checks but delivered less throughput at that load; a larger draft was not automatically better. The nonadaptive K3/64 result, 4,072.8 tokens/s at C128, left another candidate if the adaptive path did not hold up.

C256 showed saturation. Baseline32 peaked at C64; baseline64 peaked at C128. From C128 to C256, adaptive K5/64 rose only from 4,172.6 to 4,296.2 tokens/s while its median of per-run P99 first-token waits grew from 1.80 to 10.86 s. K3/64 gained little output, 4,072.8 to 4,197.6 tokens/s, while that tail grew from 1.86 to 10.52 s. The extra queue mostly added waiting.

The engine comparison needed its own recovery

We also compared engines on one four-GPU quad at a 32K ceiling, 64 sequence slots and no speculation. First we repaired the harness: a local queue.py shadowed Python’s standard-library module; a probe expected the replica proxy’s /lb_stats route on a single server; and SGLang required its metrics path to be handled correctly. We also removed an unsupported GLM launch flag. These were distinct from a model failing inference. After recovery, Qwen3.8-Flash-Next completed on both engines and GLM-5.3-Flash on vLLM, with 24 JSON and eight tool checks passing before and after each successful arm.

Compare engines within the same model. Qwen ran with thinking disabled; GLM used reasoning, so these rows are not a model ranking. The pinned images and engine versions also differ. Qwen’s SGLang arm led the single-request output rate while vLLM led the C64 and C128 medians here. Three repeats make the observed variation visible without supplying a confidence interval or a universal engine preference.

That original GLM SGLang arm passed its initial structured checks and produced C1/C16 observations, then hit a CUDA illegal-memory failure at C64. We do not promote that partial run to a completed engine comparison. The failure belongs to the tested checkpoint, image and workload; it does not establish that every GLM/SGLang combination is unusable.

We investigated custom engines too. NInfer’s upstream requirements specify RTX 5090 and a build restricted to sm_120a. That is not the H200’s Hopper target. We excluded it from this node’s measured engine table rather than presenting upstream single-GPU claims as a local result.

Returning to the GLM failure

The incomplete row bothered us enough to reopen it. On the evening of 12 October, we returned to the same GLM-5.3-Flash checkpoint and SGLang 0.5.21 image. The first useful result was another failure: the original configuration crashed during the concurrency-one test, after its initial output checks. Our earlier description, “CUDA failure at C64,” identified where that particular run stopped. It had never established a concurrency threshold.

We kept tensor parallelism within one four-GPU NVLink quad, the 32K configured ceiling, BF16 KV, TileLang DSA and DeepGEMM. The original recipe used EP1, disabled prefix caching, an 8,192-token prefill chunk, a 0.85 static-memory fraction and 64 maximum running requests. CUDA graphs and overlap scheduling were enabled. Rather than change every flag and lose the explanation, we first changed one launch option at a time.

Turning graphs off gave us a useful diagnostic, but at a large speed cost. Increasing the state pool did not remove the illegal-memory error, even when the pool was only about half occupied. Disabling overlap also failed. Those outcomes narrowed the search; they did not identify the defective kernel. The successful prefix-cache and EP4 retries each completed all three sweeps, so we could finally replace “we stopped at a crash” with two concrete ways to get this workload through.

The state pool hid a second concurrency limit

The SGLang cookbook pointed us toward keeping prefix caching enabled and treating the hybrid model’s state pool separately from its KV allocation. We combined that direction with EP4, a 4,096-token prefill chunk and a 0.78 static-memory fraction. The H200 configuration also supplied a five-step speculative direction. Our four-GPU layout was an adaptation of the upstream eight-GPU guidance, not a reproduction of its results.

The first cache-enabled EP4 recipe requested 64 active requests and 128 state slots. The startup log quietly made the practical limit 25 active requests: this path reserved five state slots per request. It passed 2,424 requests without a failure, but its C128 median was only 961.7 tokens/s. Client concurrency 128 did not mean 128 requests were decoding, or even that the requested server limit of 64 had been honored.

A speculative launch with 256 state slots was similarly capped at 51 active requests. We stopped it before measurement and recorded a configuration rejection, not an inference failure. With 384 state slots, both practical recipes reported the intended 64 active requests. The earlier 25-request result stays in the evidence, but is not presented as the same serving capacity.

Three sweeps changed the result

Each completed recipe below ran C1, C16, C64 and C128 three times: 2,424 requests with zero load failures per recipe. We also ran 24 JSON and eight tool-call probes before load, after the first sweep and after the final sweep. All 96 probes per recipe passed. These checks establish the structured-output contract we tested, not broad equivalence in reasoning quality.

C is client concurrency; all four rows above allowed at most 64 active server requests. Every row uses four GPUs, FP8 model weights, BF16 KV and a 32,768-token configured ceiling. The first two change one launch switch from the original. The last two share EP4, prefix caching, 384 state slots, a 4,096-token prefill chunk and a 0.78 static-memory fraction; only the speculative settings differ within that pair.

We reused a frozen pool of 256 short-chat prompts, kept reasoning enabled, used temperature zero and capped generation at 256 output tokens with EOS honored. Reported output includes reasoning tokens. The prompt pool repeats, so later cache-enabled sweeps can benefit from reuse. First-sweep averages also include compilation stalls. The downloadable results retain every repeat instead of hiding those starts behind the median.

Speculation bought latency with state memory

The speculative recipe used the model’s built-in MTP path through SGLang’s EAGLE setting: five steps, top-k one and six draft tokens. It did not download or test an external DFlash draft model. CUDA graphs and overlap stayed enabled.

Same practical recipe · speculation off and on

Faster short replies, less room for KV state

C1 output throughputtokens/s · median of three sweeps

No speculation107.0

Five-step MTP208.8

C128 output throughputtokens/s · median of three sweeps

No speculation1,813.6

Five-step MTP1,789.0

Reported KV capacitytoken slots · engine allocation

No speculation1,598,848

Five-step MTP241,344

MTP nearly doubled the single-request output rate in this pair. At C64 it rose from 1,419.3 to 1,886.1 tokens/s; at C128 it was slightly lower, 1,789.0 versus 1,813.6. The no-cache EP4 diagnostic already reached 1,859.9 at C64, so speculation was not necessary to reach that throughput range. The reason to keep MTP on our shortlist was its short-request latency, not a claim that it won at every load.

The memory cost was substantial. The speculative engine allocated about 13.8 GB per GPU for intermediate state in addition to its main state pool. Under the same memory target, reported KV capacity fell from 1,598,848 to 241,344 tokens. Verification graph capture took about 293 seconds, and additional kernel compilation appeared after serving began. A configuration can be faster once warm and still cost more memory and startup time.

What we would carry forward

We would keep EP4 and CUDA graphs in the practical shortlist. The cache-enabled non-speculative recipe offers more KV capacity; five-step MTP is useful when short-chat latency matters more. The original recipe with EP4 alone also passed and retained 3,342,848 reported KV token slots, although its cache policy, prefill chunk and memory fraction differ from that practical pair.

The smallest repair was a launch configuration change, not a checkpoint replacement or an unverified kernel patch. Prefix caching and EP4 were independently successful remedies in this finite test. Their success does not prove which kernel caused the original fault.

The two experiment queues used separate GPU quads and CPU placements but shared the host. Together with the cache-policy and warmup differences, that makes these results configuration observations rather than a new controlled ranking against the earlier vLLM row. This rescue covered short chat at a 32K configured ceiling; it did not extend the original long-context or two-hour evidence.

The GLM SGLang rescue JSON records all ten attempted configurations, effective limits, quality summaries and individual load points. The per-repeat CSV includes successful points from failed runs with their overall failed status intact. Raw requests, exact launch commands and server logs remain in the node archive h200-glm-sglang-rescue-20261012.tar.zst.

The next gate was time

The wider sweeps narrowed our choices but did not finish the job. Hardware monitoring recorded slowdown flags and temperatures up to 84°C in the extension and 85°C during recovery; that is context to retain, not a demonstrated cause of any particular failure. Configurations ran in sequence, and memory headroom, draft policy and workload changed between rounds.

The extended evidence JSON retains configurations, run IDs, checks and failed states. The results CSV provides the concise comparison rows. Raw source directories are h200-extended-20261011 and h200-recovery-20261012 on the node. The next step was to hold selected candidates under load long enough to test a different claim. Part 15 follows the two-hour runs and the check that came after them.