Blog
Posts tagged inference
18 posts tagged "inference".
Part 14 · One node, eight H200s
Part 14: Pushing the serving envelope
Extend DeepSeek to a million-token ceiling, compare draft lengths and engines, and separate fast benchmark rows from usable serving recipes.
ReadPart 15 · One node, eight H200s
Part 15: Two hours and one last check
A corrected load generator, two long DeepSeek runs, and the HTTP 500 that kept the faster configuration from passing the final gate.
ReadSupplementary report
After the campaign: the fixes that changed our next run
We revisit the missing GLM baseline, repair DeepSeek loaders, and test native TP4, DSpark and two replicas while keeping the limits of each result visible.
ReadSeries · Start here
Eight H200s: the inference campaign, in 15 parts
Follow our H200 NVL campaign from the first topology check to a million-token conversation: fifteen engineering stories, measured results and failures.
Explore the seriesPart 1 · One node, eight H200s
Part 1: Arrival on two H200 NVL islands
Start with the machine: storage, NUMA, NVLink topology, a broken cross-socket bridge, and the IOMMU fix that made it usable.
ReadPart 2 · One node, eight H200s
Part 2: Qwen3.8-27B, from smoke test to the 1M wall
Bring up the first model, check quality before speed, compare FP8 and INT4, and work through the attempts to reach one million tokens.
ReadPart 3 · One node, eight H200s
Part 3: gpt-oss-120b, two kernels and many users
Compare Triton and Marlin on gpt-oss-120b, follow latency as concurrency grows, and keep the failed quality gate attached to the excluded arm.
ReadPart 4 · One node, eight H200s
Part 4: Chasing one million tokens on four GPUs
Move from fitting the context to finding the needle, reading a fresh million-token document, and sustaining decode at depth.
ReadPart 5 · One node, eight H200s
Part 5: The giants: GLM, MiniMax and DeepSeek
Take the largest models across both GPU islands and follow the memory failures, long-prefill costs, and provisional MiniMax results.
ReadPart 6 · One node, eight H200s
Part 6: Parallelism on two islands
Compare TP8, independent replicas and expert parallelism on the actual topology, then investigate the anomaly and the fixes that did not help.
ReadPart 7 · One node, eight H200s
Part 7: Everything that broke, and what fixed it
Follow the node and engine failures, successful retries and unresolved paths, including the DeepSeek launches that stopped before benchmarking.
ReadPart 8 · One node, eight H200s
Part 8: Tuning and splitting prefill from decode
Test a larger prefill budget and a prefill/decode split, keeping the queueing costs and the missing baseline visible in the comparison.
ReadPart 9 · One node, eight H200s
Part 9: Speculative decoding, from one user to sixteen
Measure when drafting helps chat and code workloads, where concurrency changes the result, and which speculative paths never produced valid numbers.
ReadPart 10 · One node, eight H200s
Part 10: Cold starts and the memory below the GPU
Trace weight loading, startup time, host RAM and KV offload, separating a warm page cache from a reproducible cold start.
ReadPart 11 · One node, eight H200s
Part 11: What the tokens cost, and why users set the price
Turn the measured throughput into scenario-based costs, state the rental-price assumption, and separate fresh documents from cached conversations.
ReadPart 12 · One node, eight H200s
Part 12: Nine models, one node, one million tokens
Bring the original campaign together: compare the last turn, the cost of another user, the useful configurations and the unfinished work.
ReadPart 13 · One node, eight H200s
Part 13: Where FP8 checkpoints come from
Inspect tensor headers and scale conventions behind published FP8 names, including the difference between an expert cast and an all-FP8 model.
ReadSupplementary report
Growing one conversation to a million tokens
A supplementary report on cached, growing conversations: per-turn latency, retrieval checks and the difference from a fresh million-token prompt.
Read