Posts tagged engineering

Ling-3.1-flash: a 560B open model built mostly on linear attention

inclusionAI released Ling-3.1-flash under MIT: 560B parameters, 25B active, 7 of every 8 layers linear attention. What the benchmarks say and what to check.

Part 14 · One node, eight H200s

Part 14: Pushing the serving envelope

Extend DeepSeek to a million-token ceiling, compare draft lengths and engines, and separate fast benchmark rows from usable serving recipes.

Part 15 · One node, eight H200s

Part 15: Two hours and one last check

A corrected load generator, two long DeepSeek runs, and the HTTP 500 that kept the faster configuration from passing the final gate.

Supplementary report

After the campaign: the fixes that changed our next run

We revisit the missing GLM baseline, repair DeepSeek loaders, and test native TP4, DSpark and two replicas while keeping the limits of each result visible.

Series · Start here

Eight H200s: the inference campaign, in 15 parts

Follow our H200 NVL campaign from the first topology check to a million-token conversation: fifteen engineering stories, measured results and failures.

Part 1 · One node, eight H200s

Part 1: Arrival on two H200 NVL islands

Start with the machine: storage, NUMA, NVLink topology, a broken cross-socket bridge, and the IOMMU fix that made it usable.

Part 2 · One node, eight H200s

Part 2: Qwen3.8-27B, from smoke test to the 1M wall

Bring up the first model, check quality before speed, compare FP8 and INT4, and work through the attempts to reach one million tokens.

Part 3 · One node, eight H200s

Part 3: gpt-oss-120b, two kernels and many users

Compare Triton and Marlin on gpt-oss-120b, follow latency as concurrency grows, and keep the failed quality gate attached to the excluded arm.

Part 4 · One node, eight H200s

Part 4: Chasing one million tokens on four GPUs

Move from fitting the context to finding the needle, reading a fresh million-token document, and sustaining decode at depth.

Part 5 · One node, eight H200s

Part 5: The giants: GLM, MiniMax and DeepSeek

Take the largest models across both GPU islands and follow the memory failures, long-prefill costs, and provisional MiniMax results.

Part 6 · One node, eight H200s

Part 6: Parallelism on two islands

Compare TP8, independent replicas and expert parallelism on the actual topology, then investigate the anomaly and the fixes that did not help.

Part 7 · One node, eight H200s

Part 7: Everything that broke, and what fixed it

Follow the node and engine failures, successful retries and unresolved paths, including the DeepSeek launches that stopped before benchmarking.

Part 8 · One node, eight H200s

Part 8: Tuning and splitting prefill from decode

Test a larger prefill budget and a prefill/decode split, keeping the queueing costs and the missing baseline visible in the comparison.

Part 9 · One node, eight H200s

Part 9: Speculative decoding, from one user to sixteen

Measure when drafting helps chat and code workloads, where concurrency changes the result, and which speculative paths never produced valid numbers.

Part 10 · One node, eight H200s

Part 10: Cold starts and the memory below the GPU

Trace weight loading, startup time, host RAM and KV offload, separating a warm page cache from a reproducible cold start.

Part 11 · One node, eight H200s

Part 11: What the tokens cost, and why users set the price

Turn the measured throughput into scenario-based costs, state the rental-price assumption, and separate fresh documents from cached conversations.

Part 12 · One node, eight H200s

Part 12: Nine models, one node, one million tokens

Bring the original campaign together: compare the last turn, the cost of another user, the useful configurations and the unfinished work.

Part 13 · One node, eight H200s

Part 13: Where FP8 checkpoints come from

Inspect tensor headers and scale conventions behind published FP8 names, including the difference between an expert cast and an all-FP8 model.

Supplementary report

Growing one conversation to a million tokens

A supplementary report on cached, growing conversations: per-turn latency, retrieval checks and the difference from a fresh million-token prompt.

DeepSeek opened V4 Flash Vision weights under MIT

Ten days after the API drop, V4 Flash Vision Exp is on Hugging Face under MIT. 305B MoE, 1M context, same token price as text Flash.

AMD skipped ROCm 8 and 9. Here is what 10 ships

ROCm 10.0 jumps from 7.14 to a TheRock-built Core SDK plus ROCm.AI. The 3.3x inference claim is real lab data, not a free upgrade.

The first 3T-class open model is here. What Kimi K3 changes

Moonshot shipped Kimi K3, a 2.8 trillion parameter open-weight model you can actually self-host. Here is what the largest open model yet means for India.

The real cost of LLM inference is the GPU you are not using

Token prices have fallen roughly 600x since 2020, but real inference cost is set by GPU utilization, not the sticker price. Here is the math.

Qwen3.8-27B is the open-source model Indian teams can build on

Qwen3.8 shipped twice under two licences: an Apache 2.0 27B that keeps vision, and a 2.4T flagship under a revenue-share licence. Which can you build on?

DeepSeek V4-Flash-Vision-Exp: multimodal without the vision surcharge

DeepSeek's first vision model in the V4-Flash line matches V4-Flash on text and closes most of the gap to Opus-4.8, with no vision surcharge.