Part 13: Where FP8 checkpoints come from

The last question came from files that should have made deployment simpler. We had downloaded community checkpoints labeled FP8 to try different Hopper execution paths. Inspecting them showed that the label could mean an expert-weight cast, a different block-scaling convention, or a checkpoint whose large expert tensors were still BF16. Before explaining a speed difference, we needed to know what the engine had actually loaded.

The hardware label did not choose the kernel

H200 is a Hopper GPU. NVIDIA's spec table for the H200 NVL lists FP8 tensor-core throughput (3,341 TFLOPS with sparsity) and no FP4 rate at all [S02]. DeepSeek ships V4-Flash-0731 with its expert weights in a 4-bit format, and OpenAI ships gpt-oss in MXFP4. Our recorded vLLM DeepSeek-V4-Pro path served those expert weights through a weight-only W4A16 kernel. That does not describe every engine: the SGLang W4A8 comparison later in this chapter uses a different path. The vLLM pull request recorded during the campaign describes the available DeepSeek-V4 dispatch in that implementation: the FP4-by-FP8 MoE backends are gated to Blackwell, and "on SM90 only Marlin W4A16 passes" [S127]. Our DeepSeek-V4-Pro launch picked exactly that kernel:

$ grep -m1 "Mxfp4 MoE backend" results/runs/dsv4_pro__dep8__1024k__vllm__1010-053422/server.log
(Worker_DP0_EP0 pid=4695) INFO 10-10 05:37:04 [mxfp4.py:721] Using 'MARLIN' Mxfp4 MoE backend.

We wanted to test whether an available FP8 representation enabled a more useful kernel on Hopper. Storage precision alone could not answer that question. We planned to convert checkpoints ourselves, then cancelled the on-node conversion at 02:17 IST to keep GPU time for benchmarks. That left published FP8 checkpoints. "Published FP8" turned out to cover very different files.

Read the headers before allocating another run

A safetensors file starts with a JSON header that lists every tensor's dtype and shape. We can read that metadata without loading the tensor payload or allocating GPU memory. For each checkpoint we summed bytes by storage type and by role (expert weights vs everything else):

# what we ran on the node for each repo (simplified)
import json, struct, glob
for f in glob.glob(f"{repo}/*.safetensors"):
    with open(f, "rb") as fh:
        n = struct.unpack("<Q", fh.read(8))[0]
        hdr = json.loads(fh.read(n))
    for name, t in hdr.items():
        if name != "__metadata__":
            add(dtype=t["dtype"], role="expert" if ".experts." in name else "other", count=prod(t["shape"]))

Three things stand out.

1. The DeepSeek cast stores FP8 experts, at 84% more disk space. DeepSeek's original stores 296.4 billion expert values as 4-bit numbers packed two per byte, with one 8-bit power-of-two scale (E8M0) per 32 values. That's the shared-scale layout the OCP Microscaling spec defines for MXFP4 [S48]. AtlasCloud's checkpoint [S58] holds the same 296.35 billion expert values as one-byte FP8 E4M3 with block scales. So 167 GB became 307 GB. Casting to a wider storage format cannot restore information lost in the original quantization. The same vLLM PR describes its own bit-exact re-encoding path and reports 1.38-1.53× faster prefill than Marlin on H20 for twice the expert memory [S127]. We haven't verified bit-exactness of the AtlasCloud file ourselves. What it buys is a format Hopper computes natively.

2. The scale layout is part of the format. MiniMax's own MXFP8 release uses FP8 E4M3 elements with an 8-bit scale shared by each block of 32 values (scale tensors shaped [rows, cols/32]), again the OCP MX layout [S48]. The FP8 checkpoints from Qwen, Z.ai and DeepSeek use one scale per 128×128 block (weight_block_size: [128, 128] in their configs). Same element type, different scale granularity, and on Hopper a different execution path. The block-FP8 path we compared used 128×128 blocks with FP32 scales; that is not a definition of every FP8 kernel. For MXFP8 on Hopper, "automatic MXFP8 linear selection currently uses Marlin W8A16" vLLM PR #59866 (status recorded during the campaign), so the weights stay 8-bit in memory and the math runs in BF16. Our MiniMax-M3 log shows exactly that:

$ grep -hE "MXFP8 GEMM|MxFp8 MoE backend" results/runs/minimax_m3__fx_tep8_recipe__1024k__vllm__1010-081133/server.log
INFO 10-10 08:13:44 [__init__.py:929] Using HummingMxfp8LinearKernel for MXFP8 GEMM
INFO 10-10 08:13:44 [mxfp8.py:137] Using 'MARLIN' MxFp8 MoE backend.

So we label MiniMax-M3 MXFP8 (W8A16 on Hopper), never just "FP8".

3. A "FP8" label can cover a BF16 file. TevunahAi's gpt-oss-120b FP8 [S67] has a quantization config that targets Linear layers. In its headers, only the attention projections (0.96B params) are FP8. All 114.7 billion expert parameters are stored as BF16. Our reading is that gpt-oss keeps its experts as fused 3-D tensors rather than separate Linear layers, so a recipe that targets Linear passes over them. The result is 232.7 GB, 3.6× the original MXFP4 release (65.3 GB). vLLM also refused to load it (KeyError: 'layers.0.mlp.experts.w13_bias'), so we never served it.

Missing checkpoint scales did not tell us every runtime value

FP8 KV cache works best with calibrated scale factors for keys and values, which vLLM's FP8 documentation describes loading from the checkpoint [S54]. vLLM's own write-up on FP8 KV covers what happens without them [S115]. We checked the weight index (model.safetensors.index.json) of all 22 checkpoints on the node that have one. None ships k_scale or v_scale tensors. That establishes the absence of those named tensors in the inspected indices, not the value of every runtime KV scale. An engine can use architecture-specific handling or compute scales elsewhere. The MiniMax log below explicitly reports uncalibrated q_scale and/or prob_scale values of 1.0; those names should not be silently substituted for key/value scales:

$ grep -m1 -o "Using uncalibrated.*" results/runs/minimax_m3__fx_tep8_recipe__1024k__vllm__1010-081133/server.log
Using uncalibrated q_scale 1.0 and/or prob_scale 1.0 with fp8 attention
$ ls /mnt/data/models/*/model.safetensors.index.json | wc -l
22
$ grep -l -E "k_scale|v_scale" /mnt/data/models/*/model.safetensors.index.json | wc -l
0

We can't yet say what that costs in quality. The needle tests passed with FP8 KV for GLM-5.3, Qwen3.6, Qwen3.8-27B and both DeepSeek builds. DeepSeek-V4-Pro (TEP8) found 3/3 at ~799K. MiniMax-M3 scored 2/3 on one-shot 1M prompts but found the needle on all ten turns of its ramp to 1M (see Part 5). With BF16 KV it found 3/3 (table below).

Serving them: the original campaign results

Both DeepSeek-V4-Flash FP8 builds ran on one quad with SGLang 0.5.21 and FP8 KV. They're different model versions (V4-Flash vs V4-Flash-0731), so this isn't a precision comparison. It does show that both casts serve a 1M-token prompt and find the needle:

DeepSeek-V4-Flash-0731, FP4 vs FP8 (finished 09:01 IST, Sun 11 Oct). This is the same model in two precisions: the original FP4-expert release on SGLang 0.5.21's W4A8 path [S125] against the AtlasCloud FP8 cast, both TP4 on one quad with FP8 KV, the same 1M and 250K prompts and the same needle test. GSM8K covers all three arms, including a vLLM 0.31 run of the FP4 release as a reference:

What this does and doesn't show. The FP8 cast reached the first token about 11% faster at 1M and 21% faster at 250K, and wrote about 7-8% faster per token with one user. That fits FP8 tensor cores doing the matmuls that W4A8 has to dequantise first, but we didn't profile it, so treat that as a hypothesis. The quality comparison is less conclusive. The two FP4 runs scored 0.950 and 0.948 on 1,319 GSM8K questions; the FP8 run scored 0.960 on only 250. Unequal samples and no paired answer analysis do not establish a tie or prove no accuracy loss. These runs support a latency comparison under the listed settings, not a general quality ranking. Each performance point used one prompt; the checkpoint-scale inspection above also does not certify every runtime scale. Caveat: the FP8 1M point is from the first run (dsv4_flash_0731_fp8__tp4_a__1024k__sglang__1010-040022, before per-point seeds were added, but it was point 0 so the earlier repeated-seed issue does not by itself establish a cache hit at this first point). The 250K FP8 points are from the clean rerun dsv4_flash_0731_fp8__tp4_a__1024k__sglang__1011-012409. W4A8: dsv4_flash_0731__fx_sgl_w4a8_a__1024k__sglang__1010-225510. vLLM FP4: dsv4_flash_0731__tp4_b__1024k__vllm__1010-235127.

The DSpark speculative-decoding runs for this model (the r/LocalLLaMA SGLang DSpark + HiCache recipe on lmsysorg/sglang:dev-dspark) were not run: the FP8 checkpoint failed to boot on that image during DeepGEMM warm-up (AttributeError: 'Fp8MoEMethod' object has no attribute 'runner', run dsv4_flash_0731_fp8__fx_dspark_b__1024k__sglang__1011-032530), and there was no time left for an engine fix.

Qwen3.8-27B: BF16 KV vs FP8 KV (finished Sun 11 Oct). Same weights, same image (vLLM 0.31.0), four replicas on quad A at 256K context; the only change is --kv-cache-dtype auto (BF16) instead of fp8. The two runs were a day apart. Cells show first token / per output token unless marked:

Medians, one run per cell, our measurement. Both servers found the 256K needle 3/3. Runs qwen38_27b__dp4_a__256k__vllm__1011-005426 and qwen38_27b__dp4_a__256k__vllm__1009-230337.

FP8 KV held 1.96× the tokens, as expected from half the bytes per value. But BF16 KV was faster wherever the context was long: a 250K document's first token came in 34 s instead of 45 s, and with 16 users on 250K documents each token took 184 instead of 234 ms. With short prompts the two were within a few percent. A vLLM issue reports FP8 KV running slower than BF16 on Hopper for 256-wide attention heads [S116], and an SGLang user compared FP8 and BF16 for this model on one H200 [S117]. We didn't profile why it was slower here. For Qwen3.8-27B, these measurements suggest a capacity-versus-latency tradeoff: FP8 exposed a larger pool, while BF16 delivered lower latency on these long-context points. Because the runs were a day apart, we would confirm that choice with a controlled repeat before treating the difference as a stable property of the dtype.

GLM-5.3-Flash: FP8 KV vs BF16 KV (finished Sun 11 Oct). Both on the vLLM nightly 1a001d58 (which has the FP8 KV path for its sparse attention [S122]), TP4 at 1M context. Caveat: FP8 KV ran on quad A and BF16 KV on quad B, so the comparison does not isolate KV precision from quad placement. We did not measure how much of the difference placement contributes.

Medians, one run per cell, our measurement. Both servers found the 1M needle 3/3. Runs glm53_flash__fx_tp4_a_ngt__1024k__vllm__1011-000540 and glm53_flash__fx_tp4_b_ngt__1024k__vllm__1010-230613.

Same pattern, stronger. FP8 KV held 1.92× the tokens but was slower in nearly every cell: one 1M prompt took 79 s to first token against 62 s, and the gap grew with users. With 16 users on 250K documents the median first token was 40 s on FP8 KV against 12 s on BF16. For these measured loads, BF16 KV was the more promising starting point. Its roughly 4.4M-token pool is a nominal capacity estimate, not a guaranteed service threshold. A same-quad repeat and a workload near the usable pool would be needed to locate a crossover; we did not measure one. The missing checkpoint scales leave a calibration question that the weight-index inspection alone does not resolve; the engine’s scale behavior must be checked per configuration [S115].

MiniMax-M3 precision pairs (finished 02:18 IST, Sun 11 Oct). Same 8-GPU TP8 + EP8 layout and nightly image, at 1M context. The MXFP8 release (W8A16 on Hopper) ran with FP8 KV and with BF16 KV, and a W8A8 FP8-dynamic upload [S119] ran with FP8 KV:

What this does and doesn't show. FP8 KV held about 1.68× the tokens of BF16 KV on the same weights, which is in line with the recipe's "about 1.5×" claim and slightly above it. Decode speed per token was essentially the same in all three. The FP8-dynamic build reached the first token about 14% faster at 1M with one user, with one prompt per point. Prefill using FP8 tensor cores in W8A8 is a possible reason, but we haven't measured it, so treat it as a hypothesis. The GSM8K scores were 0.936 and 0.916 on 250 questions. The runs changed both weights and KV precision, and we did not perform a paired significance analysis, so these scores do not isolate a quantization effect. The log and checkpoint inspection establish the scale observations described above, not a universal runtime KV-scale value. Sources: results/runs/minimax_m3__fx_tep8_recipe__1024k__vllm__1010-192440 and results/runs/minimax_m3_fp8dyn__fx_tep8_recipe_fp8__1024k__vllm__1010-200919.

The label was where the investigation began

We finished the campaign with a more useful checklist than “download the FP8 version”: identify the tensor roles, inspect the scales, confirm the actual kernel, and test the output before timing it. The headers could rule out an assumption cheaply; only a working engine and measured workload could establish a serving result.

That lesson became concrete in the afterword. A later DeepSeek-V4.1 recovery exposed scale conversions in both the main and draft loaders, then led to successful native TP4 and DSpark tests. Those results concern a different model version and a 32K configuration. They do not retroactively validate the failed V4 SGLang DSpark launch recorded here.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S02] NVIDIA, H200 product page (H200 NVL column)
2. [S127] vLLM project, https://github.com/vllm-project/vllm/pull/53709
3. [S48] Open Compute Project (AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm), OCP Microscaling (MX) Formats Specification v1.0
4. [S58] Hugging Face model card (publisher = repo owner), https://huggingface.co/AtlasCloud/DeepSeek-V4-Flash-0731-FP8-DSpark
5. [S67] Hugging Face model card (publisher = repo owner), https://huggingface.co/TevunahAi/gpt-oss-120b-1024-Calibration-FP8
6. [S54] vLLM project (docs), https://docs.vllm.ai/en/stable/features/quantization/fp8/
7. [S115] vLLM blog, https://vllm.ai/blog/2026-04-22-fp8-kvcache
8. [S125] SGLang project (yuan-luo), https://github.com/sgl-project/sglang/pull/34967
9. [S116] vLLM project, https://github.com/vllm-project/vllm/issues/48786
10. [S117] SGLang project (dhruvrnaik), https://github.com/sgl-project/sglang/issues/41938
11. [S122] vLLM project, https://github.com/vllm-project/vllm/pull/55222
12. [S119] EmbeddedLLM, https://huggingface.co/EmbeddedLLM/MiniMax-M3-FP8-dynamic
13. liuyao0322, https://github.com/vllm-project/vllm/pull/59866 (vLLM PR #59866 (status recorded during the campaign))