Part 3: gpt-oss-120b, two kernels and many users

Once a model fits, the next constraint may be the kernel rather than memory. gpt-oss-120b fit on a single H200, letting us keep four copies on one quad and compare Triton with Marlin. Marlin improved aggregate output by 36% at 64 short-chat users. Longer prompts narrowed or reversed the advantage, which made this a workload-selection exercise rather than a search for one universally faster backend.

gpt-oss-120b is OpenAI's open-weight mixture-of-experts model, shipped with its expert weights in MXFP4. It fits on one 141 GB H200 (our speculative-decoding tests ran it on a single GPU), which gives a different question from the giants: not can it run but how many copies, and with which kernel. We ran four replicas on one NVLink quad (data parallel, DP4), 128K context, vLLM 0.31.0, and looked at what each user sees as the number of users grows.

The kernel question

H200 is Hopper (SM90). NVIDIA's H200 NVL spec lists no FP4 tensor-core rate [S02], so vLLM has to pick a kernel that unpacks MXFP4 experts on the fly. Three are in play on Hopper:

• Triton (OpenAI's triton_kernels; a vLLM change tuned its split-k on Hopper [S114]): what vLLM picked by default for us (Using OAITritonExperts in the server log). The vLLM gpt-oss recipe documents the Hopper path [S87].
• Marlin: a weight-only mixed-precision kernel. A vLLM pull request benchmarked Marlin against Triton for gpt-oss on H200 [S113]. We forced it with --moe-backend marlin (Using MarlinExperts in the log).
• FlashInfer CUTLASS: what NVIDIA Dynamo's gpt-oss-120b H200 recipe uses [S112].

We launched Triton on quad A and Marlin on quad B at the same time, using the same vLLM image, model files and prompts. That reduced differences in traffic and software, but did not remove the quad/NUMA confound. A same-quad repeat would have made the comparison stronger; it was planned and dropped for time. The measurements below therefore describe the two deployed arms, with backend choice as the principal intended variable.

Medians, one run per cell, our measurement. Both servers passed the needle check at 128K (3/3) and had the same KV pool: about 2.61M tokens across the four replicas.

What changes with the number of users

One user: Marlin reduced token latency from 9.6 to 5.2 ms, roughly 104 to 190 tokens/s. Small-batch execution is where kernel overhead and weight movement can matter disproportionately, but those are possible mechanisms here, not a profile of this run. The direct finding is the latency difference in the deployed arms.

64 users, short prompts. The server's total went from 4,379 to 5,951 tokens/s (+36%), and each of the 64 users still got a token every 10.1 ms instead of 13.6. That's 16 users per replica.

At 64 users with 8K prompts, the aggregate advantage shrank to 4%: 3,413 versus 3,268 tokens/s. The longer requests include more prompt-processing work, so an improvement in one MoE kernel need not translate proportionally into end-to-end throughput. Expert layers also execute during prefill; we did not profile the kernels closely enough to identify which component absorbed the gain. The first-token results, 2.66 s for Marlin versus 1.68 s for Triton, are a reason to keep that distinction visible.

Long documents (126K tokens). With one user, Marlin still decoded faster (6.2 vs 10.6 ms per token) but read the document slightly slower (7.7 s vs 7.1 s). With 8 users, Triton came out ahead on total output (130 vs 121 tokens/s) and on first token (15.7 s vs 19.2 s). At this length the work is mostly prefill, and the MoE kernel stops being the bottleneck.

This changed the decision from “force the fastest kernel” to “choose a kernel for the service mix.” Marlin was the stronger candidate for the measured short-chat traffic. With long documents and eight users, Triton led aggregate throughput despite Marlin’s faster single-user decode. The table is useful because it keeps prompt latency, token latency and server output separate; collapsing them into one speed-up would have hidden the reversal.

What didn't run

• CPU KV offload for gpt-oss (vLLM's native CPU offload [S57]): two launch failures and no result. The first failed in the Rust frontend's tool parser on this model; we set VLLM_USE_RUST_FRONTEND=0 for gpt-oss. The second ran the host out of memory: each of the four replicas pins its own CPU KV buffer, and 4 × 400 GiB plus a stale 400 GiB shared-memory file left by an earlier run (and a leftover LMCache container) was more than the node had. Details are in Part 7. A resized rerun (96 GiB per replica) was prepared and dropped when the deadline moved.
• An FP8 community checkpoint of gpt-oss-120b failed to load with a KeyError. No numbers.
• EAGLE-3 speculative decoding ran on one GPU, but those servers had no needle check of their own and inherited the FlashInfer server's 2/3 under our rule, so those rows are unpublishable too. Part 9 explains.

From a fast stream to a long wait

The kernel comparison showed that short-chat gains could disappear once prompt work dominated the request. Part 4 makes prompt length the main experiment: a near-million-token input on one quad, followed by separate checks of retrieval, first-token wait and decoding under load.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S02] NVIDIA, H200 product page (H200 NVL column)
2. [S114] vLLM project (xyang16), https://github.com/vllm-project/vllm/pull/30528
3. [S87] vLLM project (recipes), https://recipes.vllm.ai/openai/gpt-oss-120b
4. [S113] vLLM project (mmangkad et al.), https://github.com/vllm-project/vllm/pull/27523
5. [S112] NVIDIA Dynamo, https://docs.nvidia.com/dynamo/dev/recipes/gpt-oss-120b
6. [S57] vLLM project (GitHub), https://github.com/vllm-project/vllm/pull/37160