Part 6: Parallelism on two islands

After making the giant models fit, we returned to models that did not need all eight GPUs for one copy. Would the second quad make one request faster, or would it be more useful serving another request? We compared cross-socket tensor parallelism, independent replicas and MoE expert parallelism. The answer depended on both the model and the user count, and one apparent hardware limit turned out to be our proxy.

Qwen3.8-27B · FP8 weights · FP8 KV Qwen3.6-35B-A3B · FP8 weights · FP8 KV vLLM 0.31.0 · 8× H200 NVL

The question

We first ran a common 256K-configured grid on three eight-GPU deployments, using vLLM 0.31.0, FP8 weights and FP8 KV. Two layouts used dense Qwen3.8-27B; the DEP8 layout used MoE Qwen3.6-35B-A3B. That first comparison asks what those complete deployments deliver. It cannot attribute every difference to parallelism because the model changes too. The later same-model tables are the stronger evidence for choosing a layout.

• 8 replicas (TP1 × 8): Qwen3.8-27B, one independent vLLM server per GPU, each pinned to its own socket's CPUs, with a small round-robin proxy in front.
• TP8: Qwen3.8-27B as one engine with every layer split across all eight GPUs (--tensor-parallel-size 8), with NCCL_P2P_LEVEL=SYS so NCCL uses direct GPU-to-GPU traffic between the sockets [S24].
• DEP8: Qwen3.6-35B-A3B, a mixture-of-experts model, with --data-parallel-size 8 --enable-expert-parallel. Attention runs as eight data-parallel copies, and the experts are split across all eight GPUs [S05] [S04].

The grid: 1,024-token prompts with 1,024-token answers at 1, 64 and 256 users at once, 8,192-token prompts at 64 users, and for TP8 an extra 16-user point. DEP8 runs a different model (a 35B MoE with about 3B active parameters, against a 27B dense model), so TP8 vs replicas is the like-for-like comparison, and DEP8 shows what the MoE layout does on the same node.

The node: two islands

This matters more than anything else here. The eight GPUs are two groups of four. Inside a group, every pair has six bonded NVLinks (NV6). Between the groups there is no NVLink: traffic crosses PCIe and the link between the two CPU sockets (SYS) (Part 1). NVIDIA's guidance for bridged H200 NVL cards is to keep each NVLink group within one CPU domain [S01], and this server does that, so any layout that spans all eight GPUs has to cross sockets.

Cross-island GPU-to-GPU traffic didn't even work properly at first. With the Intel IOMMU in translated mode, direct writes between the sockets failed, and NCCL quietly fell back to host shared memory. We switched to IOMMU passthrough (intel_iommu=on iommu=pt) and rebooted, following NCCL's troubleshooting guidance. After that, cross-socket copies ran at 38.5 GB/s and an 8-GPU all_reduce with NCCL_P2P_LEVEL=SYS reached 83.1 GB/s of bus bandwidth, against 337 GB/s inside one quad (our measurements, Part 1 has the details and logs). The TP8 run below depends on that fix.

Results: users vs throughput and speed

All numbers are our measurement.

• Eight copies beat one engine split eight ways, everywhere. At 64 users the 8 replicas produced 3,912 tokens/s at 13.3 ms per token, and TP8 634 at 25.2 ms. At 256 users it was at least 6,388 against 2,565 (the replica point is proxy-limited, see below).
• TP8 didn't even help a single user. One request alone took 13.2 ms per token on TP8 and 11.5 ms on one GPU of the replica layout. Every layer of a TP8 model ends in an all_reduce across all eight GPUs, so every token waits on the cross-socket link, which carries about a quarter of the in-quad bandwidth by our nccl-tests numbers. We think that's why, but we didn't profile it.
• TP8 decode was spiky. At 64 users its median was 25.2 ms per token, but the p99 was 273 ms (the replicas' p99 was 13.9 ms).
• DEP8 was the fastest single user and the biggest total, at 9.9 ms per token alone and 7,582 tokens/s at 256 users (31.3 ms per token each). That's a smaller-active-parameter model, so it's not a verdict on DEP vs replicas. The vLLM docs describe this layout for MoE models: attention replicated per data-parallel rank, experts sharded across all ranks [S05].
• The replicas' 256-user point is limited by our own proxy, so it's a lower bound. Median first token was 117 ms, but the p90 was 131 s, and the engines' counters, summed over all eight, never showed more than 102 requests running while 256 were in flight. Our proxy opens its upstream connections through aiohttp's default connector, which allows 100 simultaneous connections aiohttp docs: TCPConnector, limit defaults to 100. So roughly 150 requests waited in the proxy, not in vLLM. The 64-user points are under that limit and unaffected. A fair 256-user replica number needs a rerun with that limit lifted.

Longer prompts: 8K in, 1K out, 64 users

With 8K-token prompts the replicas produced 3,001 tokens/s (17.5 ms per token) and DEP8 2,445 (23.6 ms). The TP8 point is left out: its engine counters show a 7.3% prefix-cache hit rate in a grid meant to start cold, so it fails our cache rule (cold_cache_hit in final.csv).

The 16-user anomaly

We added a 16-user point to the TP8 run because 8-GPU runs elsewhere in the campaign had shown 200-290 ms per token at batch 16. It reproduced: 182.5 ms per token and 83.5 tokens/s for the whole node. Sixteen users together got barely more than one user alone (75.0 tokens/s), and far less than 64 users (634 tokens/s). It wasn't a few bad steps. The median gap between tokens was 183.7 ms and the p99 193.8 ms, so every step was slow. No errors in the server log, all 48/48 requests completed, and the median first token was 532 ms (our measurement).

WARNING [symm_mem.py:107] SymmMemCommunicator: symmetric memory multicast operations are not supported.
WARNING [custom_all_reduce.py:265] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs.
WARNING [flashinfer_all_reduce.py:108] Failed to initialize FlashInfer All Reduce workspace: [SymmDeviceMemory] Device
        does not support multicasting.. This is expected on GPUs without NVSwitch (e.g., NVLink bridge-only or PCIe topologies).
WARNING [flashinfer_all_reduce.py:216] FlashInfer mnnvl allreduce workspace unavailable (likely no NVSwitch multicast
        support); falling back to trtllm backend for single node.
INFO    [flashinfer_all_reduce.py:224] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
...  'pass_config': {... 'fuse_allreduce_rms': True, ...}

The suspected cause. By default, vLLM fuses each layer's all_reduce with the RMSNorm that follows it (fuse_allreduce_rms). Its preferred fused kernel needs NVLink multicast across every GPU in the tensor-parallel group, which only an NVSwitch provides. Our log shows exactly that check failing, followed by a fallback to the trtllm fused backend. A public bug report on the same hardware class (8× H200 NVL, two bridged NVLink islands, TP8) describes this fallback as working but very slow: 719.6 ms median inter-token latency at 32 concurrent requests with fusion on, against 20.7 ms with fuse_allreduce_rms turned off [S96]. The proposed fix in vLLM PR #48075 checks for multicast and, if it's missing, turns the fusion off and uses plain NCCL plus a separate RMSNorm. The PR's own H200 NVL numbers: 733 ms with the forced trtllm path, 21 ms with the fix. As of this build the PR is still open.

The upstream report gave us a testable hypothesis, but our pattern did not fully match it. TP8 was slow at 16 users, near-normal at one user (13.2 ms/token), and mostly fast but spiky at 64 and 256. We therefore treated the fused all-reduce fallback as a suspect and ran two interventions. The completed A/B below did not reproduce the upstream improvement: neither disabling the fusion nor disabling FlashInfer all-reduce removed our cliff.

The fusion-off A/B: neither fix helped

Both interventions reproduced the 16-user cliff: about 192 ms/token versus 182 ms in the baseline, with little movement at one or 64 users. They failed our proposed pass rule of at least a 3× improvement at 16 users, so we did not carry the workaround into the giant-model launches. This weakens the specific fusion-fallback hypothesis for our configuration; it does not prove that every possible interaction with that path is irrelevant. The unresolved next step was a profile of the TP8 16-user cell.

Four GPUs or eight? Same users, twice the hardware

The question a buyer actually asks: if the users don't change, what do the other four GPUs buy? We ran the same 1K/1K user steps (1, 64 and 256 users at once) and the 8K-prompt step at 64 users on one quad and on the whole node. The 4-GPU layouts ran on Saturday 10 Oct; the 8-GPU ones on Saturday evening and, for the layouts marked new below, on Sunday 11 Oct. All numbers are our own measurements.

In vLLM, a MoE model with --enable-expert-parallel spreads its experts over TP × DP GPUs, so TEP4 × DP2 means four-way tensor parallel attention, two data-parallel ranks and experts split eight ways vLLM docs: expert parallel deployment. Our TP4 × DP2 for the dense Qwen3.8-27B keeps each TP4 engine inside one NVLink quad, so no tensor traffic crosses the sockets.

The extra quad paid when the workload could use it. At 256 users, Qwen3.8-27B replicas produced 11,268 tokens/s on eight GPUs versus 5,314 on four (2.1×). TP2 pairs produced 12,328 versus 5,897 (2.1×). At 64 users the replica gain was only 1.1×, 3,885 versus 3,531. One request still uses one copy, so simply adding copies is not a single-request acceleration technique.

For Qwen3.6-35B-A3B, DEP8 produced less than DEP4 at 256 users: 7,582 versus 9,228 tokens/s. Cross-socket expert communication is a plausible contributor, but we did not profile it. Eight independent one-GPU copies reached 18,391 tokens/s, 2.2× four copies. TEP4×DP2 reached 6,646, or 0.88× DEP8. On the same eight GPUs the independent copies therefore delivered 2.8× TEP4×DP2 and 2.4× DEP8 at 256 users, and also led at one and 64 users. That is the direct same-model evidence for avoiding unnecessary distributed experts in this short-prompt grid.

Qwen3.8-27B made the choice less absolute. Two quad-local TP4 engines delivered 11,439 tokens/s at 256 users (1.02× eight one-GPU copies) and 6.3 ms/token for one user. This was the strongest measured Qwen3.8-27B layout at 64 users, including 8K prompts: 3,426 tokens/s with a 344 ms first token. Keeping each tensor-parallel group within a quad was materially better than our cross-socket TP8 run. Qwen3.6 TEP4×DP2 has no valid 8K point because prefix-cache hits contaminated that run.

In our main grid, Qwen3.6's dp4 layouts are labelled data-parallel attention + TP4 experts (no EP). Without --enable-expert-parallel, vLLM forms the expert layers into one tensor-parallel group across the data-parallel ranks [S04]. The dense Qwen3.8-27B's dp4 is four plain data-parallel copies.

Replica size and precision (Sunday 02:26 to 04:08 IST)

Same Qwen3.8-27B, 256K maximum context, vLLM 0.31.0, all eight GPUs in use, with a round-robin proxy in front. We compared one copy per GPU (8 × TP1) against one copy per GPU pair (4 × TP2), each in FP8 and in an INT4 build, plus ThinkingCap-Qwen3.8-27B (a fine-tune of the same base, FP8) as 8 × TP1. Each run had a fresh server, distinct warm-up prompts, a needle gate (all passed 1.0), the 1K/1K user grid, a 250K-token document at 1 and 4 users, and 250 GSM8K questions.

What it shows. At 256 short-chat users the two layouts were close (about 11.3K vs 12.3K output tokens/s in FP8). Pairs gave the fastest single user and the shortest time to first token on a 250K-token document (pairs split the prefill), and a bigger KV pool (27.1M vs 23.3M tokens across all copies), because each copy's weights are spread over two GPUs. INT4 bought more KV room but not more speed here: it was slower than FP8 at 256 users in both layouts. ThinkingCap ran at the same speed as the base model in every cell, which is what you'd expect from a fine-tune with identical architecture and precision. Its GSM8K score was a little higher (0.768 vs 0.744), but on 250 questions that gap is about one standard error, so we don't call it a quality difference.

Every point

Before tuning further, audit the failures

The replica results were useful only after we separated a proxy cap, cache contamination and an unresolved TP8 cliff from real model behaviour. Part 7 collects that diagnostic record, including the fixes that worked and the plausible hypotheses that did not.

Sources

Bracketed ids [S..] refer to the campaign bibliography (docs/SOURCES.md). Numbers not attributed to a source are our own measurements on this node.

1. [S24] NVIDIA, NCCL environment variables
2. [S05] vLLM project, Expert Parallel Deployment (docs)
3. [S04] vLLM project, Data Parallel Deployment (docs)
4. [S01] NVIDIA, H200 NVL product brief (PB-12243): NVLink bridge topology guidance
5. [S96] vLLM project, https://github.com/vllm-project/vllm/issues/48071
6. aiohttp, https://docs.aiohttp.org/en/stable/client_reference.html#tcpconnector (aiohttp docs: TCPConnector, limit defaults to 100)
7. qtris123, https://github.com/vllm-project/vllm/pull/48075 (vLLM PR #48075)
8. vLLM, https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html (vLLM docs: expert parallel deployment)