Multi-GPU training: NVLink, InfiniBand, and when one 8× node beats two 4× servers

Two H100s are not twice one H100. How close they get depends on the wire between them, and that wire is the real difference between the SXM and PCIe versions of the same chip, and between a server and a cluster.

The wires

PCIe 5.0 x16

64 GB/s each way. What a PCIe H100 or an RTX card has to its neighbours (through the CPU, with contention).

NVLink bridge (pairs)

600 GB/s between two PCIe cards bridged together. Only pairs.

NVLink 4 (H100/H200 SXM)

900 GB/s per card, all-to-all through NVSwitch on the HGX baseboard. Any card to any card at full speed.

NVLink 5 (B200/B300)

1.8 TB/s per card, same all-to-all topology.

InfiniBand NDR (between nodes)

400 Gbit/s per link, eight links per node: 3.2 Tbps, about 400 GB/s per node, with RDMA so the GPUs talk without the CPU.

What the parallelism needs

• Data parallel exchanges gradients once per step: the whole model's worth of bytes. Tolerates PCIe for small models; wants NVLink above ~10B.
• Tensor parallel splits every layer and exchanges activations several times per layer: it needs NVLink and stays inside a node. This is how a 70B model is served on two or four cards.
• Pipeline parallel puts different layers on different cards and passes activations at the boundaries: modest bandwidth, so it is the way to cross nodes.
• FSDP / ZeRO shard weights, gradients and optimiser state across all cards: a lot of all-gather traffic. Fine inside an NVLink domain, and across nodes with InfiniBand.

Why one 8× node beats two 4× servers

Eight SXM cards on one HGX board are one NVLink domain: 900 GB/s (or 1.8 TB/s) between any two of them. Two separate four-card servers are two domains joined by the network, and everything between them runs at network speed. For a model that fits in one node, the eight-card server is always faster than two fours; for a model that needs sixteen cards, two eight-card nodes on InfiniBand are the smallest cluster that makes sense. On our catalogue the price is linear, so the eight-card server costs exactly the same as two fours: $9,072 a month for 8× H100 SXM.

When PCIe is enough

Fine-tuning with LoRA on models up to ~30B, inference of models that fit one card, and embarrassingly parallel work (many independent runs) do not need NVLink. A PCIe H100 at $810 is 29 % cheaper than the SXM card for the same compute; two of them bridged serve a 70B model at FP8. The SXM premium buys the fabric, and you should pay it only when the job crosses cards every step.

Sizing a training cluster

A rule of thumb for full fine-tuning or continued pre-training with mixed precision and an Adam-style optimiser: about 16 bytes per parameter of state, plus activations. A 70B model needs 1.1 TB of state: two 8× B200 nodes (2.9 TB) or two 8× H200 nodes (2.3 TB) with sharding. A 8B model trains comfortably on one 8× H100 node. Throughput scales close to linearly across nodes on InfiniBand up to the sizes we place (64 nodes, 512 cards); beyond that, ask.

On our catalogue

• SXM nodes: B300, B200, H200, H100 SXM, A100, and MI300X on OAM, in 1×, 2×, 4× and 8×. Every count on the same baseboard, with the full fabric.
• 8× nodes come with dual 100 Gbit/s Ethernet and, in a cluster, 3.2 Tbps InfiniBand.
• Clusters from 16 to 512 cards, on the same InfiniBand fabric, at the per-card price of the catalogue, handed over as Slurm or Kubernetes.

Check the fabric yourself at first login: nvidia-smi topo -m shows NV18 (or NV12) between every pair on an SXM node, and ibstat shows the InfiniBand ports active on cluster nodes.