NVLink, NVSwitch and InfiniBand: the fabric that makes eight GPUs act as one

Buying GPUs is the easy part. Getting eight of them to behave like one machine — so a model too big for any single card trains as if it were on one giant accelerator — is a question of fabric: the wiring that moves data between GPUs, and between nodes. It is the least glamorous line on a spec sheet and the one that most often decides whether your expensive GPUs are computing or waiting. Here is how it works, and why it matters.

The problem: a GPU that waits is a GPU you are paying for nothing

A large training or inference job is not one calculation on one chip. The work is split across GPUs, and those GPUs have to talk to each other constantly. They exchange:

• tensor- and model-parallel data — the model itself is split across GPUs;
• activations and gradients during training;
• KV-cache and model state during inference;
• collective operations like AllReduce and AllGather, where every GPU has to synchronise with every other;
• and a great deal of raw GPU-memory traffic.

If the link between GPUs is slower than the GPUs themselves, the chips sit idle waiting for data. On hardware that costs what a B300 node costs, communication that cannot keep up is money burning. This is why interconnect — not just raw FLOPS or memory — is a first-class design decision, not a footnote.

Scale-up: NVLink and NVSwitch, inside the node

NVLink is NVIDIA's direct GPU-to-GPU interconnect — a private, high-bandwidth highway between chips that bypasses the far slower PCIe bus. NVSwitch is the switch that turns a handful of point-to-point links into an all-to-all fabric, so every GPU in a node can talk to every other at full speed, not just its neighbour.

The bandwidth has climbed relentlessly with each GPU generation:

• NVLink 1 (Pascal) — 160 GB/s. Direct GPU-to-GPU.
• NVLink 2 (Volta) — about 300 GB/s. NVSwitch arrives, enabling scalable all-to-all connectivity.
• NVLink 3 (Ampere, A100) — 600 GB/s. Distributed training and large-scale HPC.
• NVLink 4 (Hopper, H100) — 900 GB/s. Pushed GPU scale-up further for LLM training and inference.
• NVLink 5 (Blackwell, B200 / B300) — 1.8 TB/s per GPU. The NVL72 paradigm — up to 72 GPUs in one tightly coupled NVLink domain.
• NVLink 6 (Rubin, on the roadmap) — 3.6 TB/s per GPU.

This is the "scale-up" fabric, and it is what makes a multi-GPU node more than the sum of its cards. In an 8-GPU B300 node, NVSwitch places all eight GPUs on a single NVLink-5 domain at 1.8 TB/s each. To your software the eight behave like one very large accelerator with pooled memory — which is exactly why a 70-billion-parameter model that will not fit or train on a single card runs comfortably across the node. (We walked through that memory maths in will your model fit?.)

Scale-out: InfiniBand and RoCE, between nodes

NVLink stops at the edge of the node. When a job is too big for eight GPUs and needs several nodes working together, a different fabric takes over: InfiniBand — or RoCE, RDMA over Converged Ethernet. This is the "scale-out" network. It carries far less bandwidth than NVLink, but it is built for low latency and remote direct memory access across a cluster, so separate nodes can run collectives together almost as if they shared memory.

East-west vs north-south traffic

There is a piece of networking vocabulary worth knowing here, because it explains why AI clusters are built so differently from ordinary ones. Picture the classic network diagram with users and the internet drawn at the top. Traffic flowing up and out to them is north-south — requests arriving, results leaving, datasets and checkpoints moving to and from storage. Traffic flowing sideways, server to server inside the cluster, is east-west.

A traditional web application is mostly north-south: a user asks, the server answers, done. An AI training run is the opposite — overwhelmingly east-west. The GPUs and nodes spend their time talking to each other, exchanging gradients and running collectives, and that internal chatter dwarfs anything going in or out of the cluster. This is the whole reason the fabric matters so much: NVLink carries east-west traffic inside the node, InfiniBand carries east-west traffic between nodes, and an AI factory is engineered east-west-first — because that is where nearly all the traffic, and nearly all the waiting, actually happens. The north-south side (loading a dataset, serving an inference response, pushing a checkpoint to object storage) is real, but it is a trickle next to the east-west flood.

The two fabrics are complementary, not competing. The clean mental model is:

• Inside a node: GPU → NVLink → NVSwitch → GPU. This is scale-up.
• Between nodes: Node → InfiniBand / RoCE → Node. This is scale-out.

Get the scale-up fabric right and eight GPUs act as one. Get the scale-out fabric right and many nodes act as one. Get either wrong and you have bought a lot of GPUs that spend their time waiting on the wiring.

Where this is heading

The frontier is pushing the scale-up domain well beyond a single node. NVIDIA's NVL72 puts 72 Blackwell GPUs into one tightly coupled NVLink domain — a whole rack behaving as a single accelerator — and the roadmap points at higher bandwidth again (NVLink 6 at 3.6 TB/s), larger scale-up domains, and in-network acceleration, where the switch itself helps run the collectives. The label the industry has settled on for all of this is the "AI Factory": compute, storage, network, scheduling and software engineered together so that every GPU cycle turns into useful work.

The point behind the buzzword is real, and it is the thread running through this whole series:

• GPU density is hardware.
• GPU utilisation is architecture.
• AI-factory efficiency is systems engineering.

More GPUs do not help if they cannot talk to each other fast enough to stay busy.

What this means for a FlexiCloud node

When you rent a bare-metal B300 node from us, the scale-up fabric is already built in: eight B300 SXM GPUs on a single NVLink-5 + NVSwitch domain, delivered as one coherent machine rather than eight cards you have to stitch together. That is what lets the node train and serve models no single GPU could. For workloads that outgrow one node, we connect multiple nodes over InfiniBand for scale-out, priced on request. To be clear about scale: our unit is the eight-GPU node, not a 72-GPU NVL72 rack — that rack-scale domain is the frontier end of the same idea, and it is where the industry, not a single node, is headed.

We deliver the hardware and the fabric; you run the software on top — the frameworks, the parallelism strategy, the scheduling. That division is deliberate: the interconnect and topology are ours to get right so your GPUs never wait on the wiring, and the stack is yours to run however your workload needs. If you want to talk through how your job maps onto a node — how it parallelises, whether it needs one node or several — see the cost breakdown, check the pricing page, or talk to an engineer.