When training or serving spans more than one GPU node, the network between the nodes decides how well the cluster scales. Here is how NVIDIA’s two AI fabrics, Quantum-X800 InfiniBand and Spectrum-X Ethernet, differ, and when each one fits.
A single GPU node is one tightly coupled machine. The moment a job spans two or more nodes, the fabric between them becomes the component that decides whether the cluster scales linearly or stalls. Distributed training spends a large share of every step exchanging gradients between GPUs on different nodes, and distributed inference moves activations and cache between stages, so the node-to-node network sits directly in the critical path. This guide is about that layer: the two fabrics NVIDIA builds for it, what “InfiniBand-class Ethernet” actually means, and how to choose.
There are two interconnect layers in a GPU cluster, and they are often confused. Inside a node, NVLink and NVSwitch connect the GPUs to each other at terabytes per second, and that layer stops at the node’s edge. Between nodes, a separate scale-out fabric carries traffic over network adapters. The fabric that compares to InfiniBand is this node-to-node network, and that is where Quantum-X800 InfiniBand and Spectrum-X Ethernet are the two choices.
Key Takeaways
- The scale-out fabric is a workload decision, not a default. For multi-node training and inference, node-to-node bandwidth and tail latency often gate cluster throughput more than the GPUs do, so choose the fabric against the job.
- InfiniBand (Quantum-X800) is the reference AI fabric. It is lossless by design and adds in-network reductions through SHARP, which offloads collective operations like all-reduce to the switch instead of the GPUs.
- Spectrum-X is Ethernet engineered to behave like InfiniBand. Per-packet adaptive routing, telemetry-based congestion control, and noise isolation give standard Ethernet the predictable behavior AI traffic needs, without a separate InfiniBand network to operate.
- One SuperNIC endpoint serves both fabrics. The NVIDIA ConnectX-8 SuperNIC runs either InfiniBand or Ethernet at up to 800 Gb/s, so the node hardware does not change when the fabric does; the choice is the switch layer and the operating model.
- “InfiniBand-class” is NVIDIA’s positioning. The published stats show a gain over traditional Ethernet, not a measured tie with InfiniBand, so weigh it against your own latency and collective-heavy workloads.
What the Scale-Out Fabric Decides
In data-parallel training, every node computes gradients on its share of the batch and then all nodes must agree on the combined result before the next step. That synchronization, the all-reduce collective, runs on every step and moves a volume of data proportional to the model size. If the fabric is slow, lossy, or prone to congestion, the GPUs sit idle waiting for the collective to finish, and adding nodes stops adding throughput. Pipeline and tensor parallelism that cross node boundaries have the same dependency on the fabric. For multi-node AI, the fabric is part of the compute budget.
Two properties separate an AI fabric from ordinary data-center Ethernet. The first is loss behavior: standard Ethernet drops packets under congestion and relies on retransmission, which is fatal to a synchronized collective. The second is routing: standard Ethernet spreads flows with static hashing, which collides large AI flows onto the same links and leaves others idle. InfiniBand was built without those problems, and NVIDIA re-engineered Ethernet to avoid them too.
InfiniBand: the Reference AI Fabric
NVIDIA’s Quantum-X800 is the current InfiniBand generation, built for exactly this job. Its Q3400-RA switch carries 144 ports at 800 Gb/s using 200 Gb/s-per-lane signaling, and a two-tier fat tree of these switches scales to more than 10,000 host connections at 800 Gb/s. InfiniBand is lossless by design through credit-based flow control, it uses adaptive routing and telemetry-based congestion control to keep links evenly loaded, and it self-heals around link failures.
Its distinguishing feature for AI is SHARP, the Scalable Hierarchical Aggregation and Reduction Protocol, now in its fourth generation. SHARP performs collective reductions inside the switch rather than on the GPUs: instead of every node shipping its full gradient to every other node, the network aggregates along the way, which cuts the data moved and frees GPU cycles. The fourth generation adds FP8 precision and new collective operations. NVIDIA attributes large collective speedups to it; as with any vendor figure, treat the headline number as a best case and validate against your own model. The trade is that InfiniBand is a specialized network with its own switches, cabling, and operational model, run alongside or instead of your Ethernet estate.
Spectrum-X Ethernet: InfiniBand-Class Behavior on Ethernet
Spectrum-X is NVIDIA’s answer to a specific question: can Ethernet, which every data center already runs, be made to behave like InfiniBand for AI. It is a matched platform, the Spectrum-4 (SN5600) switch paired with a BlueField-3 or ConnectX-8 SuperNIC, and it earns the “InfiniBand-class” label through three control loops that work together (NVIDIA engineering blog):
- Per-packet adaptive routing. The switches sample the queue depth of every candidate egress port at sub-microsecond intervals and send each packet toward the least-congested port, reacting in hundreds of nanoseconds. Because packets can then arrive out of order, the SuperNIC reorders them at the endpoint so the application sees a clean stream.
- Telemetry-based congestion control. Adaptive routing cannot fix incast, where many senders target one receiver at once. For that, switch and SuperNIC coordinate: the switch marks packets only once its routing capacity is exhausted, and senders adjust their rates using round-trip-time probes plus those marks.
- Plane load balancing and noise isolation. A third loop spreads traffic across independent network planes so one job’s bursts, or a failure, degrade the fabric gracefully instead of stalling other tenants’ jobs.
NVIDIA’s published headline is a 1.6x improvement in generative-AI network performance over traditional Ethernet fabrics (TechRadar). That figure is a gain over standard RoCE Ethernet, measured by the vendor, not an independent benchmark showing parity with InfiniBand. Spectrum-X is engineered to close the gap with InfiniBand while staying on Ethernet, and for collective-heavy training the InfiniBand path with SHARP keeps a structural advantage. Which matters more depends on your workload.
The Shared Endpoint: ConnectX-8 SuperNIC
What makes the choice cleaner than it sounds is that both fabrics terminate on the same adapter. The NVIDIA ConnectX-8 SuperNIC supports both protocols, InfiniBand at 800/400/200 Gb/s or Ethernet at 400/200/100 Gb/s, with up to 800 Gb/s of total bandwidth over a PCIe Gen6 host interface (ConnectX-8 datasheet). It carries the endpoint half of the congestion-control and routing features for both fabrics. In practice the node’s network adapters are the same hardware either way; the fabric decision is made at the switch layer and in how you want to operate the network, not by re-specifying the servers.
Choosing Between Them
- Lead with InfiniBand (Quantum-X800) when the cluster is a dedicated AI island, when collective-heavy training can exploit SHARP’s in-network reductions, and when the lowest, most deterministic latency is the priority and you are prepared to run a specialized network.
- Lead with Spectrum-X Ethernet when the AI fabric has to live inside an Ethernet operation, when you want one network technology and skill set across the data center, and when multi-job noise isolation on a shared fabric matters. Both deliver 800 Gb/s-class connectivity per adapter; the difference is the operating model and the last increment of collective performance, not the raw link speed. The right answer is set by the workload’s collective intensity, your latency targets, and whether a separate InfiniBand network is something you want to operate.
Where This Fits at OpenMetal
OpenMetal builds single-tenant GPU servers to the workload, as dedicated bare metal on fixed monthly pricing rather than a per-GPU-hour meter. Today the shipping catalog is single-card and dual-card as-built: the NVIDIA H200 NVL and the NVIDIA RTX PRO 6000 Blackwell Server Edition, where a two-card node is two discrete GPUs, each with its own memory, not a pooled accelerator. The 8x NVIDIA HGX B300 node is on the OpenMetal roadmap for the largest reasoning and training workloads; inside that node the eight GPUs share aggregate HBM over NVLink and NVSwitch, placed by the serving software, and each node carries eight ConnectX-8 SuperNICs for scale-out.
A multi-node B300 build, several of those nodes connected by a Spectrum-X Ethernet or Quantum-X800 InfiniBand scale-out fabric, is possibly available as a build-to-requirement engagement. It is a design conversation with the OpenMetal Engineering and Sales teams; the configuration is sized to your workload. Across such a cluster the GPUs exchange data and gradients over the fabric using data, tensor, and pipeline parallelism; memory is aggregated within each node over NVLink, not pooled across nodes. If a multi-node AI cluster is where your workload is heading, that is exactly the conversation to start.
Ready to Plan a Multi-Node AI Cluster?
The fabric is only worth choosing against a real workload. Tell us your model, your scale target, and your latency objective, and the OpenMetal team will help you size the nodes and the scale-out fabric together.
A Short Fabric Checklist
- How much of each training step is collective communication, and would SHARP’s in-network reductions materially cut it?
- What is your tail-latency target for the collective, and how many nodes must stay in sync?
- Do you want a dedicated InfiniBand network, or one Ethernet technology across the data center?
- Will the fabric be shared across jobs or tenants, where noise isolation matters?
- What link speed per GPU does the workload actually need, today and at your next scale step?
Answer those and the fabric usually chooses itself. If you would rather work through them against a real deployment than a spec sheet, bring your requirements to the OpenMetal team and we will size the cluster, and its fabric, to the workload.
Sources
- NVIDIA, “Giga-Scale AI: The Ethernet Evolution as Spectrum-X Ethernet Rewrites the Rules” (adaptive routing, congestion control, noise isolation): https://developer.nvidia.com/blog/giga-scale-ai-ethernet-evolution-spectrum-x-ethernet-rewrites-rules/ (verified 2026-10-20)
- TechRadar on NVIDIA Spectrum-X (1.6x over traditional Ethernet, vendor claim): https://www.techradar.com/pro/spectrum-x-promises-adaptive-routing-and-congestion-control-for-maximum-bandwidth-and-noise-isolation-ensuring-predictable-outcomes-for-thousands-of-simultaneous-ai-jobs-at-any-scale (verified 2026-10-20)
- NVIDIA ConnectX-8 SuperNIC datasheet (dual-protocol InfiniBand/Ethernet, up to 800 Gb/s, PCIe Gen6): https://www.pny.com/en-eu/file%20library/professional/datasheet/mellanox/connectx-datasheet-connectx-8-supernic-4057050-pnywebonly.pdf (verified 2026-10-20)
- NVIDIA Quantum-X800 InfiniBand switch datasheet (Q3400-RA 144x800G, fourth-generation SHARP, fat-tree scale): https://resource.naddod.com/files/products-Switches/2025-03-28/nvidia-quantum-x800-infiniband-switches-datasheet-011078.pdf (verified 2026-10-20)