NVLink

NVLink is the interconnected used between NVIDIA GPUs for memory coherence. GPUs have NVLink ports, and NVLink Switches (also referred to as NVSwitches¹) allow multiple GPUs to connect into a mesh.

NVLink 5

NVLink 5 provides 800G per link.²

NVLink 5 switches have 72x800G ports.²

One B200 GPU has up to 1.8 TB/s (14,400 Gbps) of NVLink 5 bandwidth,¹ which breaks down to 18 NVLink 5 ports.

A GB200 NVL72 rack has:

• 72 GPUs (4 per tray, 18 trays)
• 18 NVSwitch ASICs (2 per tray, 9 trays)

Since each GPU has 18 NVLink ports and each rack has 18 NVSwitches, each GPU can connect to each NVSWitch in the rack to form a single-layer, non-blocking fabric.

NVLink-C2C

NVLink C2C (chip-to-chip) is the interconnect that comes off of NVIDIA’s Grace CPUs to carry coherence traffic to either other Grace CPUs or NVIDIA GPUs which implement NVLink. It was positioned against PCIe Gen 5 and can carry Arm AMBA CHI protocol.³

Physically, NVLink-C2C is implemented using:⁴

• single-ended (ground-reference) NRZ signaling, not low-voltage differential
• nine data signals per link, each at 40 Gbps, per direction, or 45 GB/s per direction

Grace and H100 both implement ten NVLink-C2C links per socket, for a total of 450 GB/s/direction or 900 GB/s bidirectional total⁴

cNVLink

Coherent NVLink (cNVLink) is something that appeared in the marketing material surrounding the launch of the Grace CPU. The only reference to it that I could find was in the HotChips34 talk about Grace⁵ where it is described as a protocol that is supported alongside PCIe Gen 5 on two of Grace’s PCIe Gen5 x16 blocks. Maybe it’s for some future NVLink-capable add-in card like a Mellanox NIC.

NVLink 4

This is the generation of NVLink supported on H100.

• Each H100 has 18 NVLinks, each at 50 GB/s (400 Gbps)⁶
• Each H100 is connected to each NVSwitch via
  • 5 NVLinks to two NVSwitches
  • 4 NVLinks to the other two NVSwitches
  • 5+5+4+4 = 18 NVLinks per H100 GPU to four NVSwitches

Each NVSwitch has:⁶

• 100 Gbps per lane implemented as 50G PAM4
• 64 ports, each two lanes wide, or 64x200G ports
• Each “NVLink network port”⁶ is 400 Gbps, implying one “NVLink network port” is four lanes (two ports) wide
• This means each NVLink switch has effectively 32 NVLink ports at 400G each

NVSwitch in this generation also supports scale-up via external OSFP connectors.⁶

NVLink 3

This is the generation of NVLink supported on A100.

• Each A100 GPU has 12 NVLink ports⁷
• Each A100 is connected to each NVSwitch via two links
• 8x A100 $\times$ 2 links to each switch implies 16 ports per NVSwitch
• Each A100-to-NVSwitch connects at 600 GB/s bidirectionally
  • This is 300 GB/s/direction or 2,400 Gbps per direction
  • This is implemented over 12 NVLink connections
  • This implies each NVLink connection is 200 Gbps per direction

Footnotes

1. NVIDIA NVLink and NVIDIA NVSwitch Supercharge Large Language Model Inference | NVIDIA Technical Blog ↩ ↩²

2. NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference | NVIDIA Technical Blog ↩ ↩²

3. NVIDIA Opens NVLink for Custom Silicon Integration | NVIDIA Newsroom ↩

4. 9.3 NVLink-C2C: A Coherent Off Package Chip-to-Chip Interconnect with 40Gbps/pin Single-ended Signaling | IEEE Conference Publication | IEEE Xplore ↩ ↩²

5. HC2022.NVIDIA Grace.JonathonEvans.v5.pdf (hotchips.org) ↩

6. NVIDIA NVLink4 NVSwitch at Hot Chips 34 - ServeTheHome ↩ ↩² ↩³ ↩⁴

7. Introducing NVIDIA HGX A100: The Most Powerful Accelerated Server Platform for AI and High Performance Computing | NVIDIA Technical Blog ↩