AMD MI300X

MI300X is AMD’s first GPU to use CDNA 3.

Specifications

Each MI300X GPU has:¹

• 8 XCDs
  • 304 CUs (38 per XCD)
    • 19,456 Stream Processors (64 per CU)
    • 1,216 Matrix Cores (4 per CU)
  • 2.1 GHz (peak)
  • 2:4 structured sparsity
• 192 GB HBM3 (8 stacks)
  • 5.3 TB/s (max)
• 7x16 AMD Infinity Fabric (D2D)
• 1x16 PCIe Gen5 (H2D)
• 750 W maximum (TBP)²

It also has a complex memory hierarchy that I don’t yet understand.³

Performance

The following are theoretical maximum performance in TFLOPS:¹

Data Type	VFMA	Matrix	Sparse
FP64	81.7	163.4
FP32	164.4	163.4
TF32		653.7	1307.4
FP16		1307.4	2614.9
BF16		1307.4	2614.9
FP8		2614.9	5229.8
INT32
INT8		2614.9	5229.8

A single 8-way OAM MI300X UBB is capable of hosting a copy of Llama 3.1 405B in FP16.⁴

Platforms

The following platforms support MI300X GPUs:³

• Azure ND MI300X v5
• Dell PowerEdge XE9680
• HPE Cray XD675
• Lenovo SR685a V3
• Supermicro AS-8125GS-TNMR2

The following cloud providers sell MI300X:

• Vultr⁵

Footnotes

1. AMD CDNA(tm) 3 Architecture ↩ ↩²

2. AMD Instinct™ MI300X Accelerators ↩

3. AMD Instinct MI300X Hot Chips 2024_Page_15 - ServeTheHome ↩ ↩²

4. Meta’s New Llama 3.1 Launched with Day 0 Support on AMD Instinct™ MI300X (amd.com) ↩

5. https://insidehpc.com/2024/09/vultr-offers-amd-instinct-mi300x-for-ai-cloud-platform/ ↩