VAST

VAST Data makes an eponymous all-flash parallel file system that blends NFS with parallel file system concepts.

Technology

I wrote a description about some of the unique aspects of its architecture here: VAST Data’s storage system architecture (glennklockwood.com). In brief,

1. NFS clients: It allows standard NFS clients to connect to it for low performance, but also supports enhanced NFS clients including NFSoRDMA and NFS multipathing to allow a single client to issue reads and writes to multiple VAST frontend servers in parallel.¹
2. Novel write path: It lands all writes to storage-class memory (formerly Optane, but now Kioxia FL6) to preserve low latency at the cost of write bandwidth. Data accumulates in this write layer over time, building massive stripes that contain similar data (using a locality-sensitive hash on incoming data). When a write stripe is full, it is compressed and flushed to low-durability, cost-effective QLC flash.²
3. Novel data protection: VAST uses extremely wide stripes of up to 150+4 to protect data with very high efficiency. They use locally decodable codes instead of standard Reed-Solomon codes to avoid having to rebuild using 150 surviving drives; instead, they only need $\frac{150}{4}$ surviving drives to rebuild.³

Items 2 and 3 above allow VAST to get away with using all low-endurance QLC flash at an effective price-per-gigabyte rate that becomes competitive with hybrid disk+flash storage systems in overall value. VAST is neither the fastest nor the cheapest, but it does both reasonably well.

Customers

VAST is seeing broad success in midrange public HPC, private sectors including finance, and GPU-as-a-Service cloud providers.

VAST’s most notable customers include:

• TACC, which houses the NSF’s largest supercomputers⁴
• CoreWeave, a growing player in providing GPUs-as-a-Service⁵
• Lambda Labs, a major GPU-as-a-Service cloud provider⁶
• Core42, a major GPU-as-a-Service cloud provider in the middle east⁷
• xAI⁸
• Hypertec Cloud, a Canadian GPUaaS provider⁹
• Crusoe¹⁰ https://www.datacenterdynamics.com/en/news/crusoe-partners-with-vast-data-for-high-performance-storage-on-crusoe-cloud/

Footnotes

1. Meet Your Need for Speed with NFS and VAST Data ↩

2. Client-side Compression Techniques and their Impact on Similarity Reduction – VAST Data (zendesk.com) ↩

3. Breaking Resiliency Trade-Offs with Locally Decodable Codes (vastdata.com) ↩

4. VAST Data wins TACC supercomputer storage contract – Blocks and Files ↩

5. CoreWeave Partners With VAST Data ↩

6. Lambda Partners with VAST Data ↩

7. Core42 Partners with VAST Data ↩

8. Grok is being built on the VAST Data platform! ↩

9. VAST Data Accelerates Large-Scale AI Deployments with Hypertec Cloud ↩

10. ↩