WEKA

WEKA is a company that makes a super-high-performance scale-out parallel file system.

NeuralMesh Axon

I think WEKA NeuralMesh Axon is a rebranding of WEKA’s hyperconverged mode, where the WEKA file system is instantiated across the node-local SSDs of compute nodes.

Challenges

WEKA relies on transparently tiering data from its flash layer to an underlying object store to achieve economical $/TB. However, when data is tiered down, its read performance becomes very poor, resulting in unexpected/unpredictable performance when operating at scale. Huggingface showed that this behavior resulted in 40% loss of training throughput when training using a 24 TB dataset.¹

Footnotes

1. The Smol Training Playbook: The Secrets to Building World-Class LLMs ↩