I spent much of my career in HPC focused on storage and high-performance I/O. Specifically,
- 2014-2015: When I worked at 10x Genomics, I managed the FAS NetApp filers to which all the company’s DNA sequencers output and all its downstream processing relied upon.
- 2015-2022: When I joined NERSC, I was 50% funded on the TOKIO project to build tools to understand I/O performance through the full stack. I went on to help lead the Perlmutter storage subsystem design and develop NERSC’s ten-year storage strategy, dubbed Storage 2020
- 2022-2024: I joined Microsoft as a product manager in Azure Storage, the unit of Azure responsible for Azure Blob, Azure Files, and all other first-party storage services.
It is a complicated and not-well-understood aspect of HPC, so the community of researchers and practitioners around HPC storage is small and dedicated. I stopped working on storage when I joined Microsoft Azure’s AI infrastructure team, where the biggest challenges arose from compute, not storage.
I don’t miss working on storage, as I felt I had caught up to the cutting edge of it and there wasn’t enough innovation to warrant my focusing on it as a full-time job. Being “the storage guy” also required that I pay attention to some uninteresting aspects of HPC storage such as data management, compression, and topics like that. However, I miss the community and the smart people who dedicated their careers to I/O.