Foundation models for science are pretrained AI models for science that excel at transforming some class of input data into another class of output data that is scientifically relevant.
Just as a transformer is good at next-token prediction, a foundation model for science might be good at predicting the next frame in a 2D image evolving in time. That foundation model can then be fine-tuned for a specific scientific problem using physics-based simulation to create a generative model that is very good at generating outputs that mimic what the physics model would’ve produced, but at a fraction of the computational cost.
“Frontier models for science” is a term made up by FASST. I’m not sure if it is meant to be different from foundation models for science.
Characteristics
Vision transformers can be used to process scientific imaging data, but they typically require very long sequences to handle high-resolution imagery. This is different from LLMs for chatbots which downsample or crop images before analyzing them.1
Example models
The following models have been developed for science and proclaim to be foundation models. However, it is unclear how useful they are beyond the specific case for which they were fine-tuned.
| Model | Creator | Year | Domain | Parameters |
|---|---|---|---|---|
| ORBIT | ORNL | 2024 | Climate | 113 B |
| Aurora | Microsoft | 2024 | Climate | 1.3 B |
| FourCastNet | NVIDIA | 2022 | Climate | 100 M2 |
| TEDDY | Merck | 2025 | Single Cell Bio | 70-400 M |
There’s also a paper where the authors “advocate for developing [foundation models] for power grids”3 but it doesn’t actually present a trained model.
Footnotes
-
This has to be inferred since the paper only says the model requires 10 GB of GPU memory and took 1024 GPU-hours of A100 time. The model used 8 AFNO blocks (888K per block) and a 12-layer vision transformer (~7.08M per layer). ↩
-
Foundation models for the electric power grid - ScienceDirect ↩