Foundation Model
A large AI model trained on broad data at scale that can be adapted to a wide range of downstream tasks — GPT-4, Claude, Gemini, and AlphaFold 3 are all foundation models.
What it means
A foundation model is a large-scale model trained on massive, diverse datasets using self-supervised learning, designed to be fine-tuned or adapted for many downstream tasks rather than trained from scratch for each application. The term was coined by Stanford’s HAI (Human-Centered AI Institute) in 2021 to describe models like GPT-3, BERT, and CLIP that serve as a “foundation” for building specialized AI systems.
Key properties:
- Pre-trained at scale: Foundation models are trained on enormous datasets (often hundreds of billions to trillions of tokens) using significant compute
- General-purpose: The same model can be adapted to translation, summarization, code generation, question answering, and other tasks through fine-tuning or prompting
- Emergent capabilities: Abilities that weren’t explicitly trained appear at sufficient scale — multi-step reasoning, analogical thinking, in-context learning
Beyond text: The foundation model concept has expanded beyond language:
- Image: CLIP, Stable Diffusion, GPT-4V
- Protein: ESM3 (sequence, structure, function), AlphaFold 3
- Scientific: Gato (multi-task agent), GNoME (crystal structures)
- Multimodal: Models that combine text, images, and other modalities
Why it matters for researchers
The paradigm shift in scientific AI: Before foundation models, AI tools for science were narrow models trained from scratch for specific tasks. Now, large pre-trained models can be adapted with relatively little task-specific data through fine-tuning or prompting.
Practical implication: When you use Elicit, Semantic Scholar, or similar tools, the underlying capabilities are built on top of language foundation models. The tool’s quality is partly a function of which foundation model it uses and how it has been fine-tuned for the domain.
For methods sections: When reporting research that used a foundation model, specify the model family, version, and whether it was used off-the-shelf or fine-tuned. Foundation model capabilities change substantially between versions, and the version you used affects reproducibility.