Pretraining

Pretraining is the initial large-scale training phase in which a language model learns general language understanding and world knowledge by predicting tokens across a massive text corpus. The model is exposed to hundreds of billions or trillions of tokens from diverse sources — web pages, books, code, and scientific papers — and adjusts its parameters to minimize next-token prediction loss. The resulting pretrained model serves as the foundation for downstream adaptation through instruction tuning or fine-tuning.