Next-Token Prediction

Also known as: causal language modeling, CLM

Next-token prediction is the training objective used by most large language models. Given a sequence of tokens, the model is trained to assign high probability to the actual next token in the corpus. Because this objective requires no human annotation — the training signal comes directly from the text itself — it scales naturally to internet-scale datasets. Minimizing this loss across trillions of tokens forces the model to internalize grammar, facts, reasoning patterns, and style.