← All chapters
LLM Core & Architecture
Transformer internals, attention, tokenization, and decoding.
24 terms
Autoregressive GenerationAutoregressive generation produces text one token at a time: at each step the model receives the prompt plus all previously generated tokens as context, then predicts a probability distribution over the vocabulary and samples or greedily selects the next token. This token is appended to the sequence and the process repeats until an end-of-sequence token appears or a length limit is reached. The approach is simple and produces high-quality text but scales linearly in generation cost with output length.Beam SearchBeam search is a decoding algorithm that maintains a fixed number of candidate sequences — the beam width — expanding each by all vocabulary tokens at every step and keeping only the top candidates by cumulative log-probability. Wider beams explore more of the generation space and often produce higher-likelihood sequences than greedy decoding, at the cost of proportionally more compute per step. Beam search is common in machine translation but less so in open-ended generation, where diversity is valued.Byte-Pair EncodingByte-pair encoding is a subword tokenization algorithm that starts from individual bytes or characters and iteratively merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. The result is a compact vocabulary that assigns short token sequences to common words and longer sequences to rare words. BPE tokenizers handle arbitrary text including out-of-vocabulary words gracefully and are the basis for GPT-series tokenizers.Context LengthContext length is the maximum number of tokens a language model can attend to in a single forward pass, covering both the input prompt and the generated output. Models process only what fits within this limit; content beyond the boundary is truncated or must be handled by chunking and retrieval strategies. Longer context windows allow richer in-context examples, full documents, and multi-turn conversations without external memory, but they increase memory requirements and attention compute quadratically unless efficient attention variants are used.Decoder-Only ArchitectureA decoder-only architecture is a transformer variant that uses causal (left-to-right) self-attention: each token can attend only to tokens that precede it in the sequence. This design makes the model naturally suited to autoregressive text generation. GPT, LLaMA, Claude, and most leading large language models use decoder-only architectures. The encoder-decoder design used in the original transformer is more common in translation and summarization models where the full input is available upfront.Embedding LayerThe embedding layer is the first learned component in a language model. It maps each integer token ID to a dense vector of fixed dimensionality — the model's hidden size. These vectors are the continuous representations the rest of the network operates on. The embedding weights are learned during pretraining; the layer essentially serves as a lookup table where semantically or statistically related tokens tend to occupy nearby regions of the embedding space.Flash AttentionFlash Attention is an IO-aware implementation of scaled dot-product attention that fuses the attention computation into a single GPU kernel and tiles operations to avoid materializing the full N×N attention matrix in high-bandwidth memory. By keeping intermediate activations in fast SRAM, Flash Attention achieves the same mathematical result as standard attention while using substantially less GPU memory and running significantly faster — enabling training and inference over much longer context lengths without algorithm-level changes.Greedy DecodingGreedy decoding selects the single highest-probability token at every generation step. It is the simplest and fastest decoding strategy and is fully deterministic given the same input. However, locally optimal token choices do not always lead to globally optimal sequences: a sequence that starts with a slightly lower-probability token may yield a far higher total-sequence probability. For tasks where exact accuracy matters more than fluency or diversity, greedy decoding is often a sensible baseline.KV CacheThe KV cache stores the key and value tensors computed for all previous tokens during autoregressive decoding so they do not need to be recomputed at each new generation step. Without caching, generating a sequence of length N requires O(N²) attention computations; with caching, only the new token's keys and values are computed at each step. The KV cache grows linearly with sequence length and is a primary contributor to GPU memory consumption during inference with long contexts.Layer NormalizationLayer normalization is a technique applied inside each transformer block that normalizes the activations across the feature dimension to have zero mean and unit variance, then rescales with learned gain and bias parameters. It stabilizes training by preventing the distribution of activations from shifting dramatically as gradients flow through deep networks. Modern large language models typically use RMSNorm — a simpler variant that omits the mean-centering step — for both training stability and inference efficiency.LogitsLogits are the raw, unnormalized scores produced by a language model's final linear layer before any probability conversion. For each position in the vocabulary, the model outputs a single real-valued logit indicating its relative preference for that token as the next prediction. These scores are fed into softmax to produce a probability distribution. Directly manipulating logits — by adding biases or masking tokens — enables precise control over generation without retraining the model.Mixture of ExpertsMixture of Experts is an architecture that replaces the dense feed-forward layer in a transformer block with a collection of parallel expert sub-networks and a learned routing function. For each token, the router activates only a small subset of experts — typically two out of dozens or hundreds — so total parameter count is large while the compute per token remains comparable to a much smaller dense model. MoE architectures like Mixtral and GPT-4 achieve high capacity with favorable inference economics.Model ParametersModel parameters are the learned numerical values — weights and biases — stored in the neural network after training. For transformer-based language models, parameters include the token embedding matrix, the query/key/value projection matrices in every attention head, feed-forward layer weights, and layer normalization scales. Parameter count is a common proxy for model capacity: larger models can represent more complex patterns but require more memory and compute to run.Multi-Head AttentionMulti-head attention runs several self-attention operations in parallel, each with its own learned projection matrices for queries, keys, and values. Each 'head' can specialize in attending to different relationship types — syntactic structure, coreference, or long-range context. The outputs of all heads are concatenated and projected back to the model dimension. Using multiple heads consistently outperforms a single larger attention operation of equivalent parameter count.Next-Token PredictionNext-token prediction is the training objective used by most large language models. Given a sequence of tokens, the model is trained to assign high probability to the actual next token in the corpus. Because this objective requires no human annotation — the training signal comes directly from the text itself — it scales naturally to internet-scale datasets. Minimizing this loss across trillions of tokens forces the model to internalize grammar, facts, reasoning patterns, and style.Positional EncodingPositional encoding injects sequence-order information into token representations before they enter the transformer layers. Because self-attention treats its input as a set rather than a sequence, position signals must be added explicitly. Early models used fixed sinusoidal functions; modern large language models typically use learned rotary position embeddings (RoPE), which encode relative distances between tokens and generalize better to sequence lengths beyond those seen during training.PretrainingPretraining is the initial large-scale training phase in which a language model learns general language understanding and world knowledge by predicting tokens across a massive text corpus. The model is exposed to hundreds of billions or trillions of tokens from diverse sources — web pages, books, code, and scientific papers — and adjusts its parameters to minimize next-token prediction loss. The resulting pretrained model serves as the foundation for downstream adaptation through instruction tuning or fine-tuning.Self-AttentionSelf-attention is the mechanism by which each token in a sequence computes a weighted combination of all other tokens' representations. For every token, the model derives a query, a key, and a value vector. Dot products between queries and keys produce attention scores that are scaled and softmaxed into weights, which are then applied to the value vectors to produce the output. Self-attention lets the model capture long-range dependencies without the sequential bottlenecks of recurrent architectures.SoftmaxSoftmax is a function that converts a vector of raw scores (logits) into a probability distribution. It exponentiates each score and normalizes by the sum of all exponentiated values, ensuring outputs sum to one and are all positive. In language models, softmax is applied to the final layer's logit vector to yield per-token probabilities. Temperature scaling modifies the logits before softmax is applied, controlling how peaked or flat the resulting distribution is.TemperatureTemperature is a scalar hyperparameter that controls the randomness of a language model's output by dividing logits before the softmax operation. A temperature of 1.0 leaves the model's learned distribution unchanged. Values below 1.0 sharpen the distribution, making high-probability tokens more dominant and output more deterministic. Values above 1.0 flatten the distribution, increasing diversity and creativity but also raising the risk of incoherent or factually incorrect text.TokenizationTokenization is the process of splitting raw text into discrete units — tokens — that a language model can process. Modern LLMs use subword tokenizers that break text into fragments smaller than words but larger than individual characters. Each token maps to an integer ID, which is then passed to the embedding layer. The tokenizer's vocabulary determines which character sequences receive single tokens; rare strings are split into multiple tokens, making them more expensive to process and generate.Top-K SamplingTop-K sampling restricts next-token selection to the K tokens with the highest logit scores, zeroing out all other candidates before applying softmax and sampling. It prevents the model from selecting very low-probability tokens that could derail coherent generation. A small K value produces conservative, focused output while a larger K introduces more variability. Top-K is often combined with top-P sampling and temperature for fine-grained decoding control.Top-P SamplingTop-P sampling — also called nucleus sampling — selects the next token by restricting the candidate set to the smallest subset of tokens whose cumulative probability mass equals or exceeds a threshold P, then sampling uniformly from that subset. Unlike top-K, which uses a fixed count, top-P adapts the candidate set size to the model's confidence: for peaked distributions the nucleus is small, and for flat distributions it is large. Values around 0.9–0.95 are common in practice.TransformerThe transformer is a neural network architecture built around self-attention rather than recurrence. It processes all tokens in a sequence simultaneously, letting each position attend to every other position in parallel. Introduced in the 2017 'Attention Is All You Need' paper, the transformer became the dominant architecture for language modeling, machine translation, and most modern large-scale AI systems because it scales efficiently with data and compute.