Positional Encoding

Positional encoding injects sequence-order information into token representations before they enter the transformer layers. Because self-attention treats its input as a set rather than a sequence, position signals must be added explicitly. Early models used fixed sinusoidal functions; modern large language models typically use learned rotary position embeddings (RoPE), which encode relative distances between tokens and generalize better to sequence lengths beyond those seen during training.