Layer Normalization

Also known as: LayerNorm

Layer normalization is a technique applied inside each transformer block that normalizes the activations across the feature dimension to have zero mean and unit variance, then rescales with learned gain and bias parameters. It stabilizes training by preventing the distribution of activations from shifting dramatically as gradients flow through deep networks. Modern large language models typically use RMSNorm — a simpler variant that omits the mean-centering step — for both training stability and inference efficiency.