Softmax

Softmax is a function that converts a vector of raw scores (logits) into a probability distribution. It exponentiates each score and normalizes by the sum of all exponentiated values, ensuring outputs sum to one and are all positive. In language models, softmax is applied to the final layer's logit vector to yield per-token probabilities. Temperature scaling modifies the logits before softmax is applied, controlling how peaked or flat the resulting distribution is.