Self-Attention

Self-attention is the mechanism by which each token in a sequence computes a weighted combination of all other tokens' representations. For every token, the model derives a query, a key, and a value vector. Dot products between queries and keys produce attention scores that are scaled and softmaxed into weights, which are then applied to the value vectors to produce the output. Self-attention lets the model capture long-range dependencies without the sequential bottlenecks of recurrent architectures.