KV Cache

Also known as: KV cache, key-value cache

The KV cache stores the key and value tensors computed for all previous tokens during autoregressive decoding so they do not need to be recomputed at each new generation step. Without caching, generating a sequence of length N requires O(N²) attention computations; with caching, only the new token's keys and values are computed at each step. The KV cache grows linearly with sequence length and is a primary contributor to GPU memory consumption during inference with long contexts.