Tokenization

Tokenization is the process of splitting raw text into discrete units — tokens — that a language model can process. Modern LLMs use subword tokenizers that break text into fragments smaller than words but larger than individual characters. Each token maps to an integer ID, which is then passed to the embedding layer. The tokenizer's vocabulary determines which character sequences receive single tokens; rare strings are split into multiple tokens, making them more expensive to process and generate.