Byte-Pair Encoding

Also known as: BPE

Byte-pair encoding is a subword tokenization algorithm that starts from individual bytes or characters and iteratively merges the most frequent adjacent pair into a new token until the vocabulary reaches a target size. The result is a compact vocabulary that assigns short token sequences to common words and longer sequences to rare words. BPE tokenizers handle arbitrary text including out-of-vocabulary words gracefully and are the basis for GPT-series tokenizers.