Tokenization (Classical)

Classical tokenization splits raw text into discrete tokens — typically words or punctuation marks — using rule-based patterns or regular expressions. Unlike modern subword tokenizers, classical tokenizers operate at the word boundary level, treating whitespace and punctuation as delimiters. The choice of tokenization rules has downstream effects on every pipeline component: different word-splitting conventions produce different feature sets, making tokenization a foundational preprocessing decision in classical NLP systems.