← All chapters
Classical AI, NLP & Linguistics
Foundational NLP and machine-learning concepts that underpin modern systems.
22 terms
Bag of WordsThe bag-of-words model represents a document as an unordered multiset of its tokens, ignoring word order and sentence structure. Each document becomes a vector in vocabulary-sized space where each dimension counts or weights term occurrences. Despite its simplicity, bag-of-words paired with TF-IDF weighting is a strong baseline for text classification, sentiment analysis, and document retrieval. Its main limitation is that it discards all positional and semantic relationships among words.Cosine SimilarityCosine similarity measures the angle between two vectors in a high-dimensional space, computed as their dot product divided by the product of their magnitudes. It ranges from -1 to 1, with 1 indicating identical direction regardless of vector length. In NLP, cosine similarity is the standard measure for comparing document representations — TF-IDF vectors, word embeddings, or sentence embeddings — because it is insensitive to document length and captures directional similarity, which corresponds to semantic relatedness in well-trained embedding spaces.Cross-ValidationCross-validation is a model evaluation technique that partitions a dataset into K equal folds and trains K models, each leaving one fold out as a validation set. Metrics are averaged across folds to produce a more reliable performance estimate than a single train/test split. It is especially valuable in NLP when labeled data is scarce, ensuring that evaluation results are not unduly influenced by a single favorable or unfavorable data partition. The most common variant is K-fold cross-validation with K between 5 and 10.Dependency ParsingDependency parsing analyzes the grammatical structure of a sentence by identifying directed relationships between words. Each word except the root is connected to a head word by a labeled arc indicating the syntactic role — subject, object, modifier. The resulting dependency tree exposes predicate-argument structure useful for information extraction, question answering, and semantic role labeling. Classical parsers used shift-reduce or graph-based algorithms; modern parsers use deep learning and achieve near-human accuracy on standard treebanks.Feature EngineeringFeature engineering is the process of transforming raw text into numerical representations that capture the information most useful for a given task. In classical NLP, it encompasses decisions about tokenization, n-gram ranges, TF-IDF weighting, POS-tag inclusion, entity type indicators, and character-level features. Good feature engineering required deep domain knowledge and could dominate model accuracy; one of the major shifts brought by deep learning is that end-to-end models learn useful representations without manual feature design.Hidden Markov ModelA Hidden Markov Model is a probabilistic model of sequences where the system transitions between hidden states, each of which emits observable symbols according to learned probability distributions. In NLP, HMMs were the dominant approach for part-of-speech tagging and speech recognition before neural methods: hidden states represent grammatical tags or phonemes, and observations represent words or acoustic features. The Viterbi algorithm efficiently finds the most probable hidden state sequence given an observed sequence.LemmatizationLemmatization maps inflected word forms to their canonical dictionary form — the lemma — using linguistic knowledge of morphology and part-of-speech. Unlike stemming, it preserves valid words: 'better' lemmatizes to 'good', and 'ran' to 'run'. Lemmatization is more accurate than stemming but requires a lexicon and part-of-speech disambiguation, making it slower and more language-specific. It improves vocabulary consistency in downstream NLP tasks such as classification and information extraction.Logistic RegressionLogistic regression is a linear classification model that computes a weighted sum of input features, passes the result through a sigmoid or softmax function, and outputs class probabilities. It is one of the most widely used classical baselines for text classification because it is fast to train, interpretable through its feature weights, and regularizable to handle high-dimensional sparse feature spaces like TF-IDF vectors. Its linear decision boundary is a limitation for tasks with complex feature interactions.N-GramAn N-gram is a contiguous sequence of N tokens drawn from a text. Unigrams are individual words, bigrams are adjacent pairs, and trigrams are three-word sequences. N-gram models estimate the probability of a word given the preceding N-1 words, forming the statistical basis of language modeling before neural methods. N-gram features are also used directly in bag-of-words pipelines to capture local phrase context that unigram representations miss entirely.Naive Bayes ClassifierA Naive Bayes classifier applies Bayes' theorem to compute the probability of a class given observed features, under the 'naive' assumption that features are conditionally independent given the class label. Despite this rarely true assumption, Naive Bayes performs surprisingly well on text classification tasks because the violation is often mild and the model requires very little training data. It remains a strong, interpretable baseline for spam detection and sentiment analysis.Named Entity RecognitionNamed entity recognition is the task of identifying and categorizing spans of text that refer to real-world entities such as people, organizations, locations, dates, and monetary values. Classical NER systems used hand-crafted features with conditional random fields or hidden Markov models; modern systems use fine-tuned transformers. NER is a core component in information extraction pipelines, search engines, and knowledge graph construction, where structured entity data is required from unstructured text.OverfittingOverfitting occurs when a model learns patterns specific to its training data so thoroughly that it fails to generalize to new examples. In classical NLP, overfitting is common when vocabulary-sized feature vectors have far more dimensions than training examples. Countermeasures include regularization (L1/L2 penalties), dropout, data augmentation, and cross-validation to detect the gap between training and validation performance before deploying a model.Part-of-Speech TaggingPart-of-speech tagging assigns a grammatical category — noun, verb, adjective, determiner, etc. — to each token in a sentence. POS tags provide shallow syntactic structure that helps downstream components such as parsers and named entity recognizers disambiguate word meanings and identify phrase boundaries. Classical POS taggers used maximum-entropy models or hidden Markov models trained on annotated corpora; neural taggers now achieve near-human accuracy on standard benchmarks.Precision and RecallPrecision measures the fraction of predicted positives that are truly positive, while recall measures the fraction of actual positives that are correctly identified. The two metrics trade off: high-precision systems avoid false positives but may miss many true positives, while high-recall systems catch most positives but generate more false alarms. The F1 score combines them as the harmonic mean. Precision and recall are especially important in NLP tasks like named entity recognition and information retrieval where both false positives and false negatives carry distinct costs.Sequence LabelingSequence labeling assigns a discrete label to each token in an input sequence. Named entity recognition and part-of-speech tagging are canonical examples. Classical approaches used Hidden Markov Models or Conditional Random Fields that model dependencies between adjacent labels; neural approaches use BiLSTMs or fine-tuned transformers. The BIO tagging scheme — Beginning, Inside, Outside — is a standard encoding that allows multi-token spans to be represented as per-token label sequences.StemmingStemming reduces words to a common root form by stripping suffixes using rule-based heuristics, without reference to the word's grammatical context. The Porter stemmer, for example, converts 'running', 'runs', and 'runner' all to 'run'. Stemming is fast and language-independent but imprecise: it can produce non-words and conflate distinct terms. It is commonly used in search indexing where recall matters more than exact morphological accuracy.Stop WordsStop words are high-frequency function words — such as 'the', 'is', 'and', and 'of' — that carry little semantic content and are often removed during text preprocessing to reduce noise and vocabulary size. Removing stop words shrinks feature vectors, speeds up processing, and can improve precision in keyword-based retrieval by eliminating common terms that would dominate frequency-based scores. However, some tasks such as authorship attribution or detecting negation require stop words because grammatical function words carry signal.Text ClassificationText classification assigns one or more predefined category labels to a piece of text. Examples include spam detection, sentiment analysis, topic categorization, and intent recognition. Classical pipelines vectorize text with bag-of-words or TF-IDF features and train Naive Bayes or logistic regression classifiers. Modern approaches fine-tune transformer encoders on labeled examples. Evaluation requires careful attention to class imbalance, since accuracy can be misleading when one class dominates the dataset.TF-IDFTF-IDF is a numerical statistic that weighs the importance of a term within a document relative to a corpus. The term-frequency component scores how often a word appears in the document; the inverse-document-frequency component discounts words that appear in many documents, down-weighting common words like articles and prepositions. Multiplying the two components yields scores that highlight words distinctive to a particular document, making TF-IDF a robust feature representation for text classification and information retrieval.Tokenization (Classical)Classical tokenization splits raw text into discrete tokens — typically words or punctuation marks — using rule-based patterns or regular expressions. Unlike modern subword tokenizers, classical tokenizers operate at the word boundary level, treating whitespace and punctuation as delimiters. The choice of tokenization rules has downstream effects on every pipeline component: different word-splitting conventions produce different feature sets, making tokenization a foundational preprocessing decision in classical NLP systems.Word EmbeddingA word embedding is a dense, low-dimensional vector representation of a word learned from large text corpora. Unlike sparse one-hot or bag-of-words vectors, embeddings encode semantic and syntactic relatedness as geometric proximity. Pioneered by methods like Word2Vec and GloVe, word embeddings became the dominant text representation in pre-transformer NLP, serving as input to recurrent networks and convolutional text classifiers. Contextual embeddings from transformers later superseded static word embeddings for most tasks.Word2VecWord2Vec is a shallow neural network model trained to predict a word from its context (CBOW) or to predict context from a word (Skip-Gram). Training on large corpora produces dense word vectors where semantically related words cluster together and analogical relationships emerge as linear offsets — the famous 'king − man + woman ≈ queen' property. Word2Vec was a landmark advance in word embeddings, predating transformers, and its representations remain useful as input features for classical and lightweight NLP models.