The NLP Engineer's System Design Interview Guide

NLP System Design Is More Than Model Selection

Most NLP interview candidates prepare by memorizing transformer architectures. Interviewers, however, are looking for something different: the ability to reason about an end-to-end system where the model is just one component among many. The query understanding layer, the index design, the reranking stage, the fallback strategy — these are the parts of the system that determine whether a product ships successfully.

This guide walks through the design decisions that appear again and again in NLP system design rounds: how to represent text, how to retrieve and rank candidates, how to classify or extract information reliably, and how to serve all of this within latency budgets that real users actually tolerate.

Representation Choices and Their Downstream Consequences

The first decision in almost every NLP pipeline is representation: how text becomes a vector or a token sequence that a model can consume. Sparse representations like BM25 are fast and interpretable; dense embeddings capture semantic similarity but require an approximate nearest-neighbor index and careful training data curation. Interviewers want to hear you reason about this tradeoff explicitly rather than defaulting to whatever is currently popular.

• Sparse (BM25, TF-IDF): fast, lexical match, no GPU at query time
• Dense (bi-encoder): semantic search, needs ANN index (FAISS, ScaNN, Vespa)
• Cross-encoder reranking: highest quality, too slow for first-stage retrieval
• Hybrid: combine sparse and dense scores; re-rank the merged candidate set