NLP Engineer

The failures cluster in four places. A byte-level BPE vocabulary trained mostly on English spends far more tokens per unit of meaning on other scripts, so a Devanagari or Thai sentence can cost three to five times what its English translation costs. That quietly shrinks the effective context window and inflates price for exactly the users we already under-serve. Second, scripts written without spaces get segmented on statistical boundaries the tokeniser learned rather than linguistic ones, so entity spans land mid-word and character offsets stop lining up with the source string. Third, normalisation: the same Vietnamese or Arabic string can arrive precomposed or decomposed and tokenise completely differently, which is why I normalise to NFC at ingestion and record that I did. Fourth, agglutinative languages such as Finnish or Turkish shatter into fragments that respect no morpheme boundary. Before I commit to a tokeniser I measure fertility, meaning average tokens per word per language, and round-trip fidelity on held-out text. Where fertility is bad I extend the vocabulary instead of accepting the tax.

I treat it as an ordered pipeline, because the steps interact. First, repair encoding damage: text encoded as UTF-8 and then decoded as Latin-1 leaves a recognisable signature, and a library like ftfy fixes most of it. Anything ambiguous gets logged rather than guessed at. Then Unicode normalisation, NFC by default. I reach for NFKC only when downstream matching demands it, because NFKC destroys information — it folds ligatures, superscripts, and the full-width Latin that carries meaning in Japanese. Next, strip control and zero-width characters except where they are load-bearing: a zero-width non-joiner in Persian or Devanagari changes the word, so a blanket strip is a bug. After that, whitespace collapse, quote and dash folding, and optional case handling, which I keep switched off for entity recognition since capitalisation is a genuine feature. The discipline that matters most is retaining an offset map back to the raw text, so highlights and annotations still point at the original document, and keeping the raw bytes so a corrected pipeline can be rerun later.

Token accuracy is close to meaningless here because the outside class dominates; a model that tags nothing at all scores in the nineties. I evaluate at span level with exact-match micro and macro F1 per entity type, and I always read the macro figure, because the rare type people actually care about — an adverse event, say — is invisible in the micro average. Then I decompose exact match into MUC-style categories: correct, right span with wrong type, boundary drift, spurious, missed. Boundary errors and type errors demand entirely different fixes, and collapsing them into one number hides which you have. I also compute a decision-level metric matching how the output gets used; if the consumer only needs to know whether a document mentions a drug at all, document-level recall matters more than span precision. Confidence intervals come from bootstrapping over documents rather than sentences, since entities cluster within a document and sentence-level resampling understates variance. Alongside all that I keep a small hand-built set of nested, hyphenated, and non-Latin names that never enters training.

Disagreement at that rate is a guideline problem, not an annotator problem, and I would say so early. Measure it properly first: Cohen's kappa or Krippendorff's alpha rather than raw agreement, which flatters you whenever one label dominates. Then build a confusion matrix over label pairs and read the largest cells. Two or three genuinely ambiguous distinctions usually carry most of the disagreement — whether a job title attached to a person belongs inside the person span, whether a product family counts as an organisation. Those get resolved in an adjudication session where annotators argue from real examples, and every decision lands in the guideline as a rule plus a positive and a negative example, because abstract rules do not transfer. Then I re-measure on a fresh double-annotated batch of a couple of hundred documents. My target depends on the task: alpha above 0.8 for named entities, lower is tolerable for subjective labels like stance, but at that point I stop pretending a single gold label exists and retain both annotations to model the ambiguity.

More often than the current mood suggests. When the pattern is genuinely closed and well-formed — ICD codes, IBANs, dosage strings, invoice references — a validated regular expression is exact, runs in microseconds, costs nothing per call, and will never invent an entity that was not in the text. A gazetteer plus fuzzy matching wins whenever the target set is enumerable and stable: country names, ticker symbols, a client's own SKU catalogue. Both give you determinism and an audit trail, which regulated buyers explicitly ask for. A CRF or small fine-tuned encoder remains my default for high-volume sequence labelling with a fixed label set and a single-digit millisecond budget per document; it trains on a few thousand spans and serves happily on CPU. The honest limits: rules are brittle against formatting drift and accumulate into an unmaintainable thicket unless one person owns them, and a CRF needs labelled data you may not have. My usual shape is hybrid — rules for the closed part, a learned model for the open part, and an explicit precedence rule at the merge.

I resist retraining until I know what changed. Start with the inputs: are these PDFs coming through a different extraction path, so text arrives with hard line breaks mid-sentence, headers interleaved with body copy, and tables flattened into word soup? That alone explains a surprising share of these collapses, and it is a parsing fix rather than a model fix. Next I read fifty failures by hand and bucket them — missed entities, boundary drift, wrong type, spurious spans. Then I quantify the shift numerically: vocabulary overlap against the training corpus, unknown-subword rate, mean sentence length, entities per thousand tokens. Falling entity density with stable precision usually means genre shift; falling precision with unfamiliar surface forms usually means the customer names things differently. I check casing too, because a corpus that arrives uppercased destroys a cased model outright and the symptom looks like general incompetence. Only after all that do I choose between better preprocessing, a few hundred in-domain annotations, or continued pretraining on the customer's unlabelled text, in that order of cost.

I would start by refusing to build twelve pipelines. One multilingual encoder, a joint intent-and-slot head, a shared label space, and language as a feature rather than a routing key — that buys cross-lingual transfer, so a Portuguese intent with two hundred examples borrows from the Spanish five thousand. Slots get BIO tags over subword tokens, mapped back to character offsets, because the consuming system needs the substring rather than token ids. On data, I would annotate English and one morphologically rich language properly, machine-translate the rest with slot-aware alignment, then have native speakers correct rather than author, which runs several times cheaper and produces errors you can see. The tradeoff I accept knowingly: a shared model underperforms a dedicated one on the highest-volume language, typically by a point or two of F1, and I pay that for maintainability until one market justifies its own head. Evaluation uses per-language test sets written natively, never translated, because translated tests conceal precisely the idioms that break the model, and releases gate on the worst language rather than the mean.

Clinical text is a different language wearing an English surface. Abbreviations are overloaded, so PT means physical therapy, prothrombin time, or patient depending on the service; sentences are fragments; and negation and uncertainty carry the clinical meaning. My order of operations starts with continued pretraining of the encoder on the hospital's own unlabelled notes, which is cheap and reliably worth several F1 points, largely because it repairs the subword vocabulary's fertility on drug and procedure names. Then targeted annotation, perhaps fifteen hundred notes chosen by stratified sampling across services rather than randomly, since cardiology and psychiatry notes barely overlap lexically. Terminology enters as an explicit feature rather than something the model must memorise, with spans linked to SNOMED or RxNorm in a normalisation step. What I will not skip is a negation and experiencer classifier over every extracted span; a family history of diabetes recorded as an active diagnosis is worse than missing it entirely. I also hold out one department's notes wholesale, to measure how far adaptation generalises even inside a single hospital.

Almost never for a model I intend to fine-tune, and occasionally for one I pretrain. Swapping the tokeniser invalidates the embedding matrix, so you either discard pretrained weights or perform a careful re-initialisation where each new token starts as the average of the old subwords that composed it. That recovers much of the loss but not all of it. The case that genuinely justifies the move is a domain where fertility is severe: SMILES strings, genomic sequences, or a language the original vocabulary never encountered, where you might be spending eight tokens on something that should cost two. There the payoff is both quality and a straight cut in sequence length, which compounds quadratically at attention. The cheaper middle path I try first is vocabulary extension — add a few thousand domain tokens, initialise them sensibly, continue pretraining briefly — since it preserves compatibility with the base checkpoint. What people underestimate is the operational tail: every cached embedding, stored token count, and prompt-length estimate downstream is now wrong, and that migration is the real bill.

Pairwise mention scoring is quadratic and a hundred-page document yields thousands of mentions, so pruning is the first design decision. I would run a mention detector to propose spans, keep the top candidates by score within each segment, and restrict antecedent search to a sliding window, since most links are local and the long-distance ones are dominated by a handful of salient entities. For those I maintain document-level cluster state that carries across segments, which is closer to an incremental entity-based approach than pure span-pair scoring. Practically, the easy wins deserve separating out: exact string match on proper names and head-word matching on definite descriptions already resolve most links and give a strong baseline. Pronouns are where models actually differ, and where gender and animacy assumptions cause real harm, so I evaluate pronoun resolution as its own slice. I report CoNLL F1 for comparability but gate releases on a downstream metric — does resolving coreference measurably improve the extraction task that motivated it? Frequently the honest answer is that it helps one relation type and nothing else.

Overlap metrics answer one narrow question — does this share n-grams with a reference — and they are near-useless whenever many outputs are valid. I build the evaluation in layers. Adequacy and fluency get judged separately on a small rated sample, because a fluent wrong answer and an awkward correct one need opposite fixes, and I source those ratings from native speakers with at least two raters and a published agreement figure. Then automatic checks aimed at the task's real failure modes: entity preservation between source and output, number and date fidelity, terminology compliance against the client glossary, and a check that the output is in the requested language at all, which fails more often than anyone expects on low-resource pairs. I like contrastive minimal pairs where one variant is ungrammatical — agreement, negation scope, word order — because they isolate a specific linguistic capability instead of averaging across everything. And I always inspect the length distribution, since systematic truncation shows up there long before it moves any score.

Segmentation sits upstream of nearly everything, so it is a frequent culprit. The usual damage is a rule-based splitter tripping on abbreviations and decimals, turning 'approx. 3.5 mg' into three fragments, after which a chunker grouping by sentence emits chunks that begin mid-clause. Quality falls because the embedded chunk no longer carries a complete proposition. I would verify directly rather than theorise: sample chunks from before and after, then compare the length distribution and count how many begin with a lowercase word or end without terminal punctuation. A sharp rise in either is the signal. Other common causes are PDF extraction inserting hard line breaks at the column width, so paragraphs split at arbitrary points, and a language change, since Thai carries no sentence-final punctuation and a Latin-trained splitter returns either the whole document or every clause. The fix I prefer is a language-aware, abbreviation-aware segmenter plus a chunker that respects structural boundaries such as headings and list items, and a regression test asserting chunk statistics stay within a band.

Five hundred is enough to evaluate and barely enough to train, so I spend them on evaluation first. A two-hundred-and-fifty-example test set I trust is worth more than five hundred noisy training examples, because without it I cannot tell whether anything I try afterwards helped. Then I chase transfer. A multilingual encoder fine-tuned on a closely related high-resource language often performs surprisingly well zero-shot when the scripts match; when they do not, transliterating into a shared script sometimes recovers most of the gap. Continued pretraining on whatever unlabelled text exists — news, religious texts, a crawl — is usually the highest-value cheap step available. Annotation projection is worth considering too: align a parallel corpus and carry labels across, accepting the alignment noise as a cost. Active learning over any remaining annotation budget, selecting on uncertainty plus lexical diversity, stretches a small effort further than random sampling. The limitation I state plainly to stakeholders is that variance at this data scale is enormous, so I report intervals and refuse to claim a two-point gain is real.

Linking is a retrieval problem followed by a disambiguation problem, and I keep the two separate so each can be measured. Retrieval means an alias index over the ontology covering synonyms, abbreviations, and spelling variants, with candidates generated by a mix of character n-gram matching and a dense embedding of the mention in context. I aim for recall at fifty above ninety-five percent, since nothing downstream can recover a candidate that never entered the list. Disambiguation is a cross-encoder scoring the mention in context against each candidate's name and definition, which handles the cases string similarity gets exactly backwards — 'cold' as a symptom versus a temperature, or an abbreviation shared by two drugs. The piece teams skip is a calibrated decision to link nothing. Many mentions genuinely have no entry, and a system forced to always choose emits confident nonsense, so I set a threshold against a target precision and route the remainder to review. I report accuracy at rank one on linkable mentions plus NIL precision and recall separately, tracked per entity type.

You measure it by building test sets that contain it, which is the whole job — models fail on dialect largely because the evaluation never included any. I construct stratified slices: standard written register, regional dialect, informal social text with non-standard orthography, and code-switched utterances, each with enough examples to detect a meaningful difference, so several hundred per slice as a floor. Then I report per-slice scores and treat the gap between best and worst as the headline number rather than the mean, because the mean conceals the users being failed. For code-switching I inspect tokeniser behaviour first, since Hinglish written in Latin script often fragments badly and the degradation is upstream of the model entirely. I look for systematic patterns: is it dropping entities from the non-dominant language, or misclassifying intent when the sentiment-carrying words sit in the other one? The honest constraint is sourcing — this data is hard to collect ethically and cheaply, and synthetic code-switching generated by a model reproduces that model's own biases, so I use it to expand a set, never to validate one.

Because the error cost is asymmetric while the training signal is sparse. In a clinical or legal corpus, 'no evidence of pneumothorax' and 'pneumothorax' contain the identical entity, and a model trained on span labels alone feels little pressure to separate them. Negated mentions may be a small slice of examples, so getting them wrong barely moves aggregate F1 while producing exactly the errors that destroy user trust. The linguistics is also tractable, which helps: negation cues form a fairly closed class and scope is largely predictable from syntax, which is why rule systems built over dependency parses stay competitive on English clinical text. I model three attributes per extracted span — polarity, uncertainty, and experiencer, meaning whether the finding belongs to the patient or a relative — as a separate classifier with its own labelled set and its own reported metrics, so it improves independently. The failures I watch for are long-distance and double negation, and coordination where a cue scopes over only the first conjunct; both rules and classifiers still degrade there, so I sample those cases deliberately.

I would organise it around a few shared components rather than per-market ownership, because thirty independent stacks means thirty maintenance burdens and no shared learning. At the core sits one multilingual representation layer — a shared encoder and a shared tokeniser whose vocabulary has been audited for fertility across all thirty languages — with task heads above it. Around that, three services everyone reuses: language identification with calibrated confidence and an explicit unknown class, normalisation with per-script rules, and a terminology service holding each market's approved vocabulary. Data flows into a single annotation platform with one guideline per task, translated and locally adjudicated, so agreement can be compared across markets and I can detect when a guideline fails to survive translation. The organisational half matters more than the modelling half: I would staff language leads owning quality for clusters of related languages, and set a policy that no feature ships until the weakest market clears an absolute floor rather than a relative one. The cost is speed, since English-first teams ship faster, and I would hold that line because retrofitting language support costs several times more.

Month one is definitions. I would not hire annotators before a task specification exists with a label set, a written guideline, and fifty adjudicated examples produced by people who understand the domain, because everything downstream inherits those decisions permanently. Then a pilot team of four to six annotators double-annotating everything, so agreement exists from day one, with a lead who adjudicates and owns guideline changes under version control — guidelines drift, and undated guidelines render old labels uninterpretable. Tooling next: pre-annotation from the current model so people correct rather than author, which roughly doubles throughput but biases work toward the model's existing mistakes, so I keep a blind subset with no pre-annotation to quantify that bias. By month three I want an active learning loop selecting on uncertainty and diversity, plus quality control through seeded gold items rather than spot checks. By month six the deliverable I care about is not volume but a stable agreement figure and a guideline a new annotator can reach competence on within a week. Pay and career path matter here; annotator churn is what actually degrades datasets.

Empirically, and willing to lose. For each of the nine I would put four numbers on the table: quality against the incumbent, cost per million documents, p99 latency, and how frequently the task definition changes. The pattern I usually find is that a general model wins on tasks with rich context and fuzzy boundaries — summarisation, open-ended classification, anything where the label set keeps moving — and loses badly on high-volume sequence labelling, where a small encoder is roughly a hundred times cheaper, ten times faster, and a point or two better, because the task is narrow and the training data sits exactly on distribution. The consolidation argument is real, though, and I would say so: nine models means nine evaluation suites, nine retraining schedules, and a hiring profile that is hard to fill. So my counter-proposal is normally partial consolidation to two or three, keeping the highest-volume encoders and moving the long tail across. I would also name the risk nobody prices, which is that a hosted general model changes underneath you while my nine deterministic models do not.

Aggregate scores go stale, both because teams learn to optimise them and because traffic shifts underneath. What survives is a maintained error taxonomy. Every reported failure and every sampled error gets classified into a category that maps to a fix owner — segmentation, tokenisation, recognition, linking, negation, language identification — and I track the distribution over time rather than a single number. That tells you where the next engineer-month should go, which no F1 ever does. Underneath it I keep three tiers of data: a frozen benchmark for comparability that is never trained on and never edited, a rotating sample of recent production traffic to catch distribution shift, and adversarial sets that grow whenever we fix a bug class, so regressions cannot return quietly. I also budget for periodic re-annotation of the frozen set, since gold data itself decays as guidelines evolve, and a benchmark whose labels contradict the current guideline actively misleads people. The organisational discipline that makes the whole thing work is publishing the slice table in every review, not the headline figure.