Machine Learning Engineer

I try to reach a precise prediction statement before anyone opens a notebook: what entity we are scoring, at what moment, using only information available at that moment, and what decision changes as a result. Reduce churn is not a problem statement. Predicting for each active subscriber on the first of the month the probability they cancel within sixty days, so retention can target an offer, is. That framing immediately surfaces the hard parts. The prediction moment fixes which features are legal. The horizon fixes the labelling window and how long we must wait to observe ground truth. The decision fixes the metric, because a model feeding a fixed-capacity outreach queue is judged on precision at the top of the ranking, not on overall accuracy. I also ask what happens today without a model, since the existing rule is my baseline and occasionally it is already good enough. If nobody can tell me which action the score triggers, I push back — a model with no decision attached gets built, admired, and never used.

Leakage is any feature encoding information the model would not have at prediction time, and it announces itself as a validation score that feels too good. The obvious cases are columns written after the outcome: an account status field updated when the customer cancels, or a timestamp of the very event being predicted. The subtler ones do more damage — aggregates computed over the whole dataset before splitting, so target statistics from validation rows bleed into training encodings; a join pulling in a table refreshed after the label date; identifiers correlated with the label because of how rows were collected. My routine is to sort features by importance and interrogate the top few one at a time, asking when the value is written and by which process rather than trusting the schema. I also run a time-based holdout even when the problem is not framed as a time series, because leakage usually collapses under that split. Then I compare offline performance against the first production week; a large unexplained drop is leakage until proven otherwise.

The default of shuffled k-fold is wrong here, and the two reasons compound. When several rows belong to one entity — sessions from a user, images of a patient — random splitting puts near-duplicates on both sides and the score measures memorisation, so I group by entity and keep whole groups inside a fold. When the model predicts forward in time, I use forward chaining: train on everything before a cutoff, validate on the window after it, roll the cutoff, average across folds. Where both apply, group first and then order by time, and insert a gap between train and validation equal to the label maturation window, otherwise labels observed during the training period leak into the validation period. The tradeoff is fewer effective folds and a noisier estimate, particularly at early cutoffs where training data is thin. I accept that noise because the alternative estimate is confidently wrong. Every preprocessing step gets fitted inside the fold too, since scalers and target encodings fitted on the full set are a quiet form of leakage.

That is less a modelling problem than a measurement problem. Accuracy is meaningless at that rate, and ROC AUC flatters you because the false positive rate barely moves when negatives dominate, so I look at precision-recall AUC and at precision at whatever review capacity actually exists. For training I would first try the model as is with class weights, since gradient boosted trees handle skew better than people expect, and only reach for resampling if that stalls. If I resample, it happens inside the cross-validation fold and never before the split, and I remember it distorts predicted probabilities, so calibration follows on untouched data. Synthetic oversampling rarely helps me on tabular fraud data; it interpolates between points in a space where the minority class is not convex. The larger wins usually come from better features around the rare event and from the label definition itself — confirmed chargebacks and flagged-then-reversed transactions are different targets, and mixing them makes the minority class incoherent before any model sees it.

Two baselines, both before touching a real model. The first is the incumbent: whatever rule, heuristic, or human process makes this decision today, measured on the same data with the same metric. Teams skip it and then cannot answer whether an AUC of 0.82 is a win, which is the only question that matters. The second is a trivial statistical baseline — predict the base rate, predict the previous value for a time series, threshold the single strongest feature — because it shows how much signal is cheaply available and catches leakage early. If logistic regression on ten obvious features lands within a point of the sophisticated model, the sophisticated model is not paying for its complexity. If it beats the trivial baseline by an implausible margin, I hunt for a leak before celebrating. Baselines also force the evaluation harness into existence while the stakes are low, so split logic, metric code, and data loading are shaken out before anyone argues about architecture. Roughly a day of work that saves weeks of ambiguity later.

The question I care about first is why the value is missing, because that determines the treatment. Missing because a sensor dropped out is different from missing because the customer has no history yet, and the second case is often the most predictive thing on the row. So I add explicit indicator columns for the informative cases rather than silently filling them in. For tree ensembles I usually let the model handle missingness natively, since LightGBM and XGBoost learn a default direction per split and that beats mean imputation on most tabular problems I have worked on. Where imputation is genuinely needed it is fitted inside the fold and shipped as part of the model artifact. The serving half is where teams get burned: training reads a warehouse table where late-arriving data has since filled in, while serving reads a live store where it has not, so the same customer looks dense offline and sparse online. I monitor missingness rate per feature across training and live traffic, and treat divergence as an incident rather than a curiosity.

On tabular data with heterogeneous columns and a few million rows, boosting is usually my first choice and often my last. Trees handle mixed scales, monotone-but-nonlinear relationships, missing values, and high-cardinality categoricals with target encoding, and they do it without much tuning; a well-fitted LightGBM is a strong opponent that deep tabular architectures have repeatedly failed to beat convincingly. It also trains in minutes, which matters more than people admit, because iteration speed determines how many feature ideas actually get tested. I switch to neural networks when the input is genuinely unstructured — text, images, audio, raw sequences — when I need representation learning that transfers across tasks, or when one model must consume several modalities jointly. Embedding-heavy problems like large-scale recommendation are the honest middle ground. The other consideration is serving: a tree model is a small artifact with predictable single-digit-millisecond latency on CPU, while a neural model may need a GPU and batching to hit the same envelope. If the accuracy gain is a fraction of a point, that operational difference decides it.

It creeps in wherever the same feature gets computed twice. The classic version: the training feature is a thirty-day spend average written in SQL over a warehouse table, and the serving feature is the same concept written in application code over a live store, and the two disagree about timezone, about whether the current day is included, or about how refunds are treated. Nobody notices because both are individually plausible. The structural fix is one definition executed by one piece of code on both paths, which is what a feature store buys; without one I would at minimum share a transformation library rather than reimplement. Then I verify rather than assume: log the exact feature vector used at inference, join a sample back to the offline vector for the same entity and timestamp, and alert on per-feature divergence. That log is also the correct training source for the next model, since by construction it is what serving sees. The residual risk is time travel in offline joins, which point-in-time correctness has to handle explicitly.

I set the budget before starting and spend it in stages. Stage one is coarse random search over a wide space, because random beats grid — most hyperparameters barely matter and grid wastes its budget resolving them precisely. Thirty to sixty trials usually reveals where the good region is. Stage two narrows into that region with a Bayesian method or Hyperband, which kills unpromising trials early and earns its keep once a single fit takes more than a few minutes. I fix from experience the things not worth searching: for boosted trees I hold the learning rate low, use early stopping on a validation fold to choose tree count, and search depth, leaf count, subsampling, and regularisation. The discipline that matters most is a locked test set the search never touches, because searching hundreds of configurations against a validation set will overfit it and the reported gain evaporates in production. I also weigh the honest counterfactual: if two days of tuning buys half a point and one new feature buys three, tuning was the wrong place to spend the week.

AUC is invariant to monotone transformations, so a model can rank perfectly and still emit numbers nowhere near probabilities — and if downstream logic multiplies that score by an expected loss, the difference is the entire decision. I check with a reliability diagram: bin the predictions, plot observed frequency against mean predicted probability, and read expected calibration error alongside it. Boosted trees trained on log loss are usually close but overconfident at the extremes, and anything trained with resampling or aggressive class weights is systematically shifted. The fix is a calibration model fitted on data the base model never saw: Platt scaling when the distortion looks sigmoidal and data is limited, isotonic regression when I have tens of thousands of held-out examples and the distortion is irregular. I check calibration per segment as well as globally, because a model can be well calibrated overall and badly off for a subgroup that averages out. Calibration also drifts faster than ranking does, so refitting the calibrator on recent data is often a cheap alternative to full retraining.

Flat is informative, so I would resist retraining and instead find the break. First check is whether the new model is genuinely serving: traffic split correct, right artifact deployed, no fallback path silently absorbing requests. Then I compare the online score distribution against offline predictions on the same population, since a shift there points at features rather than at the model. Next I ask whether the offline gain was real — was the split time-based, did the search overfit the validation set, was there leakage inflating the lift. Then the metric chain. Ranking AUC improving while the business metric stays flat usually means the improvement landed where it does not matter: better separation deep in the tail, while the top of the ranking, which is all the user ever sees, is unchanged. So I recompute the offline metric restricted to the positions the experiment actually exposes. Finally power. Many teams read flat when the test could never have detected the effect size they hoped for, so I check the minimum detectable effect before concluding anything.

I separate three things that all get called drift. Covariate shift is the input distribution moving, which I watch per feature with population stability index or a distance measure over recent versus reference windows. Concept drift is the relationship between features and label changing, which only labels reveal. Data quality breaks — a nulled column, a renamed category, an upstream schema change — look like drift but are bugs, and they are by far the most common cause. Cadence follows label latency and business rhythm rather than calendar habit. If labels mature in a week, weekly retraining is cheap and I would automate it behind a promotion gate where the challenger must beat the incumbent on a recent holdout by a stated margin. If labels take ninety days, monthly retraining is theatre, and I would monitor proxy metrics and lean on calibration refits instead. I also keep a shadow-scored slice with delayed ground truth so true performance can be plotted over time, because drift alarms without a performance signal produce noise teams quickly learn to ignore.

I would cut in cheap passes before doing anything clever. First, mechanical removal: near-zero variance, near-duplicate columns, features whose missingness is so high that even the indicator carries nothing. Second, and more important than any statistic, a legality audit — for each family of features, when is this value written relative to prediction time. In a set that size there are almost always a few post-outcome columns, and they will dominate importance and quietly ruin the model. Third, a fast model on everything with permutation importance or SHAP on a holdout, grouping correlated features so importance is not split across near-copies. Then I take the smallest set landing within a small margin of full performance, usually somewhere between forty and a couple of hundred features. The reason to be aggressive is operational: every feature is a pipeline that can break, a serving-time dependency, and one more thing to monitor. I would trade a quarter point of AUC to avoid owning eighteen hundred production dependencies. The dropped set stays documented so the decision is revisitable.

The threshold is a business decision expressed in the model's units, so I start from costs rather than from the ROC curve. I write down what a false positive costs and what a false negative costs — a wrongly blocked transaction means a support contact and some churn risk, a missed fraud means the transaction value plus a chargeback fee — and pick the point maximising expected value against a calibrated probability. That immediately shows why calibration matters, and why a single global threshold is often wrong: if the cost of a miss scales with amount, the right cut varies by amount, and the clean formulation thresholds expected loss rather than probability. Where capacity is the binding constraint instead — a review team handling four hundred cases a day — the threshold is simply whatever fills the queue, and the metric becomes precision at that volume. I choose it on a validation window, confirm on a later one, and monitor the realised positive rate in production, since a fixed threshold on a drifting score distribution silently moves the operating point.

The first thing I want is agreement statistics per class on a shared subset — Cohen or Fleiss kappa rather than raw agreement, since raw agreement flatters an imbalanced task. Low agreement concentrated on one class usually means the guideline is ambiguous, not that the annotators are careless, so the fix is a sharper definition plus adjudicated examples rather than simply buying more labels. I would also check whether disagreement clusters on genuinely hard cases; those are worth keeping as soft labels or confidence weights rather than forcing a majority vote that pretends certainty. Then I use the model to help: train on cross-validated folds and inspect the examples the model is most confident are mislabelled. That set mixes real errors with interesting edge cases, and reviewing a few hundred of them is usually the highest-value hour available. The ceiling matters too. If two humans agree eighty-five percent of the time, a model scoring ninety against those labels is fitting noise, and I would say so before anyone sets a target above the ceiling.

Reproducible means someone rebuilds the artifact from a commit hash, so everything variable has to be pinned somewhere durable. Data first: training reads a snapshot or a versioned table through a point-in-time query, never a live table, because the same query rerun next year against a mutable warehouse silently returns different rows. Configuration lives in a file in the repository, not in notebook cells or flags somebody typed once. Seeds are set for every source of randomness — splitting, sampling, initialisation, and the training library itself — while acknowledging that GPU non-determinism makes bitwise identity unrealistic, so I define reproducible as within a stated tolerance on the test metric. Every run writes a record: code version, data snapshot identifier, config, environment, metrics, artifact hash. The step people skip is making the pipeline the only path to a shippable model, so nobody can promote something trained on a laptop. I would also add one test that trains end to end on a small fixture in continuous integration, because a pipeline exercised only on the full dataset rots quietly.

This is the hardest bias to see, because the model looks like it is improving. If we only observe outcomes for items the model surfaced, the training set becomes a picture of the model's own beliefs, and anything scored low never gets a chance to prove otherwise. In credit you never learn how the rejected applicants would have performed; in a ranker, click data is dominated by position rather than relevance. The remedy is deliberate exploration: hold out a small randomised slice, a couple of percent of traffic scored randomly or with a stochastic policy, and treat it as the unbiased evaluation set. It costs real money, so I frame it to leadership as the price of knowing whether the model works at all. Where exploration is impossible, propensity weighting using the logged score partially corrects the training distribution, though it is fragile once propensities approach zero. I also log the score and policy version with every decision, because without those the selection mechanism cannot be reconstructed and the correction becomes impossible rather than merely difficult.

My default is one model with segment identity as a feature, and I make the other side argue its way in. A single model pools data, so small segments borrow strength from large ones, and trees or embeddings will learn segment-specific behaviour wherever the data supports it. Splitting fragments the data, and the segments that most need help are exactly the ones left with too few rows to fit anything stable. The case for separate models is real when segments have genuinely different feature availability or different label semantics — a market where the outcome is defined differently, or a product line with features that do not exist elsewhere — or when regulation demands an independently auditable model per jurisdiction. Operationally, one model is one thing to monitor, retrain, and roll back; forty models mean forty ways to go stale and a monitoring surface nobody watches. When I do split, I usually go hybrid: a shared model plus a per-segment residual correction, so large segments get specialisation without stranding the small ones.

Aggregates hide almost everything interesting, so I define slices before looking, based on where harm would actually land: geography, tenure, device, language, low-volume segments, and any protected attribute we are permitted to evaluate on. Performance gets reported per slice with confidence intervals, because a small slice bounces around and somebody will over-read a difference that is pure noise. I look beyond the headline number — calibration per slice matters as much as ranking, since a model systematically overestimating risk for one group creates a different harm than one that merely ranks worse for them. Where the attribute is not collected, proxies are unreliable enough that I would rather state the limitation than publish a number I do not trust. The gating question is what to do about a gap, and that is a product decision rather than a modelling one: sometimes more data for that slice, sometimes a per-slice threshold, sometimes declining to serve a segment the model handles badly. The slice table goes in the model card so the tradeoff is documented rather than discovered later.

Most wrong conclusions I have seen came from process, not statistics. So I standardise the parts that should not be creative: one evaluation harness in the repository that everyone calls, one canonical split, one locked test set with a rule that it is touched once per milestone. If six people each write their own split, the numbers in the team channel are not comparable and nobody notices for a month. I ask for the baseline number in every writeup, because a result without the comparison it beats is not a result, and for a confidence interval or at minimum a repeat with a different seed, since half-point differences on a small validation set are noise that gets promoted as findings. Experiment tracking is mandatory but should happen automatically through the harness, because anything manual gets skipped under deadline. Culturally I make negative results cheap to report by asking about them in reviews, otherwise people quietly abandon failures and the team relearns the same dead ends. And whoever claims a win explains why it worked.