MLOps Engineer

Reproducible means I can rebuild the same artifact, not merely land on the same accuracy number. So I pin four things at launch: code by commit SHA, data by an immutable snapshot identifier rather than a table name, environment by a container digest rather than a mutable tag, and configuration by the resolved config the run actually used instead of the template it came from. All four are written into run metadata, and the job refuses to start if any of them cannot be resolved. Randomness gets a seed, but I do not pretend seeds buy determinism on GPUs; non-deterministic kernels and mixed precision give a slightly different loss curve, so I track a tolerance band rather than an exact match. The honest limit is data. If the feature store overwrites rows in place, nothing else I pin matters, and that is usually where reproducibility really dies. I test the claim instead of assuming it, re-running a randomly chosen month-old job each sprint and diffing the metrics. When that job cannot even be launched, the pipeline has drifted and I fix it before somebody needs it under pressure.

The merge starts a chain I want to be able to recite from memory during an incident. CI builds a container, runs unit tests on the transformation code, then trains against a pinned data snapshot in an isolated job with a resource ceiling. The resulting artifact lands in the registry as a candidate with its lineage attached. A separate evaluation job scores it against a frozen holdout plus behavioural checks: slice metrics for the segments that have burned us before, latency and memory under a realistic load profile, and a schema compatibility check. Only when those pass does a promotion job flip the registry stage, and that stage change is the single event deployment watches for. Serving picks up the version, starts a canary, and an automated comparison against the incumbent gates the ramp. Nothing in the chain is a human clicking a button except promotion approval, which I keep manual for a system's first few months and automate once the eval suite has caught something real. The whole path needs to finish inside an hour, because anything slower and people start finding ways to route around it.

Weights are the least interesting thing in there. Every version should carry the training data snapshot identifier, the code commit, the container digest, the resolved hyperparameters, the evaluation report that justified promotion, and an owner that is a team rather than a person who might leave. Then the operational fields: current stage, which environments have it loaded, the serving contract it expects, and a link to any incident where it was rolled back. The reason is mundane. At three in the morning nobody asks what the hyperparameters were; they ask what changed between the version that worked and this one, and a registry that stores only artifacts cannot answer that. I also keep input and output schemas beside the artifact so a serving process can refuse to load a model whose contract it does not understand rather than returning plausible garbage. The cost here is discipline. Registries rot quickly once writing to them is optional, so registration happens inside the training job itself, never as a follow-up step somebody is supposed to remember.

I treat prompts as deployable artifacts with the same lifecycle as weights, because operationally that is what they are. They live in the repository rather than in a database somebody edits through an admin panel, so every change carries a review and a commit SHA. At build time they compile into a versioned bundle that gets registered, and the running service resolves a bundle version from configuration instead of reading whatever is latest. That gives me the property I actually care about: withdrawal is a config change to a known-good version, not a revert plus a rebuild plus a cache flush. Each bundle is scored by an offline eval job over a fixed case set before promotion, and that report is attached to the version. Every request logs the resolved bundle version, so a support ticket can be traced back to the exact text the model saw. The friction point is speed, since product wants to adjust wording without a deploy. My compromise is a staged bundle they can point a small traffic slice at, under the same eval gate, rather than an unversioned live edit nobody can roll back.

A failure with a weekly period is almost never the model, so I would stop reading training code immediately. The question is what else in the system is periodic: weekend batch jobs landing late, a warehouse maintenance window, a retention policy dropping partitions, a cache expiring over a quiet period, or a credential rotation on a weekly cadence. I would pull the last eight Monday runs and the last eight Tuesday runs and diff them at the level of inputs first, comparing row counts per partition, freshness timestamps, and schema hashes, before diffing anything else. My prior is that an upstream job produced a partition with zero or partial rows because Sunday traffic is thin, and a join or a validation step then failed on empty input. The fix is rarely to make training tolerant of that. It is to make the dependency explicit, so the job waits on a data-readiness signal instead of a wall-clock schedule and fails with a clear message about the input rather than a stack trace deep inside a feature transform. Then I would add a freshness assertion so the next periodic surprise is caught by the pipeline.

I split it into four bands so whoever is paged can triage in thirty seconds. Service health first: request rate, error rate by class, p50 and p99 latency, and saturation of the accelerator or the queue in front of it. Then input health: feature null rates, schema violations, and distribution distance against the training reference for the handful of features that genuinely move predictions. Then output health: prediction distribution, score histogram shifts, and the rate of fallback or default responses, which is often the earliest honest sign something upstream broke. Last, business proxies with whatever lag they carry, clearly labelled as delayed so nobody expects them to move within minutes. The important design choice is that only the first two bands page anyone. Business metrics move for a hundred reasons and paging on them destroys trust in the alerting within a fortnight. I also pin the currently serving model version and prompt bundle version to the top of the dashboard, because the second question in every incident is what changed, and answering it should not require writing a query.

The pipeline's job is to make promotion boring. Stage one is ordinary software CI: lint, type checks, and unit tests on feature transforms with golden fixtures, because most of the model bugs I have chased turned out to be transform bugs. Stage two builds the image and records its digest. Stage three trains on a pinned snapshot inside a job with a resource ceiling and a wall-clock timeout, then registers a candidate. Stage four is the real gate: offline metrics against the incumbent on a frozen holdout, per-slice metrics with a tighter regression threshold than the aggregate one, a behavioural suite of cases we have broken before, and the non-quality checks people forget, meaning artifact size, cold-start time, memory ceiling, and throughput at target batch size. Stage five verifies that the current serving binary can load the artifact and answer a fixture request in the expected shape. Failures block promotion rather than the merge, so research can still land work. I insist that every threshold lives in the repository with an owner and a comment explaining the number, or it gets quietly relaxed the first time it is inconvenient.

Shadow means the candidate sees real traffic and its output never reaches a user. Concretely, the serving layer forks the request after feature resolution and sends it to the candidate asynchronously on a separate pool, so a slow candidate cannot add latency to the live path, with a hard timeout and a circuit breaker. I compare three things: agreement with the incumbent at the top of the ranking, latency and resource profile under real traffic shape rather than a synthetic load test, and error and timeout rates. What shadow cannot give me is a business metric, since nobody clicked the shadow's ranking, so I say plainly that it de-risks the mechanics and not the value, and the value question still needs a live experiment. Two traps I plan for up front. Side effects, meaning the candidate path must have writes, counters, and billing pointed at a sink. And feedback contamination, where the candidate's logged requests get swept into tomorrow's training set. Both have bitten teams I have worked in. I would run at least one full weekly traffic cycle before trusting the latency numbers.

It depends on whether the risk is stateless or experiential. For changes where each request is independent, such as a new serving image, a quantised artifact, or a library bump, percentage of traffic is right: it gives fast statistical signal on latency and errors, and any single user sees a bad response once. For anything a user experiences across a session, cohort routing is right, because random per-request routing produces incoherent behaviour. A user gets the new model's answer, then the old one's, and files a bug that neither version reproduces. Personalisation, conversational products, and anything with caching or memory sit in that bucket. In practice I run both: cohort routing for coherence, and a percentage ramp on the cohort's size. The price of cohorts is a worse sample, so if I hash on user ID I check the canary group is not skewed toward one region or device class before drawing conclusions. I also make sure rollback unsticks users immediately, because sticky routing that survives a withdrawal quietly becomes its own incident the following morning.

First I confirm the bump is actually responsible by checking whether p99 recovers on the previous image; a rollback is a diagnostic here, not only a remedy. Tripled p99 with a stable p50 tells me this is a tail problem, so I am hunting for something that occasionally takes a very different path rather than a uniform slowdown. My checklist: did the build lose a fused kernel or fall back to a slower attention implementation, did thread-pool or intra-op parallelism defaults change and start contending with the serving process, did a numerical library swap to a different BLAS with different threading, did tokenisation move from a compiled path to a Python one, or did resident memory grow enough to increase allocation pressure and garbage collection. Rather than reason about it, I would take a flame graph from a canary pod under real traffic and diff the fully resolved dependency tree between the two images, which usually reveals a transitive package that jumped several versions. The durable fix is a latency and memory gate in CI at target batch size, since this class of regression is completely invisible to correctness tests.

Reverting the binary is the easy half; the data the bad version emitted is the hard half, and I plan for it before the deploy rather than during the incident. The design rule is that anything a model writes downstream carries the producing model version, which turns recovery into a query, find rows written by version N and decide what to do with them, instead of an archaeology project. For scoring systems I prefer downstream state to be recomputable: keep the raw inputs, treat scores as a derived and disposable layer, and recovery becomes a backfill with the restored version. Where writes are genuinely irreversible, a message sent, a ticket raised, money moved, I do not rely on rollback at all. Those paths get a slower ramp, a human approval gate, or an idempotency key that lets me suppress duplicates. The step teams skip is telling the consumers. If a downstream team aggregated or trained on the bad scores, my rollback has silently corrupted their system, so the incident is not closed until the affected partitions are marked and every consumer has acknowledged the window.

Inference wins, always, and I would rather state that as policy than discover it during a traffic spike. Practically I partition into a guaranteed inference pool sized for peak plus headroom, a smaller burst pool, and everything remaining as preemptible capacity for training. Training runs at lower priority with mandatory checkpointing and a required tolerance for eviction; a job that cannot survive being killed does not get to use spare capacity. Gang scheduling matters for multi-GPU training, since a partial allocation deadlocks the queue while holding cards idle, so the scheduler admits a job only when the whole group is available. Fairness is per-team quota with borrowing, where an idle team's share is lendable but reclaimable inside a bounded window. The failure mode I watch for is fragmentation, plenty of single free GPUs and no free node with eight, which silently destroys throughput, so allocatable whole-node count is a first-class metric on my dashboard. I would also pack small models onto shared cards with memory limits rather than giving every service a dedicated accelerator, since that is where most of the waste actually accumulates.

Degraded needs an operational definition or it turns into a matter of opinion mid-incident. I define it as a measurable change we can observe within minutes and attribute to the model: the fraction of requests hitting a fallback, the rate of outputs failing a hard validity check such as schema conformance or refusal detection, a shift in the score distribution beyond a band derived from historical variation, or a drop in an immediate implicit signal like acceptance rate where the lag is short enough to trust. Those get thresholds and those page. Everything slower, weekly quality review, input feature drift, a labelled sample audit, produces a ticket instead, because waking somebody for a metric they cannot move at three in the morning trains the team to ignore the pager. I set thresholds by replaying the last six months rather than picking round numbers, so I know the false-page rate I am signing up for. Every page links a runbook whose first step is comparing the deployed model and prompt versions against the last known-good pair, since a recent change explains far more incidents than genuine drift does.

I run a schedule and use drift monitoring as an alarm rather than a trigger. The reason is operational. A schedule gives me a predictable, continuously exercised path: the pipeline runs weekly whether or not it is strictly needed, so it stays working, capacity is planned, and evaluation is routine. Drift-triggered retraining sounds efficient and then fires at midnight during a holiday traffic anomaly, trains on a week of unrepresentative data, and pushes a worse model toward production. The trigger also tends to be miscalibrated, since population drift and performance degradation are genuinely different things and features move constantly without the model getting worse. So drift alerts route to a human who decides whether to pull the retrain forward. The exception is adversarial domains, fraud and abuse in particular, where the label distribution shifts within days and a fixed weekly cadence loses real money; there I shorten the cycle instead of making it conditional. Either way the promotion gate is unchanged. A model ships because it beat the incumbent on the holdout and the slice metrics, not because a calendar or a detector fired.

The reliable fix is removing the opportunity for skew, not detecting it later. That means one implementation of each transformation, shared as a library between the training job and the serving path, and feature definitions that live in one place with point-in-time correct reads for training. Where unification is impossible, an online path in one language and an offline path in another, I add a parity test in CI that runs both implementations over a fixture set and asserts identical outputs, and it fails the build rather than lighting up a dashboard. Then a continuous production check: log the serving feature vector for a sample of requests, replay the same entities through the offline path at that timestamp, and alert on the mismatch rate. That catches the sneaky cases, which are nearly always time-related, typically an aggregate computed over a window containing data that was not available at prediction time, so the offline model looks wonderful and production disappoints. The tell is a model beating its holdout by an implausible margin. I now treat a suspiciously good offline number as a leakage report until somebody proves otherwise.

Environment drift is a build-system problem, so I fix it at the build. One image, built once, promoted unchanged through staging into production, addressed by digest and never by a mutable tag, so the same bytes run everywhere and works in staging finally means something. Dependencies resolve from a lockfile that covers transitive packages including the accelerator-adjacent layers, and the image build fails on anything unpinned. Configuration is the only thing permitted to differ; it is injected at runtime and validated against a schema at startup, so a missing key crashes the process immediately instead of at the first request that needs it. The parts people forget sit outside the container: the host driver, the model artifact fetched at boot, and the base image rebuilt nightly by a job that silently pulls new system packages. I pin the artifact version in config and log driver and image digests at startup. The tradeoff is friction, since upgrading anything becomes deliberate work and teams complain, so I pair it with an automated dependency-bump pull request that runs the full gate. Otherwise the pins rot and everyone ends up two years behind.

I start by making spend attributable, because you cannot control what you cannot assign. Every workload carries labels for team, environment, and purpose, and a daily job turns cluster utilisation into a per-team cost that lands on that team's own dashboard rather than in a central finance report nobody opens. Then the structural controls. Development gets hard quotas plus idle reaping, since notebooks holding a large accelerator overnight are frequently the single biggest line item, so idle sessions die after a fixed window with a warning first. Training defaults to preemptible capacity and must checkpoint; opting into on-demand requires a justification field somebody reads. Inference is where the money actually lives at steady state, so I attack utilisation there: batching and concurrency tuning, right-sizing the accelerator to the model instead of defaulting to the largest card, quantising wherever the eval gate says quality holds, and autoscaling to a real overnight floor. Finally I alert on rate of change rather than absolute spend, since the expensive incidents are misconfigured autoscalers running all weekend. Aggressive scale-to-zero trades money for cold starts, and that call belongs to the product owner.

Silent degradation is an organisational problem as much as a technical one, because the usual incident process assumes something is down and this is something being subtly wrong. I would establish that the model has an on-call owner inside the same rotation as the service, not a research team reachable on weekdays. The runbook's opening steps are mechanical: confirm the serving model and prompt versions, compare them with the last known-good pair, and check input health before anybody theorises about the model. Mitigation precedes diagnosis, so roll back or route to a conservative fallback and then investigate, because the urge to debug a live degradation while users suffer is strong and always costs more than it saves. I push for a severity definition that covers quality incidents, since teams default to the lowest severity when nothing is throwing errors, which means nobody joins the call. Afterwards the review question I care about is not root cause but detection gap: how long between degradation and someone noticing, and which signal would have caught it sooner. Every quality incident should exit with a new promotion gate or a new alert.

The hard part is not running the model in three regions, it is being certain all three are running the same thing. I would make deployed configuration a single declarative source covering model version, prompt bundle, routing weights, and feature service endpoints, reconciled per region by an agent that reports drift, rather than three pipelines each applying changes and hoping. Artifacts replicate to regional stores ahead of any config change, and the promotion event waits on replication completing, since the classic failure is a region pointing at a version it cannot pull and quietly falling back. Rollouts go region by region with a bake period between them, never simultaneously, and rollback is one config revert applied by the same reconciler. I expose a single view of what each region actually has loaded, read from the running processes rather than from what we intended to deploy, because those two diverge and the gap is exactly where incidents live. Two tradeoffs I would name aloud: consistency costs deployment latency, and if residency rules forbid replicating some inputs, regions will legitimately behave differently, so monitoring must separate drift we must eliminate from difference we chose.

Handover fails when it is treated as a stage rather than a constraint, so I move the requirements upstream. Practically, researchers develop against the same container image and the same feature interfaces that serving uses, from the very first experiment, so it works in my notebook is already it works in the runtime. I provide that as a paved path: a project template, a training entry point that registers artifacts automatically, and an evaluation harness they want to use because it is faster than whatever they would build themselves. Bribery works considerably better than policy here. Then a small number of non-negotiables checked in CI, meaning pinned dependencies, no notebook in the deployment path, a declared input schema, and a latency and memory budget handed to them at the start of the project rather than discovered at the end. Culturally, the change that mattered most in my experience was putting a researcher into the on-call rotation for their own model. Nothing improves deployability faster than being paged for it. The pushback is real, it reads as a tax on research velocity, so I carry that load jointly rather than dumping it.