LLM Engineer

Both reshape the same next-token distribution, but at different points. Temperature divides the logits before the softmax, so it flattens or sharpens the whole distribution: at low values the top token's probability approaches one and generation becomes near-deterministic, at high values the tail gains real mass and you start seeing tokens the model barely considered. Top-p truncates instead. It sorts tokens by probability, keeps the smallest set summing to p, renormalises, and the rest of the tail is gone regardless of temperature. The practical difference shows up when the model is uncertain: with top-p at 0.9 and a flat distribution you might sample from forty candidates, while a peaked distribution leaves you two. Temperature does not adapt that way. In production I pin temperature at or near zero for extraction, classification, and anything feeding a parser, and raise it only on genuinely generative surfaces. Tuning both at once mostly wastes time, so I fix top-p at a sane default and move temperature, because two interacting knobs make regressions hard to attribute.

In order of how much I trust them: constrained decoding, then a provider's structured output mode, then a schema in the prompt, then parsing heuristics. If the serving stack lets me apply a grammar or JSON-schema constraint at the sampling step, that is the real fix — invalid tokens get masked out, so malformed output becomes impossible rather than merely unlikely. Failing that, native structured output modes do roughly the same job server-side. A schema described in the prompt alone gets you to maybe ninety-eight percent, which sounds fine until it is two failures per hundred calls at a million calls a day. Schema design matters more than people expect: shallow beats nested, enums beat free strings, required fields with obvious names beat clever ones. I avoid unions and deeply optional structures, since drift concentrates exactly there. I still validate every response and retry once with the validator error appended, and I log failures with the schema version, because a rising invalid rate is often the first sign a model version moved underneath us.

Several reasons, and they are worth separating. The most common is that the examples are not representative: six examples that all follow one shape teach the model that shape, and it will force the seventh input into it rather than handle the case honestly. Format bleed is real too — if every example answer runs two sentences, you get two sentences even when the correct answer needs a paragraph. There is label bias from ordering and distribution as well; four of six examples in class A skews predictions toward A. Then the mechanical costs: more examples means more prefill latency and more input tokens on every call, which at volume becomes the dominant line on the bill. Long example blocks also push the actual instruction further from the end of the prompt, where models tend to attend more strongly. My practice is to ablate — run the eval at zero, two, four, and eight examples and find where the curve flattens. It usually flattens near three or four, and the examples worth keeping are edge cases rather than typical ones.

Prompts live in the repository as versioned artifacts, not as strings inline in application code and certainly not in a table someone edits through an admin panel. Each template gets an id and a content hash, and that hash is recorded on every request alongside the model id, the decoding parameters, and the tool schema version — because a prompt that produced a good answer under one model and temperature is not the same experiment under another. That tuple is what makes a production trace reproducible six weeks later when someone reports a regression. I store rendered prompts too, at least sampled, since a template plus variables reconstructed from memory is never quite what was sent. Changes go through review like code, with the eval suite running on the diff. I push back hard on hot-editing prompts in production without a deploy: it feels fast and it destroys your ability to attribute anything. If non-engineers need to iterate, give them a staging path with the same versioning rather than a bypass around it.

Four things usually compete: the system instruction, tool definitions, retrieved content, and conversation history. Tool schemas surprise people — twenty tools with verbose descriptions can run several thousand tokens on every single call, and they are pure overhead on turns where no tool gets used. So my first cut is tool definitions, filtered to those plausibly relevant to the current turn. Next is history, but not naively. I keep the first turn, since it usually carries the actual task framing, keep the last several verbatim, and compress the middle into a running summary regenerated every few turns. Dropping the oldest turns blindly is the classic mistake: the user stated their constraint in turn two and now the model has forgotten it. Retrieved passages I would rather reduce by reranking harder than by truncating, because half a passage is worse than none. The system prompt stays whole. I track a token budget per section explicitly instead of letting them fight, and log when a section gets squeezed, since silent truncation is invisible until quality drops.

Never by vibes on ten examples, which is how most of these decisions actually get made. I run the frozen eval suite against both with identical prompts and decoding parameters, then look at the aggregate — which almost always moves by less than the noise, so the per-case diff is what matters. I want the list of cases that flipped from pass to fail, because a model gaining two points overall while breaking every function-calling case is a downgrade for us. Then the non-quality axes, which frequently decide it. Tokens generated per response often shifts, and a chattier model can raise cost twenty percent at identical quality. The latency profile changes. Refusal behaviour changes. Formatting adherence changes, which quietly breaks parsers. And prompts tuned against the old model may be actively counterproductive on the new one, so a fair comparison needs at least a light re-tune before I call it. If it stays close, I shadow the new model on live traffic and compare offline before routing anyone.

I map it to what is missing. If the model lacks facts, retrieval is the answer and fine-tuning is the wrong instrument, because training facts into weights gives you a model that is confidently stale and expensive to correct. If the model has the knowledge but will not produce the form we need — a house style, a rigid structure, a domain's terminology, a classification boundary hard to describe in words — that is where fine-tuning earns its keep, and it earns more by letting a smaller model do the job, which is often the real motivation. Prompting is where I always start, because it costs a day and tells you whether the task is feasible at all. So the order is: prompt until the curve flattens, add retrieval if the errors are factual, then consider fine-tuning if the errors are behavioural and I have a few thousand clean examples. The honest cost of fine-tuning is not training, it is the treadmill — every base model upgrade means redoing it, and teams underestimate that until they are two versions behind.

First I get the confusion matrix — which tool was chosen instead of which — because the fix differs completely per pattern. Overwhelmingly the cause is overlapping descriptions: two tools whose one-line summaries could both plausibly serve the same request. Rewriting descriptions to state what each tool is not for fixes more cases than any amount of work on the system message. The second pattern is a missing tool, where the model picks the nearest available thing because nothing matches; the honest fix is adding the tool or offering an explicit none-of-these option, which models take readily once it exists. Third is parameter confusion rather than selection — the right tool with a wrong argument, usually from an ambiguous parameter name or a missing enum. I would also check the count, since beyond roughly fifteen or twenty tools selection accuracy degrades noticeably and routing to a subset first beats one flat list. And I build the eval before the fix, several hundred labelled requests, so I can distinguish improvement from noise.

Treat it as a composed document with owned sections rather than a string everyone appends to. In practice that means a stable base layer holding identity, safety rules, and the output contract owned by one team; a domain layer per use case; and a request-time layer for retrieved content and user input. Each section carries a token budget and a rendering position, and composition happens in code against a schema, so a team adds a section instead of editing the middle of someone else's paragraph. What actually causes breakage is instruction conflict: one team writes "always answer concisely" while another writes "explain your reasoning step by step", and the model splits the difference unpredictably. So conflicting directives need a precedence rule stated in the base layer, and I make the rendered prompt inspectable in a diff view so reviewers see the whole thing rather than their fragment. Every layer needs its own eval cases, and the composition needs an integration eval, because sections passing individually tells you nothing about the combination.

Continuous batching rather than static batching is the starting point: sequences join and leave the running batch each decode step, so a short request is not stuck behind a long one, which is exactly where static batching's tail comes from. Beyond that it is a scheduling problem. The main knob is how aggressively you admit new requests — a large batch raises throughput and utilisation, but every sequence decodes slightly slower, and prefill of a newly admitted long prompt stalls decode for everyone unless you chunk it. So I cap admitted prefill tokens per step. I also separate traffic classes, because interactive requests sharing a replica with bulk offline jobs means the bulk job sets your p99; either split deployments or give the scheduler priorities. KV cache memory is the real ceiling, since batch size is bounded by cache, so long contexts collapse concurrency and paged attention softens that without removing it. And I load-test with the real length distribution, because synthetic uniform prompts make every configuration look fine.

Less than the benchmarks suggest and more than the demos do. With a decent modern method — weight-only quantisation, per-group scales, a sensible calibration set — aggregate benchmark scores typically drop a point or two, which sounds negligible. The interesting part is that degradation is uneven. Long-context recall, multi-step arithmetic, low-resource languages, and strict format adherence tend to suffer before general fluency does, so a chat demo looks fine while your JSON parse failures quietly triple. That is why I evaluate on our own task suite rather than published numbers, and specifically on the structured and long-context slices. The upside is concrete: roughly a quarter of the memory, which buys KV cache headroom and larger batches, and for memory-bandwidth-bound decoding that is usually a real throughput gain rather than a theoretical one. It is not automatic, though — you need a kernel that runs the quantised format fast, and some setups dequantise and gain nothing. Calibration data matters too; calibrating on generic web text when your traffic is code puts the error where you least want it.

Two different symptoms with different causes, so I separate them first. Trailing off is nearly always a limit rather than model behaviour: max tokens set too low, a stop sequence appearing legitimately inside the content, or a client-side truncation nobody logged. I check the finish reason before theorising anything, because length versus stop answers the question immediately. Repetition is a sampling and context issue. At temperature zero, degenerate loops are a known failure mode, especially on smaller models and long generations, and the standard levers are a modest repetition or frequency penalty and a small nonzero temperature. But before reaching for penalties I look at the context, since repetitive input produces repetitive output — a history containing three near-identical turns, or retrieved passages that duplicate each other because the same document sits in the index under two ids. Deduplicating retrieval fixes more repetition than any decoding parameter I have tried. If it is a fine-tuned model, I would also suspect training data with repeated structure the model learned to reproduce faithfully.

In layers, cheapest first. The bottom layer is assertions needing no model at all: does it parse, does it obey the length cap, does it contain the required fields, does it avoid a forbidden phrase, does the tool call carry valid arguments. Fast, deterministic, and they catch most real breakage. The middle layer is task-specific scoring wherever a correct answer exists — exact match on extraction, accuracy on classification, recall of required facts. The top layer is a judge model for open-ended cases, used only after measuring its agreement against human labels on a couple of hundred examples, and run pairwise rather than as an absolute score because absolute scores drift. Cases come from production traffic, weighted toward past failures; every incident adds a case, which is what stops the same regression twice. It runs on every prompt diff in CI within a wall-clock budget under ten minutes, or people quietly stop running it. And I report per-case diffs, since aggregates hide exactly the failures that matter.

It depends on whether the task is local or global. If the answer lives in one part of the input — find a clause, extract a figure — chunking with retrieval is right and cheap: split, rank against the query, feed the top few. If the task genuinely needs the whole document, like summarising a two-hundred-page report or checking consistency across it, I use hierarchical processing: summarise each section against a fixed schema so the intermediate representation is structured rather than prose, then reason over those summaries. The failure mode there is compounding loss, since details dropped in the first pass cannot be recovered later, so the section schema should be designed around what the final task needs rather than being a generic summary. For sequential tasks I prefer a running structured state to a rolling text summary, because text summaries degrade turn over turn while a state object degrades gracefully. And I check whether the input is long for a real reason; often it is boilerplate, and stripping navigation cuts forty percent before any machinery is needed.

The whole game is making the shared prefix as long as possible and byte-stable. Caching keys on an exact prefix match, so anything varying early — a timestamp in the system message, a user id, tools serialised in nondeterministic order — destroys the hit rate completely and silently. So I order the prompt strictly: static system instructions, then tool definitions with a stable serialisation, then few-shot examples, then retrieved content, then the user turn. Anything dynamic goes last, without exception. In conversations the prefix grows monotonically, which caches well as long as history is never rewritten, and compaction breaks exactly that, so I compact on a schedule rather than every turn. Measurement matters: I track cache hit rate as a first-class metric, because it degrades quietly the moment someone adds a variable field near the top. On self-hosted stacks the equivalent is prefix-aware routing, so requests sharing a prefix land on the replica already holding that KV cache — round-robin balancing throws the benefit away. Savings are real only above steady traffic, since cold prefixes cost more to write than they save.

When the task is narrow, the volume is high, and a large model already does it well. Those three conditions together are what make the arithmetic work: a well-distilled student on a specific task can track the teacher closely at an order of magnitude less per call, and at millions of calls a day that gap funds a team. Narrowness is the load-bearing condition, because distillation transfers task behaviour rather than general capability, so the student falls apart the moment the product asks for something adjacent. I would generate training data from the teacher over real production inputs rather than synthetic ones, keep the teacher's full output distribution where the stack allows it, and evaluate the student on the tail rather than the average, since that is where the gap opens first. The cost people underestimate is ownership: you now maintain a training pipeline, a model artifact, and a serving deployment, and every upstream change means redoing all three. Below a certain volume, paying the API is cheaper than paying an engineer.

LoRA adapters over a shared base, assuming the variants are fine-tunes of the same base — that is the design that makes this affordable. One copy of the base weights sits in GPU memory, adapters load per request and swap at negligible cost, so twenty variants cost roughly one model's memory instead of twenty. Requests carry a variant id resolved at the gateway from tenant or feature config, never from client input, and the batcher groups by adapter where it can, since mixed-adapter batches cost more than homogeneous ones. A variant that is a full fine-tune, or built on a different base, needs its own replica pool, and I would push hard against letting those proliferate, because every distinct base is a fixed memory cost and a separate upgrade path. Operationally the hard parts are eviction policy for cold adapters, per-variant metrics so you can see which one regressed, and versioning: an adapter id must pin to an immutable artifact, since a floating latest in a serving path means you cannot reproduce yesterday's output.

By treating it as a measurement instrument needing calibration, not as an oracle. First I establish agreement against human labels on a couple of hundred cases and report that number alongside every result the judge produces, because a judge at sixty percent agreement is telling you almost nothing and people quote its scores anyway. Known biases need designing around: judges prefer longer answers, prefer whichever option is presented first, and rate outputs from their own model family higher. So I use pairwise comparison rather than absolute scores, randomise position and run both orders, and where the stakes justify it use a judge from a different family than the model under test. The rubric should be specific and checkable — does the answer cite a passage supporting the claim, rather than is the answer good — because vague rubrics collapse into a fluency score. I revalidate whenever the judge's own version changes, since that silently shifts the entire metric baseline. And for anything consequential, humans stay in the loop on a sample.

Quality and coverage beat volume badly. A thousand carefully curated examples routinely outperform fifty thousand scraped ones, because the model learns noise as readily as signal. I source from production traffic where possible, since that distribution is the one that matters, and label with domain experts against a written rubric with measured inter-annotator agreement. Before training I run the dataset through checks: near-duplicate detection, because duplicates silently reweight the objective; length and label distribution, because skew becomes bias; and contamination against the eval set, which is the mistake that makes results look wonderful and production look broken. Coverage I assess by clustering the inputs and checking that clusters match production traffic rather than the cases someone found interesting. I deliberately include examples of the behaviour I want when the model should decline or ask for clarification, because a dataset of only confident answers trains a model that never admits uncertainty. And I hold out a slice by time rather than randomly, so the evaluation reflects drift.

I start by finding out whether it actually did, because half these reports turn out to be one memorable bad output rather than a shift. So the first thing I bring is data: the metric over time, and either it moved or it did not. If it did not, the honest answer is that outputs vary run to run by design, and I explain sampling in plain terms — the system picks among plausible continuations, so the same question can come back differently worded, and here is where we have pinned that down for the parts that must stay stable. If it did move, I name the cause concretely: a provider model update, a prompt change, an index refresh, a traffic shift toward a harder segment. What I avoid is the two failure modes I have watched damage teams — hand-waving about nondeterminism as though nothing is knowable, which destroys confidence, and promising exact reproducibility we cannot deliver against a hosted model. Then I say what we are adding so we detect the next occurrence first.