AI Engineer

I start from the question of where the answer lives. If everything the model needs is in the user's current input or in a small, stable body of text — a policy page, a product taxonomy that changes quarterly — I would put it in the prompt and skip the infrastructure entirely. Retrieval earns its keep when the corpus is too big to fit, changes faster than we can redeploy prompts, or is tenant-specific so the same prompt has to see different content per customer. The cost is real: an index to keep fresh, an embedding model to version, a whole extra hop that can fail or return junk. I have watched a team stand up a vector database for four hundred documents that would have fit in twenty thousand tokens. Before building it I would sample fifty real user questions and check how many are actually answerable from the corpus. If that number is under half, retrieval is not the missing piece — the content is, and no amount of clever chunking fixes that.

The request arrives with a user id, a conversation id, and the message. First I resolve the tenant and pull conversation history from our store, trimmed to the last few turns plus a running summary. Then retrieval: embed the query, search the help-centre index filtered to that tenant's locale and product, pull roughly fifty candidates, rerank down to six or eight passages. Those go into a prompt template with system instructions, the passages tagged with source ids, and the history. The model call streams back through our gateway, which owns the timeout, the retry on a 5xx, and token accounting. As tokens stream to the browser we also buffer the full text so we can post-process it — resolve citation markers into real links, strip anything resembling a fabricated URL. Finally we write the turn, the retrieved ids, latency, and cost to a trace store. That trace is the piece I care about most; without it you cannot debug a bad answer a week later, and someone always asks about one.

I chunk on structure before I chunk on length. Internal docs almost always have headings, and a section boundary is a far better cut point than a five-hundred-character window that slices a table in half. So: split on headings, then split any oversized section on paragraph boundaries, targeting around three hundred tokens with a small overlap so a sentence spanning a cut still appears whole somewhere. Every chunk carries its heading path, document title, owner, and last-updated date prepended as a short header. That context is what lets the model say "according to the deployment runbook" rather than quoting a naked paragraph. Tables and code blocks I keep intact even when they blow the target, because half a table is worse than a long one. The failure I watch for is orphaned pronouns: a chunk starting with "This process requires..." is useless alone. Prepending the heading path usually repairs it. I validate by sampling thirty chunks and asking whether a competent human could answer from that chunk by itself.

Four groups. Health first: p50 and p95 end-to-end latency, error rate split by stage — retrieval, model call, post-processing — because "the feature is slow" means nothing until you know which hop is slow. Then cost: tokens in and out per request, cost per conversation, and the tail, since a handful of runaway conversations usually dominates the bill. Then quality proxies we can compute without labels: how often we returned zero retrieved passages, how often the model produced its fallback, and the average number of turns before a user gives up. Then user signal: thumbs ratings, copy-to-clipboard, and whether the user escalated to a human afterwards. That last one is the honest metric for a support assistant, and it is the one I would defend in a review. Thumbs ratings are thin — maybe two percent of sessions rate anything, and the people who bother are usually annoyed. So I treat them as a smoke alarm rather than a score, paired with a weekly manual read of twenty sampled conversations.

It should never be a spinner that spins forever, and never a raw error string. My default is a hard client-side deadline shorter than the server's, so the user sees something deliberate rather than a dead socket. If we were streaming and got partial text, I keep what arrived, mark the message incomplete, and offer a retry that resumes rather than restarting from scratch. If nothing arrived, I fall back — for a support assistant that means showing the top retrieved articles directly, which is worse than a synthesised answer but far better than a blank box, and it reuses work we already paid for. Retries need care: one retry with jitter, never on a timeout where the model may still be generating and billing us. I also make the failure legible on our side, so the trace records which stage expired. The thing I push back on in design review is a cheerful "Something went wrong" with no path forward. Give the person a next action, even a degraded one.

Streaming buys perceived latency, and for anything conversational that is most of the win: time to first token of four hundred milliseconds feels immediate even when the full answer takes eight seconds. So chat gets streamed by default. But streaming costs you the ability to inspect output before the user sees it. If I need to validate a JSON structure, confirm citations resolve to real documents, run a safety filter, or reorder anything, I cannot un-show text someone already read. So structured output driving UI — a generated form, a set of filters, a table — I wait for and render once. There is a middle path I like: stream the prose, hold back anything inside a tagged block until it closes and validates. Cost is the factor people forget. A streamed request the user abandons still bills for tokens generated after they left, so I cancel server-side on disconnect. And if the answer is short, under a second or so, streaming adds machinery and complexity for no perceptible gain.

Event-driven where I can, scheduled reconciliation always. The source — a wiki, a ticket system, a CMS — emits change events; a worker consumes them, re-chunks the affected document, embeds only chunks whose text actually changed, and upserts by a deterministic id derived from document id plus chunk index. Deletes matter more than people expect: a deleted document lingering in the index will get retrieved and cited, and the reader has no way to know the source is gone. So deletes are first-class, with a tombstone so I can distinguish "removed" from "never indexed". Events get lost, so a nightly job walks the source, compares content hashes against the index, and repairs drift. I alarm on the size of that repair set, because a spike means the event pipeline broke quietly. Reindexing after an embedding model change is the painful case — you cannot mix vector spaces — so I build a new index, shadow-query both, compare recall on a fixed question set, then flip an alias. Staleness gets an explicit target.

I pull the trace for that conversation and work backwards, because a hallucination is usually a retrieval bug wearing a costume. First question: was the cited chunk actually in the context we sent? If it was not, we have a citation-formatting bug — the model invented a plausible id — and the fix is post-processing that drops any citation outside the retrieved set. If it was in context, I read the chunk itself. Often it is real but truncated mid-sentence, or lost a negation at the cut point, or it is a superseded version of a policy still sitting in the index. Then I check whether the correct passage was retrieved at all and simply ranked below the cutoff, which is a recall problem, and I would look at whether conversation history is polluting the query embedding. Only once all of that is clean do I conclude the model ignored its context, which does happen when the relevant passage sits in the middle of a long one. Either way the case joins the regression set.

I start from the user-facing number the product needs and work inward. Say we agree time to first token must stay under a second at p95 for an inline assistant. That budget gets divided: auth and tenant resolution get maybe thirty milliseconds, embedding the query eighty, vector search a hundred, reranking a hundred and fifty, prompt assembly is negligible, and whatever remains belongs to the model's prefill. Now every design decision carries a price tag. A cross-encoder reranker over fifty candidates might cost two hundred milliseconds, so I either drop to twenty-five candidates, move to a cheaper reranker, or find the time elsewhere. Parallelism helps — retrieval and history loading run concurrently rather than in sequence. The step people forget is that prefill scales with context length, so stuffing eight passages instead of four is a latency decision as much as a quality one. I instrument each stage separately and alert on stage-level p95, because an end-to-end number tells you that you missed the budget and never where.

You cannot assert equality, but you can assert plenty else. Before anything ships, the change runs against a frozen set of a few hundred real queries and I diff outputs against the current version — not for exactness, but to see which cases moved at all. Usually only ten or fifteen percent change, and I read those by hand. Automated checks cover what is genuinely testable: every citation resolves, no output exceeds the length cap, structured fields parse, refusal rate stays inside a band. Then a graded rollout — internal users, one percent, ten — with guardrail metrics watched at each step: escalation rate, fallback rate, cost per conversation, p95 latency. I keep the previous prompt and retrieval config addressable by version so rollback is a config flip rather than a deploy. The honest limitation is that a frozen query set goes stale. It stops representing what users actually ask within a couple of months, so refreshing it from production traffic is recurring work, not one-time setup.

The main thing I have learned is that internal APIs are almost never the right tool surface. They were designed for engineers who read documentation, so they carry twelve optional parameters, overloaded semantics, and error messages that assume context. I wrap them in a thin tool layer exposing narrow, intention-shaped operations — find_orders_for_customer rather than a generic search endpoint — with few required arguments and descriptions written for a model rather than a developer. Every tool is classified read or write. Reads execute freely; writes go through a confirmation step or an idempotency key so a retried call does not refund someone twice. Errors get translated into something actionable: "no customer matched that email" instead of a raw 404 body. I cap the loop hard, six tool calls and then we stop and hand off, because a stuck agent will happily hit the same endpoint forty times. And every call is logged with the arguments the model chose, since that log is the only way to learn which descriptions are ambiguous.

I look for a behaviour that only happens when the answer was useful. For a support assistant that is deflection: did the session end without a human ticket, and did the person avoid coming back with the same question within forty-eight hours. For a coding assistant it is whether the suggested change survived in the codebase a week later. For search-and-summarise it is whether the user clicked through to a source, which sounds like failure but usually means they trusted the summary enough to verify it. These beat any text-quality score because fluency cannot game them. The catch is attribution — deflection moves for a dozen reasons — so I want a holdout slice that does not get the feature, running long enough to read a real difference. Product teams hate holdouts, and I have had to argue for one more than once. Judged quality scores still have a place, but as leading indicators telling me something changed, not as evidence that anyone is better off.

The gap is almost always in what production queries look like next to the ones in the offline set. Offline questions tend to be clean and self-contained. Real ones are three words, or a follow-up like "and the second one?", or they contain a product name spelled the way customers spell it rather than the way marketing does. So I would sample two hundred production queries, push them through the offline harness, and see whether recall collapses. It usually does. Second, I check whether the query we embed is the query the user typed; if conversation history is being concatenated into the embedding input, a long thread drags the vector somewhere useless. Third, filters — tenant, locale, and date filters applied in production but absent offline will silently remove the correct document, and that shows up as a miss the harness never reproduces. Fourth, index freshness, since the offline snapshot may hold documents production has not indexed. Then I rebuild the eval set from sampled traffic and stop trusting the curated one.

When the number of steps genuinely depends on the input and I cannot enumerate the branches. If I can draw the flowchart, I write the flowchart — a fixed pipeline is cheaper, faster, testable, and its failure modes are boring. Agents earn their cost when a request might need one lookup or five, in an order that varies, and hand-coding that routing would mean a switch statement that grows forever. The price is steep and worth naming: latency becomes unpredictable because you are paying for several sequential model calls, cost per request grows a long tail, and debugging means reading a trajectory rather than a stack trace. I have watched a well-behaved three-call pipeline get replaced by an agent that reached the same answer at four times the cost. My usual compromise is a constrained loop — a fixed skeleton with one adaptive step in the middle, capped iterations, and a deterministic fallback when the cap is hit. That keeps flexibility where it matters and keeps p95 something I can commit to.

First, know where the money goes before cutting anything. In most retrieval features the input tokens dominate, and the largest single lever is how many passages we stuff — moving from ten to five often costs a point or two of answer quality and halves the bill. Caching is next: exact-match caching on common questions handles a surprising share of support traffic, and provider-side prefix caching on the static system block helps if the hit rate holds. Then routing — send classification-shaped requests to a small model and reserve the large one for genuinely open questions, with a measured quality check on that split rather than a hopeful guess. Operationally I want per-tenant budgets and rate limits in place before the spike rather than after, plus a circuit breaker that degrades to retrieval-only results instead of failing outright. The uncomfortable part is that several of these trade quality for money, so I bring the numbers to the PM and let them pick the acceptable degradation rather than deciding quietly.

Early, and with real outputs. The most useful thing I do in week one is generate fifty actual responses from a rough prompt over real inputs and put them in a shared document, bad ones included. A designer who has only seen a happy-path mock will design a card assuming three sentences, and then production hands them nine paragraphs or an apology. Showing that spread changes the design while changing it is still cheap. I also push for failure states to be designed rather than bolted on: what an empty result looks like, what a partial answer looks like, what low confidence looks like. With PMs the conversation I insist on is where the quality bar sits, stated as a rate — eight in ten answers usable — and what happens to the other two, since that determines whether we need a human-review path at all. And I am explicit about which requests are cheap and which are expensive, so scope decisions happen with cost visible instead of after I have built it.

I would make isolation structural rather than a filter someone can forget to apply. Tenant identity is resolved once at the edge from the authenticated session, never from anything in the request body, and travels as part of a request context the retrieval client requires as a constructor argument — so a query without a tenant simply cannot be constructed. Depending on scale that is either separate indexes or namespaces per tenant. Namespaces are cheaper to operate, separate indexes are far easier to prove correct to an auditor, and for regulated customers I have defaulted to the expensive option. Caches are the sharp edge people miss: any cache key omitting tenant id is a cross-tenant leak waiting to happen, and that includes embedding caches and response caches. Same for eval fixtures and debug tooling, which have a habit of running with elevated access. I would back all of it with a continuous probe that queries as tenant A for a string existing only in tenant B, and treat a hit as an incident rather than a bug.

I stop hunting for correct answers and start defining what a wrong one looks like. For most product features, failure is enumerable even when success is not: it cited a source that does not support the claim, it answered a question outside the corpus instead of declining, it surfaced another tenant's data, it contradicted the policy document. Those become assertions checkable on any output, and they catch more real regressions than a similarity score against a golden answer ever did. On top of that I build a small human-labelled set, a hundred to two hundred cases drawn from real traffic and deliberately weighted toward the hard tail rather than sampled uniformly, since uniform sampling hands you a pile of easy questions. Labelling is done by whoever owns the domain, against a written rubric, and I measure agreement between two labellers before trusting any of it. A judge model can then be calibrated against that human set and used to scale, but only once I know its agreement rate.

The hard part is deciding what stays central and what each team owns. My split: the platform owns ingestion, chunking, embedding, the index, retrieval with tenancy enforced, the model gateway with its budgets and rate limits, and the trace store. Teams own their prompts, their tool definitions, and their evals, because those encode product judgment a central team will always get subtly wrong. The interface between them needs versioning from day one. The first time the platform changes a default chunk size and silently degrades someone's feature, the trust is gone and it does not come back cheaply. So retrieval configs are explicit and pinned, and changes ship behind a version teams opt into. I would invest early in shared observability too, because the first platform question is always whether the problem is your layer or mine, and without per-stage traces that argument costs a week. The failure mode I watch for is the platform becoming a queue: if adding a document source needs a central ticket, teams route around you within a quarter.

I would characterise that five percent properly first, because "the hard cases" is usually three distinct clusters with three different fixes. Typically one cluster is retrieval failure, one is genuinely ambiguous input, and one is a real reasoning limit. The first is engineering work, the second wants a clarifying question rather than a guess, and only the third is actually a model problem. Then I would argue for scoping rather than heroics. If those cases are high-stakes — pricing, entitlements, anything a customer escalates — the right design is often to detect them and route to a human instead of squeezing the model into competence it does not have. That means building a confidence signal worth trusting, which is usually retrieval-derived rather than the model's stated certainty, since stated confidence is close to useless. The organisational half is harder than the technical half: a demo sets expectations, and someone has to tell the stakeholder that the last five percent costs more than the first ninety-five. Better before launch than after.