AI Product Manager

I ask what the correct output is and who knows it. If a competent person could write the rule down in an afternoon, a rule will be cheaper, faster, testable, and explainable, and I should say so even though it is a less exciting answer. Models earn their cost when the input space is open-ended, when the mapping is easier to demonstrate with examples than to specify, and when being approximately right is genuinely useful. Classification with twenty stable categories and clean labels is often better served by a small trained classifier than by a language model, and I try to separate those two cases rather than treating AI as one option. The test I apply is a thought experiment about failure: if a wrong answer costs the user more than the right answer saves them, the economics rarely work regardless of accuracy. I also look for the hybrid shape, which is usually the real answer. Deterministic logic handles the ninety percent that is structured, and the model handles the messy remainder with a path to a human.

I stop writing acceptance criteria as single outputs and start writing them as properties plus a rate. Instead of describing the summary the feature should produce, I specify that a summary must not contain a claim absent from the source, must stay under a length bound, and must meet those conditions on a stated share of a defined evaluation set. That converts an unspecifiable requirement into something engineering can measure. The document then needs a section conventional specs do not have: what the feature must never do, written as hard constraints rather than quality targets, covering things like fabricating a price or acting without confirmation. I also include a labelled set of about fifty real examples with the output I would accept for each, because that communicates intent far better than prose and becomes the seed of the evaluation set. And I write the degraded path explicitly, since the interesting design work is what the product does when the model is unsure or unavailable, and specs that omit it get that behaviour designed by accident.

I commit to one outcome metric, two or three quality metrics, and a set of guardrails I would roll back for. The outcome metric has to be about the user's job rather than the feature, so for a support assistant it is resolved without a human rather than messages sent, and for a drafting tool it is drafts sent without heavy editing rather than drafts generated. Engagement alone is a trap here, because a feature that answers badly generates more usage, not less. The quality metrics come from the offline evaluation set and are the ones I expect to move if a model or prompt changes. Guardrails are the ones I watch for regressions I am not trying to cause: latency at the ninety-fifth percentile, cost per completed task, escalation rate, and thumbs-down rate on the surface. I also insist on one qualitative commitment, which is that I read a sample of real sessions weekly. Metrics tell me something changed; the transcripts tell me what, and I have never regretted that time.

The number alone cannot answer it. What matters is the shape of the other fifteen percent and who absorbs it. If the failures are visible to the user and easy to reject, such as a suggested draft they can read before sending, 85% can be a strong product, because the user is a cheap and reliable filter. If the failures are silent and downstream, such as a value written into a record nobody rechecks, 85% is unacceptable at any level of enthusiasm. So I look at the failure distribution before the headline: are errors concentrated in an identifiable segment I can route away from, are they plausible-looking or obviously wrong, and what does a wrong answer actually cost. Plausible errors are much worse than absurd ones. Given a favourable answer I would ship narrowly, to a segment where the task is checkable, with the interface framing output as a suggestion rather than an answer, and with a visible correction path. Then I would spend the next cycle on the specific failure cluster rather than on the average.

It owes them the ability to check cheaply. A disclaimer is the weakest version of this and mostly protects the company rather than the user, since people stop reading it within a day. What actually works is making verification part of the flow: showing the source passage next to the claim, highlighting which part of the input drove the answer, and letting the user jump to the original in one click. Framing matters too. Calling the output a draft or a suggestion sets a different expectation than presenting it as an answer, and the verbs on the buttons carry more weight than any warning text. I want an obvious, low-friction way to reject or correct, both because it protects the user and because those corrections are the most valuable evaluation data the product generates. Where the model is genuinely unsure, I would rather it say so or hand off than produce a fluent guess, even though that reduces the apparent coverage. Trust lost to a confident wrong answer takes far longer to rebuild than the coverage gained.

I buy information before I commit scope. The first week is not development, it is building a small evaluation set from real user data and testing the crudest possible version against it, usually a prompt and an off-the-shelf model. That gives a floor, and the floor tells me whether the remaining work is tuning or invention. If a naive version reaches sixty percent on a task where eighty is the bar, the gap is engineering and I can plan it. If it reaches fifteen, the problem is probably not shaped the way I assumed and I would rather find that out in week one. I then scope in stages with explicit decision points rather than committing to a launch date up front, and I tell stakeholders which parts of the plan are estimates and which are conditional. The part I protect hardest is the evaluation set, because without it every subsequent scoping conversation becomes an argument about anecdotes. The honest limitation is that this front-loading is unpopular when a date has already been promised externally.

I try to build the chain in advance rather than argue it afterwards. The chain has links: an offline quality metric, a user-visible behaviour that metric should change, and a business outcome that behaviour should move. For a support assistant that runs from answer correctness, to fewer escalations to a human, to reduced cost per ticket and better resolution time. If I cannot name the middle link, the accuracy gain is unproven value regardless of how real it is. Then I measure the middle link directly in an experiment, because the relationship is rarely linear. Quality gains often do nothing until they cross a threshold where users stop double-checking, and then they matter a great deal, which is why three points can be worthless in one range and transformative in another. I also check for the counterintuitive case where better quality reduces a metric someone is targeting, such as engagement falling because users get their answer in one turn. That conversation is much easier to have before the experiment than after.

I refuse the framing that it is a quarter of either, and I ask what the first useful increment of evaluation buys. Usually that is a few hundred labelled examples and a script that runs on every change, which takes a week or two, not a quarter, and it immediately unblocks the feature work because nobody can ship confidently without it. So I fund the increment and put the ambitious tooling on the shelf. The argument I make to sales is concrete rather than principled: without evaluation, every model or prompt change is a coin flip that could regress a customer-visible behaviour we cannot detect, and the recovery cost of one such incident exceeds the delay. If I have had such an incident, I use it, because a real example moves the conversation in a way that a risk register does not. Where I concede is scope. Evaluation for the two features with regulatory or reputational exposure, sampled review for the rest. Perfect coverage on a feature nobody has adopted is a real waste and I have made that mistake.

I write them before the rollout starts, because criteria invented mid-incident are always generous. The launch gate has an offline component and an online one. Offline: no regression beyond a stated tolerance on the evaluation suite, and specifically no regression on the safety-relevant slice, which is a hard gate rather than a tolerance. Online: a staged rollout at one, ten, and fifty percent with a minimum soak at each step long enough to accumulate a meaningful sample, which for a low-traffic surface may be days rather than hours. Rollback triggers need to be automatic where possible and unambiguous where not: latency past a stated percentile bound, error rate above baseline, a spike in negative feedback, or any confirmed instance of a harm category we have listed. I name a person who can roll back without convening a meeting, since ambiguity about authority is what turns a ten-minute rollback into a two-hour one. And I require that rollback be tested before launch, because a rollback path nobody has exercised usually does not work when it is needed.

I start from cost per completed task rather than cost per call, because a task that takes six model calls including retries and a rerank is the real unit, and per-call thinking consistently underestimates by a factor of several. Then I look at the distribution of usage rather than the mean, since heavy users in these products are often twenty times the median and a flat subscription priced off the average loses money on exactly the customers who love the feature most. Practically that pushes me toward a subscription with a generous included allowance and transparent overage, or toward seat pricing with per-seat limits, rather than pure usage pricing that makes buyers nervous about an unpredictable bill. I also model the cost curve downward deliberately, because inference prices fall and a margin that only works after a price drop is a bet I should make consciously rather than implicitly. The constraint I hold firmly is that cost reduction must not silently degrade quality, so any cheaper model swap goes through the same evaluation gate as a feature change.

I give a date and decline the guarantee, in that order, because leading with the caveat makes it sound like an excuse. The date is for a staged rollout to a defined segment, not general availability, and I explain that the second date depends on what the first stage shows. On the guarantee I am direct: no one can promise a generative system never produces bad output, and anyone who does is either misinformed or planning to be somewhere else when it happens. What I can commit to is a bounded risk, expressed concretely. Here are the harm categories we test against, here is the measured rate on our evaluation set, here is what we block deterministically rather than probabilistically, here is the detection time and rollback path if something lands. Executives are generally fine with residual risk when it is quantified and someone owns it; what they cannot tolerate is being surprised. I also tell them the failure story in advance, the specific worst plausible headline, so the first time they think about it is not during the incident.

Not the ones people usually reach for. Click-through and session length are actively misleading on generative surfaces, since a bad answer produces more turns and more time on page. I want task completion measured at the level of what the user came to do, and where the product allows it, a downstream signal like whether the drafted message was actually sent or the suggested fix was accepted and not reverted. I pair that with a quality-of-experience readout: latency at the tail rather than the median, because a p50 of one second with a p95 of fourteen feels broken, and regeneration rate, which is a good proxy for the first answer being wrong. Novelty effects are stronger here than in ordinary product experiments, so I want at least two weeks and I look at week two separately from week one. I also check heterogeneity across segments before rolling out, because a positive average frequently hides a segment where the feature is worse than the old flow, and shipping over that is how a launch turns into churn.

I treat it as a scheduled migration and I am glad it came with notice, because the version that arrives without notice is a quality regression nobody announced. The first step is running the candidate replacements against our existing evaluation suite, which is the moment that suite either proves its value or reveals it was never built. I expect a mixed result rather than a uniform one, so I look at the slices where the new model is worse and decide whether those are recoverable with prompt changes or are structural. Prompts do not transfer cleanly across model families and I budget real time for that rather than treating it as a config change. In parallel I check the second-order effects: cost per task, latency profile, and any behaviour our guardrails were implicitly relying on. Then a shadow period running both in production on live traffic with only the incumbent's output served, which surfaces distribution differences the evaluation set missed. The longer-term lesson I would push is keeping the model boundary swappable so this is a fortnight rather than a quarter.

Fast acknowledgement, specific diagnosis, no defensiveness about how models work. The customer does not want an education about probabilistic systems, they want to know whether it will happen again and what protects them meanwhile. So the first response is same-day and factual: we have reproduced it, here is what happened, here is the immediate mitigation. Mitigation usually means narrowing rather than fixing, such as disabling the assistant for that query class or forcing it to answer only from retrieved sources with a citation, and I would rather ship a visible restriction quickly than promise a quality improvement in a month. Then the actual diagnosis. Was the source content wrong, was retrieval missing the right document, or did the model contradict a document it retrieved? Those are three different fixes and conflating them wastes a cycle. I add the case to the evaluation set permanently, which is the only credible answer to whether it will recur, and I tell the customer that is what happened. Internally the harder question is whether this class of failure was known and accepted, and if so I own that.

I make it by naming the failure the change is supposed to fix, because each option addresses a different one and the debate is usually happening at the wrong level. If the model lacks facts about our domain, that is retrieval, and no amount of fine-tuning reliably installs knowledge that changes weekly. If it has the facts but produces the wrong format, tone, or structure, that is prompting first and fine-tuning if prompting plateaus. If it is genuinely bad at the task shape itself, fine-tuning is the honest answer and I should expect to fund data labelling rather than treating it as an engineering task. So I ask for an error analysis on fifty real failures before the decision, categorised that way. It takes two days and it settles most of these arguments. I also weigh the ongoing cost, not just the build: fine-tuning creates an asset that has to be maintained across base model upgrades, which is a real commitment that teams underestimate. My default order is prompting, then retrieval, then fine-tuning, and I want evidence to skip a step.

The design question is not whether to have review, it is how to keep it from becoming rubber-stamping, which is what happens by default once the model is right most of the time. So I specify the reviewer's job as making a decision they are equipped to make, with the evidence in front of them: the source material, what changed, and what the model was uncertain about. If the reviewer has to leave the screen to verify, they will stop verifying. I would route by risk rather than reviewing everything, since uniform review across thousands of items guarantees inattention, and I would rather have careful review of the risky ten percent plus sampled audit of the rest. Throughput has to be modelled honestly as a staffing cost, and if the review time exceeds the time saved the feature does not work economically no matter how good the model is. I also build in measurement of the reviewers themselves, seeding known-bad items at a low rate to detect when approval has become automatic, and I would tell the reviewers that is happening.

I sort planned work by whether a better model helps it or replaces it. Anything that is essentially a workaround for a current model limitation goes on a short leash, built cheaply and expected to be deleted, and I say that out loud when we build it so nobody defends it later out of attachment. Anything that compounds regardless of model quality gets the real investment: proprietary data and feedback loops, evaluation infrastructure, workflow integration, permissions and audit, and the distribution that puts the feature where the work already happens. Those hold their value across a model generation. I keep the model boundary genuinely swappable so adopting a new release is an evaluation run rather than a rewrite, which is the difference between capturing an improvement in a fortnight and missing it entirely. I also plan for capability arriving rather than only for it being absent, keeping a short list of features currently just out of reach with a defined trigger for revisiting. The mistake I try hardest to avoid is a twelve-month bet on a limitation being permanent.

First I check the uncomfortable possibility that it genuinely does not matter, because the assumption that users notice every improvement we notice is usually wrong and killing the work is sometimes the right call. If I still believe it matters, the likely explanation is that the topline metric is too coarse or too slow. Aggregate retention will not move on a quality change affecting five percent of sessions, so I look for the segment that experiences the improvement and measure there, and I look at leading indicators that respond faster: repeat use of the specific surface, regeneration rate, correction rate, escalation volume. Trust changes tend to show up as slow shifts in whether people come back to a feature after a bad experience, which is a survival curve rather than a weekly number. I also argue some of these on risk rather than growth, since reducing the rate of a failure that could cost an enterprise renewal is worth funding even when it never appears as a positive movement. What I will not do is fund it indefinitely on faith.

I treat the return rate as the truth and the demo as marketing, including my own enthusiasm. The diagnosis starts with talking to people who used it twice and stopped, which is a more informative group than either heavy users or non-adopters. Usually one of three things is true. The feature solved a problem people do not have often, in which case usage is correct and the roadmap was wrong. It worked but not reliably enough to depend on, so people tried it, got burned once, and rationally stopped, which shows up as a sharp drop after a first bad session. Or it was faster to do the task manually once you account for checking the output, which is the failure mode fluent demos hide best. Each has a different response, and only the second is worth a quality investment. If it is the first, I would argue for killing it rather than iterating, and I would rather make that argument at week six than after two more quarters of sunk effort. Being the person who proposes the kill is part of the job.

I would set it by reversibility and blast radius rather than by product, so teams are not each inventing a policy. Reading and drafting are open. Actions that change state get graded: reversible and single-subject, like adding a calendar entry, can proceed with a visible undo; irreversible or affecting other people, like sending an external message, moving money, or deleting data, needs explicit confirmation with the specific consequence shown rather than a generic dialog. Anything touching money, legal commitments, or another person's data sits outside autonomous action entirely until we have operating evidence. I would also cap breadth even where the individual action is permitted, since an agent doing one wrong thing is an incident and one doing four hundred is a crisis, so rate limits and batch ceilings are policy rather than implementation detail. Every autonomous action needs an audit record and a way to reverse it in bulk. The part I would push hardest on is that exceptions get granted centrally with evidence, because otherwise the boundary erodes one reasonable-sounding product request at a time.