AI Research Scientist

An honest baseline is the strongest thing a reasonable person would try instead of my method, tuned with the same effort I gave my own. Most baseline dishonesty is not fraud, it is asymmetric effort: twelve learning rates swept for the new idea, the paper default used for the comparison. So I fix the protocol before I look at any results. Same search budget, same training data, same early-stopping rule, same evaluation code path. I write that down first, because deciding afterwards is how a fair comparison quietly becomes a flattering one. I also add a deliberately unsophisticated arm, since it is uncomfortably common for nearest-neighbour retrieval or a linear probe to land within a point of an elaborate model, and I would rather learn that privately than in review. When the strongest baseline is genuinely out of reach at my compute scale, I say so explicitly instead of substituting a weaker proxy and staying silent. What I never do is lift a number from another paper's table; preprocessing and split differences are usually bigger than the effect I am claiming.

First question: 1.4 points relative to what spread? A single run against a single run tells me nothing, so before I get attached I rerun both arms across at least five seeds and look at the distributions rather than the means. On many mid-sized benchmarks the seed-to-seed standard deviation is itself over a point, which makes a 1.4 gap unremarkable. I report mean with a bootstrap interval and, if the sample supports it, a paired test on per-example outcomes rather than aggregate scores, because pairing removes a lot of variance for free. Then I check the boring explanations: did the arms see the same number of tokens, was the baseline's checkpoint selected on the same criterion, did I accidentally evaluate with different decoding settings. Half of the surprising gains I have chased died at that step. If the effect survives, I look at where it lives. A gain concentrated in one subset of examples is a much more interesting finding than a uniform lift, and often reveals the mechanism I would otherwise have guessed at.

I report the number of seeds, the mean, and a dispersion measure, and I refuse to publish a table where those are missing, including my own drafts. Standard deviation across seeds is the minimum; for small seed counts I prefer showing every run, because five raw numbers are more informative than a mean plus a standard deviation computed from five points. If the comparison matters, I use a paired setup where both arms share seeds and data ordering, then bootstrap over evaluation examples to get an interval on the difference rather than on each arm separately. That difference interval is the thing readers care about and it is almost never reported. I also separate the sources of variance deliberately: seed variance from initialisation and data order, split variance from which examples landed in the test set, and decoding variance from sampling temperature. They have different fixes. The honest failure mode I watch for in myself is running extra seeds until the interval excludes zero, so I fix the seed count before starting and stick to it.

An ablation exists to attribute an effect, not to decorate a paper. The question it answers is which component is doing the work, and the only way to get that is to change one thing at a time while holding everything else fixed, including parameter count and training compute where possible. That last constraint is the one people skip. Removing a module and reporting a drop proves little if the removal also shrank the model by fifteen percent, so I either add a matched-capacity control or state clearly that the comparison is not capacity-controlled. I plan the ablation grid before running the main experiment, because deciding afterwards means I only ablate the things that flatter the story. I also include the reverse direction where feasible: adding my component to a different architecture. If it only helps inside my exact setup, that is worth knowing and worth saying. And I ablate my own evaluation choices, since prompt format or judge model can move results more than the architectural change under study.

I read for claims, not for papers. First pass is title and abstract with one question: what would have to be true for this to change something I am doing? Almost everything fails there and I let it. Second pass, for maybe fifteen survivors, is figures and the main results table, because the table tells me whether the comparison is fair faster than the prose does. I look for a missing baseline, a suspiciously narrow benchmark set, or an absent variance column, and most candidates die at one of those. Third pass, for three or four papers, is a real read including the limitations and appendix, where the interesting caveats hide. I keep a running note per research thread rather than per paper, so a preprint enters as one line of evidence for or against a position I already hold. That structure stops me hoarding PDFs I will never open. I also deliberately read one paper a week that argues against my current direction, because otherwise my reading list becomes a confirmation machine.

I assume the gap is mine until proven otherwise, because usually it is. I start with the boring layer: exact dependency versions, tokenizer version, dataset revision, and whether the evaluation script normalises answers the way the paper's did. Metric implementation differences alone explain a startling share of reproduction gaps. Then I check whether the released checkpoint reproduces the number, which separates a training problem from an evaluation problem in one step. If evaluation is clean and training is not, I look at the schedule details papers routinely omit, things like warmup length, gradient clipping, or whether the reported number came from the best checkpoint on the test set rather than a validation set. Only after that do I email the authors, with a specific diff rather than a vague complaint, and most respond helpfully. If the gap survives all of it, I document exactly what I ran and treat the published number as unconfirmed rather than wrong. And I note it in my own work, because building on an unreproducible result quietly poisons everything downstream.

Contamination is the default assumption now, so I try to bound it rather than rule it out. The direct check is n-gram overlap between benchmark items and any training corpus I can access, though with closed models that option disappears. The indirect checks are more useful in practice. I compare performance on items published before and after the model's training cutoff, and a sharp discontinuity is strong evidence. I test memorisation directly by giving the model a truncated benchmark question and seeing whether it completes the canonical continuation verbatim. I also perturb items in ways that preserve difficulty but break surface form, renaming entities or reordering multiple-choice options, and a model that drops sharply under perturbation was pattern-matching rather than reasoning. Where the stakes justify it, I build a small fresh evaluation set on the same task after the cutoff. That is expensive and it is often the only credible option. What I avoid is treating a canary string or a vendor statement as sufficient, since neither is verifiable from outside.

I would define the decision first: adopt only if the variant improves loss at matched training compute and does not degrade long-context behaviour. That framing sets the arms. I train at three model sizes rather than one, because architectural changes routinely help at 100M and vanish at 3B, and a single-size result cannot distinguish a real gain from an optimisation-difficulty artefact. Both arms get identical data order, identical token budgets, and separately tuned learning rates, since a variant that only wins because the baseline is at a bad learning rate is a tuning result, not an architecture result. I measure loss curves rather than final loss alone, so I can see whether the gap widens or closes with scale, and I fit a simple scaling trend across the three sizes to extrapolate. Alongside that I run targeted probes, retrieval over long inputs and position generalisation beyond the training length, because average loss can improve while the specific capability I care about regresses. If the trend narrows with size, I recommend against, however good the small-scale plot looks.

I try to set the kill criterion at the start, while I am still unattached. Usually it is a milestone with a date: if by week six the approach has not beaten the baseline on the small proxy task, I stop. Written in advance, that is a discipline; invented afterwards, it is rationalisation. Beyond the milestone, the signal I trust most is whether my understanding is improving. A direction where each failed experiment teaches me something specific about why is alive even when the numbers are flat. A direction where failures are unexplained and I am reaching for new tricks is usually dead, and I have learned that adding one more component to a method that does not work is the classic sign of denial. I also weigh what the result would be worth if it succeeded. A modest gain on a saturated benchmark is not worth another month even at high probability of success. The hardest part is social: killing a direction the team has invested in requires saying so plainly and early, before sunk cost hardens.

No, it should make me nervous. A table where every removal hurts by a similar margin is more consistent with a tuning artefact than with genuine component contribution, because the full configuration is the one I spent the most time tuning and every ablated variant inherits hyperparameters chosen for a different architecture. The fix is to retune each ablated arm with the same budget I gave the full system. That is expensive and it is frequently where impressive ablations collapse. I also check whether the drops are within seed variance, since a table of one-run numbers with 0.4 point gaps and a 0.6 point seed standard deviation is decorative. Another tell is that the drops are suspiciously additive; real components usually interact, so perfectly linear degradation suggests I am measuring capacity rather than mechanism. If after retuning one component carries most of the effect and the rest are noise, that is a better paper than the original story. It is a smaller claim, but it is one that will survive someone else running it.

The first move is to name the confound in writing rather than hoping nobody notices, because an unequal-compute comparison that is labelled honestly is still useful and one that is not is misleading. Then I try to bound the direction. If the arm with less compute won, the comparison is conservative and the conclusion probably survives; if the arm with more compute won, I cannot separate method from budget and I say so. Where possible I convert the comparison into a curve instead of a point, running the cheaper arm at several budgets and showing where its trend would have to be to explain the gap. That turns an unfair comparison into a falsifiable claim. I also look for a matched sub-comparison hiding inside the data, such as an equal-token checkpoint from each run, which is often available even when the final checkpoints are not comparable. And I distinguish compute spent on training from compute spent on search, because unequal hyperparameter search is the confound people forget and it is usually the larger one.

I present it as information, not apology, and I lead with what we now know rather than what we lost. Concretely: the hypothesis was that X would improve Y, here is the design, here is the result, here is the region of the space we can now rule out and here is the region we cannot. That last distinction matters, because a null result at one scale on one dataset does not license a universal claim, and overstating a negative is as sloppy as overstating a positive. Before the meeting I do the adversarial pass myself, checking that the implementation is correct, the baseline was not accidentally advantaged, and the effect would have been detectable at my sample size. A power analysis is the cheapest defence against the reasonable question of whether we simply failed to see a small real effect. Then I make a recommendation rather than leaving the room to sit with disappointment. Usually that is a specific next bet, occasionally it is stopping. Teams handle bad news well when it arrives with a decision attached.

The honest boundary conditions, first. Engineers will run my method on data distributions I never tested, so the most valuable artefact is a written statement of where it was validated and where I expect it to degrade. Beyond that: a reproducible training and evaluation script that runs end to end from a clean checkout, the exact dataset revision, and the evaluation harness so improvements are measured the same way after I stop being involved. I also hand over the failure set, the specific examples where the method loses to the baseline, because that is what will show up in production first and it saves the team rediscovering it under time pressure. Cost characteristics matter too, since a result that needs three forward passes per query may be a research win and a product non-starter, and it is better for me to say that than for them to find it in a load test. Finally I stay on the hook for a few weeks. Handoffs that end at the merge tend to come back as bug reports six months later.

Only when I have measured the judge, not merely used it. That means a human-labelled calibration set, agreement between judge and humans reported as a real number, and a check that agreement holds specifically on the examples where my method and the baseline differ, since overall agreement can be high while the judge is unreliable exactly in the contested region. I control for the known biases: position effects, which I handle by evaluating both orderings and averaging; length preference, which I check by regressing scores on output length; and self-preference, which is why I avoid using a model from the same family as either system under test. I prefer pairwise comparison to absolute scoring because judges are more consistent at ranking than at calibrated numbers. Even then I treat the judge as a screening instrument that lets me test more conditions cheaply, with a human evaluation on the final comparison that goes into the paper. If a claim rests entirely on an unvalidated judge, I would not defend it in review and I do not expect to be able to.

I report all seven, including the losses, in the main table rather than an appendix. The version of this that damages a field is choosing the four after seeing the results and calling the suite a benchmark. So I look for structure: is there a property shared by the winning tasks, such as longer inputs or more compositional outputs, that predicts where the method helps? If there is, that becomes the actual contribution, a method that helps under a stated condition, which is more useful than an unconditional claim and much more likely to replicate. If there is no pattern, I say the results are mixed and I mean it. I also correct for multiple comparisons, because with seven tasks and no correction, a couple of significant wins are the expected outcome of noise alone. Then I check the losses for a bug, since a large unexplained regression is often an implementation problem rather than a finding. Reviewers trust a paper with visible losses more than one with a clean sweep, and so do I when reading.

I would make the run configuration the single source of truth and make it impossible to launch anything outside it. Every run records a content hash of its config, the git commit, the container image digest, the dataset snapshot identifier, and the resolved random seeds, and all of that is written next to the results rather than into a personal notebook. Data gets versioned by immutable snapshot rather than by path, because the most common cause of a failed internal reproduction is that a dataset directory was updated in place. I would keep evaluation code in a separate versioned package so a metric change cannot silently rewrite history, and I would store per-example predictions, not just aggregate scores, since almost every follow-up question needs them and rerunning to get them is wasteful. The organisational half matters more than the tooling: a weekly habit of picking one recent result and having someone else rerun it from the recorded config finds drift while it is still cheap. Bit-exactness across hardware is often unattainable, so I target reproducing conclusions within stated tolerance instead.

I treat it as a portfolio with explicit tranches rather than a queue of requests. Roughly two thirds goes to directions with demonstrated early signal, where the remaining risk is execution; a quarter to speculative bets that would matter a great deal if they worked; and a deliberate slice to infrastructure and replication, which nobody requests and everybody benefits from. The speculative tranche gets small budgets with hard checkpoints instead of open-ended allocations, because the failure mode is a promising idea consuming a third of the cluster for a quarter on the strength of one plot. I also require that any request above a threshold states what result would cause the author to stop, which filters proposals more effectively than any review committee. Reallocation happens on a fixed cadence so people can plan, and I protect small exploratory jobs from being starved by one large training run, since queue behaviour quietly decides research culture. The uncomfortable part is cutting a direction mid-quarter, and doing that visibly and with reasons is what keeps the process credible.

I would make replication a routine obligation rather than an accusation. The mechanism I have seen work is a standing rule that any result headed for a paper, a launch, or a leadership deck gets rerun from its recorded config by someone who did not produce it, budgeted as normal work rather than as a favour. That single practice catches most evaluation bugs and split leaks. Alongside it, a short pre-experiment review where the author states the hypothesis, the arms, and the stopping rule, which is far cheaper than reviewing conclusions after a month of compute has been spent. I would also normalise a results meeting where negative and null findings are presented with the same seriousness as wins, because a group that only presents successes is a group that has learned to hide the rest. The cultural risk is turning this into a gate that slows everyone down, so I keep the bar proportional to the claim: an internal exploratory plot needs nothing, an external claim needs the full pass.

I would design a scaling study whose output is an extrapolation with error bars, not a single comparison. That means training both architectures at four or five sizes spanning at least an order of magnitude, each at compute-optimal token counts, with learning rate tuned per size rather than transferred blindly, because untuned small models systematically flatter whichever architecture is more forgiving. From those points I fit loss against compute for each architecture and compare the fitted trends, since the question is not which is better at 1B but whether the gap persists or closes by the target scale. I would report the extrapolation with the uncertainty of the fit made explicit, because the honest answer is often that the study cannot resolve the difference at the target scale, and leadership needs to hear that rather than a false precision. Alongside loss I would measure the things loss hides: throughput and memory at the target size, stability under the planned batch size, and whether the variant has known pathologies at long context. A small loss advantage that costs twenty percent throughput is not a win.

I start from what the finding actually enables. A capability result that meaningfully lowers the cost of misuse gets a staged treatment: the analysis and defences published, the specific recipe withheld, and a decision made with people who do that assessment for a living rather than by me alone. Most work is not in that category. For the rest the tension is commercial, and I argue it on concrete grounds instead of principle. Publishing costs a lead time that is usually shorter than people assume, since the hard part is rarely the idea. It buys recruiting, external scrutiny that catches errors we would not, and credibility that compounds. Method details that are inseparable from our data or infrastructure often cannot be usefully published anyway. Where I push hardest is against publishing a result we cannot support, since a paper the group cannot reproduce on request is worse than no paper. And I try to get the decision made early, at the point the direction starts working, because deciding after the draft exists turns a strategy question into an argument about somebody's effort.