AI Security Engineer / Red Teamer

I start from the data and the actions, not the model. I draw the feature as a graph: every source of text that reaches the context window, every tool the model can call, every place output lands, and who owns each of those. Then I mark which text is trusted and which is attacker-influenceable — a support ticket, a scraped page, a PDF a customer uploaded are all attacker-influenceable even though they look like ordinary content. From that map the questions write themselves. What is the worst action an attacker could cause if they fully controlled the untrusted text? Whose data sits inside the blast radius? Is there a path from untrusted input to a privileged tool call with no human in between? I use STRIDE loosely for the plumbing — auth, logging, tenancy — but the AI-specific risks all come from that trust map. The output is a ranked list of scenarios with a proposed control for each, plus an explicit list of what we are consciously not defending against this quarter, because pretending coverage is complete is how teams get surprised later.

Direct injection is the user typing something to subvert the system prompt. Annoying, but attacker and victim are the same person, so the blast radius is usually their own session. Indirect injection is the one that keeps me up. There the hostile instructions ride inside content the model reads on someone else's behalf: a web page it browses, an email it summarises, a document sitting in the retrieval index, the readme of a repository it is refactoring. The user never sees the instruction and never consented to it, and the model acts with that user's privileges. That is a confused-deputy problem, which is why I treat it as an authorisation bug rather than a content-moderation bug. Filtering helps at the margin but cannot be the primary control, because the attacker gets unlimited attempts at rephrasing while I get one classifier. The controls that actually hold are structural: keep retrieved content out of the instruction channel, require confirmation for irreversible actions, and grant the agent only permissions the current user already had.

Everything that comes back from a fetch is data, never instruction. Concretely, the fetched text enters the context inside a clearly delimited region, labelled as untrusted third-party content, with a standing instruction that nothing inside it changes the task. I do not pretend delimiters are a control on their own — a determined attacker will try to escape them — so the real boundary lives outside the model. The fetching service runs in its own network segment with an egress allowlist so it cannot be turned into a proxy for internal endpoints; it strips scripts and normalises the document before the model sees it; and the model's tool set during a browse turn is deliberately narrowed, usually to more reading. If the workflow needs the model to act on what it read, that action happens in a separate turn with a fresh minimal permission grant and, for anything irreversible, a human click. I would rather the browsing agent be boring than trust prose to hold a security boundary that prose has never held.

Systematically, and with a written scope. I start from a taxonomy of the harms this product actually cares about rather than a generic list, because a coding assistant and a health chatbot fail in different ways. For each harm I write out attacker goals, then attack each goal along several independent dimensions: framing the request as fiction or research, splitting it across turns, wrapping it in translation or encoding, appealing to a persona the system prompt already established, and pushing on the seams between the system prompt and tool output. Every attempt gets logged with the exact input, model version, sampling settings, and a verdict, because refusals are stochastic and a single success proves much less than a success rate. I usually run each variant several times and report a rate rather than a binary. The point is not collecting trophies. It is finding which classes of framing generalise across goals, since those are the ones a patch to one system prompt will not fix and the ones worth escalating.

Those are two separate problems that teams often merge. On the response side, a model can only reveal what reached its context, so the first control is retrieval-time authorisation: filter documents by the caller's entitlements before they are assembled into the prompt, not after generation. Post-generation detection — entity matching, structured patterns for cards and national identifiers, a small classifier for freeform ones — is a second net, and I measure its recall against a seeded corpus rather than assuming it works. On the logging side, prompts and completions are the highest-density personal-data store most AI teams accidentally build. I redact before write, keep raw traces in a short-retention bucket with its own access control, and make sure evaluation sets built from production traffic pass through the same redaction. The honest limitation is that redaction is lossy for debugging: the trace you most want during an incident is the one that got scrubbed. I handle that with a break-glass path that is time-boxed, logged, and reviewed, rather than quietly retaining raw logs.

Abuse of an AI endpoint is usually economic before it is anything else. One request can cost a thousand times what a normal API call costs, so someone who can loop it does real damage without ever defeating a security control. I put limits at several layers: quotas measured in tokens rather than requests, since request counts hide long-context abuse; caps on output length and on agent loop iterations; and a hard ceiling on tool calls per session so a runaway agent cannot spend all night retrying. Anonymous traffic gets a much tighter budget than authenticated traffic, and fresh accounts get less than aged ones. I also watch for the pattern where someone uses the product as a free frontier-model proxy: high volume, shallow sessions, prompts unrelated to the product's domain. Alerts fire on cost per principal per hour, not on request rate. The tradeoff is that legitimate power users hit these ceilings first, so I make the limits visible in the product and cheap to raise on request.

The principle I hold to is that the agent never has permissions the requesting user does not already have. Tool calls carry the user's identity down to the underlying service and are authorised there, not in the agent framework, because a check living in the prompt layer is a check an injection can talk its way around. Beyond that I split tools by consequence. Read-only calls against data the user can already see run freely. Writes, spend, and anything that leaves the system — sending mail, posting to a webhook, opening a pull request — need an explicit grant scoped to the current task that expires with it. Irreversible or high-value actions get a confirmation showing the exact resolved parameters, not a summary the model wrote, since letting the model narrate its own action defeats the purpose. I also cap fan-out per turn and per tool. If a compromised context can only read a document the user could already read and then ask for a confirmation, an injection has nowhere useful to go.

Assume every line is hostile, because the model can be talked into writing hostile code by whatever it just read. The execution unit is a short-lived container per session on a hardened runtime such as gVisor or a microVM, with no network by default, a read-only root filesystem, one writable scratch mount, dropped capabilities, a seccomp profile, and hard limits on CPU, memory, process count, and wall clock. Nothing sensitive lives in that environment: no reachable cloud metadata endpoint, no ambient credentials, no mounted source tree beyond the files the task genuinely needs. If the code must install packages, that goes through a proxy pinned to an internal mirror with allowlisted names, since dependency installation is the easiest way to smuggle in an arbitrary payload. Results come back as size-capped data, because a multi-megabyte return value is its own denial of service. I monitor for the tells — outbound connection attempts, unusual syscalls, processes surviving the turn — and treat any escape as full compromise of that host, so the host holds nothing worth stealing.

The insight that changes how people design these systems is that any channel where the model can influence an outbound request is an exfiltration channel. The classic shape is rendered output: if a client renders arbitrary links or remote images the model composed, an attacker controlling part of the context can get secrets appended to a URL the victim's browser fetches on its own. The same applies to tool calls hitting attacker-chosen destinations, to webhooks, and to anything writing model output into a shared document some other party can read. So I close them structurally. Client renderers allowlist link and image hosts and never auto-load remote resources from model output; outbound HTTP tools have a fixed destination set rather than a URL parameter the model fills in; cross-tenant writes go through a service that authorises the destination against the caller. Then I test it: seed a canary secret into a session, replay the injection corpus, and alert if that canary ever appears in an egress log. Detection matters because new render surfaces keep appearing.

Uploads make the index an untrusted input, and every document in it is a message addressed to some future model turn. Three things worry me. First, instruction smuggling: a document carrying text aimed at whichever agent later retrieves it, potentially inside a different user's session if the index is shared. Second, retrieval hijacking, where a document is engineered with the right keywords to rank first for a valuable query, so the attacker chooses what the model grounds on. Third, plain leakage through weak tenancy, where one customer's chunks are retrievable by another. My controls are tenancy filters applied inside the vector store at query time rather than as a post-filter on results, provenance metadata carried through retrieval so both the model and the interface know a chunk came from an uploaded file, and the same untrusted framing I use for the open web. I also scan uploads at ingest for imperative-instruction patterns — not as a hard block, since false positives are real, but as a signal feeding review and a per-tenant abuse score.

That is where an injection stops being a single-turn problem. If the agent writes summaries, preferences, or notes into a store it reads on later turns, one successful injection can plant text that reactivates days afterwards, which looks to everyone like the model spontaneously misbehaving. Debugging starts with provenance: every memory record needs the session, the source turn, and a flag for whether untrusted content was in that context. Without those fields you cannot answer where a bad instruction came from and you end up blaming the base model. Once I have provenance I can query records written during turns that touched external content and diff them against the expected shape. The structural fixes are writing memory as structured fields rather than free text wherever possible, never letting stored text back into the instruction channel on read, and expiring it. I also give support a way to clear one user's memory without a deploy, because during an incident the ability to reset state fast matters more than a perfect root cause.

Four percent is a real product problem, so I would not simply relax the threshold and declare victory. First I look at what is actually blocking. In my experience most of that volume comes from a handful of clusters — security researchers discussing vulnerabilities, medical or legal wording, one language the classifier saw little of — and the fix is targeted rather than global. I would sample a few hundred blocked requests, label them properly, and get an honest precision-recall curve instead of reasoning from complaint tickets. Then I would separate the decision from the action: rather than one threshold that blocks, use a band where low-confidence hits get a softer response, a reduced tool set, or asynchronous review, and reserve the hard block for the high-confidence tail where precision is good. I would also check whether the classifier is doing work the architecture should do. If it is the only thing standing between untrusted content and a privileged tool call, the threshold argument is unwinnable and the permission model is the actual bug.

With an adversarial benchmark I maintain like a test suite, not with a vendor's marketing number. I build a corpus of attack cases grouped by technique class and by objective — get the agent to call a tool it should not, get it to reveal system instructions, get it to send data outward — and score on whether the objective was achieved, not on whether the model said something odd. The headline metric is attack success rate per class, tracked per model version, because upgrades move it in both directions and I want that visible before rollout rather than after. I pair it with the cost side: false-block rate on a held-out sample of real traffic, so nobody buys a lower success rate with a product that refuses everything. Two honest caveats. A static corpus decays as defences fit to it, so I refresh part of it each cycle and keep a private holdout the application team cannot tune against. And zero on a benchmark is never zero in the world, so I report which structural controls would contain a success anyway.

Downloading a checkpoint means running someone else's artifact, and people forget that. Three concrete risks. Serialisation formats that execute code on load — pickle-backed formats do this by design — which is why I require safetensors and block the rest at the loader. Provenance: hub accounts get compromised and names get typosquatted, so I pin by commit hash rather than by tag, mirror the artifact internally, and verify the digest at deploy time instead of trusting the registry at pull time. And behaviour: a model can be tuned to act normally except on a trigger, which no amount of hashing detects. On that last one I am honest that detection is weak. What I do instead is prefer sources we can audit, run every candidate through our own behavioural evaluation suite before promotion, and keep it inside the same sandbox and permission model as any other untrusted component so a backdoor has limited reach. The same discipline applies to datasets and to the inference stack, which usually carries a much larger dependency tree than the model itself.

The uncomfortable part is that most defences trade directly against data volume, which is the thing making the model good. I think about it as who has a write path into the training set. If examples come from production traffic or user feedback, an attacker has a cheap one, so those examples need identity, per-contributor rate limits, and a sampling policy that stops any single account dominating a slice. Then I look for statistical tells: near-duplicate clusters, examples whose loss is anomalously low or high, label-flip patterns concentrated around a rare token or phrase. None of that is conclusive, and a well-constructed poison at a low injection rate is genuinely hard to see, so I do not sell detection as the control. What I rely on is a curated evaluation set the training pipeline never touches, plus a trigger-hunting pass where we probe the tuned model with rare token sequences and compare against the base model's behaviour. And I keep the base model deployable, so rollback is a config change rather than a retraining cycle.

Three things in writing: scope, rules of engagement, and success criteria. Scope names the exact surfaces, environments, and accounts in play, and explicitly names what is out — production customer data, third-party services, anything that would page someone at three in the morning. Rules of engagement cover who knows the exercise is happening, whether the detection team is being tested too, what stops the exercise, and how we handle genuinely sensitive material we stumble into. Success criteria are the part teams skip. I want objectives written as attacker goals tied to business impact, so the report can say we obtained another tenant's documents rather than the model said something bad. I timebox it, usually one to two weeks with a defined crew, and I pair a security person with someone who knows the product deeply, because domain knowledge finds more than technique does. Afterwards there is a live debrief with engineering in the room rather than a document thrown over a wall, since the fix discussion is where most of the value is.

A report that lands is written for the engineer who has to fix it and the director who has to fund it, and those are two documents stapled together. The top is short: what we tried, what worked, what the business consequence is, and the two or three changes that would close the most findings. Then each finding gets an exact reproduction — inputs, model version, configuration, timestamps — with the success rate stated up front, because an AI finding that reproduces one time in ten gets dismissed unless you name the rate yourself. I report observed impact rather than hypothetical impact, and I say plainly where I could not determine it. Severity is argued, not asserted. Every finding carries a proposed mitigation with a rough cost, and I mark which are architectural versus which are a prompt change, since those get funded through different doors. I also list what we did not reach and which defences held, because a report of pure failure gives leadership no way to judge whether the system is improving.

First move is containment, and for agents that means a kill switch not requiring a deploy: a flag disabling the specific tool, or dropping the agent to read-only. Then scope. I want the full trace for that session — context, retrieved documents, tool calls with arguments and results — and then the same query run across every session that touched the same content source, because if injected material caused it, one incident is rarely one incident. From there it is ordinary response work: what did the action change, who is affected, is anything still running, what needs reverting. The AI-specific twist is attribution. Non-determinism means you often cannot reproduce it on the first attempt, so I resist concluding it was a fluke and instead replay the exact context with the same seed and sampling settings. The retro question I push hardest is not why the model did that; it is why the model was able to. If the answer is that a prompt was the only thing preventing it, the fix is a permission change, not a prompt change.

I treat tenant identity as something carried by the request rather than something the application remembers. The tenant is on the auth token, propagated to every downstream call, and enforced at the storage layer — a partition in the vector store or a mandatory filter applied inside the query, never a filter applied to results in application code, because that is one refactor away from disappearing. Caches are where this usually breaks: prompt caches, embedding caches, and semantic response caches all key on content, and a content-only key is a cross-tenant leak waiting to happen, so tenant goes into every cache key. Same for evaluation corpora and tuning sets built from traffic. Then there is the shared-model question. Per-tenant tuning gives cleaner isolation but multiplies operational cost; one model with retrieval-based personalisation is cheaper but stakes everything on the retrieval layer being correct. I usually take the second and invest heavily in verifying it, with automated cross-tenant probes running continuously against production and a canary document per tenant that must never surface elsewhere.

By making the fast path the safe path. If review is a meeting teams have to book, they route around it. So most of my effort goes into things that run without me: a threat-model template a team fills in during design in about an hour, a library of pre-approved patterns for the common shapes — sandboxed execution, tool authorisation, untrusted-content framing — and an adversarial test suite wired into continuous integration so regressions surface as a build failure rather than as a person's opinion. Reviews then concentrate on what is genuinely new. I am also disciplined about severity, because the fastest way to lose credibility is filing everything as critical; when I say something must block a launch it has to be rare and obviously right. Where a team wants to ship something I consider risky, I write down the risk, the alternative, and the decision owner, and let them own it rather than fighting it. The measure I care about is how often teams come to me before they have built the thing.