Prompt and Semantic Caching
Also known as: prefix caching, KV cache reuse, response cache
Cache at two levels: the stable prompt prefix the provider can reuse, and whole responses keyed by meaning rather than exact text.
The problem
LLM systems repeat themselves relentlessly. The same system prompt, the same tool definitions, and the same retrieved documents are re-sent on every turn and re-processed from scratch, while users ask semantically identical questions thousands of times a day in slightly different words. Both forms of repetition are paid for in full, every time, in both money and latency.
The pattern
Cache at two levels because they solve different problems. Prompt caching reuses the provider's computation over a stable prefix — order your prompt so everything static comes first and mark the cache breakpoint after it. Semantic caching skips generation entirely by matching a new question against previously answered ones by embedding similarity, subject to a high threshold and correct scoping.
Use when
- A large stable prefix — system prompt, tools, documents — is re-sent across turns
- Traffic contains many semantically duplicate questions, as in support and FAQ workloads
- Cost or latency is a binding constraint
- Answers are stable over a useful time window
Avoid when
- Prompt caching: the prefix changes every request, so nothing is reusable
- Semantic caching: answers are personalized, time-sensitive, or depend on live state
- Near-miss cache hits would be dangerous — medical, legal, or financial specifics
Tradeoffs
GainPrompt caching cuts input cost substantially and improves time to first token
CostPrompt order becomes rigid; any change to the prefix invalidates the whole cache
GainSemantic cache hits are effectively free and near-instant
CostA false hit returns a subtly wrong answer with total confidence
GainBoth are transparent to the model and require no prompt-quality tradeoff
CostCache invalidation is now your problem, and staleness is invisible until someone complains
These are two different mechanisms that get conflated. Prompt caching makes each call cheaper; semantic caching removes the call. Use both, and understand that only one of them can return a wrong answer.
Order the prompt for cacheability
Provider prompt caches match on an exact prefix, so a single changed token near the top invalidates everything after it. Order strictly from most stable to least: system instructions, then tool definitions, then long-lived documents, then conversation history, then the current turn. A timestamp in the system prompt is the classic mistake — it guarantees a miss on every request.
messages = [
# ---- stable prefix: cached across every request in the session ----
{"role": "system", "content": SYSTEM_PROMPT, # never changes
"cache_control": {"type": "ephemeral"}},
{"role": "user", "content": [
{"type": "text", "text": render_tools(TOOLS)}, # changes on deploy
{"type": "text", "text": knowledge_base_excerpt, # changes daily
"cache_control": {"type": "ephemeral"}}, # <- breakpoint
]},
# ---- volatile suffix: re-processed every time ----
*conversation_history,
{"role": "user", "content": current_question},
]
# Anti-pattern: this single line moves on every request and destroys every hit above it.
# SYSTEM_PROMPT = f"You are a helpful assistant. Current time: {datetime.now()}"Put volatile values at the end of the prompt, never the beginning. A timestamp, a request id, or a per-user greeting in the system prompt costs you the entire cache — and it is the most common reason a team reports that prompt caching "doesn't work".
Semantic caching needs a high bar and correct scoping
A semantic cache returns a previous answer for a new question, which means a false positive is a wrong answer delivered with full confidence. Two things keep it safe: a high similarity threshold, and a cache key that includes everything the answer depends on — tenant, user permissions, locale, and a version of the underlying corpus. Sharing a cache across tenants is a data-leak bug, not an optimization.
SIMILARITY_FLOOR = 0.94 # high on purpose; tune down only with measured evidence
def cache_scope(ctx: Context) -> str:
# Everything the answer depends on. Omitting any of these leaks or staleness.
return "|".join([ctx.tenant_id, ctx.locale, ctx.permission_digest, ctx.corpus_version])
async def cached_answer(question: str, ctx: Context) -> Answer | None:
q = embed(normalize(question))
hit = await cache.nearest(q, scope=cache_scope(ctx))
if not hit or hit.similarity < SIMILARITY_FLOOR:
return None
if hit.age > TTL_FOR.get(ctx.topic, timedelta(hours=6)):
return None
metrics.increment("cache.semantic.hit", tags={"topic": ctx.topic})
return hit.answer.marked_cached(similarity=hit.similarity)Negation is the classic false hit
"Can I cancel my subscription?" and "Can I not cancel my subscription?" are extremely close in embedding space and have opposite answers. So are pairs differing only in a number, a date, or a plan name. Guard the cache with a cheap lexical check on top of the similarity score: if the two questions differ on a negation, a numeral, or a named entity, treat it as a miss regardless of how similar the vectors are.
NEGATIONS = {"not", "n't", "never", "without", "cannot", "can't", "unable"}
def safe_to_reuse(new_q: str, cached_q: str) -> bool:
a, b = tokenize(new_q.lower()), tokenize(cached_q.lower())
if (set(a) & NEGATIONS) != (set(b) & NEGATIONS):
return False # opposite meaning, near-identical vectors
if numbers_in(a) != numbers_in(b):
return False # "30 days" vs "60 days"
if named_entities(a) != named_entities(b):
return False # different plan, product, or person
return True- Order the prompt most-stable to least; keep every volatile token at the end
- Include tenant, permissions, locale, and corpus version in the semantic cache key
- Set the similarity floor high and lower it only against measured false-hit rates
- Guard hits with lexical checks on negation, numbers, and named entities
- Set TTLs per topic — pricing changes weekly, policy changes yearly
Sample cache hits into your evaluation set. Serving a stale or near-miss answer is invisible in latency and cost dashboards — the only way you find out is by grading a sample of what the cache actually returned.