Caching Strategies for LLM Systems

LLM inference is expensive and slow relative to most database or API calls. Caching is one of the highest-leverage optimizations available: by avoiding redundant model calls, you can cut costs by 50–90% in read-heavy workloads and reduce p99 latency by an order of magnitude. Three distinct caching patterns apply at different layers of the stack.

Prompt prefix caching

Most LLM providers support prompt prefix caching: if the first N tokens of your prompt are identical across requests, the provider caches the key-value attention computation for those tokens and skips recomputing it. Since system prompts and RAG preambles are often identical across requests, this can cut effective input token costs by 80% or more. You typically pay a reduced rate for cache-hit input tokens.

# Anthropic prompt caching example
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": long_system_prompt,   # This prefix will be cached
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_message}],
)
# Check cache usage
print(response.usage.cache_read_input_tokens)    # tokens served from cache
print(response.usage.cache_creation_input_tokens) # tokens written to cache

Semantic caching

Semantic caching stores the model's response to a query and serves it for future queries that are semantically similar. When a new request arrives, you embed it and search the cache for similar prior queries. If a match exceeds your similarity threshold, return the cached response instead of calling the model. This works well for FAQ-style applications where users ask similar questions repeatedly.

Result caching

For deterministic or near-deterministic tasks — document classification, entity extraction, structured data parsing — cache the output keyed by a hash of the input. If the same document comes in twice, return the cached result. This is the highest cache-hit-rate strategy and works even when semantic matching would be unreliable.

• Prompt prefix caching: provider-level, automatic, best for long repeated system prompts
• Semantic caching: vector-similarity-based, best for FAQ and search workloads
• Result caching: key-value based on input hash, best for deterministic extraction tasks
• Always set TTLs on cached responses — stale answers are often worse than no cache