Semantic Caching in Depth

Exact-match caching only helps when users send byte-for-byte identical queries — useful for programmatic pipelines but limited in conversational applications. Semantic caching extends this by storing a query's embedding alongside the response and matching future queries by vector similarity rather than string equality. If an incoming query is close enough to a cached one, the stored response is returned immediately without calling the model.

When semantic caching is a good fit

Semantic caching delivers the highest return in FAQ-style applications, documentation assistants, and product knowledge bots where a finite set of questions recurs with natural variation in phrasing. It is less appropriate for highly personalized responses, queries where small wording differences imply meaningfully different answers, or creative generation tasks where returning a stale response would feel wrong to the user.

• High-fit: customer support bots with common questions rephrased by different users
• High-fit: document Q&A where the corpus changes infrequently
• Low-fit: personalized recommendations that depend on per-user context
• Low-fit: queries about real-time data where freshness is required
• Low-fit: creative tasks like drafting emails where identical responses feel robotic

Implementing a semantic cache

import numpy as np
from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheEntry:
    query_embedding: list[float]
    query_text: str
    response: str
    created_at: float

class SemanticCache:
    def __init__(self, similarity_threshold: float = 0.92, ttl_seconds: float = 3600):
        self.entries: list[CacheEntry] = []
        self.threshold = similarity_threshold
        self.ttl = ttl_seconds

    def _cosine(self, a: list[float], b: list[float]) -> float:
        a_arr, b_arr = np.array(a), np.array(b)
        return float(np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)))

    def lookup(self, query_embedding: list[float]) -> Optional[str]:
        import time
        now = time.time()
        for entry in self.entries:
            if now - entry.created_at > self.ttl:
                continue
            if self._cosine(query_embedding, entry.query_embedding) >= self.threshold:
                return entry.response
        return None

    def store(self, query_embedding: list[float], query_text: str, response: str):
        import time
        self.entries.append(CacheEntry(query_embedding, query_text, response, time.time()))

Choosing a similarity threshold

The similarity threshold is the most critical configuration parameter. Too high and the cache barely helps — only near-identical phrasings match. Too low and semantically different questions receive the same cached response, which is a quality failure. Start at 0.90 to 0.95 cosine similarity and evaluate the false positive rate on a set of query pairs you have manually labeled as same-intent versus different-intent. Tune until the false positive rate is below 1%.

Cache invalidation and TTLs

Every cached entry should carry a TTL that expires it when the underlying knowledge base may have changed. For product documentation that updates weekly, a TTL of one to two days balances hit rate against staleness risk. For real-time data like pricing, the TTL should be minutes or the cache should be bypassed entirely for those query categories.