Semantic Caching

Semantic caching stores previous model responses indexed by the embedding of the corresponding request, then reuses a cached response when a new query is semantically similar enough to a cached key — measured by cosine similarity above a threshold. Unlike exact-match caching, semantic caching captures paraphrases and near-duplicate queries that identical string matching would miss. It reduces cost and latency for repetitive workloads such as FAQs or support applications but requires careful threshold tuning to avoid returning stale or mismatched answers.