Also known as: embedding cache, approximate query cache, fuzzy LLM cache
A semantic cache returns a cached LLM response when an incoming query is similar enough — by embedding cosine similarity — to a previous query, rather than requiring exact-string match.
A semantic cache returns a cached LLM response when an incoming query is semantically close to a previously-seen query, not just an exact-string match. The mechanism is straightforward — embed the query, do a nearest-neighbor lookup in a small index of past queries, and if cosine similarity exceeds a threshold, return the cached response. The economics are compelling on repetitive workloads: a 30% cache hit rate translates to 30% lower LLM cost and most of the latency budget back.
The cache stores tuples of (query_embedding, query_text, response, metadata). On request:
cosine_similarity ≥ threshold, return the stored response. Cache hit.The threshold is the only knob, and it’s the only thing that matters for quality.
For these workloads, a semantic cache cuts cost-per-token and tail latency simultaneously. For research assistants, novel content generation, or any workload where queries are inherently unique, hit rates approach zero and the cache is dead weight.
The procedure:
correct / incorrect.The trade-off: lower threshold = higher hit rate = higher cost savings, but more wrong answers. Higher threshold = fewer hits but safer.
A second guard: pair the semantic cache with an LLM-as-judge verification on a sample of cache hits. If the judge flags >X% as incorrect, retighten the threshold.
The cache’s quality depends entirely on whether the embedding model places semantically equivalent queries close and semantically distinct queries far. A bad embedding produces 0.95 cosine similarity between unrelated queries; a good one keeps near-paraphrases at 0.95+ and distinct queries below 0.85.
In practice: use a strong general-purpose embedding (E5, BGE, OpenAI text-embedding-3, zembed) and tune threshold per-deployment. Don’t reuse a generic embedding for high-stakes flows without measuring per-domain false-positive rate.
Cached responses go stale when the underlying knowledge changes — product changes, policy updates, retrieved-document churn. Two strategies:
For RAG-style workloads, source-aware is correct but engineering-heavy. TTL is the default; tune based on how fast your knowledge base changes.
The full LLM-serving cache hierarchy (see caching strategies ):
These compose: exact-prompt first (always-correct cheap win), then semantic (approximate, more savings). Prefix KV-cache lives at the serving layer ( vLLM serving ) and is orthogonal — it speeds up cache misses.
Monitor: cache hit rate, false-positive rate (sample-checked via LLM-as-judge), threshold drift, cost-per-query before/after cache. Drift detection on the input-query embedding distribution is the early warning when your cache stops being useful — if today’s queries embed to a different region than yesterday’s, your hit rate is about to drop.
Lower thresholds (e.g., cosine 0.85) give more cache hits but more false positives — the cached answer doesn't actually fit the new query. Higher (0.97+) is safe but rarely hits. Tune empirically: sample 1000 cache-hit pairs, manually label whether the cached response answers the new query, and pick the threshold where false-positive rate drops below your tolerance.
When your query distribution is repetitive — customer support FAQs, common SQL/data questions, common-pattern code generation. Workloads with high query diversity (research assistants, novel prompts) get near-zero hit rates and aren't worth the cache infrastructure. Measure hit rate on a representative traffic sample before committing.
Yes — re-embedding the query on every request is wasted work. Cache (query string → embedding) at the request edge. The semantic cache lookup then becomes one nearest-neighbor query, no embedding cost. For very-high-QPS systems, cache embeddings in Redis or a small ANN index in process memory.
