Semantic caching is a caching technique for large language model applications that stores prompts and their responses, then returns a stored response when a new prompt means the same thing as an earlier one, even if the wording differs. A traditional cache matches on an exact key, so “What were Q3 sales?” and “Show me sales for the third quarter” are two misses. A semantic cache compares the meaning of the two prompts using embeddings and can serve the second request from the first answer, skipping a model call.
How Semantic Caching Works
A semantic cache sits between the application and the model provider and follows a short loop:
- Embed the prompt. The incoming prompt is converted to a vector by an embedding model.
- Search for neighbors. The vector is compared against vectors of previously cached prompts in a vector index, usually with cosine distance.
- Apply a threshold. If the closest cached prompt falls within a configured distance threshold, the cache returns its stored response. RedisVL’s
SemanticCache, for example, exposes adistance_thresholdwhere a lower value means a stricter match. - Fall through on a miss. If nothing is close enough, the request goes to the model, and the new prompt, embedding, and response are stored for next time.
Production caches add controls on top of that loop. Time-to-live settings expire entries so answers do not outlive the data behind them. Metadata filters scope lookups by user, tenant, or data domain so one user’s cached answer is not returned to another. Eviction policies such as LRU or LFU, offered by projects like GPTCache, cap the cache size.
The Trade-Off: Hit Rate Versus Correctness
The threshold is the central design decision. A loose threshold raises the hit rate and cuts cost, but it also raises the chance of a false hit, where two prompts look similar yet ask different things. GPTCache’s own documentation warns that users may see false positives on hits and false negatives on misses. “Revenue for the EU last quarter” and “revenue for the US last quarter” can sit close together in embedding space while needing different answers. Teams usually tune the threshold against a set of known question pairs and keep it strict for numeric or regulated answers.
| Approach | Matches on | Typical use | Main risk |
|---|---|---|---|
| Exact-match cache | Identical prompt text | Repeated identical calls | Low hit rate |
| Semantic cache | Embedding similarity | Paraphrased questions to chatbots and agents | False hits, stale answers |
| Query result cache | Identical SQL plan and data version | BI dashboards and engines | Invalidation on data change |
| Materialized view or reflection | Precomputed relation the optimizer can substitute | Repeated aggregations | Refresh cost |
Semantic Caching in the Lakehouse
Data agents that answer business questions over a lakehouse are a natural fit for semantic caching, since many users ask the same few questions in different words. The cache has to respect the same rules as the data, though:
- Freshness. If the answer came from an Apache Iceberg table, the cached response is only valid for the snapshot it read. Tying the cache entry to a table snapshot ID or a short TTL keeps agents from serving numbers that changed after a new commit.
- Access control. A cached answer built from rows a user could see should not be served to a user with narrower permissions. Scoping entries by role or by row-level policy keeps the cache aligned with catalog governance.
- Where to cache. Caching the generated SQL rather than the final answer lets the engine rerun the query against current data, so the result stays fresh while the expensive generation step is reused. Engines then add their own acceleration below that, such as Dremio Reflections, which speed up the SQL itself.
A semantic layer helps here too. When agents map questions to governed metric definitions, two paraphrases tend to resolve to the same metric and filters, which makes cache matching more reliable.
Related Reading
- Semantic Caching for Data Agents Without Serving Stale or Leaked Numbers
- Caching
- Retrieval-augmented generation
Learn More
To learn more about the Data Lakehouse, read The Lakehouse Built for Everyone by Alex Merced. You can find this and other books by Alex Merced at books.alexmerced.com.