A multi-layer caching architecture in the agentic search pipeline boosts hit rates and freshness: Layer 1 caches planning decisions, Layer 2 caches summarized tool outputs, and Layer 3 caches final outputs, with respective hit rates around 50%, 35%, and 3%.

The system uses a vector search pipeline with 768-d embeddings, cosine similarity, and an HNSW ANN index, plus a compatibility layer tied to a contextual hash to ensure correctness and freshness, with embedding latency of roughly 15–20 ms and ANN search 10–15 ms.

Storage footprint estimates show about 8 GB per 1 million cache entries (3 GB for vector DB and 5 GB for KV store), projecting 50–60 GB for 10 million monthly queries under a 7‑day TTL.

Freshness control relies on a two-stage gating: a BERT-based classifier separates evergreen from time-sensitive queries, and post-execution validation prevents stale results, caching evergreen queries while refreshing time-sensitive or uncertain ones.

In an agentic search flow, queries trigger multiple LLM steps—planning, tool execution, summarization, rewriting—driving high token costs and a P95 latency near five seconds, with traditional string caches failing on semantic equivalence and freshness.

Operational knobs include tuned thresholds (≈0.98 for near-exact matches), layer-specific TTLs (planner 7 days, summarization 2 days, end-to-end 1 day), context-based invalidation, and metrics tracking hit/miss rates, latency saved, cost saved, false positives for evergreen classifier, plus targeted purges when tool or model updates occur.

Production results on 10 million-plus queries per month show Planner 44% hit rate, Summarization 35%, End-to-End 18%, yielding about 48% cost reduction and a P95 latency drop from 3.2 to 1.9 seconds with zero stale responses thanks to the two-stage gating.

Layer 1 details: PlannerCache stores tool-call plans rather than outputs, using embeddings and a context hash that includes model/tool versions and locale to invalidate stale plans.

Layer 3 details: The End-to-End cache supports both exact matches and high-threshold semantic matches to serve identical queries rapidly when possible.

The article highlights a multi-layer semantic caching approach for production LLM systems, achieving significant cost savings and sub-two-second P95 latency by caching at multiple pipeline layers, not just end-to-end.

Core cache components consist of an Embedding Generator (768-d vectors), a Vector Database with an ANN index using cosine similarity, a Key-Value Store for payloads, and a Query Classifier that labels queries as evergreen or time-sensitive.

Layer 2 specifically caches summarized results with freshness validation, caching only static data sources and maintaining a conservative 35% hit rate.