The ephemeral acceleration layer for frequent reads: what to cache, what
to key on, when contents go stale. The cache is a fast derivative that
rebuilds if lost (Redis ⟂ Postgres hard boundary).
Cache policy
The only consumer today is search candidates from
Query Engine
(exact match on the query). We cache the expensive retrieval layer
(cold start ~10 min), not the model's final answer — that answer is
personalized and access-filtered, so it holds no shared value for a cache.
Cache
retrieval candidates — the shared, expensive layer
Don't cache
the final LLM answer — personalization · ACL
-
TTL-only. Expiry by time, no event-driven invalidation — stale
candidates are cheap and the cost of a mistake is low. The exact TTL is
a tuning knob.
-
ACL isn't held at the cache. Access is re-checked by the
late-binding trim on the retrieval side, after candidates are returned —
so the cache needs no strict ACL invalidation, and access correctness
doesn't depend on it.
-
Home —
redis-cache (evictable, LRU policy): under
memory pressure old keys are dropped and the cache refills over time
(two instances).
-
Value carries chunk bodies, not just ids + score. A hit returns
the candidates whole, skipping the Postgres row lookup entirely — simpler
and faster while the working set is modest. The lean alternative (cache
only ids + score, read bodies back from Postgres on a hit) packs an order
of magnitude more queries into the same memory
(cache memory) —
a v2 lever once user count or Redis memory
pressure start to hurt the hit-rate.
-
No single-flight lock — the per-user key makes it moot. A cache
entry is keyed by query + identity,
so a "hot key" is one user's exact query: two concurrent misses on it
would need the same user firing the identical query at the same instant.
With no key shared across users there is no thundering herd — a miss just
computes and writes, no rebuild lock. A shared cross-user retrieval layer
guarded by a single-flight lock
(
lock:)
is a v2 lever, should that layer be introduced.
Key composition
The cache gate sits at the entrance to retrieval, before model
and prompt selection — those don't affect the candidate set, so
they aren't part of the key. The result is determined only by the query
itself and the identity; stale candidates expire by TTL, no separate
invalidation needed
(cache gate in the RAG pipeline).
standalone query
+
identity
Query
standalone query + identity
→
Key
assembled from segments
→
hit
ready candidates from redis-cache
miss
expensive retrieval → write with TTL