← Cache & Workers

Redis: instances and keys

cache-workers · workzone

The Redis topology and a single key registry: which physical instances the module runs, which keys live in them and how long. It rests on the “system of record vs fast derivative” axis — the hard boundary below.

Hard boundary: Redis ⟂ Postgres

The axis is system of record vs fast derivative. Postgres holds the system of record and correctness-critical mutual exclusion (run uniqueness — transactional, next to the journal). Redis holds the fast / expiring / in-flight derivatives: the cache (rebuildable), rate-limit counters (durable — losing them would open a guessing window), dedup, queue payload (recoverable from the journals). This is a deliberate choice across all modules, not an accident. Hence too the rejection of Redlock: coarse, rare locks are held by Postgres, a distributed lock service isn't needed and, without fencing tokens, is dangerous for correctness. Instant access-token revocation via a jti blacklist in Redis — v2: in v1 the access token is stateless and expires on its own lifetime.
Redis — ephemeral

loss is painful but recoverable over time

  • queue payload
  • rate-limit counters durable
  • blacklist of revoked access tokens v2
  • webhook / job dedup
  • search-candidate cache
Postgres — durable

source of truth, must not be lost

  • run uniqueness (partial UNIQUE + heartbeat)
  • run journals · DLQ of failed tasks
  • AI budgets (SQL aggregate, not a counter)
  • sessions · refresh tokens · conversation history

Two instances

Cache and durable — in separate instances. They can't be kept in one instance with memory-based auto-eviction: eviction doesn't distinguish disposable cache from data that must not be lost — under pressure it might silently drop a task from the queue, a rate-limit counter, or a revoked-token blacklist entry (a security hole). So the roles are separated physically — two processes with different memory policies, not two namespaces in one.

Access. Both instances are on the internal network only, never exposed outward; entry is password-protected (requirepass), and in prod the channel runs under TLS. Redis holds rate-limit counters and the token blacklist v2 — they must not be readable from outside.

redis-durable
maxmemory + noeviction · AOF persistence · survives restart
  • SAQ queue
  • rate-limit counters
  • jti blacklist v2
  • dedup keys
  • coordination lock — maintenance mode · KS restore
  • Must not be lost — memory isn't evicted, state is on disk.
  • The cap is a write rejection, not eviction: on reaching maxmemory, the noeviction policy rejects a new write with an error instead of dropping data. Producers hit the rejection and slow down (backpressure) — the instance doesn't grow unbounded and doesn't run into the OOM killer. We still keep queue depth under monitoring so it never reaches the cap.
redis-cache
maxmemory + allkeys-lru · no persistence
  • search-candidate cache
  • pub/sub channel for push notifications
  • Grows unbounded — eviction is mandatory.
  • Every key has a TTL; persistence isn't needed (everything rebuilds).
  • pub/sub is an ephemeral signal, not a key; a miss is cured by polling.

Ephemeral keys (TTL)

Some state lives exactly as long as the window in which it means something — the TTL removes the key itself, no separate cleanup needed.

jti-blacklist v2 Instant access-token revocation — groundwork for v2. The key will live exactly the token's lifetime: past that it would expire on its own, no reason to keep it. In v1 there's no early revocation — the access token is stateless and fades on expiry. The consumer is Auth & Security.
webhook dedup One event — one delivery. A receipt mark keeps the same inbound delivery from being processed twice within the window.
idempotency window One job_id enqueue — one run. The window cuts off a duplicate enqueue of the same task; the full mechanics are on uniqueness.

Key registry

A single prefix convention: the colon-delimited namespace fixes the instance, lifetime and owner. A new key is a new row here, not a free-form name in code.

Namespace Purpose Instance TTL Owner
q: task queue — run payloads durable until completion queues
rl: rate-limit counters (token-bucket · sliding window) durable limit window rate-limit
brute: brute-force barrier state — per-IP window, per-account counter and delay, binding-code attempts durable barrier window Auth & Security
grace: refresh-rotation grace — old token → new pair for ~10 s durable grace window Auth & Security
bl: v2 blacklist of revoked jti durable access-token lifetime Auth & Security
cache: search-candidate cache cache per-key (+ LRU) cache
lock: coordination lock for maintenance — one holder at a time (maintenance mode · KS restore) durable operation lease runs
dedup:webhook: dedup of inbound webhook deliveries — one event, one processing durable receipt window Harvester
dedup:slack-event: dedup of inbound Slack-bot events — a repeat doesn't answer twice durable receipt window Slack
dedup:telegram-update: idempotency of repeated Telegram updates (by update_id) — a repeat doesn't answer twice durable receipt window Telegram
telegram:active-conv:chat_id pointer to the active conversation for a chat_id — replies continue it; /new resets it, and the next message opens a new one (the conversation's home of truth is Postgres conversations) cache sliding, for the conversation's duration Telegram
cache:mattermost:listener health flag of the Mattermost WebSocket listener — a heartbeat JSON; an expired key reads as “not running” (derived state — safe to evict) cache 90 s Mattermost
dedup:job: idempotency of a job_id enqueue — one run durable ~24h (successful run) lifecycle
push: pub/sub channel — signal of a new row in the feed cache ephemeral (pub/sub) Notifications

Cache memory. maxmemory — via env: dev 256mb · prod 512mb, capped at ≤ 70 % of the container's memory. Tuned by runtime metrics (keyspace_hits/misses, evicted_keys), not a fixed number — we fit the hot working set, not every query. Metric-driven tuning is about redis-cache; on redis-durable the same maxmemory serves as a hard cap with write rejection (noeviction) rather than eviction.