Achilles — Cache & Workers: Tests

Decisions made

We test the layer's conventions, not the libraries Redis and SAQ as libraries are trusted to their authors — what's checked here are our rules on top of them: task uniqueness, cleanup of stale runs, retries and backoff, cache policy, lane isolation. We don't duplicate the broker's and driver's tests.

Uniqueness and cleanup — on a real DB and fake time The lock of a single active run is held by a partial-UNIQUE Postgres index, not a check in code — so the test brings up a DB and hits the index. Expiry of heartbeat_at and the TTL/cron windows are driven through a stubbed clock (time-machine), not by real waiting.

Rate-limit Lua — under concurrent load Token-bucket / sliding-window atomicity is only visible under a race: the test fires parallel requests and checks that not a single one leaked past the limit — a sequential check wouldn't catch that bug.

Stack and infrastructure

In place done pytest, pytest-asyncio

Redis fakeredis for unit key logic; a running Redis for Lua atomicity and restart survival

Time time-machine — cache TTL, heartbeat_at expiry, cron windows and next-tick recomputation

Worker the SAQ handler by direct call — without bringing up a worker or really polling the queue

Markers @pytest.mark.unit / @pytest.mark.integration — by type; p0 / p1 — by priority, orthogonal to type

Unit · P0 Enqueue idempotency deterministic · every PR

test_enqueue_idempotency.py

A repeat of the same job_id is dropped, window after terminal

UnitP0 → Uniqueness

Cases

duplicate in queuea repeat enqueue of the same job_id while the task is still queued or running is dropped — no second task appears

window after successthe job_id of a completed succeeded task is held for ~24h (time-machine) — a repeat within the window is dropped, past the window it goes through

window after failureafter failed the window is short — the task can be re-enqueued sooner than after a success

Integration · P0 Uniqueness and stale cleanup DB · separate CI step

test_uniqueness_reaping.py

A second active run is rejected by the index, a stale heartbeat is cleaned up

IntegrationP0 → Uniqueness

Cases

second active rejectedwith a run still active, an attempt at a second is turned away by the partial-UNIQUE index — the race is absorbed by the database itself, not by a check in code alone

stale heartbeat → stalea worker died without closing the run → on heartbeat_at expiry (time-machine) the run is cleaned up to failed/stale (cleanup)

lock released after cleanupafter a stale run is cleaned up the lock is released and the next run starts — a zombie doesn't lock the task forever

reaping stalls with the singleton, drains on returnthe reaper lives on the singleton — while it's down, reaping stalls too and stale runs pile up; on return several accumulated stale runs are cleaned up in one pass, not one per tick (reaper on the singleton)

graceful drain on SIGTERMa normal stop (not a crash): the worker finishes what it started, and an unfinished task returns to the queue and is picked up again — the repeat is safe by job_id idempotency, not via reaping (graceful drain)

Unit · P0 Retries and backoff logic is deterministic · every PR

test_retry_backoff.py

Short vs long retry, classification, exp-backoff + jitter

UnitP0 → Retries

Cases

short retry in-memorydelay ≤60s — a retry within the same slot, without returning to the queue

long — defer/re-enqueuea delay beyond the threshold → the task is deferred and re-enqueued, the worker slot is freed for the wait

transient vs permanenta temporary failure is classified as transient and retried; a permanent one (permanent) goes to terminal at once, without attempts

exp-backoff + full jitterthe delay grows exponentially with full jitter — two series of attempts don't coincide in timing

cap and attempt ceilingthe delay hits an upper cap; on exhausting the attempt count the task goes to terminal, it doesn't retry forever

Integration · P1 Terminal and DLQ DB

test_terminal_dlq.py

Outcome in the consumer's row, failure by policy (not in Redis)

IntegrationP1 → Terminal and DLQ

Cases

outcome isn't lostthe run's result is written to the Postgres row of the consumer module — durable, surviving the task's completion in the broker

failure by policya failure that has exhausted its attempts is recorded by the consumer's policy — a DLQ table / a “not delivered” status / failed-items — not into the durable Redis queue

the queue doesn't accumulate dead workthe Redis queue doesn't accumulate terminal tasks — after terminal the record of it lives in Postgres, not in the broker

Integration · P1 Rate-limit under load running Redis · concurrent

test_rate_limit.py

Lua atomicity under a race, restart survival

IntegrationP1 → Shared primitive

Cases

nothing leaks past the limita batch of parallel requests on token-bucket / sliding-window — exactly the limit is let through, none over: Lua counts atomically

one shared primitive for allone primitive serves different keys (provider, source) — the counters don't cross between keys

survives restartlimit state lives in durable Redis — after a process restart the counter isn't reset, the window continues

failure mode on unavailabilityredis-durable unavailable at decision time → the primitive doesn't swallow the failure but hands it to the consumer; the safe default is fail-closed (protection blocks the guessing), a relaxation to fail-open is something the consumer declares deliberately (failure mode)

Unit · P1 Cache — TTL and key fakeredis + time-machine

test_cache.py

Hit/miss, separation by key, TTL expiry

UnitP1 → Cache policy

Cases

hit / missthe first request is a miss, it computes and stores; a repeat with the same key is a hit, the source isn't touched

key from query and identitya different identity or a different query yields a different key → miss: one user's candidates don't leak to another through the shared cache (key composition)

TTL expiryon TTL expiry (time-machine) the entry disappears — the next request misses and recomputes

the cache doesn't insistthe cache is a hint, not the source of truth: on a discrepancy the trim re-checks against the source, the cached value isn't imposed

Integration · P1 Cache — single-flight running Redis · concurrent miss

test_cache_singleflight.py

One rebuild per batch of misses, lock in durable

IntegrationP1 → Cache policy

Cases

one computes, the rest waita batch of parallel misses on a hot key — the expensive retrieval runs exactly once, the rest wait for the ready result, the source isn't rebuilt N times

lock in durable, not in cachethe rebuild lock (lock:) lives in redis-durable — the LRU of the evictable cache won't drop it under memory, single-flight doesn't fall apart

the lock is releasedafter a rebuild (success or failure) the lock is released — the next miss after TTL takes it anew, the key doesn't get stuck

Integration · P1 Lane isolation separate worker pools

test_lane_isolation.py

The agents lane doesn't starve interactive, a lane failure stays local

IntegrationP1 → Three lanes

Cases

the lane cap holdsthe agents lane, with its own cap, is saturated with tasks — the interactive lane is still served, it doesn't starve

a lane failure stays locala worker failure in one lane doesn't bring down the workers of another — the topology keeps the lanes apart (worker topology)

Unit · P1 Cron windows — timezone and tick time-machine · deterministic

test_cron_timezone.py

Naive window → UTC by zone, NULL → organization's zone

UnitP1 → Windows in UTC

Cases

org-zone window → UTCa naive HH:MM is unfolded to UTC via platform_settings.timezone (IANA) — the tick lands on the right moment, not on local hours taken as UTC

owner's zone, NULL → organ agent schedule is computed by the owner's timezone; NULL falls back to the organization's zone — not UTC and not the server's zone

DST boundaryrecomputing the next tick across a daylight-saving transition (time-machine) doesn't shift the window by an hour or lose a tick

Integration · P1 Singleton scheduler several replicas · one tick

test_scheduler_singleton.py

Cron tick and next_run_at scan: once per N replicas, batch of due runs

IntegrationP1 → Scheduler

Cases

one tick — one taska cron tick with N scheduler replicas publishes the task once, not N times (time-machine on the cron window)

double run absorbeda race of two replicas on one tick is absorbed by the uniqueness lock — the second publish is turned away (uniqueness)

next_run_at scan — all duethe second trigger source: one pass over agent schedules publishes every run whose next_run_at has already arrived — not the first one found and not one per tick (two sources)

a missed tick isn't caught upa tick that fell on the singleton's downtime is not published retroactively on return — we don't backfill, we wait for the next window (time-machine skips the scheduled window) (missed tick)

Integration · P1 pub/sub for push notifications fan-out to subscribers · poll fallback

test_pubsub_push.py

The signal reaches all subscribers, a miss is harmless

IntegrationP1 → pub/sub channel

Cases

fan-out to subscribersa published signal is received by all hanging subscribers — an event from one api replica reaches a connection on another

a miss is harmlessa subscriber missed a publish (no connection at the moment of the event) — the next poll catches up the counter, the truth in Postgres isn't lost

Integration · P1 Dedup of inbound deliveries running Redis · time-machine

test_webhook_dedup.py

One event — one delivery; receipt window, key in durable

IntegrationP1 → Key registry

Cases

a repeat in the window is droppedthe same inbound delivery repeated (dedup:webhook: / dedup:slack-event: / dedup:telegram-update:) within the window is dropped — the event is processed exactly once

after TTL it goes through againon the receipt window's expiry (time-machine) the mark fades — the same delivery goes through again, it doesn't get stuck forever

key in the durable instancethe dedup key lives in redis-durable, not in the cache — the LRU won't drop it within the receipt window and a repeat won't leak through

orthogonal to dedup:job:inbound-delivery dedup (an event) and the idempotency of dedup:job: (enqueuing a task) — different namespaces, they don't overlap (one event — one delivery)

Integration · P1 Durable memory policy running Redis · maxmemory

test_memory_policy.py

noeviction: write rejection instead of silent eviction, backpressure

IntegrationP1 → Two instances

Cases

cap → rejection, not evictionon reaching maxmemory on redis-durable a new write is rejected with an error (noeviction) — data isn't dropped silently

producer backpressurethe producer hits the write rejection and slows down — the instance doesn't grow unbounded and doesn't run into the OOM killer

durable survives pressure, cache goes under LRUunder the same memory pressure a durable key stays put, while a cache key is evicted by LRU — the roles are separated physically, not by the policy of one instance

Structure Test file structure

The split is by type. unit/ runs on every PR deterministically — fakeredis and a stubbed clock, without a running DB or real waiting: cache key logic, idempotency, backoff. integration/ — as a separate, rarer step on a running DB and Redis: uniqueness and cleanup hit the index, rate-limit is raced concurrently, lanes and the scheduler are checked on real pools. Priority (P0–P1) is orthogonal to the directories and set by markers (pytest -m p0).

tests/cache_workers/module directory
- conftest.pyfakeredis / running Redis · time-machine · task and run factories · SAQ handler directly
- unit/deterministic, every PR — without a DB or real waiting
  - test_enqueue_idempotency.pyjob_id repeat and the window after terminal
  - test_retry_backoff.pyclassification, exp-backoff + jitter, cap and ceiling
  - test_cache.pykey composition, isolation by identity, TTL
  - test_cron_timezone.pyunfolding a cron window to UTC, NULL → org zone, DST
- integration/running DB and Redis
  - test_uniqueness_reaping.pyuniqueness on the partial-UNIQUE index, stale cleanup by heartbeat, graceful drain on SIGTERM
  - test_terminal_dlq.py · test_rate_limit.pyconsumer terminal/DLQ, rate-limit under concurrent load and failure mode
  - test_cache_singleflight.pyone rebuild per batch of misses, lock in durable
  - test_lane_isolation.py · test_scheduler_singleton.pylane isolation, singleton scheduler across N replicas
  - test_webhook_dedup.py · test_memory_policy.pydedup of inbound deliveries within the receipt window, noeviction on durable and backpressure
  - test_pubsub_push.pyfan-out of a signal to subscribers, poll catch-up on a miss