← Cache & Workers

Queues and workers

cache-workers · workzone

The substrate for running background jobs: how traffic is split by load and which processes chew through it. The queue payload lives in redis-durable; a job's path inside the queue (heartbeat, retries, cleanup) is on the job lifecycle.

Three lanes by load

Grouped by the nature of the job: we separate what contends with each other — fast from heavy, bulk mailings from transactional mail (so a campaign doesn't delay an invite or a password reset), and agents (which hit the LLM provider's rate limit) apart from live traffic.

interactive
fast · low latency
  • Email — invite · reset
  • Notifications — emails · webhooks
background
heavy and bulky · high concurrency
  • Harvester — sync
  • KS — embedding · curation · backup
  • Email — bulk mailing
agents
own cap on LLM calls
  • Agent Engine — runs
  • agent_max_concurrency separate from chat

The agents lane keeps its own cap on concurrent LLM calls, so background work doesn't eat the provider's rate limit out from under live chat.

Worker topology one image · several services · API scales independently
Same image, different command
A worker is the same image as the backend with a different command (saq configured for its lane). Three worker services with different concurrency; the API is a separate service.
  • Load isolation: a heavy run doesn't slow down requests, and one service crashing doesn't take another down.
  • Start at 3, not 1 or 5: merging conflated lanes back together costs more than splitting further later.
  • Split on demand: SAQ supports named queues, so breaking a lane into sub-queues is trivial.
api
FastAPI · serves requests · scales on its own
worker · interactive
low latency · moderate concurrency
worker · background
high concurrency · heavy jobs
worker · agents
cap agent_max_concurrency

Queue registry

Queue Lane Consumer Nature of jobs
Harvester sync background Harvester full · incremental · partial-resync · dlq-retry · reconcile · health
Email · transactional interactive Email invite · reset (test email — inline, bypasses the queue)
Email · bulk mailing background Email bulk campaigns — by volume, not weight; separated so they don't crowd out transactional mail
KS background runs background Knowledge Store ingest · re-embedding · graph curation · backup — bounded concurrency (online search calls the runtime directly, bypassing the queue)
Agent runs agents Agent Engine runs under the cap on concurrent LLM calls
Notifications · external delivery interactive Notifications email + webhook POST (in-app is written synchronously to Postgres, bypassing the broker)
Slack bot · inbound interactive Slack DM event → Query Engine → post back to the thread (ack<3s on receipt, dedup by event_id)
Telegram bot · inbound interactive Telegram DM update → Query Engine → reply as a message (received by webhook, dedup by update_id)
Mattermost bot · inbound interactive Mattermost DM post → Query Engine → reply in the thread (no webhook — enqueued by the scheduler's WebSocket listener, dedup by post id)