← Cache & Workers

Job lifecycle

cache-workers · workzone

The canonical convention: what happens to a background job from enqueue to terminal — the same for every lane and consumer. Queue and worker topology is on queues and workers; where the lock gets its correctness is on the Redis ⟂ Postgres hard boundary.

Job path

Enqueue is idempotent by job_id: a repeat with the same id spawns no second job. A retry is the same job, the same id, a new attempt — not a second row in the journal.

queued in the queue, waiting for a worker
enqueue
running a worker took it · stamps a heartbeat
pick
succeeded outcome in the consumer's row
failed error · into the DLQ policy
terminal
reaping post-terminal · lock released, idempotency window

enqueue → queued → running → succeeded / failed → reaping. Terminal and lock release are a separate step (see terminal and DLQ, uniqueness); a worker that crashes without cleanup doesn't free the job — the watchdog reaps it (see heartbeat).

Heartbeat and reaping

The heartbeat is a live job's pulse: a long run stamps heartbeat_at into its own Postgres row every ~30s. By that pulse the watchdog tells a working job from a dead one.

pulse present
job alive
heartbeat_at is fresh — the worker is running; the uniqueness lock holds and the next such job waits.
pulse gone stale
worker dead
No pulse past the threshold → the watchdog reaps the run to failed / stale, the uniqueness lock is released, the slot is free.

Without a pulse a crashed job would stay running forever and block the next one — reaping closes that deadlock.

The watchdog lives on the singleton scheduler. Scanning for stale heartbeats is a periodic pass, not work every worker does: otherwise N replicas would race each other on cleanup. Its home is the same dedicated singleton that ticks cron. The cost of that pairing: while the singleton is down, reaping stalls too — a crashed job's lock isn't released, and that job type waits for the replica to come back. The outage is bounded: on return the watchdog immediately clears the stale work, and cadences are coarse — acceptable for v1.
A planned stop is a graceful drain, not reaping. The watchdog catches a crash; on an orderly restart (deploy, SIGTERM) a worker drains cleanly: it stops taking new jobs, finishes the ones in flight, then exits. If it doesn't finish in time, the job returns to the queue and is picked up again: at-least-once delivery plus idempotency by job_id make the retry safe.

Retries

Two levels by pause length — a short one we wait out inside the job, a long one we return to the queue rather than keep a worker sleeping.

short · ≤60s
in-memory, queue untouched
The pause is tiny — the job waits and retries on its own; the broker and the slot are unchanged. Cheaper than sending the job through the queue and back.
long · minutes–hours
defer / re-enqueue with delay
A large Retry-After — the job returns to the queue with a deferred start, and the worker slot is freed. We don't sleep inside the job.
Reliability defaults are platform-wide, overridable by config. Error classification decides whether to retry at all: transient (timeout, 429, temporary failure) → retry; permanent (validation, 4xx logic) → straight to terminal, no wasted attempts. Backoff is exponential with full jitter (the random spread damps the thundering herd — a synchronized rush of retries after a shared outage), pause capped at ~300s, no more than ~5 attempts. The exact numbers are load tuning, not dogma.
classify transient → retry · permanent → terminal at once backoff exp + full jitter · cap ~300s attempts ≤ 5, then terminal → DLQ policy

Uniqueness

Mutual exclusion is held by Postgres, not the broker: a partial unique index next to the journal plus a heartbeat. The lock is correctness-critical — that's why it lives where the system of record is.

The lock = a partial UNIQUE on active states. UNIQUE … WHERE state IN ('queued','running') — while a job is active, the index won't admit a second one with the same key. A repeat launch (double click, scheduler race, retry) hits the index and gets skipped, not a duplicate. After terminal the row falls out from under the condition — the index is free again.
succeeded · ~24h
idempotency window
A successful job_id is held for the window, so a repeat enqueue under at-least-once delivery doesn't run the job again.
failed · briefly
resubmit allowed
A failed job_id is released quickly — an operator or the retry policy is free to resubmit the same work.

The window's source of truth is the journal in Postgres; the key in Redis (see ephemeral keys) is only a fast path, duplicating the journal rather than replacing it.

Rejected: a distributed lock service (Redlock / SETNX). Mutual exclusion is already held by the partial unique index plus a heartbeat — a Redis lock isn't needed and, without fencing tokens, is dangerous to correctness (a hung holder doesn't lose the lock). The same axis as the Redis ⟂ Postgres hard boundary.

Terminal and DLQ

The consumer reads the terminal: a run's or delivery's outcome is a status in its own Postgres row, not something lost in the broker. Each consumer's failed jobs settle in its own home, from which a curation pass re-picks them.

DLQ table Harvester — failed items into a dead-letter table; reconcile / dlq-retry re-picks.
"not delivered" status Email and Notifications — a failure is marked on the delivery row, retry per the retry policy.
run row Knowledge Store — the run's terminal (failed with cause and step stats) in curation_runs; the next curation pass reprocesses the graph anew.
run row Agent Engine — the run's terminal (failed with cause) in the agent_runs run row.
Rejected: a durable DLQ queue in Redis. A failure's durable home is Postgres at the consumer, not a separate queue in the ephemeral layer. A Redis DLQ stays only a transport backstop on the way to that home, not the place where failures are triaged.

Platform run contract

We unify the contract. The lifecycle skeleton is one for all consumers; the data home and the meaning of a re-pick are domain-owned. That way the monitor and the watchdog work over any run without collapsing different domains into one store.

state queued · running · succeeded · failed (+ stale as the watchdog's sub-outcome) attempts attempt counter against the retry cap heartbeat_at a live run's pulse; by it the watchdog spots a dead one last_error terminal cause + step stats next_retry_at deferred start of a retry; NULL if none scheduled
shared · platform
skeleton — one for all
The fields above are a shared vocabulary (mixin), names aligned where applicable. Not every consumer carries all of them: attempt counter and deferred start only for those that retry by counter; KS re-picks with a new run, not a retry. The heartbeat → stale watchdog and the uniqueness lock are shared by any table under the contract.
private · consumer
meaning — each its own
The row's payload and the meaning of "redo the work" are domain-owned: reconcile / next pass / resend differ per consumer. The data home stays with the consumer — the proximity principle.

The monitor reads each run table in its own domain home and merges them in the admin summary (module, job_id, state, attempts, age, error): one view of "what failed and where" without a central table. A single UNION ALL projection across the run tables is a v2 convenience once the run tables grow many.