← Harvester

Reliability

harvester · workzone

What happens on failures and at high volume: the module does not lose data and survives crashes. Any sync is a background job on SAQ over Redis, not a request within an HTTP session. The administrator starts a sync and can close the tab: no connection needs to be held, breaking the session does not stop the work — the run is carried by a worker. What follows is the resilience path of a single run: SyncRun → checkpoint → retries → DLQ → visibility. The modes themselves that drive these runs are on the sync modes page.

1 SyncRun — run state One record per run.
The entity of a single run
Every sync launch creates a SyncRun — the single source of truth for how the work is going. It holds the state (queued / running / succeeded / failed / cancelled), the “N of M processed” progress, the current checkpoint, the worker's heartbeat, and accumulated errors. The UI computes nothing itself: it reads the SyncRun and renders progress and outcome from it.
progress N / M
checkpoint cursor
heartbeat alive at
errors K in DLQ
2 Checkpoint and resume Large volume syncs in parts.
Page by page, with resumption
A source with hundreds of thousands of entities is pulled page by page, and after each batch the run records its position — a checkpoint. An interrupted run resumes from the last checkpoint, not from scratch: already-processed work is not redone. A page not written through to a checkpoint at the moment of the crash is re-processed by resume — without duplicates, because the write to storage is idempotent (Load). The cursor and page-fetch mechanics themselves come from sources; here we cover only how the run leans on them to survive volume.
Heartbeat: worker crash and restart
While a worker is active, it updates the heartbeat in SyncRun. A stopped heartbeat means the worker has died (pod restart, container recreation): the run is picked back up and continues from the last checkpoint. The heartbeat interval is 30 s, and silence is counted as a crash after three missed beats (90 s): enough to ride out a GC pause or a brief pod restart, while a stuck run is reclaimed within a minute or two. Both values are fixed worker constants, not a source setting. A worker crash turns into a pause, not a loss. The heartbeat beats from a separate coroutine, independent of the retry stack: otherwise a long pause inside an attempt would look like the worker dying and trigger a false restart. A short backoff (60 s cap) is safe — it stays below the 90 s silence threshold; long pauses do not hold the worker at all, they go into deferred re-enqueue.
Resume — on a freshness budget, not forever
Resume undoes the effect of a crash, it does not replace a full run: continuing from a checkpoint makes sense while the gap is short. If the break exceeds the checkpoint's freshness budget (6 hours — a fixed module constant, not a source setting), the cursor and the source catalog have time to go stale and objects to disappear: the checkpoint is discarded and the run restarts from scratch. Auto-resume applies only to a crash (a stopped heartbeat): a run cancelled by an administrator (cancelled) is terminal, its checkpoint is not resumed — the next run starts from scratch. This way a day-long source pause does not lead to loading a stale selection. Resume applies only to a connector's incremental run whose stream is globally ordered by modification time (a manifest flag): with an unordered stream a watermark does not guarantee everything earlier has been processed. Reconciliation and a targeted DLQ retry do not resume and do not move the cursor — their selection does not reflect the stream frontier.
crash · within budget run continues from the last checkpoint
page 1 page 2 ✗ crash ↻ resume page 3 …
crash · budget expired checkpoint discarded, run from scratch
page 1 page 2 ✗ crash ⏳ budget expired ↺ from scratch page 1 …
cancelled · admin run is terminal, next one starts from scratch
page 1 page 2 cancelled new run page 1 …
3 Retries with backoff Transient source failure.
Retry with a growing pause
A network drop, a timeout, a 429 or 5xx from the source API — a reason to retry, not to fail. The request is retried with backoff: the pause between attempts grows, so as not to overload the source and to wait out a spike. After the attempts are exhausted, the item is not lost — it moves further along the path, into the DLQ.
What to retry, what goes straight to the DLQ — by error class
The “retry or defer” decision depends on the error class, not the specific code. Transient errors (they pass on their own: a load spike, brief unavailability) — are retried. Permanent ones (a retry is pointless: the request is invalid, no permission, no object, the data does not normalize) — go to the DLQ without attempts. This is not tied to HTTP: the classification is a layer above the protocol, so sources of a different nature (S3, a database, file shares, mail — with no HTTP codes) fit the same mechanism without reworking the core. The connector declares its error classifier, the core applies a single retry/DLQ cycle.
transient → retry 429 · 500 · 502 · 503 · 504 · 408 · network timeouts and drops
permanent → straight to DLQ 400 · 401 · 404 · 422 · normalization and validation errors
The HTTP mapping is only a default. The code table above is the standard “code → class” mapping for HTTP sources, which the connector may override: a code that means different things across APIs is classified by the behaviour of the specific source, not by the letter of the protocol. Example — 403 on GitHub/GitLab: more often a rate limit than an access denial, so before sending to the DLQ the connector checks Retry-After / X-RateLimit-Remaining and treats such a 403 as transient.
Backoff parameters
Attempts per HTTP page — 5 (the first + 4 retries), in-memory within the worker job. The pause is exponential with full jitter: base 1 s, factor ×2, cap 60 s. The cap is deliberately set below the heartbeat silence threshold (90 s), so a short retry never looks like the worker dying. All four are fixed worker constants, not a source setting.
attempts 5 (1 + 4)
base 1 s
factor ×2
cap 60 s
jitter full
Two levels of retries: memory versus queue
Short delays (≤ the 60 s cap) are an in-memory retry of the page right in the job: it heals a transient failure without losing the run's progress. Long pauses (minutes to hours: a large Retry-After, a degraded source) the worker does not sleep off — it re-enqueues the job (SAQ defer / re-enqueue) and frees the slot. This is how the checkpoint freshness budget (6 hours) is realized through deferred re-enqueue, not by holding a worker on sleep. If the source sent a Retry-After, it overrides the computed backoff.
4 Dead Letter Queue A single failed item.
Partial failure, not all-or-nothing
An item that could not be processed even after retries is set aside in the DLQ together with its reason — and the run carries on with the rest. This is a partial-failure strategy: a thousand entities loaded, three failed ones sit in the triage queue. The run then terminates as succeeded with error_count > 0 — finished successfully, but not flawlessly: the error count separates a partial failure from a total one (failed). The DLQ is the durable table dead_letters in Postgres, not ephemeral run state: the backoff retries above live in the worker job's memory and are measured in seconds, while a failed item awaits manual triage and survives a worker restart.
processed → storage
failed → DLQ
5 Failure visibility A failure is visible, understandable, and fixable.
Run status
SyncRun shows “N/M processed, K in DLQ” — it reveals both the progress and the outcome of any run.
Failure notification
A failed run raises a notification — the admin learns of the failure without opening the panel. The delivery channel (in-app, by email) is chosen by the notification system, not by Harvester.
Re-run
“Retry failed” launches a manual incremental run scoped to the queue (dlq retry) — after the source or permissions are fixed, they need not be collected anew. Processed ones leave the DLQ, failed ones again await the next triage; the fields and lifecycle are held by the data model.
Where this is viewed
The sync monitoring screen and the failed-item triage live in the Admin Panel — Harvester only supplies the SyncRun data here.