Achilles — Harvester: Sync

When and how a source's data gets refreshed. Three modes — Full Sync, Incremental, and Reconciliation — answer one question in different ways: take everything, take only the delta, or reconcile against the source as a whole. The mode selects only the fetch window (since) and the aggressiveness of entity resolution; the pipeline itself is one and the same for all three. Every mode runs on a background worker — run durability (SyncRun, checkpoint, DLQ) is held by reliability.

1 Full Sync A full import from the beginning of time.

Take everything: since = epoch connection wizard sync wizard

The fetch window is open from zero — the pipeline passes every entity in the source. This mode kicks in when a new source is connected or an existing one is asked to be rebuilt from a clean slate. The run is long but safe: the output is the same idempotent upsert, so a repeat doesn't breed duplicates but updates what's already collected — Load.

2 Incremental Deltas only: what changed since last time.

Webhooks + polling source card

The working mode after the initial load. The source reports its own changes via webhooks; whatever it doesn't send, we pick up by polling on since from the previous run. The pipeline sees only the delta — that's cheap and fast. The mechanics of events and of the since-based increment are held by sources. The two cases below — partial re-sync and dlq retry — aren't separate modes but the same incremental with a different trigger and scope; the "type" in the run history is derived from this pair, not stored as a separate column.

Lost events → partial re-sync auto run history

A webhook may fail to arrive, a source may sit idle; you can't rely on a single event stream. A watchdog watches each source: silence longer than the quiet threshold (12 hours — the platform's global default, see Scheduling) is grounds for suspicion, not a verdict. First comes a cheap check by polling on since — no delta means it was just quiet, and the pass was for nothing; a delta found is the missing data. The watchdog then runs an ordinary incremental with since = incremental_cursor — there's no separate "re-sync window": the cursor only advances on success, and Load is idempotent, so re-requesting the delta is safe. All without a full import and without a human. In the run history such a re-sync is tagged with its own type — a string of them in a row reads as an unhealthy webhook, not the norm.

Failed items → dlq_retry failed-item review

A targeted retry over a list, not over a window. When items pile up in the DLQ, "Retry failed" raises a run whose scope comes not from since but from the specific identities in the review queue. Kin to partial_resync — both patch in a targeted way — but launched manually, after the cause (permissions, source) is fixed, rather than by the watchdog. Processed items leave the queue; its mechanics are held by reliability and the data model.

3 Reconciliation A scheduled reconciliation: audit and cleanup.

Comprehensive source audit auto reconciliation schedule

Runs on a schedule — weekly by default (see Scheduling). Reconciles what's collected against the source as a whole: what's no longer in the source is cleaned up as stale, discrepancies are corrected. The same pipeline as Full Sync and Incremental, but with an aggressive resolve. The difference is the size of the visibility window, not the logic: the resolve area is always bounded by what falls into the run's fetch window. Incremental sees a narrow window by since → dedup is local only; reconciliation pulls the source whole in one run and collapses intra-source duplicates scattered across earlier increments — invisible to the narrow window. There's no conflict with the pipeline here: the upsert key (source_id + source_type + source_entity_id) covers the same source_entity_id across runs, while the aggressive resolve works on one entity under different ids within the visible full source (cross-source stitching is held by Knowledge Store). This is a hygiene mode, not a mode for delivering fresh data.

Per-source independence

A mode is always the state of one source, not of the platform. Each source syncs on its own and holds its own since, its own schedule, and its own current mode. "Sync everything" isn't a special shared run but a launch of all per-source syncs at once.

In parallel with delivery runs the platform-level curation of the graph in Knowledge Store. By default the lanes don't get in each other's way; only during destructive curation steps (merge, retention) does the affected source's run go to queued — mutual exclusion is held by Knowledge Store · Lane coordination.

Scheduling → Cache & Workers

When to run Incremental and Reconciliation is set by the schedule: the platform's global default plus a per-source override. A source without its own schedule inherits the shared one; with its own, it follows that.

Incremental polling — one default interval (15 min) for every source, with a per-source override. A webhook-aware split (a slower ~6 h safety-net poll for sources that push their own changes) is a v2 refinement — v1 keeps a single cadence and leans on the per-source override where a source needs a different rhythm.
Reconciliation — weekly, in a set window (day of the week and hour; for a daily cadence — hour only) in the organization's timezone; by default — Sunday, 03:00.
Lightweight health probe — once a day.
Watchdog quiet threshold — 12 h; silence longer than that launches a partial re-sync.
Home of the defaults — platform_settings (the same platform-settings table as timezone; edited from Admin without a deploy). The per-source override lives in the sync_interval / reconcile_interval / reconcile_window fields: NULL = inherit the global default. The reconciliation window is stored as a minute of the week without a zone — the scheduler unfolds the hour from the setting in the org-timezone into a UTC launch moment.

Managing the schedule is UI over the sync backend domain, not part of the pipeline. Both schedules are held by the screens — Admin Panel · incremental and Admin Panel · reconciliation.