A single module for ingesting data from external sources. It runs in three
modes — full import, incremental sync, and periodic
reconciliation — over one internal pipeline:
extract → normalize → load.
Architecture
Full Sync
full
Full import from the beginning of time — on connecting a source or
a manual rebuild.
start: epoch
Incremental
incremental
Only the deltas since last time — webhooks plus scheduled
polling.
start: last run
Reconciliation
reconciliation
Express audit over the whole branch: clears what has vanished, aggressive
resolve.
start: whole source
→Knowledge Store — 3 projections + ACL + identities
→AI model registry — embedding model (built-in or your own; assignment is a precondition for ingestion)
↔Auth crypto core · Cache & Workers SAQ + Redis
→Query Engine — pre-filter by ACL
E/T/L hub (interchange) Transform station AI station
API
GET · POST/api/v1/sources
POST/api/v1/sources/sync
GET · PATCH · DELETE/api/v1/sources/{id}
POST/api/v1/sources/{id}/test-connection
GET/api/v1/sources/{id}/catalog
POST/api/v1/sources/{id}/sync
POST/api/v1/sources/{id}/cancel
GET/api/v1/sources/{id}/runs
GET/api/v1/sources/{id}/dead-letters
GET/api/v1/sources/{id}/health
Run status and progress reach open screens as push nudges over the platform
live-updates stream;
the endpoints above stay the single source the client refetches.