What a data source is and how it connects. A connection is a source model filled in by the admin: type, address, credentials, and scope. From there the platform authenticates, tucks secrets away in the crypto core, verifies the connection via Test Connection, and begins fetching only the objects in scope. From there the source runs through its own lifecycle — pause, disconnect, delete. The connection screen lives in Admin Panel.
connector_type
The source type — Jira, Confluence, GitLab, Slack, and
others. Selects a
connector
from the registry; its manifest defines which fields and
what kind of credentials the form asks for.
base URL
The address of the source instance — where the connector
sends its requests.
connector_type. Details
below.
Signing in to a source breaks down into two independent questions: as whom the platform accesses the source and with what it proves that. The axes are orthogonal and combine — a service account presents its own token.
Service-account tokens and passwords are stored
encrypted — the platform needs the original back (the connector
uses it to reach the source), so this is encryption, not a hash. The
key and algorithm aren't home-grown: the same primitive
(AES-256-GCM) that encrypts the SMTP password in the Email module is
provided by the Auth crypto core. Harvester keeps no secret store of
its own — it writes the secret to the
sources.credential_enc field and calls the core to
decrypt it at the moment it queries the source.
Before the first fetch, the connection is checked step by step — so
that on failure it's immediately clear where the problem is: the
address, the credentials, or the permissions. This is the same
availability-check approach used by model providers and Email
(is_available()): not "works / doesn't work," but a clear
diagnosis. The "credentials" and "permissions" steps are the
connector's
check_connection() method; the URL-availability step is a shared layer, the same for every
type, not the connector's job.
base URL responds. If not — the
address is wrong or the source is unreachable over the network.
The same probe runs on two triggers.
On demand — the admin
hits it at connection and edit time, with all three steps.
On a schedule
— the platform runs a lightweight version itself (steps 1–2:
connectivity and credential validity) between syncs, separate from the
heavy run. On a fetch failure: the probe drops the source into a
health state of (error)
and raises a notification through the same channel as a run failure
(visibility). The triggers and the probe's outcome show up in the "Test
Connection" block of the
source card.
Scope is a mode, not a list typed in by hand. As soon as the
credentials are accepted
(Test Connection,
step 2), the connector builds a catalog of the source's objects
— projects, spaces, channels — via its
list_catalog() method, and the admin picks from a live list on the connection screen. No
typos, no guessing at keys.
Scope is stored as a policy — a mode plus lists, not a frozen snapshot: the platform reconciles the specific objects against the catalog on every sync, or else newly created and deleted spaces would drift from the settings. A note on permissions: to build the catalog, the account needs to list the instance, even when the fetch is later narrowed. Scope is part of the source model and the boundary within which the fetch operates.
How the
connector
pulls data from the source and turns a single loop: the listing method
returns a page, the connector feeds it into the
pipeline and goes
for the next one by cursor. A single failed request — a timeout or a
429 from the source — doesn't break the loop: retry with
backoff keeps
reliability.
cursor
The source returns data in pages; the cursor is a bookmark to the
next one. The connector pages to the end, rather than trying to
grab everything in one request.
fetch(since)
The "fetch only what changed after" boundary. On a full import
since = epoch; on an incremental it's the moment of
the previous sync.
Concurrency is between sources: their runs are independent and go in parallel (per-source). Within a single source the listing is sequential — the cursor keeps it from running ahead — but object processing is order-independent.
The same source APIs also return users with their access rights — but that's a separate topic: how identity and ACL move into the platform is owned by acl-identity.
A connected source outlives any single run: it gets paused, disconnected, deleted. Behind the screen's buttons stand states and rules. Authentication splits into "as whom" and "with what."
Resume.
The axes are independent: an active source can be either idle or syncing, and error speaks to the last run, not to the admin's intent. The badges on the Admin Panel · Sources and Admin Panel · Sync screens are a rendering of these axes, not a separate truth.
During a run the source is locked. While it's
syncing, operations on it are unavailable — editing the
configuration, pausing, disconnecting, deleting, and re-running: one
source, one active run
(single-flight per-source). The only available lever is Cancel.
Cancel is safe and therefore needs no rollback. The run
stops at the nearest checkpoint, what's already processed sits
consistently —
Load is idempotent
on source_id + source_type + source_entity_id, and no
half-assembled state arises. A rollback to a snapshot would be harmful
and unnecessary: the data self-reconciles through idempotent upsert,
Incremental, and Reconciliation.
Pause, meanwhile, silences the schedule, rather than freezing a live run. Resume from a checkpoint heals a worker crash within a short window, subject to a freshness budget; a run cancelled or paused for long is terminal — the next one starts fresh instead of finishing off a stale fetch.
Pause and Disconnect are reversible and don't touch data; the fate of what was collected rests with Delete alone. The "configuration + data" cleanup is an explicit hard teardown, unlike the soft archiving of a record that vanished from the source itself (that one is kept for audit). An entity fed by another source as well doesn't disappear here — only the deleted source's contribution is detached; cross-source stitching and its boundary are owned by data-model → Knowledge Store.