A page about what data flows through the module and how it lives over time. Two contracts carry the pipeline — RawItem → Entity — then Entity fans out across stores, and four decisions set its fate after loading: what we remember from history, how we handle deletes and attachments, and where the boundary with the Knowledge Store runs. Beyond this content plan the module holds its own control layer — the source configuration and the run journal; the Admin Panel edits it through the API.
RawItem
Source material as is, together with its metadata. The connector's output and the input to transform: not yet normalized, each source in its own format.
Entity
A normalized entity — a single shape for everything the module
collects. Different types within a source (ticket, page, message) are
subtypes of one Entity and pass through
one pipeline.
source_id + source_type + source_entity_id — by
which the
Loader
upserts: exactly one row per source record
draft · final ·
archived — the content's lifecycle in the source,
set by the classify step. A record disappearing from the source is
a separate axis (deletes),
not a status value
refs)
One logical contract, Entity, fans out at load time across three storage paradigms — this is not one record in one database. The exact layout (which field goes to the relational, vector, or graph store) is held by the → Knowledge Store; here — a preliminary picture, refined as the storage is designed.
EntityThe record body — identity, type, status, metadata, the text served in results — and ACL as the source of truth for rights. Everything filtered and joined exactly.
Fragment (chunk) embeddings — child rows of the entity (keyed by its id) with a content hash: the backbone of semantic search by meaning rather than exact word match.
The entity as a node and the within-source relationships the
pipeline builds. Mentions of other sources (refs)
become edges later — already on the Knowledge Store side.
Rights at result time are applied as a filter: a fragment inherits the ACL of its entity (keyed by its id), and the ACL sits in the relational store next to the vectors — so filtering by rights runs in the same query as the search by meaning. Capturing rights and membership is held by acl-identity, and their model and storage by the Knowledge Store.
Three different questions about changes, and each gets its own answer. They are easy to conflate, so the decision is spelled out explicitly: one we store now, another we defer, the third we deliberately do not store.
Open and became Done,
step by step — we do not track. Only updated_at marks
the fact of a change; we do not reconstruct a lifelong feed of
edits to each field.
A record that has disappeared from the source is marked deleted and
hidden from results, but physically retained — this gives an audit
trail (what existed and when it left) and recovery, if the record
returns or the deletion was a mistake. The delete marker is a separate
axis from status, not one of
its values: a disappearance from the source is seen by the full scan of
reconciliation
(a delete never arrives in the increment) — which then records the
moment of disappearance. The classification status (draft ·
final · archived) is preserved as it was, so
recovery is unambiguous: remove the marker — the prior status returns
on its own. archived belongs to content (the author
archived it — the record still exists in the source), a delete belongs
to presence; the two meanings cannot be held in one value.
In v1 we take only what is already text: the body of an email, a text file, a message with a text attachment. It is immediately fit for indexing and embedding.
Extracting text from PDF and docx, recognizing scans (OCR) — a separate processing layer. Deferred to v2: binary attachments are for now stored as links, without parsing their content.
Relationships and deduplication are split between two modules. The boundary rule is simple: what can be established exactly and within a single source — Harvester does; what requires a guess across sources — it hands to the → Knowledge Store.
refs), not materialized into edges: the target node
may not be loaded yet. The format is
below.
source_id + source_type + source_entity_id: a repeat run recognizes
the same record and updates it.
refs format
the entity_ref table · Knowledge Store · before materialization
refs live as a separate row in the
entity_ref table (home — Knowledge Store) — a claim of a
relationship, not yet an edge. Filled in by
normalization;
each carries the minimum sufficient for the
→ Knowledge Store
to later resolve the target and materialize the edge in
entity_edge, even if the target node was not yet loaded at
capture time.
mentions · blocks ·
parent · author ·
duplicate · …
issue · page ·
user · commit · mr ·
message
"PROJ-123" · a URL · "@user" ·
a sha
jira · gitlab · … ·
null if unknown
Three concrete tables whose home is Harvester: the connected source, the journal of its runs, and the queue of failed items for triage. The Admin Panel screens (the hub and the source card) read and edit them through the API — the behaviour is owned by the module.
| id | BigInteger | PK | — |
| name | Text | NOT NULL | display name · “Jira · Acme” |
| connector_type | Text | NOT NULL | connector key · open set, no CHECK · → manifest |
| base_url | Text | NULL | instance address · not needed for some types |
| auth_account | Text | NOT NULLDEFAULTCHECK | as whom · service · personal |
| auth_method | Text | NOT NULLDEFAULTCHECK | by what · static_token · oauth v2 |
| credential_enc | Text | NULL | AES-256-GCM ciphertext · NULL when Disconnected · → secrets |
| state | Text | NOT NULLDEFAULTCHECK | admin intent · active · paused · disconnected |
| scope_mode | Text | NOT NULLDEFAULTCHECK | everything / only selected |
| scope_list | JSONB | NOT NULLDEFAULT | deny-list (all) or allow-list (selected): container ids |
| content_filters | JSONB | NOT NULLDEFAULT | “beyond the rules” toggles · the set is defined by the manifest |
| sync_interval | Integer | NULL | sec · increment interval · overrides the global default · NULL = inherits |
| reconcile_interval | Integer | NULL | sec · reconciliation cadence · overrides the global default · NULL = inherits |
| reconcile_window | Integer | NULL | minute of the week reconciliation starts, org-tz · a daily cadence → minute of day (mod 1440) · overrides the global default · NULL = inherits |
| authority_tier | Text | NULLCHECK | trust level in the source · low · normal · high · input to KS decay · the numeric multiplier is derived by code · NULL = the manifest's default_authority for the type |
| webhook_enabled | Boolean | NOT NULLDEFAULT | real-time channel · DEFAULT false |
| webhook_secret_enc | Text | NULL | ciphertext of the HMAC secret · → secrets |
| incremental_cursor | JSONB | NULL | the since position of the last run · NULL until the first |
| last_probe_at | DateTime(tz) | NULL | time of the scheduled connection probe · NULL until the first |
| last_probe_status | Text | NULLCHECK | probe outcome · ok · unreachable · auth_failed |
| created_at | DateTime(tz) | DEFAULT | server_default=now() |
| updated_at | DateTime(tz) | DEFAULT | now() + trigger |
state — the admin's intent (active ·
paused · disconnected). Health (idle · syncing · error) is not a
separate column: it is derived from the last
run and the
scheduled connection probe. The probe outcome itself is persisted
(last_probe_at · last_probe_status) —
the one facet of health that cannot be derived from a run (a
light probe between runs fails on its own). The hub badges are a
rendering of the two axes, not a third truth.
scope_mode + scope_list and
content_filters are flexible policy, not a frozen
snapshot: which objects scope offers and which “beyond the rules”
toggles are available is declared by the connector manifest, so
the shape of the columns is not fixed — hence JSONB.
credential_enc and webhook_secret_enc
are AES-256-GCM ciphertext; the cipher and the key are held by
the Auth crypto core, Harvester keeps no store of its own.
Outward (API, export) the values never leave — the UI receives
only a mask.
sync_interval (increment interval),
reconcile_interval (reconciliation frequency), and
reconcile_window (when in the week to launch it) —
on a source override the global defaults
(NULL = inherits). The home of the defaults
themselves is platform_settings (core): one global
set per platform, and the source field merely overrides it
pointwise. The window belongs to the heavy, infrequent
reconciliation (night / weekend); the interval-based increment
has none. The minute is stored as an offset without a zone —
the hour is set by the admin in the organization's timezone
(platform_settings.timezone), and the scheduler
unrolls it to the nearest UTC launch moment. The global window
default is Sunday 03:00.
| id | BigInteger | PK | — |
| source_id | BigInteger | FK→sourcesIDX | CASCADE |
| mode | Text | NOT NULLCHECK | fetch strategy · full · incremental · reconciliation |
| trigger | Text | NOT NULLCHECK | what initiated it · connect · schedule · webhook · watchdog · manual |
| state | Text | NOT NULLDEFAULTCHECK | queued · running · succeeded · failed · cancelled |
| scope | JSONB | NULL | the run's subset · NULL = the whole source · window · item list (DLQ) · containers |
| entities_done | Integer | NULL | processed (N) · the total on succeeded |
| entities_total | Integer | NULL | volume estimate (M) for the “N of M” progress |
| checkpoint | JSONB | NULL | resume position within the run |
| error_count | Integer | NOT NULLDEFAULT | count in the DLQ · DEFAULT 0 |
| error_detail | Text | NULL | a brief cause of failure |
| started_at | DateTime(tz) | NULL | NULL while queued · duration = finished − started |
| finished_at | DateTime(tz) | NULL | terminal time |
| heartbeat_at | DateTime(tz) | NULL | worker liveness |
| created_at | DateTime(tz) | DEFAULT | now() |
UNIQUE (source_id) WHERE state IN ('queued','running'):
a source carries at most one unfinished run, and for its duration
the card is locked. Cancel terminalizes the run
(cancelled) — the next starts fresh.
succeeded with error_count > 0 —
the data arrived, some was set aside in the
DLQ for
triage. failed is a total failure of the run (the
source unavailable, the run torn down entirely). So
state and error_count are read
together: green with an errors note is not red.
mode is the fetch strategy (full ·
incremental · reconciliation, exactly
the three sync modes): the since window and the
aggressiveness of resolve.
trigger is what raised the run. scope is
which subset we pull. All three live on the run, not on the
source: the source holds only
incremental_cursor and the schedule. So
partial re-sync and dlq retry
are not separate modes, but incremental with
different trigger and scope: the first — watchdog +
a recovery window, the second — manual + an item
list from the
DLQ.
The “type” of a row in the history is derived from this triple —
it is not stored as a separate column. A run failure raises a
notification through the same channel as the platform's other
alerts.
trigger carries what raised the run
(connect · schedule · webhook · watchdog · manual) — visible in
the run history, complementing mode. But who
among the admins launched a manual run or changed the source
configuration, Harvester does not duplicate itself: the actor's
identity, the action, and the result are written by the
platform's append-only journal of admin actions — its home is
Auth & Security. The audit lives in one place, not smeared
across module tables.
entities_done out of the estimate
entities_total), the “N of M” in progress. But the
source's total volume (the “Entities” column in the hub) is not a
Harvester counter — it is a COUNT of the source's
nodes in the Knowledge Store; the module keeps no column of its
own for it.
| id | BigInteger | PK | — |
| source_id | BigInteger | FK→sourcesIDX | CASCADE · the queue lives as long as the source does |
| run_id | BigInteger | FK→sync_runsNULL | the run of the last failure · SET NULL survives a journal cleanup |
| source_type | Text | NOT NULL | the record's type at the source (ticket · page · message) |
| source_entity_id | Text | NOT NULL | the record's id at the source · what a retry re-pulls |
| reason | Text | NOT NULLCHECK | triage category · permission · not_found · malformed · rate_limited · unknown |
| error_detail | Text | NULL | error text · for diagnostics |
| attempts | Integer | NOT NULLDEFAULT | how many times it failed · grows on a repeat hit |
| created_at | DateTime(tz) | DEFAULT | first set aside |
| updated_at | DateTime(tz) | DEFAULT | last failure · now() + trigger |
UNIQUE (source_id, source_type,
source_entity_id): the same item, failing again in the
next run, does not spawn a duplicate but updates the row
(attempts, updated_at, a fresh
reason) via upsert. The queue is a snapshot of what
is currently unresolved, not a feed of every failure.
reason category encodes the triage, but whether
an item lands in the queue depends on another axis — a transient
error or a permanent one. That classification is declared by the
connector: the default HTTP mapping (429 · 5xx → transient;
403 · 404 → permanent) it may override to suit its own API. The
attempt-and-backoff mechanics themselves are held by
reliability — here arrives only what the retries exhausted.
incremental run manually
(trigger=manual) with a scope from the
source's unresolved rows — that is the dlq retry. Item processed
— the row is deleted (its current state now
lives in Entity via
upsert); failed again — it stays and is updated; gone from the
source — deleted, nothing left to triage. This is a working
queue, not a journal: the audit of “how much failed and when” is
already carried by sync_runs.error_count, so we
clean up the resolved without a duplicating history.
sync_runs.error_count is an immutable snapshot: how
much failed in that run. The “K in DLQ” counter in the
hub and the triage by reason are the live
dead_letters of the source
(COUNT and GROUP BY reason); as things
get fixed it shrinks, whereas the run's error_count
stays as it was. The set of reason categories is the
backend's source of truth, the UI merely renders it.