Lifecycle

Harvester delivers data and repairs each source individually; Curation Pass curates the graph across all sources at once. It's the mirror of Harvester · Reconciliation: Reconciliation reads the source API and brings the per-source snapshot into agreement with it; Curation Pass reads the Knowledge Store database itself and reconciles what's visible only when you look at all sources together. The boundary is exactly the one at the boundary with the producer: "certain within a source → Harvester, a guess across sources → Knowledge Store".

The schedule is a single platform one, not per-source: the run covers the whole graph, and there's no point splitting it by source. The cadence is set by → Admin · Knowledge Store. Under one orchestrator — three scheduled steps in v1 (retention v2 being the fourth) plus one by event:

A deterministic step, not a guess. During capture Harvester leaves explicit references as claims in entity_ref, when the target node isn't in the database yet (another source, or a forward within its own). The run walks the unresolved claims, looks up the target by its natural identifier and, on a match, writes an edge to entity_edge (origin=curation) and clears the claim; if the target isn't found, the claim waits for the next run. So the graph is eventually complete: a relation appears the moment both of its ends are in the database. This separates the deterministic assembly of explicit relations from the fuzzy resolution below.

This is deep entity resolution — what Harvester's exact-match didn't catch at load time. Within one source, duplicates are collapsed by Reconciliation's full scan; across sources there's no exact key, and resolution becomes a guess — a job for Curation Pass. Jira's "Ivan Petrov" and Slack's "ipetrov" are one person; two pages about the same release in different spaces are one topic.

The run doesn't discard the unresolved: whatever couldn't be linked fuzzily stays for manual review in → Admin · Account linking — so automation and manual review work off one queue.

Data goes stale and isn't equal in reliability: a ticket closed a year ago and yesterday's release, an official policy and a chat aside — all carry different weight. Curation Pass maintains the ranking entities.trust_score — no deletion, only a lower priority in results. The base is set by source authority, with age and demand modulating it: trust = source weight, lowered by staleness and disuse.

Decay is neither a status nor a soft-delete: the status and is_deleted axes are driven by ingestion, while trust_score is a continuous weight the run recomputes. A record's disappearance from a source is recorded by Reconciliation, not by staleness.

What to keep forever, what to archive, what to delete physically — the closing step of the chain, deferred to v2: in the first version data simply accumulates (the source purges only what it marked deleted itself — via Reconciliation), and our own retention policy comes later. The policy depends on the entity type and the customer's requirements (regulatory periods, volume, sensitivity), so it's set not in code but in → Admin · Knowledge Store alongside the run schedule.

This step stands outside the scheduled chain: it's triggered not by a timer but by an event — a change of embedding model. Vectors computed by the old model are incompatible with the new one: their proximity in a shared space loses meaning, and search breaks while the database is mixed. So on a model change Curation Pass regenerates chunks.embedding and updates chunks.embedding_model — a bulk run that is itself inference AI.

A subtlety of dimensionality: at equal dimensionality old and new embeddings coexist row by row while the run proceeds (told apart by embedding_model). Changing the dimensionality N itself is not a row-by-row refresh but a schema operation: chunks.embedding is typed by its dimensionality (halfvec(N)) and can't hold two Ns at once, so a new size requires reindexing the column.

Cancellation is safe — the swap is the only commit point. Since the new generation is assembled in a separate column while the old index stays authoritative until the swap, aborting the run simply drops the half-built column: no mixed row-by-row state visible to search arises, and the platform stays on the previous model. The cost isn't in integrity but in the inference spent — which is why in Admin cancelling a re-embedding goes through a confirmation, and restarting is a repeat model change.

The mirror of Harvester's run journal (sync_runs) on the Knowledge Store side: one row per run, visibility of progress and outcome. The UI computes nothing — it reads curation_runs and draws progress, status and per-step stats.

Two background lanes write to one graph: delivery — Harvester pulling sources, and curation — Curation Pass tidying the already-assembled graph. There's one rule: normally they run side by side, without getting in each other's way, and only for a short destructive curation window does source sync yield the road to it.

UI. The buttons simply reflect this state, computing nothing. A source under a run — "Sync" is disabled; a destructive curation window is in progress — the start is marked "queued · graph curation in progress". The overall activity indicator is on → Admin · Knowledge Store, once the screen leaves stub state.

Backup runs automatically on a schedule, separately from graph curation. The destination (external storage / S3), frequency and the backups' own retention are set in → Admin · Backups; restore is launched from there too. One Postgres database under the whole module means the snapshot is consistent by construction: body, vectors, graph and permissions land in the backup at one consistent point, with no assembly from several DBMSs.

The relational body, pgvector and the graph live in one Postgres, so the projections and the ACL change transactionally — no cross-database synchronization, no window where permissions and content have diverged across DBMSs. The source of truth is the relational body; vectors and edges are its projections.

Synchronous loading (one transaction, a fixed projection order) is held by the data model. Curation, by contrast, is asynchronous: Curation Pass reconciles cross-source edges and fuzzy relations in a separate run, on top of the already-consistent body. So the graph is eventually complete — an edge is visible at once when both nodes are already in the database; a reference to a node that hasn't arrived yet (as with cross-source) is reconciled by the run — while the relational body is always correct.