Achilles — Harvester: Pipeline

The internal processing pipeline: how raw source material becomes a normalized entity and lands in storage. Three stages — Extract → Transform → Load. Source-specific work covers only the entry (extract) and the first transform step (normalize); everything past that is shared across every source and every sync mode.

1 Extract Pipeline entry: a stream of RawItem.

A thin connector per source connector

One connector per source — a thin async client (httpx) over the REST API; official SDKs are not used: they are synchronous and carry their own retry layers, whereas the platform's reliability lives on the raw HTTP response.
Every connector hides behind a single interface fetch(since) → RawItem and emits a stream of RawItem — that is the entry into Transform. The fetch mechanics themselves — connection, authentication, pagination, and the since increment — are held by sources.

2 Transform The chain for full sync and incremental.

1 normalize connector

Raw source data → the unified Entity model. The one transform step written for a specific source.

2 filter

Drops what should not be indexed: empty entities, system events (“X joined the channel”), bot messages. The baseline rules are built in and not configurable. What to pull beyond them — a set of toggles declared by the connector manifest (as with the scope objects): overrides (bring back what was filtered — bots in chats) and extra inclusions (attachments, comments and threads, private channels and DMs). The card shows the ones relevant to the type — source settings. Object-level scope is a neighbouring topic, sources.

3 classify

Entity status: draft / final / archived.

4 resolve

Collapsing duplicates within a single run and picking the winner on a version conflict: one fetch can return an entity twice. Idempotency across runs is not here — Load provides it via upsert on source_id + source_type + source_entity_id. Deep cross-source entity resolution is held by data-model.

5 enrich

Relationships. An edge is drawn immediately when the target node is already loaded; otherwise — a link to another source or to a not-yet-arrived node of its own — it is stored as a mention (refs) in the metadata and materialized later, once both ends are in the database. The boundary is data-model.

6 clean

Stripping markup for the vector: marked-up text → flat. Done late, after enrich: links and mentions in the markup are needed for relationships, and headings for splitting into fragments. Only the copy bound for embedding is cleaned; the original in Entity stays untouched.

7 chunk

Long text → fragments for embedding. One Entity → many fragments, each with its own vector. Target size — 512 tokens (working range 256–1024), a tunable parameter. The model window is not a target but a cap: effective = min(target_chunk_tokens, model_max_tokens). A large embedder window (text-embedding-3 has 8192) buys headroom against clipping long sections, not permission to make a fragment fill the whole window: the longer the text under a single vector, the more it “averages out” and the worse search hits become.

The strategy is by content type, not one global size for everything: Slack — a thread or window, messages stay whole; a Jira ticket or GitLab MR — the ticket together with its comments as one document; code — by syntax (functions, AST), not by a token window; a long Confluence page takes the default route below. This is a valid modern default — just routed by type.

structural The primary cut — by the structure from normalize: headings, sections, a message or a thread. The boundary is natural — no overlap needed.
recursive Fallback: a block larger than the target size is broken down by paragraphs until it fits.
overlap ~15% (≈75 tokens) — overlap at the seam so the boundary context is not torn. Engaged only on the recursive fallback: on a structural cut the boundary is natural, and short content (a Slack message) stays whole, overlap = 0.

8 contextualize v2 AI

Before embedding, each fragment is prepended with a short context that grounds it in the document — which source and section, what it is about; the note is written by a model that sees the whole document and the fragment itself. Embedding the fragment together with that note markedly raises semantic search hits: the fragment stops being torn out of context.

A v2 subtlety for the content cache: the note changes the final string, so the string is hashed together with it — re-embedding happens on its own. The generated blurb is stored alongside the fragment: the model is non-deterministic, and without the saved note the hash would “drift” on every run.

9 embed AI content cache

Each fragment → a vector. The model comes from the AI-model registry — Admin assigns it to the embedding function (assignment by function); this is the first concrete function and the point where Harvester meets model management. Harvester keeps no setting of its own: it uses whichever model is assigned in the registry. The assignment is a precondition for ingestion: there is no default, and until an embedding model is chosen (the platform's built-in models are available out of the box or one of your own), ingestion will not start — otherwise there is no way to pin the dimensionality of chunks.embedding.

The content-cache key is a hash of the exact final string that goes to the embedder, plus everything that affects the vector: SHA256(model_id@version | instruction_prefix | normalized(fragment text)). Changing the model, the tokenizer, the required prefix (query:/passage:), or the normalization rules changes the key → re-embedding. So an unchanged fragment reuses its vector, and a real change hits inference precisely where it matters. How the shared runtime holds this flow and scales under load — embeddings runtime.

Where the parameters live. The “no hardcoding” convention splits the chunk and embed knobs across two homes. Model intrinsics — the max window tokens, the required prefix (query:/passage:), the vector dimensionality — live in the AI-model registry; the chunker reads the cap from there. Tunable knobs — target_chunk_tokens, overlap_pct, per-content-type sizes — are in config. At the seam sits a validator: target_chunk_tokens ≤ model.max_input_tokens.

3 Load Idempotent upsert — the pipeline exit.

Loader

Idempotent upsert on the key source_id + source_type + source_entity_id: each entity is written to exactly its own row, a repeat run updates it rather than spawning duplicates. Idempotency extends to fragments too: their vectors are child rows of the entity, and the same run reconciles the whole set — stale ones are deleted, new ones added, no orphaned vectors from a previous version remain. Unchanged fragments are not re-embedded: the vector is reused by content hash, and inference hits only what actually changed. The module's core guarantee: a repeat sync, or connecting a new source later, does not break already-collected data. One entity meanwhile lands in several storage sinks at once:

relational The record body, metadata, and ACL — structure for exact filters.
vector Fragment embeddings — semantic search.
graph The entity node and within-source relationships — graph traversal.

The full layout — which field goes to which database, plus fields, history, and deletes — is held by data-model.

RawItem → Entity → saved

Contracts

The pipeline rests on two contracts. Each stage knows only them — which is why everything after normalize stays shared.

RawItem

Connector → Transform

Raw source material together with its metadata. The connector's output and the transform's input.

Entity

Transform → Loader

A normalized entity. Different types within a source (ticket, page, message) are subtypes of one Entity and pass through the same pipeline. The full model is held by data-model.