What we test and with what
Transform → Load is identical for all sources — a deterministic unit
test takes it on synthetic RawItems. Only Extract and
normalize are source-specific: we test them against
recorded responses and images, and live — only by hand.
Three levels by reproducibility
Reproducibility grows from the live source to the image to the HTTP
fixture: live free-tiers — manual debugging, outside CI; Docker images
— integration as a separate, rarer CI step; HTTP fixtures
(VCR.py / WireMock) — unit, contract, and end-to-end replay by source
on every PR. make test includes only the fixture level;
Docker integration runs as a separate step.
The API layer tests the edge, not a rerun of the domain
HTTP tests hold the boundary: request shape and schema validation,
response codes, secret masking, authorization on every route.
Behavioral depth — run locks, delete cascades, the three Test
Connection steps — is held by the domain tests above; the endpoint
need only reflect them outward correctly. End-to-end RBAC / JWT / CORS
mechanics are owned by Auth & Security; here it's only that
Harvester's routes sit behind that guard.
Contract cases — and a kit for the connector author
The same set the platform uses to check its built-in connectors
(test_connector_contract +
test_manifest_registry) is run by an external connector's
author against their own class — a conformance check against the
contract before shipping. The contract isn't frozen before
1.0: the kit changes along with it.
Stack and infrastructure
On hand donepytest, pytest-asyncio, pytest-cov, httpx
filter — built-in rulesempty entities, system events, and bots are filtered out; the base rules aren't configurable
filter — manifestoverrides and extra inclusions are taken from the manifest's content_filters
classify → statusthe entity is assigned draft / final / archived
resolve — duplicates within a runcollapsing duplicates within a single run + picking the winning version; cross-run idempotency — NOT here
enrich — links and refsan edge is built at once when the target node is already loaded; otherwise (a foreign source or a forward ref) — as a refs claim, the edge isn't materialized
clean — a copy for embeddingcleaned after enrich and only the copy; the original in Entity is untouched
chunk — fragmentsone Entity → many fragments; the structural / recursive / overlap mechanics
embed — model and cacheeach fragment → a vector from the AI registry's model; cache by content hash, unchanged ones aren't re-embedded
embed — assignment is requiredwithout an assigned embedding function ingestion won't start (there's no default); with the platform's built-in model assigned → ingestion proceeds, fragments are embedded with it
upsert — one keyby source_id + source_type + source_entity_id exactly one row; a repeat run updates, doesn't breed duplicates
fragment syncthe vector set is reconciled as a whole: stale ones removed, new ones added, no orphans
unchanged fragmenta vector by content hash — isn't re-embedded on a repeat
three projectionsone Entity → relational + vector + graph (the field layout is owned by Knowledge Store)
fragment ACLa fragment inherits the entity's ACL, has none of its own
deletion — a separate axisa record gone from the source gets a deletion marker; the classification status (draft / final / archived) is kept as it was — hidden from results, physically retained
restorationthe marker is cleared (the record returned / the deletion was a mistake) → the prior status returns on its own, without reclassification
ContractConnectors — a unified interface on HTTP fixturesVCR.py + WireMock · every PR
contract — 4 methodsthe connector implements fetch · normalize · list_catalog · check_connection; here — the pipeline core fetch + normalize, catalog and probe are under the api/scope tests
output shapefetch yields a stream of RawItem with payload, identifiers, and collection metadata
increment by sinceFull → since = epoch; Incremental → the moment of the previous run
cursor paginationlisting → page → emit; next cursor exists → back to listing
empty / last responseno cursor → the loop finishes cleanly, without an extra request
request failure (WireMock)timeout / 429 on a page → retry with backoff, the loop doesn't break
normalize → Entitythe sole source-specific transformation step yields source_id + source_type + source_entity_id, type, status
test_manifest_registry.py
The manifest declares, the registry self-registers
form from the manifestthe UI builds the connection form from the declared config fields and credential kinds — the frontend knows nothing about the specific connector
not in the formrate_limit (starting rate) and webhook (behavior) don't go into the form; from webhook, only the secret itself reaches the UI
self-registrationa connector registers via @register / discovery; the registry is the single source of truth about types
external packagea connector from an installed package with the achilles.connectors entry point takes its place in the registry as a built-in — discovery doesn't distinguish the class's home
new typeadding a type = adding a class, without touching the wizard / frontend / dispatcher
dispatch by typethe dispatcher raises the connector by connector_type
Replay · P0End-to-end pipeline on cassettes — by source typeVCR.py · offline · every PR
test_pipeline_replay.py
Full E→T→L over recorded responses, a golden reference per source
parametrized by typeone run per built-in type — Jira / GitLab / Slack — from its own committed cassette
the whole pipeline offlinefetch from a cassette → Transform → Load with no network and no image; end-to-end coverage where Docker gives only the Git family (Slack/Jira have no image for regular CI)
entity golden referencethe count, types, and keys of entities are checked against a golden snapshot; a mismatch is a failure, not a warning
run determinisma fixed seed for chunking and embedding → a bit-for-bit repeat, a diff readable in review
re-recording a cassetteAPI drift is fixed by re-recording from manual/; a cassette is a committed file, touches no network in CI and doesn't "fall over"
Integration · P0End-to-end pipeline against an imageDocker · separate CI step
test_pipeline_e2e.py
Extract → Transform → Load against a seeded source
checkpoint resumecommit after a batch → resume from the last checkpoint, an unfinished page without duplicates
heartbeat (time-machine)30 s interval; a crash = silence after 3 missed beats (90 s)
freshness budgeta gap > 6 h → the checkpoint is discarded, the run starts from scratch; cancelled isn't resumed
retry → DLQbackoff on a drop / timeout / 429 / 5xx; after exhaustion the item goes to dead_letters, isn't lost
permanent → straight to DLQ400 / 401 / 404 / 422 and normalization errors — a retry is pointless, the element goes to dead_letters without retries
transient → retrya load spike / brief unavailability — retry with backoff; the error class decides, not the letter of the code
connector overrides the class403 at GitHub / GitLab — more often rate-limit: by Retry-After / X-RateLimit-Remaining the connector treats it as transient, not permanent
two retry levelsa short delay (≤ 60 s) — in-memory holding of the page in the task; a long pause (Retry-After in minutes-to-hours) → SAQ defer / re-enqueue, the worker slot freed, not a sleep
partial failurea succeeded run with error_count > 0 — distinct from failed
failure notificationa failed run raises a notification; the channel choice is owned by the notifications module, Harvester only signals
one active runpartial unique index UNIQUE (source_id) WHERE state IN ('queued','running')
DLQ dedupthe same item updates the row (attempts, reason), doesn't breed a duplicate; a processed one is removed
Full Syncsince = epoch, passes every entity; a repeat doesn't breed duplicates
Incrementalwebhooks + polling by since — the working mode after the initial load
Reconciliationreconciles the whole, aggressive resolve, cleanup of what vanished
deletion detection is mode-specificIncremental doesn't set a disappearance marker (deletion doesn't arrive in the delta); Reconciliation marks the missing via a full scan
partial_resync (12 h)silence > 12 h → cheap polling → delta present → window re-sync, auto, no human
dlq_retry — manuala targeted retry by identity from the queue after a fix; processed ones leave the DLQ
per-sourceeach source holds its own since / schedule / mode; "sync everything" = launching them all at once
authenticityHMAC over the body (GitHub / Slack), a static token in the header (GitLab), fallback — a secret endpoint
verifier by providerparametrization of the pluggable verifier across 4 providers, including Atlassian (JWT Connect apps); the core runs a single order of freshness → signature → dedup
unverified → rejecta forged signature / token → the call is rejected
timestamp freshnesswhere there's a timestamp (Slack, GitLab) — a call older than the 5 min window is discarded before the signature; for a source with no timestamp (GitHub, Atlassian) the step is skipped, anti-replay is carried entirely by dedup (a larger window TTL — 24 h)
anti-replayby timestamp (Slack) / delivery-id (GitHub) — a repeat is discarded (TTL memory in Redis)
TLS onlythe channel is terminated by the reverse proxy over TLS
spike → notificationrejections to the log; a spike → a Security-type notification
starting rate feels out the limitstart from the manifest's rate; on calm responses AIMD ramp-up of +10% per window — the rate feels out the real limit upward, not only backing off on 429
provider signal beats AIMDRetry-After — a hard pause over the whole scope, overrides the adaptive rate; respected by all workers
pre-emption by RemainingX-RateLimit-Remaining low → slow down early: safe_rps = Remaining / (Reset − now), take the min with the AIMD rate
accounting by costthe limiter charges cost-units, not a request count; how many units a request costs is declared by the connector (1 by default)
key by rate_limit_scopethe limiter key is built from the manifest's rate_limit_scope (tenant / account_token / workspace_method / site); two sources on one token share a budget
rate outlives runsthe learned API capacity isn't reset between runs — it lives in Redis with a TTL in hours, outlives a worker restart
ACL in source termsproject / space / channel are kept without mapping onto platform levels
two-sided capturethe group tag on an entity + group membership
full import of peopleon source connection people are imported in full; incremental picks up new ones
identity by exact emailmerging people into an identity by an email match; non-matching ones → manual review in Admin, Harvester doesn't guess
bridge to users on upsertan identity upsert with an existing users of the same lower(email) → identity.user_id is set; no users → NULL
fuzzy merge — not heredifferent emails → different identities; "the same person under different emails" Harvester doesn't stitch — deep resolution is owned by Knowledge Store
rights syncthere's no separate mechanism: Incremental in a targeted way, Reconciliation via a full pass
revocation is eventually consistentwhat's revoked is visible until the next incremental / resync / reconciliation — an accepted trade-off
enforcement — owned by Query Enginethe pre-filter on results isn't here; Harvester only stores the ACL
Integration · P1Secrets, DB control layer, scope, scheduling
cascade from the sourceDELETE sources → sync_runs and dead_letters go via ON DELETE CASCADE
run_id → SET NULLclearing the run journal nulls dead_letters.run_id, the queue row lives on — a different ondelete, easy to mix up
CHECK fieldsan invalid value for an enumerable column under CHECK (state, mode, trigger, scope_mode, reason, auth_method, last_probe_status, …) is rejected by the DB
open set of typesthere's no CHECK on connector_type — the manifest validates it, not the DB
health is derivedin the DB only state; idle / syncing / error are derived from the run + probe, not stored as a column
run type is derivedpartial_resync / dlq_retry — a triple (mode + trigger + scope), no separate column
schedule resolutionper-source sync_interval / reconcile_interval = NULL → inherits the global default; set → follows its own
reconciliation on schedulea daily / weekly reconciliation run is launched by the scheduler, not by hand
health check on schedulea lightweight probe (Test Connection steps 1–2) between syncs; failure → health error + notification
watchdog on a timersilence > 12 h → the watchdog raises a partial_resync itself, no human
"sync everything" = fan-outlaunching all per-source at once; independent sources run in running in parallel — a partial index on source_id, not a global one
API · P1Source endpoints — HTTP contracthttpx · ASGI app · DB · fake connector · → HTTP API · → Conformance
createa valid form → 201 + a body with the id; followed by an auto Full Sync (trigger=connect)
form from the manifestfields and credential kinds are validated against the connector's manifest; an extra / foreign field → 422 VALIDATION_ERROR
CHECK surfaces as 4xxan invalid auth_method / scope_mode, an empty name → 422, not 500
type outside the registryan unknown connector_type → 422 (checked against the registry, not a DB enum)
readlist and detail → state + derived health + last run; entity count from Knowledge Store
secret maskGET / list return the credential and webhook masked; the plaintext value — only in the create response
edit configPATCH partial; a credential change → re-encryption, a new mask outward, not the value
no resourcea nonexistent id → 404 NOT_FOUND
health — a light probeGET /health → state + computed health (idle / syncing / error) + last_probe_status, without the full source detail — a cheap endpoint for frequent polling
draft and existinga probe on a config from the wizard and by source id → 200 with a step-by-step result
3 steps separatelyURL unreachable / credentials invalid / too few permissions → distinct machine codes per step, not a generic 500
step 2 failedstep 3 in the response is marked "not checked"
catalog after step 2GET of the object catalog is available only after successful credentials; before — 409 / empty
the scheduled probe shares the contractthe background health check writes last_probe_status (ok / unreachable / auth_failed) and drops health to error — the same logic as the manual one
test_api_sync.py
Launching runs, fan-out, reading the journal and DLQ
anonymousno token → 401 UNAUTHORIZED (parametrized across all routes)
expired tokenan access JWT older than 15 min → 401
no "bare" routesevery write endpoint actually sits behind require(permission) — a parametrized audit of guard coverage
API key — read onlya read-only key on a write operation → rejected; never wider than the owner's rights
webhook outside the sessionpublic intake — a signed channel, not JWT; its contract is held by test_webhook_security
ManualLive free-tier sources — outside CImanual debugging · source of truth for fixtures
Free tiers of real SaaS with seeded sample data. The purpose — manual
debugging of a new connector live: the API's actual behavior, response
shapes, and the quirks of pagination and permissions that a mock can't
reproduce are all visible. They're unstable, require tokens and a
network — not part of the automated run; they serve as the source of
truth when recording HTTP fixtures (tokens are scrubbed on record).
AtlassianAtlassian Cloud free — Jira + Confluence via
atlassian-python-api; the real permission model and
pagination
Slacka free workspace via slack-sdk — channels, threads,
rate-limit live
GitLabgitlab.com + a seeded project via
python-gitlab — issues, MRs, wiki
StructureTest file structure
The split is set not by folders but by purpose. unit/,
contract/, and replay/ are part of make test
and run on every PR (deterministically). integration/ and
api/ — as a separate, rarer step: the first on Docker
images, the second on a running app and DB. Live sources remain a
manual debugging tool and don't enter the automated run. Priority
(P0–P1) is orthogonal to the catalogs and is set by markers
(pytest -m p0).