Connectors

Behind a single interface, a connector covers only what's unique to each source. The contract surface is four source-specific capabilities: fetch the raw material and bring it to the shared model (the pipeline core), plus two operational capabilities without which a connection can't be set up — a catalog of scope objects and a staged probe. Everything after normalize is written once and works for any source.

A connector doesn't just know how to reach a source — it declares what type it is and what it needs to connect. The registry serves the manifest over the API, and the add-source wizard in Admin builds a form from it: not a line of frontend per specific connector, the fields appear on their own. The backend is the source of truth, the form is its projection.

The remaining manifest fields don't go into the form — they're declarative extensibility hooks: the connector describes its behavior with data, and the core applies a single mechanism. That's how a diverse source slots in without core edits — the limiter, webhook intake, retry/DLQ.

• rate_limit (requests/sec) sets a safe starting rate — the rate collection begins with on a cold start, while the real plan limit is still unknown. From there, adaptive throttling ramps higher while the source responds calmly and backs off on a sign of overload. There's no hard ceiling here — the real boundary is dictated by the source's own responses, not this number.
• rate_limit_scope — what the limiter key is built on: tenant · account_token · workspace_method · site. Providers cap the limit per account, token, or workspace, not per “source”, so two sources on one token share one budget — the key must reflect this, or the limiter miscounts.
• request_cost (default 1) — the limiter counts cost units, not requests: one heavy call spends the budget of a dozen light ones. At Atlassian these are points (search = 51), at Slack it's a per-method weight. The connector assigns a cost to its calls, the core deducts it from the key's shared budget.
• webhook — the incoming-call verifier as a declaration: the signature scheme (HMAC over the body at GitHub and Slack, a static token in the header at GitLab, and where the source doesn't sign — a fallback to a secret unguessable endpoint), extraction of the dedup key, and the presence of a timestamp for anti-replay. The connector declares a ready verifier or overrides for the exotic; the core applies a single intake mechanism. This is behavior, not a value — it doesn't go into the UI; the only UI-facing field here is the secret itself. The intake policy (authenticity + once) is held by security.
• error_class — the connector's error classifier transient / permanent with a default HTTP mapping (429/5xx → transient, 4xx → permanent). Non-HTTP sources (S3, DB, files) override the mapping for their own error codes — and slot into the shared retry/DLQ without reworking the core: whether to retry or drop to the DLQ, the core decides by class, not by source type.
• default_authority — the starting trust level by source type from a closed set of low · normal · high: policy and wiki bases — high, tickets — normal, chat and bots — low. The connector declares a sensible default, the admin overrides it on the source card (sources.authority_tier), and KS decay unfolds the level into a numeric ranking multiplier — the level → weight mapping itself lives in code in one place, not in the source row. The closed set is the very limit: the admin doesn't enter a “raw” number and doesn't miss the scale. A value, not behavior — unlike the rest of this list, it is UI-facing.

A connector self-registers on import — by a decorator or via discovery at startup: from the built-in package and from external packages that declare a Python entry point. The registry is the single source of truth about supported types: the type selection in the wizard, the connection form, and the dispatcher that spins up the right connector by connector_type all read the same registry.

• Adding a source type = adding a connector class; removing = removing the class. No edits in the wizard, in the frontend, or in the dispatcher. The class lives in the built-in package or arrives as an external library — the registry doesn't tell them apart.

The built-in connectors. Each is a thin async client (httpx) over the source's REST API: the official SDKs are synchronous and carry their own retry layers, whereas platform reliability (AIMD, Retry-After, error_class) lives on the raw HTTP response. The scope objects define what the admin selects at the Scope step.

• Starting rate — a conservative req/s that collection begins with on a typical (not top-tier) plan; from there adaptive throttling ramps higher while the source responds calmly and backs off on a 429. This is a manifest field of the specific connector (rate_limit), not a global constant — each type has its own ceiling and its own counting unit.
• Atlassian (Jira, Confluence) counts not in requests but in points (request_cost): a search costs ~51 units, so the starting rate here is a points/s budget, not req/s.
• Slack is special: the limit is per method (rate_limit_scope = workspace_method), so there's no single req/s — the rate is per-method. The sharp edge is reading channel history: for non-Marketplace apps the provider caps it down to 1 request/min for 15 messages. That's why the install model is internal-app: the client registers an app in their own workspace, and internal apps aren't subject to the cap — history is collected without the Tier-1 ceiling.
• Starting authority — the type's default trust level (default_authority): wikis and knowledge bases higher, trackers mid, chat lower. It's a starting label, not a verdict — the admin overrides it per specific source, and KS decay takes the level as a ranking baseline.

The source isn't in the catalog — write your own: one class on the same contract. Three steps — and the whole skeleton below; from there the connector slots into the type selection and the connection form on its own.

✦ canonical
  pip package → entry point  achilles.connectors
  core untouched · platform upgrade stays clean

○ quick dev
  in-tree → achilles/harvester/connectors/custom/
  own folder · doesn't touch the built-ins

# 3 · discovery picks it up on its own
@register
class GitHubConnector(BaseConnector):   # contract: 4 methods

    manifest = Manifest(                  # defines the form and the card
        type         = "github",
        credentials  = [PersonalToken],
        config       = [BaseURL],
        scope        = [Repository],      # the Scope step
        collect      = [Comments, Attachments],
        capabilities = [Incremental, Webhooks],
        rate_limit   = 1.0,               # starting req/sec (GitHub: 5000/hour)
        rate_limit_scope = AccountToken,  # limit per token, not per source
        request_cost = 1,                 # cost units per call (default)
        webhook      = Hmac("X-Hub-Signature-256", dedup="X-GitHub-Delivery"),
        error_class  = HttpErrors,        # 429/5xx → transient, 4xx → permanent
        default_authority = Normal,       # starting trust level: Low · Normal · High
    )

    def fetch(self, since):              # extract → stream of RawItem
        ...                               # httpx or any SDK — the contract doesn't mandate HTTP

    def normalize(self, raw):            # transform → Entity
        return Entity(...)               # then — the shared pipeline

    def list_catalog(self):              # catalog of scope objects
        ...                               # live selection + scope reconciliation

    def check_connection(self):          # staged probe: credentials + rights
        ...                               # URL reachability — shared layer
#  either way below — in-tree or as a pip package — the connector is visible
#   only after the image is rebuilt and the container redeployed:
#   discovery reads classes at process startup, not from Admin

├── base.py        BaseConnector — the 4-method contract
├── registry.py    discovery: package + entry points
├── jira.py        ┐
├── confluence.py  │ built-in
├── gitlab.py      ┤ connectors
├── slack.py       ┘
└── custom/        ← the platform doesn't touch this folder
    └── github.py  ← your class goes here

# core untouched — discovery finds it via the entry point
├── connector.py   ← the same class
└── pyproject.toml [project.entry-points."achilles.connectors"]

The code is your own and trusted — the operator decides what to install: a shared process, no sandbox, a bespoke integration, not a marketplace. The < 1.0 contract isn't frozen — the package is rebuilt for the platform version.