Achilles — Harvester: Security

Harvester pulls data from external networks, and what needs protecting is the ingestion channel itself — two vectors: verifying incoming webhooks and throttling the rate of calls to third-party APIs. Narrowly about the channel: encryption of credentials is held by sources, and carrying access rights over from the source by acl-identity.

1 Webhooks Accept only what genuinely comes from the source.

The core applies — the connector declares

No hardcoded if per provider: the core holds a single verification mechanism, and the connector brings its own pluggable verifier — it declares the signature scheme, how to extract the dedup key, and whether the delivery carries a timestamp. The core runs three steps in a strict order — freshness → signature → dedup (cheap → expensive): first filter by timestamp and headers, and only what passes the signature is let into dedup — otherwise an inauthentic stream would itself become a DoS on the dedup store. A new source = a new verifier; the core stays untouched.

Call authenticity

Every incoming call must be proven to come from the source — otherwise foreign content gets into the pipeline. The signature scheme is declared by the connector: HMAC over the body (Slack, GitHub), a static token in a header (GitLab), or a Connect-app JWT (Atlassian); where the source does not sign at all — fall back to a secret, unguessable endpoint. Authenticity not confirmed — the call is rejected.

Timestamp freshness

Where the provider sends a timestamp (Slack, GitLab), a call older than the freshness window (uniformly 5 minutes) is discarded at once, before the signature — this is both the first barrier against replay and the cheapest way to reject junk. For sources without a timestamp (GitHub, Atlassian) the step is skipped, and dedup alone carries replay protection.

One call — once

A signature does not guard against replaying an intercepted call. The dedup key of each delivery (extracted by the connector's verifier) is placed in Redis with an atomic SET key NX EX — the first time sets it, a repeat sees the key and is discarded. The TTL depends on the timestamp: 15 minutes where there is a timestamp (freshness has already cut off the old), and 24 hours where there is none — to catch manual redeliveries too. The channel itself is TLS-only (terminated at the reverse proxy).

Declarations of today's connectors

Slack — HMAC over v0:ts:body, timestamp present (5-min window), dedup by event_id.
GitHub — X-Hub-Signature-256, no timestamp, dedup by X-GitHub-Delivery.
GitLab — static token (a plan to migrate to an HMAC signing token), dedup by Idempotency-Key / Event-UUID.
Atlassian — X-Hub-Signature (admin) or JWT (Connect), no timestamp, dedup by X-Atlassian-Webhook-Identifier.

Fend off junkv2

Knowing the endpoint address, one can flood it with junk — each call costs a signature check. That is why the checks run from cheap to expensive (freshness and headers — before HMAC, and inauthentic traffic never reaches dedup), a rate limit sits on the endpoint at the reverse proxy, and the path itself is unguessable (a secret in the address).

A rejection — to the log and an alert

Every rejected call goes to the log: a lone one is noise, a spike is a signal that a secret has drifted or the endpoint is being probed. On a spike a Security-type notification is raised — through the same channel as password brute-forcing.

Where webhooks are configured

A webhook is tied to a specific source — the secret and the address are set up in the same place as the connection; which events to send is chosen on the source's side, not ours. That mechanic is held by sources; here — only the verification of what arrived over the channel.

2 Rate limiting per scope → Cache & Workers Don't get us banned — and don't take the source down.

A limit by scope, not “per source”

Providers limit not a “source” in general, but an account, token, workspace, or site — and the thresholds all differ, with their own tier (tier) caps. So the limiter key is built from rate_limit_scope in the connector manifest (tenant / account_token / workspace_method / site), not from an abstract “source” — there is no single figure across them all. The manifest also declares the starting and maximum rate: a safe starting point for a typical tier, while the runtime feels out the real boundary from the source's own responses.

Count cost, not requests

“One request” is not the unit of load: on Atlassian a search costs 51 points, on Slack the limit differs per method. So the limiter charges not requests but cost units (cost per request, default 1) — and how many units a given request costs is declared by the connector. This way the providers' heterogeneous billing maps onto a single counter without reworking the core.

Adaptive rate (AIMD)

The rate adapts to the source via an AIMD controller: on calm responses it ramps up +10% of the lower bound per evaluation window (≈10 s / 20 requests), on a 429 it drops sharply ×0.5 (additive-increase / multiplicative-decrease). The floor and start are the rate from the manifest, the cap is its maximum. The current rate is not reset between runs — it is the API's learned capacity; in Redis it lives with a TTL of hours. The executor is a distributed token bucket in Redis, with atomic (Lua) charging, a per-scope key, surviving a worker restart.

Provider signals outweigh AIMD

When the source speaks directly — we listen to it, not to our own guess. Retry-After is a hard pause on the entire scope, respected by all workers, and it overrides AIMD. And if X-RateLimit-Remaining is low — we slow down ahead of time: safe_rps = Remaining / (Reset − now), taking the min with the AIMD rate.

The seam with retries

Throttling and retries are one line: bounced by the limit — the retry goes out already with a delay. The retries themselves and their backoff are held by reliability.