← Knowledge Store

Embeddings runtime

knowledge-store · workzone

The embedder is one shared service for the whole platform: it turns text into a vector, and Harvester, Knowledge Store, Query Engine, and Agent Engine all call it. The load-bearing decision was already made on the model registry side: the embedder is factored out as a separate network service (OpenAI-compatible, weights loaded lazily), not baked into the backend. So scale here is an operations question, not a redesign.

Load paths

Four sources of load converge on one shared runtime.

Harvester · ingest
the embed stage — chunks in batches at ingestion
background · can wait
Knowledge Store · re-embedding
a bulk run on model change
background · can wait
Query Engine · search
a query vector for every user query
user awaits the answer
Agent Engine · search
the KS search primitive in the agent loop — an intent vector
background · small · online path
→
embeddings :80
OpenAI-compatible · model-agnostic · weights lazy, the assigned model warmed
→vector
Path Volume Latency Waiting
Search · QE a small request user awaits the answer critical
Search · Agent a small request background, but small tolerates · online
Ingest · Harvester large batches background tolerates
Re-embedding · KS a bulk run, rare background tolerates

Per-source unevenness is not the embedder's concern. The chunk stream no longer carries a source: the embedder cares only about the total volume, while the pace and steadiness of ingestion per source are held by Harvester's ingestion workers.

The bottleneck is contention, not capacity → Cache & Workers

The risk is not the volume itself but the different shape of load across the paths. A big reindex of hundreds of thousands of chunks can occupy the runtime entirely — and a user's query queues up behind it, search slows down. It's a matter of priority, not of hardware count.

Search does not share a limit with the bulk run. Background load (ingest + re-embedding) goes through a task queue with bounded concurrency; search requests call the runtime directly, outside that limit. So the interactive path doesn't queue behind the whole reindex — the limits are separate, not shared. Ingest already runs on background workers; their concurrency cap is simply not shared with search. This doesn't give full decoupling in v1: the runtime is one, and an online request will still wait for the current bulk batch in flight — but the delay is bounded by one batch, not the whole queue; the flows are fully separated by distinct online/bulk pools in v2 (below).
Agent search takes the direct path, not the bulk queue. Although the agent loop is background, its call to the embedder is a small query vector through the same KS search primitive as a user's: queueing it behind a reindex would freeze the loop for minutes for the sake of a millisecond task. So it shares the online path with search, not the bulk limit. It adds no volume: the pace of agent requests is capped from above by the agent concurrency cap, not by the embedder's throughput.
The first query doesn't wait for the weights. Lazy loading protects deployment, but on the latency-critical path it would translate into seconds of cold start. So the runtime warms the assigned model proactively — on its assignment in Admin and on its own startup, not on the first search call: by the first live query the weights are already in memory. The desired model is persisted next to the weights cache, so a restarted container reloads it by itself — a restart mid-download resumes rather than coming back empty.
Memory is a budget, not a hope. The runtime reads its container limit at startup and keeps loaded models as long as they fit: on a switch the new model loads alongside the old one, which keeps serving search until the new weights are ready — eviction (LRU) fires only under real pressure, so a roomy host gets a seamless switch and a small one peaks at a single resident model as before. A model that cannot fit even alone is rejected at assignment (MODEL_TOO_LARGE; the preflight runs before anything commits) instead of surfacing later as an OOM kill. The per-model phase (loading / ready / error) is served on a dedicated status endpoint — the Admin screens show “loading weights” and “re-indexing” as distinct stages — while /healthz stays pure liveness: a container busy loading weights is alive, and the autoheal watchdog must not shoot it.
If the embedder doesn't answer — search degrades, it doesn't fail. The runtime call on the online path runs under a bounded timeout: hanging in wait for a vector while the user awaits an answer is unacceptable. If it misses the deadline or is unavailable, search returns a result from the remaining primitives that don't need the vector (lexical, graph, filters): the vector is just one of the two text legs; without it the results get poorer, but the query doesn't go unanswered — a vector failure is visible honestly, not hidden behind a hang. On the bulk path a failure brings nothing down: the chunk goes back into the queue and is picked up by the ingestion workers' retry; no user is waiting on it.
Scale: v1 → v2

The service has two capacity levers. Batching — the runtime accumulates requests for a few milliseconds and computes them as one batch; it gives a multiple throughput gain on a single container and comes free with a batching-capable runtime. Replicas behind a load balancer — N copies: embedding is stateless (text in, vector out, no state between requests), so any copy can serve any request. Squeeze batching first, then replicas.

Doing now v1
search ───→
embeddings :80
ingest · re-emb. → queue · limit →
  • One logical service behind a configured address.
  • A runtime with dynamic batching — as a requirement.
  • One host → scale vertically.
Deferring v2
load balancer
⤋
replica
replica
replica
  • A pool of N replicas behind the same address.
  • Autoscaling with load.
  • Pool split: “online” (search) vs. “bulk” (ingest).
  • Requires orchestration (k8s / GPU).
The v1 → v2 transition — with no edits on the consumer side. Both deployments live behind one address (base_url): growing the pool means changing the address's target; the code of Harvester, Knowledge Store, and Query Engine doesn't change.