pgvector: ACL
pre-filter, graph traversal, and upsert are verified on a live database, not
on a substituted store. Mocks cover only pure logic without a database
(chunking, fusion, decay).
pytest -m p0).
pgvector extension (vectors + recursive SQL in one database);
Alembic migrations are applied when the container starts
Entity / chunks /
entity_edge and the ACL five-tuple — building the graph and
scene rights without going through Harvester
@pytest.mark.unit / @pytest.mark.integration /
@pytest.mark.api — by type;
@pytest.mark.p0 / p1 — by priority,
orthogonal to type
body is sliced into pieces; a short one — a single chunk; an empty one — noneUNIQUE (entity_id, ordinal) holdsentity_id, the best chunk stays with the entity; one entity appearing in several lists is deduplicatedchunks → is_empty is true; the source of truth is the presence of chunks to search, not the count of sources or entitieschunk accepted → is_empty is false; the property changes by itself, without a separate command or flagharvester_embedding with zero chunks does not make the base non-empty — emptiness is defined by chunks, not by ingestion readinessis_empty is computed from the presence of chunks (the implementation may cache), never set by hand — like is_available() on SMTPauthority_tier (high > normal > low) yields a higher trust_score; the tier is read by a JOIN on source_idsource_updated_at yields a lower trust_score, a fresh one — higher (time-machine)access_counter (hits · last_accessed_at), long-unqueried decays toward neutraltrust_score, does not touch status / is_deleted — the stale stays visiblesrc→dst with a filter on relation type is built correctlyentity_acl on visited ids — rights land in the same querychunks returns only chunks of allowed entities — the selection by the same SQL, before rankingtext_tsv + GIN) is filtered by the same ACL JOIN — an exact word from a disallowed entity does not leakentity_id → entity_aclgroup_membership) OR a direct grant (entity_acl.source_principal_id); the resolution chain from usersscope='public' (both principal/group NULL) is visible to all — to a user with no memberships and to an anonymous caller under the source's rules; it goes into the filter as scope='public', without a synthetic groupscope IN (group/principal/public) plus consistency: group → source_group_id filled, principal → source_principal_id, public → both NULL; a malformed combination is rejected by the DBentity_acl — nodes without access do not enter the result, without a separate rights projectionweight threshold are dropped from the traversal — weak links are not pulled into context; the filter applies before ACL, holding back inflation from a hub node alongside the fanout capUNIQUE (source_id, source_type, source_entity_id) a repeated upsert updates, does not spawn duplicateschunks → entity_edge in one transaction; FKs holdbody → re-embedding of the affected chunks; unchanged ones are not re-embeddedchunks are deleted, none orphaned (CASCADE on entity_id)source_principal nulls entities.author_principal_id (ON DELETE SET NULL) — the entity survives; deleting a source carries away all its content via CASCADEstatus value outside (draft/final/archived) is rejected by CHECK; NULL is allowed — until the classify stepis_deleted = true → the record and its chunks drop out of all four primitives, the row is physically alivechunks.is_deleted is set in the same transaction; partial HNSW/GIN (WHERE NOT is_deleted) exclude the deleted from the index traversalstatus (draft/final/archived) — the axes are independentusers → KS sets identity.user_id by exact lower(email); Harvester upserts identity → KS backfills the link via users; KS holds both pointsALICE@CORP ↔ alice@corp link up — reconciled by UNIQUE (lower(email)), no case collisions(user_id) WHERE user_id IS NOT NULL — one identity per users; a repeated link is idempotent, does not spawn a second bindinguser_id = NULL, the identity stays unmatched for manual linking in Admin, not an erroruser_identity_edge with origin = curation once both nodes are already in the graph; an entity_ref request until then, and after materialization the row is deletedentity_ref request; the run materializes the edge (origin = curation) after the target arrives in the next increment — the request is clearedtarget_kind = user (e.g. author → identity) is settled by writing author_principal_id into the entity body; no entity_edge is created — not every request becomes an edgeduplicate_ofsource_principal under different emails are matched, identity_id is set; the bridge to users is backfilled across different emailsentities.trust_score into the DBchunks via CASCADEidentity_id = NULL) for manual review, not discardedchunks.embedding_model with the current model → a bulk run recomputes chunks.embeddingembedding_model = the platform's current modelcontent_hashhalfvec(N)) is a migration (new column → CREATE INDEX CONCURRENTLY → atomic swap), not an in-place refresh, and is out of scope hererunning + started_at; the finish — succeeded / failed + finished_atJSONB: edges materialized, duplicates reconciled, entities loweredstate = failed + a brief reason in error; steps already run are reflected in stepsqueued/running/succeeded/failed/cancelled) is rejected by the DBqueued/running on the platformcancelled + finished_at, releases the singleton lock; the next run starts freshschedule/model_change/manual) is rejected by the DB; event-driven re-embedding is written as a row with trigger = model_changequeued, does not fail, and starts when the step finishesON CONFLICT DO NOTHING)duplicate_of is consistentheartbeat_at going stale the lock is taken away, a new run starts; fencing rejects a late writechunks are written with the current model; mixed embedding_model generations converge, ingestion does not stallis_deleted lifts off the partial index, the ACL-JOIN — for a manual measurementchunks.is_deleted + a partial HNSW (WHERE NOT is_deleted) keeps the deleted out of the index — this predicate causes no under-recall, it is not a post-filterentities → the top-K of the approximate HNSW is partly cut off already after the scan: fewer allowed candidates than requested, without an errorhnsw.iterative_scan (pgvector 0.8.0+) only tops up by conditions on chunks itself; a predicate via a JOIN on entities is not compensated by it — hence ACL-recall does not close automaticallyvector_cosine_ops), not by pgvector's L2 default — the metric is fixed as a contract, not inherited by accident from the indexchunks.embedding and the HNSW index are restored — proximity computes as before the backupbackup_snapshots row: running + started_at → succeeded + finished_at + size_bytes + location; the CHECK on state rejects a foreign valuebackup_settings.retention_count → the oldest is removed both from the journal and from storagerunning → on heartbeat_at going stale (~90s) the watchdog reaps it into failed, the single-flight lock releases, the next scheduled one startsWHERE state = 'running': a second backup on top of an unclosed one is rejected, no parallel dumpdestination_creds_enc is stored encrypted, not returned in the API response or data export (like api_key_enc)backup_settings CHECK (id = 1): a second row is rejected; the application reads/updates, does not create422, not 500K hits; K is bounded by a server cap — a request over the cap is truncated to it, not unboundedscore and chunk/entity ids, ordered by descending relevance; an empty result → 200 with an empty list, not 404401 (parametrized across all retrieval routes)202 + run id; a curation_runs / backup_snapshots row in queued, a task in SAQbackup_snapshots → 202; an unknown snapshot → 404409, not data corruptionPOST reindex / re-embed / backup while a run is already queued/running → 409 CONFLICT (single-flight over curation_runs / backup_snapshots) — not a second run and not 500422, not 500200/202; Member → 403401 (parametrized across all KS admin routes)
The absolute recall of the approximate HNSW index under a hard
ACL filter
depends on the distribution of real data and the shape of rights — general
benchmark estimates cannot be trusted, and synthetics do not reproduce it. The
automated test catches only the fact of under-recall and the
boundary of the cure: hnsw.iterative_scan tops up by
conditions on chunks itself, but the ACL comes as a JOIN on
entities and is not compensated by the iterative scan — so the
value itself, that the ACL filter does not drop completeness below acceptable,
is taken by hand on a representative corpus. The purpose is to raise recall
under the actual load via index parameters (m,
ef_search), since iterative_scan does not close the JOIN
predicate — not in the automated run.
Split by type: unit/ — pure logic without a database, runs
on every PR. integration/ and api/ — against a
test Postgres with pgvector in Docker, as a separate, rarer
step. Priority (P0–P1) is orthogonal to the folders and set by
markers (pytest -m p0). The live recall measurement stays a
manual tool and does not enter the automated run.
tests/knowledge_store/module directory
conftest.pytestcontainers Postgres+pgvector · Entity/chunks/edge factories + the ACL five-tuple · fake embedding model · Curation Pass steps directlyunit/pure logic without a database
test_chunking · test_fusion · test_staleness_decay · test_traversal_builderslicing, fusion, decay, traversal builderintegration/against Postgres+pgvector
test_acl_prefilter · test_identity_bridge · test_graph_traversal · test_entity_upsert · test_soft_deleteP0 — load-bearing invariants: ACL, identity bridge, traversal, upsert, soft-deletetest_curation_pass · test_embedding_refresh · test_curation_runs · test_coordination · test_hnsw_recall · test_backup_restoreP1 — curation, refresh, journal, lane coordination, recall, backupapi/HTTP contract — httpx + ASGI + Postgres+pgvector
test_api_retrieval.pyprimitives as tools + the assembled hybridtest_api_access.pyresults are always ACL-filteredmanual/outside CI — a live recall measurement on real data