The core of Knowledge Store is the knowledge entity. One logical
Entity contract, produced by Harvester, is persisted here and fans out into
three projections within one
Postgres database: the relational body (entities), chunks with vectors and full text (chunks), and a graph of relations (entity_edge).
Everything else hangs off entities: permissions
(entity_acl → entities), search
(four primitives), and graph curation
(Curation Pass). How the Harvester contract lands in these tables is covered at the
boundary with the producer.
An entity is neither a single row in one database nor three copies across three DBMSs. It is one record with three projections inside a single Postgres: the body is filtered and joined exactly, chunks are searched by meaning, edges are traversed by relation. All three live side by side, so the single ACL pre-filter and the fusion of four paradigms run in one SQL query, without cross-database sync.
Identity, type, status, metadata, text for retrieval — and the
source of truth for permissions (entity_acl points here).
Everything filtered and joined exactly.
An entity's chunks are child rows (keyed by its id),
each with two indexes: a dense embedding for semantic search and a
sparse tsvector for keyword search. The backbone of
text retrieval — both signals side by side.
An entity is a node, there is no separate node table; relations are edges between entities. Structural ones come from Harvester; cross-source ones are materialized by Curation Pass.
entities row (the others reference it by FK),
then its chunks (re-embedding only when
content_hash changes), then structural entity_edge
rows whose two nodes are already in the database. Cross-source edges and
curation are asynchronous, driven by
Curation Pass.
entities
The module's central table — the canonical snapshot of a source record and the source of truth referenced by all the other projections and permissions. Its identity matches the idempotency key of the Harvester contract: one row per source record, and a repeat run upserts it.
| id | BigInteger | PK | — |
| source_id | BigInteger | FK→sourcesIDX | source · CASCADE |
| source_type | Text | NOT NULL | subtype · ticket · page · message |
| source_entity_id | Text | NOT NULL | native record id · part of the upsert key |
| title | Text | NULL | title |
| body | Text | NULL | text for retrieval · split into chunks |
| url | Text | NULL | permalink in the source |
| status | Text | NULLCHECK | draft · final · archived · set by the classify step |
| author_principal_id | BigInteger | FK→principalNULLIDX | author · bridge into the world of identities |
| source_created_at | DateTime(tz) | NULL | created in the source |
| source_updated_at | DateTime(tz) | NULLIDX | edited in the source · staleness input |
| is_deleted | Boolean | NOT NULLDEFAULT | soft-delete · a separate axis from status |
| deleted_at | DateTime(tz) | NULL | when it vanished from the source · set by reconciliation |
| content_hash | Text | NULL | change detection · saves re-embedding |
| trust_score | Real | NULL | ranking weight · driven by Curation Pass: source authority, reduced by age and demand (decay) |
| meta | JSONB | NULL | flexible attributes by entity type |
| created_at | DateTime(tz) | DEFAULT | now() |
| updated_at | DateTime(tz) | DEFAULT | now() + trigger |
updated_at). A record vanishing from the
source is is_deleted, a separate axis from
status: the record is hidden from retrieval but kept
physically for audit and recovery. The moment of disappearance is
set by the full scan of
reconciliation — deletions don't arrive in the incremental stream.
chunks
Embeddings are not one vector per record, but per chunk: a long
document is
split into pieces, each embedded separately. That same chunk text also carries a
sparse full-text index (tsvector + GIN) —
lexical keyword search alongside vector search by meaning. A chunk is a
child row of an entity and has no ACL of its own: it inherits the
parent's permissions
via entity_id, so the permission filter runs in the same
SQL query as the proximity search.
chunks by source_id for
volume is deceptive: each partition gets its own HNSW index, while semantic
search runs across all sources at once — a query with no predicate on the
partition key is forced to scan every partition's index and merge, and the ACL
shortfall multiplies across partitions (partition pruning won't kick in until
the query is narrowed to a source — a rare case). The only gain from partitions
is cheap source deletion (DROP PARTITION), but that is already
covered by ON DELETE CASCADE. So it is one global
store, not sharded by source. Dedicated sharding
(v2) only on proven volume and a
pattern shift toward source-scoped search: an addition, not a rewrite.
| id | BigInteger | PK | — |
| entity_id | BigInteger | FK→entitiesIDX | parent · CASCADE · carrier of the ACL |
| is_deleted | Boolean | NOT NULLDEFAULT | mirror of entities.is_deleted · condition of the partial HNSW/GIN |
| ordinal | Integer | NOT NULL | the chunk's position within the entity |
| text | Text | NOT NULL | chunk text |
| embedding | halfvec(N) | pgvectorHNSW | N — the dimensionality of the assigned embedding model (built-in ones are 1024) · half-precision · ANN by proximity (cosine) |
| text_tsv | tsvector | GENERATEDGIN | sparse index from text · lexical search (ts_rank) |
| token_count | Integer | NULL | chunk length in tokens |
| content_hash | Text | NULL | re-embed only when the text changes |
| embedding_model | Text | NULLIDX | what it was embedded with · refresh key |
| created_at | DateTime(tz) | DEFAULT | now() |
| updated_at | DateTime(tz) | DEFAULT | now() + trigger · updated on re-embedding |
hnsw.iterative_scan, pgvector 0.8.0+) — but only
for conditions on chunks itself; an iterative scan can't
compensate for a predicate via a JOIN to
entities. And we have two such predicates —
is_deleted and the ACL itself — and they are solved
separately. is_deleted is denormalized: the axis is
mirrored onto chunks, and both text indexes (HNSW and GIN)
are built partial — WHERE NOT is_deleted,
so the deleted are outside the traversal entirely. The ACL can't be
removed this way — it stays the hardest filter, arriving via a JOIN; its
recall under our permissions must be measured on real data, not trusted
to general estimates. See
ACL pre-filter.
N of the
halfvec(N) column and the search metric are inherited from
it — Knowledge Store doesn't pick them, it consumes them
(N = meta.embedding_dim of the assigned
model). There is no default: the platform ships built-in models out of
the box (Platform), but N is fixed on the first assignment —
built-in or your own — before the first ingestion. The metric is
cosine (embeddings are normalized, the
vector_cosine_ops operator class on HNSW). One
global choice also sets the boundary between updates: switching to
a model of the same dimensionality is a
refresh
(regenerating vectors in place); changing the dimensionality
rebuilds the column type and index — that is a schema migration, not a
refresh. In v1 the column is provisioned as
halfvec(1024) (the dimensionality of both built-in
models), and assigning a model of a different — or undeclared —
dimensionality is rejected at the registry level (409); a dynamic
N will arrive together with a dimensionality change as
a schema operation.
halfvec by default: half the storage and index
halfvec (half-precision, 2 bytes per dimension vs.
4 for vector; pgvector 0.7.0+) halves both — the
index sits in memory better and search is faster. The recall loss from
half-precision is practically nil, but measure the absolute value on
your own data.
N is set by the model — and when
choosing it, prefer an MRL-compatible one (Matryoshka Representation
Learning: earlier dimensions carry more meaning, so truncating the tail
barely drops recall). That preserves the
v2 option — indexing on the truncated
vector (subvector) while storing the full one: a smaller
index without re-embedding. Truncation is valid
only for MRL models — you can't cut the tail of an
ordinary model, it breaks the space.
halfvec is the RAM for
HNSW; the v2 escalation path:
binary quantization (binary_quantize →
bit + Hamming distance) as a coarse first pass,
then rerank the top candidates on halfvec. Binary is
1 bit per dimension vs. 16 for halfvec: ~16× on the
vector data volume relative to our default
(the full 32× is only against fp32 vector, which we
don't keep), and that on the data, not on the HNSW index size 1:1;
halfvec stays for rerank. An addition of a second
index, not a rewrite — pointless in v1.
simple configuration
text_tsv is generated with the
simple configuration — tokenization and lower-casing
without stemming or stop words, language-agnostic. Two
reasons. The content is mixed (RU + EN), and
GENERATED … STORED requires an
immutable expression — a per-row language in a
generated column is impossible. And the role: in the hybrid, the lexical
primitive carries exact matches — names, IDs,
acronyms, code — while meaning and cross-language come from the
vector primitive, so losing stemming for lexical is not critical.
Per-language stemming (detection + per-language configuration) is
deferred curation, not a schema rewrite.
entity_edge
The entity graph lives in the same Postgres: a node is the
entities record itself (there is no separate node table), and
relations are edges between entities. Traversal is recursive SQL (CTE) over
near context (1–3 hops), under the same ACL
pre-filter as the other projections. A dedicated graph engine is
v2, if it's ever needed.
| id | BigInteger | PK | — |
| src_entity_id | BigInteger | FK→entitiesIDX | source entity · CASCADE |
| dst_entity_id | BigInteger | FK→entitiesIDX | target entity · CASCADE |
| rel_type | Text | NOT NULLIDX | relation type · mentions · replies_to · links_to · child_of · duplicate_of |
| weight | Real | NULL | edge weight / confidence |
| origin | Text | NOT NULLCHECK | harvester (structural, at load) · curation (materialized by a run: cross-source and deferred forward-references) |
| created_at | DateTime(tz) | DEFAULT | now() |
| updated_at | DateTime(tz) | DEFAULT | now() + trigger · weight updated during Curation Pass |
origin distinguishes who drew the edge. Structural ones
(harvester) come with the load — relations within a
source that are already visible in the record (a thread, a page
hierarchy). Cross-source ones (curation) are materialized by
Curation Pass: a mention becomes an edge once both nodes are in the graph.
Since nodes are entities, traversal and its ACL are
an ordinary JOIN of entity_acl on visited ids: the single
pre-filter covers the graph too, without a separate permissions
projection. Traversal starts from a node and follows the relation type,
so edges carry a composite index (src_entity_id, rel_type),
not just single-column FKs — the recursive CTE takes the next hop over
it. Traversal is bounded not only by depth (1–3 hops) but also by
width: a cap on fanout from a node, pruning by
weight, an explicit visited-set against cycles — otherwise a
hub node (a popular author, a space's root page)
fans traversal out into tens of thousands of edges before the ACL even applies.
rel_type is the graph's canonical vocabulary; relation
requests (entity_ref)
are translated into it at materialization:
parent → child_of,
duplicate → duplicate_of,
mentions → mentions, … An edge is born once
both nodes are in the database — otherwise the request
waits in entity_ref and materializes later, on par with
cross-source: the FKs on both ends don't tolerate a dangling target,
including for a forward-reference within a single source. A request whose
target is not an entity (author → user) resolves into the
relational body (author_principal_id), not into an
edge: entity_edge links entities only.
A reference whose target can't be linked yet (the node isn't loaded or lives in another source). Harvester writes the request on capture, Curation Pass resolves the target and clears the row, materializing the edge.
| id | BigInteger | PK | — |
| src_entity_id | BigInteger | FK→entitiesIDX | the entity that owns the request · CASCADE |
| relation | Text | NOT NULL | relation type in source terms · mentions · blocks · parent · author · duplicate · … |
| target_kind | Text | NOT NULL | target kind · issue · page · user · commit · mr · message · … |
| target_ref | Text | NOT NULL | the target's natural id in the source · resolution key |
| source_hint | Text | NULL | presumed source · jira · gitlab · NULL if unknown |
| created_at | DateTime(tz) | DEFAULT | now() |
| updated_at | DateTime(tz) | DEFAULT | now() + trigger · on repeat capture |
dead_letters: a row lives while the
request can't be linked. Once resolution finds the target, Curation Pass
writes the edge into entity_edge and deletes
the row; provenance stays on the edge
(origin=curation) — no point duplicating it in the request.
A repeat capture of the same reference doesn't breed a duplicate — an
upsert on the natural key
(src_entity_id + relation + target_kind + target_ref).
The contract between the modules is already fixed: Harvester produces the Entity and sketches the three projections, Knowledge Store gives them their exact shape and stores them. The same boundary on the producer's side is in its cross-source.
Normalizes a record into an Entity, computes chunks and their
embeddings, draws structural relations within a source, and
captures permissions. Upserts on the idempotency key — exact-match
deduplication at load.
Holds the three projections and permissions as the source of truth, serves ACL-filtered search, and curates the graph across all sources (Curation Pass): cross-source edges and fuzzy merging — cross-source inference that Harvester can't make from a single source.
The boundary: “exact within a source → Harvester, inference across sources → Knowledge Store.” Applying permissions at retrieval — → Query Engine.