Achilles — AI Foundation: Test cases

Decisions made

Registry in core, readers only read Rows are created by Admin (providers / models) and the runtime (usage); the consumers — Harvester, Query Engine, Agent Engine — only read from the registry, they create no rows. Constraints are exercised through the ORM straight against the test DB: testcontainers PostgreSQL, the real schema from the migration, per-test rollback.

RESTRICT — the source of truth for assignment integrity A model in use can't be deleted or reassigned while it is referenced by model_assignments / chat_models / agent_models. The lock is held by the DB (FK RESTRICT), not just a code check — the test bypasses the service layer and hits the database directly.

Scope boundaries — DB only This is the control layer: constraints, cascades, the upsert aggregate, seed, and reversibility. The provider connectivity check and model discovery are module logic, covered by a separate iteration; not touched here.

Stack and infrastructure

Available done pytest, pytest-asyncio

Test DB testcontainers (PostgreSQL) — the real schema, per-test rollback; constraints are checked against a live database, not mocks

Seed verified by running the migration itself — the Platform provider, built-in models, tool presets; not a hardcoded fixture

Markers @pytest.mark.integration — the whole layer needs a DB; priority (p0 / p1) is orthogonal to type

Integration · P0 Lock on a model in use — RESTRICT

test_model_lifecycle_lock.py

Model in use — deletion / reassignment rejected by the DB

IntegrationP0 → Assignments

Cases

in use by an assignmentmodel in model_assignments → DELETE of ai_models rejected (FK RESTRICT → IntegrityError)

in use by chatmodel in chat_models → deletion / disabling rejected by RESTRICT

in use by agentsmodel in agent_models → rejected by the same RESTRICT

free model deletesreferenced nowhere → DELETE goes through as normal

lock in the DBbypassing the service layer — a direct DELETE against the database is rejected too; not just a code check

Integration · P1 Default, cascades, usage upsert, catalog, seed

test_default_model.py

Exactly one default — partial UNIQUE WHERE is_default

IntegrationP1 → chat_models / agent_models

Cases

two defaults → rejecteda second is_default=true in chat_models rejected by the partial unique

default switch is atomicsetting a new default in one transaction clears the old one

can't be left without a defaultclearing the only is_default without setting a new one → forbidden (the "exactly one" invariant is enforced by the application)

same for agentsthe same three cases on agent_models — the list shape is shared

test_provider_cascade.py

Provider / model: cascade, CHECK, duplicate, key mask

IntegrationP1 → ai_providers / ai_models

Cases

provider cascadeDELETE of ai_providers → its ai_models go with it via ON DELETE CASCADE

local without base_url → rejectedCHECK (kind='cloud' OR base_url IS NOT NULL) — a local without an endpoint is rejected by the DB

duplicate model → rejectedUNIQUE(provider_id, model_id) — the same model_id twice under a provider is rejected

key is write-onlywhat leaves outward (serialization / API) is the mask ••••xxxx, not api_key_enc and not the plaintext key

assignment CHECK — system subsetmodel_assignments.function is CHECKed against the vocabulary {harvester_embedding · query_rag}; INSERT with function='chat' / 'agent_engine' → rejected by the DB (assignment is possible only to system functions)

usage — full function vocabularymodel_usage.function='chat' goes through: usage is recorded across all four functions, including chat / agent_engine; two different vocabularies on one column → Function vocabulary

test_model_usage_upsert.py

Usage: upsert by bucket, SET NULL, cost derivation

IntegrationP1 → model_usage

Cases

upsert into the same bucketa repeat on UNIQUE(model_id, function, bucket_date) → one row, counters (request_count · input_tokens · output_tokens) grow

a different slice → its own rowa different function / bucket_date → a separate aggregate row

model deletion → SET NULLmodel_id is nulled via ON DELETE SET NULL, the usage row survives — cost outlives it

cost derivationcost = input·price_input + output·price_output; any one price NULL → cost NULL

test_tools_registry.py

Tool catalog: UNIQUE, secret, CHECKs

IntegrationP1 → tools

Cases

name is uniqueUNIQUE(name) — a duplicate tool key is rejected

secret is write-onlycredential_enc never leaves outward — only the is_set flag; config (JSONB) is non-secret and returned as is

CHECK sourcesource ∈ (preset · custom · mcp · openapi) — anything else is rejected by the DB

CHECK accessaccess ∈ (read_only · write) — anything else is rejected

test_seed_migration.py

Seeding Platform + built-ins, downgrade reversibility

IntegrationP1 → Migration

Cases

Platform providerthe migration seeds the is_system provider Platform — non-deletable

direct DELETE rejecteddeleting the is_system provider Platform bypassing the service layer — a direct DELETE against the database is rejected by the DB lock, not just a code check

built-ins + intrinsicsembedding models seeded with meta: bge-m3 {embedding_dim:1024, max_input_tokens:8192}, Qwen3-Embedding-0.6B {1024, 32768}

tool presets disabledtwo presets (web_search · fetch_url), both chat_enabled=false and agents_allowed=false

downgrade mirrorsDROP in reverse order; after the rollback the tables are gone

API · P1 Registry — the HTTP contract httpx · ASGI app · → HTTP API → Conformance

test_api_registry.py

CRUD of providers / models / tools and assignments

APIP1 → ai_providers / ai_models

Cases

registry CRUDcreate / edit / delete of a provider, model, tool over HTTP → codes; the response carries the mask ••••xxxx / the is_set flag, not the secret

function assignmentswitching the chat / agent / embedding default via an assignment → 200, the old one cleared atomically

in-use under RESTRICT → 409deleting a provider / model assigned to a function → 409, not 500 (the lock on a model in use)

CHECK surfaced as 422local without base_url, duplicate model_id, a bad tool source/access → 422, not 500

type governs assignment → 422model_type gates it: an embedding model in chat_models/agent_models or on query_rag → 422; a chat model on an embedding function → 422 — assignment across the wrong type is rejected, not 500 → Type governs assignment

unknown {id} → 404GET/PATCH/DELETE of a provider / model / tool by a nonexistent {id} → 404, not 500

CRUD — Owner/Admin onlyMember on POST/PATCH/DELETE of the registry (providers · models · tools · assignments) → 403; anonymous → 401; parametrized over all write routes, not just discovery

test_api_discovery.py

Polling a provider's model catalog · access

APIP1 → provider adapter

Cases

discovery by adaptera catalog request → the list of models from a cloud (OpenAI-compatible /v1/models) or local (Ollama /api/tags) provider; discovery is config, not inference

provider unreachablea bad key / unreachable endpoint → a clean upstream error surfaced outward, not 500

Owner/Admin onlyCRUD and discovery — Owner/Admin → 200; Member → 403; anonymous → 401 (parametrized over the registry routes)

Out of scope Where it's covered nearby

prompt_settings the singleton and its CHECK (id = 1), override / reset / token cap — covered from the Admin side (test_prompt_settings.py); not duplicated here

Structure Test file structure

The whole layer is integration with a DB. Priority (P0–P1) is orthogonal to the directories and is set by markers (pytest -m p0), not by separate folders.

tests/ai_foundation/module directory
- conftest.pytestcontainers PG, provider / model factories
- integration/with a DB, per-test rollback
  - model-lifecycle-lockRESTRICT lock on a model in use
  - default-model · provider-cascadeexactly one default, cascade / CHECK / mask
  - model-usage-upsert · tools-registryusage upsert, tool catalog
  - seed-migrationseed + downgrade reversibility