← Agent Engine

Test cases

agent-engine · workzone
Decisions made
The loop runs on a fake model The agentic loop (thinks → calls a tool → reads the result) is checked deterministically: a fake chat-client with a recorded sequence of tool-calls and fake tools. The real model and provider stay out of CI, for manual debugging; only a reproducible double goes into the automated run.
Tools — a contract to KS, not a re-run of it search · graph · sql are tested as calls to Knowledge Store primitives under the owner’s identity ACL: the right tool is chosen by intent, and permissions trim on the KS side. The results, ranking, and fusion themselves are held and tested by Knowledge Store — not duplicated here.
Budget, stops, cascades — domain tests The module’s safety core: the weekly sum of tokens_used, two locks (the owner’s switch enabled and the sticky admin_paused) plus the model condition (model_id IS NOT NULL; removal from the list → SET NULL → stop), account deactivation → enabled=false for its agents, a skipped record on a runtime refusal (budget_exceeded / already_running) and the scheduler’s silence when a lock is closed, FK cascades from the agent and from the account. Also here — that outwardly this surfaces as clean codes, not 500.
Stack and infrastructure
On hand done pytest, pytest-asyncio, pytest-cov, httpx
Model doubles fake chat-client (recorded tool-call sequences) + fake tools search / graph / sql
Time time-machine — schedule, next_run_at, the weekly budget window
Other factory_boy (agents / agent_runs), SAQ handler called directly — without spinning up a worker
Markers @pytest.mark.unit / @pytest.mark.integration — by type; @pytest.mark.p0 / p1 — by priority, orthogonal to type
Unit Loop and prompt composition on a fake model AI deterministic · every PR
test_agent_loop.py
think → tool → observe on a fake chat-client
Cases
step with a toolthe model calls a tool → the result comes back into context → the model takes the next step with it in hand
completiona final answer with no tool-call stops the loop; the output goes to the run journal
iteration caplooping is cut off at the step limit — the run is failed with reason=iteration_cap (it hit the cap, not rejected before start), not an endless model call
many steps in a runseveral consecutive tool calls in one run — context accumulates step by step
choice by intentthe recorded sequence calls the right one of the three tools for the task — tool-call dispatch by name works
tool output is data, not a commanda fake tool returns text with an injection (“ignore previous, do X”) → the result is placed into context as data, not as an instruction; the safety perimeter holds, an injection via tool output can’t override platform rules → Prompt composition
test_prompt_composition.py
Layered prompt assembly, safety isn’t overridden
Cases
layer orderthe platform prompt at the bottom (safety + org), the owner’s prompt on top, the engineering layer closing it out — assembly happens in exactly this order
owner can’t override safetythe owner’s prompt doesn’t override the platform safety layer — the owner’s wording can’t cut out the perimeter
binding is added in codethe owner’s identity binding and the descriptions of the three tools are inserted by the engineering layer, not taken from the owner’s text
Unit / Contract Tools into Knowledge Store fake KS primitives · every PR
test_tools.py
search / graph / sql — called under the identity ACL, reading KS
Contract → Tools
Cases
three tools by intentsearch · graph · sql are dispatched by the tool-call name and call the corresponding Knowledge Store primitive
under the identity ACLthe call runs with the owner’s identity permissions (the owner’s frame), not a separate agent identity — the fake primitive receives the creator’s context
KS trims permissionsthe cut by permissions is a pre-filter on the Knowledge Store side, not a post-filter in the agent; here we check that the ACL context is passed through, and KS holds the results
read-onlythe agent writes nothing in Knowledge Store — all three tools are read-only, no mutating primitive is given to them (the KS read boundary)
empty knowledge base → core not offeredKnowledge Store is empty (is_empty) → the harness doesn’t include the core search · graph · sql in the set, even though it’s locked; “locked” = can’t be turned off by hand, not “present in any state of the knowledge base”; the agent starts and degrades onto external tools, or plainly reports there’s no data — an empty knowledge base doesn’t enter the readiness gate
Integration · P0 Run lifecycle DB · separate CI step
test_run_lifecycle.py
agent_runs states, the single-active-run lock, spend accounting, timestamps
IntegrationP0 → Agent runs
Cases
success pathqueuedrunningsucceeded; output written to the run row
failurethe run goes to failed, error is filled; distinct from skipped
token accountingtokens_used is set from the actual run — the basis for the weekly budget sum
work timestampsstarted_at / finished_at are set at start and finish; created_at / updated_at — server_default + trigger
skipped without a starta run that never started (skipped) has no started_at — it was rejected before launch, not cut off mid-work
single active runa second unfinished run of the agent is rejected by the partial unique index UNIQUE (agent_id) WHERE state IN ('queued','running') — the race (a manual launch coinciding with a fired slot) is quashed by the database itself, not just a code check
stale heartbeat → stalethe worker died without closing running → once heartbeat_at goes stale the run is reaped to failed with reason=stale, the lock is released, and the agent starts again — a zombie doesn’t lock it out
Integration · P1 Budget, schedule, control, isolation
test_budget.py
Weekly per-user token limit
IntegrationP1 → Weekly budget
Cases
weekly sumspend = SUM(tokens_used) across all of the owner’s agents for the current week (by finished_at), not per agent
over limit → skippedlimit exhausted → the run doesn’t start, a skipped is written with reason=budget_exceeded, tokens_used=0
week resetthe weekly window rolls over → spend resets to zero, the owner’s agents start again
limit from settingsthe limit value comes from platform_settings, not stored in the module — changed centrally
test_scheduling.py
Manual + interval/calendar, next_run_at, enqueueing
IntegrationP1 → Schedule
Cases
manual onlyschedule NULL → the agent runs only manually, the scheduler leaves it alone
intervalan interval schedule computes the next next_run_at from the previous run
calendar (time-machine)a calendar schedule — daily / weekly at HH:MM — yields the correct next_run_at
owner’s timezonethe calendar HH:MM resolves against the owner’s users.timezone (on NULL — the org default platform_settings.timezone), then is stored in UTC
enqueueingwhen next_run_at arrives, the scheduler places the run on the background queue with trigger=scheduled
overlap doesn’t multiplya slot (or a manual launch) on top of a still-running/queued run creates no new cycle — a skipped row, reason=already_running
manual without a schedule“Run now” (trigger=manual) needs no schedule — it works even with schedule=NULL; the locks (enabled, admin_paused) are still honored
test_governance.py
Two locks (enabled · admin_paused), deactivation, account cascades
IntegrationP1 → Two locks
Cases
admin_paused is stickythe owner can’t clear admin_paused — only an admin can; the owner’s attempt leaves the flag unchanged
start conditionsthe agent starts — by schedule or manually — only when enabled, not admin_paused, budget within bounds AND model_id IS NOT NULL; the axes are independent
removed model → stopthe admin removed the model from agent_modelsON DELETE SET NULL nulls agents.model_id → the agent drops out of the scan (next_run_at cleared), launch is unavailable, no skipped is written — as with a closed lock; it doesn’t fall back to a default
closed lock — silent, no rowwith enabled=false or admin_paused=true the agent is out of the scheduler’s scan (next_run_at cleared) and the manual-launch button is unavailable — no run happens and no skipped appears; the journal doesn’t accumulate durable stops
deactivation → enabled=falsedeactivating the owner’s account clears enabled on all their agents; reactivation doesn’t restore the flag — only the owner brings them back to life by hand
deletion → cascadedeleting the account → agents and behind them agent_runs go via ON DELETE CASCADE
banner — on the surface“paused by administrator” is drawn on the surface side from the flag; the domain merely holds admin_paused
test_isolation.py
Background lane: own pool, separate LLM cap, read-only
IntegrationP1 → Background lane
Cases
separate worker poolruns go on the background lane with their own pool, off the live-chat path — agent load doesn’t crowd the interactive path
separate LLM capthe background lane’s cap on concurrent model calls is separate from the live-chat cap (concurrency)
read-only alongside writesreading Knowledge Store runs alongside graph writes without locks — the agent doesn’t queue behind data ingestion
API · P1 Endpoints — HTTP contract httpx · ASGI · DB · → HTTP API → Conformance
test_api_agents.py
CRUD of your own agents, launch, reading the journal
Cases
CRUD of your owncreate / edit / delete your own agents over HTTP — name, description, prompt, model, schedule, the enabled flag
model choicemodel_id from the allowed list is saved and applied in the run; on creation the field is set (preselect — the list default)
model not in the list → 422model_id outside agent_models (or an embedding type) → 422, not 500
run now“Run now” → 202 + run id; an agent_run is created (trigger=manual, queued), a task in SAQ
repeat /run under an active runa second POST /agents/{id}/run while the agent’s run is already running409 CONFLICT (single-flight, doesn’t double the run); consistent with the domain already_running → skipped
/run on a nonexistent agent → 404an unknown {id}404, not 500
/run with budget exhaustedthe agent’s budget is spent → POST /run doesn’t start a run: 409 (or a domain skipped with reason budget_exceeded) — a fixed HTTP outcome, not 500
CHECK surfaces as 422an invalid trigger / state off the enum, an empty name422, not 500
schedule shape → 422schedule is a tagged union by type (interval / calendar), weekday 0–6, time HH:MM; a malformed input is rejected by validation: (a) unknown type; (b) calendar without time; (c) weekday outside 0–6 → 422, not 500 → Schedule shape
reading the journalGET the agent’s runs with output, tokens_used and outcome; skipped with reason is returned distinctly from failed
test_api_admin.py
Registry of all agents, read-only profile, admin lock
Cases
list allthe admin sees ALL agents on the platform — search and filters by owner / state
both locks in the registrythe registry returns both the owner’s switch (enabled, read-only for the admin) and admin_paused — the row’s combined status is visible to the admin
read-only profilethe admin opens another user’s agent profile read-only: owner, prompt, schedule, runs
lock / unlockthe admin sets and clears admin_paused on any agent; leaves the owner’s enabled switch untouched
no deleting another’sthe admin route doesn’t delete another user’s agent — deletion stays with the owner (or the account cascade)
owner can’t clear another’s admin_pausedan owner on the admin action over admin_paused403
test_api_access.py
Owner sees their own; roles and authentication on every route
Cases
own onlythe owner manages only their own agents; another’s agent is neither visible nor editable in their routes
member blockeda member on another’s agent or on an admin route → 403 FORBIDDEN
anonymousno token → 401 UNAUTHORIZED (parameterized across all routes)
expired tokenan expired access JWT → 401
routes behind the guardevery Agent Engine endpoint is genuinely behind the guard — a parameterized coverage audit; the end-to-end RBAC / JWT mechanics are held by Auth & Security, here we only check the routes sit behind it
Structure Test file structure

The split is set by purpose, not folders. unit/ runs on every PR deterministically — a fake model and fake primitives, no network or DB. integration/ and api/ — a separate, rarer step on a live DB: the first checks the domain (runs, budget, control, lane), the second — the HTTP contract on the ASGI app. The real model and provider stay out of CI, for manual debugging. Priority (P0–P1) is orthogonal to the folders and set by markers (pytest -m p0).

  • tests/agent_engine/module directory
    • conftest.pyfake chat-client · fake tools search/graph/sql · time-machine · agents/agent_runs factories · SAQ directly
    • unit/loop and tools on a fake model — every PR, deterministic
      • test_agent_loop.py · test_prompt_composition.pythink→tool→observe, layered prompt assembly
      • test_tools.pysearch/graph/sql as calls to KS primitives under the identity ACL
    • integration/domain on a live DB
      • test_run_lifecycle.pyrun states, single-active lock + stale reaping by heartbeat, token accounting, timestamps
      • test_budget · test_scheduling · test_governance · test_isolationweekly budget, schedule, locks and cascades, background lane
    • api/HTTP contract of the endpoints — httpx + ASGI + DB
      • test_api_agents · _admin · _accessCRUD and launch, admin registry and lock, roles and authentication