Testing strategy¶
The determinism boundary (the key idea)¶
An AI system is probabilistic; a good test suite is not. We split work by what kind of question it answers:
- Plumbing tests (unit, component, integration, API, contract, security, UI)
run against a deterministic
FakeLLM+ local hash embeddings. Same input → same output, every PR, no keys, no cost. These verify the system works. - Model-quality evaluation (RAG eval + LLM-as-Judge) is inherently
probabilistic. It's isolated in
ai_evaluation/, gated with tolerance bands, and the deterministic reference metrics (retrieval math, citation, fragments) run every PR while the semantic judge runs with a real model nightly.
This is why CI is fast, free, and stable while still measuring AI quality.
Layer → tool → cadence¶
| Layer | Tool | PR | Nightly |
|---|---|---|---|
| Unit / component / integration | pytest | ✅ | ✅ |
| API (smoke/functional/negative/auth/schema) | pytest + httpx + Pydantic | ✅ | ✅ |
| Contract | Schemathesis (property-based vs OpenAPI) | ✅ | ✅ |
| E2E / UI | Playwright (chromium) | smoke | full (+FF/WebKit) |
| Accessibility | Playwright + axe-core | critical | full |
| RAG evaluation | custom metrics + LLM-as-Judge | deterministic | full |
| Security (API + AI) | pytest + local attack corpus | ✅ | ✅ |
| Performance | k6 | — | load |
Test identity & observability¶
Every test carries a stable id (API-RAG-001, UI-AUTH-002, SEC-AI-004, …) in
its docstring. A pytest hook (conftest.py, RECORD_RESULTS=1) records each
outcome to artifacts/test-results/results.json, which feeds the MCP tools,
failure triage, flaky detection, and the unified dashboard.
Quality gates¶
quality/gates/gates.yaml sets per-layer thresholds; quality.gates.run_gates
aggregates all artifacts and fails the build on any breach. The dashboard
(artifacts/dashboard/quality.html) summarises the run.
What we deliberately did NOT do (and why)¶
- No LangChain in the core RAG path — a small hand-rolled pipeline is more explainable and avoids a heavy fast-moving dependency (ADR-002).
- No Kubernetes runtime — "cloud-ready" is shown via 12-factor config + containers.
- No full chaos engineering — reduced to app-level resilience tests.
- LLM eval is never the only signal — deterministic checks always run alongside.