Skip to content

ADR-001: Test framework & language split

Status: Accepted · Date: 2026-08-22

Context

The platform tests an AI app across seven pyramid layers. No single tool or language fits all layers well.

Decision

  • Python + pytest for unit, component, integration, API, contract, security, and the evaluation harness. Rationale: the app is Python, so tests share models/types; pytest's fixtures + parametrization suit data-driven API/eval testing; httpx + Pydantic give typed request/response validation.
  • TypeScript + Playwright Test for UI/E2E/accessibility. Rationale: Playwright is the 2026 standard for cross-browser, auto-waiting, trace-first UI automation; TS matches the web app's language; the Trace Viewer is a first-class debugging + interview artifact.
  • Separate API tests from UI tests because they answer different questions at different cost/stability. API tests are fast, deterministic, and cover logic/contracts; UI tests cover the integrated user journey and are reserved for what only the UI can verify. Pushing everything through Playwright would be slow and flaky.
  • Determinism boundary: framework/plumbing tests are deterministic and run every PR (FakeLLM + hash embeddings). Probabilistic model quality evaluation is isolated in the eval layer, gated with tolerance bands, and run nightly against a real model.

Consequences

  • (+) Right tool per layer; fast PR feedback; stable CI.
  • (+) Two languages showcase both stacks a 2026 SDET is expected to know.
  • (−) Two toolchains to maintain — accepted, and mirrors real orgs.