Skip to content

Phase 1 — Architecture & Technology Decision Review

Project: ai-sdet-platform — an Enterprise AI/LLM SDET Automation & Evaluation Platform Status: Proposal, awaiting approval before Phase 2 implementation Author role: Principal SDET / QA Architect / AI Evaluation Engineer

This is a decision document, not code. It states what we will build, what we will not build, and — importantly — where I am changing your requested stack and why. Nothing in Phases 2–8 gets written until this is approved.

0. TL;DR — What I recommend

Build a realistically scoped monorepo with one working RAG application and a multi-layer test/eval framework around it. The single biggest risk to this project is over-scope: the original brief lists ~15 major capability areas. If we build all of them shallowly, the repo looks like resume-keyword soup and fails the one test that matters — "can another SDET clone this and run it in 10 minutes?"

So my headline recommendation is: build a vertical slice that is 100% real and runnable, then add breadth. Every feature in the demo workflow (§36 of the brief) must actually execute. Features that can't be made real on a laptop are explicitly marked "illustrative" rather than faked.

Key stack decisions (details in §3):

Area Brief asked for My recommendation Why
Vector DB Qdrant or pgvector pgvector (Postgres) One datastore for app data + vectors = simpler Compose, fewer moving parts, still production-credible. Qdrant added as optional profile.
LLM provider OpenAI/Anthropic/Local Provider abstraction + deterministic FakeLLM default CI and clone-and-run must work with zero API keys. Real providers are opt-in.
Embeddings (unspecified) fastembed (local ONNX) default, OpenAI optional No network needed for the core demo.
Python deps uv or Poetry uv 2026 standard, fastest, single lockfile.
API framework FastAPI FastAPI (agreed) —
UI (unspecified) Minimal React + Vite SPA (or HTMX fallback) Just enough real DOM for Playwright to be meaningful. Not a product.
Reporting Allure Allure for pytest + Playwright HTML + a custom unified JSON→HTML dashboard Allure is great but heavy; the unified JSON dashboard is the portfolio centerpiece.
LLM-as-Judge Required Judge runs against FakeLLM in CI (deterministic), real model nightly Otherwise CI is flaky and costs money — the exact anti-pattern the brief warns against.

1. Guiding principles (the non-negotiables)

These come straight from §47 of the brief and drive every later decision:

  1. Clone-and-run in one command, no secrets. docker compose up + make demo must work offline. Real LLM providers are strictly opt-in via .env.
  2. Nothing fake claimed as real. If a component is illustrative (e.g. a metric we can't truly compute without a paid model), it is labeled as such in code and docs.
  3. Determinism where it matters. LLM output is non-deterministic; our tests of the framework must not be. We separate "test the plumbing" (deterministic, every CI run) from "evaluate the model" (probabilistic, nightly, gated with tolerance bands).
  4. Every AI decision has an audit trail. Self-healing events, judge verdicts, and failure-triage classifications are all logged to inspectable JSON artifacts.
  5. The test framework can test the app independently of the eval framework, and vice versa. Clean module boundaries (§43).

2. Final architecture

2.1 System context

flowchart TD
    subgraph Client
        UI[Web UI - React/Vite]
    end
    subgraph Application
        API[FastAPI - REST + Auth]
        RAG[RAG Service]
        MCP[MCP Server]
    end
    subgraph Data
        PG[(Postgres + pgvector)]
        OBJ[Local object store - docs]
    end
    subgraph Providers
        LLM[LLM Provider abstraction<br/>FakeLLM default / OpenAI / Anthropic]
        EMB[Embeddings<br/>fastembed default / OpenAI]
    end

    UI --> API
    API --> RAG
    API -->|JWT auth| API
    RAG --> EMB
    RAG --> PG
    RAG --> LLM
    API --> OBJ
    MCP --> API
    MCP --> TR[(Test & Eval result store - JSON)]

    subgraph Observability
        OTEL[OpenTelemetry Collector]
        LF[Langfuse - optional profile]
    end
    API -.traces.-> OTEL
    RAG -.traces.-> OTEL
    RAG -.-> LF

2.2 RAG data flow

flowchart LR
    D[Document] --> P[Parser] --> C[Chunker] --> M[Metadata enrich] --> E[Embed] --> V[(pgvector)]
    Q[Question] --> QE[Embed query] --> R[Retriever top-K] --> RR[Reranker] --> CB[Context builder] --> G[LLM generate] --> A[Answer + citations + confidence]

2.3 Test pyramid → tool mapping

Layer Tool Runs in
Unit pytest / vitest every PR
Component pytest (RAG modules in isolation) every PR
Integration pytest + testcontainers/Compose every PR
Contract Schemathesis + JSON Schema every PR
API functional/negative/security pytest + httpx + Pydantic every PR
E2E / UI Playwright + TypeScript smoke on PR, full nightly
Accessibility Playwright + axe-core critical pages on PR
AI / LLM evals custom evaluator + LLM-as-Judge deterministic on PR, full nightly
Performance k6 smoke on PR, load nightly
Security (AI + API) pytest fixtures + curated attack corpus every PR

3. Technology stack (with justifications and exclusions)

Application

  • FastAPI + Pydantic v2 + uvicorn — async, typed, OpenAPI for free (feeds contract tests).
  • Postgres 16 + pgvector — app data and vectors in one store. Qdrant offered as an optional Compose profile so we can demonstrate the swappable-store abstraction without requiring two databases for the happy path.
  • React + Vite + TypeScript for the UI — real component tree, data-testid + ARIA-first, enough surface for meaningful Playwright + self-healing + a11y tests. Fallback: server-rendered HTMX if we want to cut Node from the app runtime.
  • LLM provider abstraction with three implementations:
  • FakeLLM (default) — deterministic, templated, keyless. Powers CI + demo.
  • AnthropicProvider / OpenAIProvider — opt-in via env.
  • Embeddings: fastembed (local ONNX, keyless) default; OpenAI optional.

Automation

  • Python 3.12 + uv + pytest + httpx + Pydantic for API/contract/security/eval.
  • TypeScript + Playwright Test for UI/E2E/a11y, POM + fixtures + projects per browser.
  • Ruff + Mypy (Python), ESLint + Prettier + tsc --strict (TS), pre-commit hooks.

Evaluation

  • Custom evaluator package (ai-evaluation/) — retrieval metrics (Precision@K, Recall@K, MRR, Hit Rate) computed deterministically from the golden dataset; generation metrics (faithfulness, relevance, citation accuracy) via LLM-as-Judge with structured output.
  • Golden dataset versioned as JSONL with a schema + a dataset version field.

Infra / CI / Observability

  • Docker Compose (profiles: core, qdrant, observability).
  • GitHub Actions — split workflows per the brief.
  • OpenTelemetry SDK + collector; Langfuse as optional profile (not required to run).

Deliberate exclusions (the brief said to justify these)

  • LangChain / LlamaIndex — excluded from the core RAG path. Hand-rolling a small, readable RAG pipeline is more valuable for an SDET portfolio (you can explain every line) and avoids a fast-moving heavy dependency. May reference them in docs as alternatives.
  • Chaos-mesh / full chaos engineering — reduced to app-level resilience tests (§41): injected timeouts, DB-down, rate-limit, malformed LLM output. True infra chaos is out of scope for a laptop repo; documented as a roadmap item.
  • Kubernetes / cloud deploy — excluded from runtime; "cloud-ready" is demonstrated via clean 12-factor config + containers + stateless services, documented in an ADR. Adding real K8s manifests would be scope with little interview payoff.
  • LangSmith — Langfuse chosen instead (self-hostable, keyless-friendly); LangSmith mentioned as an alternative only.
  • Multiple real LLM providers wired live — abstraction is real and tested with FakeLLM; real providers documented, one (Anthropic) wired as the reference impl.

4. Repository structure (refined from the brief)

ai-sdet-platform/
├── apps/
│   ├── web/                 # React+Vite UI
│   └── api/                 # FastAPI: auth, docs, chat, agent, RAG endpoints
├── rag/                     # ingestion, chunking, embeddings, retrieval, reranking, generation
├── llm/                     # provider abstraction (FakeLLM/Anthropic/OpenAI), prompts
├── mcp/                     # MCP server exposing test/eval tools
├── ai-evaluation/           # datasets, evaluators, judges, metrics, prompts, reports
├── automation/
│   ├── playwright/          # UI/E2E/a11y (TS)
│   ├── api/                 # pytest API/negative/schema
│   ├── contract/            # Schemathesis
│   ├── performance/         # k6
│   ├── security/            # AI + API security corpus & tests
│   └── self-healing/        # engine + safety + history artifacts
├── quality/
│   ├── triage/              # AI failure analyzer
│   ├── flaky/               # flaky detection
│   ├── gates/               # quality-gate config + evaluator
│   └── dashboard/           # unified JSON -> HTML report
├── tests/                   # unit/integration/e2e for the framework itself
├── infra/docker/            # Dockerfiles + docker-compose.yml (+ profiles)
├── docs/                    # architecture (ADRs), strategy, per-feature guides
├── scripts/                 # make targets, seed data, demo runner
├── .github/workflows/
├── .env.example  .gitignore  Makefile  README.md  LICENSE  pyproject.toml

Change from brief: added top-level quality/ (triage, flaky, gates, dashboard) and llm/ (provider abstraction) as first-class modules — they were buried before and are central selling points.

5. Test, evaluation, self-healing, MCP & CI strategies (summary)

  • Test strategy: deterministic plumbing tests everywhere; probabilistic model evals isolated and gated with tolerance bands. Each test carries a stable ID (API-RAG-023) and emits a structured result record → unified dashboard.
  • Evaluation strategy: retrieval metrics are math (deterministic); generation metrics use LLM-as-Judge with structured JSON verdicts + configurable thresholds; in CI the judge runs on FakeLLM so results are reproducible, with a nightly real-model run.
  • Self-healing: locator fails → gather DOM/a11y candidates → score by similarity → confidence bands (≥0.90 auto, 0.70–0.89 heal+warn, <0.70 fail+suggest) → every event logged to artifacts/self-healing/. Never silent.
  • MCP: read-only/safe testing tools (get_failed_tests, get_self_healing_events, get_rag_evaluation, …) over the JSON result store; demonstrated agent-investigates-failure flow.
  • CI/CD: PR pipeline = lint → unit → api → contract → ui-smoke → security → rag-eval (deterministic) → quality gate. Nightly = full UI/API/perf/security/full-RAG-eval.
  • Security: curated local attack fixtures (prompt injection, jailbreak, IDOR, JWT, injection). No destructive actions, no live targets.
  • DEMO_FAILURE_MODE: env flag injects safe, detectable failures (changed selector, bad retrieval, schema mismatch) so the framework can be shown catching them live.

Full detail per area lands as ADRs in docs/architecture/ during Phase 2+.

6. What's missing from the brief that a 2026 SDET should add

  • Evaluation dataset versioning + drift tracking (brief mentions versioning; I'll make it concrete: dataset hash in every eval report, diff between runs).
  • Cost & token budget as a first-class gate — fail if $/query regresses.
  • Reproducibility manifest — every eval report records model id, dataset version, commit SHA, seed.
  • SBOM + secret scanning + dependency pinning (§42) wired into CI via GitHub-native tooling (Dependabot, gitleaks, syft).
  • A "why-this-not-that" ADR set so the repo teaches, not just runs.

7. Proposed build order (unchanged from brief's phases, rescoped)

Phase 2 foundation (Compose + app + API + pgvector + RAG + minimal UI, runs offline) → 3 (Playwright + pytest API + contract + a11y) → 4 (RAG eval + judge + metrics + report) → 5 (self-healing + triage + flaky) → 6 (MCP + agent + AI test-gen) → 7 (security + perf + observability + resilience) → 8 (CI + gates + docs + ADRs). Tests run and pass at the end of each phase before moving on.

8. Decisions I need from you before Phase 2

  1. Vector store: pgvector as primary (my rec) — OK? Or must Qdrant be primary?
  2. UI: React+Vite (my rec) vs HTMX — either works; React gives richer Playwright demos.
  3. Real LLM: Wire Anthropic as the reference real provider (keyless FakeLLM stays default)? Or keep it provider-agnostic with no live wiring?
  4. Scope dial: Full breadth (all 15 areas, some illustrative) vs deep vertical slice first (my rec) then expand? This decides how much is real vs documented-stub. ```