Skip to content

ADR-003: LLM provider abstraction & evaluation determinism

Status: Accepted (abstraction implemented in Phase 2; evaluation harness in Phase 4) Date: 2026-08-22

Context

LLM output is non-deterministic and calling real models costs money and requires keys. Yet CI must be reproducible, free, and runnable by anyone who clones the repo. We also must evaluate real model quality — the whole point of an AI QA platform.

Decision

  • Provider abstraction (llm/provider.py): the app depends on an LLMProvider interface, never a vendor SDK. Implementations: FakeLLM (default), AnthropicProvider (reference real provider, opt-in), room for others. Every response carries tokens/cost/latency so eval + observability measure uniformly across providers.
  • FakeLLM is deterministic and keyless. It answers from the retrieved context by selecting the most relevant sentence and emitting a real [source: N] citation, so the RAG plumbing (retrieval, citation parsing, API contract, agent) is genuinely exercised without a network call.
  • Two-track evaluation:
  • Deterministic track (every PR): retrieval metrics (Precision@K, Recall@K, MRR, Hit Rate) computed as math against the golden dataset — no model needed. Plus judge harness smoke-tested against FakeLLM.
  • Probabilistic track (nightly / opt-in): LLM-as-Judge with a real model, structured JSON verdicts, thresholds with tolerance bands, reproducibility manifest (model id, dataset version, commit SHA).

Consequences

  • (+) CI is free, fast, reproducible; clone-and-run needs no keys.
  • (+) Real-model quality is still measured, just isolated where non-determinism belongs.
  • (+) Directly satisfies the brief's rules: don't treat LLM output as deterministic; don't make LLM eval the only form of testing; don't let CI be flaky/costly.
  • (−) FakeLLM answers are lexical, not reasoned — so it validates plumbing, not model quality. That separation is intentional and documented.