Skip to content

Metric definitions

Every quality metric used by the gates has a precise definition here. "A metric you can't define, you can't defend."

Retrieval metrics (deterministic — computed from ranked results vs a relevant set)

Metric Definition
Hit Rate 1 if any relevant document appears in the retrieved list, else 0 (averaged over queries).
Precision@K (# relevant in top-K) / K. "Of what I retrieved, how much was relevant?"
Recall@K (# relevant in top-K) / (total relevant). "Of what was relevant, how much did I retrieve?"
MRR Mean Reciprocal Rank = mean of 1/(rank of first relevant hit). Rewards ranking the right doc first.

Generation / answer metrics

Metric Definition How we measure it
Correctness Does the answer contain the expected fact? Reference-based: expected fragments present (deterministic).
Faithfulness / Groundedness Is every claim supported by the retrieved context (no hallucination)? LLM-as-Judge dimension (0–1).
Answer Relevance Does the answer actually address the question? LLM-as-Judge dimension (0–1).
Citation Accuracy Are the cited sources correct (⊆ expected) and present? Deterministic: cited vs expected sources.
Hallucination Rate Fraction of non-abstaining answers whose faithfulness < 0.5. Derived from judge faithfulness.
Pass Rate Fraction of golden examples meeting all their criteria. Deterministic + judge combined.

Abstention (refusing an out-of-scope/injection question) is the correct answer for those categories and is never counted as a hallucination.

Cost / performance metrics

Metric Definition
Tokens/request prompt + completion tokens per query (proxy for cost).
Cost/request estimated USD per query (0 with the local FakeLLM).
Latency (p50/p95/p99) request-duration percentiles under load (k6).
Time-to-first-token latency to first streamed token (real-provider config).

Test-health metrics

Metric Definition
Flip rate fraction of run-to-run pass↔fail transitions in a test's history.
Retry passes times a test failed then passed on retry (a flakiness signal, never hidden).
Self-healing confidence similarity of a healed locator to the recorded fingerprint (0–1).

Reproducibility

Every evaluation report records: dataset version + hash, provider + model, and the judge used — so a score is always traceable to exactly what produced it.