Skip to content

02 โ€” Evaluation, LLM-as-Judge & RAG (the core of this role)

This is the heart of the job (JD responsibility #2: "AI Evaluation & Quality Pipelines"). Expect to be questioned on it in depth. Taught from zero.

Part A โ€” Why evaluating AI is different from normal QA

Normal QA: you put in X and expect exactly Y. You check they're equal.

AI/LLM QA: you put in X and get back something Y-ish. The same correct answer can be worded a thousand different ways. So: - assert response == "expected" is useless โ€” any reword fails it. - Instead you ask how good the output is along a few quality dimensions, and judge it statistically. (In plain words: you grade it like an essay, not mark it right/wrong like a math quiz.)

The four families of techniques, cheapest โ†’ most powerful:

  1. Deterministic heuristics (run first, cheap): simple yes/no checks. Is it valid JSON? Is the length sensible? Does it contain a required keyword? Does it avoid banned words or personal info (PII)? Fast, real, and they catch a lot.
  2. Reference-based overlap metrics (need a known-correct "gold" answer): BLEU, ROUGE (these count how many words/phrases overlap with the gold answer), Exact Match. Reliable but dumb โ€” they don't understand meaning, so "The cat sat" vs "The feline sat" scores poorly even though they mean the same thing.
  3. Semantic similarity (uses embeddings, see below): BERTScore, cosine similarity. These spot meaning no matter the wording. Set a tolerance band (e.g., raise an alert if similarity < 0.85).
  4. LLM-as-a-Judge (most flexible): ask a strong LLM to score the output against a rubric for things the simple metrics can't see โ€” helpfulness, coherence, faithfulness, safety. This is the centerpiece of the role.

Embeddings, quickly: an embedding turns a piece of text into a list of numbers (a vector) so that similar meanings land close together. Cosine similarity (โˆ’1 to 1, usually 0โ€“1 for text) measures how close two vectors are โ€” closer = more alike in meaning. The same trick powers RAG retrieval and contamination checks.

Remember: word-overlap metrics are cheap but literal; embeddings catch meaning; an LLM judge catches quality.

Part B โ€” LLM-as-a-Judge

What it is

Use a capable LLM (the "judge" or "autorater") as a cheap, scalable stand-in for a human grader that can also explain its scores. The foundational study (Zheng et al., 2023, MT-Bench) showed strong judges like GPT-4 agree with humans >80% of the time โ€” about as often as two humans agree with each other.

The three scoring modes (know all three + when to use each)

Mode What the judge does Output Best for
Pointwise (single-answer grading) Scores ONE response against a rubric A score, e.g. 1โ€“5, + explanation Absolute quality of one system; scalable; objective tasks (factuality, toxicity)
Pairwise (A/B preference) Sees TWO responses, picks the better "A better / B better / tie" โ†’ win-rates Comparing two prompts/models ("is the new version better?"); subjective tasks (tone, style)
Reference-based Also given a gold answer to grade against Score anchored to the ground truth Objective domains (math, QA) where a single correct answer exists

Rule of thumb: pairwise for subjective quality, pointwise/direct for objective dimensions, reference-based when you have a gold answer.

The judge's biases (THE classic interview question โ€” know all three + the fix)

A judge is an LLM, so it has predictable blind spots:

Bias What happens Real evidence Mitigation
Position / order bias Judge favors whichever answer sits in slot A (or B), no matter its quality One model's win rate jumped 2.5% โ†’ 82.5% just by moving its answer to second place Swap-and-average: run it both ways (A first, then B first); only call a winner if it wins both, otherwise tie
Verbosity bias Judge prefers the longer answer even if the extra words add nothing A "padded list" trick fooled weaker judges ~91% of the time Normalize for length; prefer direct scoring; tell the judge to ignore length
Self-enhancement / self-preference Judge favors answers from its own model family GPT-4 preferred its own outputs ~10% more; some studies far higher Use a different model as the judge than the one you're testing; or use an ensemble of varied judges

Fixes that help across all three: chain-of-thought (make the judge explain before giving a score), few-shot calibration examples (show it a few pre-graded samples in the prompt), reference-guided grading, temperature=0 (makes the judge less random), and multi-sampling (judge several times, then combine the scores).

Calibrating the judge against a golden dataset (JD: "Calibrate LLM-as-Judge rubrics using human-labeled golden datasets")

A judge is only trustworthy if it agrees with humans. The calibration loop (think of it as training and checking your grader):

  1. Build a human-labeled golden set โ€” have 2โ€“3 people label each item. First confirm the humans agree with each other (inter-annotator agreement). If they don't, your rubric is too vague โ€” fix the rubric before blaming the judge.
  2. Run the LLM judge over the same items.
  3. Measure how often judge and human agree (metrics below).
  4. Feed the disagreements back as few-shot correction examples and sharpen the rubric.
  5. Re-check now and then โ€” judges drift as the underlying models get updated (recalibrate roughly every quarter).

Agreement metrics (know which to use when):

Metric Use when
Percent / raw agreement Quick check; does NOT subtract out lucky guesses (so it overstates)
Cohen's kappa (ฮบ) 2 raters, category labels; corrects for chance agreement โ€” the default to cite
Krippendorff's alpha (ฮฑ) 3+ raters, missing data, or ordered/numeric scales
Weighted kappa Ordered labels where "off by one" should hurt less than "off by three"
Spearman / Pearson correlation Continuous scores (e.g., 1โ€“10 ratings)

Remember: 2 raters + categories โ†’ Cohen's kappa. More raters or numbers โ†’ Krippendorff's alpha.

The Landis & Koch kappa scale (memorize):

ฮบ Agreement
0.00โ€“0.20 Slight
0.21โ€“0.40 Fair
0.41โ€“0.60 Moderate
0.61โ€“0.80 Substantial โ† common minimum bar for a judge
0.81โ€“1.00 Almost perfect

Two humans usually agree at about ฮบ โ‰ˆ 0.80 โ€” so that's your realistic ceiling, not 1.0. A judge that hits ฮบ โ‰ฅ 0.6 against humans is generally "good enough to ship, but keep watching it." (The practice repo computes Cohen's kappa for you โ€” run it.)

Part C โ€” RAG, explained from zero

What RAG is

Retrieval-Augmented Generation = before the LLM answers, hand it fresh, relevant, trusted information instead of letting it rely only on what it memorized during training.

In plain words: it's an open-book exam. The model looks up the right pages first, then answers from them.

The pipeline: 1. Ingest (done ahead of time): chop your documents into small chunks, turn each chunk into an embedding (vector), and store them in a vector database. 2. Retrieve (when a question comes in): embed the question, then find the chunks whose vectors are most similar (this is semantic search). 3. Augment: paste those chunks into the prompt as context. 4. Generate: tell the model to answer using only that context.

This is how most enterprise chatbots (including Bell's) answer questions about internal or up-to-date info.

Why RAG can fail โ€” and this maps straight to the metrics

A RAG answer can be wrong for two completely different reasons. You must test them separately:

  • Retrieval failed โ€” it pulled the wrong or incomplete chunks. (Measured by Context Precision & Context Recall.)
  • Generation failed โ€” good chunks were pulled, but the model ignored them, misread them, or made something up (hallucinated). (Measured by Faithfulness & Response Relevancy.)

Saying "a RAG bug is either a retrieval bug or a generation bug, and the metrics let me pinpoint which" is exactly the systems thinking they're listening for.

Remember: open-book exam โ€” did the student grab the wrong page (retrieval), or read the right page and still answer wrong (generation)?

Part D โ€” RAGAS (the tool the JD names)

RAGAS ("Retrieval-Augmented Generation Assessment") is the standard open-source Python library for evaluating RAG systems. Its key trick: most of its metrics are LLM-as-Judge based and reference-free (they need no gold answer) โ€” so you can run them on live production traffic.

Version note: current is RAGAS v0.4.x (early 2026). The API changed a lot. Two renames to know so you don't sound out of date: - "Answer Relevancy" โ†’ "Response Relevancy" (class ResponseRelevancy). - Dataset fields: ground_truth(s) โ†’ reference, contexts โ†’ retrieved_contexts. - The standalone context_relevancy metric was removed โ€” use Context Precision instead.

The core four (know the formula for each)

A RAG test case has 4 parts: user_input (the question), retrieved_contexts (the chunks pulled), response (the answer), and optionally reference (the gold answer).

1. Faithfulness โ€” reference-free. "Is the answer backed by the context, or is it making things up?" - Split the answer into small individual claims โ†’ check each one against the retrieved context. - Faithfulness = (claims supported by context) / (total claims) โ€” range 0โ€“1. - This is your hallucination detector. Needs no gold answer โ†’ great for production monitoring.

2. Response Relevancy (formerly Answer Relevancy) โ€” reference-free. "Does the answer actually address the question, or is it evasive / padded / off-topic?" - The judge reads the answer, generates N (default 3) questions from that answer, embeds them, and measures cosine similarity back to the original question. - Relevancy = average cosine_similarity(generated_questions, original_question). - Catch: it measures on-topic-ness, NOT correctness. An answer can be perfectly on-topic and still factually wrong.

3. Context Precision โ€” a reference-free version exists. "Of the chunks we retrieved, are the useful ones ranked at the top?" (i.e., signal-to-noise and ranking quality of retrieval) - Mean Precision@k across the retrieved chunks, weighted so that relevant chunks ranked high score better.

4. Context Recall โ€” needs a reference. "Did retrieval grab all the info needed to answer?" - Split the gold reference answer into claims โ†’ count how many are supported by the retrieved context. - Context Recall = (reference claims found in context) / (total reference claims).

Memory hook: - Faithfulness + Response Relevancy = generation quality (both reference-free โ†’ run them in prod). - Context Precision + Context Recall = retrieval quality (Recall needs a gold reference โ†’ run it offline on a golden set). - One-liner: Precision = "is the good stuff on top?" Recall = "did we get all the good stuff?"

Other RAGAS metrics worth naming

  • Factual Correctness / Answer Accuracy โ€” compares answer claims to a reference (claim-level F1). Needs a gold answer.
  • Noise Sensitivity โ€” does feeding in irrelevant context push the model into wrong claims?
  • Agent/tool metrics: Tool Call Accuracy, Topic Adherence, Agent Goal Accuracy โ€” for testing multi-step agents (JD responsibility #2: "validate multi-step agent workflows, tool usage").
  • Aspect Critic โ€” a simple yes/no on a custom criterion you write in plain English (e.g., "Is this answer harmful?").

Minimal RAGAS code (the still-current evaluate() flow)

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas import EvaluationDataset, evaluate
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.metrics import (Faithfulness, ResponseRelevancy,
                           LLMContextPrecisionWithoutReference, LLMContextRecall)

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
evaluator_emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

dataset = EvaluationDataset.from_list([{
    "user_input": "When was the Eiffel Tower completed?",
    "retrieved_contexts": ["The Eiffel Tower was completed in 1889 for the World's Fair."],
    "response": "The Eiffel Tower was completed in 1889.",
    "reference": "1889",   # only needed for reference-based metrics
}])

result = evaluate(
    dataset=dataset,
    metrics=[Faithfulness(), ResponseRelevancy(),
             LLMContextPrecisionWithoutReference(), LLMContextRecall()],
    llm=evaluator_llm, embeddings=evaluator_emb,
)
print(result)            # aggregate scores
df = result.to_pandas()  # per-row scores to inspect failures

RAGAS limitations (be honest โ€” it scores you points)

  • Non-deterministic โ€” same input can get a different score on each run. Mitigate: temperature=0, pin the model version, multi-sample and average.
  • Imperfect human correlation โ€” moderate at best; use it for relative comparisons and trends, not as absolute truth.
  • Inherits the judge's biases (verbosity, etc.) and costs several LLM calls per row (use HHEM / embedding / non-LLM variants to cut cost).
  • Always keep a human spot-check / calibration set. A 0.0โ€“1.0 score means nothing until you've checked the judge actually agrees with humans.

Part E โ€” Embedding evals into CI/CD as release gates (JD #2)

The pattern (it's regression testing, just adapted to AI): - Golden eval set = your "test suite" (a curated mix of representative + edge + adversarial cases, kept in the repo / GCS). - Metric thresholds = your "assertions" (e.g., faithfulness โ‰ฅ 0.90, context recall โ‰ฅ 0.85). - On every PR / prompt change / model bump โ†’ run the eval suite. - Gate: if a score falls below threshold (or drops too far below the last good run) โ†’ fail the build, block the deploy. - Steady the probabilistic gate so it doesn't flap: temperature=0, multi-sample, pass-rate thresholds over N runs, and tolerance bands instead of exact equality. (The practice repo's tests/ shows exactly this.)

Part F โ€” One worked example (ties it all together)

Follow a single Bell support chatbot through every concept above. This is the fastest way to make the metrics stick โ€” memorize this one story.

The setup: - User question: "How many days do I have to return my Bell phone?" - The real policy (source of truth): "Bell allows returns within 15 days of purchase for a full refund. A restocking fee may apply after 7 days." - The bot's answer: "You can return your Bell phone within 15 days for a full refund. A restocking fee applies after 7 days." - The chunks the system retrieved: โ‘  refund policy (relevant), โ‘ก data-plan info (junk), โ‘ข store hours (junk).

Why normal testing breaks (Part A)

assert answer == "15 days" fails, because "within 15 days," "you have 15 days," and "fifteen days" are all correct but different strings. So you grade how good it is, not whether it's equal.

LLM-as-Judge modes (Part B)

  • Pointwise: "Score 1โ€“5: does this correctly and completely answer using the policy?" โ†’ 5/5.
  • Pairwise: Bot v1 "You have 15 days" vs Bot v2 (our fuller answer) โ†’ judge picks v2 (more complete). Answers "is the new version better?"
  • Reference-based: give the judge the gold answer and grade against it.

Judge biases (Part B)

  • Position bias: show v2 first and the judge may favor it just for being first โ†’ fix: run both orders, swap-and-average.
  • Verbosity bias: a rambling 200-word answer beats our crisp one โ†’ fix: tell it to ignore length.
  • Self-preference: a GPT-4 judge quietly prefers GPT-4 answers โ†’ fix: use a different model as judge.

RAG = open-book exam (Part C)

The bot looked up a page, then answered from it. A wrong answer is either "grabbed the wrong page" (retrieval) or "read the right page wrong" (generation).

The 4 metrics, with numbers (Part D)

1. Faithfulness = supported claims รท total claims (catches hallucination): - "return within 15 days for a full refund" โ†’ policy says 15 โœ… - "restocking fee after 7 days" โ†’ policy says "may apply after 7 days" โœ… - Faithfulness = 2/2 = 1.0 ๐Ÿ‘ - Hallucinating bot says "Return within 30 days, shipping is free" โ†’ both unsupported โ†’ 0/2 = 0.0 (metric catches it).

2. Response Relevancy = does it address the question? (on-topic, not correctness) - Answer gives the return window โ†’ high. If it said "Bell has great unlimited plans!" โ†’ low (off-topic), even though that's true.

3. Context Precision = are relevant chunks ranked at the top? - Refund chunk โ‘  is on top, junk below โ†’ high. If โ‘  were buried under โ‘ก and โ‘ข โ†’ low.

4. Context Recall = did we fetch everything needed? (needs the gold answer) - Gold has 2 facts (15-day window + restocking fee); both are in the retrieved chunk โ†’ 2/2 = 1.0. - If the retriever missed the restocking-fee chunk โ†’ 1/2 = 0.5 (retrieval missed something).

Remember the split: Faithfulness + Relevancy = did it write a good answer? (generation, reference-free โ†’ run in prod). Precision + Recall = did it fetch the right info? (retrieval; Recall needs a gold answer โ†’ offline).

The gate (Part E)

Collect ~50โ€“200 cases like this into a golden set, set bars (faithfulness โ‰ฅ 0.9, recall โ‰ฅ 0.85), and run them on every change. A new prompt that drops faithfulness to 0.7 โ†’ build fails, release blocked.

โ†’ Next: 03 โ€” Red-Teaming & AI Safety