11 — RAG & Ragas Evaluation¶

JD (nice-to-have): "LLM evaluation frameworks such as Ragas or equivalent for RAG output quality." and "RAG output validation" in the main duties. Know what RAG is, the four core metrics, and which failure each catches.

1. RAG in plain words¶

RAG = Retrieval-Augmented Generation. Instead of relying only on what the model memorised, the system retrieves relevant documents (from a vector DB) and puts them in the prompt as context, so the model answers grounded in your data. In finance: retrieve the customer's policy / account docs, then answer from them.

Pipeline:

question -> [embed] -> [retrieve top-k chunks from vector DB] -> [stuff into prompt] -> [LLM] -> answer

Two failure surfaces to test separately: 1. Retrieval — did we fetch the right documents? (bad retrieval → right model, wrong context → wrong answer) 2. Generation — given the context, did the model answer faithfully and relevantly? (good context but model ignores it or hallucinates)

Interview line: "RAG has two independent failure points — retrieval and generation — so I evaluate them separately. Ragas gives me metrics for each: context precision/recall for retrieval, faithfulness and answer relevancy for generation."

2. The four core Ragas metrics (memorise: what + which failure)¶

Each is scored 0–1, usually computed with an LLM-judge under the hood.

Metric	Plain-English question	Catches
Faithfulness	Is every claim in the answer supported by the retrieved context?	Hallucination — the model made something up not in the docs.
Answer relevancy	Does the answer actually address the question (not padded/off-topic)?	Evasive, incomplete, or rambling answers.
Context precision	Are the retrieved chunks relevant, and are the relevant ones ranked high?	Noisy retrieval — junk chunks / poor ranking.
Context recall	Did retrieval fetch all the info needed to answer (vs a ground-truth answer)?	Missing retrieval — the key doc wasn't fetched.

Memory hook: - Faithfulness + Answer relevancy = judge the generation (answer vs context / answer vs question). - Context precision + Context recall = judge the retrieval (chunks vs question / chunks vs ground truth).

Rough formulas (be able to sketch)¶

Faithfulness = (# claims in answer supported by context) / (# claims in answer).
Answer relevancy = similarity between the question and questions "reverse-generated" from the answer.
Context precision = precision@k weighted by whether relevant chunks are ranked high.
Context recall = (# ground-truth claims attributable to retrieved context) / (# ground-truth claims).

3. Using Ragas (the shape)¶

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

data = Dataset.from_dict({
    "question":     ["What's my overdraft limit?"],
    "answer":       ["Your overdraft limit is £500."],          # model output
    "contexts":     [["Overdraft limit for standard accounts is £500."]],  # retrieved
    "ground_truth": ["The overdraft limit is £500."],           # for recall/precision
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy,
                                 context_precision, context_recall])
print(result)   # {'faithfulness': 1.0, 'answer_relevancy': 0.97, ...}

Gate on it (file 06/10):

assert result["faithfulness"] >= 0.90      # tolerate almost no hallucination in finance
assert result["context_recall"] >= 0.85    # must fetch the needed info

(The practice-repo implements faithfulness & relevancy from scratch so you understand the mechanics without the dependency.)

4. Which metric to prioritise (finance context)¶

Faithfulness is king in a compliance setting — a hallucinated financial figure is worse than a slightly-less-relevant one. Gate it high (≥0.9).
Context recall guards against "we never retrieved the regulation that applies."
Combine with abstention testing (file 07): when context lacks the answer, the system must say "I don't have that", not fabricate — this is a faithfulness + safety concern.

5. Building a RAG test suite¶

Golden Q&A set with question, ideal answer, and the doc(s) that should be retrieved.
Retrieval tests — assert the expected doc is in top-k (context recall/precision).
Generation tests — faithfulness + answer relevancy on the produced answer.
Negative/abstention — question with no supporting doc → must abstain.
Regression — run the whole thing over model/version changes (file 10) and index changes (re-embedding, chunk-size changes).
Gate on aggregate metric thresholds.

Also test the plumbing deterministically: chunking, embedding dimension, top-k, dedup, metadata filters — these are ordinary code and shouldn't need an LLM to verify.

Rapid-fire recall¶

RAG = retrieve docs → ground the answer. Two failure points: retrieval and generation.
Ragas 4: Faithfulness (hallucination), Answer relevancy (on-topic), Context precision (retrieval quality/ranking), Context recall (retrieval completeness).
Generation metrics = faithfulness + answer relevancy; retrieval metrics = context precision + recall.
Finance: gate faithfulness highest; test abstention when context is missing.