Practice Repo — LLM Evaluation & Quality Gates¶

A small, runnable project that demonstrates the exact skills the BCE "AI Testing" role asks for. It runs offline out of the box (a deterministic mock LLM), so you can pytest with zero setup — then flip a flag to run against real Gemini or OpenAI.

The point isn't the code volume — it's that you can open each file in the interview and explain it. That's what makes the concepts stick.

What it shows¶

File	Concept it teaches
`llm_client.py`	Provider-agnostic LLM client (mock / Gemini / OpenAI) + JSON output validation with retry (a guaranteed coding-round question)
`rag_metrics.py`	Faithfulness, Response Relevancy, Context Precision, Context Recall implemented from scratch so you understand the formulas
`judge.py`	LLM-as-Judge: pointwise rubric scoring + pairwise with position-swap to beat order bias
`calibration.py`	Cohen's kappa to validate a judge against human labels
`eval_pipeline.py`	End-to-end: load golden set → score → aggregate (the "regression suite for AI")
`tests/test_eval_gates.py`	pytest release gates: thresholds + pass-rate over N runs + tolerance — the CI/CD pattern
`tests/test_metrics.py`	Unit tests proving the metrics behave correctly
`redteam_demo.py`	A tiny prompt-injection test harness (the red-teaming idea in code)
`agent.py`	A tiny tool-using agent (plan → call tool → observe → repeat) with guardrails: loop guard, budget cap, and blocking unauthorized mutating actions
`tests/test_agent.py`	How you test an agent: tool-call accuracy, trajectory order, sandboxed side-effects, blocked unauthorized actions, loop/budget guards
`ragas_example.py`	The real RAGAS + Vertex/Gemini API (commented; needs keys)
`coding_drills.py`	Coding-round solutions (AI primitives + must-know-8 Python + data processing) for ../09-coding-questions.md
`tests/test_coding_drills.py`	Verifies every coding-drill solution (practice by deleting a body and re-implementing)
`data/golden_set.json`	A small labeled RAG golden dataset (good + hallucinated + off-topic cases)

Quick start (offline, no API key)¶

cd practice-repo
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt          # only pytest needed for offline mode
python eval_pipeline.py                   # run the full eval, prints a report
pytest -v                                 # run the release gates

Everything passes/works offline because llm_client.py defaults to a deterministic mock backend.

Run against a real model (optional)¶

export LLM_BACKEND=gemini   # or: openai
export GOOGLE_API_KEY=...    # for gemini   (or OPENAI_API_KEY for openai)
pip install google-genai     # or: pip install openai
python eval_pipeline.py

How to study with it (do this)¶

Run python eval_pipeline.py and read the per-case report. Note which cases fail which metric and why.
Open data/golden_set.json — see the deliberately hallucinated and off-topic answers. Predict their scores before running.
Run pytest -v. Then break a gate on purpose: lower a threshold in tests/test_eval_gates.py or corrupt a golden answer, and watch it fail. This is the CI gate in action.
Read rag_metrics.py top to bottom — each metric has a docstring with its formula. Be able to recite faithfulness from memory.
Read calibration.py and run it — understand why kappa ≥ 0.6 matters.

⚠️ The mock metrics use simple lexical heuristics so they run offline and deterministically. Real RAGAS/Vertex use an LLM judge + embeddings (see ragas_example.py). The structure, formulas, and interfaces here mirror the real thing — that's what's being taught.