Skip to content

Practice Repo — runnable, offline

A tiny fake agent platform so the concepts in the notes become physical. No API keys, no network — it runs with only pytest.

Run it

cd practice-repo
python -m pip install pytest          # that's all you need
python -m pytest -v

You should see 50 tests pass across five test files (agent graph, trace/non-determinism, audit/eval, safety/red-team, and the action harness).

What each file demonstrates

File Maps to notes Shows
agent.py 08, 09, 12 Fake agent graph (classify→policy→approval→execute→audit), mockable tools, a Langfuse-shaped Trace, and a hash-chained AuditLog.
evals.py 04, 07, 10, 11 cosine, safe_json, semantic assertion, Ragas-style faithfulness/answer_relevancy/context_recall, and a regression harness with baseline comparison.
conftest.py 01 Fixtures: scoped audit, a factory make_payment with cleanup.
tests/test_agent_graph.py 08 Assert path/order, mock tools to force branches, the approval gate, exact tool args, boundary parametrize.
tests/test_trace_and_nondeterminism.py 07, 09 Trace path/score/cost assertions, structural invariants, semantic (not exact) matching, messy-JSON parsing.
tests/test_audit_and_eval.py 10, 11, 12 Audit chain intact / append-only / tamper detected, Ragas metrics catch hallucination, regression gate passes and trips.
safety.py 16 A guarded fake chatbot (secure vs vulnerable=True) + graders: refusal check, PII/secret regex, canary-token leak, identity denylist, Attack Success Rate.
tests/test_safety_redteam.py 16 Red-team: harms refusal + ASR gate + over-refusal, system-prompt/secret leakage (canary), jailbreak incl. indirect injection via poisoned tool output, identity ("I'm ChatGPT") checks — plus tests proving the detectors catch a real leak.
action_harness.py 17 The action containment harness for real-world actions: least-privilege allowlist, human gate for irreversible actions, dry-run, spend cap, idempotency, kill switch.
tests/test_action_harness.py 17 One test per control (OWASP LLM06): disallowed tool blocked, irreversible→approval, retry→no double-execute, over-budget→blocked, dry-run→no side effect, kill switch→halts.

Learn by breaking things (do this before the interview)

  1. In agent.py, move the check_policy span to after release_payment → test_policy_checked_before_release fails. That's why order matters (file 08).
  2. Change HIGH_VALUE_THRESHOLD to 100000 → the approval-gate test fails. That's a missing human gate (file 08/12).
  3. In tests/test_audit_and_eval.py::test_tampering_breaks_the_chain, comment out the tamper line → verify_chain() stays True, showing the hash chain only catches actual edits (file 12).
  4. Make degraded return the correct answer in test_regression_is_detected → the regression gate no longer trips (file 10).
  5. Raise the semantic threshold to 0.99 → paraphrase test fails, showing threshold tuning (file 07).
  6. In safety.py, make the secure bot echo the system prompt (or say "I'm ChatGPT") → the red-team tests in test_safety_redteam.py go red, proving the guardrail + detector work (file 16).

Then extend it (interview talking points)

  • Add a tool_error path: make release_payment raise and assert the agent degrades gracefully (file 08).
  • Add pytest-asyncio and an async def run_agent variant (file 04).
  • Add an HTTPX + Pydantic contract test against a stub endpoint (file 03).
  • Add a --min-score CLI around run_eval and wire the exit code as an Azure DevOps gate (file 06/10).