Skip to content

bkg-chat-eval — Interview Prep

One-liner: A Python framework that automatically tests an LLM-powered analytics chatbot — grading each answer on multiple quality dimensions and fact-checking its numbers against the real database, because you can't assertEquals a non-deterministic AI answer.

1. Elevator pitch

30-second version:

"bkg-chat-eval is a test harness for an AI chatbot. The chatbot answers business questions in natural language by generating SQL and running it against a data warehouse. The problem is its answers are non-deterministic and can be 'fluent but wrong' — hallucinated numbers, wrong tables, wrong filters. A normal assertion suite is useless there. So I built a framework that grades each response on six quality dimensions, and the key piece is an oracle agent that re-executes the dev team's canonical business logic against the live database to verify the bot's actual numbers. Scores are persisted so we can track regressions across runs, and it produces an HTML report."

2-minute version adds: the framework is deliberately project-agnostic — it's driven by a "Business Knowledge Graph" (BKG) JSON file plus a golden dataset and a DB connection string. Swap those three and the same code evaluates a completely different chatbot/domain. In this instance the domain is Nokia 5G macro-tower deployment for T-Mobile (sites, modernization milestones, HSE compliance percentages, e911/NAS readiness). It uses an LLM-as-judge for the subjective dimensions (content quality, on-topic, grounding, safety) and an agentic tool-calling "BKG-truth" oracle for the objective dimension (are the numbers actually right?).

2. The problem it solves

Traditional test automation assumes a deterministic system: same input → same output, so you assert exact equality. An LLM analytics chatbot breaks every one of those assumptions:

  • Non-deterministic phrasing — the same question yields different wording each time.
  • Live data — the warehouse refreshes, so even the "right" number changes day to day. A hardcoded expected value goes stale.
  • Fluent-but-wrong failure mode — the most dangerous bug is a confident, well-formatted answer with a wrong number (wrong table joined, wrong filter, wrong aggregation grain). It passes a human eyeball test.

So the QA problem becomes: how do you define and automate "correct" when there's no single correct string? The answer is multi-dimensional scoring + an independent oracle that recomputes ground truth.

3. Tech stack & why

Layer Tech Why this choice
Language Python 3.11 LLM SDK ecosystem, data tooling, pytest
Test runner pytest (+ pytest-asyncio) Tests are the harness; one parametrized test per golden case
Config pydantic / pydantic-settings One typed Settings object loaded from .env; fail-fast on bad config
LLM judge OpenAI (gpt-4.1 default) + Anthropic + Azure OpenAI Provider-agnostic LLMJudgeClient so we can swap models/providers via one env var
Embeddings sentence-transformers (MiniLM) Semantic retrieval of relevant BKG tables for schema-aware judging
Database PostgreSQL via psycopg2 The live ground-truth warehouse the oracle queries
SQL parsing sqlglot Deterministically validate generated SQL (tables/columns) before judging
Transport requests with SSE streaming Chatbot streams answers via Server-Sent Events
Observability Langfuse Read the chatbot's tool-call traces; write eval scores/traces
Reporting Jinja2 → HTML Self-contained shareable report
UI path Playwright Optional UI-channel evaluation via page objects
Quality gates black, isort, flake8, mypy, pre-commit, GitHub Actions Lint → type-check → test in CI

Talking point: "I treated the LLM provider as a swappable dependency behind one client interface — same reason you'd abstract a payment gateway. Today it's GPT-4.1, tomorrow Claude, with no change to evaluators."

4. Architecture walkthrough

                 ┌─────────────┐
   golden        │ run_eval.py │  CLI entrypoint (--mode api|ui)
   dataset ─────▶│  (pytest)   │
   (30 cases)    └──────┬──────┘
                        │ one test per case
                        ▼
                 ┌──────────────┐     normalized      ┌──────────────────┐
   Chatbot  ◀───▶│   Adapter    │────  payload  ─────▶│ ResultAggregator │
  (SSE/HTTP)     │ api / ui     │  (uniform dict)     │  .evaluate()     │
                 └──────────────┘                     └─────┬────────────┘
                                                            │ runs 6 evaluators
                       ┌────────────────────────────────────┼───────────────────────┐
                       ▼                ▼                     ▼                       ▼
                 ┌───────────┐   ┌──────────────┐    ┌────────────────┐      ┌──────────────┐
                 │ oracle /  │   │content_quality│    │ on_topic / bkg │      │ safety /     │
                 │ bkg_truth │   │ (LLM judge)   │    │  (LLM judge)   │      │ latency      │
                 │ (agent +  │   └──────────────┘    └────────────────┘      └──────────────┘
                 │  live DB) │
                 └─────┬─────┘
                       │ weighted score + gates
                       ▼
        ┌──────────────────────────┐     ┌─────────────────┐
        │ store: eval_results.json  │     │ Langfuse (write │
        │  + regression.db (SQLite) │     │ eval traces)    │
        └─────────────┬─────────────┘     └─────────────────┘
                       ▼
              Jinja2 → HTML report

Package responsibilities:

  • adapters/ — how to talk to the chatbot. api_adapter.py (HTTP + SSE) and ui_adapter.py (Playwright). Both emit the same normalized payload (question, ai_response, tool calls/results, latency, streaming metrics like TTFT and tokens/sec). This decouples evaluation from transport — the single most important design contract.
  • api/ — auth_client.py (login → JWT) and chat_client.py (parses the SSE event grammar: chunk/reasoning/agent_status/tool_call/tool_result/done/error, with retry + backoff).
  • eval/ — the six evaluators + aggregator.py (the brain) + the BKG oracle agent + supporting modules.
  • business_context/ — the BKG JSON files + context_loader.py, which extracts a compact markdown summary injected into judge prompts (passive knowledge).
  • config/ — single pydantic Settings.
  • store/ — eval_store.py (current run JSON), regression_store.py (SQLite history with compare_runs / get_trend / cost tracking).
  • report/ — Jinja2 HTML report.

5. End-to-end flow (say this as a story)

  1. run_eval.py launches pytest; tests/api/test_chat_flow.py is parametrized over the golden dataset (data/golden_dataset.json, 30 cases TC_001–TC_030).
  2. For each case, the adapter sends the question to the chatbot and collects the streamed answer + the structured tool trace (which tables/SQL it actually used).
  3. The adapter returns a normalized payload to ResultAggregator.evaluate().
  4. The aggregator runs six evaluators, computes a weighted overall score, and applies gates (some hard, most soft).
  5. Results are persisted to JSON + SQLite (regression history) and a Langfuse trace is written with per-evaluator scores.
  6. An HTML report is rendered.

6. The evaluation methodology (the heart — know this cold)

Six active evaluators, combined as a weighted sum into overall_score:

Evaluator What it checks How
oracle / bkg_truth Are the numbers factually correct? Agentic tool-calling agent that re-runs canonical logic vs live DB. Score = supported / (supported + contradicted). Hard-gated.
content_quality Correctness, completeness, grounding One consolidated LLM judge on a 3-axis rubric
on_topic Relevance + context adherence LLM judge
bkg Entity/schema grounding 0.4 heuristic + 0.6 schema-aware judge over embedding-retrieved top-K tables
safety PII / prompt-injection Regex guard. Hard-gated.
latency Response-time SLA Banded score

The oracle is the clever part. It's a domain-blind OpenAI tool-calling agent. It doesn't know anything about 5G towers — it learns the project at runtime through tools: list_kpis, call_kpi, call_decision, call_core_node, verify_tool_result, execute_python (sandboxed), execute_sql, and submit_verdict. The KPI/decision tools execute the python_function source code stored in the BKG (bkg_kpi_runner.py exec()s it into a shared namespace) — i.e., it re-runs the dev team's own reference implementation of each metric against the live DB, then compares to what the chatbot claimed. Per-claim verdicts: supported | contradicted | unverifiable.

Two-pass claim extraction (claim_extractor.py): a dedicated LLM pass first enumerates every claim the answer makes into a "contract" (C1, C2, …) before verification. This fixed a real measurement bug where a single-pass agent silently dropped 30–50% of unverifiable claims — which inflated PASS rates exactly where hallucination was most likely.

Gating model is a soft-gate design: only (a) safety, (b) oracle-with-tool-results, and (c) overall_score < pass_threshold (0.60) are hard fails. Every other threshold miss is a warning, not a failure — so one flaky judge can't fail the whole case.

Weight profiles auto-switch: weights when a real expected_answer exists; weights_bare (oracle 0.50 / bkg 0.30 / on_topic 0.10) when expected_answer is a placeholder. In this dataset expected answers are "N/A", so the bare profile (oracle-dominant) is used.

Golden dataset: 30 cases, each with question, context, and evaluation_criteria containing "Unforgivable / Acceptable / Partial credit" rubrics, plus tags.

7. Notable design decisions (interview gold)

  • Project-agnostic by design. No domain words in any prompt — everything comes from the BKG at runtime. Git history literally has a refactor: make eval project-agnostic commit.
  • Two uses of one BKG file: passive context injection into judge prompts vs. active grading by executing its reference functions. Be ready to articulate the difference.
  • Trust boundary on exec(). BKG python_functions are trusted (controlled dev pipeline) and exec()'d directly; LLM-supplied code goes through a sandboxed execute_python with an import allow/block list. Security-conscious tradeoff.
  • Cost engineering: SQLite judge cache (7-day TTL), whole-agent-run cache with optional DB-replay re-verification on cache hits (catches silent data drift), temperature=0 for determinism, per-call USD cost tracking.
  • Streaming UX metrics (TTFT, tokens/sec) — a single latency number hides UX regressions.
  • Determinism for CI: temp 0 + strict jsonschema validation of judge output + raise JudgeParseError instead of silently scoring 0.5, so malformed responses fail loudly.

8. Interview Q&A

Q1. What is this project in one sentence?

An automated evaluation framework that grades an LLM analytics chatbot's answers across six quality dimensions and fact-checks its numbers against the live database using an agentic oracle.

Q2. Why can't you just write normal assertion-based tests?

Because the system under test is non-deterministic and data-backed. The wording changes every run, and the "correct" number itself changes as the warehouse refreshes. The dangerous failure mode is a fluent answer with a wrong number. So instead of assertEquals, I score multiple dimensions and recompute ground truth independently rather than comparing to a frozen expected string.

Q3. What's the "oracle" and why is it trustworthy?

The oracle is a tool-calling agent whose tools execute the dev team's canonical metric definitions — the python_function for each KPI stored in the BKG — against the same live database. So it's not a second opinion from another LLM; it's the organization's agreed-upon business logic, re-run. The chatbot's claimed numbers are compared to that. Score = supported claims / (supported + contradicted).

Q4. How do you handle the fact that the database changes?

Two ways. First, I never assert exact equality against a stale expected value — the expected_answer is treated as a hint, not authority. Second, the oracle recomputes against the current DB at eval time, so it's always comparing apples to apples. And the cache layer re-runs cached DB fetches to detect data drift instead of trusting a stale cached result.

Q5. Walk me through what happens when a single test case runs.

(Recite the 7-step flow in section 5.) CLI → pytest parametrized over the golden case → adapter sends question, gets answer + tool trace → normalized payload → aggregator runs six evaluators → weighted score + gates → persist to JSON/SQLite + Langfuse → HTML report.

Q6. What's the difference between the LLM-as-judge evaluators and the oracle?

The judges (content_quality, on_topic, bkg) score subjective quality — is it relevant, complete, grounded, on-topic — using an LLM with a strict rubric at temperature 0. The oracle scores objective correctness of the actual figures by recomputing them. Judges answer "is this a good answer?"; the oracle answers "are these numbers true?"

Q7. Why a "soft gate" model instead of failing on any threshold miss?

LLM judges are themselves slightly noisy. If every dimension were a hard gate, a single judge's off-day would fail a perfectly good case and erode trust in the suite. So only the things I'm confident about — safety, the oracle when it has tool evidence, and the aggregate score — are hard fails. Everything else surfaces as a warning. It's the same reason you don't fail a build on one flaky low-priority assertion.

Q8. Two-pass claim extraction — what problem did that solve?

Originally one agent both found and verified claims, and it would quietly skip claims it couldn't verify — so the pass rate looked better exactly when the bot was hallucinating unverifiable stuff. I split it: one pass enumerates every claim into a contract first, then the verifier must return a verdict for each. Dropping a claim is now visible as unverifiable, not a silent omission. It measured ~30–50% of claims were being dropped before.

Q9. How do you keep API costs and runtime under control?

A content-hashed SQLite cache for judge calls (7-day TTL), a whole-agent-run cache, structural SQL pre-checks (sqlglot) that short-circuit before spending an LLM call on obviously-broken SQL, and per-call cost tracking so I can see spend per run. Cache hits optionally re-verify against the DB so caching never masks data drift.

Q10. You're exec()-ing code from a JSON file — isn't that dangerous?

It's a deliberate trust boundary. The BKG comes from a controlled internal dev pipeline, so its reference functions are trusted the same way you trust your own repo. Code that the LLM generates at runtime is not trusted — that goes through a sandboxed executor with an import allow/block list. So trusted-source code and model-generated code are treated differently.

Q11. How is this "project-agnostic"? Prove it.

No evaluator or prompt contains domain terms like "tower" or "5G." All domain knowledge is loaded at runtime from the BKG JSON, the golden dataset, and the DB connection. To point it at a new chatbot I swap those three inputs — zero code change. There's an explicit refactor commit that made this true; the same codebase backs the FMCG and HOAD evaluators.

Q12. What metrics or evidence show the framework works?

It catches the failure modes a human reviewer misses — wrong-grain aggregations, dropped claims, fabricated verification steps. It tracks per-dimension scores over time in the regression store so we can see drift. And it produces per-case verdicts with the supporting/contradicting evidence, so a failure is actionable, not just a red X.

Q13. How do you trace and debug a failing evaluation?

Every case writes a Langfuse trace with per-evaluator scores and the oracle's tool calls, so I can see exactly which claim was contradicted and which DB query proved it. The HTML report surfaces the reason string per dimension. And because the oracle records its actual tool calls (not the LLM's self-reported steps), I can tell a real contradiction from a fabricated verification.

Q14. What's the hardest bug you found in the eval framework itself?

The agent fabricating its own verification steps — reporting "re-ran query, no change" without actually calling a tool. I caught it by recording the real tool-call log inside the loop and comparing counts. That's why the framework now trusts its own instrumentation over the model's self-report.

Q15. How does this relate to your QA background?

It's the same discipline with a harder oracle problem. Golden dataset = test cases. Aggregator = assertion engine. Regression store = regression suite. Gates = pass/fail criteria. CI = the same shift-left gate. The only new muscle was reasoning about non-determinism and defining "correct" probabilistically with tolerances instead of exact match.

Q16. If you had more time, what would you improve?

The docs drifted from the code (the README still says "13 evaluators" but we consolidated to 6), so I'd regenerate docs from code. I'd add inter-rater reliability checks across judge models to quantify judge noise, expand the golden dataset, and wire the UI (Playwright) path into CI which currently isn't running against the live backend.

Q17. Why six evaluators and not one big judge?

Separating dimensions de-correlates the signals — a relevant-but-wrong answer should fail on oracle while passing on on-topic, and I want to see that. One blended score hides which axis failed and makes regressions un-diagnosable. It's the same reason you write focused assertions instead of one giant assert.

9. STAR story (memorize one)

Situation: Our LLM analytics chatbot was shipping answers that looked right but sometimes had wrong numbers, and we had no automated way to catch it. Task: Build a regression-safe way to verify chatbot answers without a human reviewing every one. Action: I built an eval framework with an agentic oracle that re-runs the canonical KPI logic against the live DB, plus LLM judges for subjective quality, two-pass claim extraction to stop silent claim-dropping, and a regression store to track scores over time. Made it project-agnostic via a BKG file so it could be reused across products. Result: We could grade every release automatically, caught fluent-but-wrong answers (and even surfaced bugs in the dev team's own KPI definitions), and turned "vibes-based" AI QA into a gated, trend-tracked pipeline.

10. Honest caveats (don't get caught off guard)

  • The README/docs/ARCHITECTURE_OVERVIEW.md are partly stale (claim "13 evaluators," reference FMCG/HOAD lineage). Code consolidated to 6 — present the code as ground truth.
  • A committed .env may contain real-looking keys/DSN — confirm they're local/dev-only before any live demo or screen-share.
  • The Playwright UI path exists but isn't wired into CI against the live backend.