FMCG-chat-evaluation — Interview Prep¶

One-liner: The same project-agnostic LLM-evaluation framework as bkg-chat-eval, applied to an FMCG (fast-moving consumer goods) analytics chatbot — and the project where the eval suite actually caught real bugs in the dev team's own KPI code.

Note: This shares its architecture with bkg-chat-eval and hoad-ai-automation. Read those for the shared pieces. This file focuses on what's distinctive about the FMCG instance, because that's where you'll have the strongest, most concrete stories.

1. Elevator pitch¶

30-second version:

"FMCG-chat-evaluation tests an AI data-analyst chatbot for a consumer-goods company called SimCo — it answers questions like 'which zone is dragging down target achievement?' by generating SQL over a sales/distribution warehouse. My framework grades each answer and, crucially, fact-checks the numbers by re-running the company's canonical KPI logic against the live Postgres DB. The standout outcome: the eval didn't just grade the chatbot — it surfaced actual bugs in the dev team's KPI implementations, like a metric calling an undefined helper and another using the wrong SQL parameter syntax."

2-minute version adds the FMCG domain detail: SimCo sells brands like CrunchBite, DermaPure, GlowMax, FreshWash across Skin Care, Fabric Care, Home Care, and F&B. The data covers primary/secondary sales, market share, days-of-inventory, target achievement by region/zone, sell-through rate, channel stuffing. The chatbot answers things like "North zone is at 99.9% achievement; Kerala at 73.8% is the drag." Because those are numeric, data-grounded answers over a daily-refreshing DB, exact-match scoring is impossible — so the framework uses semantic + tolerance-aware verification (±5% on raw values, ±1 percentage point on percentages).

2. What's distinctive vs. the sibling projects¶

Aspect	FMCG specifics
Domain	SimCo FMCG sales/distribution — brands, zones, target achievement, sell-through, inventory
Transport	WebSocket-only chat (`websocket-client`) — connects to `ws://.../ws/{session_id}`, sends `{"type":"question"}`, assembles streaming `text`/`tool_call`/`tool_result`/`done` events
Trace fetch	Chatbot emits a `trace_id`; adapter pulls the structured tool trace from Langfuse (`langfuse_trace_loader.py`). If trace fetch fails, the row hard-fails — the verifier needs the tool evidence
Wrong-grain detector	A signature feature (see below)
dev_findings.md	Documents real KPI bugs the eval surfaced — your best concrete story
Golden dataset	20 cases (`TC_001`–`TC_020`) with explicit "unforgivable vs acceptable drift" criteria

3. Tech stack & why (deltas from the shared stack)¶

Shared with siblings: Python 3.11, pytest (+asyncio/timeout), pydantic config, OpenAI/Anthropic/Azure behind LLMJudgeClient (gpt-4.1 default), PostgreSQL via psycopg2, sqlglot, Langfuse, Jinja2 HTML reports, sentence-transformers, black/isort/flake8/mypy/pre-commit + GitHub Actions.

FMCG-specific: - websocket-client — the chatbot's chat channel is WebSocket, not SSE. api/chat_client.py assembles streamed events. - Playwright page objects (pages/login_page.py, pages/chat_page.py) for the UI channel. - pre-commit detect-secrets in addition to the usual hooks.

4. Architecture (same shape — recite this)¶

run_eval.py (pytest, parametrized over golden_dataset.json)
        │
        ▼
api/chat_client.py  ──WebSocket──▶ Chatbot  (assembles answer + trace_id)
        │
        ▼
adapters/api_adapter.py ── pulls tool trace from Langfuse ──▶ normalized payload
        │                                                     (hard-fail if no trace)
        ▼
eval/aggregator.py  →  6 evaluators  →  weighted score + hard gates
        │
        ▼
store/eval_store.py (JSON)  +  regression_store.py (SQLite history)
        │
        ▼
report/report_generator.py → HTML   (+ Langfuse eval trace per case)

Package roles are identical to bkg-chat-eval: adapters/ (how to call the bot), api/ (auth + WebSocket clients), eval/ (evaluators + aggregator + oracle), business_context/ (BKG + loader), config/ (pydantic), store/ (persistence), report/ (HTML), data/ (golden dataset, results, SQLite caches).

5. The two evaluation pipelines¶

Pipeline A — answer evaluation (the main one). Six evaluators, weighted:

Evaluator	Threshold	Method
oracle (BKG-Truth)	0.70	Agentic verification vs live DB; hard-gated; score = supported / (supported + contradicted)
bkg	0.50	Schema/entity grounding via the BKG graph
content_quality	0.60	LLM judge (correctness / completeness / grounding)
on_topic	0.50	LLM judge (relevance + context adherence)
safety	0.80	Regex PII / prompt-injection guard; hard-gated
latency	warn 15s / fail 60s	SLA banding

Pipeline B — SQL evaluation (evaluate_sql.py). A separate path that audits the SQL the agent generated, not the final prose answer: 1. Fetch process_message traces from a local Langfuse. 2. Extract per-"leg" SQL (a complex question generates multiple SQL legs). 3. Deterministically validate each leg with bkg_sql_validator.py (sqlglot parse → check tables/columns/joins/relevance against the BKG, with severity tiers). 4. LLM-judge each leg. 5. Combined score = 0.6 × BKG + 0.4 × judge, aggregated across legs by min-score (weakest leg) — one bad leg drags the score, which is the conservative correct choice.

Talking point: "I evaluate two layers — the final answer and the SQL that produced it. Validating the SQL deterministically first means I only spend an LLM judge call on syntactically valid, schema-consistent queries."

6. Signature feature: the wrong-grain detector¶

When the oracle marks a claim "contradicted," the aggregator checks whether the actual/expected ratio is a clean ½, 2×, ⅓, 3×, ¼, or 4×. If so, it demotes the contradiction — because that pattern almost always means the evaluator's own pandas aggregation computed at the wrong grain (e.g., a pooled mean instead of sum-per-period-then-mean), not a genuine chatbot error.

Why this is interview gold: it shows you understand the difference between mean-of-rates and SUM(numerator)/SUM(denominator) — a classic, real data-engineering trap. If you average daily percentages naively you get a different (wrong) number than aggregating the underlying counts. Being able to explain that distinction is a strong analytics-QA signal.

7. The dev_findings.md story (your single best concrete example)¶

The eval framework surfaced real bugs in the dev team's BKG KPI code: - sell_through_rate called an undefined helper _month_rank → NameError at runtime. - fill_rate used SQLAlchemy-style :placeholder bind syntax against psycopg2, which needs %s → query failure.

These BKG runtime errors propagated into false unverifiable verdicts, dragging aggregate scores (~15% case-level miss in one run). So the framework added a bkg_runtime_errors diagnostic banner so reviewers don't blame the chatbot for the dev team's broken reference code.

Why interviewers love this: it's cross-team root-cause debugging. The QA tool found bugs upstream in code that defines correctness itself, and instead of silently scoring them as failures, you built a diagnostic to attribute blame correctly. That's mature QA thinking — distinguishing "system under test is wrong" from "my oracle is wrong."

8. Other notable engineering (shared with siblings, worth naming)¶

Trusting the agent's own tool log, not its self-report. The agent demonstrably fabricated verification_steps (placeholder args, "Repeated call, no change"). The framework records real_tool_calls inside the loop and flags fabrication by comparing counts.
Diagnostic flags battery: inspection_only, lazy_verification, fabricated_verification_steps, fake_retry_pattern, zero_row_scope_mismatch, bkg_runtime_errors, contract_drift, no_verdicts_produced — each a named failure mode the aggregator detects.
Cache with DB-replay verification: cache key is "timeless," so cached DB fetches are re-run to detect data drift; ~$0.00001/hit.
Two weight profiles (weights vs weights_bare) depending on whether a real expected_answer exists.
Consolidation history: eight legacy evaluators collapsed into two LLM judges (content_quality, on_topic) to cut ~3 LLM calls/case and de-correlate signals; legacy fields kept via pydantic AliasChoices for migration safety (oracle_* → bkg_truth_*).
Determinism for CI: temp 0 + seed + strict schema validation + JudgeParseError (fail loud, not silent 0.5).

9. Interview Q&A¶

Q1. What's different here vs. a normal API/UI automation suite?

The system under test is a non-deterministic LLM over live data. So there's no fixed expected output. I score multiple quality dimensions and independently recompute the ground-truth numbers, with tolerances (±5% raw, ±1pp percentages) instead of exact match.

Q2. The chatbot uses WebSocket — how do you test a streaming WebSocket response?

The client connects to ws://.../ws/{session_id}, sends a question message, then assembles the streamed text/tool_call/tool_result/done events into a complete answer plus latency/streaming metrics. I also capture the trace_id it emits so I can pull the structured tool trace from Langfuse afterward — that tells me which tables and SQL it actually used.

Q3. Why hard-fail a case if the trace can't be fetched?

Because the oracle needs the tool evidence to verify the numbers. Without the trace I can't tell how the bot got its answer, so I can't fact-check it. Scoring it anyway would be guessing — better to fail explicitly and fix the trace pipeline than to emit a false PASS.

Q4. Tell me about a bug you caught with this framework.

(The dev_findings story.) The eval surfaced two broken KPI definitions in the dev team's reference code — one calling an undefined helper, one using SQLAlchemy bind syntax against psycopg2. These were producing false "unverifiable" verdicts and dragging scores ~15%. I added a bkg_runtime_errors diagnostic so we'd attribute those to the reference code, not the chatbot. The QA tool found bugs in the thing that defines correctness.

Q5. What's the wrong-grain detector and why does it matter?

If the oracle says a number is contradicted but the ratio is a clean half or double or third, that's almost always my own aggregation being at the wrong grain — averaging rates instead of summing the underlying counts. So I demote those contradictions. It matters because mean-of-rates ≠ SUM/SUM; naively averaging daily percentages gives a wrong total. The detector stops the eval from falsely blaming the chatbot for my pandas mistake.

Q6. You evaluate the SQL separately from the answer — why both?

The final prose can be right by luck or wrong for subtle reasons. Auditing the generated SQL catches why — wrong table, missing join, wrong filter. I validate SQL deterministically with sqlglot first (cheap, catches structural errors), then LLM-judge only the valid ones, and I take the weakest leg's score because one bad sub-query taints the whole answer.

Q7. How do you keep the eval itself honest — what if the judge or agent lies?

I don't trust the agent's self-reported steps; I record its actual tool calls inside the loop and compare. I have a battery of diagnostic flags for fabricated steps, fake retries, inspection-only behavior. Judges run at temperature 0 with strict JSON-schema validation and raise an error on malformed output instead of silently scoring 0.5. The eval is itself tested.

Q8. How is correctness defined when the data is live?

The expected_answer in the golden dataset is explicitly a stale snapshot hint, not authority. The real ground truth is the BKG's python_function re-executed against the current DB by the oracle. Correctness is semantic with tolerances; the criteria distinguish "unforgivable" errors (wrong entity, wrong direction, order-of-magnitude) from "acceptable drift."

Q9. What does the regression store give you?

A SQLite history of runs with compare_runs and get_trend, plus cost tracking. So I can see if a model change or prompt change regressed scores on specific cases — exactly like a regression test suite catching a reintroduced bug, but for AI quality.

Q10. How did you reduce cost and runtime?

Content-hashed judge cache (keyed on prompt + model + schema + BKG fingerprint), a whole-agent-run cache, deterministic SQL pre-checks that short-circuit before any LLM call, and per-model cost tracking. Cache hits re-run the underlying DB fetch so caching never hides data drift.

Q11. Why collapse eight evaluators into two judges?

The eight were correlated and each cost an LLM call. Merging accuracy/completeness/correctness/hallucination into one content_quality 3-axis judge and relevance/context-adherence into on_topic cut ~3 calls per case and gave cleaner, less redundant signals. I kept the legacy field names via pydantic aliases so historical data and dashboards didn't break.

Q12. How does CI use this?

GitHub Actions runs black/isort/flake8 → mypy → pytest (excluding UI/slow) on push/PR to main, secrets injected as env. Pre-commit mirrors the linters plus detect-secrets. So formatting, types, and the fast eval tests all gate merges.

Q13. What would you improve?

Quantify judge noise with inter-rater agreement across models; expand the 20-case golden set; rotate the committed .env secrets out of the repo into a secret store; and wire the SQL-eval pipeline into CI alongside the answer-eval pipeline.

10. STAR story (memorize)¶

Situation: SimCo's analytics chatbot gave numeric answers over a live FMCG warehouse, and scores were mysteriously low on certain cases. Task: Find out whether the chatbot was wrong or the eval was wrong. Action: I traced the low scores to unverifiable verdicts, then to runtime errors in the BKG reference KPIs — sell_through_rate hit a NameError and fill_rate used the wrong DB parameter syntax. I added a bkg_runtime_errors diagnostic banner to attribute these to the reference code, and a wrong-grain detector to stop false contradictions from aggregation mismatches. Result: Scores stopped unfairly penalizing the chatbot, the dev team fixed their KPI code, and the eval became trustworthy enough to gate releases. It proved the QA tool could find bugs upstream of itself.

11. Honest caveats¶

The committed .env contains real-looking keys/DSN — confirm dev-only before any demo/screen-share.
Golden dataset is small (20 cases) — be ready to say how you'd grow and balance it.
Like the siblings, some docs lag the code; cite the code, not the README.