hoad-ai-automation — Interview Prep¶

One-liner: The same LLM-evaluation framework family as bkg/FMCG, applied to a retail-analytics chatbot for House of Anita Dongre (HoAD) — and the instance with the richest multi-channel (API and UI) testing and a hybrid RAG retrieval layer.

Note: Architecture is shared with bkg-chat-eval and FMCG-chat-evaluation. This file emphasizes what's distinctive: the dual API+UI adapter testing, the OTP→WebSocket protocol, and the hybrid retrieval with Reciprocal Rank Fusion.

1. Elevator pitch¶

30-second version:

"hoad-ai-automation is an evaluation framework for an AI retail-analytics assistant built for House of Anita Dongre — an Indian fashion retailer with brands like AND, Global Desi, and Anita Dongre. The chatbot answers questions about live sales, inventory, and replenishment systems by querying SQL databases. My framework tests it through both the API and the real UI using the same evaluation logic, scores each answer on up to thirteen quality dimensions, and uses a hybrid keyword-plus-semantic retrieval layer to ground judgments in the right database tables."

2-minute version adds: "HOAD" = House of Anita Dongre; brands include AND, Global Desi, Anita Dongre, AD Mens, Grassroot. The chatbot covers replenishment systems — Core Replenishment, Core Reorder, Fashion Replenishment, Fashion Consolidation, Zero-Sale. The framework's design contract is a normalized payload: the API adapter and the Playwright UI adapter produce the exact same dict shape, so 100% of the evaluation logic is shared across channels. It also has a second mode (evaluate_sql.py) that audits the agent-generated SQL pulled from Langfuse traces.

2. What's distinctive vs. the sibling projects¶

Aspect	HOAD specifics
Domain	House of Anita Dongre — fashion retail; sales, inventory, replenishment systems
Multi-channel testing	First-class API + UI evaluation; `tests/api` and `tests/ui`, two adapters, one evaluation core
Auth/protocol	OTP login (`request-otp` / `verify-otp`), thread/message REST CRUD, then HTTP-trigger-then-WebSocket-stream
Retrieval	Hybrid retrieval (keyword + embeddings) fused with Reciprocal Rank Fusion (k=60) over a BKG of 39 entities, 74 relationships, ~148 tables across 3 DBs
Evaluators	Up to ~13 dimensions (this instance kept the larger set), default weights documented
Consistency mode	`eval/consistency.py` for multi-run determinism checks
Logging	`structlog` + `colorlog`

3. Tech stack & why (deltas from the shared stack)¶

Shared with siblings: Python 3.10/3.11, pytest (+asyncio/timeout), pydantic config, OpenAI/Anthropic/Azure behind one judge client, Langfuse, Jinja2 HTML, sqlglot SQL validator, SQLite + JSON storage, black/isort/flake8/mypy/pre-commit + GitHub Actions.

HOAD-specific: - Playwright as a first-class channel (Page Object Model in pages/, network interception in adapters/ui_adapter.py) — not an afterthought. - requests + httpx + websocket-client — the chatbot uses an OTP REST handshake then streams tokens over WebSocket. - OpenAI text-embedding-3-small for BKG table retrieval (cached to data/bkg_embeddings.pkl) and sentence-transformers MiniLM (all-MiniLM-L6-v2) for optional semantic accuracy. - structlog + colorlog for structured logs. - Default judge model here is gpt-4o-mini (cheaper), with claude-haiku-4-5 as the Anthropic option.

4. Architecture (the normalized-payload contract is the star)¶

run_eval.py  --mode api | ui | all
        │ (pytest drives tests/api or tests/ui)
        ▼
┌──────────────┐         ┌──────────────┐
│ APIAdapter   │   OR    │  UIAdapter   │   ← network interception (Playwright)
└──────┬───────┘         └──────┬───────┘
       │  SAME normalized payload (identical dict shape)         │
       │  { question, ai_response, context, status, source,     │
       │    latency_ms, ui_latency_ms, ttft_ms, chunk_count,    │
       │    stream_duration_ms, tokens_per_sec }                │
       └───────────────────────┬────────────────────────────────┘
                               ▼
                  eval/aggregator.py :: ResultAggregator.evaluate()
                   ├─ load business context (BKG)
                   ├─ run ~13 evaluators (heuristic and/or LLM-judge)
                   ├─ no_data_detector  (don't fail legit "no records")
                   ├─ weighted overall score + soft/hard gates
                   └─ open Langfuse trace
                               ▼
              store/ (eval_results.json, regression.db, judge_cache.db)
                               ▼
                report/report_generator.py → HTML

The key design point: because both adapters emit the same payload, the entire evaluation pipeline is channel-agnostic. Adding a new transport (say, a mobile API) means writing one adapter — zero new evaluation code. This is the cleanest thing to whiteboard in an interview.

5. Hybrid retrieval with Reciprocal Rank Fusion (your differentiator here)¶

eval/bkg_loader.py parses the BKG into an in-memory BKGGraph singleton with two indexes: - a keyword index (exact matches like the column NetSalesValue), and - a semantic index (MiniLM / OpenAI embeddings, so "revenue" matches NetSalesValue).

For a given response it retrieves candidate tables from both and fuses the two ranked lists with Reciprocal Rank Fusion (RRF, k=60) — score = Σ 1/(k + rank) across lists. The fused top-K tables are then fed to the schema-consistency judge in bkg_evaluator.py.

Why RRF: keyword search nails exact identifiers but misses synonyms; embeddings catch semantics but can rank an exact match below a fuzzy one. RRF combines both without needing to tune score-scale normalization between them. Embeddings are cached by a signature hash, so re-embedding only happens when the schema changes.

Talking point: "This is RAG, but for grounding the evaluator — I retrieve the schema context the judge needs to decide whether the chatbot used the right tables, fusing lexical and semantic search with RRF."

6. Evaluation methodology¶

Golden dataset (data/golden_dataset.json) drives parametrized tests; each case has question / context / expected_answer / evaluation_criteria / tags.
Up to 13 dimensions in [0,1], weighted sum. Default weights: accuracy 0.25, hallucination 0.15 (inverted), context_adherence/relevance 0.15, completeness/latency/safety 0.10; correctness/conciseness/helpfulness/bkg 0.0 (opt-in). Plus the oracle/bkg_truth agent for objective number-checking.
Soft-gate model: only two hard gates fail a test — a safety violation (threshold 0.80) or overall score < pass_threshold (0.60). Everything else is a non-fatal warning in the reason string.
no_data_detector — prevents a legitimate "no records found" answer from being unfairly failed (a real edge case in analytics chat).
eval/consistency.py — when num_eval_runs > 1, checks response similarity, numerical consistency, and latency coefficient-of-variation across repeated runs. This is how you measure non-determinism directly.
LLM-judge prompts explicitly treat expected_answer as a stale hint (DB is live), same philosophy as the siblings.

7. Notable engineering (name these)¶

Normalized-payload adapter pattern — the cleanest design point; API and UI share all evaluation logic.
Hybrid retrieval with RRF — lexical + semantic fusion; embeddings cached by signature hash.
Two uses of the same BKG files — passive context injection into judge prompts vs. active grading in bkg_evaluator.py.
Concurrency in the chat client — double-checked locking for lazy thread creation; a daemon-thread + threading.Event WebSocket reader with a hard timeout and partial-response fallback.
Graceful degradation — every LLM evaluator falls back to a heuristic or a neutral 0.5 when no API key is present, so the suite still runs offline.
Cost engineering — content-hashed judge cache (SHA-256 key, 7-day TTL), per-evaluator cost tracking, structural SQL pre-checks that short-circuit the LLM call.
Multi-format chunk extraction — the WebSocket reader handles message_chunk/full_response plus OpenAI/Anthropic/SSE fallbacks, because the backend's streaming format isn't fully standardized.

8. Interview Q&A¶

Q1. What does this project test and how is it different from your other automation?

It evaluates an LLM retail-analytics chatbot for House of Anita Dongre, through both its API and its real UI, using shared evaluation logic. Unlike deterministic UI/API automation, it scores non-deterministic answers on ~13 quality dimensions and recomputes ground-truth numbers, because there's no fixed expected output.

Q2. You test the same chatbot through API and UI — how do you avoid duplicating evaluation logic?

The normalized-payload contract. Both the API adapter and the Playwright UI adapter return the exact same dict shape — question, answer, context, status, latency, streaming metrics. The aggregator and all evaluators consume that shape, so they don't know or care which channel produced it. A new transport is one new adapter and zero evaluation changes.

Q3. Why test through the UI at all if you have an API?

The UI catches things the API can't: rendering bugs, the streaming experience the user actually sees, auth/session handling in the browser, and end-to-end integration. The API path is faster and used for bulk scoring; the UI path validates the real user journey. Same answer should pass both — and if it doesn't, that's a UI-layer bug.

Q4. Explain the retrieval layer.

To judge whether the chatbot used the right tables, the evaluator needs the relevant slice of a ~148-table schema. I retrieve candidate tables two ways — exact keyword match and embedding-based semantic match — and fuse the ranked lists with Reciprocal Rank Fusion at k=60. Keyword nails exact column names; embeddings catch synonyms like "revenue" → NetSalesValue. RRF combines them without having to normalize score scales. Embeddings are cached by a schema-signature hash.

Q5. What is Reciprocal Rank Fusion and why use it over just averaging scores?

RRF ranks each item by the sum of 1/(k + rank) across the lists it appears in. It uses rank position, not raw scores, so I don't have to reconcile a cosine-similarity scale with a keyword-hit count — which are not comparable. It robustly rewards items that rank high in either method. k=60 is the common default that dampens the influence of low-ranked items.

Q6. How do you handle the WebSocket streaming protocol?

After an OTP REST handshake and thread creation, the chatbot streams tokens over WebSocket. A daemon thread reads chunks into a buffer, signaled by a threading.Event, with a hard timeout and a partial-response fallback so a stalled stream doesn't hang the suite. The reader handles multiple chunk formats — message_chunk/full_response plus OpenAI/Anthropic/SSE shapes — because the backend isn't fully standardized.

Q7. What's the no_data_detector and why does it exist?

Sometimes the correct answer is "no records found for that filter." A naive judge would score that as incomplete or wrong. The detector recognizes legitimate empty-result answers so they aren't unfairly failed. It's a real analytics edge case — absence of data is a valid answer.

Q8. How do you actually measure the chatbot's non-determinism?

consistency.py runs the same question N times and computes response similarity, numerical consistency across runs, and the coefficient of variation on latency. If the same question gives materially different numbers across runs, that's a stability bug — exactly the kind of thing a single-shot test would miss.

Q9. What happens if there's no LLM API key — does the suite break?

No, it degrades gracefully. Every LLM evaluator has a heuristic fallback or returns a neutral 0.5, and the structural checks still run. So the suite is runnable offline or in a restricted CI environment, just with reduced fidelity.

Q10. Why ~13 evaluators here when the sibling projects consolidated to 6?

This instance kept the granular set; the FMCG/bkg lineage later consolidated correlated judges to cut LLM calls. The tradeoff: more dimensions give finer diagnostics but cost more calls and can be correlated. I'd point to the consolidation as the direction of travel — fewer, de-correlated judges.

Q11. How does the soft-gate model work and why?

Only a safety violation or an overall-score-below-threshold hard-fails a case. Every other dimension miss is a warning. Rationale: individual LLM judges are slightly flaky, so gating on each one would produce false failures and erode trust. Gate on the aggregate and on safety, surface the rest.

Q12. How do you keep costs down with 13 evaluators per case?

Content-hashed judge cache (free reruns of identical prompts), per-evaluator cost tracking so I can see the expensive ones, structural SQL pre-checks that short-circuit before an LLM call, the cheaper gpt-4o-mini as default judge, and opt-in (0.0-weighted) evaluators that don't run unless enabled.

Q13. What would you improve?

The top-level README is stale (describes a 3-metric React+FastAPI version) — docs/ARCHITECTURE_OVERVIEW.md is the accurate source, so I'd regenerate docs from code. I'd consolidate correlated evaluators like the sibling projects did, wire the UI path fully into CI, and add inter-rater reliability metrics for the judges.

9. STAR story (memorize)¶

Situation: We needed to evaluate the HOAD retail chatbot through both its API and its real UI, but couldn't afford two separate evaluation codebases. Task: Build one evaluation engine that works identically across channels and grounds its judgments in a 148-table schema. Action: I defined a normalized-payload contract so the API and Playwright UI adapters emit identical output, then built a hybrid keyword+embedding retrieval layer fused with Reciprocal Rank Fusion to feed the right schema context to the judges, plus a consistency mode to measure run-to-run determinism. Result: 100% of evaluation logic is shared across channels, judges are grounded in the correct tables, and we can quantify the chatbot's non-determinism directly — turning "does it feel right?" into measured, gated quality.

10. Honest caveats¶

README is stale/generic (3-metric React+FastAPI framing, localhost URLs) and contradicts the implementation — cite docs/ARCHITECTURE_OVERVIEW.md as the truth.
README mentions a correctness.py requiring Anthropic, but the default provider is OpenAI — minor inconsistency to acknowledge.
Git log shows CI was disabled then re-added — be ready to explain it was toggled during development.