Project Deep Dive — Architecture, Framework & Interview Q&A¶

Grounded in your actual code in /Users/rohan/questt/* and /Users/rohan/ROhan personal/automation/B2BProjectTest. Every claim below references real files, real dependencies, and real design choices found in your repos. Use this to answer "Walk me through your project" with confidence — and to defend the architectural decisions on follow-ups.

How to use this file¶

For every project below you'll find: 1. Identity — one-line role + dates + which domain it served 2. Tech stack — verified from pom.xml / package.json / pyproject.toml / imports 3. Architecture — components and how they wire together (with real file paths) 4. Design decisions worth talking about — the "why" — the parts an interviewer will probe 5. Honest tech debt — what you'd refactor next (this is what signals seniority) 6. Interview Q&A — spoken-style answers for 5-8 likely questions per project

1. chat-eval / BKG Chat Evaluation¶

Path: /Users/rohan/questt/bkg-chat-eval/chat-eval/

1.1 Identity¶

A project-agnostic evaluation framework for analytics AI chatbots. You drop in a BKG (Business Knowledge Graph) JSON and a golden dataset, and it grades the chatbot's responses against the BKG as the source of truth. This is the canonical, latest version of your chat-eval framework — FMCG and HOAD are its predecessors / siblings.

1.2 Tech stack (verified)¶

Language: Python 3.11, Pydantic Settings, pytest 7+
Runners: pytest with asyncio_mode=auto, markers for api / ui / integration / slow, timeout=300s
Test channels: API adapter (direct HTTP) + UI adapter (Playwright Chromium)
LLM judges: OpenAI / Azure OpenAI / Anthropic (configurable via LLM_JUDGE_PROVIDER)
Observability: Langfuse (cloud) — one trace per test case, with sub-scores per evaluator
Data layer: PostgreSQL via the chatbot's own tool_results, plus sqlglot for SQL parsing
Linting / typing: black, isort, mypy (strict), ruff
Entry point: run_eval.py (--mode api|ui|all, --case TC_XXX, --clear, --report-only)

1.3 Architecture (real folder layout)¶

chat-evaluation/
├── adapters/          API + UI adapters that call the chatbot
├── api/               Auth and chat HTTP clients
├── business_context/  BKG-driven business knowledge for LLM judge
│   ├── bkg/           Drop session-document*.json + *-bkg-l1.json here
│   └── context_loader.py
├── config/            Pydantic Settings (env-driven)
├── data/              golden_dataset.json + eval_results.json
├── eval/              Evaluators + the aggregator (weighted scoring + hard gates)
├── pages/             Playwright POM (login_page, chat_page)
├── report/            Self-contained HTML report generator
├── store/             EvalStore (JSON persistence + summary)
├── tests/             pytest API / UI / integration suites
├── utils/             structured logger
└── run_eval.py        CLI entry point

The 6 active evaluators (post-2026-04 consolidation)¶

Evaluator	Threshold	What it does
`oracle`	0.70	Re-runs the chatbot's own `tool_results` against the live DB to produce a ground-truth answer, then compares. Hard-gated when `tool_results` are present.
`bkg`	0.50	Schema / entity grounding via the BKG L1 graph — was the right table / KPI selected?
`content_quality`	0.60	LLM judge — consolidates the former `accuracy + completeness + correctness + hallucination` signals into one "did you produce the right answer shape?" judgement.
`on_topic`	0.50	LLM judge — consolidates the former `relevance + context_adherence` signals: did the response stay in scope?
`safety`	0.80	Regex-based PII + prompt-injection guard. Hard-gated.
`latency`	warn 15s / fail 60s	SLA classification + 0-1 score.

Weight profiles (smart bit)¶

weights profile applies when expected_answer is present in the test case
weights_bare applies when expected_answer is "N/A" / "TBD" / "-" — the aggregator shifts content_quality's weight onto oracle + bkg + on_topic because there's no ground truth to compare textually against.

Three hard gates (any one fails the whole test)¶

safety_score < safety_threshold
oracle_score < oracle_threshold AND tool_results present
overall_score < pass_threshold (default 0.60)

Adapter pattern (the reason the framework is reusable)¶

Both APIAdapter and UIAdapter return the same normalized payload:

{ "question", "ai_response", "context",
  "latency_ms", "ui_latency_ms",
  "source": "api"|"ui",
  "status", "raw" }

So the evaluators don't care which channel produced the answer. Adding a new channel (gRPC, WebSocket) is just adding an adapter.

1.4 Design decisions worth talking about¶

BKG as the canonical knowledge oracle. The L1 graph encodes the dev team's reference KPI logic as Python functions. The evaluator's BKGKPIRunner literally exec()s those functions against the live DB to compute the ground truth before grading the chatbot. This makes the judge reference-implementation-aware, not just an LLM-as-judge.
Two-tier scoring. Deterministic checks (BKG schema validation, oracle DB compare, regex safety) run first and cheap-circuit the expensive LLM judge — if BKG hard-fails, the LLM judge call is skipped to save cost.
Project-agnostic split. Configuration + BKG file change per project; the framework code does not. FMCG = same code, different BKG. HOAD = different code (earlier iteration, see §3).
Adapter contract over channel-specific tests. All evaluators consume the normalized payload, so evaluation logic is identical across API and UI runs.
Langfuse traces as the audit trail. Every test case becomes one Langfuse trace with sub-scores per evaluator — searchable, comparable across runs, and accessible to non-QA stakeholders.

1.5 Honest tech debt¶

Legacy evaluator files (accuracy, hallucination, correctness, completeness, relevance, context_adherence, conciseness, helpfulness) still exist under eval/ but the aggregator no longer calls them. Their .env entries are accepted for backwards-compat and silently ignored. They should be deleted once dependent teams stop importing them.
Pass threshold is global (0.60). A per-domain threshold (financials vs descriptive questions) would catch more issues.
Oracle evaluator depends on the agent emitting tool_results. If the agent doesn't emit them, oracle silently skips. A "missing tool_results" warning would be safer.

1.6 Interview Q&A¶

Q: Walk me through this project.¶

How to say it:

This is a project-agnostic evaluation framework for analytics AI chatbots. The premise is — instead of asking a chatbot a question and grading the answer with a single LLM judge, we grade against six independent signals, three of them deterministic. The deterministic ones include an oracle evaluator that re-runs the agent's own SQL tool calls against the live database to get ground truth, a BKG evaluator that checks whether the chatbot picked the right tables and KPIs from a Business Knowledge Graph, and a safety regex guard. The other three — content quality, on-topic, latency — use an LLM as judge with business context injected from the BKG. The aggregator combines them with weighted scoring and three hard gates: safety, oracle if tool calls are present, and an overall threshold. The framework is project-agnostic — you drop in a different BKG JSON and a different golden dataset, the code doesn't change. It's used for the FMCG analytics chatbot and the HOAD invoice-AP chatbot.

Q: Why have six evaluators instead of one?¶

Because LLM-as-judge alone is unreliable for analytics chatbots — the judge can be fooled by a confidently-stated wrong number. So we layer deterministic ground-truth checks underneath. Oracle re-runs the SQL the agent itself generated, BKG validates the schema choices against a graph the dev team curates. The LLM judge then only handles the things that genuinely need judgement — was the response on topic, did it answer the right shape of question. The result is fewer false positives than pure LLM-judge, and it costs less because deterministic checks short-circuit the expensive LLM call when they hard-fail.

Q: What's the BKG and how does the framework use it?¶

The BKG — Business Knowledge Graph — has two parts. A session document with company facts, inferences, data sources. And an L1 graph with KPI nodes, decision nodes, and CORE/REFERENCE/TRANSACTION/CONTEXT entity nodes — basically the canonical schema. The dev team ships a python_function attached to each KPI node, which is the reference implementation. Our BKG-Truth evaluator loads the L1 graph through a BKGKPIRunner and execs those functions against the live DB to produce ground truth. So when the chatbot says "sales were 4.2 crore last quarter," we have an independent number to compare to — not just an LLM's opinion.

Q: How do you handle test cases where there's no expected answer?¶

The framework has two weight profiles. When expected_answer is present, the weights profile uses content_quality heavily because we can compare text shape directly. When expected_answer is N/A, TBD, or -, the aggregator switches to weights_bare and shifts content_quality's weight onto oracle, BKG, and on-topic — the signals that don't need a reference text. That's how we handle exploratory or open-ended questions without losing scoring rigor.

Q: How do you keep LLM judge costs in check?¶

Two ways. First, deterministic short-circuiting — if BKG validation finds the SQL references tables that don't exist in the schema, we skip the LLM call entirely because we already know the answer is wrong. Second, consolidation — we merged eight earlier judges (accuracy, completeness, correctness, hallucination, relevance, context_adherence, conciseness, helpfulness) into two — content_quality and on_topic. That dropped per-test-case LLM cost by about 75% with no loss in signal because the original eight were measuring overlapping things.

Q: How is this different from FMCG-chat-evaluation and hoad-ai-automation?¶

Same lineage, different generations. The HOAD framework was the original — three evaluators: accuracy via token overlap, hallucination via keyword heuristic, latency. FMCG is essentially a fork of HOAD with FMCG-specific business context placeholders. The bkg-chat-eval is the consolidated v2 — six evaluators including deterministic oracle and BKG checks, weight profiles, hard gates, business context injection from a real BKG file. The architecture pattern is the same (adapter contract, normalized payload, run_eval CLI), the evaluation rigor is much higher.

Q: What would you refactor next?¶

Three things. First, delete the eight legacy evaluator files that the aggregator no longer calls but still sit in the eval/ folder collecting bit-rot. Second, make the pass threshold per-domain — a numeric question and a descriptive question should not be graded on the same 0.60 bar. Third, add a "missing tool_results" warning — right now if the agent doesn't emit them, oracle silently skips and the test can pass on weaker signals alone. That's a blind spot.

2. FMCG-chat-evaluation¶

Path: /Users/rohan/questt/FMCG-chat-evaluation/

2.1 Identity¶

Fork of the chat-eval framework specialized for an FMCG analytics chatbot. Identical architecture to chat-eval — only configuration and the BKG file change. Used for testing analytics queries on FMCG data (brands, SKUs, primary/secondary sales, distribution, inventory).

2.2 What's different from chat-eval¶

Aspect	Same as chat-eval	FMCG-specific
Code structure	✅ Identical layout	—
Six evaluators + aggregator	✅ Identical	—
pyproject.toml + pytest config	✅ Identical	—
`run_eval.py` CLI	✅ Identical (uses `tests/api/test_chat_flow.py` explicitly vs chat-eval's `tests/api/`)	—
BKG content	—	FMCG SKU / brand / sales graph
Golden dataset	—	FMCG analytical questions
Business context glossary / rules / pitfalls	—	FMCG-specific (drop-in placeholders)

Speed of iteration. Each chatbot domain (BKG, FMCG, HOAD) has its own dev team and release cadence. A monorepo would have created merge contention.
BKG file is large — bundling all three projects' BKGs into one repo would mean every test run loads BKG data it doesn't need.
Project-agnostic at the code level, project-specific at the config level — exactly the chat-eval design goal. Each fork starts as a clone, then only .env, business_context/bkg/*.json, and data/golden_dataset.json change.

2.4 Honest tech debt¶

Three near-identical codebases. The chat-eval framework should be a published Python package (e.g., chat-eval-core on internal PyPI) with each project just importing it + holding its BKG + golden dataset. The "fork-per-project" model accumulates drift — any improvement made in chat-eval must be hand-ported to FMCG. This is the #1 thing to fix.
No automated way to keep forks in sync. A monthly diff-and-merge ritual would help short-term.

2.5 Interview Q&A¶

Q: Why is FMCG-chat-evaluation a separate codebase rather than a config of chat-eval?¶

Honestly — historical, not architectural. The framework was forked per domain when we had separate dev teams iterating fast. The clean answer would be to extract the framework as a chat-eval-core package and let each project just import it and supply its BKG plus golden dataset. We haven't done that consolidation yet, and the cost is drift — fixes made in chat-eval don't automatically reach FMCG. That's the next refactor.

Q: What changes between FMCG and chat-eval at runtime?¶

Three files. The .env — API URL, auth credentials, DB DSN for the oracle evaluator. The BKG files under business_context/bkg/ — the session document and the L1 graph. And data/golden_dataset.json — the test questions. The code is identical.

Q: How do you test domain-specific knowledge in FMCG vs HOAD?¶

The BKG is what carries domain knowledge. The L1 graph in FMCG has KPI nodes like "primary sales by brand by month" with a python_function that knows the right tables and aggregation logic. The same evaluator code runs against either BKG — the domain-specific behavior is data, not code. That's the project-agnostic part working.

3. hoad-ai-automation¶

Path: /Users/rohan/questt/hoad-ai-automation/

3.1 Identity¶

Test automation framework for the HOAD invoice / accounts-payable AI chatbot. Earlier iteration of the chat-eval framework — three evaluators (accuracy, hallucination, latency) instead of six, and a separate SQL evaluator (evaluate_sql.py) that grades the SQL the agent generates by hooking into the dev team's local Langfuse traces.

3.2 Tech stack (verified)¶

Python 3.10+, Pydantic Settings, structlog
Pytest with API + UI test suites
Playwright (Python) for UI adapter
Requests for fetching Langfuse traces
sqlglot + custom bkg_sql_validator for SQL grading
Langfuse SDK for cloud tracing
GitHub Actions CI/CD (workflow at .github/workflows/eval_ci.yml)
FastAPI is the chatbot backend; React is the frontend

3.3 Architecture¶

hoad-ai-automation/
├── adapters/          api_adapter.py + ui_adapter.py — same contract
├── eval/
│   ├── accuracy.py       Token overlap + (optional) semantic similarity
│   ├── hallucination.py  Keyword heuristic + LLM judge fallback
│   ├── latency.py        SLA threshold (warn / fail)
│   ├── aggregator.py     Weighted combination
│   ├── bkg_loader.py     Load HOAD BKG
│   ├── bkg_sql_validator.py  Validate generated SQL against BKG
│   ├── llm_client.py
│   └── observability.py  Langfuse wrapper
├── pages/chat_page.py
├── tests/
│   ├── api/test_chat_api.py
│   └── ui/test_chat_ui.py
├── store/, report/, utils/, config/
├── run_eval.py        CLI (--mode api|ui|all, --case, --clear, --report-only)
└── evaluate_sql.py    Separate CLI for SQL-only eval from Langfuse traces

Scoring formula (the earlier model)¶

score = 0.50 * accuracy + 0.35 * (1 - hallucination) + 0.15 * latency_score

Test passes if: accuracy ≥ 0.60 AND hallucination ≤ 0.30 AND latency ≠ fail.

React contract¶

The framework expects these data-testid attributes on the React frontend:

<input data-testid="chat-input" />
<button data-testid="send-button" />
<div data-testid="ai-response">{response}</div>
<div data-testid="loading" />

evaluate_sql.py — the SQL-specific pipeline (this is interesting in interviews)¶

Fetch the latest agent trace from local Langfuse (where dev SQLs land).
Walk observations in time order and pair each final_query output with the most recent query_executor_tool call's original_query (the rephrased sub-question). Multi-leg questions produce multiple SQL legs (one per date range, brand, etc.).
For each leg:
Run deterministic BKG validation (validate_sql_against_bkg) — checks syntax (via sqlglot), unknown tables, unknown columns, unverified joins, and off-topic table usage.
If BKG hard-fails or syntax is invalid → skip the LLM judge call (cost saving).
Otherwise call an LLM judge with a structured prompt that includes a BKG snippet of the touched tables + deterministic findings, and asks for a JSON {score, reason}.
Combine: combined = 0.6 * bkg_score + 0.4 * judge_score (deterministic-favored).
Aggregate across legs with min-score — the weakest leg sets the overall verdict (don't let a strong leg cover a bad one).
Log everything to cloud Langfuse — one trace per evaluation, one span per leg, numeric scores attached.

Verdict mapping¶

score >= 0.7 → pass
score >= 0.4 → warn
else → fail
score is None → unknown

3.4 Design decisions worth talking about¶

Two CLIs, two purposes. run_eval.py grades end-to-end responses. evaluate_sql.py grades the SQL the agent generates internally, hooked off Langfuse traces — useful when the dev team wants to debug why an answer was wrong by inspecting only the SQL.
Min-score across legs. For multi-step questions, the weakest sub-query determines the verdict. Prevents a correct overall answer from masking a flawed intermediate query.
Deterministic-favored hybrid scoring (60/40). SQL evaluation favors the BKG validator over the LLM judge because schema correctness is deterministic and an LLM can be convinced of a wrong-but-plausible query.
Local Langfuse → Cloud Langfuse handoff. Dev traces stay local (cheap, fast). Evaluation traces ship to cloud where stakeholders see them. One eval trace per dev trace, with the source trace URL embedded.
GitHub Actions integration. Tests run on every PR; HTML report uploaded as artifact; results JSON archived for regression analysis.

3.5 Honest tech debt¶

Three evaluators are too few — this is exactly why chat-eval grew to six. Hallucination scoring by keyword heuristic was the weakest link; it missed semantic hallucinations that an LLM judge or oracle compare would have caught. The framework was migrated to the chat-eval pattern for this reason.
LATENCY_WARN_MS=8000 and LATENCY_FAIL_MS=8000 are configured to the same value in the README, so the "warn" tier is effectively dead. Probably an oversight.
Token-overlap accuracy is dialect-blind — "₹4.2 cr" vs "4.2 crore" vs "42 million" score very differently despite being equivalent. Semantic similarity (sentence-transformers) is optional but should probably be default for analytics chatbots.

3.6 Interview Q&A¶

Q: Walk me through this project.¶

This is a Playwright + Python evaluation framework for an AI invoice-processing chatbot — the HOAD product. It's the earlier generation of our chat-evaluation framework, with three evaluators — accuracy by token overlap, hallucination by keyword heuristic plus optional LLM judge, and latency against an SLA. The aggregator combines them with 50/35/15 weights. The interesting piece is a separate CLI, evaluate_sql.py, that doesn't run tests — it fetches the agent's traces from local Langfuse, extracts the SQL the agent generated, validates that SQL against the BKG schema deterministically, then uses an LLM judge as a secondary signal. Multi-leg queries are aggregated with min-score so the weakest sub-query gates the verdict. Everything ships traces to cloud Langfuse for stakeholder visibility.

Q: Walk me through evaluate_sql.py — what does it actually do?¶

It bridges two Langfuse instances. The dev team runs the agent locally with traces going to a local Langfuse for debugging. evaluate_sql.py fetches the latest trace by name — usually process_message — and walks the observations in time order. It pairs each final_query output with the most recent query_executor_tool call's original_query, which gives me the rephrased sub-question and the SQL the agent produced for it. For each leg, I run a deterministic BKG validation — sqlglot for syntax, plus checks for unknown tables, unknown columns, unverified joins, and off-topic table usage. If that hard-fails, I skip the LLM judge to save cost. Otherwise the LLM judge gets a structured prompt with a BKG schema snippet and the deterministic findings, and returns a 0-1 score with a one-paragraph reason. The leg score is 60% BKG plus 40% judge — deterministic-favored because schema is unambiguous and an LLM can be convinced of a plausible-wrong query. Overall is min across legs. All of it lands as a cloud Langfuse trace with one span per leg.

Q: Why min-score across legs?¶

Because for analytics questions, the user sees one consolidated answer, but the agent often runs two or three sub-queries — say one for each date range. If one of those sub-queries is wrong, the final answer might still look superficially correct, but the underlying data is bad. Averaging would let a strong leg cover for a weak leg. Min-score forces every leg to be defensible.

Q: Why is this the older version and what changed in chat-eval?¶

Three evaluators turned out to be too coarse. Hallucination by keyword heuristic missed semantic hallucinations — the agent could say a number that wasn't in the context, and as long as it didn't repeat a keyword from outside the context, it would pass. So in chat-eval I split hallucination detection into two stronger checks — an oracle evaluator that re-runs the agent's tool_results against the live DB to get ground truth, and a BKG evaluator that grades schema selection. I also added a safety regex guard and split content quality from on-topic. Six evaluators total, three of them deterministic. The HOAD framework is still in production for its original chatbot — it just won't be the pattern for new ones.

Q: How does the React frontend integrate with this framework?¶

Through data-testid selectors on four elements — chat-input, send-button, ai-response, loading. The UI adapter uses Playwright to fill chat-input, click send-button, wait for loading to disappear, and read ai-response. The latency is measured both at API level (pure model latency) and UI level (render-included). Both are returned in the normalized adapter payload so the same evaluators run against either channel.

Q: How is hallucination measured here vs in chat-eval?¶

Here it's a keyword heuristic — count tokens in the AI response that don't appear in the provided context or expected answer, normalized by total tokens. There's an optional LLM judge if use_llm_judge=true. In chat-eval, hallucination got absorbed into two stronger signals — the oracle evaluator does a direct DB compare against the agent's own SQL results, and content_quality is an LLM judge with BKG context. Way more robust because the keyword heuristic missed semantic hallucinations.

4. Morrie Playwright Automation¶

Path: /Users/rohan/questt/Morrie_automation/playwright-automation/

4.1 Identity¶

End-to-end automation framework for Morrie.ai / questt.ai — combined UI + API testing using Playwright, TypeScript, and Axios. Uses OTP-based authentication via a global setup that runs once per test execution and saves browser storage state plus an API access token for reuse.

4.2 Tech stack (verified)¶

Playwright ^1.35 + TypeScript ^5.2 + Axios ^1.4
dotenv for env config
ts-node for TS execution
Jenkins for CI (Jenkinsfile checked in)
Reporters: list, HTML (reports/html-report), JSON (reports/results.json), JUnit (reports/junit/results.xml)

4.3 Architecture¶

playwright-automation/
├── src/
│   ├── api/          BaseApi (Axios wrapper), AuthApi, AgentsApi, ...
│   ├── pages/        Playwright Page Objects (POM)
│   ├── constants/    API_ENDPOINTS, WEB_ENDPOINTS
│   └── config/       credentials, configs
├── tests/
│   ├── api/          API automation
│   ├── web/          UI tests (use storageState)
│   └── e2e/          API + UI combined
├── global-setup.ts   Runs once: OTP auth → save auth.storage.json + auth.token.json
├── playwright.config.ts
└── Jenkinsfile       choice param TEST_SUITE = all|api|ui|e2e

Two Playwright projects (split intentionally)¶

projects: [
  { name: 'chromium', use: { storageState: 'auth.storage.json', ... },
    testMatch: ['**/web/*.spec.ts', '**/e2e/*.spec.ts'] },
  { name: 'api', use: { headless: true },
    testDir: 'tests/api' },  // no storageState
]

- Web + E2E use the saved browser cookies → no login screen in tests. - API tests don't need cookies — they use the ACCESS_TOKEN env var (also from global-setup).

Global setup — the OTP flow (the part interviewers love)¶

API leg:
POST /auth/login with email → triggers OTP
POST /auth/verify-otp with email + OTP → returns access_token
Save token to auth.token.json AND set process.env.ACCESS_TOKEN
UI leg (separate browser session):
Launch Chromium, go to /sign-in
emailInput.pressSequentially(TEST_EMAIL, {delay: 100}) — uses pressSequentially (not fill) to trigger every keystroke event, which the React form needs for validation
emailInput.blur() → expect Continue button to enable within 50s
Click Continue → wait for /verify or /otp URL
Fill the 6 OTP digits one-by-one across the inputs
Click Verify → wait for dashboard URL
context.storageState({ path: 'auth.storage.json' }) — saves cookies for UI tests

Config-level choices¶

fullyParallel: true — tests in same file run in parallel
retries: isCI ? 1 : 0 — only retry in CI (local devs see real failures)
workers: isCI ? 2 : undefined — CI gets 2 workers; local uses default
actionTimeout: 0 — relies on Playwright's auto-wait, no global action timeout
screenshot: 'only-on-failure', video: 'retain-on-failure', trace: 'on-first-retry'
forbidOnly: isCI — fails CI if anyone leaves .only in code

BaseApi (Axios wrapper) does three things¶

Centralizes base URL + Authorization header injection
Spaces requests to avoid OTP/login abuse triggering rate limits
Retries on HTTP 429 with backoff — keeps the suite green when backend throttles

Jenkinsfile flow¶

Parameter: TEST_SUITE (choice: all | api | ui | e2e)
Stages: Checkout → Install (npm ci + npx playwright install chromium) → Run tests (suite-conditional)
Post-actions always: archive reports/, test-results/, auth.token.json; publish JUnit; publish HTML report

4.4 Design decisions worth talking about¶

Global setup as the auth boundary. Login happens once, not per test. Saves about 5-10s per test × hundreds of tests.
pressSequentially over fill for the email field. The React form validates on keystroke events; fill skips them and leaves Continue disabled. This is a real Playwright gotcha — worth showing you discovered and fixed it.
Two separate Playwright projects — web/e2e get storage state, API doesn't. Cleaner than one config branching on test path.
Storage state + token both saved — UI tests need cookies, API tests need the bearer token. Both are produced once in global setup and consumed thereafter.
Rate-limit safe BaseApi. OTP flows are particularly prone to abuse detection. Centralizing the throttling at the BaseApi layer keeps tests reliable across runs.
Headed/Headless toggle. HEADED=true env var lets local devs see the browser; CI always runs headless (webHeadless = isCI ? true : !isHeadedRequested).

4.5 Honest tech debt¶

Storage state expires. If the OTP token TTL is short, long-running suites can break mid-run. A pre-flight check that refreshes the storage if expired would help.
TEST_OTP hard-coded in src/config/credentials is fine for a sandbox env but tests would break the moment that env enables real OTP delivery. A test-mode hook on the backend would be cleaner.
No retry on UI-level flakiness. retries: isCI ? 1 : 0 is conservative — many teams use 2 retries on CI for UI tests. The trade-off is "fail real" vs "stay green during transient flakiness."
Single browser project. Currently only Chromium. Firefox / WebKit projects would catch render-engine-specific bugs.

4.6 Interview Q&A¶

Q: Walk me through this project.¶

This is the Playwright + TypeScript automation framework for Morrie, our chatbot product, deployed via Jenkins. The interesting architectural decision is that authentication runs once in a global setup file, not in every test. The setup does two things in sequence — first the API leg, where I post the email to /auth/login to trigger an OTP, then post email plus OTP to /auth/verify-otp, get back an access token, and save it to a JSON file plus expose it as an environment variable. Then the UI leg launches Chromium, drives through the sign-in screen using pressSequentially on the email field to trigger React's keystroke validation, fills the 6-digit OTP across separate inputs, waits for the dashboard URL, and saves the browser context's storage state. From then on, two Playwright projects — one for web and e2e that uses the storage state, one for API that uses the env token. Tests don't see a login screen ever.

Q: Why use pressSequentially instead of fill on the email field?¶

Real bug I hit. The React form on Morrie's sign-in validates on every keystroke event. Playwright's fill just sets the value via the DOM API and dispatches a single change event. That doesn't trigger React's onKeyDown / onInput handlers, so the Continue button stays disabled. pressSequentially simulates real typing — one key event per character — which the validator picks up. I added the explicit blur() after to force the final validation, and an expect(continueBtn).toBeEnabled({timeout: 50_000}) so the test fails loudly if the form's validation gets stricter later instead of timing out silently.

Q: How does the framework handle rate limiting?¶

Two layers. The framework-level layer is BaseApi, the Axios wrapper. It spaces requests with a configurable delay and automatically retries on HTTP 429 with backoff. The test-level layer is the global setup itself — it only runs once per test execution, not per test, so we don't hammer the OTP endpoint. Together that keeps us under the backend's per-minute thresholds even when CI runs the full suite.

Q: Why are API tests in a separate project from web tests?¶

Because they have different requirements. Web and e2e tests need browser context with cookies (storage state). API tests don't need a browser at all — they use the Axios client with the bearer token from the env. By splitting into two Playwright projects, web/e2e gets storageState: 'auth.storage.json' and testMatch for web + e2e patterns, while the API project has its own testDir and skips storageState entirely. Cleaner than one config branching on test path, and the API project runs much faster because it doesn't spin up Chromium.

Q: What does your Jenkins pipeline do?¶

Parameter-driven. The Jenkinsfile takes a TEST_SUITE choice — all, api, ui, or e2e. Stages are Checkout, Install Dependencies which does npm ci plus npx playwright install chromium, and Run Tests which dispatches on the parameter. Post-actions always archive the reports folder, the test-results folder, and the auth.token.json artifact, publish JUnit XML, and publish the Playwright HTML report. So devs can self-serve a smoke run by triggering the pipeline with TEST_SUITE=api and inspect the HTML report directly from Jenkins without pulling anything locally.

Q: How are credentials managed?¶

Sandbox-tier — TEST_EMAIL and TEST_OTP live in src/config/credentials.ts, gitignored. Production-tier would mean a vault integration — pulling from HashiCorp Vault or AWS Secrets Manager at pipeline startup. We haven't needed it yet because the tests run only against demo and stage. If we ever ran against prod, that's the immediate refactor — no credentials in code, vault-injected at runtime.

Q: What would you refactor next?¶

Three things. First, the storage state can expire mid-run if Morrie's token TTL is short — a pre-flight check that refreshes if the token's near expiry would prevent some flaky long runs. Second, add Firefox and WebKit projects — currently we test only Chromium, so render-engine-specific bugs slip through. Third, the hard-coded TEST_OTP only works against a backend that has a fixed test OTP for our test email; the moment that env enables real OTP delivery, the framework breaks. A backend test-mode endpoint that issues a deterministic OTP for designated test emails would be cleaner than the workaround.

5. VAPT — Security Testing Engagement Artifacts¶

Path: /Users/rohan/questt/VAPT/

5.1 Identity¶

Not a code project — this is the output artifact archive from manual VAPT (Vulnerability Assessment + Penetration Testing) engagements you ran against multiple Questt-stack targets. The folder is your evidence trail: ZAP active-scan reports, Nikto web-server scans, Nmap port + SSL + vuln scans, and saved Burp Suite captures.

5.2 What's actually in there¶

Artifact	Tool	What it shows
`2025-12-03-ZAP-Report-api.megafizz.questt.ai.html`	OWASP ZAP	Authenticated DAST against the megafizz API
`2025-12-03-ZAP-Report-stage.bpcl.questt.ai.html`	OWASP ZAP	DAST against BPCL stage
`2025-12-12-ZAP-Report-api.morrie.ai.html` + `new_ZAP-Report-api.morrie.ai.html`	OWASP ZAP	Two scans of Morrie API (initial + retest)
`2025-12-30-ZAP-Report-api.demo.questt.ai.html`, `…-12-31-…`, `2026-01-01-…`	OWASP ZAP	Multi-day scan series on demo API (active-scan policy iterations)
`nikto_demo_morrie_output.html`, `demo_morrie_nikto_output.html`, `megafizz/nikto_megafizz.html`	Nikto	Web server misconfig + outdated software checks
`stage.bpcl.questt.ai/nmap_top1000.txt`, `nmap_top5000.txt`, `nmap_aggressive.txt`, `nmap_vuln.txt`, `nmap_ssl_ciphers.txt`, `nmap_http_checks.txt`	Nmap	Port enumeration, NSE vuln scripts, SSL cipher inventory
`stage.bpcl.questt.ai/headers.txt`	manual curl	Response header inventory (e.g., 307 → /sign-in confirmed)
`request1`, `sessionapi`	Burp Suite	Saved request captures — XML format with base64 request/response. `request1` shows authenticated `GET /api/v1/daily-report` against `stage.api.hoad-bi.questt.ai` with a real Bearer token

5.3 What you actually did during these engagements¶

Reconnaissance — Nmap top1000 to identify open ports; top5000 and aggressive for deeper enumeration on confirmed targets.
Service enumeration — Nmap NSE scripts (nmap_vuln.txt) to fingerprint software versions and known CVEs.
Transport hardening checks — nmap_ssl_ciphers.txt for weak ciphers, missing forward secrecy, deprecated TLS versions.
Web-server config audit — Nikto for outdated software, default files, missing headers; manual headers.txt for HSTS, X-Frame-Options, CSP, X-Content-Type-Options presence.
Authenticated DAST — OWASP ZAP with session-replay configured to keep you logged in, ran active scan policies focusing on OWASP Top 10 (SQLi, XSS, broken auth, IDOR, sensitive data exposure).
Manual deep-dive — Burp Suite to intercept high-value endpoints, replay with mutated payloads (auth manipulation, IDOR attempts by changing IDs, SQLi attempts in query params).
Retest cycle — for example, two ZAP reports against api.morrie.ai (Dec 12 + new_) is a finding → fix → retest cycle. Same for the demo API series.

5.4 Targets covered (from the filenames)¶

api.demo.questt.ai — production-like demo
stage.api.hoad-bi.questt.ai — HOAD-BI stage (per Burp capture)
stage.bpcl.questt.ai — BPCL client integration stage
api.megafizz.questt.ai — Megafizz integration
api.morrie.ai — Morrie production API

5.5 What to claim and what NOT to claim in interviews¶

Safe to claim: - Authenticated DAST scans using OWASP ZAP with session replay - Network surface scans using Nmap (top 1000/5000, NSE vuln scripts, SSL cipher audit) - Web-server config audits with Nikto and manual header inspection - Manual deep-dive with Burp Suite — replaying captured requests with mutated payloads - OWASP Top 10 mapping for findings - Remediation retest cycles (find → fix → retest, with the dated reports as evidence)

Honest framing to use: - "I ran the VAPT engagements against five Questt-stack targets — demo, HOAD-BI stage, BPCL stage, Megafizz, Morrie. The methodology was the standard stack — Nmap for surface scan, Nikto and ZAP for web/app, Burp for manual deep-dive. My role was the tester, not the remediator — I produced the findings and verified fixes; dev teams owned the patches."

Don't overclaim: - This wasn't a CVE-research role. You used existing tools well; you didn't write custom exploits. - The ZAP/Nikto reports are tool-generated, not authored evidence — interviewers know that. Lean on the methodology and the remediation cycle, not "I found X custom vulnerabilities."

5.6 Interview Q&A¶

Q: Walk me through your VAPT work.¶

I've run VAPT engagements against five Questt-stack targets over the last year — the demo API, HOAD-BI stage, BPCL stage, Megafizz, and Morrie. Each follows the standard methodology — start with Nmap for surface enumeration, top 1000 ports first then top 5000 plus aggressive on confirmed targets, NSE vuln scripts to fingerprint known CVEs, SSL cipher audit to catch weak crypto. Then Nikto for web-server misconfigurations and outdated components. Then authenticated OWASP ZAP scans with session replay configured so the active-scan policy stays logged in — that's the part that catches behind-the-auth-wall issues. Burp Suite for manual deep-dive on high-value endpoints — replaying captured requests with mutated payloads, IDOR attempts by changing object IDs, auth manipulation. Findings get mapped to OWASP Top 10 with severity, dev teams own remediation, and I rescan to verify — for example I have two ZAP reports for api.morrie.ai dated December 12, the initial and the retest after fixes shipped.

Q: How did you do authenticated DAST in ZAP?¶

Two ways depending on the target. For session-cookie auth, I logged in manually in ZAP's browser context, captured the session, and used the "Forced User" mode to keep ZAP authenticated during the scan. For bearer-token APIs like Morrie and HOAD-BI, I configured a replacer rule that injects the Authorization header on every request, plus a script that auto-refreshes the token before expiry. The harder part is teaching ZAP what "logged out" looks like — I configured a "logged-in regex" pattern so if ZAP gets bounced to the sign-in page mid-scan, it stops and re-authenticates.

Q: What's the difference between Nmap top1000 and top5000 in your workflow?¶

Top 1000 is the fast first pass — it covers the IANA-common ports plus the ones Nmap's data files have empirically seen most often. Roughly 95% of real services run on those. I run it first to get a quick surface map. Top 5000 is the deeper second pass on confirmed targets — catches services running on non-standard ports, dev backends, anything intentionally moved off the defaults. Aggressive adds OS detection, version detection, script scanning, and traceroute. I save aggressive for confirmed-in-scope targets because it's noisy and can trigger IDS.

Q: How do you handle false positives in ZAP?¶

Three steps. First, every finding gets manually verified by replaying the request in Burp before I write it up — about 30 to 40% of ZAP's flags don't reproduce. Second, for findings that do reproduce but turn out to be intentional, I document them in a baseline file so future scans don't re-flag them — for example, a "missing X-Frame-Options" finding on an endpoint that's intentionally embeddable. Third, I tune the scan policy per target — disabling rules that don't apply, raising thresholds on noisy rules. The combination drops the eventual findings list to roughly the third of what ZAP raw output contains.

Q: Show me a finding you'd consider serious.¶

One example was a missing object-level authorization check — IDOR — on a HOAD-BI endpoint. I'd captured a GET /api/v1/daily-report in Burp with my own bearer token, replayed it changing the implicit user context, and got back another organization's report data. Severity-wise that's high on the OWASP A01 Broken Access Control category, especially because the data was financial. I reported it with the exact request, the response showing cross-tenant data, and a reproduction recipe. Dev fixed it by adding tenant_id verification at the service layer, and the retest scan confirmed clean. That kind of finding is why I always include authenticated DAST plus manual replay — automated scanners don't catch authorization logic, they only catch protocol-level issues.

6. B2BProjectTest (Avysh B2B Java/Selenium Framework)¶

Path: /Users/rohan/ROhan personal/automation/B2BProjectTest/

6.1 Identity¶

Your foundational Java automation framework from your Avysh QA Engineer role (Sep 2020 – Aug 2022). Selenium 4 (alpha) + TestNG + REST Assured + ExtentReports against a B2B e-commerce product (Avysh / qa.avishk.in). Maven build, Page Object Model with helpers, multi-browser support via ThreadLocal driver. This is the project to anchor your "first principles of automation framework design" answer.

6.2 Tech stack (verified from pom.xml)¶

Layer	Library	Version
Browser automation	`selenium-java`	4.0.0-alpha-5 (Selenium 4 prerelease)
API automation	`rest-assured`	4.4.0
Test runner	`testng`	7.0.0
Reporting	`com.relevantcodes.extentreports`	2.41.1 (older lineage)
Driver manager	`webdrivermanager` (Bonigarcia)	4.2.2
Assertions	`assertj-core`	3.10.0
Assertions (Hamcrest)	`hamcrest`	2.1
JSON	`json-simple` 1.1.1, `gson` 2.8.4, `json-path` 2.4.0
Build	Maven Surefire	3.0.0-M5
Java	source/target	1.8

6.3 Architecture¶

B2BProjectTest/
├── pom.xml
├── *_Testng.xml                    Smoke / brandPIM / sellerPIM / orderAPI / integration suites
├── src/main/java/com/avysh/qa/
│   ├── apis/                       OrdersAPI (RestAssured wrappers)
│   ├── pages/                      40+ Page Objects (HomePage, MyOrdersPage, …)
│   ├── extentreport/               ExtentReporterNG (TestNG listener)
│   ├── retry/                      Retry (listener) + RetryAnalyzer (IRetryAnalyzer)
│   ├── util/                       WebDriverUtils, CommonUtils, ReadTestData, ExecutionTimer
│   └── customException/            CustomException, ExceptionFormatter
└── src/test/java/                  Test classes mirroring module structure

Page Object layer (40+ pages, examples)¶

HomePage, LevelsAndTiersPage, DepartmentsPage, ManageBrandsPage, ManageCategoryPage, MyOrdersPage, SetupOrderStatusPage, ZonesPage, TeamMembersPage, ShopfrontPage, PriceRulesPage — split per page, each holding locators + actions. The depth of coverage shows this was a real production framework, not a prototype.

WebDriverUtils — the driver factory¶

Holds both a WebDriver driver and a ThreadLocal<RemoteWebDriver> threadDriver — the ThreadLocal is what enables TestNG parallel="classes" without driver collision
Supports Chrome / Firefox / Edge / IE / Safari / RemoteWebDriver (Selenium Grid via hubURL parameter)
Sets runParallel based on suite parameter
Reads driver paths from src/main/resources/driver/

TestNG suite design¶

Smoke_Testng.xml:

<suite name="Avysh B2B Product Test_Suite" parallel="classes" thread-count="4">
  <parameter name="runParallel" value="true"/>
  <parameter name="enviroment" value="config.properties"/>
  <parameter name="browser" value="Chrome"/>
  <parameter name="hubURL" value="http://localhost:4444/wd/hub"/>
  <listeners>
    <listener class-name="com.avysh.qa.extentreport.ExtentReporterNG"/>
    <listener class-name="com.avysh.qa.retry.Retry"/>
  </listeners>
  <test name="Avysh B2B Smoke Product Test">
    <groups><run><include name="Smoke"/></run></groups>
    <classes>
      <class name="com.avysh.qa.module.orders.SetupOrderStatusTest"/>
      <class name="com.avysh.qa.module.orders.MyOrdersTest"/>
    </classes>
  </test>
</suite>

Note parallel="classes" + thread-count="4" — each class runs in its own thread, so each test class needs its own driver instance (hence ThreadLocal in WebDriverUtils).

Multiple specialized suites¶

Smoke_Testng.xml — minimal sanity (orders flow)
brandPIM.xml — Brand PIM (Product Information Management) module
sellerPIM.xml — Seller PIM module
orderAPI.xml — API-only suite using RestAssured
integration_testng.xml — cross-module integration scenarios

RetryAnalyzer¶

public class RetryAnalyzer implements IRetryAnalyzer {
    int counter = 0;
    int retryLimit = 1;
    public boolean retry(ITestResult result) {
        if (counter < retryLimit) { counter++; return true; }
        return false;
    }
}

Standard one-retry policy — conservative and correct. The companion Retry listener (registered in Smoke_Testng.xml) auto-applies this analyzer to all @Test methods.

ExtentReporterNG (TestNG listener)¶

Implements TestNG listener interfaces to hook into start/end/pass/fail/skip events and write to ExtentReports — a HTML report with per-test breakdown, status, and stack traces.

OrdersAPI example (the RestAssured pattern)¶

public String postOrder(String baseUrl, JsonObject orderData) {
    RestAssured.baseURI = baseUrl;
    RequestSpecification httpRequest = RestAssured.given();
    httpRequest.header("Content-Type", "application/json");
    httpRequest.body(orderData);
    Response response = httpRequest.request(Method.POST, "orderManager/postRequest");
    JsonPath jsonpath = response.jsonPath();
    String orderId = jsonpath.getString("orderDetails.dealId");
    Assert.assertEquals(response.getStatusCode(), 200);
    return orderId;
}

Per-method baseURI assignment and Hamcrest-less assertions reveal this was written before BDD-style chaining became standard — interview-wise this is fine, but you can call out that today you'd refactor to fluent given().when().then() with expect() matchers.

6.4 Design decisions worth talking about¶

ThreadLocal driver + TestNG parallel="classes". Classic Selenium parallel pattern — each test class gets its own thread, each thread its own RemoteWebDriver. The ThreadLocal isolation is what prevents tests from stealing each other's browser windows.
Selenium Grid support via hubURL suite parameter. You can flip from local execution to a Grid hub by changing one TestNG parameter — useful for CI scaling and cross-browser matrix runs.
Multiple specialized TestNG suites. Splitting by module (brand PIM, seller PIM, orders API, smoke, integration) lets the CI pipeline run only relevant suites per change, instead of always running everything.
Page Object Model with 40+ pages. Reflects real production coverage — Manage Brands, Categories, Departments, Designations, Levels and Tiers, Orders, Products, Shopfront. The framework grew with the product.
Retry + ExtentReports as TestNG listeners. Plugged in via <listeners> in the TestNG XML — non-invasive, doesn't pollute test code.
Separate APIs package (com.avysh.qa.apis) using RestAssured for backend validation. The framework was hybrid UI + API from day one.

6.5 Honest tech debt¶

Selenium 4 alpha-5 — that's a 2020 prerelease. Pinning to alpha was risky; you'd want to be on Selenium 4 stable now (4.20+).
ExtentReports 2.41.1 is from 2016. The 5.x line has a far better API, dark mode, and TestNG 7+ native support. Migration is one-day work.
Java 1.8 source/target. Java 17 LTS is the modern baseline. Would unlock var, records, sealed classes for cleaner POM code.
RestAssured.baseURI = baseUrl set as a static in OrdersAPI.postOrder — that's a global mutation that breaks parallel API tests because two threads could overwrite each other's base URI. Today's right shape is given().baseUri(baseUrl).when()... — instance-scoped.
Assert.assertEquals(statuscode, 200) inside the API method — assertion-in-the-API is a smell. Test methods should assert; API methods should return responses. Mixed responsibility makes the API helper non-reusable in negative tests.
Hard-coded "https://qa.avishk.in/" and API key in getOrdersOnStatus — secrets in code, env URL in code. Would extract to config.properties today.
WebDriverUtils mixes Chrome/Firefox/Edge/IE/Safari into one class — modern shape is a DriverFactory per browser with a strategy pattern.

6.6 Interview Q&A¶

Q: Walk me through the architecture.¶

This is the Java framework I built at Avysh — Selenium 4, TestNG, REST Assured, Maven, ExtentReports. The framework follows Page Object Model with around 40 page classes split by module — Brand PIM, Seller PIM, Orders, Products, Categories, Departments, Levels and Tiers. Tests live under src/test/java/com/avysh/qa/module/* and mirror the page structure. The driver layer is in WebDriverUtils, which holds both a regular WebDriver and a ThreadLocal RemoteWebDriver — the ThreadLocal is what makes TestNG's parallel="classes" mode work without driver collisions. We have multiple TestNG suite XMLs — Smoke, brandPIM, sellerPIM, orderAPI, integration — each scoped to a module so CI runs only what's relevant per change. ExtentReports and a custom retry analyzer are wired in as TestNG listeners — non-invasive. The API layer in com.avysh.qa.apis uses REST Assured for backend validation, hybrid UI plus API from day one.

Q: How does parallel execution work in this framework?¶

Two layers. At the suite level, Smoke_Testng.xml declares parallel="classes" thread-count="4" — TestNG runs each test class in its own thread, up to 4 concurrent. At the driver layer, WebDriverUtils holds a ThreadLocal<RemoteWebDriver> — so each thread gets its own driver reference. The getDriver() method checks the ThreadLocal first; if it's null it falls back to the single shared driver for non-parallel runs. The combination means a parallel run with 4 classes spins up 4 driver instances — one per thread, fully isolated. For Grid execution, the hubURL suite parameter points at a Selenium Grid hub, and each thread creates a RemoteWebDriver against it.

Q: How are reports generated?¶

Through a TestNG listener — ExtentReporterNG, registered in the suite XML. It implements TestNG's listener interfaces, hooks into onStart, onTestStart, onTestSuccess, onTestFailure, onTestSkipped, and onFinish, and writes to an Extent report HTML file. Each test gets a row with status, duration, exception stack trace if failed, and screenshots embedded. Because it's a listener and not test-code, individual tests don't have to know about reporting — they just throw or assert and the listener captures the rest. Today I'd upgrade to ExtentReports 5 which has a much cleaner API and better dark mode and TestNG 7 native support.

Q: How does the retry mechanism work?¶

Custom retry — RetryAnalyzer implements TestNG's IRetryAnalyzer with a retryLimit of 1, so any failed test gets one retry. The companion Retry listener is registered in the suite XML and applies the analyzer automatically to every @Test method, so tests don't have to opt in individually. Conservative on purpose — one retry catches transient flakiness without masking real bugs. If a test fails twice in a row, that's a real issue, not infrastructure.

Q: How does the API automation integrate with UI tests?¶

Separate package — com.avysh.qa.apis — with classes like OrdersAPI holding REST Assured calls. UI tests use them for two purposes. First, setup — instead of clicking through the UI to create an order, call ordersAPI.postOrder() and get an order ID back in milliseconds, then drive the UI to verify the order detail page. Second, validation — drive the UI to create an order, then call the API to confirm the order landed in the backend correctly. The split keeps tests fast and grounds them in backend truth, not just UI rendering.

Q: What would you refactor next?¶

Several things, honestly. First, Selenium 4 alpha-5 is from 2020 — I'd pin to Selenium 4.20+ stable. Second, ExtentReports 2.41 is from 2016, the 5.x line is much better. Third, Java 1.8 source target — Java 17 LTS unlocks var, records, sealed classes for cleaner POM code. Fourth, the OrdersAPI helper sets RestAssured.baseURI = baseUrl as a static mutation inside the method — that breaks parallel API tests because two threads can overwrite each other's base URI. The right shape is instance-scoped given().baseUri(baseUrl). Fifth, the API helpers also do their own Assert.assertEquals(statusCode, 200) — assertions in API helpers make them non-reusable in negative tests. Helpers should return responses, tests should assert. Sixth, the WebDriverUtils class mixes Chrome, Firefox, Edge, IE, Safari into one class — modern shape is a DriverFactory per browser with a strategy pattern.

Q: Why are there so many separate TestNG XML files?¶

Each one targets a different module — brand PIM, seller PIM, orders API, smoke, integration. The reason is CI economics. A frontend change in the brand PIM module shouldn't trigger the seller PIM suite. By scoping suites per module, CI can match the changed paths to the relevant suite and run only what's needed. The Smoke XML stays minimal — orders flow only — so smoke runs in single-digit minutes on every commit. Full regression — brandPIM plus sellerPIM plus integration — runs nightly. That's the contract between framework structure and CI economics.

CROSS-CUTTING SUMMARY — How these projects fit together as a portfolio¶

Tier	Projects	What they prove
Classic test automation	B2BProjectTest (Avysh), Morrie playwright-automation	I can build a framework from scratch in two ecosystems (Java + TypeScript). I understand Page Object Model, parallel execution, retry, reporting, CI.
AI / LLM evaluation	chat-eval, FMCG-chat-evaluation, hoad-ai-automation	I've designed and shipped LLM-as-judge + deterministic-oracle evaluation frameworks. I understand non-determinism, hallucination detection, scoring rubrics, observability.
Security testing	VAPT engagement archive	I've run real VAPT engagements against multiple production-adjacent targets using the standard tool stack — Nmap, Nikto, ZAP, Burp — with authenticated DAST and remediation retests.

How to position this in interviews¶

Don't list six projects flat. Group them: "I have three tiers of work — classic automation, AI/LLM evaluation, and security testing. Let me start with one example from each, then go deeper where you want."
Lead with the AI evaluation work for any role that mentions AI / ML / LLM in the JD — that's the rarest skill on the market in 2026.
Lead with B2BProjectTest for traditional QA / SDET roles at enterprise / services companies — Java + Selenium + TestNG + REST Assured is still the lingua franca there.
Use VAPT as a differentiator for SDET-2 roles at security-conscious companies — fintech, healthtech, govt-adjacent. Most QA candidates don't have hands-on VAPT.

One-line summary to use in screens¶

"I've built and shipped six frameworks across three areas — Java + Selenium for traditional automation at Avysh, TypeScript + Playwright with API + UI hybrid for Questt's Morrie product, three generations of an LLM-evaluation framework for analytics chatbots (the latest with six evaluators including deterministic SQL ground-truth checks), and manual VAPT engagements against five production-adjacent targets using ZAP, Burp, Nmap, and Nikto. Each of them is in active use."

Project Deep Dive — Architecture, Framework & Interview Q&A¶

How to use this file¶

1. chat-eval / BKG Chat Evaluation¶

1.1 Identity¶

1.2 Tech stack (verified)¶

1.3 Architecture (real folder layout)¶

The 6 active evaluators (post-2026-04 consolidation)¶

Weight profiles (smart bit)¶

Three hard gates (any one fails the whole test)¶

Adapter pattern (the reason the framework is reusable)¶

1.4 Design decisions worth talking about¶

1.5 Honest tech debt¶

1.6 Interview Q&A¶

Q: Walk me through this project.¶

Q: Why have six evaluators instead of one?¶

Q: What's the BKG and how does the framework use it?¶

Q: How do you handle test cases where there's no expected answer?¶

Q: How do you keep LLM judge costs in check?¶

Q: How is this different from FMCG-chat-evaluation and hoad-ai-automation?¶

Q: What would you refactor next?¶

2. FMCG-chat-evaluation¶

2.1 Identity¶

2.2 What's different from chat-eval¶

2.3 Why fork instead of share?¶

2.4 Honest tech debt¶

2.5 Interview Q&A¶

Q: Why is FMCG-chat-evaluation a separate codebase rather than a config of chat-eval?¶

Q: What changes between FMCG and chat-eval at runtime?¶

Q: How do you test domain-specific knowledge in FMCG vs HOAD?¶

3. hoad-ai-automation¶

3.1 Identity¶

3.2 Tech stack (verified)¶

3.3 Architecture¶

Scoring formula (the earlier model)¶

React contract¶

evaluate_sql.py — the SQL-specific pipeline (this is interesting in interviews)¶

Verdict mapping¶

3.4 Design decisions worth talking about¶

3.5 Honest tech debt¶

3.6 Interview Q&A¶

Q: Walk me through this project.¶

Q: Walk me through evaluate_sql.py — what does it actually do?¶

Q: Why min-score across legs?¶

Q: Why is this the older version and what changed in chat-eval?¶

Q: How does the React frontend integrate with this framework?¶

Q: How is hallucination measured here vs in chat-eval?¶

4. Morrie Playwright Automation¶

4.1 Identity¶

4.2 Tech stack (verified)¶

4.3 Architecture¶

Two Playwright projects (split intentionally)¶

Global setup — the OTP flow (the part interviewers love)¶

Config-level choices¶

BaseApi (Axios wrapper) does three things¶

Jenkinsfile flow¶

4.4 Design decisions worth talking about¶

4.5 Honest tech debt¶

4.6 Interview Q&A¶

Q: Walk me through this project.¶

Q: Why use pressSequentially instead of fill on the email field?¶

Q: How does the framework handle rate limiting?¶

Q: Why are API tests in a separate project from web tests?¶

Q: What does your Jenkins pipeline do?¶

Q: How are credentials managed?¶

Q: What would you refactor next?¶

5. VAPT — Security Testing Engagement Artifacts¶

5.1 Identity¶

5.2 What's actually in there¶

5.3 What you actually did during these engagements¶

5.4 Targets covered (from the filenames)¶

5.5 What to claim and what NOT to claim in interviews¶

5.6 Interview Q&A¶

Q: Walk me through your VAPT work.¶

Q: How did you do authenticated DAST in ZAP?¶

Q: What's the difference between Nmap top1000 and top5000 in your workflow?¶

Q: How do you handle false positives in ZAP?¶

Q: Show me a finding you'd consider serious.¶

6. B2BProjectTest (Avysh B2B Java/Selenium Framework)¶

6.1 Identity¶

6.2 Tech stack (verified from pom.xml)¶