Frameworks Per Project — Detailed (for interviews)¶
Verified by scanning each repo's actual manifests (
pom.xml,package.json,requirements.txt,testng.xml,playwright.config.ts, workflows). Use this to answer "what's your tech stack on that project, and why?" For each project: the stack, the "why these frameworks" talking points, and an honest "what I'd improve."⚠️ Accuracy corrections vs older notes are at the bottom — read them so you don't over-claim (e.g. RAGAS, Jenkins, Selenium version).
1. LLM Evaluation Framework — bkg-chat-eval / FMCG / HoAD (3 repos, one architecture)¶
On the resume these are merged into one "AI/LLM eval framework reused across telecom, FMCG, retail." That's fair — they share the design.
One-line pitch: "A Python LLM-evaluation framework that certifies chatbot answers against a ground-truth oracle — it re-runs the business logic in a read-only database and uses LLM-as-judge across quality dimensions, gated in CI."
Stack (detailed):
| Layer | What I used |
|---|---|
| Language | Python 3.10/3.11 |
| Test runner | pytest 7.4.3 + pytest-asyncio (async mode) + pytest-timeout |
| Config/validation | Pydantic 2.5 + pydantic-settings (typed config from env) |
| LLM judges | OpenAI (gpt-4o-mini, strict JSON-schema output) + Anthropic (claude-haiku-4-5, forced tool-use for structured output) + Azure OpenAI optional; temperature 0 |
| Embeddings | OpenAI text-embedding-3-small or sentence-transformers (all-MiniLM-L6-v2), cached to a pickle |
| Retrieval fusion | Reciprocal Rank Fusion (RRF, k=60) — FMCG repo only — fusing keyword + dense semantic ranking |
| Oracle (the key idea) | BKG-Truth agent re-executes the business KPI/decision Python functions against a live read-only PostgreSQL (psycopg2); SQL claims validated with sqlglot |
| Channels tested | REST API (requests/httpx), WebSocket chat (websocket-client), Playwright (Python) UI |
| Caching | SQLite (judge cache 7-day TTL, regression DB, agent cache) |
| Observability | Langfuse (writes eval traces to cloud, reads chatbot traces) |
| Reporting | Self-contained Jinja2 HTML report + Excel + JSON + regression trend charts |
| CI | GitHub Actions (eval_ci.yml: lint → mypy → pytest -m "not ui and not slow") |
| Quality tooling | black, flake8, mypy, isort, pre-commit (HoAD adds detect-secrets) |
Why these frameworks (talking points): - pytest + asyncio because the chatbot streams over SSE/WebSocket — async lets me test streaming responses without blocking. - Pydantic for typed, validated config so a bad env var fails fast, not mid-run. - LLM-as-judge with temperature 0 + strict JSON schema for repeatable, parseable scores; multi-provider so I'm not locked to one model and can cross-check. - The BKG oracle is the differentiator — instead of trusting an LLM judge alone, I re-run the actual business logic against the DB to get ground truth, then the judge only grades against that. That's how I avoid "an LLM grading an LLM with no anchor." - SQLite caching to keep eval cost/latency down (judge calls are expensive) — a 7-day TTL so re-runs are cheap. - Langfuse for tracing so I can debug which step of the chatbot produced a bad answer. - CI gate runs the fast subset on every push; UI/slow tests excluded to keep it quick.
What I'd improve (honest): the three repos duplicate ~80% of code — I'd extract a shared core package. And I'd scrub the committed .env/DB password before any public sharing.
2. Morrie_automation — Playwright + TypeScript (questt.ai AI-agent platform)¶
One-line pitch: "A hybrid UI+API automation framework in Playwright + TypeScript using the Page Object Model and custom fixtures, with an Axios API layer that handles rate limits — run fully in parallel with HTML/JUnit reports."
Stack (detailed):
| Layer | What I used |
|---|---|
| Language | TypeScript 5.2 |
| Test framework | Playwright Test (@playwright/test ^1.35) |
| API client | Axios 1.4 — a BaseApi with 700ms rate limiting + 429 Retry-After handling, subclassed by Auth/User/Chats/Agents |
| Design | Page Object Model (Login, Dashboard, Agent, PageBuilder) + 8 custom fixtures |
| Auth | global-setup.ts logs in once — API token + UI cookie (storageState reused across tests) |
| Config | Two projects: chromium (uses saved storageState) and api; fullyParallel: true; CI 2 workers, 1 retry |
| Evidence | screenshot + video only-on-failure, trace on-first-retry (Trace Viewer) |
| Reporting | Playwright HTML + JSON + JUnit XML |
| CI | GitHub Actions (checkout → Node LTS → npm ci → install browsers → build) |
| Data | src/data/ JSON (agents/chats/testUsers) — data-driven |
Why these frameworks (talking points):
- Playwright over Selenium for this greenfield: built-in auto-waiting (less flakiness), parallel by default, the Trace Viewer for debugging, and network interception — and one API can drive both browser and API tests.
- POM + fixtures so tests read like English and locators live in one place; fixtures give clean, reusable setup (a logged-in page, an API client) instead of repeated beforeEach.
- The Axios BaseApi with rate limiting is the standout — the AI platform's endpoints rate-limit, so I built a 700ms throttle + Retry-After retry so setup doesn't trip limits during parallel runs. I reuse the same API layer to seed state fast, then verify through the UI.
- storageState so I log in once in global-setup and skip the login flow in every test — big speed win.
What I'd improve: the lastRequestTime throttle is a single global lock; for high-concurrency I'd move to a per-host token bucket.
3. B2BProjectTest — Java / Selenium / TestNG / REST Assured (Avysh B2B e-commerce)¶
One-line pitch: "A Java UI+API automation framework — Selenium 4 + TestNG + REST Assured, Page Object Model with a Page/Helper/Test three-tier design, ThreadLocal parallel execution, auto-retry, ExtentReports, and screenshot-on-failure."
Stack (detailed):
| Layer | What I used |
|---|---|
| Language / build | Java 1.8, Maven (surefire, compiler plugins) |
| Test framework | TestNG 7.0.0 (suite XMLs, groups, listeners) |
| UI | Selenium 4.0.0-alpha-5, WebDriverManager 4.2.2, ngwebdriver (Angular sync) |
| API | REST Assured 4.4.0 + json-path + gson |
| Assertions | AssertJ (SoftAssertions) + Hamcrest |
| Parallel | ThreadLocal<RemoteWebDriver>, suites parallel="classes" |
| Retry | Custom RetryAnalyzer (retryLimit=1) + IAnnotationTransformer to auto-apply it |
| Reporting | ExtentReports 2.41.1 via a custom IReporter listener; screenshot-on-failure via a CustomException (TakesScreenshot) |
| Design | POM — 44 page classes + 24 helper (business-workflow) classes + OrdersAPI; 52 @Test across 25 classes, 7 modules |
| Data | data-driven via JSON files (custom ReadTestData) |
| CI | none in the repo |
Why these frameworks (talking points):
- TestNG over JUnit for its suite XML control, groups (smoke/regression), dependsOnMethods, data providers, and listeners — better fit for a large UI suite.
- The Page + Helper + Test three-tier design so tests read like a user journey (regionsHelper.createRegion(...)), page structure changes ripple only to the Page class, and workflows are reused across tests.
- ThreadLocal driver so parallel="classes" gives each thread its own browser with no cross-talk.
- Auto-retry via IAnnotationTransformer so every test gets one retry on flakiness without per-test opt-in — one retry only, so a real bug (fails twice) isn't masked.
- XPath heavily because the app is built on Webix (custom webix_tm_id/view_id attributes) needing parent/child navigation.
What I'd improve (honest, and a good signal): it's on Selenium 4.0.0-alpha and legacy ExtentReports 2.x — I'd upgrade both, replace System.out.println with a logging framework, add a BaseTest, move the hardcoded credentials in config.properties to env/vault, and wire it into CI. (Saying this shows a quality/maintenance mindset.)
4. RestAssured_API — REST Assured + TestNG + Maven (questt.com APIs)¶
One-line pitch: "A Java API-automation framework — REST Assured 5.1 + TestNG + Maven — with a helper/service layer, GET/POST/PATCH request wrappers, token/OTP auth flows, and JSON-schema validation on responses. 62 tests across API v1–v4."
Stack (detailed):
| Layer | What I used |
|---|---|
| Language / build | Java 1.7, Maven |
| Test framework | TestNG 7.5 |
| HTTP/assertions | REST Assured 5.1.1 + Hamcrest matchers |
| JSON | Jackson 2.13 (databind) + gson + org.json |
| Schema validation | json-schema-validator 4.3.1 — 21+ JSON schema files, matchesJsonSchema() |
| Data | OpenCSV + Apache POI (Excel) + JSON test data |
| Design | helper/service layer (Login/Profile/Challenges/StudyPlan/AccessToken…) + requests/GET|POST|PATCH.java wrappers; 62 @Test across 15 classes |
| Auth | token- and OTP-based flows |
| Reporting | TestNG/Surefire native (JUnit XML) |
| CI | none |
Why these frameworks (talking points): - REST Assured for its given/when/then BDD-style syntax and built-in JSON parsing + Hamcrest assertions — API tests read cleanly. - JSON-schema validation is the standout — I validate every response against a committed schema, so a backend rename or type change fails the test immediately, not silently later. That's contract-drift protection within one repo. - Helper/service layer + request wrappers to keep tests DRY — auth, base URI, and common calls live once. - Data-driven via CSV/Excel/JSON so non-engineers can extend cases.
What I'd improve: it's on Java 7 with assertions sometimes inside helpers (breaks negative-test reuse) and a hardcoded API key — I'd move to instance-scoped specs, return responses from helpers, and pull secrets from env.
5. VAPT — Security testing (questt.ai APIs)¶
One-line pitch: "Hands-on VAPT (Vulnerability Assessment & Penetration Testing) across multiple production APIs — DAST + recon with OWASP ZAP, Burp Suite, Nmap, and Nikto — over retest cycles, with reports per target."
Stack / tools (detailed):
| Tool | Version | Use |
|---|---|---|
| OWASP ZAP | 2.16.1 | DAST — automated web/API vuln scanning |
| Burp Suite | 2025.10.4 | Intercepting proxy, manual request tampering |
| Nmap | 7.98 | Port/service recon; -A -sV, NSE ssl-enum-ciphers/ssl-cert |
| Nikto | — | Web-server misconfiguration scanning |
Why / talking points: - ZAP + Burp are the standard DAST + manual pair — ZAP for automated coverage, Burp to hand-craft and replay attacks the scanner can't. - Nmap + Nikto for the recon layer — open ports, TLS ciphers, server fingerprinting (found nginx 1.18 / Varnish / TLS 1.2–1.3 on GCP). - Retest cycles (scans dated Dec 2025 → Jan 2026) show I re-verified fixes, not just a one-off scan. - Ties directly to the AI red-teaming side of AI-testing roles — same adversarial mindset applied to LLMs (prompt injection, jailbreaks).
What I'd improve: add an OWASP Top 10 mapping and a remediation-tracking sheet so findings link to fixes and status. (Note: the Burp exports embed base64 auth tokens — treat the folder as sensitive.)
6. bce-ai-testing/practice-repo — Python LLM-eval + agent-testing lab (built this session)¶
One-line pitch: "A runnable, offline-first LLM-evaluation lab — provider-agnostic client (mock/Gemini/OpenAI), LLM-as-judge (pointwise + pairwise with position-swap), RAG metrics (faithfulness/relevancy/precision/recall), Cohen's kappa judge calibration, a tool-using agent with guardrails, and pytest release gates."
Stack: Python, pytest ≥7.4 (43 tests); optional real-model deps google-genai (Gemini 2.5-flash), openai, ragas, langchain-openai. Golden set JSON; stdout gate report (faithfulness ≥0.70, tolerance 0.05, pass-rate ≥90%).
Talking point: "It mirrors the real eval frameworks but runs offline so anyone can pytest it — it's how I show I can build eval + agent-testing from scratch." (This is your BCE-interview centrepiece — see ../../interview-prep/bce-ai-testing/practice-repo/.)
7. cicd-cloud-lab — CI/CD + cloud lab (built this session)¶
One-line pitch: "A hands-on CI/CD lab — a Flask app, pytest API + smoke tests, a Docker image (gunicorn), a GitHub Actions pipeline (test → artifact → gated image build), and a CD template that deploys to AWS and smoke-tests the live URL."
Stack: Python 3.12, Flask ≥3, gunicorn, pytest; Docker + docker-compose; GitHub Actions (ci.yml + cd.yml). See ../../cicd-cloud-lab/ and the guide ../21_CICD_Cloud_DevOps_for_QA.md.
Cross-project cheat sheet (memorize this table)¶
| Project | Type | Language | Framework | Standout thing to mention |
|---|---|---|---|---|
| LLM Eval (bkg/FMCG/HoAD) | LLM-eval | Python 3.10/3.11 | pytest + asyncio, Playwright | BKG oracle re-runs business logic in DB; LLM-as-judge temp 0; RRF (FMCG) |
| Morrie | UI+API | TypeScript 5.2 | Playwright Test 1.35 | Axios BaseApi with rate-limit + 429 retry; POM + fixtures; storageState |
| B2BProjectTest | UI+API | Java 8 | Selenium 4 + TestNG 7 + REST Assured | Page/Helper/Test 3-tier; ThreadLocal parallel; auto-retry via IAnnotationTransformer |
| RestAssured_API | API | Java 7 | REST Assured 5.1 + TestNG | JSON-schema validation (21+ schemas) = contract-drift protection |
| VAPT | Security | — | ZAP / Burp / Nmap / Nikto | DAST + recon, retest cycles; red-teaming mindset |
| bce practice-repo | LLM-eval lab | Python | pytest | LLM-judge + RAG metrics + agent guardrails + kappa, offline |
| cicd-cloud-lab | CI/CD lab | Python | Flask + pytest + Docker + GH Actions | test-gated pipeline + deploy-then-smoke-test |
⚠️ Accuracy corrections — don't over-claim these¶
- RAGAS / DeepEval: your real Questt eval repos do not use RAGAS or DeepEval — the judge/metrics are custom-built. RAGAS appears only as a reference in the practice-repo. So say "I built the eval metrics myself; RAGAS is the open-source equivalent I know" — don't claim you used RAGAS in production.
- RRF (Reciprocal Rank Fusion): only FMCG-chat-evaluation has an explicit RRF (k=60). bkg and HoAD use cosine + keyword fusion but not a literal RRF — don't claim RRF on those.
- Morrie CI: the repo copy uses GitHub Actions, not Jenkins. If you used Jenkins on the company's pipeline, say that explicitly — but the code shows GH Actions, so don't claim Jenkins for this repo.
- B2BProjectTest versions: it's Java 1.8, Selenium 4.0.0-alpha-5 (a pre-release alpha), TestNG 7.0.0, ExtentReports 2.41.1 (legacy 2.x), REST Assured 4.4.0. Get the version right or frame it as "Selenium 4 (early build) — I'd upgrade it."
- RestAssured_API: Java 1.7, REST Assured 5.1.1, TestNG 7.5 (newer than the B2B repo — don't mix them up).
🔒 Do this before any interview / public share (and it's a great "quality mindset" talking point)¶
Several repos have committed secrets — scrub them: bkg-chat-eval .env (OpenAI/Anthropic/Langfuse keys + Postgres password), FMCG .env + hardcoded localdev123, B2BProjectTest & RestAssured_API config.properties API keys, and the VAPT Burp XML (base64 auth tokens). Rotate anything real. If asked about secrets management, this is your story: "I found committed credentials in older repos, so I moved to env vars / a vault and added a detect-secrets pre-commit hook (which I already use in the HoAD repo)."