Interview Q&A — defending this project

Concise, honest answers to the questions a strong 2026 SDET interview will ask, each pointing at where it's demonstrated in the repo.

Why Playwright (not Selenium)?

Auto-waiting (kills a whole class of flakiness), true cross-browser (Chromium/ Firefox/WebKit) from one API, first-class tracing/video/screenshots, and network interception. See automation/playwright/ + ADR-001.

Why pytest for API/eval and not "everything in Playwright"?

Different layers answer different questions at different cost/stability. API tests are fast, deterministic, and cover logic/contracts; UI tests are reserved for the integrated journey. Pushing everything through the browser is slow and flaky.

Why separate API tests from UI tests?

An API failure and a UI failure have different root causes and different owners. Separation localises failures and keeps the fast feedback loop fast.

Why does RAG evaluation need more than exact string matching?

A correct answer can be phrased infinitely many ways; exact match fails correct answers and passes wrong-but-matching ones. We combine deterministic reference checks (fragments, citations, retrieval math) with an LLM judge for semantic dimensions. See ai_evaluation/.

What is LLM-as-a-Judge?

A model scores an answer against a rubric (correctness, faithfulness, relevance) and returns a structured verdict (score, pass, reason, violations). We parse strict JSON and validate it — never trust free text. ai_evaluation/judges/.

What are hallucination and faithfulness?

Faithfulness = every claim is supported by the retrieved context. A hallucination is a claim that isn't. We measure faithfulness per answer and derive a hallucination rate; abstaining on out-of-scope questions is not a hallucination.

How do you evaluate an AI agent?

By asserting on behaviour: which tools it calls, with what arguments, in what order — plus safety (it can't call forbidden tools) and injection resistance (malicious input doesn't change the tool plan). tests/unit/test_agent.py.

How does self-healing work, and why is it dangerous?

On a good run we record an element fingerprint; when a locator breaks we score candidates by similarity to it, apply confidence bands, and require uniqueness before healing — logging every decision. It's dangerous because naive healing can silently bind to the wrong element and hide a real regression; our mitigations are fingerprint similarity, a uniqueness gate, refusing below 0.70, and a full audit trail. ADR-004.

What is MCP and how does it help QA?

Model Context Protocol exposes typed, discoverable tools to AI clients. We expose read-only test/eval tools so an agent can investigate a failure (get_failed_tests → get_test_case → get_self_healing_events → get_application_logs) without the ability to mutate anything. ADR-005.

How do you test prompt injection (direct and indirect)?

Direct: a corpus of injections asserts no system-prompt leak and no compliance. Indirect: an uploaded malicious document is neutralised by a context-sanitisation guardrail (verified it can't plant a canary). automation/security/ + ADR-006.

How do you keep AI evaluation reliable (not flaky/expensive)?

Two tracks: deterministic plumbing (FakeLLM + local embeddings) on every PR, and probabilistic model-quality evaluation nightly with tolerance bands. Every report records dataset hash, model, and judge for reproducibility. ADR-003.

How do you implement quality gates?

quality/gates/gates.yaml defines thresholds per layer; quality.gates.run_gates aggregates all artifacts and exits non-zero on any breach — CI blocks the merge.

How do you measure AI application cost?

Every provider response carries tokens + estimated USD; the eval report and k6 custom metrics aggregate tokens/cost per request so cost is a first-class, gate-able metric.

How do you detect flaky tests without hiding them?

quality/flaky/ classifies a test from its history; a retry-pass is treated as flaky, never as a green. Retries are reported, not used to mask failures.

How does failure triage work?

quality/triage/ classifies a failure (PRODUCT_BUG, SELECTOR_FAILURE, …) from collected signals with a confidence and probable root cause — as a recommendation for a human, never auto-closing bugs or editing tests.

How is everything wired into CI/CD?

.github/workflows/ci.yml: lint → tests → contract → security → RAG eval → unified quality gate, publishing reports/traces/dashboards. Nightly adds full cross-browser UI, k6 load, and full RAG regression. Supply-chain: gitleaks, pip-audit, SBOM, Dependabot.