Skip to content

Senior Automation QA Engineer — AI Platform (Agent Development Squad)

A complete, deep-but-plain-English study kit built directly against the job description. It assumes you already have a strong QA/automation foundation (Selenium, Playwright, RestAssured, VAPT, some LLM-eval) and closes the specific gaps this JD tests: Pytest depth, Playwright-on-Angular with streaming, HTTPX contract/schema testing, SQL/NoSQL state validation, Azure DevOps gates, and the AI-agent-testing layer (non-determinism, agent graphs, Langfuse, model regression, Ragas, SOX audit trails).

Plain-English style: every technical term is glossed the first time it appears ("In plain words: …"). Each concept ends with a short "Interview line:" you can say almost verbatim. The practice-repo runs offline (no API keys needed) so you can make gates pass/fail on purpose.

What this role actually is (read first)

Strip the buzzwords and it is two jobs fused into one senior role:

  1. Classic senior automation (the "Primary Gate"). They will not hire you on AI buzzwords. First they verify you can build and own real suites: Pytest for Python backends and agent logic, Playwright for an Angular UI, HTTPX/Postman for API contract + schema validation, SQL/NoSQL for test data and DB-state assertions, all wired into Azure DevOps as automated quality gates. This is where most interview time goes.

  2. Testing AI agents (the "Required Awareness"). On top of the automation, you must demonstrate the single most important mental shift: moving from deterministic assert actual == expected to probabilistic, rubric-based validation of non-deterministic systems — and knowing how to assert on intermediate agent states (tool calls, handoffs), read Langfuse traces, and catch quality regressions when a model version changes.

The context is audit-critical financial workflows (SOX/GDPR). So "quality" here also means: immutable audit logs, correlation-ID chains, and compliant event emission — not just "does the feature work."

The one sentence that wins the interview: "I treat the agent's execution trace as a first-class test surface — I assert on the reasoning path and intermediate tool calls, not just the final response, and I gate on evaluation scores over a golden set rather than exact-match."

Likely interview structure

Round Focus Where to study
1. Screening / experience Your automation narrative, "why AI agent testing", fit README + 14 (behavioral)
2. Automation core (heaviest) Pytest, Playwright/Angular, API contract/schema, SQL 01–05, practice-repo
3. Python coding + CI/CD Live Python, fixtures, Azure DevOps gates 04, 06, practice-repo
4. AI/agent testing Non-determinism, agent graphs, Langfuse, regression, Ragas 07–11
5. Compliance + system design Audit-log integrity, design an agent test strategy 12, 08, 10

Because the JD says "interviews are taken by the experts", expect follow-up drilling: after any answer they will ask "why", "what breaks", "how would you test that". The model answers in 14 are written to survive two follow-ups deep.

Files in this kit

File What it covers
01-pytest-mastery.md Learn-from-scratch → interview-ready in one file. Assert, discovery, fixtures/yield/scopes with analogies, parametrize, markers, mocking/monkeypatch, async, plugins, structure, flaky-test control, a hands-on exercise, and a Rapid-fire recall to revise.
02-playwright-angular-streaming.md Learn-from-scratch → interview-ready in one file. Why timing breaks UI tests, auto-wait, locators, expect vs assert, page.route mocking, then the hard parts for this role: streaming/SSE + WebSocket and approval-queue flows, POM, Trace Viewer — with analogies, an exercise, and a Rapid-fire recall.
03-api-testing-contract-schema.md HTTPX & Postman/Newman; contract testing vs schema validation (Pydantic, JSON Schema, OpenAPI), status/headers/auth, negative & idempotency tests.
04-python-for-test-scripting.md The Python testers get grilled on: data structures, comprehensions, async/await, typing, decorators, context managers, retries/backoff, cosine similarity.
05-sql-nosql-test-data.md Test-data setup/teardown, transactions & rollback fixtures, DB state validation, joins/aggregations, MongoDB & Redis basics.
06-azure-devops-cicd.md Azure Pipelines YAML, stages/jobs, quality gates, PublishTestResults, parallel/matrix, secrets, environments & approvals.
07-llm-nondeterminism-testing.md The concept the whole job rests on — learn-from-scratch → interview-ready. Why LLM output is non-deterministic and how to adapt assertions: the five strategies (structural invariants, semantic similarity, LLM-as-Judge, snapshot, metamorphic), pass-rate thresholds, mock-vs-eval split — with intuition, an exercise, and a Rapid-fire recall.
08-agent-graph-testing.md Multi-step agent graphs (mental model), asserting intermediate states / tool calls / handoffs, tool mocking, loop & termination testing, determinism controls.
09-langfuse-traces.md What a trace / span / generation / observation / score is, how to validate reasoning chains, confidence scores and execution paths, eval pipelines, CI assertions on traces.
10-model-regression-detection.md Golden datasets, offline eval harness, detecting quality degradation on model-version change, drift, canary/shadow, statistical significance.
11-ragas-rag-eval.md RAG explained, Ragas metrics (faithfulness, answer/context relevancy, context precision/recall) with formulas and how to gate on them.
12-audit-sox-gdpr-observability.md Immutable log structure, correlation-ID chains, SOX-compliant event emission, GDPR data handling, Azure Monitor / Application Insights for debugging.
13-mock-interview.md ~70 drill questions by topic (self-test).
14-mock-interview-answers.md Full model answers (behavioral + technical), STAR stories.
15-real-world-challenges-and-solutions.md Real internet-sourced questions + 16 practical challenges QA hits testing chatbots/tools/agents, each with an easy solution. Cited sources.
16-safety-redteam-testing.md Red-team/safety: testing harmful-content handling, sensitive-info/system-prompt leakage (canary tokens, PII/secret scan), jailbreaks (DAN/encoding/multi-turn/indirect injection; Garak/Promptfoo/PyRIT), and wrong-identity disclosure ("I'm ChatGPT"). Runnable in the repo.
17-agent-action-harness.md Agents that take real-world actions (OWASP LLM06 Excessive Agency): irreversibility, blast radius, over-agency. The containment harness — least-privilege tools, scoped creds, human gate, dry-run/shadow, sandbox, spend caps, idempotency, kill switch, audit — and how to test each. Runnable in the repo.
practice-repo/ Runnable Python: pytest patterns, HTTPX contract tests, a fake agent graph + Langfuse-style trace assertions, semantic-assert helpers, Ragas-style metrics, all offline.

Suggested 6-day study plan

  • Day 1 — Automation core I. 01 Pytest (reads from scratch → interview-ready) + 04 Python. Run the practice-repo; make a fixture, a parametrize, a mock. Be able to explain fixture scopes and conftest from memory.
  • Day 2 — Automation core II. 02 Playwright/Angular (reads from scratch → interview-ready; nail streaming + approval queue) and 03 API contract vs schema. Know the difference cold.
  • Day 3 — Data + pipeline. 05 SQL/NoSQL state validation and 06 Azure DevOps gates. Write a YAML pipeline from memory.
  • Day 4 — The AI shift. 07 non-determinism (reads from scratch → interview-ready; the differentiator), 08 agent-graph testing, and 17 the action harness (agents that take real actions — OWASP LLM06). Practise the "trace as a test surface" and "contain the action" lines.
  • Day 5 — AI depth. 09 Langfuse, 10 model regression, 11 Ragas. Know what a trace contains and the 4 Ragas metrics.
  • Day 6 — Compliance + safety + mock. 12 audit/SOX and 16 safety/red-team (harms, leakage, jailbreaks, identity — run the repo's test_safety_redteam.py), then read 15 (real-world challenges), self-test with 13, check against 14.

The 10 things to be able to say without thinking

  1. Fixture scopes (function/class/module/session) and when to use each.
  2. Difference between contract testing (does the API honour the agreed shape/behaviour between two services) and schema validation (does this one payload match a JSON Schema/Pydantic model).
  3. How Playwright auto-waits and why that still isn't enough for Angular's async zones + streaming responses.
  4. Why LLM output is non-deterministic (sampling: temperature/top-p) and the 5 assertion strategies that replace exact-match.
  5. "Assert on intermediate states, tool calls, and handoffs, not just the final answer."
  6. What a Langfuse trace contains (spans, generations, inputs/outputs, latency, cost, scores) and how you'd assert on it in CI.
  7. How you detect model-version regression: frozen golden set + eval scores + threshold gate + statistical check.
  8. The 4 Ragas metrics and which failure each catches.
  9. Immutable audit log + correlation-ID chain + SOX event emission — how you'd verify each.
  10. An Azure DevOps quality gate: tests run in a stage, results published, deployment blocked on failure/threshold.