Skip to content

13 — Mock Interview (self-test)

~70 questions by topic. Cover the answer, say it aloud, then check 14-mock-interview-answers.md. ⭐ = high-probability.

A. Screening / behavioural

  1. ⭐ Walk me through your automation experience and why you want to test AI agents.
  2. ⭐ You've done deterministic automation. What changes when the system under test is non-deterministic?
  3. Tell me about a flaky test you root-caused. What was it and how did you fix it?
  4. Describe owning a suite end-to-end in CI. What did you gate on?
  5. How do you decide what to automate vs test manually?
  6. A dev says "the test is wrong, my code is fine." How do you handle it?

B. Pytest ⭐

  1. ⭐ Explain fixtures and the scopes. When would you use session vs function?
  2. What's conftest.py and why is it useful?
  3. ⭐ Difference between yield and return in a fixture?
  4. How does parametrize work? How do you parametrize a fixture?
  5. What are markers? How would you run only smoke tests in CI?
  6. ⭐ How do you mock an external dependency? Where do you patch — definition or use site?
  7. monkeypatch vs unittest.mock.patch?
  8. How do you test async agent code?
  9. How do you run tests in parallel, and what must be true for that to be safe?
  10. How do you handle a genuinely flaky test vs a badly-written one?
  11. pytest.approx — when and why?

C. Playwright / Angular ⭐

  1. ⭐ How does Playwright auto-waiting work? Why is it more stable than Selenium sleeps?
  2. What's your locator strategy for an Angular app? Why avoid CSS classes?
  3. ⭐ Web-first assertions vs plain asserts — what's the difference?
  4. ⭐ How do you test a streaming (token-by-token) agent response?
  5. How do you prove the response streamed incrementally, not dumped at once?
  6. ⭐ How would you test a human-in-the-loop approval queue?
  7. How do you test two approvers acting on the same item concurrently?
  8. How do you mock a backend/SSE response in Playwright?
  9. How do you handle auth once and reuse it across tests?
  10. How do you debug a Playwright failure in CI?

D. API / contract ⭐

  1. ⭐ Difference between contract testing and schema validation?
  2. How do you validate a response schema in Python?
  3. What is consumer-driven contract testing (Pact)? Why does it help independently-deployed services?
  4. How would you use an OpenAPI spec in testing?
  5. ⭐ Beyond happy path, what API cases do you cover?
  6. How do you test idempotency?
  7. Why validate error response bodies?
  8. ⭐ How do you validate an agent API response when the content is non-deterministic?

E. Python

  1. Mutable default argument trap — explain.
  2. Write a retry decorator with backoff.
  3. What's a context manager and when do you write one?
  4. asyncio.gather — what does it do?
  5. Write cosine similarity from scratch.
  6. Parse possibly-malformed JSON from an LLM.

F. SQL / NoSQL ⭐

  1. ⭐ How do you isolate test data between tests?
  2. Transaction rollback vs truncate vs containers — trade-offs?
  3. ⭐ How do you validate DB state after an agent action? What exactly do you assert?
  4. INNER vs LEFT join.
  5. How do you store money and why?
  6. Mongo vs Redis vs SQL — when each?
  7. What's eventual consistency and how does it affect your assertions?

G. Azure DevOps ⭐

  1. ⭐ Sketch a pipeline that runs Pytest as a quality gate.
  2. Stage vs job vs step?
  3. How does one stage gate another?
  4. How do you publish and surface test results?
  5. ⭐ How would you add an LLM eval as a quality gate?
  6. How do you handle secrets in a pipeline?

H. LLM non-determinism ⭐⭐

  1. ⭐⭐ Why are LLM outputs non-deterministic? Is temperature 0 deterministic?
  2. ⭐⭐ Why do exact-match assertions fail, and what replaces them?
  3. Explain semantic-similarity assertion and its limitation.
  4. What is LLM-as-Judge? What biases does it have and how do you mitigate them?
  5. ⭐ What is metamorphic/property testing for LLMs? Give an example.
  6. How do you test that an agent abstains instead of hallucinating?
  7. How do you set a pass criterion for a non-deterministic suite?

I. Agent graphs ⭐⭐

  1. ⭐⭐ Why isn't asserting the final answer enough for an agent? What else do you assert?
  2. ⭐ How do you deterministically test an agent's branching logic?
  3. How do you test tool-error handling and infinite loops?
  4. What agent-specific failure modes do you test for?
  5. How would you test that a high-value action can't execute without approval?

J. Langfuse / regression / Ragas / audit ⭐

  1. ⭐ What does a Langfuse trace contain? What's a span vs a generation vs a score?
  2. ⭐ How would you assert on a trace in CI?
  3. ⭐⭐ How do you detect quality degradation when a model version changes?
  4. Shadow vs canary vs A/B rollout?
  5. ⭐ Name the 4 Ragas metrics and which failure each catches.
  6. Which RAG metric matters most in finance and why?
  7. ⭐ How do you verify audit-log immutability and a correlation-ID chain?
  8. What does SOX require of the audit trail, and how do you test segregation of duties?

K2. Safety / red-team ⭐ (see file 16)

  1. ⭐ How do you test that a chatbot handles harmful content? What do you gate on?
  2. ⭐⭐ How do you test that a chatbot isn't revealing sensitive information / its system prompt?
  3. What's a canary token and how does it help leakage testing?
  4. ⭐ How do you test for jailbreaks in a chatbot or its tools? Name techniques and tools.
  5. ⭐ What's the highest-value safety test for a tool-calling agent? (indirect injection via tool/RAG output)
  6. ⭐ The model says "I'm ChatGPT, made by OpenAI." How does QA catch and prevent that?
  7. Why must you also test over-refusal, not just refusal?
  8. Does QA build the guardrails? (No — QA validates them; defense-in-depth verification.)

K3. Agents that take real-world actions ⭐ (see file 17)

  1. ⭐⭐ An agent can release payments / send emails / delete records. How is testing that different from testing a chatbot?
  2. ⭐ What is Excessive Agency (OWASP LLM06)? Name its three root causes.
  3. ⭐ How do you test an irreversible action is contained? (human gate, dry-run, reversibility)
  4. ⭐ How do you stop a retried step from executing a payment twice? (idempotency)
  5. How do you bound blast radius / runaway cost? (least privilege, spend/rate caps, kill switch)
  6. How do you test an acting agent safely without touching production? (sandbox + synthetic data + dry-run/shadow)
  7. What's the safe-rollout pipeline for an agent that takes actions?

K. System design

  1. ⭐ Design a test strategy for a new financial agent feature end-to-end.
  2. Design an eval/regression pipeline for the agent platform.
  3. How do you gate a model upgrade safely?