14 — Model Answers¶
Full answers to the highest-value questions from 13. These are written to survive follow-ups. Adapt the STAR stories to your real projects.
The two answers that decide the interview¶
⭐⭐ "Why are LLM outputs non-deterministic, and how does that change your testing?" (Q55/56)¶
"An LLM produces a probability distribution over the next token and samples from it — temperature and top-p control how much variation. So the same prompt can yield different, equally-valid wording. And even at temperature 0 you aren't fully deterministic: floating-point non-associativity on GPUs, batching, and silent provider model updates cause drift. So exact-match assertions produce false failures on correct answers — a flaky, untrustworthy suite.
I adapt the assertions. First I assert the deterministic invariants that still hold — response schema, that the right tool was called, confidence in range, no PII, length bounds. Then for the fuzzy content I use semantic similarity against a reference, or an LLM-as-Judge grading against a rubric, or metamorphic properties like 'a rephrased question gives the same answer' and 'an unanswerable question triggers abstention, not a fabrication.' And I set the pass criterion over a dataset — say ≥95% of golden cases pass — not on a single sample. Logic tests mock the model so they stay deterministic and run on every PR; quality is measured probabilistically as an eval gate."
Follow-up "is temperature 0 deterministic?" → "Closer, but not guaranteed — batching and FP effects still cause variation, and the provider can update the model under the same name. That's exactly why I pin and log the model version."
⭐⭐ "Why isn't the final answer enough for an agent?" (Q62)¶
"Because a correct-looking answer can come from a wrong path — the agent might skip the policy check, call the wrong tool, hallucinate a number instead of calling get_balance, or approve without authorization. In audit-critical finance the path is the product. So I treat the execution trace as the test surface: I assert on intermediate state, which tools were called with which arguments and in what order, handoffs between sub-agents, and that the graph terminated within a step budget. The killer pattern is mocking the tools and asserting the agent's branching — deterministic, and it catches the dangerous bugs like 'released funds when policy said no.' Then I validate the final answer with the probabilistic methods."
Behavioural¶
⭐ "Walk me through your background and why AI agent testing." (Q1)¶
"I've spent [X] years in QA automation across API and web — Selenium/TestNG, Playwright with TypeScript, RestAssured, plus VAPT and some LLM evaluation work. I've owned suites end-to-end and wired them into CI as gates. What draws me here is that AI agents break the core assumption my career was built on — deterministic expected outputs. Testing non-deterministic, multi-step agents in a compliance context is genuinely unsolved, and I've already been building that muscle: evaluation harnesses, semantic and judge-based assertions, and trace-based validation. This role lets me combine the automation rigor I have with the AI-evaluation discipline I'm deliberately growing."
"Non-deterministic — what changes for you?" (Q2)¶
Compress the Q55 answer: shift from assert == to rubric/threshold over a dataset; assert invariants + path; mock for logic, eval for quality.
"A flaky test you root-caused." (Q3, STAR)¶
Situation: a UI suite went red intermittently in CI. Task: stop the rerun-to-green culture. Action: I categorised flakes by cause — found the main one was an implicit timing assumption; replaced fixed sleeps with waits on the actual outcome, and fixed a shared-state fixture that leaked between parallel workers by moving it to function scope with proper teardown. Result: flake rate dropped from ~8% to near zero and the team trusted the gate again. Lesson: reruns hide real bugs; triage by cause — state, timing, order, or genuine externality.
Pytest¶
⭐ Fixtures & scopes (Q7)¶
"A fixture is reusable setup/teardown a test requests by name; Pytest injects it. Everything after yield is teardown. Scopes control lifetime: function (default, per test) for anything mutable so state doesn't leak; class/module for grouped setup; session for expensive read-only things like a DB container, browser, or auth token. The trade-off is speed vs isolation — wider scope is faster but risks cross-test contamination, so I keep mutable state at function scope."
⭐ Mocking / where to patch (Q12)¶
"I replace external dependencies — network, DB, the LLM — with controlled fakes using patch or pytest-mock's mocker, so logic tests are fast and deterministic. The key rule is patch where the object is used, not where it's defined — app.service.get_quote, not the module it came from — because the import already bound the name in the using module. I use spec= so the mock rejects calls the real object wouldn't accept, and I keep a small live suite behind a marker for realism."
Playwright¶
⭐ Streaming response (Q21)¶
"I test three things. One: first-token latency — the response element becomes non-empty within a budget. Two: it renders progressively — I capture the text early, wait for the completion signal, and assert it grew, proving incremental streaming rather than a single dump. Three: the final settled content is correct, which I assert after waiting on a completion signal the UI exposes — a data-streaming=false attribute or a disabled stop button — never a fixed sleep. For determinism I mock the SSE body with page.route so the test doesn't depend on a live model, and I also test the failure UX by returning a 500 or aborting mid-stream."
⭐ Approval queue (Q23)¶
"I test the state machine, not just the click. Seed a pending high-value action, assert its status, approve through the confirm dialog, then assert the UI transitions to Approved AND the backend state changed AND an audit event was emitted with the correct actor and correlation ID. Then the edge cases: reject/request-changes with reason capture, authorization (a non-approver shouldn't even see the button — I test with different storage-state sessions), and concurrency — two browser contexts on the same item, where the second approver must see it already actioned."
API¶
⭐ Contract vs schema (Q28)¶
"Schema validation asks 'does this one payload conform to a declared shape?' — I do it with Pydantic or JSON Schema on a response. Contract testing asks 'do two independently-deployed services still agree on the interface?' — it protects the boundary over time so a backend change doesn't silently break the Angular UI or a downstream service. I'd validate individual responses with Pydantic and protect the boundary with consumer-driven contracts via Pact, or by validating live responses against the OpenAPI spec with something like Schemathesis, run as a CI gate so a breaking change fails the provider build, not production."
⭐ Validating a non-deterministic agent API (Q35)¶
"I split it. The envelope is a hard schema contract I assert exactly — trace_id present, tool_calls array, confidence in [0,1], model_version equals the pinned one. The natural-language content is validated probabilistically — semantic similarity or a judge — never exact-match. That way the structural regression (a renamed field, a missing trace id) fails loudly while the fuzzy content is judged on a rubric."
SQL¶
⭐ DB state validation (Q44)¶
"After the API says an action succeeded, I prove the side effect landed. I assert the row transitioned — status and timestamps — and, crucially, that there's exactly one side effect, not zero or two, which catches double-writes from retries. I check foreign keys aren't orphaned, money is stored as DECIMAL, and the audit row was emitted with the right event type and correlation ID. For isolation I wrap each test in a transaction and roll back, or use an ephemeral container for true integration."
Azure DevOps¶
⭐ Eval as a quality gate (Q53)¶
"Same principle as a unit-test gate. I add a stage that runs the eval harness over a frozen golden set and computes an aggregate score; the step exits non-zero — failing the build — if the score is below an absolute threshold or regresses against the recorded baseline, with a per-category guard so one slice can't collapse under a healthy average. It publishes the scorecard as an artifact, and on a model-version change it also requires human sign-off via an environment approval before deploy."
Langfuse / regression / Ragas / audit¶
⭐ What a trace contains (Q67)¶
"A trace is one end-to-end run. Inside it are observations of three kinds: spans (a unit of work with duration, like a retrieval or a graph node), generations (a specific LLM call capturing model version, prompt, completion, tokens, cost, latency), and events (point markers). You attach scores — numeric/boolean/categorical evaluations — to a trace or observation, and datasets drive offline eval runs. So the trace gives me the execution path, the reasoning chain, confidence scores, cost and latency, and the model version — everything I want to assert on."
⭐⭐ Detecting model-version regression (Q69)¶
"Model version is an uncontrolled dependency, so I pin it and log it on every generation. The detection system is a frozen, version-controlled golden set; an offline eval harness that scores the agent over it — structural pass, judge/semantic/Ragas scores, latency and cost; and a baseline comparison gate that fails if the aggregate drops below an absolute floor or regresses versus the recorded baseline, with a per-category guard so an averaged score can't hide a domain that collapsed. I evaluate over the whole set and use a significance check so I'm not blocking on sampling noise. For silent provider changes I alert on unexpected model_version values and run continuous canary evals in production. Upgrades roll out shadow, then canary, with a config-flag rollback."
⭐ Ragas metrics (Q71)¶
"Faithfulness — is every claim supported by the retrieved context; catches hallucination. Answer relevancy — does the answer actually address the question; catches evasive/padded answers. Context precision — are the retrieved chunks relevant and well-ranked; catches noisy retrieval. Context recall — did we fetch everything needed versus a ground truth; catches missing retrieval. Faithfulness and answer relevancy judge the generation; context precision and recall judge the retrieval. In finance I gate faithfulness highest and add abstention tests, because a hallucinated financial figure is the worst failure."
⭐ Audit immutability + correlation chain (Q73)¶
"For immutability I verify tamper-evidence: the log is append-only or hash-chained, so each event stores the hash of the previous one — I assert the chain is intact, that sequence numbers have no gaps, that timestamps are ordered, and that an attempted update or delete is rejected; flip a byte and the hash check must fail. For the correlation chain I send a request with a correlation ID and assert the same ID appears end-to-end — response header, application logs, the agent's Langfuse trace, the DB rows, and the audit event — so any transaction is fully reconstructable. A broken chain is itself a defect in a SOX context."
System design — "Test strategy for a new financial agent feature" (Q75)¶
Structure the answer as layers: 1. Clarify — what action, is it money-moving, does it need approval, which tools/models, what's the SLA. 2. Deterministic core (per-PR gate): unit tests for each tool; component tests for node logic; graph tests with mocked tools asserting path/branching/handoffs/termination; API contract + schema; DB state + audit assertions; Playwright for the UI incl. streaming + approval queue. 3. Probabilistic layer (eval gate): golden set with judge/semantic/Ragas scoring; abstention & safety; thresholds + baseline regression. 4. Trace layer: Langfuse assertions on path, confidence, cost/latency in CI. 5. Compliance layer: audit completeness, correlation chain, immutability, segregation of duties. 6. Adversarial: prompt injection via tool output, unauthorized-action attempts, tool-abuse. 7. CI/CD: deterministic + contract on every PR; eval + trace gates before deploy; model-version change forces full eval + sign-off; production monitors feed drift detection. 8. Rollout: shadow → canary with metric watch and instant rollback.
"The through-line: deterministic tests gate every commit, probabilistic evals and trace assertions gate deployment, compliance is verified as a first-class requirement, and I treat the agent's trace and the audit log as primary test surfaces — not just the API response."