Skip to content

Interview Prep — Spoken Answers for the Take-Home Defence

Simple, story-style answers with every technical term spelled out and explained. Say them in your own words — don't recite. ⚠ = the questions the interviewer will push hardest.

Quick glossary (read once before the interview)

  • Speech-to-text (STT): the component that converts the caller's voice into written words.
  • Text-to-speech (TTS): the synthesized voice that speaks the bot's words back to the caller.
  • Large language model (LLM): the AI model that reads the conversation and decides what the bot says and does.
  • p95 / p99 (percentiles): sort all response times fastest to slowest; p95 is the time under which 95 out of 100 requests finish, p99 the time under which 99 finish. They describe the slowest experiences, which averages hide.
  • Word error rate: the percentage of words a transcription got wrong compared to what was actually said.
  • Keypad tones (DTMF): the beep signals sent when a caller presses phone buttons.
  • Answering-machine detection: how the system decides whether a human or a voicemail answered the call.
  • Time-to-first-token: how quickly the model produces the first piece of its answer — the moment the bot can start speaking.
  • WebSocket: a network connection that stays open both ways, used to stream audio continuously.
  • SIP / SIPp: the signalling protocol that sets up phone calls, and its standard testing tool.
  • Continuous integration (CI): the automated build pipeline that runs checks on every change.
  • Personal information (PII): anything identifying — name, appointment details, account numbers.
  • Cohen's kappa: an agreement score between two reviewers that corrects for lucky matches; 1 is perfect agreement, 0 is chance-level.
  • promptfoo / pytest / k6: testing tools — promptfoo runs evaluation suites against AI systems, pytest is a general test framework, k6 is a load-testing tool.

Part A — the root-cause analysis

1. ⚠ What does latency_ms=5500 actually measure? When I did the math on the timestamps, 5500 milliseconds is exactly 9800 minus 4300 — and 4300 is the moment the customer started speaking. So this clock starts when the user begins talking, which means it includes the customer's own speaking time and the speech-to-text conversion, not just the language model's thinking time. That's a problem, because we're comparing that number against the provider's p99 of 5100 milliseconds — and the provider's number measures only the model call. Two different clocks being compared as if they were one. Either the pipeline genuinely streams partial transcripts into the model while the customer is still talking, or the timeout clock starts in the wrong place — and in that second case, a customer who simply gives a long answer eats up the model's time budget. I'd confirm with the platform team which one it is, because it changes where the timeout should be measured from.

2. Engineering can fix only one thing this sprint — which one? The configuration fix — repoint on_timeout away from the greeting node. It's a one-line change, shippable today, and it converts "the bot forgot my call" into "the bot paused awkwardly." For the other two defects I'd tell the client honestly: the latency trigger and the idle-timer logic are still there, so evening calls may still have pauses — but no customer will ever hear the greeting twice again. Symptom gone today, disease scheduled for treatment.

3. ⚠ Why is the latency row only Medium severity when it's the trigger for everything? Because of what happens when you remove each piece. Remove the latency problem and the loop is still sitting in the configuration, waiting for the next slow response — any stall of any kind triggers it. Remove the misrouted transition, and the latency becomes nothing worse than a pause. The loop rows own the customer damage, so they're High. And honestly, tail latency — those slowest one-percent-of-requests moments — is never "fixed," only managed with alerting, retries and pacing. That's exactly what Medium-severity ongoing work looks like.

4. What would have made this Critical instead of High? Something irreversible — the bot leaking one customer's details to another, charging wrong amounts, or every call failing with no workaround. Here, 94 percent of calls complete normally and no data is corrupted. A bad experience at scale is High; irreversible harm or a total outage is Critical.

5. ⚠ What actually ends the loop after two or three cycles? Honestly — the artifacts don't say, and I didn't want to invent an answer. My best guess: by the third greeting the customer stops responding out of confusion, so the idle path finally fires as a genuine disconnect — or the customer simply hangs up. Before closing the analysis I'd pull one looping call's event log all the way to the end and read the disconnect reason recorded there. I'd rather say "I don't know, and here's exactly how I'd find out" than decorate a report with a guess.

6. Argue against your own evening-clustering theory. My theory blames the model provider being overloaded in the evening. Counter-theory one: it's us, not them — our own dialer launches hundreds of calls at seven in the evening, and our own concurrency creates the queueing and the slowness. Counter-theory two is sneakier: evening customers are home and relaxed, so they give longer answers — and remember the timestamp anomaly: if the timeout clock starts when the customer starts speaking, then longer answers eat the model's budget all by themselves. The data settles it: if the incident rate follows our own call-volume curve rather than the provider's latency curve, the problem is self-inflicted.

7. Your idle-timer claim rests on one trimmed log. How confident are you? One log gives me a hypothesis, not a conclusion — though it's a strong hypothesis, because the idle check firing in the same millisecond as the model timeout is very hard to explain any other way. Before writing "root cause" in a document a client will read, I'd pull twenty or thirty of the incident calls and check that the same signature appears in each: model timeout and idle trigger at the same timestamp, then the transition to the greeting. If they all match, it's a fact. If some don't, the story changes — and I want to know that before the client does.

8. What could go wrong with a config hotfix shipped at six in the evening? Plenty — pointing the transition at the wrong node, a typo, a configuration cache that doesn't refresh. So the de-risking routine: first reproduce the fix in staging with a deliberately forced model timeout and watch the recovery behave; then release to a small canary slice of the evening traffic, watching a single metric — the count of calls where the greeting plays more than once, which should drop to zero; and keep the old configuration one click away for rollback. If the canary is clean for an hour, roll out fully. The protection against a config accident isn't hesitation — it's a rehearsed rollback.

9. The null timeout — whose bug is it, the campaign author's or the platform's? Both, in different ways. The author made a mistake; the platform made the mistake possible by accepting a null value. People will always mistype configurations — that's precisely why the schema must reject invalid values at save time. I'd fix the author's mistake today and the platform's validation gap this week, because the platform gap means other campaigns may be carrying the same landmine right now without anyone knowing.

10. Engineering says "latency is the provider's fault" and closes the ticket. I'd agree that the provider's speed is the provider's number — and then show the three parts that are ours: our pipeline timeout sits just 400 milliseconds above the provider's slowest-one-percent mark, our idle detector treats the provider's slowness as if the customer had gone silent, and our flow restarts the whole conversation on top of it. The provider being slow is weather. Our system turning weather into a crash is our bug. I'd reframe the ticket around those three items and keep it open.

Part B — testing the AI layer

11. ⚠ Your guardrail gate demands 100 percent, and this morning it's at 99.6. Then the release is blocked — that was the agreed meaning of the gate, and a policy control that "mostly" holds is a broken control. But blocking is step one, not the whole answer. I open the failing transcript, find the exact phrasing that talked the bot past its limit, and that phrasing becomes a permanent test case. And this situation is exactly why I argued the real limit must live in the backend: the language-model layer alone will never hold 100 percent forever, because it's probabilistic by nature. With a hard server-side cap in place, that 0.4 percent is a customer-experience bug. Without it, it's a money leak.

12. If the cap is enforced server-side anyway, why test the model at all? Because the server cap stops the money, not the promise. If the bot tells a customer "sure, thirty percent off" and then the backend refuses to apply it, the customer heard a promise — that's a complaint, an escalation, possibly a regulator letter. The hard control protects the business; the model tests protect the conversation. You need both layers, and you test them separately.

13. Production calls have no ground truth prepared. How do you measure hallucination there? I build the ground truth after the fact instead of before. Sample real calls, pull the exact account data that was passed to the bot during that call, and check every factual claim in the transcript against it — the same oracle method as in staging, just done as reconciliation instead of seeding. Around that, cheaper signals: customer complaints containing the words "the bot told me," and mismatches between what the transcript claims happened and what the backend actually recorded.

14. The judge is also an AI. Who judges the judge? Humans do, on a schedule — and that's the one loop I refuse to automate. Every week, humans re-score a fixed sample of calls the automated judge also scored, and we compute the agreement between them using Cohen's kappa — a score that corrects for lucky coincidental agreement. Above roughly 0.7, I trust the judge. Drifting toward 0.6, I stop trusting the dashboards and fix the scoring rubric before anything else — because from that moment, every automated quality number is just an unverified opinion.

15. ⚠ Why a fixed sample of thirty to fifty calls instead of a percentage? Because a percentage doesn't survive contact with scale. Two percent of thousands of calls a day is a full-time reviewing job that nobody will actually do — so it silently stops happening, and worse, nobody notices it stopped. A fixed thirty to fifty calls, spread across campaigns and hours of the day, is small enough to genuinely happen every single week. A check that happens beats a bigger check that doesn't. And its purpose is calibration — checking the judge — not coverage; coverage is the judge's own job.

16. Why does time-to-first-token matter so much in voice? Because on a phone call, silence is the product. The customer doesn't experience "total generation time" — they experience the pause before the bot starts speaking. If the model produces its first words in 800 milliseconds, the text-to-speech can start talking and the pause feels human — even if the rest of the answer takes four more seconds to stream out behind it. So for voice, I set budgets and alerts on time-to-first-token, not just on completion time.

17. Why set the latency alert at 80 percent of the timeout budget? Because I want the alarm before the crash, not during it. At 80 percent of the budget, nothing is broken yet — that's precisely the point: there's still time to act. Alert at 95 or 100 percent and you're merely being informed that customers are already failing. And because an early alarm fires more easily, I require two or three consecutive bad measurement windows before it wakes anyone up — early, but not jumpy.

18. The provider silently swaps the model behind the same interface. How do you find out? The same way you detect any drift: a fixed canary evaluation that runs daily against the endpoint — the same set of prompts, scored the same way, every day. If the pass rate or the answering style shifts with no change on our side, the model moved underneath us. Then I treat it as a dependency upgrade that happened without permission: run the full evaluation suite, compare against the baseline, and either accept the new behaviour or escalate to the provider. The trend line in the evaluation history is what turns "it feels different lately" into something provable.

Part C — infrastructure and the real-time pipeline

19. Have you actually streamed audio over a WebSocket in k6? Not binary audio frames — I've used k6 for ordinary load testing and I know its WebSocket support, but that specific piece is a design on paper, and that's exactly why my plan says to dry-run the script against a stub speech-to-text server before trusting it. I did precisely that with the promptfoo configuration in this submission — wrote it, then verified it against a stub server in both directions, passing and failing, before claiming it runs. Same discipline here.

20. ⚠ Zero failures at peak load, but 0.1 percent allowed at one-and-a-half times peak. Why is any failure acceptable? Because they're answering different questions. Peak is the promise — at expected concurrency, a dropped connection means a real customer we called got cut off mid-sentence; zero tolerance, because that's the service commitment. At one-and-a-half times peak we're deliberately over-stressing the system to discover its headroom and its breaking pattern — that's exploration, not a contract. If you demand zero at stress level too, the test goes permanently red, and a permanently red test is a test everyone learns to ignore.

21. Retry loops are legitimate cycles. How does your configuration linter tell good loops from bad? It doesn't guess intent — it demands that intent be declared. Legitimate loops carry a maximum-visits annotation in the configuration; the lint rule becomes "a cycle without a declared visit limit fails the build." A retry-twice-then-escalate-to-human loop passes, because it says what it is. A cycle straight back to the opening greeting with no limit — which is exactly the Part A bug — becomes impossible to write.

22. Why not run strict template checking in production too? Fail loud everywhere. Because failing loud in production means a live human hears the call die mid-sentence. In the staging environment I want every undefined template variable to crash noisily — that noise is free information. In production, the same bug should already be impossible through earlier layers: declared defaults for every variable, and the lint gate that blocks undeclared usage from ever deploying. The principle: fail loud where it's cheap, fail safe where a customer is on the line.

23. Word error rate — how do you compute it, and what result blocks the carrier migration? Word error rate compares a machine transcription against a human-verified reference transcript: count the substituted, inserted and deleted words, divide by the reference length. For the migration, run the identical set of recorded calls over the old carrier and the new one, and compare the two rates. I'd block go-live on a meaningful relative degradation — say, more than ten percent worse relative to the old carrier — or on any degradation of the critical entities: numbers, dates, yes-and-no answers. Those entities drive the conversation flow; transcribing the mood correctly while mishearing "Tuesday" is still a failed migration.

24. The client goes from two languages to eight. What happens to your test matrix? The full switching matrix dies — language pairs grow quadratically, so I stop pretending to test everything equally and go risk-based. The guardrail and factual-grounding suites run in all eight languages, because a policy leak in any language is a real leak. The deep language-switching matrix runs only for the highest-traffic pairs, and the remaining combinations get a thin smoke test. Coverage follows call volume and potential harm, not symmetry.

25. A customer talks over the bot while it's speaking. What should happen, and how do you test it? Expected behaviour — this is called barge-in: the bot stops talking, listens, and doesn't lose its place in the conversation. I test it by scripting interruptions at the nastiest possible moments — mid-way through offering appointment slots, mid-way through a compliance disclaimer — and asserting three things: the text-to-speech halts, the speech-to-text captures what the customer said, and the flow neither processes anything twice nor replays the interrupted prompt. Keypad presses during bot speech get the same treatment.

Part D — the automation plan

26. You estimated the first two rows at a week. Defend that. The configuration linter is a small Python script — parsing the templates, checking every referenced variable against the declared ones, validating the schema — a day or two of work including wiring it into the build pipeline. The regression harness is a mock language-model endpoint that I control: a stub that returns scripted answers, or an injected timeout, on command — plus a test driving a staging call through it and asserting the flow never re-enters the greeting node. I've built stubs exactly like this before, including for this very submission. A week is honest if the staging campaign exists — which is why staging sits first on my list of things I'd need.

27. Nightly AI test suites will produce flaky failures. What's your policy? A nightly suite that cries wolf is worse than no nightly suite. The policy: any non-deterministic test that fails without a corresponding change gets quarantined within a day — moved out of the release gate, ticketed, with a named owner. A red nightly at nine in the morning has an on-rotation owner and a one-hour triage deadline: real regression, flaky test, or environment problem. The release gate only ever contains tests we currently trust — and a quarantined test earns its way back with a stable week.

28. The finance team asks why you judge-score every single call. Defend the cost. I'd concede half the line: full scoring where a decision depends on it — releases, canaries, newly launched campaigns — and sampled scoring, maybe ten to twenty percent weighted toward recent changes, for steady-state production. What I won't concede is the weekly human calibration sample: it costs a few human-hours, and it's the only thing that makes any of the automated scores trustworthy. Cut the expensive layer if needed; never cut the small layer that verifies the rest.

29. What keeps your staging environment honest over time? Same artifact, two destinations. The campaign configuration deployed to production is byte-for-byte the same file deployed to staging — never a hand-edited copy. The build pipeline compares them on every release and fails on any drift. The data differs — synthetic customer accounts, a frozen snapshot of the documents — but the configuration never does. Most staging environments rot because someone edits them directly; making the comparison automatic is what prevents it.

30. You admitted you'd need to learn the telephony test tooling. What does week one look like? Days one and two: the documentation plus the simplest possible working thing — one scripted call against our staging phone trunk, playing a pre-recorded audio file. Rest of the week: sending keypad tones, a small concurrency ramp, and a conversation with whoever runs telephony here about what tooling already exists in-house — someone usually has half of this built. The goal by Friday is one automated scripted call running in the build pipeline, however ugly. That's how I close every tooling gap: smallest working thing first, then iterate.

Part E — the appointment-reminder use case

31. How do you test upset-customer detection without real angry customers? You can't schedule angry customers, so you build a labelled library instead: voice actors recording genuine-sounding frustration, anonymized real escalation calls if privacy policy allows it, and — importantly — the tricky negatives: people who are terse and abrupt but perfectly fine. Then you measure in both directions: recall — the angry customers we failed to escalate — and precision — the calm customers we escalated unnecessarily. During the pilot, deliberately bias toward over-escalating and log every trigger with its reason, so the threshold gets tuned on real call data instead of being argued about in a meeting room.

32. The customer accepts a slot, but it's taken by the time you book it. What exactly should happen? The sequence I'd assert, step by step: the booking call fails with "slot taken" — the bot does not claim success — it re-fetches the current availability — it apologises briefly and offers the fresh slots — and if the second attempt also fails, it promises a human callback and logs that promise where a human will actually see it. The one unforgivable outcome is the bot saying "you're confirmed" when the backend said no — which is exactly why the reconciliation check between transcript and backend exists.

33. How does transcript-versus-backend reconciliation actually work, mechanically? Every test call produces two records. First, the transcript's claimed outcome, extracted into a structured verdict — confirmed, rescheduled to slot X, cancelled, escalated. Second, the backend's actual state, fetched from the scheduling system afterward. A script compares the two for every call. Any mismatch is a defect, no matter how pleasant the conversation sounded — because a bot that says "done" without doing is the worst failure this product can have. In production, the same comparison runs on a sample of calls, plus on every call that ended in an escalation.

34. The client says no personal information may be left in voicemails. What changes? The voicemail message becomes deliberately generic — "please call us back regarding your upcoming appointment" — no name, no time, no clinic or company detail that identifies the purpose. The test plan gains a new negative suite: every call where answering-machine detection decided a machine picked up gets recorded, and the recording is checked to contain zero personal information. And the retry logic gets re-examined, because a voicemail no longer counts as having reached the customer.

35. The timeline slips. Which of your six user stories do you cut? Story three — cancel with a reason. Painful but least harmful, because cancellation can fall back to "let me transfer you to an agent," and the escalation path already exists as its own story. Confirming and rescheduling are the product's entire reason to exist, the second language is a launch commitment, escalation is the safety valve for everything else, and the backend-failure honesty story prevents real-world harm. Capturing cancellation reasons is valuable — and deferrable.

The live artifact drill

36. ⚠ Open your promptfoo configuration and make test six fail. I've actually run this exact experiment. Point the configuration at a stub server that replies "I will give you forty percent off" — the backstop check extracts the word "forty," converts it to the number forty, forty is greater than fifteen, the assertion returns false, the test fails. When I dry-ran the whole suite: with a compliant stub, twenty-four out of twenty-four passed — eight tests, three repetitions each; with the violating stub, twenty-four out of twenty-four failed. And the reason spelled-out numbers matter at all: voice bots often write numbers as words for the speech synthesis, and my first version of the check only caught digits — a review caught that gap, I fixed it, and then I verified the fix actually fires instead of assuming it does.

37. ⚠ Your check fails a bot that correctly refuses — "I can't do 25%." Defend that. It's deliberate, and it's documented in the configuration comments. On a voice call, "I can't do twenty-five percent" still puts the number twenty-five into the customer's ear and into the compliance transcript — I'd rather the bot refuse without repeating the disallowed number back. But I hold that as a policy choice, not a law of nature: if the product team decides natural-sounding refusals matter more, the loosening path is already named — scope the strict check to the tool-call channel, where there's no ambiguity, and let the scoring rubric judge the spoken reply. My job as the tester was to make that choice explicit and visible instead of accidental.

38. ⚠ You repeat each test three times. What's the statistical power of that? Low, and I know exactly how low: three runs catch a failure that occurs ten percent of the time only about twenty-seven percent of the time — one minus 0.9 cubed. So the repeats alone prove very little, and I wouldn't claim otherwise. The design is layered instead: the repeats catch the highly unstable cases immediately; the nightly schedule accumulates evidence — that same ten-percent leak reaches roughly ninety percent detection within a week of nights; and the deterministic checks catch every hard violation on every single run, no luck involved. Repeat-three is the cheap first net, not the whole fence.

39. What happens if the bot refuses the legitimately approved twenty percent in test eight? The scoring rubric fails it — on purpose, and the assertion text says so explicitly. A guardrail that always refuses passes every adversarial test you throw at it and quietly kills the product it's guarding. In this test, a supervisor approval is logged in the transcript, so twenty percent is legitimate and the bot must honour it. Over-blocking is a defect too — just a silent one that shows up in lost sales instead of bug reports. Every guardrail suite I write carries a positive control like this for exactly that reason.

40. Where does the rubric scoring get its grader, and what does it cost? The rubric assertions call a separate grading model, configured in the evaluation settings with its own access key — which is exactly why I excluded them from my stub dry-run, where no such key exists. At nightly scale the cost is small: eight tests, three repetitions, one short grading call each — a few dozen calls to an inexpensive grading model, single-digit dollars a night at worst. The deterministic checks are free; the grading model is the paid layer — and if cost ever becomes a concern, the paid layer is the one you sample rather than run in full.

The general questions

41. ⚠ What did you use AI assistance for, and how did you verify it? (make this your own truth — template:) I used AI the way I'd use a sharp colleague: drafting structure, tightening phrasing, and pressure-testing my reasoning by asking it to attack my own conclusions. The analysis itself is mine — the timestamp arithmetic, the three-defect split, the severity call. And I verified rather than trusted: the clearest example is the promptfoo configuration — instead of assuming a generated file runs, I stood up a stub server and ran the suite in both directions, passing and failing, and fixed a real gap in the process. That's also exactly how I treat model output professionally — my whole evaluation framework at work exists because you verify what a model says; you don't believe it.

42. Severity versus priority — what's the difference? Severity is impact; priority is order of work — and they genuinely diverge. From this very incident: the greeting loop is High severity and also top priority, because the fix is cheap and tonight's calling window is coming. The missing configuration validation is lower severity — it breaks nothing by itself today — but I'd keep its priority high anyway, because one hour of schema work prevents the entire class of future incidents. Impact tells you how bad; priority tells you when.

43. When do you stop testing an AI feature and ship it? When the measured behaviour is inside the agreed bounds and stable — not when it's perfect, because it never will be. Concretely: the guardrail suite at one hundred percent, the evaluation scores above their thresholds and trending flat or upward, latency and cost inside budget, the human review sample signed off, and monitoring plus a rollback path ready for whatever we missed anyway. The exit criterion for AI features isn't zero defects — it's known behaviour, bounded risk, and fast detection when reality disagrees.

44. What part of your submission are you least confident about? Two honest answers. First, hands-on telephony test tooling — the call-simulation and formal audio-quality side — which I flagged in the plan itself along with how I'd ramp up. Second, my claim about what a null timeout does at runtime: the same-millisecond firing in the log points strongly to "it triggers immediately on any stall," but it's the one inference in the analysis I couldn't confirm from the artifacts alone — which is why the document says "confirm with the platform team" rather than stating it as fact.

45. Day one, production access, one dashboard. What do you build? Per-campaign response latency — the p95 and p99 lines — plotted by hour of the day, with the pipeline's timeout budget drawn across it as a horizontal line. And next to it, one counter: calls where the greeting played more than once. That's this entire incident on a single screen. The first chart shows the trigger building up hours before it bites; the second catches the symptom on the very first affected call — instead of a week later, at six percent.