05 — System Design (the senior-level filter)¶

For a 5-9 year role, expect open-ended "design X" questions. The interviewer wants to see structured thinking, not a perfect answer. So lead with the skeleton below for any AI-eval design question, then borrow from the worked examples.

In plain words: AI-eval = checking whether an AI system gives good, safe answers. A "certificate" is a sign-off report saying it passed.

The universal answer skeleton (say it out loud first)¶

Remember: Clarify → Corpus → Metrics → Score → Red-team → Gate → Certify → Monitor.

Clarify — what kind of AI is it (RAG? agent? classifier?), how risky is it (the "risk tier"), which quality dimensions matter, English or French, and who reads the certificate?
Build the test corpus — your "golden set" of test cases: normal + edge + adversarial (tricky/attack) cases. Start from real questions ("seed intents"), grow them with an LLM, remove duplicates, and have humans label the ones that need a correct answer to compare against.
Define metrics + thresholds for each dimension (faithfulness, relevancy, retrieval recall, safety, latency…). In plain words: pick what to measure and the passing score.
Score — use plain computation where you can, plus an LLM-as-Judge (one AI grading another), checked against human grades using kappa.
Red-team — actively attack it: injection, jailbreak, scope/agency abuse, toxicity/bias (in EN + FR). Tool: garak.
Gate — wire it into CI/CD (the automated build pipeline); set thresholds + tolerance bands and fail the build if quality drops.
Certify — produce a report: scores, risks, and conditions.
Monitor — watch for drift in production, re-certify when needed, and feed real failures back into the corpus.

Q1. "Design an end-to-end AI Quality Certification pipeline for a new RAG chatbot before it ships."¶

In plain words: RAG = the bot looks up documents, then writes an answer from them.

Clarify: It's a RAG support bot, English + French, customer-facing → so it's high-risk tier. Dimensions that matter: faithfulness (no made-up answers), retrieval quality, answer relevancy, safety/toxicity, no leaking private data (PII), and latency (speed).

1. Corpus design. Start from seed intents — the top real questions users actually ask. Grow them with an LLM into a big, varied set, plus adversarial cases (questions the knowledge base can't answer, questions with a false assumption built in, and documents with hidden injected instructions). Cover EN & FR. Have humans label a golden subset with both the correct answer and the correct source text.

2. Retrieval eval. This checks the lookup step. Context Precision (did the relevant chunks rank near the top?) + Context Recall (did we fetch everything we needed? — this one needs the reference answer). Example threshold: recall ≥ 0.85.

3. Generation eval. This checks the writing step. Faithfulness (every claim is backed by the fetched text — this is your no-hallucination gate, ≥ 0.90) + Response Relevancy (on-topic and complete). These need no reference answer, so you can also use them live in production.

4. Judge calibration. The LLM-as-Judge doing the scoring above must itself be trusted. Check its grades against the human golden labels using Cohen's kappa ≥ 0.6 (a measure of agreement). Re-check whenever the models change. Remember: grade the grader.

5. Safety/red-team. Run garak (promptinject, latentinjection, dan, encoding, realtoxicityprompts) against the live endpoint; add counterfactual bias tests and PII-leak probes; do all of it in EN + FR. Example thresholds: jailbreak hit-rate < 2%, zero PII leaks.

6. CI/CD gate. Run it on every prompt, model, or index change. Use temperature=0 (most predictable output), sample multiple times, measure the pass-rate over N runs, and compare against the last known-good baseline (tolerance band). Below threshold → block the deploy. Run a fast subset per pull request, the full suite nightly. (Use Cloud Build or GitHub Actions.)

7. Certificate. Auto-generate an AI Quality Certificate: scores per dimension, the risks you found, and the conditions (e.g., "approved for EN; FR conditional pending a toxicity fix"). JD: "Produce clear AI Quality Certificates with scores, risks, and conditions."

8. Drift monitoring. Sample real production outputs, re-run the reference-free metrics (faithfulness, relevancy), and alert when they drift → trigger re-certification. Feed real production failures back into the corpus.

Q2. "Design drift monitoring for a model already in production." (JD #4)¶

In plain words: drift = the model quietly getting worse or facing new kinds of inputs over time.

Sample live traffic, spread evenly across intents and EN/FR (stratified).
Run reference-free metrics continuously (faithfulness, relevancy, safety) — these need no correct answer, so they work on live traffic.
Track the distribution over time, not a single score; alert when a metric's moving average drops past a tolerance band, or when the inputs themselves shift (new topics showing up).
On alert → trigger re-certification and produce a quality delta report comparing against the last certified version (JD: "Track model/version changes and generate quality delta reports").
Close the loop: add the production failures that caused the alert into the golden corpus, so the next certification covers them.
Watch for judge drift too — periodically re-check the judge against humans (kappa).

Q3. "How would you certify a Salesforce Agentforce / ServiceNow Now Assist agent?" (named in JD)¶

Agents don't just answer — they act (call tools, take multiple steps). So on top of the RAG metrics, add:

Tool Call Accuracy — right tool, right order, right arguments (a RAGAS metric).
Topic Adherence — stays in scope and refuses out-of-scope requests.
Agent Goal Accuracy — did it actually finish the user's goal, start to end?
Escalation behavior — does it hand off to a human when it should?
Excessive Agency (OWASP LLM06) red-team — try to push it into unauthorized or runaway actions; enforce least privilege (minimal permissions), action budgets (caps on what it can do), and a sandbox that fakes the tools so tests cause no real side effects.
Trajectory eval — grade the whole sequence of steps, not just the final answer. Remember: for agents, judge the journey, not only the destination.

Q4. "Generate a large, diverse, adversarial test corpus from a few seed intents." (JD #2)¶

Seed = a handful of real intents/questions to start from.
Expand with an LLM: reword them (different phrasing, formality, EN/FR), vary the difficulty, and add edge cases (ambiguous, multi-part, no-answer-exists, false-premise).
Adversarial layer: add injection/jailbreak variants, toxicity triggers, and PII-bait (inputs that try to lure out private data).
Ensure diversity — turn every generated case into an embedding (a numeric fingerprint of meaning) and drop near-duplicates by cosine similarity; check topic coverage so one intent isn't over-represented.
Avoid contamination — keep a private held-out set; never let it leak into any training or few-shot examples (otherwise the model has "seen the exam").
Human-validate a sample, especially anything used as a correct/gold answer.
Keep it living — keep folding in real production failures over time.

Q5. "A model passed all your evals but failed in production. What happened?"¶

A strong, structured answer walks through the likely causes:

Corpus gap — production hit an intent, edge case, or language your golden set didn't have → add it. This is exactly why you feed production failures back.
Metric blind spot — you measured faithfulness, but the failure was tone, latency, or a safety edge you never scored → add that dimension.
Judge miscalibration / drift — the judge said "good," but humans disagree → re-calibrate (kappa).
Distribution shift — real inputs drifted away from your test inputs → drift monitoring should have caught this; tighten it.
Non-determinism — it passed at temp=0 in testing but failed at the real production temperature → test at production settings with pass-rate thresholds.
Takeaway: evals lower risk, they don't erase it. The fix is to turn the production failure into a permanent regression test so it can never come back unnoticed.

Tips for the design round¶

Think out loud, structure first. State the skeleton, then go deep wherever they push.
Always split retrieval vs generation for RAG, and add a red-team + drift layer — these are the parts generic candidates forget.
Quantify — give example thresholds: kappa ≥ 0.6, pass-rate ≥ 9/10.
Name the EN/FR dimension without being asked — it's Bell-specific and shows you actually read the role.

→ Next: 06 — Model Answers to the ⭐ Questions