07 — Mock Interview Drill¶

60+ questions grouped by topic. Cover the right column, answer out loud, then check. The cue is a reminder — full plain-English answers to every numbered question are in 10-mock-interview-answers.md (deeper background in files 01–06). Aim to answer 90% cold before the interview.

How to drill: do one section per sitting; say answers out loud (or record). Anything you fumble, re-read the linked file, retry next day.

A. LLM fundamentals¶

#	Question	Answer cue
1	What is an LLM at its core?	Next-token predictor; probabilistic; no built-in truth
2	What is a token? Rule of thumb?	Sub-word chunk; ~4 chars / ¾ word
3	Why do tokens matter for testing?	Cost/limits in tokens; truncation; FR uses more tokens
4	Temperature vs top-p vs top-k?	Randomness knob; nucleus (cumulative p); top-k count
5	Best temperature for reproducible tests?	0 — but not truly deterministic
6	Why isn't temp 0 deterministic?	FP non-associativity, batching, backend (`system_fingerprint`)
7	What is the context window?	Max tokens in view: system+history+RAG+answer
8	"Lost in the middle"?	Models miss facts buried mid-context; needle-in-haystack test
9	What is grounding?	Tie output to a provided trusted source (RAG)
10	Hallucination mechanism?	Optimizes plausible not true; gaps→interpolation; no "I don't know"
11	4 ways to test hallucination?	Faithfulness, reference checks, adversarial/no-answer probes, numeric
12	RAG vs fine-tuning vs prompting?	Knowledge / behavior / cheapest-first
13	The 3 message roles?	System, user, assistant

B. Evaluation & LLM-as-Judge¶

#	Question	Answer cue
14	Why does exact-match testing fail for LLMs?	Many valid phrasings; non-deterministic
15	4 eval technique families?	Heuristics → overlap (BLEU/ROUGE) → semantic (BERTScore) → LLM-judge
16	BLEU/ROUGE weakness?	n-gram overlap, ignores meaning
17	What's an embedding?	Text→vector; similar meaning = close (cosine)
18	LLM-as-Judge — 3 modes?	Pointwise, pairwise, reference-based
19	When pairwise vs pointwise?	Pairwise=subjective/compare; pointwise=objective/absolute
20	3 judge biases + fixes?	Position→swap; verbosity→normalize; self-pref→diverse judge
21	How calibrate a judge?	vs human golden set; Cohen's kappa ≥0.6; feedback loop
22	Kappa scale milestones?	.41–.60 moderate, .61–.80 substantial, .81+ almost perfect
23	Cohen vs Krippendorff?	2 raters vs 3+/mixed/ordinal
24	What if humans don't agree on golden set?	Rubric too vague — fix rubric first
25	Why does a judge need recalibration?	Model drift over time

C. RAG & RAGAS¶

#	Question	Answer cue
26	What is RAG? (4 steps)	Chunk+embed → retrieve → augment → generate
27	Two ways RAG fails?	Retrieval (precision/recall) vs generation (faithfulness/relevancy)
28	What is RAGAS?	OSS RAG eval lib; LLM-judge; mostly reference-free
29	Faithfulness formula?	supported claims / total claims; reference-free
30	Response Relevancy — measures what?	On-topic/complete (NOT correctness); gen Qs→cosine
31	Context Precision?	Are relevant chunks ranked high (Precision@k)
32	Context Recall?	reference claims found in context / total; needs reference
33	Which 2 are reference-free?	Faithfulness, Response Relevancy
34	Retrieval vs generation metric grouping?	Precision+Recall / Faithfulness+Relevancy
35	Agent metrics in RAGAS?	Tool Call Accuracy, Topic Adherence, Agent Goal Accuracy
36	"Answer Relevancy" — gotcha?	Renamed Response Relevancy in current RAGAS
37	RAGAS limitations?	Non-determinism, imperfect human correlation, cost, bias

D. Red-teaming & safety¶

#	Question	Answer cue
38	Prompt injection root cause?	Instructions + data share one channel
39	Direct vs indirect injection?	User input vs hidden in retrieved/external content
40	Why is RAG exposed to indirect?	Retrieved text pasted into prompt, unsanitizable
41	Name a real indirect-injection incident	EchoLeak (M365 Copilot, zero-click)
42	Jailbreak vs injection?	Defeats safety training vs hijacks instruction-following
43	Name 4 jailbreak techniques	DAN/role-play, prefix injection, refusal suppression, many-shot, crescendo
44	OWASP LLM01 / 06 / 09 / 07?	Injection / Excessive Agency / Misinformation / System Prompt Leakage
45	NIST AI RMF 4 functions?	Govern, Map, Measure (TEVV), Manage
46	What is garak?	"nmap for LLMs"; NVIDIA vuln scanner
47	garak's 4 plugin types?	Generators, probes, detectors, buffs
48	3 garak probe families?	dan, encoding, promptinject/latentinjection, realtoxicityprompts
49	4 injection defenses?	Spotlighting, filtering, least privilege, dual-LLM/CaMeL
50	Testing bias/fairness?	Counterfactual swaps; demographic parity; EN+FR
51	Why test EN and FR separately?	Safety weaker in lower-resource lang
52	Stress-test an agent for?	Runaway actions, scope violations, tool accuracy, escalation

E. Vertex AI / Gemini¶

#	Question	Answer cue
53	What is Vertex AI? Gemini?	GCP managed ML platform; Google's model family
54	Current SDK name?	Google Gen AI SDK (`google-genai`); old `vertexai` deprecated Jun 2025
55	Gen AI Eval Service — 2 metric families?	Computation-based vs model-based (autorater)
56	What's an autorater?	Gemini model acting as LLM-judge
57	AutoraterConfig anti-bias options?	Response flipping, multi-sampling, tuned judge
58	How auth on Vertex?	ADC: `gcloud auth application-default login` / service account
59	Key generation params?	temperature, top_p, top_k, max_output_tokens, safety_settings

F. AI-in-QA tooling¶

#	Question	Answer cue
60	AI test generation approach?	Story/spec → draft cases; review before commit
61	Self-healing locators — how + risk?	Multi-signal ranker; risk = masks real regressions, log heals
62	Predictive test selection inputs?	git diff + defect history + code-to-test map; metric APFD
63	Failure clustering for triage?	Embed stack traces → cluster → triage 1 per cluster
64	NL test authoring reliability?	Constrain to vetted step library; human gate; EN/FR

G. System design / scenario¶

#	Question	Answer cue
65	Design a RAG certification pipeline	Corpus→retrieval+gen metrics→calibrate→red-team→gate→certificate→drift
66	Design drift monitoring	Sample prod, reference-free metrics, alert on band, re-certify, feed back
67	Certify an agent (Agentforce)?	Tool accuracy, topic adherence, goal, escalation, least-privilege red-team
68	Generate adversarial corpus from seeds?	Paraphrase+edge+adversarial; dedup by embedding; EN/FR; held-out
69	Passed evals, failed in prod — why?	Corpus gap / metric blind spot / judge drift / shift / non-determinism; add regression test

H. Behavioral¶

#	Question	Answer cue
70	Building QA in ambiguity (no textbook)?	STAR; from-scratch framework; quantified result
71	Convince devs an eval gate is mandatory?	Data not authority; fast+transparent; warn→block; partner early
72	Why AI testing / why this role?	Two halves of job; security ties to red-team; discipline being written now
73	How do you stay current?	Technology radar; papers; tools (RAGAS/garak/PyRIT)

Final-day routine¶

Recite the README's "10 things to know cold."
Drill sections B, C, D (the technical core) until smooth.
Run the practice repo once; explain each file out loud.
Rehearse 2 STAR stories (ambiguity + a bug you caught).
Prepare 3 questions to ask them (e.g., "What's the current Tier 2/3 certification bottleneck?", "How mature is the EN/FR eval coverage today?", "Vertex-only or multi-cloud?").