12 — Latest LLM / GenAI Interview Questions (2025–2026)¶

Researched from online sources (mid-2026): real reported interviews where they exist, plus current curated banks and vendor guides. This file focuses on what's new since the rest of the kit was written, and flags where each topic is already answered in files 01–11.

Source honesty: 🟢 = aggregated from real reported interviews (names companies) · 🟡 = curated bank / vendor guide (credible, some SEO/product bias) · 🔵 = new-topic reference. The 2026 web is full of recycled "questions 2026" lists, so a question on 10 sites ≠ asked 10× in real loops. The 🟢 items are the strongest signal.

The 10 most-repeated questions across sources (drill these first)¶

Explain the transformer / self-attention. (also a live-coding ask at OpenAI/Anthropic) → new, see §A below.
Fine-tuning vs RAG — when each? → 01 §6 ✅
Design a RAG customer-support chatbot AND how you'd evaluate it. 🟢 → 05 Q1 ✅
How do you detect & mitigate hallucinations in production? 🟢 → 01 §5, 02 ✅
How do you handle non-determinism when testing LLMs? — the signature QA question → 01 §2, 02 Part A ✅
LLM-judge biases + how do you validate the judge? → 02 Part B ✅
Why do BLEU/ROUGE fail for LLMs, and what do you use instead? → 02 Part A ✅
1M queries/day — optimize cost/latency. 🟢 → new, see §F below.
Agent vs LLM chain — what's the difference? 🟢 → 03 §8, 05 Q3 ✅
What is MCP / tool poisoning? — the standout NEW topic → new, see §G below.

Takeaway: your kit already covers 6 of the top 10 in depth (evaluation is your home turf — most candidates are weak there, so lean on it). The gaps are the new topics below.

§A. LLM fundamentals — the deeper-than-the-kit asks¶

The kit covers tokenization, temperature, context window, grounding, hallucination, fine-tuning. Add these, which now come up:

Question	Simple answer to have ready
Explain self-attention / the transformer.	Each token looks at every other token and decides who to "pay attention" to. Score = softmax(Q·Kᵀ/√d)·V. Multi-head = several attention views at once. Why it beat RNNs: processes all tokens in parallel.
What are positional encodings? Which is modern?	Transformers don't know word order by default, so you add position info. RoPE (rotary) is the current default (saying "sinusoidal" only = slightly dated).
What is the KV cache?	The model caches past tokens' keys/values so generating each new token is fast (O(n) not O(n²)). It's what makes streaming/long outputs affordable.
LoRA / PEFT / QLoRA?	Cheap fine-tuning: freeze the big model, train a tiny add-on layer (low-rank). QLoRA = same but on a 4-bit compressed base. Adapters are small and swappable.
Quantization & distillation?	Quantization = store weights in fewer bits (INT8/INT4) to cut memory/latency. Distillation = train a small "student" model to copy a big "teacher." Both trade a little quality for speed/cost.
Catastrophic forgetting?	Fine-tuning on new data can erase old skills. Prevent with PEFT/LoRA, mixing in original data, or rehearsal.
Faithfulness vs factuality (important distinction).	Faithfulness = the answer matches the provided source. Factuality = the answer is true in the real world. RAG faithfulness can be perfect while the source itself is wrong.

Coding variants seen in real loops (OpenAI/Anthropic tier): implement multi-head attention, a transformer layer, a LoRA adapter, or top-k/top-p/beam decoding from memory. (Not likely for BCE, but good to know the shape.)

§B. RAG — newer angles beyond the kit¶

The kit covers RAGAS + retrieval-vs-generation. Add: - Sparse vs dense retrieval: sparse = keyword match (BM25); dense = embedding/meaning match. Hybrid search fuses both (via Reciprocal Rank Fusion) — wins when queries mix exact IDs/codes with meaning. - Re-ranking (two-stage retrieval): fast approximate search grabs ~50 candidates, then a slower cross-encoder re-ranker (e.g., Cohere Rerank) re-orders the top few. Big quality win. - ANN algorithm question: "HNSW, IVF, or brute force?" → HNSW is the usual answer (graph-based, fast, high recall). - Chunking trade-offs: small chunks = precise but lose context; big chunks = context but noisy. Know semantic chunking and late chunking (embed the whole doc first, then split — 🔵 newer). - Advanced RAG variants (name them): CRAG (corrective), Self-RAG (model critiques its own retrieval), Adaptive RAG, CAG (Cache-Augmented — preload static data into context, skip retrieval), GraphRAG and Agentic RAG (see §G).

§C. Agents & tool use — newer framing¶

Kit covers agent testing (03 §8, 05 Q3). Add the concepts interviewers now probe: - ReAct = Reason + Act loop: think → act (call a tool) → observe → repeat. But modern models (Claude tool use, GPT function calling) often decide tool calls natively — so "ReAct prompting is often unnecessary now" is a current-sounding point. - Tool use vs function calling: function calling = the structured JSON mechanism; tool use = the broader pattern. - Agent memory types: short-term (this conversation), long-term (across sessions), episodic, working. - Single vs multi-agent: default to single-agent; only go multi when roles are genuinely separable or work is parallel. (Saying "multi-agent by default" is a red flag.) - Tool hallucination (🔵 newer): the agent invents a tool or wrong arguments. Prevent with a short tool list, schema validation, and feeding tool errors back. - "Agent stuck in a reasoning loop in prod — debug it": pull the trace, check for an unclear completion condition, add an iteration cap.

§D. Prompt engineering¶

Chain-of-thought (CoT): "think step by step" — helps on reasoning/math; interviewers want when it DOESN'T help (simple tasks, adds cost/latency).
Few-shot vs zero-shot: add examples when zero-shot output is inconsistent in format/tone.
Structured/JSON output: use schema/function-calling + validation + retry (your 09 A1 code).
Prompt injection defense → already in 03 §1, §6 ✅.

§E. LLM evaluation & testing — YOUR home turf (drill hard)¶

This is the differentiator for a QA/AI-testing candidate, and most AI-engineer candidates are weak here. The best single artifact online is a "51 LLM Evals Interview Questions" handbook (vibeengines.com). The highest-value questions, mapped to your kit:

Question	Where it's covered
Evaluation vs testing for LLMs? (testing = pass/fail; eval = distribution of quality)	02 Part A ✅
Why do public benchmarks (MMLU) fail as your eval? (contamination, wrong distribution)	08, new nuance below
Golden set: how to build, how big? (~50–100 per slice, not total; sealed holdout)	02, 05 Q4 ✅
LLM-judge biases + validate the judge (kappa vs human-human agreement)	02 Part B ✅
Single vs pairwise vs reference-based judging	02 Part B ✅
Judge drift (vendor updates shift scores → pin versions, keep an anchor set)	02, 05 Q2 ✅
Non-determinism: what replaces exact assertions? (score 0–1 + threshold gate + N-run consensus; assert properties not exact strings)	01 §2, 02, practice-repo ✅
Regression: what actually regresses? (prompt, model, retrieval, tools, post-processing — test the whole pipeline)	new nuance below
CI eval tiers (Tier1 deterministic/security → Tier2 per-PR → Tier3 nightly → Tier4 release)	02 Part E, 11 ✅
Shadow eval (run new system on real traffic, log only)	new below
Evals vs observability ("is the output good?" vs "what happened?" — joined by a trace ID)	new below
Silent degradation in prod (rolling 7-day averages; treat provider model updates as a deploy event)	05 Q2 ✅

New nuances worth adding to your answers: - "Is your eval real or vibes-based?" — reportedly asked at OpenAI. A great answer names a golden set + LLM-judge validated by kappa + CI gate + production monitoring (your whole kit). - Shadow eval = deploy the new prompt/model silently alongside the old one, log its outputs on real traffic without showing users, compare. Safer than a straight A/B. - Evals vs observability = evals answer "is this output good?"; observability (tracing/spans, e.g. Langfuse/LangSmith) answers "what happened inside the request?" You need both, joined by a trace ID. - Name-and-compare the tooling (interviewers ask): a CI eval framework (DeepEval / Promptfoo / RAGAS) + an observability platform (Langfuse / LangSmith / Braintrust).

§F. LLM system design — scaling & cost (newer emphasis)¶

Kit has the eval-pipeline design (05). Add the scaling/cost design question, reported "across multiple companies":

"Your app gets 1M queries/day — optimize cost and latency." Answer toolkit: - Tokens, not users, are the unit you scale and pay for. - Model routing/tiering: cheap small model for easy intents, frontier model only for hard ones. - Caching: semantic cache (reuse answers to similar questions) + prompt caching + KV cache. - Batching + concurrency (GPU throughput is the real constraint) and streaming responses. - Pre-compute embeddings, warm inference pools, autoscaling, guardrails at scale. - The drill: "answer the same prompt three ways — cheapest, fastest, most accurate."

§G. Genuinely NEW topics (2025–2026) — not yet in the kit¶

These are the freshest and most likely to catch you out. Learn the one-liners.

MCP (Model Context Protocol) 🔵 — the standout new topic¶

What it is: an open standard (from Anthropic) that lets an AI app connect to tools/data through one common connector instead of custom-coding every integration. Solves the "N apps × M tools" wiring explosion. Think "USB-C for AI tools."
vs REST API: MCP is built for an LLM to discover tools at runtime (tools/list) and call them, not for humans to hardcode.
tools vs resources vs prompts: model-controlled actions / app-controlled data / user-controlled templates.
Security (heavily tested): tool poisoning = malicious instructions hidden in a tool's description/metadata; rug pull = a tool changes its behavior after you approved it. Defenses: pin/verify tool definitions, sandbox, least privilege.
QA angle to volunteer: "I'd red-team MCP tools for poisoning and rug-pulls, validate tool schemas, and enforce least privilege — it's the agent Excessive-Agency risk (03) at the connector layer."

Reasoning / "thinking" models 🔵¶

What: models that spend extra compute thinking before answering (o1/o3, DeepSeek-R1, Claude extended thinking). They emit internal reasoning tokens.
Test-time compute: the idea that you can improve answers by spending more compute at inference (explore, verify, self-correct) rather than only by training bigger.
Trade-off: better on hard reasoning/math, but slower and pricier — don't use for simple tasks.
QA angle: eval them on reasoning correctness and cost/latency; the "thinking" makes outputs even more non-deterministic.

GraphRAG & Agentic RAG 🔵¶

GraphRAG: retrieve over a knowledge graph (entities + relationships) instead of loose text chunks. Use it for multi-hop / relationship questions ("how is X connected to Y?"). Default to vector/hybrid; reach for GraphRAG when relationships matter.
Agentic RAG: retrieval becomes a tool the agent chooses to call, deciding when and what to fetch — not a fixed pipeline stage. This is the biggest RAG framing shift of 2024→2026.

LLMOps / observability 🔵¶

CI/CD for LLM apps: prompt versioning, model/judge version pinning, eval pipelines, tracing & spans, drift detection. "Design a CI/CD pipeline for an AI agent with a feedback loop" is a recurring advanced ask (your 05 + 11 cover the eval-gate half).

§H. What changed in interview loops (2025–2026)¶

Loop shape unchanged (screen → technical → system design → behavioral), but content shifted from classical ML to LLMs / RAG / agents / evals.
Behavioral is now AI-specific: "a time you reduced hallucinations/cost in prod," "how do you stay current in a field that changes weekly," "an AI safety/ethics concern you raised." (Anthropic leans hard on safety/values.)
Take-homes are real and heavy: build a RAG chatbot, build an agent, ship a production MVP with safety requirements.
Eval maturity is the differentiator — and it's exactly your edge. Connect every RAG/agent/system-design answer back to "…and here's how I'd test and evaluate it."

Study priority for you¶

Lean on §E (evaluation) — it's your moat; most candidates are weak, you're strong.
Close the NEW gaps in §G — MCP (esp. tool poisoning), reasoning models/test-time compute, GraphRAG vs Agentic RAG. These are the only truly unfamiliar bits.
Add the §A deepening (attention, RoPE, KV cache, LoRA, faithfulness-vs-factuality) so fundamentals follow-ups don't catch you.
Prep the §F cost/latency design answer — it's a common one the kit didn't have.

→ Back to README · pairs with 08-real-world-questions.md