Skip to content

BCE Global Tech — "AI Testing" (QA AI Specialist) Interview Prep

A complete, beginner-friendly study kit for the AI Testing / QA AI Specialist role at BCE Global Tech (Bell Canada's Global Quality Engineering team). It assumes you have not used these AI concepts before and teaches each one from scratch, then gives you model interview answers and a runnable practice project.

Everything here was written against the actual job description and grounded in current (early-2026) documentation for RAGAS, NVIDIA garak, Google Vertex AI / Gemini, OWASP, and NIST.

Plain-English style: the teaching docs (01–05, 09) explain every technical term in everyday words the first time it appears (look for "In plain words:" glosses and short "Remember:" hooks). The goal is that you understand and remember each idea, not just recognize it.

What this role actually is (read this first)

Strip away the buzzwords and the job is two jobs in one:

  1. Bring AI into QA — make the existing 250-engineer test org faster/smarter using AI: AI-generated tests, self-healing locators, predictive test selection, failure clustering for triage, natural-language test authoring (English and French).
  2. Do QA on AI — build evaluation systems that certify other teams' AI before it ships: LLM-as-Judge pipelines on Vertex AI, RAG evaluation (RAGAS), agent/tool validation, adversarial red-teaming (garak), and embedding all of this into CI/CD as release gates.

The single most important mental shift they're testing for: moving from deterministic, assertion-based testing (assert actual == expected) to probabilistic, rubric-based evaluation of non-deterministic systems. Every answer you give should reflect that you get this shift.

You already have the QA/automation foundation (Selenium, Playwright, RestAssured, VAPT, some LLM-eval). This kit closes the AI-specific gaps.

The likely interview structure

Round Focus Where to study
1. Screening Experience narrative, "why AI testing", fit 06-model-answers.md (behavioral), this README
2. Core AI / LLM evaluation LLM internals, LLM-as-Judge, RAGAS, metrics 01, 02
3. Automation + Python coding AI-in-QA tooling, Python eval scripting, pytest 04, the practice repo
4. Red-teaming + System design + Behavioral Adversarial testing, design an eval pipeline, scenarios 03, 05, 06

Files in this kit

File What it covers
01-llm-fundamentals.md How LLMs work: tokens, temperature/top-p, context windows, grounding, hallucination, fine-tuning vs RAG, system/user/assistant roles. Taught from zero.
02-evaluation-llm-judge-and-rag.md LLM-as-Judge (pointwise/pairwise/reference), judge biases & calibration (Cohen's kappa, golden datasets), RAG explained, RAGAS metrics (faithfulness, relevancy, context precision/recall) with formulas.
03-red-teaming-safety.md Prompt injection (direct vs indirect), jailbreak taxonomy, OWASP LLM Top 10 (2025), NIST AI RMF, garak deep-dive, defenses, bias/toxicity/fairness, agent stress-testing.
04-vertex-gemini-and-automation.md Vertex AI + Gemini (Gen AI Eval Service, autorater, SDK), generation params, auth, plus AI-in-QA tooling (test generation, self-healing, predictive selection, failure clustering) and CI/CD gating.
05-system-design.md Walkthroughs: design an end-to-end AI certification pipeline, drift monitoring, certifying an Agentforce/Now Assist agent, adversarial corpus generation.
06-model-answers.md Full written model answers to the highest-probability ("⭐") questions, plus behavioral STAR answers.
07-mock-interview.md 60+ drill questions by topic with short ideal-answer cues. Use to self-test.
08-real-world-questions.md Real reported questions from online research (named-company interviews) + the honest BCE finding + 5 gaps to add (abstention, metamorphic testing, cost budgeting).
09-coding-questions.md Coding round with full answers: AI primitives (JSON+retry, backoff, rate limiter, cosine, eval scripts), the must-know-8 Python/DSA, pytest/RestAssured/Selenium, data processing. All solutions runnable in the repo.
10-mock-interview-answers.md Full plain-English answers to all 73 mock-interview questions in file 07. Use 07 to self-test, then check here.
11-jd-gap-coverage.md The last 10%: Tier 2/3 certification, RAG-over-a-database (Text-to-SQL) testing, Agentforce/Now Assist, living benchmark + playbooks, Azure OpenAI/Bedrock/Jenkins. Takes the kit to 100% JD coverage.
12-latest-llm-questions-2026.md Latest 2025–2026 questions from online research: top-10 most-repeated, deeper fundamentals (attention/RoPE/KV-cache/LoRA), and the NEW topics (MCP + tool poisoning, reasoning models, GraphRAG/Agentic RAG, cost/latency design). Flags what's already covered vs new.
practice-repo/ A runnable Python project: LLM-as-Judge, RAG metrics from scratch, judge calibration, and pytest release gates. Runs offline (mock mode) or against Gemini/OpenAI.

Suggested 1-week study plan

  • Day 1 — 01-llm-fundamentals.md. Be able to explain temperature, hallucination, and RAG vs fine-tuning to a 10-year-old.
  • Day 2 — 02. Memorize the 4 RAGAS metrics and their formulas; understand LLM-as-Judge biases.
  • Day 3 — Run the practice-repo. Make the pytest gates pass and fail on purpose. This makes the concepts physical.
  • Day 4 — 03-red-teaming-safety.md. Memorize OWASP LLM Top 10; be able to describe a direct vs indirect injection and 3 jailbreak techniques.
  • Day 5 — 04. The Vertex AI vocabulary + the AI-in-QA tooling answers.
  • Day 6 — 05-system-design.md. Practice the pipeline design out loud, on a whiteboard.
  • Day 7 — 06 + 07. Drill answers; record yourself.

The 10 things to know cold (the night-before list)

  1. Temperature controls randomness; set it to 0 for reproducible tests, but temp=0 is not fully deterministic (floating-point + batching).
  2. Hallucination = fluent but unsupported output; you test it with faithfulness (claims supported by context ÷ total claims).
  3. RAG = retrieve relevant docs, stuff them into the prompt, generate a grounded answer. You test retrieval and generation separately.
  4. RAGAS core 4: Faithfulness, Response (Answer) Relevancy, Context Precision, Context Recall. Know which need a reference answer.
  5. LLM-as-Judge = use a strong LLM to score outputs. Calibrate it against a human-labeled golden set using Cohen's kappa (aim ≥ 0.6).
  6. Judge biases: position, verbosity, self-preference — and the fix for each.
  7. Prompt injection: direct (malicious user input) vs indirect (malicious instructions hidden in retrieved content — the RAG threat).
  8. OWASP LLM Top 10 (2025) — at least LLM01 Prompt Injection, LLM06 Excessive Agency, LLM09 Misinformation.
  9. garak = "nmap for LLMs"; runs probes (attacks), detectors judge hit/miss, reports a failure rate; goes in CI as a gate.
  10. Non-deterministic testing: replace exact-match with semantic similarity + rubric scoring + pass-rate over N runs with tolerance bands.