Skip to content

04 — Vertex AI / Gemini & AI-in-QA Tooling

Two job-description (JD) areas in one file: the platform they use (Vertex AI / Gemini, JD #2) and bringing AI into the existing QA stack (JD #1). Plus what they expect for Python and CI.

The big picture (read this first)

  • Part A is about the tool Bell uses to run and grade AI — Google's Vertex AI. Analogy: Vertex AI is a rented AI workshop in the cloud — Google owns the machines, you just walk in and use them. You don't need to be an expert; you need the right vocabulary and to know its "grade my AI" service.
  • Part B flips it around: using AI to make normal testing faster — auto-writing tests, fixing broken locators by itself, picking which tests to run, and grouping failures. Think: an AI assistant for the QA team.
  • Part C is the Python + pipeline skills they expect you to code.

Part A — Vertex AI & Gemini (you only need working knowledge + correct vocabulary)

You don't need to be a Google Cloud (GCP) expert — the JD says "working knowledge of at least one platform (Vertex AI, Azure OpenAI, or AWS Bedrock)." So just know the words and the eval service.

What it is

  • Vertex AI = Google Cloud's managed machine-learning platform. In plain words: a place where you host, tune, run, and evaluate models through an API, with no servers for you to manage.
  • Gemini = Google's top family of models. "Multimodal" means it handles text, images, and more. On Vertex (early 2026): Gemini 2.5 Pro (strongest reasoning, ~1M-token context — it can read a lot of text at once), Gemini 2.5 Flash (fast and cheap everyday model, often used as a judge), and Flash-Lite (cheapest). You pick a model by its id, e.g. model="gemini-2.5-flash".
  • Remember: Pro = smartest, Flash = fast and cheap, Flash-Lite = cheapest.

Two naming facts that make you sound current

  1. The SDK got renamed. (An SDK is just the library you pip install to talk to the service.) Old one: Vertex AI SDK (google-cloud-aiplatform, import vertexai) — its gen-AI parts were deprecated (retired) on June 24, 2025. New/current one: Google Gen AI SDK (pip install google-genai, from google import genai) — one library that works with both the Gemini Developer API and Vertex AI.
  2. The product is being rebranded toward "Gemini Enterprise Agent Platform" in the docs. Same eval service underneath.

The Gen AI Evaluation Service (this is the JD's "LLM-as-Judge on Vertex AI")

A managed service that scores AI outputs for you. Analogy: it's an automatic exam-grader for AI answers — you hand it the answers, it returns the marks. Two families of metrics (a metric = one way to score): - Computation-based (deterministic = same input always gives same score, no LLM involved): exact_match, bleu, rouge/rouge_l_sum, tool-call correctness. - Model-based = LLM-as-a-Judge (a Gemini model called an "autorater" does the grading): groundedness, coherence, fluency, safety, instruction_following, summarization_quality, question_answering_quality, etc. - Remember: "autorater" = the Gemini model acting as the judge.

Pointwise vs pairwise (same idea as 02, just Vertex's words for it): - Pointwise — grade one model's answer against a rubric (e.g., give it 0–5). - Pairwise — show the judge two answers and let it pick the better one: a candidate (the new thing) vs a baseline (what you compare against). The service reports candidate_model_win_rate vs baseline_model_win_rate (how often each one won).

Custom rubrics: you write a metric prompt template that has criteria (what "good" means) plus a rating rubric (the score levels). AutoraterConfig lets you cut down on judge bias using response flipping (swap the A/B order so the judge can't favor whatever comes first — counters position bias), multi-sampling (ask more than once), or a tuned judge. Managed rubric-based metrics (e.g. RubricMetric.INSTRUCTION_FOLLOWING) split one big dimension into small, checkable items, so the score can be audited (you can see why it scored that way).

Minimal eval code (current google-genai SDK)

from google import genai
from google.genai import types

client = genai.Client(vertexai=True, project="my-project", location="us-central1")

# Generate candidate answers (or bring your own responses / "BYOR")
ds = client.evals.run_inference(model="gemini-2.5-flash", src="gs://.../prompts.jsonl")

result = client.evals.evaluate(
    dataset=ds,
    metrics=[
        types.RubricMetric.TEXT_QUALITY,                # LLM-judge / rubric
        types.RubricMetric.QUESTION_ANSWERING_QUALITY,
        types.Metric(name="bleu"),                      # computation-based
    ],
)
(Older tutorials show vertexai.evaluation.EvalTask(...).evaluate() — recognize it if you see it, but say client.evals is the current way.)

Calling Gemini + generation params

These params shape the model's answer. Quick gloss for each below. 

resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Why is the sky blue?",
    config=types.GenerateContentConfig(
        temperature=0.0,      # 0 for reproducible evals/judges
        top_p=0.95, top_k=20,
        max_output_tokens=500,
        safety_settings=[...],  # per HarmCategory block thresholds
    ),
)
print(resp.text)
In plain words: - temperature = how random/creative the answer is. 0 = boring but repeatable — use it for evals and judges so you get the same answer every time. - top_p / top_k = two other ways to limit which words the model can pick from. - max_output_tokens = the longest answer allowed (a token is roughly part of a word). - safety_settings = how strictly to block harmful content, set per HarmCategory.

Auth (on Vertex, GCP uses IAM, not API keys)

"Auth" = how you prove you're allowed in. IAM = Google's permission system. - Local dev (your laptop): run gcloud auth application-default login. This creates ADC (Application Default Credentials — saved login your code reuses). - Running on GCP (Cloud Build/Run/VM): the service account (a robot account) attached to it is used automatically — no login step. - Prefer service-account impersonation / Workload Identity Federation over downloaded JSON key files. In plain words: borrowing an identity safely beats keeping a secret key file on disk that can leak.

Part B — Bringing AI into the QA stack (JD #1)

These are "how would you modernize our test team with AI?" questions. Have a clear answer ready for each.

1. AI-driven test generation (Selenium / Playwright / Postman)

  • Give the LLM a user story / requirement / API spec (OpenAPI = a standard file describing an API) → it generates test cases, step skeletons, edge cases, and boundary/negative cases.
  • For APIs: generate Postman/Newman collections or RestAssured tests straight from the OpenAPI schema.
  • Critical QA point to say out loud: AI-generated tests are a first draft only — they make up (hallucinate) selectors and assertions. So you review first, then commit. Judge their value by coverage added and real bugs found, not by how many tests it spat out. Saying this shows judgment, not hype.

2. Self-healing automation (locators that survive UI/API changes)

  • The problem: one renamed CSS selector breaks 50 tests overnight. (A locator/selector = how a test finds an element on the page.)
  • Self-healing: store several clues per element (id, text, role, position, accessibility attributes, a DOM "fingerprint"). When the main locator fails, an ML/heuristic ranker picks the best clue still working and auto-updates the test.
  • Caveat to raise: self-healing can hide real bugs — it might "heal" past an element that was actually supposed to be gone. So it must log every heal for a human to review and fail loudly when it isn't confident.

3. Predictive test selection / prioritization (from code changes + defect history)

  • Don't run all 10,000 tests on every commit. Use a model on the git diff + past defect/flake data + a code-to-test map to predict which tests are most likely to catch a bug for this specific change, and run those first.
  • Metrics: APFD (Average Percentage of Faults Detected — how fast your test order finds bugs), time-to-first-failure, and the % of failures caught inside the chosen subset.

4. Failure clustering for defect triage / root-cause (JD #1)

  • 500 tests fail overnight, but it's usually only ~5 real causes. So: turn each failure (stack trace + error message + logs) into a vector with embeddings (numbers that capture meaning), cluster them (group similar ones — e.g. DBSCAN/k-means on those embeddings), and look at one example per cluster instead of filing 500 tickets.
  • Then add an LLM to summarize each cluster's likely root cause. This matches "automate defect triage and root-cause analysis using failure clustering" almost word for word.
  • Remember: 500 failures → ~5 buckets → 5 tickets.

5. Natural-language test authoring (English/French) for non-technical QA

  • Let a non-coder type "Log in as a Bell mobility customer and verify the bill shows the correct French date format" → an LLM turns it into runnable Playwright/Gulp steps using a library of ready-made steps.
  • Keep it reliable: restrict the LLM to a fixed set of vetted step functions (not free-form code), check the generated steps, and keep a human approval gate. Bilingual (English/French) support is explicitly required.

6. Technology radar

  • "Continuously pilot emerging AI testing tools." Be ready to name a few you'd try: Playwright's AI features, RAGAS, garak, PyRIT, promptfoo, Giskard, DeepEval. A radar = a living scorecard that rates tools by maturity and fit, tried out and re-reviewed every quarter.

Remember the 6 (one line each): (1) generate tests from a story/spec, (2) self-heal broken locators, (3) predict which tests to run, (4) cluster failures into a few buckets, (5) plain-English test authoring (EN/FR), (6) radar of new tools. Each one says: let AI do the boring/slow part, but keep a human checking it.

Part C — Python + CI/CD expectations

The JD requires Python (building frameworks, API integration, data processing, eval scripting) and CI/CD (GitHub Actions, Google Cloud Build, Jenkins). CI/CD = the pipeline that builds and tests your code automatically on every change.

Be ready to live-code: - Call an LLM API and parse/validate the JSON it returns, with a retry if it comes back malformed. (Pattern: ask for JSON, json.loads, check it against a schema, retry while feeding the error back in.) - Write a simple eval metric (exact-match %, semantic similarity, or a faithfulness-style claim check). - pytest parametrized over a dataset of prompts, asserting on pass-rate / tolerance, not exact strings. (Parametrized = run the same test over many inputs.) - Data processing with pandas (load a results CSV/JSONL, add up scores per metric, spot regressions). - API integration with retry + exponential backoff + rate-limit handling. (Exponential backoff = wait longer after each failed retry so you don't hammer the API.)

CI/CD framing (say this): "I treat the eval suite like a regression suite: golden dataset = test data, thresholds = assertions, a failing eval = a failing build. The difference is the assertions are probabilistic, so I stabilize them — temperature=0, multi-sampling, pass-rate thresholds over N runs, and tolerance bands against the last good baseline — and run a fast subset per-PR with the full suite nightly." In plain words: a normal test is pass/fail; an AI test is "passes often enough," so you steady it (temperature 0, run it several times, require a pass-rate, allow a small tolerance vs the last good run) and run the cheap subset on each PR, the full suite overnight.

The practice-repo does exactly this: an LLM client (mock/Gemini/OpenAI), an LLM-as-Judge, RAG metrics built from scratch, kappa calibration, and pytest release gates. Running it is the best prep for the coding round.

→ Next: 05 — System Design