11 — JD Gap Coverage (the last 10%)¶

This file closes the five things the job description mentions that weren't fully covered elsewhere in the kit. Same plain-English style: a quick gloss the first time a term appears, plus short "Remember:" hooks. Each section ends with the interview soundbite to say out loud.

Honesty note: some of this (Bell's exact certification tiers, the internal AIQC playbooks) is Bell-internal and not public. Where that's true, I teach the industry-standard version of the idea so you can speak about it sensibly and adapt on the job.

1. Tier 2 & 3 Certification (JD: "Own Tier 2 & 3 certification testing from corpus design to red-teaming")¶

What "tiered certification" means. Not every AI system is equally risky, so you don't test them all the same way. You sort systems into tiers by risk, and the higher the tier, the more rigorous (and more expensive) the testing. This is exactly how safety frameworks work — the EU AI Act and NIST both tier AI by risk. Bell's program (AIQC = AI Quality Certification) does the same.

In plain words: a small internal helper bot gets a quick check; a customer-facing agent that can move money gets the full treatment.

A sensible, industry-standard reading of the tiers (use this if asked — and say you'd confirm Bell's exact definitions):

Tier	Risk level	Example	What testing looks like
Tier 1	Low	Internal text summarizer, no customer impact	Automated metrics + basic safety checks; mostly self-serve
Tier 2	Medium	Customer-facing RAG chatbot (RAG = fetches docs to answer)	Full eval suite (faithfulness/relevance/recall), golden dataset, red-teaming, CI/CD release gate
Tier 3	High	Agent that takes actions (issues refunds, changes accounts), or anything safety/financial/legal	Everything in Tier 2 plus deep adversarial red-teaming, human-in-the-loop review, EN/FR fairness, strict sign-off with conditions

Why the JD says you "own Tier 2 & 3": those are the systems that actually need a specialist — the high-stakes ones. Tier 1 can be largely automated; Tiers 2–3 need someone who designs the corpus, runs the red-team, calibrates the judge, and signs the certificate. That someone is this role.

What "from corpus design to red-teaming" means: you own the whole certification flow for those systems — build the test set (05 Q4), score it (02), attack it (03), gate it in CI/CD, and issue the certificate (05 Q1).

Remember: higher tier = higher risk = deeper testing. Tier 2 = full eval + red-team; Tier 3 = that plus human review and stricter sign-off.

Soundbite: "I'd treat certification as risk-tiered — light automated checks for low-risk tools, and for Tier 2/3 systems the full pipeline: golden corpus, RAGAS metrics, calibrated LLM-judge, adversarial red-team in EN and French, a CI/CD gate, and a signed certificate with scores, risks, and conditions. Tier 3 adds human-in-the-loop review because the blast radius is bigger."

2. RAG over a Database / Text-to-SQL (and how to test it)¶

You asked this directly. RAG isn't only for PDFs — the source of truth can be a database. The big idea (retrieve, then answer) stays the same; only how you retrieve changes.

Two retrieval styles: - Unstructured data (docs/PDFs/web) → vector search (find chunks with similar meaning). Classic RAG. - Structured data (a SQL database) → Text-to-SQL or tool calls (fetch exact rows with a query).

In plain words: vectors for fuzzy text, SQL/tools for exact structured data.

The patterns for a DB-backed system: - Text-to-SQL (NL2SQL): you give the model the database schema (the table/column names) + the user's question; it writes a SQL query; your app runs it; the model answers from the rows. - Example: "How many customers churned last month?" → SELECT COUNT(*) FROM customers WHERE churned=true AND churn_date >= '2026-05-01' → DB returns 1432 → "1,432 customers churned last month." - Tool / function calling (safer, common in production): instead of raw SQL, the model picks a pre-written function like get_churn_count(month) and fills in the arguments; your trusted code runs the DB query. The model never touches SQL. (This is the JD's "tool usage.") - Hybrid RAG: vector search over policy docs and a SQL query for live account data, combined into one answer.

Why you don't just embed the rows: embeddings are bad at exact filters, aggregations (count/sum/average/latest/top-5), and joins. SQL does these precisely; embeddings only find "similar-sounding" rows. That's the whole reason Text-to-SQL exists.

How you test it (this is the part interviewers want):

What to test	How
Is the generated SQL correct? (retrieval step)	Compare against a gold SQL, or better, run both and compare the result sets. RAGAS has SQL metrics for this: Execution-based DataCompy (compare query results) and SQL Query Equivalence.
Is the answer faithful to the rows? (generation step)	The final answer must only use values the query actually returned — no invented numbers.
Tool-call accuracy (if function calling)	Right function, right arguments, right order (RAGAS Tool Call Accuracy).
Security (ties to your VAPT background)	SQL injection via the prompt; the model must never generate `DROP`/`DELETE`/`UPDATE`; no access to unauthorized tables/columns; no runaway/expensive queries. Enforce a read-only DB user + a table allow-list, then test those guardrails.
Hard query types	Aggregations, multi-table joins, date math, NULL handling — Text-to-SQL gets these subtly wrong (same spirit as numeric hallucination).

Remember: it's still retrieve-then-generate, so still test retrieval vs generation separately — but here "retrieval" = a generated SQL query (or tool call), so you add SQL correctness + a security layer that doc-RAG doesn't need.

Soundbite: "For a DB-backed assistant it's structured RAG — Text-to-SQL or tool calls instead of vector search. I test SQL correctness against gold queries and result sets, faithfulness of the answer to the returned rows, tool-call accuracy, and a security layer for injection and destructive or unauthorized queries, with a read-only user and a table allow-list."

3. Salesforce Agentforce & ServiceNow Now Assist (named in the JD)¶

The JD says you'll evaluate "Salesforce Agent Force workflows and ServiceNow Now Assist integrations." You don't need to be an admin of either — know what they are and how you'd certify them.

Salesforce Agentforce — Salesforce's agentic AI platform (an agent = an AI that can take actions, not just chat). Agents are built from Topics (jobs they can do) and Actions (the actual operations — e.g., "update a case," "issue a refund"), with guardrails on what they're allowed to do. Salesforce even ships an Agentforce Testing Center for running test cases against agents.

ServiceNow Now Assist — ServiceNow's GenAI assistant built into their products (IT service management, customer service, HR). It does things like summarize incidents, draft replies, and suggest resolutions, using ServiceNow's "Now LLM" plus partner models.

In plain words: both are big enterprise platforms with AI agents bolted on that can act inside the company's systems — so the risk isn't just a bad sentence, it's a bad action.

How you'd certify either (it's agent testing — reuse 03 §8 + 05 Q3): - Tool/Action accuracy — does it call the right action with the right inputs? (e.g., refund the correct amount to the correct account.) - Topic adherence / scope — does it stay within its allowed jobs and refuse out-of-scope requests? - Goal accuracy — did it actually complete the user's task end-to-end? - Escalation — does it hand off to a human when it should? (The JD calls this out.) - Excessive Agency red-team (OWASP LLM06) — try to make it take unauthorized or runaway actions; enforce least privilege (minimum permissions) and a sandbox that mocks the real actions so tests cause no real damage. - Faithfulness/grounding — for the summarize/draft features, the output must match the underlying records (no invented details). - EN/FR — test both languages (Bell requirement).

Remember: these are agents that act, so "judge the journey, not just the destination" — test the sequence of actions, with sandboxed tools and least privilege.

Soundbite: "Agentforce and Now Assist are agentic — they take actions in Salesforce/ServiceNow, so I certify them as agents: tool/action accuracy, topic adherence, goal completion, escalation, and an Excessive-Agency red-team for runaway or unauthorized actions, all with sandboxed tools, least privilege, and EN/FR coverage."

4. Living Benchmark + AIQC Playbooks + Advising (the "program-building" side)¶

The JD asks you to "maintain a living benchmark of Bell-specific AI quality standards," "contribute to AIQC playbooks, documentation, and knowledge sharing," and "advise teams on AI testability, prompts, and evaluation instrumentation." This is the build-the-practice, not just run-the-tests part. Mostly Bell-internal, so here's the standard version.

Living benchmark = a test set that never stops growing. In plain words: a shared, version-controlled collection of Bell's test cases + quality bars that you keep updating as you learn. It's "living" because: - You add every new production failure to it (so the same bug can't escape twice). - You add new attack types as red-teaming finds them. - You refresh it so it doesn't go stale, and keep a private held-out portion so teams can't "study to the test." - It encodes Bell-specific standards — e.g., "customer-facing bots must score ≥ 0.9 faithfulness in EN and FR," "zero PII leaks," "agents need human sign-off for money actions."

Remember: a living benchmark = "the bar + the tests, kept fresh forever."

AIQC playbooks = the how-to guides so the practice scales beyond you: step-by-step "how to certify a RAG app," "how to red-team an agent," rubric templates, and checklists. The 250 QA engineers and the AI teams follow these. (Your kit's 05 skeletons are basically mini-playbooks.)

Advising teams ("quality by design") = shift-left for AI. You partner with AI/ML teams early so systems are built to be testable: - Testability: ask them to expose the retrieved context, tool-call logs, and confidence scores so you can actually measure faithfulness and trace failures. - Prompts: review system prompts for guardrails and injection resistance. - Instrumentation: make sure the system logs inputs/outputs/decisions so you can monitor drift and feed failures back.

In plain words: instead of testing at the end, you help them build it test-ready from the start.

Soundbite: "I'd treat the benchmark as a living asset — every production failure and new attack gets folded back in, encoding Bell-specific bars like EN/FR faithfulness thresholds and zero-PII rules. I'd write playbooks so the 250 QA engineers can self-serve Tier 1, and partner with AI teams early so systems log their context and tool calls and are testable by design."

5. Other Platforms (Azure OpenAI, AWS Bedrock) & Jenkins¶

The JD requires one of Vertex AI / Azure OpenAI / AWS Bedrock — and you have Vertex covered in 04. But interviewers may ask "what about the others?" One paragraph each so you're never blank.

Azure OpenAI — Microsoft's service that hosts OpenAI models (GPT-4o, etc.) on Azure, with enterprise security. For safety it adds Azure AI Content Safety and Prompt Shields (filters that catch jailbreaks and injection). Evaluation lives in Azure AI Foundry (its eval/prompt-flow tooling). Mental map: "Vertex's Gemini ≈ Azure's GPT models; both have a managed eval service and built-in safety filters."

AWS Bedrock — Amazon's service giving API access to many foundation models (Anthropic Claude, Meta Llama, etc.) behind one API. Key pieces: Guardrails for Bedrock (safety filters + denied topics + PII redaction), Knowledge Bases (managed RAG), and Agents for Bedrock (tool-using agents). Mental map: "Bedrock = model marketplace + managed RAG + guardrails."

In plain words: all three do the same jobs — host models, offer a RAG option, an eval option, and safety filters. The concepts you learned for Vertex transfer directly; only the names change.

Jenkins (CI/CD) — the JD lists GitHub Actions, Google Cloud Build, and Jenkins. Jenkins is an older, self-hosted automation server: you define a pipeline in a Jenkinsfile (written in Groovy) with stages (build → test → deploy). For this role, the point is identical to the others: add an eval stage that runs your golden-set evaluation and fails the build if scores drop below threshold.

// Jenkinsfile — an "AI eval gate" stage
stage('AI Eval Gate') {
    steps {
        sh 'pytest tests/test_eval_gates.py'   // fails the build if metrics regress
    }
}

Remember: GitHub Actions / Cloud Build / Jenkins are interchangeable for our purpose — each just needs one stage that runs the eval suite and blocks the release on a regression.

Soundbite: "I've worked with Vertex AI; Azure OpenAI and Bedrock map to the same concepts — hosted models, managed RAG, an eval service, and safety guardrails — so the skills transfer. And for CI, whether it's GitHub Actions, Cloud Build, or a Jenkins pipeline, the pattern is the same: an eval stage that runs the golden-set gate and fails the build on a regression."

Coverage check — kit is now 100% against the JD¶

Every JD responsibility and required skill now has a home: - Responsibilities 1–5 → files 01–05 + this file (Tier 2/3, living benchmark, playbooks, advising). - Structured/DB RAG → this file §2. - Agentforce / Now Assist → this file §3. - Vertex AI (+ Azure/Bedrock awareness) and Jenkins → 04 + this file §5. - Python, automation frameworks, non-deterministic test strategy, LLM concepts → 01, 02, 04, 09 + practice-repo.

→ Back to README