QA for Traditional and LLM-Based Applications — Working Strategy & Framework¶
Author: Rohan Dsouza, Senior QA Engineer Audience: QA team, engineering leads, product Status: Working doc, v1 — comments welcome
To keep this concrete I'll anchor everything to a product I'll call Shopfront: a fairly standard e-commerce web app (catalog, checkout, accounts, admin panel, public REST API) that's now shipping an AI support assistant — an LLM chatbot with RAG over our help-center docs, plus tool access to the orders API so it can answer "where's my order" with real data. One release train, two very different testing problems. I've written this as one document rather than two playbooks, because that's how we'll actually work: the same team signs off both, in the same sprint.
1. Test Strategy / Test Plan¶
1.1 Scope¶
In scope: checkout and payment flows, account management, admin panel, the public REST API, and the assistant end-to-end — chat UI, retrieval quality, tool calls into the orders API, and the guardrails around all of it.
Out of scope: model training or fine-tuning quality (we consume a hosted model — we test the system around it, not the weights), deep penetration testing beyond an OWASP-basics pass plus our prompt-injection suite, and load beyond the agreed seasonal peak profile.
1.2 Objectives¶
For the traditional surfaces the objective is the familiar one: catch functional regressions before release, and keep the defect escape rate on money paths as close to zero as we can. Checkout gets the most paranoia. That's a habit from fintech and I'm keeping it.
For the assistant, the objective has to be stated differently, and this matters: we're not verifying correct output, we're measuring and bounding behavior. You can't verify a generated paragraph is "correct" the way you verify an order total. What you can do is measure whether its factual claims match the data it was given, whether it stays inside policy, how it sounds, what it costs, and how long it takes — and hold every release to thresholds on those measurements.
1.3 Where LLM testing diverges from traditional testing¶
This list shapes everything else in the document, so it goes up front:
- Non-determinism. The same input produces different phrasing run to run. So eval cases run N times (we use 3–5) and gate on pass rate, not a single pass/fail. Guardrail cases are the exception: those gate at 100%, because a policy control that holds "usually" is broken.
- No fixed expected output. Assertions become rubrics and graders instead of string equality — with deterministic backstops wherever possible. If the bot quotes a refund amount, we extract the number and diff it against the source data exactly. The LLM-as-judge never gets the final word on anything numeric.
- Evaluation instead of pass/fail. Quality is a score and a trend line, not a green tick. A release can "pass" at 96% and still be a regression from last week's 99% — the trend is the signal.
- Prompts are code. A one-line prompt edit can change behavior more than a hundred-line code change. Prompt changes get the same treatment as source: version control, review, and a full eval-suite run before merge. Teams that skip this get burned within a month, usually by a "harmless wording tweak."
- Hallucination is its own failure class. There's no traditional equivalent of software confidently inventing a fact. It needs dedicated tests (grounding checks, fabrication-rate measurement), not just functional coverage.
- Latency and cost are test criteria, not ops concerns. An answer that's correct but takes 12 seconds, or burns 10x the token budget, is a failing answer. Budgets go in the exit criteria.
- The platform drifts under you. A hosted model can change behind the API. We pin model versions where the provider allows it, and re-run the eval suite as a canary whenever we deliberately take a model update — treating it like a major dependency bump, because it is one.
1.4 Entry and exit criteria¶
| Traditional surfaces | LLM assistant | |
|---|---|---|
| Entry | Build deployed to QA env, smoke suite green, feature behind a flag, test data seeded | Same, plus: model + prompt versions pinned and tagged, golden dataset updated for any new intents, eval harness green against the current-production baseline |
| Exit | All planned cases executed; zero open P1/P2; regression suite green; p95 page/API latency within budget | Eval pass rate ≥ 95% overall; 100% on the guardrail/adversarial suite; zero fabricated facts on the grounding set; p95 response latency and cost-per-conversation within budget; human review sample signed off by the domain reviewer |
Note the shape difference: the traditional exit is "nothing known is broken." The LLM exit is "measured behavior is inside agreed bounds." Both are sign-offs; only one of them is a promise of correctness.
1.5 Environments, tools, and roles¶
Environments. Shared QA env and a prod-like staging with sandboxed payments — standard. The LLM-specific additions: staging must call the real model at a pinned version, not a mock. I'd push back on anyone proposing we mock the LLM for behavioral testing — mocks are fine for testing the surrounding flow logic (retries, timeouts, UI states) but tell you nothing about what the model will actually say. We also need seeded synthetic customer accounts (ground truth we control for hallucination checks) and a frozen snapshot of the help-center docs per release, so grounding results are comparable week over week.
Tools.
| Layer | Tooling |
|---|---|
| UI (web + chat widget) | Playwright (TypeScript); Selenium only where legacy suites already exist |
| API | Postman for exploratory work; pytest + requests (and REST Assured for the Java services) in CI |
| Load | k6 — checkout under peak, and concurrent chat sessions against the assistant |
| LLM eval | promptfoo for prompt-regression suites; custom pytest harness with LLM-as-judge + deterministic diffs for grounding; Ragas for retrieval-specific metrics; Langfuse for tracing and eval tracking over time |
Roles. Two QA/SDETs cover both surfaces — I don't want a separate "AI QA" silo; the assistant ships in the same releases as everything else. What's genuinely new: a prompt owner (one named person accountable for prompt changes — the eval gate is ours, but someone has to own the file), a domain reviewer for the weekly human-review sample (our support lead is the right person; she knows what a wrong answer looks like better than any of us), and a golden-dataset curator — I'll own that to start, since the dataset is the eval framework's foundation and it decays fast if nobody tends it.
2. Test Cases / Test Scenarios¶
(to be drafted)
3. QA Process / Framework¶
(to be drafted)
4. Bug / Defect Report Analysis¶
(to be drafted)