07 — Testing Non-Deterministic LLM Output (learn from scratch → interview-ready)¶

This is the single most important concept in the JD: "Understands that LLM outputs are non-deterministic and knows why standard assertion patterns must be adapted." Nail this and you've cleared the "required awareness" bar.

How to use this file: read top-to-bottom the first time (it builds the whole idea from "same input → same output" up). Later, jump to Rapid-fire recall to revise. You need basic pytest (01); no ML background required.

0. The core problem, in one picture¶

Every test you've ever written assumes:

same input → same output. Call add(2, 3), you always get 5. So assert add(2, 3) == 5 works forever.

An LLM breaks that. Ask "what's my balance?" twice and you might get: - "Your account balance is $100.00." - "You currently have $100 in your account."

Both are correct. But assert answer == "Your account balance is $100.00" fails on the second — marking a correct answer as a bug. Do that across a suite and every run is randomly red. This is why traditional testing falls apart on AI, and why this role exists: you have to change what "pass" means.

Everything below is (a) why it varies and (b) what to assert instead.

1. Why the output changes (say the mechanism, not "it's random")¶

An LLM generates text one token (≈ a word-piece) at a time. For each next token it produces a probability distribution over the whole vocabulary — e.g. "balance" 40%, "account" 25%, "current" 10%, … Then, crucially, it samples from that distribution rather than always taking the top choice. Different sample → different word → different sentence.

The knobs that control this (know the names — interviewers ask): - Temperature — how "adventurous" the sampling is. 0 ≈ greedy/most-likely (near-deterministic, focused); higher (e.g. 0.7) = more varied/creative. - top-p (nucleus) — sample only from the smallest set of top tokens whose probabilities sum to p. - top-k — sample only from the k most likely tokens.

The gotcha that catches people out: even at temperature 0 output isn't guaranteed identical — GPU floating-point non-associativity, request batching, mixture-of-experts routing, and silent provider model updates cause drift. So "just set temperature 0 and exact-match" is not reliable.

Interview line: "LLMs sample the next token from a probability distribution controlled by temperature and top-p, so identical prompts yield different text. And even at temperature 0 you're not fully deterministic — floating-point/batching effects and silent model updates cause drift — so I never assert exact string equality on model output."

2. The mindset shift: from "is it exact?" to "is it acceptable?"¶

Traditional test	LLM test
"Is the output exactly this string?"	"Is the output acceptable — correct, safe, right shape?"
One right answer	Many valid answers
`==`	invariants + meaning + rubric + thresholds
Pass/fail on one run	Pass-rate over many runs / a dataset

assert response == "Your account balance is $100.00"   # WILL flake — both wordings are correct

The test's job shifts from "is it this exact string?" to "is it acceptable?" against a rubric.

3. The five assertion strategies that replace exact-match¶

A toolkit — you usually combine two or three. Ordered cheapest-and-most-reliable first.

Strategy 1 — Structural / deterministic invariants (assert what IS fixed — do this first, always)¶

Even non-deterministic answers have deterministic properties. Assert these exactly — cheap, fast, catch most real regressions:

assert isinstance(resp.confidence, float) and 0 <= resp.confidence <= 1   # shape
assert resp.trace_id                       # envelope present
assert "$" in resp.answer                  # a currency answer must contain a figure
assert resp.tool_calls[0]["name"] == "get_balance"   # correct tool chosen
assert len(resp.answer) < 500              # length bound
assert not contains_pii(resp.answer)       # safety invariant

Intuition: you can't predict the sentence, but you can be certain a balance reply contains a "$", used the balance tool, and leaks no PII.

Strategy 2 — Semantic similarity (meaning, not words)¶

Embed the response and a reference, compare with cosine, pass above a threshold:

sim = cosine(embed(response), embed(reference_answer))
assert sim >= 0.85            # "close enough in meaning"

Catches paraphrase-equivalent answers. Caveat: semantically similar ≠ factually correct — combine with Strategy 1.

Strategy 3 — LLM-as-Judge (rubric grading)¶

Use a model to grade the answer against a rubric, returning a structured verdict:

verdict = judge(
    question=q, answer=response, rubric="correct, grounded, no advice beyond policy",
)   # -> {"pass": True, "score": 4, "reason": "..."}
assert verdict["pass"]

Best for open-ended quality. The judge is itself an LLM with biases (position, verbosity, self-preference) → calibrate against ~50 human labels (Cohen's kappa), prefer pairwise where possible, and pin the judge model/version. (More in 11.)

Strategy 4 — Deterministic-mode + snapshot (for regression, not correctness)¶

Set temperature 0, seed if supported, snapshot the output, review diffs on change. Good for prompt-regression detection, not for proving correctness — a change-detector, not a correctness check.

Strategy 5 — Property / metamorphic testing (assert relationships, no fixed answer needed)¶

Assert rules that must hold across inputs, without a fixed expected output: - Invariance — rephrasing the question shouldn't change the answer's meaning. - Consistency — "balance of A" then "of A again" → same figure. - Monotonicity — adding more relevant context shouldn't reduce faithfulness. - Negation / abstention — an unanswerable question must trigger "I don't know", not a fabrication (your front-line hallucination test):

def test_abstention_on_unknown(agent):
    r = agent.invoke("What's the CEO's home address?")
    assert r.refused or "don't have" in r.answer.lower()   # must NOT hallucinate

Intuition: you don't need the right answer to know a relationship — "rephrasing shouldn't flip the meaning" is checkable even when wording is unpredictable.

4. Handling flakiness deliberately¶

Threshold over N runs / a dataset, not a single sample — e.g. "≥ 95% of golden cases pass the judge." One-off dips don't fail the build; a real regression does.
Lower temperature (0–0.2) on tested paths to reduce variance while still measuring real behaviour.
Aggregate, don't spot-check: report pass-rate + mean score + variance. A healthy suite tracks a distribution, not one boolean.
Split the suite (the key architectural idea):
Logic tests — mock the LLM so it returns a fixed answer; the agent's code (routing, tool calls, error handling) is deterministic and runs on every PR. (Why 01 §6 mocking matters here.)
Eval tests — call the real model, measure quality probabilistically over the golden set, run on a schedule / pre-deploy gate, not per-commit.

Interview line: "I split the suite: agent logic is tested deterministically by mocking the model, so it runs on every PR; model quality is measured probabilistically over a golden set with a pass-rate threshold, run as an eval gate rather than per-commit."

5. The decision table (which strategy when)¶

What you're checking	Strategy
Response shape, tool chosen, confidence range, safety	Structural invariants (always start here)
"Means the same as the reference"	Semantic similarity
Open-ended quality / helpfulness / tone	LLM-as-Judge + rubric
Prompt regression across a code change	Deterministic snapshot
Robustness to rephrasing / consistency / abstention	Metamorphic / property
Factual grounding on retrieved docs	Ragas faithfulness (file 11)

Real tests combine several: invariants to catch shape/safety cheaply, plus a judge or similarity score for content quality, aggregated over a dataset.

6. Common beginner mistakes (say you avoid these)¶

Exact-matching model text → guaranteed flaky suite; false failures on correct answers.
"Temperature 0 makes it deterministic, so I can exact-match" → wrong; drift still happens (§1).
Trusting the LLM judge blindly → calibrate against humans and pin its version first.
Judging on a single run → use a pass-rate over a golden dataset.
Mixing logic and quality tests → keep mocked-deterministic logic tests separate from live probabilistic evals.
Only checking the final answer → for agents, also assert intermediate steps/tool calls (file 08).

7. Try it yourself (makes it concrete)¶

Against any chat function ask(prompt) -> str:

import re

# 1) invariant: a balance answer must contain a currency figure, whatever the wording
def test_balance_answer_has_figure():
    a = ask("what's my balance?")
    assert "$" in a and len(a) < 300           # has a figure, not rambling

# 2) abstention: an unanswerable question must NOT be fabricated
def test_refuses_unknown():
    a = ask("what is my neighbour's account balance?").lower()
    assert "can't" in a or "don't have" in a or "not able" in a

# 3) consistency: same question twice -> same figure
def test_consistent_number():
    n1 = re.search(r"\d+", ask("balance of account A"))
    n2 = re.search(r"\d+", ask("balance of account A again"))
    assert n1 and n2 and n1.group() == n2.group()

Notice: not one == "exact string" anywhere. You asserted shape, abstention, and a relationship — the three moves that replace exact-match. See it running for real in practice-repo/ test_trace_and_nondeterminism.py.

Rapid-fire recall¶

Traditional testing assumes same-input→same-output; LLMs break that (token sampling via temperature/top-p).
Even at temperature 0 it can drift (floating-point/batching + silent model updates).
Never exact-match; assert invariants first, then semantic / judge / snapshot / metamorphic.
Judge biases exist → calibrate + pin the judge; prefer pairwise.
Threshold over a dataset, report a distribution — not one run.
Mock the model for logic (deterministic, per-PR); eval the live model for quality (probabilistic, gated).
For agents, assert intermediate states/tool calls too (08); for grounding use Ragas (11).