10 — Model-Version Regression Detection¶

JD (required awareness): "Awareness of model version regression risks and how to detect quality degradation when model versions change." Providers update models silently, or you upgrade deliberately — either way, behaviour can shift. Your job: catch degradation before it reaches production.

1. Why this is a real risk¶

In plain words: the same prompt sent to gpt-4o last week and this week (or after you switch model versions) can produce meaningfully different, sometimes worse, answers — because the provider retrained/updated the model, or you changed the version/params. Nothing in your code changed, but quality dropped.

Failure examples: a model update makes the agent more verbose (breaks length gates), less willing to call tools (hallucinates numbers), or changes JSON formatting (breaks parsing), or degrades on a niche domain while improving overall.

Interview line: "Model version is an uncontrolled dependency. A silent provider update or a deliberate version bump can change output distribution, so I pin the version, log it on every run, and run a frozen golden-set eval as a regression gate — I treat a model change like a dependency upgrade that must pass CI."

2. The detection system (this is the answer they want)¶

a) Golden / regression dataset (the baseline)¶

A frozen, version-controlled set of representative inputs with expected outcomes or rubrics.
Cover the important paths + known past failures + edge cases + adversarial cases.
Immutable so results are comparable over time (never edit silently; version it).

b) Offline eval harness¶

Run the agent over the golden set and compute metrics per case: - Structural pass (schema/tool/path — file 07/08). - Semantic similarity / LLM-judge score / Ragas (files 07/11). - Latency, cost, token usage. Aggregate into a scorecard: pass-rate, mean score, per-category breakdown.

c) Baseline comparison + threshold gate¶

current  = run_eval(golden, model="claude-sonnet-4-6")   # or new version
baseline = load("baseline.json")

assert current.pass_rate >= THRESHOLD                    # absolute floor
assert current.mean_score >= baseline.mean_score - 0.02  # no meaningful regression
# per-category guard: no single slice collapses even if the average holds
for cat in current.by_category:
    assert current.by_category[cat] >= baseline.by_category[cat] - 0.05

The per-category guard matters: an average can hide a domain that got much worse.

d) Wire it as a gate (file 06)¶

On any model-version change, block deploy unless the eval passes and requires human sign-off. Record the new scorecard as the next baseline once accepted.

3. Detecting silent provider changes in production¶

You didn't change anything, but the model did. Detect via: - Log model_version on every generation (Langfuse captures it, file 09). Alert when it changes unexpectedly. - Continuous eval / canary: periodically re-run the golden set in prod; alert on score drop. - Production monitors: track distributions of output length, tool-call rate, refusal rate, confidence, user thumbs-down, latency, cost. A shift in these = investigate. (This is drift — see §5.)

4. Safe rollout strategies (mention if asked "how would you upgrade a model?")¶

Strategy	In plain words
Shadow	Run the new model alongside prod on real traffic, don't serve its output; compare offline. Zero user risk.
Canary	Serve the new model to a small % of traffic; watch metrics; ramp up if healthy.
A/B	Split traffic, compare quality/business metrics statistically.
Blue-green	Instant switch with instant rollback path.

Always keep the rollback trivial (config-flag the model version).

5. Drift vs regression (know the distinction)¶

Regression — quality dropped due to a known change (you bumped the version). Caught by the eval gate.
Drift — behaviour degrades over time due to changing inputs (new user phrasing, new products) or a silent model update. Caught by production monitoring + periodic re-eval.
Data drift (input distribution changes) vs concept drift (the right answer changes) vs model drift (the model itself changed).

6. Statistical rigor (senior signal)¶

A single sample isn't evidence — evaluate over N cases and report pass-rate with a confidence interval.
For "is the new model actually worse?", compare score distributions with a significance test (e.g. bootstrap the difference in means, or a proportion test on pass-rate) rather than eyeballing two numbers.
Account for LLM non-determinism: run each case a few times, or fix temperature low, so the comparison measures the model change, not sampling noise.

Interview line: "I don't call a 1% score drop a regression — I evaluate over the whole golden set, run cases multiple times to average out sampling noise, and use a significance test before failing the gate, so I'm not blocking on statistical jitter."

Rapid-fire recall¶

Model version = uncontrolled dependency → pin + log it.
Detection = frozen golden set → eval harness → baseline comparison → threshold gate (absolute floor + no-regression + per-category guard).
Silent provider change → log/alert on model_version, continuous canary eval, production distribution monitors.
Rollout safely: shadow → canary → A/B, trivial rollback.
Regression (known change) vs drift (over time). Use significance, not eyeballing.