01 — Pytest (learn from scratch → interview-ready)¶
The JD names Pytest as the primary framework for Python backend + agent logic, so expect deep drilling. This one file takes you from "what even is a test runner" to "I own a Pytest suite with fixtures, mocking, async, parallelism and CI reporting" — and the interview one-liners to say out loud.
How to use this file: read top-to-bottom the first time (it builds every concept from zero with analogies and examples). Later, jump to the Rapid-fire recall at the bottom to revise. You only need basic Python (functions, import, dicts).
0. What is pytest, really?¶
When you write code you want to prove it works — and keep proving it as you change things. A test is a small piece of code that checks "given this input, my code does the right thing." A test runner finds all your tests, runs them, and reports pass/fail.
Pytest is that test runner for Python. Its whole selling point: tests are plain functions with plain assert. No ceremony.
def add(a, b):
return a + b
def test_add_two_numbers(): # a test = a function whose name starts with test_
assert add(2, 3) == 5 # "assert" = "I claim this is True; fail if it isn't"
Save as test_math.py, run pytest, and you see:
. is one passing test. Everything below is doing this well at scale.
Install and run¶
pip install pytest
pytest # discover and run every test
pytest -v # verbose: print each test name + PASS/FAIL
Compared with unittest (Java-style self.assertEqual), pytest is less boilerplate (plain assert), has powerful fixtures for setup/teardown, parametrize for data-driven tests, and a huge plugin ecosystem.
Interview line: "Pytest gives me plain-assert readability, fixtures for composable setup, and parametrize for data-driven coverage — with plugins for async, parallelism and coverage that plug straight into CI."
1. assert — how a test decides pass or fail¶
A test passes if it runs to the end without raising; it fails if any assert is False (or the code crashes).
def test_examples():
assert 2 + 2 == 4 # passes
assert "cat" in "concatenate" # passes (substring)
assert [1, 2, 3][0] == 1 # passes
The magic: when an assert fails, pytest rewrites it to show why. You write assert x == 5 and on failure see:
Checking that code raises on purpose¶
Sometimes correct behaviour is an error (dividing by zero should blow up). Use pytest.raises — the test passes only if the error happens in the block:
import pytest
def test_divide_by_zero_is_rejected():
with pytest.raises(ZeroDivisionError):
1 / 0 # if this does NOT raise, the test fails
2. How pytest finds tests (discovery)¶
You never register tests. Pytest scans using naming rules — follow them and your test is found automatically:
| Thing | Rule | Example |
|---|---|---|
| File | starts test_ or ends _test.py |
test_login.py |
| Function | starts test_ |
def test_valid_user(): |
| Class (optional) | starts Test, no __init__ |
class TestLogin: |
Name a file login_tests.py or a function check_login() and pytest silently ignores it — the classic "why aren't my tests running?" moment. When unsure, run pytest --collect-only to see exactly what was found.
Running just what you want¶
pytest # everything
pytest test_login.py # one file
pytest test_login.py::test_valid_user # one specific test (:: = file::test)
pytest -k "login and not admin" # any test whose NAME matches
pytest -m smoke # tests tagged with the 'smoke' marker (§5)
pytest -x # stop at first failure (fast feedback)
pytest --lf # re-run only last-failed
-k and --lf are the two you'll use hourly once a suite grows.
3. Fixtures — the concept most people find confusing (read slowly)¶
The problem fixtures solve¶
Most tests need setup first (a DB connection, a logged-in user, a configured client) and often cleanup after (close the connection, delete the test user). Copy-pasting that into every test makes them long, repetitive, and easy to forget to clean up.
The idea¶
A fixture is reusable setup (and optional cleanup) you write once. A test says "I need that" by naming the fixture in its parameter list. Pytest sees the name, runs the fixture, and hands the result in. This "ask by naming, pytest hands it to you" is dependency injection — fancy term, simple idea.
import pytest
@pytest.fixture # this decorator makes a fixture
def sample_user():
return {"name": "alice", "role": "admin"}
def test_user_is_admin(sample_user): # naming it = "give me sample_user"
assert sample_user["role"] == "admin"
sample_user() — you named it as a parameter and pytest called it for you. That's the whole trick.
Setup AND cleanup: yield¶
If a fixture needs cleanup, use yield instead of return. Before yield = setup; the yielded value = what the test gets; after yield = teardown, which runs after the test even if it failed.
@pytest.fixture
def db_connection():
conn = connect_to_test_db() # SETUP
yield conn # the test receives conn
conn.close() # TEARDOWN, guaranteed
def test_user_count(db_connection):
assert db_connection.count("users") >= 0
# conn.close() runs automatically when the test ends
yield is a "pause". Pytest runs setup, pauses to let the test run, then resumes for cleanup — "open ... use ... close" in one place.
Fixtures can use other fixtures¶
A fixture can itself name another fixture; pytest builds the chain:
@pytest.fixture
def base_url():
return "http://localhost:8000"
@pytest.fixture
def client(base_url): # depends on base_url
return HttpClient(base_url) # pytest resolves base_url first
def test_homepage(client): # only asks for client; base_url comes along
assert client.get("/").status == 200
Scope — "how often should this setup run?"¶
By default a fixture runs once per test (fresh setup, total isolation — the safe default). But some setup is expensive (a DB container, a browser, an auth token) and you don't want to redo it 500 times. Scope controls the frequency:
| Scope | Created… | Use for |
|---|---|---|
function (default) |
once per test | anything mutable — keeps tests isolated |
class |
once per test class | shared setup for a group |
module |
once per .py file |
e.g. one HTTP client for the file |
package |
once per package | rarer |
session |
once per whole run | expensive + read-only: DB container, browser, auth token |
@pytest.fixture(scope="session") # created ONCE, reused by the whole run
def auth_token():
return log_in_and_get_token() # slow — do it once
session fixture holds mutable state and test A changes it, test B secretly sees the change and now tests depend on each other. Rule of thumb: anything mutable stays function-scoped; only expensive, read-only things go wider.
conftest.py — share fixtures without importing¶
Put shared fixtures in a special file called conftest.py. Pytest auto-loads it, and every test in that directory tree can use those fixtures without importing anything — "fixtures on tap for this folder."
# conftest.py (at the test root)
import pytest
@pytest.fixture
def client():
return HttpClient("http://localhost:8000")
def test_x(client):.
Two power-ups you'll meet often¶
autouse — a fixture that runs without being asked. Good for "reset global state before every test."
@pytest.fixture(autouse=True) # every test gets this automatically
def reset_state():
Cache.clear()
yield
Factory fixture — when a test needs several things. Return a function that makes objects, and clean them all up at the end.
@pytest.fixture
def make_user():
created = []
def _make(name="alice", role="user"): # the test calls this as often as it likes
u = User.create(name=name, role=role)
created.append(u)
return u
yield _make
for u in created: # teardown removes everything the test made
u.delete()
def test_two_users(make_user):
a, b = make_user("alice"), make_user("bob")
assert a.id != b.id
4. Parametrize — run one test against many inputs¶
Checking the same logic against many input/output pairs? Don't copy-paste. @pytest.mark.parametrize runs the body once per row and reports each as a separate test:
@pytest.mark.parametrize("a, b, expected", [
(2, 3, 5), # case 1
(-1, 1, 0), # case 2
(0, 0, 0), # case 3
], ids=["positives", "mixed", "zeros"]) # friendly names in the output
def test_add(a, b, expected):
assert add(a, b) == expected
test_add[positives], [mixed], [zeros]. If one fails you know exactly which input broke — far better than one test with five asserts (which stops at the first failure and hides the rest).
Two advanced moves:
- Stacking parametrize decorators = the Cartesian product (all combinations run).
- Parametrize a fixture with params= to run a whole suite across variants (e.g. two DB backends):
@pytest.fixture(params=["sqlite", "postgres"])
def db(request):
return make_db(request.param) # every test using `db` runs twice, once per backend
5. Markers — labels you put on tests¶
A marker is a tag you stick on a test to group or control it, applied with @pytest.mark.<name>:
import pytest, sys
@pytest.mark.smoke # a custom label you invent
def test_login_works(): ...
@pytest.mark.skip(reason="feature not built yet") # never run
def test_future_feature(): ...
@pytest.mark.skipif(sys.platform == "win32", reason="POSIX only") # skip conditionally
def test_posix_paths(): ...
@pytest.mark.xfail(reason="known bug TICKET-42", strict=True) # we EXPECT this to fail
def test_known_bug(): ...
--strict-markers errors on typos):
[tool.pytest.ini_options]
markers = ["smoke: quick sanity gate", "slow: long-running", "live: hits real model"]
@pytest.mark.smoke, run pytest -m smoke on every PR for fast feedback, run the full suite nightly.
6. Mocking — faking the slow/external stuff (heavily asked)¶
The problem¶
Your code talks to things you don't control in a test: a payment API, a DB, an LLM. Calling them for real makes tests slow, flaky (network), and sometimes expensive/dangerous (real charges). You don't want that in a unit test.
The idea¶
Mocking replaces a real dependency with a fake you control: "when my code calls this, don't actually call it — return this — and let me check my code behaved correctly."
from unittest.mock import patch
def test_agent_uses_price_from_api():
with patch("app.agent.get_quote") as fake_quote: # replace get_quote with a fake
fake_quote.return_value = {"price": 100.0} # tell the fake what to return
result = run_agent("what's the price?")
fake_quote.assert_called_once() # our code actually called it
assert result["price"] == 100.0 # and used the value
The one rule everyone gets wrong¶
Patch where the thing is used, not where it's defined.
If app/agent.py does from app.api import get_quote, then inside agent.py the name lives at app.agent.get_quote — so patch "app.agent.get_quote", not "app.api.get_quote". Patching the original definition won't affect the copy the agent already imported. Also use spec= so the mock rejects calls the real object wouldn't accept; pytest-mock's mocker fixture is a cleaner wrapper than with patch(...).
monkeypatch — the built-in fixture for env vars / attributes¶
For simple swaps, pytest gives you monkeypatch, which auto-undoes the change after the test:
def test_reads_model_from_env(monkeypatch):
monkeypatch.setenv("MODEL", "claude-sonnet-4-6")
monkeypatch.setattr("app.config.TIMEOUT", 5)
assert load_config().timeout == 5
# env var + attribute restored automatically after the test
pytest -m live) run rarely. (More in 07 and 08.)
7. Async tests — agent code is usually async¶
Modern agent backends are async. Plain pytest can't await, so install pytest-asyncio and mark async tests:
import pytest
@pytest.mark.asyncio # run this coroutine properly
async def test_async_agent():
result = await agent.ainvoke({"input": "hi"}) # note: await
assert result["status"] == "ok"
asyncio_mode = "auto" in config to drop the marker. Async fixtures work the same with async def + yield. If you've never used async, all you need here: async functions must be awaited, and this plugin lets pytest do that.
8. Asserting on more than equality¶
assert takes any boolean expression:
assert response.status_code == 200
assert "error" not in response.text # negative check
assert set(payload) == {"id", "name", "role"} # exact key set, order-independent
assert response.elapsed.total_seconds() < 2.0 # a latency budget
assert result == pytest.approx(0.3) # float comparison with tolerance
0.1 + 0.2 == 0.3 is False in floating-point, so for anything numeric-with-rounding (including AI similarity/embedding scores) use pytest.approx. For LLM outputs, exact == doesn't work at all because text varies each run — that's the whole of 07.
9. Project structure, config & plugins¶
tests/
conftest.py # shared fixtures
unit/
test_pricing.py
integration/
test_agent_graph.py
e2e/
test_ui_approval.py
pyproject.toml # pytest config
[tool.pytest.ini_options]
addopts = "-ra -q --strict-markers" # -ra: summary of skips/fails; strict: typo'd markers error
testpaths = ["tests"]
asyncio_mode = "auto" # async tests need no marker
markers = ["smoke", "slow", "live"]
Plugins you'll add as the suite grows (each is pip install):
| Plugin | Purpose | Command |
|---|---|---|
| pytest-xdist | run tests in parallel | pytest -n auto |
| pytest-cov | coverage | pytest --cov=app --cov-report=xml |
| pytest-html | HTML report | pytest --html=report.html |
| pytest-asyncio | async tests | marker / auto mode |
| pytest-mock | mocker fixture | — |
| pytest-rerunfailures | retry flaky | --reruns 2 |
| pytest-timeout | kill hangs | --timeout=30 |
Parallelism caveat: -n auto only works if tests are truly independent (no two fighting over the same DB row) — use per-worker schemas or unique data. This is exactly why the fixture-scope discipline in §3 matters.
10. Flaky tests — the mindset that marks you as senior¶
A flaky test passes sometimes and fails other times with no code change. It's tempting to just re-run until green — don't; that hides real bugs. Find the cause:
| Cause | The real fix |
|---|---|
| Shared state between tests | narrower fixture scope + proper teardown (§3) |
| Timing (waiting on something) | wait for a condition, never time.sleep() |
| Test-order dependence | make each test self-contained; expose it with pytest-randomly |
| Genuine externality (real network) | mock it (§6) — or, if it must be live, a bounded retry (--reruns) |
Only that last row deserves a retry; pytest-rerunfailures is a safety net for genuine externalities, not a fix for bad tests.
Special note for AI testing: LLM tests are expected to vary run-to-run, so "flaky" isn't a defect there — you stop asserting exact text and assert with tolerance/meaning and a chosen threshold instead (see 07). That's the reason this role exists.
Interview line: "A rerun-to-green culture hides real bugs. I triage flakes by cause — state, timing, order, or genuine externality — and only the last one gets a bounded retry."
11. Try it yourself (10 minutes — cements everything)¶
Create test_cart.py and make these pass:
import pytest
class ShoppingCart:
def __init__(self):
self.items = []
def add(self, name, price):
if price < 0:
raise ValueError("price cannot be negative")
self.items.append((name, price))
def total(self):
return sum(p for _, p in self.items)
# 1) a fixture giving each test a fresh, empty cart
@pytest.fixture
def cart():
return ShoppingCart()
# 2) a basic test using the fixture
def test_empty_cart_total_is_zero(cart):
assert cart.total() == 0
# 3) parametrize: several price sets -> expected total
@pytest.mark.parametrize("prices, expected", [
([10, 20], 30),
([], 0),
([5], 5),
])
def test_totals(cart, prices, expected):
for i, p in enumerate(prices):
cart.add(f"item{i}", p)
assert cart.total() == expected
# 4) pytest.raises: a negative price must fail
def test_negative_price_rejected(cart):
with pytest.raises(ValueError):
cart.add("bad", -1)
pytest -v and read the output. You just used fixtures, parametrize, and pytest.raises together — ~80% of everyday pytest.
Then open practice-repo/ (50 runnable tests) and read the real agent-testing tests — they use every idea here.
Rapid-fire recall¶
- Test =
def test_*()withassert; runner finds it by thetest_name. pytest.raises= "I expect this error."- Fixture = setup you request by naming it;
yieldsplits setup/teardown; teardown = everything afteryield. - Scope = how often a fixture runs (
functiondefault →session); wide = fast but risky (keep mutable statefunction-scoped). - Share fixtures without import:
conftest.py. - Data-driven:
@pytest.mark.parametrize(stacking = Cartesian;params=on a fixture too). - Marker = a tag (
-m smoke) to select/skip tests; register in config with--strict-markers. - Fake a dependency:
patch/mocker— patch where used, not defined;monkeypatchfor env/attrs. - Run in parallel:
pytest -n auto(tests must be independent). - Float/embedding compare:
pytest.approx. Async:pytest-asyncio. - Flaky ≠ rerun-to-green; triage by cause (state/timing/order/externality).