09 — Langfuse Traces for Agent Validation¶

JD (required awareness): "Understanding of what Langfuse traces contain and how they can be used for agent validation." and (nice-to-have) hands-on trace validation, eval pipeline setup, confidence-score assertion. You don't need production hours — you need to speak the model precisely.

1. What Langfuse is¶

In plain words: Langfuse is an LLM observability & evaluation platform. Your agent emits a detailed record of every run — the prompts, model calls, tool calls, latency, cost, and outputs — and Langfuse stores it as a structured trace you can inspect, score, and build eval pipelines on. Think "distributed tracing (like APM) but for LLM/agent execution", plus a datasets + scoring + evaluation layer.

Interview line: "Langfuse is observability plus evaluation for LLM apps: every agent run becomes a structured trace of nested spans — model generations, tool calls, retrievals — with inputs, outputs, latency, cost and attached scores. That trace is exactly the execution surface I want to assert on."

2. The Langfuse data model (know these terms cold)¶

Object	In plain words
Trace	One end-to-end request/run (e.g. one user turn through the agent graph). Top-level container.
Observation	A step inside a trace. Three types below.
↳ Span	A unit of work with a duration (e.g. "retrieval", "policy check", a graph node). Can nest.
↳ Generation	A specific LLM call — captures model name/version, prompt, completion, token usage, cost, latency.
↳ Event	A point-in-time log marker.
Score	A numeric/categorical/boolean evaluation attached to a trace or observation (e.g. faithfulness=0.9, `pass`=true). From humans, code, or an LLM-judge/eval.
Session	Groups multiple traces (a multi-turn conversation).
Dataset / Dataset item	Golden inputs + expected outputs used for offline eval runs.
Metadata / tags	Arbitrary context (user, env, model_version, correlation_id).

So a single agent run's trace looks like:

Trace: "release payment 42"   (metadata: model_version, correlation_id, user)
├─ Span: classify_intent
│   └─ Generation: gpt/claude call (tokens, cost, latency, prompt, output)
├─ Span: check_policy (tool)      input={amount,acct}  output={allowed:false}
├─ Span: enqueue_approval
└─ Generation: final_answer       output="..."
  Scores: faithfulness=0.92, tool_correct=true, latency_ok=true

3. What a trace lets you validate (the JD's "reasoning chains, confidence scores, execution paths")¶

Execution path — which spans/tools fired and in what order (ties directly to file 08). "Was check_policy a span, and did it precede release_payment?"
Reasoning chain — the sequence of generations + their inputs/outputs; you can inspect whether each step's reasoning used the right context.
Confidence scores — assert the agent's confidence is within range and correlates with correctness; flag low-confidence answers that were nonetheless acted on.
Cost & latency — token usage and per-step latency (budget gates: file 06).
Model version — captured on every generation → detect silent model changes (file 10).
Grounding — retrieved context vs final answer (feed to Ragas, file 11).

4. Two ways to use Langfuse in testing¶

a) Observability-assisted debugging (read traces)¶

When a test fails or a user complains, open the trace: see the exact prompt, which tool returned bad data, where latency spiked, what the model actually output. Far faster than reproducing locally. (Pairs with Azure Monitor/App Insights, file 12.)

b) Trace-based assertions & eval pipelines (test on traces)¶

Fetch traces via the SDK/API and assert programmatically in CI:

from langfuse import Langfuse
lf = Langfuse()   # keys from env

def test_trace_took_correct_path():
    trace = lf.get_trace(trace_id)                     # or fetch by tag/session
    span_names = [o.name for o in trace.observations]
    assert "check_policy" in span_names
    assert span_names.index("check_policy") < span_names.index("release_payment")

    # confidence score assertion
    conf = next(s.value for s in trace.scores if s.name == "confidence")
    assert conf >= 0.7

    # cost / latency budget
    assert sum(o.calculated_total_cost or 0 for o in trace.observations) < 0.05

Attach scores from your eval so they're tracked over time:

lf.score(trace_id=trace_id, name="faithfulness", value=0.92)
lf.score(trace_id=trace_id, name="tool_correct", value=True)

c) Dataset-driven offline eval (the eval pipeline)¶

Put golden inputs + expected outputs in a Langfuse dataset.
Run the agent over every item, creating a trace per item linked to the dataset run.
Attach scores (judge/Ragas/exact where valid).
Compare the run's aggregate scores to the previous baseline → regression gate (file 10).

(In this kit's practice-repo, agent_graph.py + trace.py emit a Langfuse-shaped trace object so you can practise these exact assertions offline, without Langfuse installed.)

5. How Langfuse assertions become CI gates¶

# Azure DevOps step (file 06)
- script: pytest tests/traces --junitxml=trace.xml   # assert path/scores/budget on traces
- script: python -m eval.langfuse_run --dataset golden --min-score 0.85 --baseline baseline.json

Fail the build if: a required span/tool is missing, path order is wrong, a score is below threshold, cost/latency exceeds budget, or aggregate score regressed vs baseline.

6. Honest framing if asked "have you used it in production?"¶

"Not in production yet — the JD notes that's developable on the job. But I understand the data model precisely: traces made of spans, generations and events, with scores attached, and datasets for offline eval runs. I've practised the assertion patterns — validating execution path, confidence scores, cost/latency budgets, and regression against a baseline — and they map directly onto the trace-as-test-surface approach I use for agent graphs." (Then reference this kit's practice-repo.)

Rapid-fire recall¶

Langfuse = observability + eval for LLM/agent apps.
Data model: Trace → Observations (Span / Generation / Event) → Scores; Sessions group traces; Datasets drive offline eval.
A Generation captures model version, tokens, cost, latency — key for regression detection.
Validate: execution path/order, reasoning chain, confidence score, cost/latency, grounding, model version.
Trace assertions + dataset eval runs → CI gate on missing spans / low scores / regression.