Skip to content

ADR-005: MCP server for test/eval investigation

Status: Accepted · Date: 2026-08-23

Context

When a test fails, an engineer (or an AI assistant) must gather context from several places: the result record, the self-healing log, the RAG evaluation, the app logs. The Model Context Protocol (MCP) is the 2026-standard way to expose such capabilities to AI clients through typed, discoverable tools.

Decision

Expose a small MCP server (mcp_server/) with read-only tools: get_failed_tests, get_test_case, query_test_metrics, get_self_healing_events, get_rag_evaluation, get_application_logs.

  • Read-only by design. No tool mutates state, runs commands, or deletes data. A conformance test asserts no tool name implies a mutating action. This is the MCP analogue of the agent tool allowlist (ADR-004): an AI can investigate, not act destructively.
  • Tools are plain functions (mcp_server/tools.py) wrapped by the server (mcp_server/server.py). This keeps them unit-testable without a stdio client and decouples the capability from the transport.
  • Optional dependency. The mcp SDK is an extra; the platform installs and tests without it. The server supports both mcp 1.x (FastMCP) and 2.x (MCPServer).
  • Typed errors. Unknown ids / invalid params raise ToolError rather than returning misleading empty data.

How it helps QA

An AI agent can answer "why did UI-AUTH-002 fail?" by chaining get_failed_tests → get_test_case → get_self_healing_events → get_application_logs and summarising a probable root cause — the same signals the failure-triage classifier uses, now available to any MCP client.

Consequences

  • (+) Standard, discoverable interface; testable core; safe by construction.
  • (−) The result store must be populated (a pytest hook / CI step writes it).