MCP Server Testing: The Honest Harness Approach

Why "it responded" isn't "it works"

An MCP server's test surface is unusual: its primary consumer is a model, not a fixed client. That means correctness has two separate layers that get conflated constantly. Integration tests verify the server responds — the process starts, the tool is discoverable, a call returns something. Evals verify it responds correctly — the tool description is unambiguous enough for a model to pick the right tool and pass the right arguments, and the result is actually useful for the task. A server can pass every integration test and still be unusable because the tool description confuses the model calling it.

A five-layer shape that holds up in practice

You don't need a novel framework here — the shape that keeps showing up in production MCP testing write-ups (and matches what I run) is roughly five gates, cheapest first:

• Smoke — does the server start, initialize, and advertise its tools/resources correctly? Cheapest test, catches the most embarrassing failures.
• Conformance — does it behave like a spec-compliant MCP server (correct error shapes, correct capability negotiation, no malformed responses)?
• Scenario / eval — given real prompts a user would actually send, does the model pick the right tool, with the right arguments, and get a usable result? This is the layer most homegrown test suites skip entirely, and it's non-deterministic — run it more than once and look at the distribution, not a single pass/fail.
• Fail-soft / load — what happens when a dependency is slow, rate-limited, or down? Does the server degrade to a clean error, or does it crash the session? What's the honest p99 under real concurrency, including the requests that stalled out — not just the ones that returned fast?
• Adversarial — what happens when the caller (or an injected prompt) sends malformed, oversized, or scope-violating input? This is the layer that catches the over-broad-credential problem before a real agent does.

A concrete fail-soft test, not just the concept

Talking about "fail-soft" in the abstract is easy; testing it is one assertion, forced against a simulated failure:

import pytest

async def test_refund_returns_clean_error_when_upstream_down(monkeypatch):
    async def raise_down(*a, **kw):
        raise billing_client.UpstreamDown()
    monkeypatch.setattr(billing_client, "refund", raise_down)

    result = await refund(order_id="ord_123", amount_cents=500)

    assert result["status"] == "error"
    assert result["retryable"] is True
    # the real assertion: the tool call did not raise past this boundary

That last comment is the point. A test that only checks the happy path never exercises the code path that actually causes 2am pages. If you don't have a test that forces each known failure mode (rate-limited, unauthorized, timeout, malformed upstream response) through the tool boundary, you don't know your fail-soft handling works — you're hoping.

The part that's actually hard: the kill bar

Here's where most testing write-ups stop, and where the honest version of this keeps going. Writing tests is a skill. Being willing to act on what they say — including "this doesn't clear the bar, park it" — is a discipline, and it's the one that's easy to skip under deadline pressure. The fix isn't more tests. It's writing the kill criterion down before you run the suite, so there's no room to move the goalposts after you see the number.

I run this discipline on my own systems, not just client work: a forward-testing harness that took one strategy to a real, unflattering verdict — sage-bot: -$735.66 over 1,002 paper trades — and retired it, because the cost-reality gate was written down before the trades ran, not adjusted after. The harness did its job by producing a trustworthy "no." An MCP server's kill bar looks the same in shape: pick the p99 latency, the fail-soft coverage percentage, or the eval pass rate that would mean "not ready," write it down, then run the suite and honor what it says.

What this looks like as a checklist

• Smoke + conformance tests exist and run on every change, not just before a demo.
• At least one forced-failure test per known dependency failure mode (down, slow, unauthorized, rate-limited).
• Scenario evals run against real prompts, more than once, with results tracked as a distribution — not a single green checkmark.
• A written kill bar exists before the suite runs, and someone other than the person who built the server has seen it.
• The results — including any that miss the bar — are recorded, not quietly rerun until they pass.