Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mcp-eval.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
How to run

Use native pytest to execute pytest-style tests. With uv: 
uv run pytest -q tests
# Or a single file/function
uv run pytest tests/test_pytest_style.py -v
uv run pytest tests/test_pytest_style.py::test_basic_fetch -v
Note: mcp-eval run executes decorator- and dataset-style tests. Use pytest for plain pytest tests.

​
Fixtures

  • mcp_session: access the session (metrics, assertions)
  • mcp_agent: a TestAgent bound to the session
Source: pytest_plugin.py

​
Markers

  • @pytest.mark.mcp_agent(<Agent|AgentSpec|name>): per‑test agent override
  • @pytest.mark.network, @pytest.mark.slow

​
Example

import pytest
from mcp_eval import Expect

@pytest.mark.asyncio
@pytest.mark.network
async def test_basic_fetch(mcp_agent):
    resp = await mcp_agent.generate_str("Fetch https://example.com")
    await mcp_agent.session.assert_that(Expect.tools.was_called("fetch"), response=resp)
    await mcp_agent.session.assert_that(Expect.content.contains("Example Domain"), response=resp)
Full examples: test_pytest_style.py

​
Per-test agent override

import pytest
from mcp_eval import Expect
from mcp_agent.agents.agent import Agent

@pytest.mark.asyncio
@pytest.mark.mcp_agent(Agent(name="custom", instruction="You fetch", server_names=["fetch"]))
async def test_with_custom_agent(mcp_agent):
    resp = await mcp_agent.generate_str("Fetch https://example.com")
    await mcp_agent.session.assert_that(Expect.tools.was_called("fetch"), response=resp)

​
Parametrization

@pytest.mark.asyncio
@pytest.mark.parametrize(
    "url,expected",
    [
        ("https://example.com", "Example Domain"),
        ("https://httpbin.org/html", "Herman Melville"),
    ],
)
async def test_urls(mcp_agent, url, expected):
    resp = await mcp_agent.generate_str(f"Fetch {url}")
    await mcp_agent.session.assert_that(Expect.content.contains(expected, case_sensitive=False), response=resp)

​
Parametrization

Use standard @pytest.mark.parametrize for breadth, or mix with mcp-eval’s @parametrize in decorator tests.