Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mcp-eval.ai/llms.txt

Use this file to discover all available pages before exploring further.

​
Run action

The recommended approach is to use the reusable workflow which handles all the setup, testing, and deployment. For the complete workflow configuration, visit mcpeval.yml. 
name: MCP-Eval CI

on:
  push:
    branches: [main, master, trunk]
  workflow_dispatch:

jobs:
  call-mcpeval:
    uses: lastmile-ai/mcp-eval/.github/workflows/mcpeval-reusable.yml
    with:
      deploy-pages: true
    permissions:
      contents: read
      pages: write
      id-token: write
      pull-requests: write
    secrets: inherit
Alternatively, you can directly use the action in your workflow: 
steps:
  - name: Checkout
    uses: actions/checkout@v4

  - name: Run MCP-Eval (uv)
    id: mcpeval
    uses: lastmile-ai/mcp-eval/.github/actions/mcp-eval/run
    with:
      python-version: "3.11"
      working-directory: .
      run-args: "-v"
      tests: tests/
      reports-dir: mcpeval-reports
      json-report: mcpeval-results.json
      markdown-report: mcpeval-results.md
      html-report: mcpeval-results.html
      artifact-name: mcpeval-artifacts
      pr-comment: "true"
      set-summary: "true"
      upload-artifacts: "true"
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  - name: Generate and upload badges
    uses: lastmile-ai/mcp-eval/.github/actions/mcp-eval/badges
    with:
      report-path: ${{ steps.mcpeval.outputs.results-json-path }}
      output-dir: badges
      format: "both"
      upload-artifacts: "true"
      artifact-name: mcpeval-badges
Sources:

​
Publish HTML and Badges via Pages

The workflow automatically deploys both the HTML report and badges to GitHub Pages when pushing to main/master branches. After deployment, badges are accessible at https://<username>.github.io/<repo>/badges/. To enable GitHub Pages deployment:
  1. Enable GitHub Pages in your repository settings
  2. The workflow will automatically deploy on pushes to main/master
  3. Badges and reports will be available at your Pages URL

​
Badges

After deployment to GitHub Pages, reference badges using your Pages URL: 
[![mcp-tests](https://img.shields.io/endpoint?url=https://YOUR_USERNAME.github.io/YOUR_REPO/badges/mcp-tests.json&cacheSeconds=300)](https://YOUR_USERNAME.github.io/YOUR_REPO/)
[![mcp-cov](https://img.shields.io/endpoint?url=https://YOUR_USERNAME.github.io/YOUR_REPO/badges/mcp-cov.json&cacheSeconds=300)](https://YOUR_USERNAME.github.io/YOUR_REPO/)
For example, mcp-tests mcp-cov