MCP Selection Lab · server.json · ard.json · llms.txt

10 non-owned MCP cross-provider selection benchmark

Read-only public MCP metadata validation. No target business tool was executed, and this is not external-customer adoption evidence.

• 10/10 public MCP metadata surfaces completed
• 35 tools in the combined distractor pool
• Development routing accuracy: 0.943 → 0.957
• Untouched holdout routing accuracy: 0.886 → 0.857
• Frozen calibrated holdout no-tool false-positive rate: 0.333
• Selected surface: baseline

Top machine-only issue candidates

1. deepwiki: 5 combined collision losses
2. gitmcp-docs: 3 combined collision losses
3. aws-knowledge: 2 combined collision losses
4. cloudflare-docs: 2 combined collision losses
5. microsoft-learn: holdout positive recall 0.833

Interpretation

A metadata rewrite that improved development routing accuracy reduced untouched holdout accuracy, so the benchmark retained the baseline. This bounded experiment measures routing among supplied MCP metadata; it does not measure global ChatGPT or Claude discovery ranking and is not customer-adoption evidence.

Agents can connect directly at https://mcp-selection-lab-production.up.railway.app/mcp. The scan reads public MCP metadata and does not execute target tools.