{"title":"10 non-owned MCP cross-provider selection benchmark","checked_at":"2026-09-06T16:13:55Z","validation_type":"read_only_public_mcp_metadata","customer_adoption_proven":false,"tool_execution":false,"sample_count":10,"success_count":10,"combined_tool_count":35,"combined":{"development_before_accuracy":0.9428571428571428,"development_after_accuracy":0.9571428571428572,"holdout_before_accuracy":0.8857142857142857,"holdout_after_accuracy":0.8571428571428571,"calibrated_holdout_positive_recall":1.0,"calibrated_holdout_false_positive_rate":0.33333333333333337,"calibration_used_holdout":false,"raw_negative_false_positive_rate":1.0,"selected_surface":"baseline"},"top_issue_candidates":[{"sample_id":"deepwiki","combined_collision_losses":5},{"sample_id":"gitmcp-docs","combined_collision_losses":3},{"sample_id":"aws-knowledge","combined_collision_losses":2},{"sample_id":"cloudflare-docs","combined_collision_losses":2},{"sample_id":"microsoft-learn","holdout_positive_recall":0.8333333333333334}],"interpretation":"A metadata rewrite that improved development routing accuracy reduced untouched holdout accuracy, so the benchmark retained the baseline. This bounded experiment measures routing among supplied MCP metadata; it does not measure global ChatGPT or Claude discovery ranking and is not customer-adoption evidence."}