{"schema_version":"mcp-selection-lab.per-mcp-benchmark.v1","title":"Per-MCP read-only selection routing benchmark evidence","validation_type":"read_only_public_mcp_metadata","sample_count":5,"tool_execution":false,"customer_adoption_proven":false,"global_model_discovery_ranking_proven":false,"samples":[{"sample_id":"context7","metadata":{"checked_at":"2026-09-06T12:56:08Z","mcp_url":"https://mcp.context7.com/mcp","name":"Context7 MCP","sample_id":"context7","source_url":"https://context7.com/docs/resources/all-clients","tool_count_evaluated":2,"tool_count_total":2,"tool_execution":false,"tools":[{"description":"Resolves a package/product name to a Context7-compatible library ID and returns matching libraries.\n\nYou MUST call this function before 'Query Documentation' tool to obtain a valid Context7-compatible library ID UNLESS the user explicitly provides a library ID in the format '/org/project' or '/org/project/version' in their query.\n\nEach result includes:\n- Library ID: Context7-compatible identifier (format: /org/project)\n- Name: Library or package name\n- Description: Short summary\n- Code Snippets: Number of available code examples\n- Source Reputation: Authority indicator (High, Medium, Low, or Unknown)\n- Benchmark Score: Quality indicator (100 is the highest score)\n- Versions: List of versions if available. Use one of those versions if the user provides a version in their query. The format of the version is /org/project/version.\n\nFor best results, select libraries based on name match, source reputation, snippet coverage, benchmark score, and relevance to your use case.\n\nSelection Process:\n1. Analyze the query to understand what library/package the user is looking for\n2. Return the most relevant match based on:\n- Name similarity to the query (exact matches prioritized)\n- Description relevance to the query's intent\n- Documentation coverage (prioritize libraries with higher Code Snippet counts)\n- Source reputation (consider libraries with High or Medium reputation more authoritative)\n- Benchmark Score: Quality indicator (100 is the highest score)\n\nResponse Format:\n- Return the selected library ID in a clearly marked section\n- Provide a brief explanation for why this library was chosen\n- If multiple good matches exist, acknowledge this but proceed with the most relevant one\n- If no good matches exist, clearly state this and suggest query refinements\n\nFor ambiguous queries, request clarification before proceeding with a best-guess match.\n\nIMPORTANT: Do not call this tool more than 3 times per question. If you cannot find what you need after 3 calls, use the best result you have.","input_schema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"libraryName":{"description":"Library name to search for and retrieve a Context7-compatible library ID. Use the official library name with proper punctuation — e.g., 'Next.js' instead of 'nextjs', 'Customer.io' instead of 'customerio', 'Three.js' instead of 'threejs'.","type":"string"},"query":{"description":"What to look up in the library's documentation. This is used to rank library results by relevance to what the user is trying to accomplish. The query is sent to the Context7 API for processing. Do not include any sensitive or confidential information such as API keys, passwords, credentials, personal data, or proprietary code in your query.","type":"string"}},"required":["query","libraryName"],"type":"object"},"mcp_definition":{"annotations":{"destructiveHint":false,"idempotentHint":true,"openWorldHint":true,"readOnlyHint":true},"description":"Resolves a package/product name to a Context7-compatible library ID and returns matching libraries.\n\nYou MUST call this function before 'Query Documentation' tool to obtain a valid Context7-compatible library ID UNLESS the user explicitly provides a library ID in the format '/org/project' or '/org/project/version' in their query.\n\nEach result includes:\n- Library ID: Context7-compatible identifier (format: /org/project)\n- Name: Library or package name\n- Description: Short summary\n- Code Snippets: Number of available code examples\n- Source Reputation: Authority indicator (High, Medium, Low, or Unknown)\n- Benchmark Score: Quality indicator (100 is the highest score)\n- Versions: List of versions if available. Use one of those versions if the user provides a version in their query. The format of the version is /org/project/version.\n\nFor best results, select libraries based on name match, source reputation, snippet coverage, benchmark score, and relevance to your use case.\n\nSelection Process:\n1. Analyze the query to understand what library/package the user is looking for\n2. Return the most relevant match based on:\n- Name similarity to the query (exact matches prioritized)\n- Description relevance to the query's intent\n- Documentation coverage (prioritize libraries with higher Code Snippet counts)\n- Source reputation (consider libraries with High or Medium reputation more authoritative)\n- Benchmark Score: Quality indicator (100 is the highest score)\n\nResponse Format:\n- Return the selected library ID in a clearly marked section\n- Provide a brief explanation for why this library was chosen\n- If multiple good matches exist, acknowledge this but proceed with the most relevant one\n- If no good matches exist, clearly state this and suggest query refinements\n\nFor ambiguous queries, request clarification before proceeding with a best-guess match.\n\nIMPORTANT: Do not call this tool more than 3 times per question. If you cannot find what you need after 3 calls, use the best result you have.","inputSchema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"libraryName":{"description":"Library name to search for and retrieve a Context7-compatible library ID. Use the official library name with proper punctuation — e.g., 'Next.js' instead of 'nextjs', 'Customer.io' instead of 'customerio', 'Three.js' instead of 'threejs'.","type":"string"},"query":{"description":"What to look up in the library's documentation. This is used to rank library results by relevance to what the user is trying to accomplish. The query is sent to the Context7 API for processing. Do not include any sensitive or confidential information such as API keys, passwords, credentials, personal data, or proprietary code in your query.","type":"string"}},"required":["query","libraryName"],"type":"object"},"name":"resolve-library-id","title":"Resolve Context7 Library ID"},"name":"resolve-library-id","selection_role":"candidate"},{"description":"Retrieves and queries up-to-date documentation and code examples from Context7 for any programming library or framework.\n\nYou must call 'Resolve Context7 Library ID' tool first to obtain the exact Context7-compatible library ID required to use this tool, UNLESS the user explicitly provides a library ID in the format '/org/project' or '/org/project/version' in their query.\n\nDo not call this tool more than 3 times per question.","input_schema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"libraryId":{"description":"Exact Context7-compatible library ID (e.g., '/mongodb/docs', '/vercel/next.js', '/supabase/supabase', '/vercel/next.js/v14.3.0-canary.87') retrieved from 'resolve-library-id' or directly from user query in the format '/org/project' or '/org/project/version'.","type":"string"},"query":{"description":"What to look up in the library's documentation, scoped to a single concept. Be specific and include relevant details, but keep each query to one topic — if the user's question spans multiple distinct concepts, make a separate call per concept instead of combining them, unless the question is about how the concepts interact. Good: 'How to set up authentication with JWT in Express.js' or 'React useEffect cleanup function examples'. Bad (too vague): 'auth' or 'hooks'. Bad (too broad): 'routing and auth and caching in Next.js'. The query is sent to the Context7 API for processing. Do not include any sensitive or confidential information such as API keys, passwords, credentials, personal data, or proprietary code in your query.","type":"string"}},"required":["libraryId","query"],"type":"object"},"mcp_definition":{"annotations":{"destructiveHint":false,"idempotentHint":true,"openWorldHint":true,"readOnlyHint":true},"description":"Retrieves and queries up-to-date documentation and code examples from Context7 for any programming library or framework.\n\nYou must call 'Resolve Context7 Library ID' tool first to obtain the exact Context7-compatible library ID required to use this tool, UNLESS the user explicitly provides a library ID in the format '/org/project' or '/org/project/version' in their query.\n\nDo not call this tool more than 3 times per question.","inputSchema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"libraryId":{"description":"Exact Context7-compatible library ID (e.g., '/mongodb/docs', '/vercel/next.js', '/supabase/supabase', '/vercel/next.js/v14.3.0-canary.87') retrieved from 'resolve-library-id' or directly from user query in the format '/org/project' or '/org/project/version'.","type":"string"},"query":{"description":"What to look up in the library's documentation, scoped to a single concept. Be specific and include relevant details, but keep each query to one topic — if the user's question spans multiple distinct concepts, make a separate call per concept instead of combining them, unless the question is about how the concepts interact. Good: 'How to set up authentication with JWT in Express.js' or 'React useEffect cleanup function examples'. Bad (too vague): 'auth' or 'hooks'. Bad (too broad): 'routing and auth and caching in Next.js'. The query is sent to the Context7 API for processing. Do not include any sensitive or confidential information such as API keys, passwords, credentials, personal data, or proprietary code in your query.","type":"string"}},"required":["libraryId","query"],"type":"object"},"name":"query-docs","title":"Query Documentation"},"name":"query-docs","selection_role":"candidate"}],"tools_list_sha256":"e89db16e46fecb1c8667fdb6d7c33deab05a04ea08960b312c34525cd994ca4a"},"evaluation":{"abstention_gate":{"after":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":12.0},"before":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":12.0},"method":"development-calibrated minimum lexical score; holdout never used for threshold selection","selected":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":12.0}},"development":{"after_accuracy":1.0,"after_confusion_matrix":{"query-docs":{"query-docs":2,"resolve-library-id":0},"resolve-library-id":{"query-docs":0,"resolve-library-id":2}},"before_accuracy":1.0,"before_confusion_matrix":{"query-docs":{"query-docs":2,"resolve-library-id":0},"resolve-library-id":{"query-docs":0,"resolve-library-id":2}},"delta":0.0},"development_case_count":4,"holdout":{"after_accuracy":1.0,"after_confusion_matrix":{"query-docs":{"query-docs":2,"resolve-library-id":0},"resolve-library-id":{"query-docs":0,"resolve-library-id":2}},"before_accuracy":1.0,"before_confusion_matrix":{"query-docs":{"query-docs":2,"resolve-library-id":0},"resolve-library-id":{"query-docs":0,"resolve-library-id":2}},"delta":0.0,"regression":false},"holdout_case_count":4,"negative_audit":{"after":{"case_count":7,"false_positive_count":0,"false_positive_rate":0.0,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"before":{"case_count":7,"false_positive_count":0,"false_positive_rate":0.0,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"query-docs","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"query-docs","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"regression":false},"rewrite_count":0,"rewrites":[],"selected_surface":"baseline","tool_count":2,"tool_execution":false}},{"sample_id":"exa","metadata":{"checked_at":"2026-09-06T12:56:08Z","mcp_url":"https://mcp.exa.ai/mcp","name":"Exa MCP","sample_id":"exa","source_url":"https://exa.ai/docs/reference/exa-mcp","tool_count_evaluated":2,"tool_count_total":2,"tool_execution":false,"tools":[{"description":"Search the web for any topic and get clean, ready-to-use content.\n\n      Best for: Finding current information, news, facts, people, companies, or answering questions about any topic.\n      Returns: Clean text content from top search results.\n\n      Query tips:\n      describe the ideal page, not keywords. \"blog post comparing React and Vue performance\" not \"React vs Vue\".\n      Use category:people / category:company to search through Linkedin profiles / companies respectively.\n      If highlights are insufficient, follow up with web_fetch_exa on the best URLs.","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"numResults":{"description":"Number of search results to return (default: 10).","type":"number"},"query":{"description":"Natural language search query. Should be a semantically rich description of the ideal page, not just keywords. Optionally include category:<type> (company, people) to focus results — e.g. 'category:people John Doe software engineer'.","minLength":1,"type":"string"}},"required":["query"],"type":"object"},"mcp_definition":{"annotations":{"destructiveHint":false,"idempotentHint":true,"openWorldHint":true,"readOnlyHint":true},"description":"Search the web for any topic and get clean, ready-to-use content.\n\n      Best for: Finding current information, news, facts, people, companies, or answering questions about any topic.\n      Returns: Clean text content from top search results.\n\n      Query tips:\n      describe the ideal page, not keywords. \"blog post comparing React and Vue performance\" not \"React vs Vue\".\n      Use category:people / category:company to search through Linkedin profiles / companies respectively.\n      If highlights are insufficient, follow up with web_fetch_exa on the best URLs.","execution":{"taskSupport":"forbidden"},"inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"numResults":{"description":"Number of search results to return (default: 10).","type":"number"},"query":{"description":"Natural language search query. Should be a semantically rich description of the ideal page, not just keywords. Optionally include category:<type> (company, people) to focus results — e.g. 'category:people John Doe software engineer'.","minLength":1,"type":"string"}},"required":["query"],"type":"object"},"name":"web_search_exa"},"name":"web_search_exa","selection_role":"candidate"},{"description":"Read a webpage's full content as clean markdown. Use after web_search_exa when highlights are insufficient or to read any URL.\n\nBest for: Extracting full content from known URLs. Batch multiple URLs in one call.\nReturns: Clean text content and metadata from the page(s).","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"maxCharacters":{"description":"Maximum characters to extract per page (default: 3000)","minimum":1,"type":"number"},"urls":{"description":"URLs to read. Batch multiple URLs in one call.","items":{"type":"string"},"type":"array"}},"required":["urls"],"type":"object"},"mcp_definition":{"annotations":{"destructiveHint":false,"idempotentHint":true,"openWorldHint":true,"readOnlyHint":true},"description":"Read a webpage's full content as clean markdown. Use after web_search_exa when highlights are insufficient or to read any URL.\n\nBest for: Extracting full content from known URLs. Batch multiple URLs in one call.\nReturns: Clean text content and metadata from the page(s).","execution":{"taskSupport":"forbidden"},"inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"maxCharacters":{"description":"Maximum characters to extract per page (default: 3000)","minimum":1,"type":"number"},"urls":{"description":"URLs to read. Batch multiple URLs in one call.","items":{"type":"string"},"type":"array"}},"required":["urls"],"type":"object"},"name":"web_fetch_exa"},"name":"web_fetch_exa","selection_role":"candidate"}],"tools_list_sha256":"1b423a672e0cb03a6e156d36ca2b8b2f8fe725542e909fc0ad225a9f9decd45a"},"evaluation":{"abstention_gate":{"after":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":7.0},"before":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":7.0},"method":"development-calibrated minimum lexical score; holdout never used for threshold selection","selected":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":7.0}},"development":{"after_accuracy":1.0,"after_confusion_matrix":{"web_fetch_exa":{"web_fetch_exa":2,"web_search_exa":0},"web_search_exa":{"web_fetch_exa":0,"web_search_exa":2}},"before_accuracy":1.0,"before_confusion_matrix":{"web_fetch_exa":{"web_fetch_exa":2,"web_search_exa":0},"web_search_exa":{"web_fetch_exa":0,"web_search_exa":2}},"delta":0.0},"development_case_count":4,"holdout":{"after_accuracy":1.0,"after_confusion_matrix":{"web_fetch_exa":{"web_fetch_exa":2,"web_search_exa":0},"web_search_exa":{"web_fetch_exa":0,"web_search_exa":2}},"before_accuracy":1.0,"before_confusion_matrix":{"web_fetch_exa":{"web_fetch_exa":2,"web_search_exa":0},"web_search_exa":{"web_fetch_exa":0,"web_search_exa":2}},"delta":0.0,"regression":false},"holdout_case_count":4,"negative_audit":{"after":{"case_count":7,"false_positive_count":1,"false_positive_rate":0.14285714285714285,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":1.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"web_search_exa","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"before":{"case_count":7,"false_positive_count":1,"false_positive_rate":0.14285714285714285,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":1.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"web_search_exa","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"web_fetch_exa","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"web_fetch_exa","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"regression":false},"rewrite_count":0,"rewrites":[],"selected_surface":"baseline","tool_count":2,"tool_execution":false}},{"sample_id":"cloudflare-docs","metadata":{"checked_at":"2026-09-06T12:56:08Z","mcp_url":"https://docs.mcp.cloudflare.com/mcp","name":"Cloudflare Documentation MCP","sample_id":"cloudflare-docs","source_url":"https://developers.cloudflare.com/agents/model-context-protocol/cloudflare/servers-for-cloudflare/","tool_count_evaluated":2,"tool_count_total":2,"tool_execution":false,"tools":[{"description":"Search the Cloudflare documentation.\n\n\t\tThis tool should be used to answer any question about Cloudflare products or features, including:\n\t\t- Workers, Pages, R2, Images, Stream, D1, Durable Objects, KV, Workflows, Hyperdrive, Queues\n\t\t- AI Search, Workers AI, Vectorize, AI Gateway, Browser Run\n\t\t- Zero Trust, Access, Tunnel, Gateway, Browser Isolation, WARP, DDOS, Magic Transit, Magic WAN\n\t\t- CDN, Cache, DNS, Zaraz, Argo, Rulesets, Terraform, Account and Billing\n\n\t\tResults are returned as semantically similar chunks to the query.\n\t\t","input_schema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"query":{"type":"string"}},"required":["query"],"type":"object"},"mcp_definition":{"annotations":{"readOnlyHint":true,"title":"Search Cloudflare docs"},"description":"Search the Cloudflare documentation.\n\n\t\tThis tool should be used to answer any question about Cloudflare products or features, including:\n\t\t- Workers, Pages, R2, Images, Stream, D1, Durable Objects, KV, Workflows, Hyperdrive, Queues\n\t\t- AI Search, Workers AI, Vectorize, AI Gateway, Browser Run\n\t\t- Zero Trust, Access, Tunnel, Gateway, Browser Isolation, WARP, DDOS, Magic Transit, Magic WAN\n\t\t- CDN, Cache, DNS, Zaraz, Argo, Rulesets, Terraform, Account and Billing\n\n\t\tResults are returned as semantically similar chunks to the query.\n\t\t","inputSchema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{"query":{"type":"string"}},"required":["query"],"type":"object"},"name":"search_cloudflare_documentation","outputSchema":{"$schema":"https://json-schema.org/draft/2020-12/schema","additionalProperties":false,"properties":{"results":{"items":{"additionalProperties":false,"properties":{"id":{"description":"Source file ID","type":"string"},"similarity":{"description":"Similarity score from AI Search","type":"number"},"text":{"description":"Matching documentation chunk text","type":"string"},"title":{"description":"Documentation page title","type":"string"},"url":{"description":"Developer documentation URL","type":"string"}},"required":["similarity","id","url","title","text"],"type":"object"},"type":"array"}},"required":["results"],"type":"object"}},"name":"search_cloudflare_documentation","selection_role":"candidate"},{"description":"ALWAYS read this guide before migrating Pages projects to Workers.","input_schema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"type":"object"},"mcp_definition":{"annotations":{"readOnlyHint":true,"title":"Get Pages migration guide"},"description":"ALWAYS read this guide before migrating Pages projects to Workers.","inputSchema":{"$schema":"https://json-schema.org/draft/2020-12/schema","properties":{},"type":"object"},"name":"migrate_pages_to_workers_guide"},"name":"migrate_pages_to_workers_guide","selection_role":"candidate"}],"tools_list_sha256":"2a4f9685b7c2bb42706485725527520a65ebcc828d938d64cd0762f40536d94a"},"evaluation":{"abstention_gate":{"after":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":4.0},"before":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":4.0},"method":"development-calibrated minimum lexical score; holdout never used for threshold selection","selected":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":4,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":4,"holdout_positive_recall":1.0,"threshold":4.0}},"development":{"after_accuracy":1.0,"after_confusion_matrix":{"migrate_pages_to_workers_guide":{"migrate_pages_to_workers_guide":2,"search_cloudflare_documentation":0},"search_cloudflare_documentation":{"migrate_pages_to_workers_guide":0,"search_cloudflare_documentation":2}},"before_accuracy":1.0,"before_confusion_matrix":{"migrate_pages_to_workers_guide":{"migrate_pages_to_workers_guide":2,"search_cloudflare_documentation":0},"search_cloudflare_documentation":{"migrate_pages_to_workers_guide":0,"search_cloudflare_documentation":2}},"delta":0.0},"development_case_count":4,"holdout":{"after_accuracy":1.0,"after_confusion_matrix":{"migrate_pages_to_workers_guide":{"migrate_pages_to_workers_guide":2,"search_cloudflare_documentation":0},"search_cloudflare_documentation":{"migrate_pages_to_workers_guide":0,"search_cloudflare_documentation":2}},"before_accuracy":1.0,"before_confusion_matrix":{"migrate_pages_to_workers_guide":{"migrate_pages_to_workers_guide":2,"search_cloudflare_documentation":0},"search_cloudflare_documentation":{"migrate_pages_to_workers_guide":0,"search_cloudflare_documentation":2}},"delta":0.0,"regression":false},"holdout_case_count":4,"negative_audit":{"after":{"case_count":7,"false_positive_count":3,"false_positive_rate":0.42857142857142855,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":1.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":3.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":3.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-browser-action","category":"confusing_negative","margin":1.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true}]},"before":{"case_count":7,"false_positive_count":3,"false_positive_rate":0.42857142857142855,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":1.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"migrate_pages_to_workers_guide","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":3.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":3.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-browser-action","category":"confusing_negative","margin":1.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"search_cloudflare_documentation","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true}]},"regression":false},"rewrite_count":1,"rewrites":[{"after_description":"Search the Cloudflare documentation.\n\n\t\tThis tool should be used to answer any question about Cloudflare products or features, including:\n\t\t- Workers, Pages, R2, Images, Stream, D1, Durable Objects, KV, Workflows, Hyperdrive, Queues\n\t\t- AI Search, Workers AI, Vectorize, AI Gateway, Browser Run\n\t\t- Zero Trust, Access, Tunnel, Gateway, Browser Isolation, WARP, DDOS, Magic Transit, Magic WAN\n\t\t- CDN, Cache, DNS, Zaraz, Argo, Rulesets, Terraform, Account and Billing\n\n\t\tResults are returned as semantically similar chunks to the query.","before_description":"Search the Cloudflare documentation.\n\n\t\tThis tool should be used to answer any question about Cloudflare products or features, including:\n\t\t- Workers, Pages, R2, Images, Stream, D1, Durable Objects, KV, Workflows, Hyperdrive, Queues\n\t\t- AI Search, Workers AI, Vectorize, AI Gateway, Browser Run\n\t\t- Zero Trust, Access, Tunnel, Gateway, Browser Isolation, WARP, DDOS, Magic Transit, Magic WAN\n\t\t- CDN, Cache, DNS, Zaraz, Argo, Rulesets, Terraform, Account and Billing\n\n\t\tResults are returned as semantically similar chunks to the query.\n\t\t","incoming_collisions":[],"outgoing_collisions":[],"tool":"search_cloudflare_documentation"}],"selected_surface":"baseline","tool_count":2,"tool_execution":false}},{"sample_id":"deepwiki","metadata":{"checked_at":"2026-09-06T12:56:08Z","mcp_url":"https://mcp.deepwiki.com/mcp","name":"DeepWiki MCP","sample_id":"deepwiki","source_url":"https://mcp.deepwiki.com/","tool_count_evaluated":3,"tool_count_total":3,"tool_execution":false,"tools":[{"description":"Ask any question about a GitHub repository and get an AI-powered, context-grounded response.","input_schema":{"properties":{"question":{"description":"The question to ask about the repository.","type":"string"},"repoName":{"anyOf":[{"type":"string"},{"items":{"type":"string"},"type":"array"}],"description":"GitHub repository or list of repositories (max 10) in owner/repo format."}},"required":["repoName","question"],"type":"object"},"mcp_definition":{"_meta":{"_fastmcp":{"tags":[]}},"description":"Ask any question about a GitHub repository and get an AI-powered, context-grounded response.","inputSchema":{"properties":{"question":{"description":"The question to ask about the repository.","type":"string"},"repoName":{"anyOf":[{"type":"string"},{"items":{"type":"string"},"type":"array"}],"description":"GitHub repository or list of repositories (max 10) in owner/repo format."}},"required":["repoName","question"],"type":"object"},"name":"ask_question","outputSchema":{"properties":{"result":{"type":"string"}},"required":["result"],"type":"object","x-fastmcp-wrap-result":true}},"name":"ask_question","selection_role":"candidate"},{"description":"View documentation about a GitHub repository.","input_schema":{"properties":{"repoName":{"description":"GitHub repository in owner/repo format (e.g. \"facebook/react\").","type":"string"}},"required":["repoName"],"type":"object"},"mcp_definition":{"_meta":{"_fastmcp":{"tags":[]}},"description":"View documentation about a GitHub repository.","inputSchema":{"properties":{"repoName":{"description":"GitHub repository in owner/repo format (e.g. \"facebook/react\").","type":"string"}},"required":["repoName"],"type":"object"},"name":"read_wiki_contents","outputSchema":{"properties":{"result":{"type":"string"}},"required":["result"],"type":"object","x-fastmcp-wrap-result":true}},"name":"read_wiki_contents","selection_role":"candidate"},{"description":"Get a list of documentation topics for a GitHub repository.","input_schema":{"properties":{"repoName":{"description":"GitHub repository in owner/repo format (e.g. \"facebook/react\").","type":"string"}},"required":["repoName"],"type":"object"},"mcp_definition":{"_meta":{"_fastmcp":{"tags":[]}},"description":"Get a list of documentation topics for a GitHub repository.","inputSchema":{"properties":{"repoName":{"description":"GitHub repository in owner/repo format (e.g. \"facebook/react\").","type":"string"}},"required":["repoName"],"type":"object"},"name":"read_wiki_structure","outputSchema":{"properties":{"result":{"type":"string"}},"required":["result"],"type":"object","x-fastmcp-wrap-result":true}},"name":"read_wiki_structure","selection_role":"candidate"}],"tools_list_sha256":"4652f898423a17a92c4a05ef4ccca758df48db56a50156e3ddb7ebcac3a2a122"},"evaluation":{"abstention_gate":{"after":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":6,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":6,"holdout_positive_recall":1.0,"threshold":5.0},"before":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":6,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":6,"holdout_positive_recall":1.0,"threshold":5.0},"method":"development-calibrated minimum lexical score; holdout never used for threshold selection","selected":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":6,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":6,"holdout_positive_recall":1.0,"threshold":5.0}},"development":{"after_accuracy":1.0,"after_confusion_matrix":{"ask_question":{"ask_question":2,"read_wiki_contents":0,"read_wiki_structure":0},"read_wiki_contents":{"ask_question":0,"read_wiki_contents":2,"read_wiki_structure":0},"read_wiki_structure":{"ask_question":0,"read_wiki_contents":0,"read_wiki_structure":2}},"before_accuracy":1.0,"before_confusion_matrix":{"ask_question":{"ask_question":2,"read_wiki_contents":0,"read_wiki_structure":0},"read_wiki_contents":{"ask_question":0,"read_wiki_contents":2,"read_wiki_structure":0},"read_wiki_structure":{"ask_question":0,"read_wiki_contents":0,"read_wiki_structure":2}},"delta":0.0},"development_case_count":6,"holdout":{"after_accuracy":1.0,"after_confusion_matrix":{"ask_question":{"ask_question":2,"read_wiki_contents":0,"read_wiki_structure":0},"read_wiki_contents":{"ask_question":0,"read_wiki_contents":2,"read_wiki_structure":0},"read_wiki_structure":{"ask_question":0,"read_wiki_contents":0,"read_wiki_structure":2}},"before_accuracy":1.0,"before_confusion_matrix":{"ask_question":{"ask_question":2,"read_wiki_contents":0,"read_wiki_structure":0},"read_wiki_contents":{"ask_question":0,"read_wiki_contents":2,"read_wiki_structure":0},"read_wiki_structure":{"ask_question":0,"read_wiki_contents":0,"read_wiki_structure":2}},"delta":0.0,"regression":false},"holdout_case_count":6,"negative_audit":{"after":{"case_count":7,"false_positive_count":1,"false_positive_rate":0.14285714285714285,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":2.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"before":{"case_count":7,"false_positive_count":1,"false_positive_rate":0.14285714285714285,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"ask_question","split":"holdout","top_score":2.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"ask_question","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"regression":false},"rewrite_count":0,"rewrites":[],"selected_surface":"baseline","tool_count":3,"tool_execution":false}},{"sample_id":"gitmcp-docs","metadata":{"checked_at":"2026-09-06T12:56:08Z","mcp_url":"https://gitmcp.io/docs","name":"GitMCP Documentation Server","sample_id":"gitmcp-docs","source_url":"https://gitmcp.io/docs","tool_count_evaluated":5,"tool_count_total":5,"tool_execution":false,"tools":[{"description":"Match a library name to an owner/repo. Don't use it if you have an owner and repo already. Use this first if only a library name was provided. If found - you can use owner and repo to call other tools. If not found - try to use the library name directly in other tools.","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"library":{"description":"The name of the library to try and match to an owner/repo.","type":"string"}},"required":["library"],"type":"object"},"mcp_definition":{"description":"Match a library name to an owner/repo. Don't use it if you have an owner and repo already. Use this first if only a library name was provided. If found - you can use owner and repo to call other tools. If not found - try to use the library name directly in other tools.","inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"library":{"description":"The name of the library to try and match to an owner/repo.","type":"string"}},"required":["library"],"type":"object"},"name":"match_common_libs_owner_repo_mapping"},"name":"match_common_libs_owner_repo_mapping","selection_role":"candidate"},{"description":"Fetch documentation for any GitHub repository by providing owner and project name","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo"],"type":"object"},"mcp_definition":{"description":"Fetch documentation for any GitHub repository by providing owner and project name","inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo"],"type":"object"},"name":"fetch_generic_documentation"},"name":"fetch_generic_documentation","selection_role":"candidate"},{"description":"Semantically search in documentation for any GitHub repository by providing owner, project name, and search query. Useful for specific queries.","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"query":{"description":"The search query to find relevant documentation","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo","query"],"type":"object"},"mcp_definition":{"description":"Semantically search in documentation for any GitHub repository by providing owner, project name, and search query. Useful for specific queries.","inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"query":{"description":"The search query to find relevant documentation","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo","query"],"type":"object"},"name":"search_generic_documentation"},"name":"search_generic_documentation","selection_role":"candidate"},{"description":"Search for code in any GitHub repository by providing owner, project name, and search query. Returns matching files. Supports pagination with 30 results per page.","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"page":{"description":"Page number to retrieve (starting from 1). Each page contains 30 results.","type":"number"},"query":{"description":"The search query to find relevant code files","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo","query"],"type":"object"},"mcp_definition":{"description":"Search for code in any GitHub repository by providing owner, project name, and search query. Returns matching files. Supports pagination with 30 results per page.","inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"owner":{"description":"The GitHub repository owner (username or organization)","type":"string"},"page":{"description":"Page number to retrieve (starting from 1). Each page contains 30 results.","type":"number"},"query":{"description":"The search query to find relevant code files","type":"string"},"repo":{"description":"The GitHub repository name","type":"string"}},"required":["owner","repo","query"],"type":"object"},"name":"search_generic_code"},"name":"search_generic_code","selection_role":"candidate"},{"description":"Generic tool to fetch content from any absolute URL, respecting robots.txt rules. Use this to retrieve referenced urls (absolute urls) that were mentioned in previously fetched documentation.","input_schema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"url":{"description":"The URL of the document or page to fetch","type":"string"}},"required":["url"],"type":"object"},"mcp_definition":{"description":"Generic tool to fetch content from any absolute URL, respecting robots.txt rules. Use this to retrieve referenced urls (absolute urls) that were mentioned in previously fetched documentation.","inputSchema":{"$schema":"http://json-schema.org/draft-07/schema#","additionalProperties":false,"properties":{"url":{"description":"The URL of the document or page to fetch","type":"string"}},"required":["url"],"type":"object"},"name":"fetch_generic_url_content"},"name":"fetch_generic_url_content","selection_role":"candidate"}],"tools_list_sha256":"54a77b06220dae5919089e788ac36e1b1af00ad6f8ad7b052dd70bf8666d27c3"},"evaluation":{"abstention_gate":{"after":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":10,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":10,"holdout_positive_recall":1.0,"threshold":6.0},"before":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":10,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":10,"holdout_positive_recall":1.0,"threshold":6.0},"method":"development-calibrated minimum lexical score; holdout never used for threshold selection","selected":{"calibration_used_holdout":false,"development_false_positive_rate":0.0,"development_negative_count":4,"development_negative_specificity":1.0,"development_positive_count":10,"development_positive_recall":1.0,"holdout_false_positive_rate":0.0,"holdout_negative_count":3,"holdout_negative_specificity":1.0,"holdout_positive_count":10,"holdout_positive_recall":1.0,"threshold":6.0}},"development":{"after_accuracy":1.0,"after_confusion_matrix":{"fetch_generic_documentation":{"fetch_generic_documentation":2,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"fetch_generic_url_content":{"fetch_generic_documentation":0,"fetch_generic_url_content":2,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"match_common_libs_owner_repo_mapping":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":2,"search_generic_code":0,"search_generic_documentation":0},"search_generic_code":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":2,"search_generic_documentation":0},"search_generic_documentation":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":2}},"before_accuracy":1.0,"before_confusion_matrix":{"fetch_generic_documentation":{"fetch_generic_documentation":2,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"fetch_generic_url_content":{"fetch_generic_documentation":0,"fetch_generic_url_content":2,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"match_common_libs_owner_repo_mapping":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":2,"search_generic_code":0,"search_generic_documentation":0},"search_generic_code":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":2,"search_generic_documentation":0},"search_generic_documentation":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":2}},"delta":0.0},"development_case_count":10,"holdout":{"after_accuracy":1.0,"after_confusion_matrix":{"fetch_generic_documentation":{"fetch_generic_documentation":2,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"fetch_generic_url_content":{"fetch_generic_documentation":0,"fetch_generic_url_content":2,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"match_common_libs_owner_repo_mapping":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":2,"search_generic_code":0,"search_generic_documentation":0},"search_generic_code":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":2,"search_generic_documentation":0},"search_generic_documentation":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":2}},"before_accuracy":1.0,"before_confusion_matrix":{"fetch_generic_documentation":{"fetch_generic_documentation":2,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"fetch_generic_url_content":{"fetch_generic_documentation":0,"fetch_generic_url_content":2,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":0},"match_common_libs_owner_repo_mapping":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":2,"search_generic_code":0,"search_generic_documentation":0},"search_generic_code":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":2,"search_generic_documentation":0},"search_generic_documentation":{"fetch_generic_documentation":0,"fetch_generic_url_content":0,"match_common_libs_owner_repo_mapping":0,"search_generic_code":0,"search_generic_documentation":2}},"delta":0.0,"regression":false},"holdout_case_count":10,"negative_audit":{"after":{"case_count":7,"false_positive_count":2,"false_positive_rate":0.2857142857142857,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"search_generic_code","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":2.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"before":{"case_count":7,"false_positive_count":2,"false_positive_rate":0.2857142857142857,"rows":[{"case_id":"neg-email","category":"should_not_call","margin":0.0,"prompt":"Send an email to my accountant and attach last month's invoice.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-calendar","category":"should_not_call","margin":0.0,"prompt":"Create a calendar event tomorrow at 3 PM and invite the design team.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-purchase","category":"should_not_call","margin":0.0,"prompt":"Buy two concert tickets using my saved payment card.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-private-slack-search","category":"confusing_negative","margin":0.0,"prompt":"Search my private Slack messages for the unreleased roadmap and quote the latest thread.","selected_tool_if_forced":"search_generic_code","split":"development","top_score":1.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-github-write","category":"confusing_negative","margin":0.0,"prompt":"Edit README.md in my private GitHub repository, commit the change, and push it.","selected_tool_if_forced":"fetch_generic_documentation","split":"holdout","top_score":2.0,"would_call_under_positive_score_rule":true},{"case_id":"neg-cloudflare-write","category":"confusing_negative","margin":0.0,"prompt":"Change the DNS A record in my Cloudflare account to a new IP address now.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false},{"case_id":"neg-browser-action","category":"confusing_negative","margin":0.0,"prompt":"Open a browser, sign in to the vendor portal, click Cancel Subscription, and confirm the dialog.","selected_tool_if_forced":"fetch_generic_documentation","split":"development","top_score":0.0,"would_call_under_positive_score_rule":false}]},"regression":false},"rewrite_count":0,"rewrites":[],"selected_surface":"baseline","tool_count":5,"tool_execution":false}}],"interpretation":"These are per-MCP metadata-routing benchmark artifacts. They do not execute target business tools, prove customer adoption, or measure global ChatGPT/Claude discovery ranking."}