← Back to Reliability Explorer
Measured selection evidence
GitMCP Docs
gitmcp-docs
Measured historical tool-selection reliability from frozen benchmark evidence.
Multi-batchS2-backedObserved missesMixed provenance
Mixed live and version-bound evidence. Historical evidence does not establish current endpoint availability or current production behavior.
3independent benchmark batches
35measured frozen cases
S1, S2evidence level(s)
3observed semantic misses
First measurement: 2026-09-10 13:23:12.077611 UTC · Last measurement: 2026-09-10 17:45:38.245198 UTC · Observed strict accuracy range: 66.7%–100.0%
Batch history
Each immutable benchmark batch is shown separately. No pooled accuracy is calculated here.
| Benchmark / batch | Type | Frozen cases | Strict score | Wilson 95% CI | Provider-valid coverage | Infrastructure failures | Evidence / measured |
|---|---|---|---|---|---|---|---|
selection-evidence-expansion-v1selection-evidence-expansion-v1 |
CONSTRUCTED_FROZEN_SELECTION_BENCHMARK | 24 | 24/24 100.0% |
Not recorded–Not recorded | 24 / 24 (100.0%) | 0 | S1 2026-09-10 13:23:12.077611 UTC |
external-gold-expansion-v2-batch2external-gold-expansion-v2-batch2 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 5 | 4/5 80.0% |
37.6%–96.4% | 5 / 5 (100.0%) | 0 | S2 2026-09-10 16:49:53.346446 UTC |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 6 | 4/6 66.7% |
30.0%–90.3% | 6 / 6 (100.0%) | 0 | S2 2026-09-10 17:45:38.245198 UTC |
Observed semantic misses
These are preserved adjudicated historical observations, attributable to a benchmark, batch, and case.
| Benchmark / batch | Case ID | Expected tool | Selected tool | Boundary family | Adjudication |
|---|---|---|---|---|---|
external-gold-expansion-v2-batch2external-gold-expansion-v2-batch2 |
gitmcp-hard-005 | search_generic_documentation |
fetch_generic_documentation | DOCUMENTED_BEHAVIOR_VS_IMPLEMENTATION | SEMANTIC_MISS_GOLD_REMAINS_VALID |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
b3-gm-005 | fetch_generic_url_content |
fetch_generic_documentation | DOCUMENTATION_VS_REFERENCED_URL | GOLD_REMAINS_VALID; semantic miss preserved in strict score. The public source supports the requested neighboring operation, and the selected tool is a plausible but incorrect neighboring retrieval mode. |
external-gold-expansion-v2-batch3external-gold-expansion-v2-batch3 |
b3-gm-006 | search_generic_documentation |
fetch_generic_documentation | DOCUMENTED_BEHAVIOR_VS_IMPLEMENTATION | GOLD_REMAINS_VALID; semantic miss preserved in strict score. The public source supports the requested neighboring operation, and the selected tool is a plausible but incorrect neighboring retrieval mode. |
Evidence provenance
Technical identities are secondary detail for auditability.
GitMCP Docs
- Canonical subject ID
gitmcp-docs- Provenance type
- CONSTRUCTED_FROZEN_SELECTION_BENCHMARK
- Identity mode
- LIVE_TOOLS_LIST_SNAPSHOT
- Identity source
LIVE_TOOLS_LIST
GitMCP
- Canonical subject ID
gitmcp-docs- Provenance type
- EXTERNAL_INDEPENDENT_GOLD_BENCHMARK
- Identity mode
- LIVE_TOOLS_LIST_SNAPSHOT
- Identity source
CURRENT_LIVE_TOOLS_LIST_FROZEN_BY_EXISTING_IMMUTABLE_SNAPSHOT
GitMCP
- Canonical subject ID
gitmcp-docs- Provenance type
- EXTERNAL_INDEPENDENT_GOLD_BENCHMARK
- Identity mode
- VERSION_BOUND_SNAPSHOT
- Identity source
CURRENT_OR_VERSION_BOUND_SNAPSHOT_REUSED_FROM_QUALIFIED_CORPUS
selection-evidence-expansion-v1 immutable identities
- case_freeze_sha256
5d248e845fc1b52ef53585682a97a592fecd0d3c1b8c9e561c758a4222273aed- freeze_commit_sha
d2937086b419ba0bf77517da772540de1975c45f- measurement_artifact_sha256
27f281b49b50fe5218102274c57db0110f3119091ca5dd53ebe954997f4fe826- model_lock_sha256
a8df78457b76e0dea73f66e2a75907feab3c42885db23658caada6973c5cb478- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d
external-gold-expansion-v2-batch2 immutable identities
- case_freeze_sha256
ce21fc085bd9a0185c9e37d0d795221f286397cbc477ee941e876be24f7ba1b4- final_freeze_receipt_commit
d3c55d2f96a4e45bf2577225ea16929877f2ded1- freeze_input_commit
61f6511c41562c24019cd441b37bc6ee07283951- measurement_artifact_sha256
dd13aec8a90b2b5490c934c9ad9650c37733c65f18d824e2ae6066202176ced9- measurement_commit
f0b6cd33be044d8bad800c20b4f2684e6d8431ce- model_lock_sha256
153647f94eed0ee3d3069229545846eef864fd43fa821f2d311dc3fb5d265ba9- policy_evaluation_commit
f0b6cd33be044d8bad800c20b4f2684e6d8431ce- policy_evaluation_sha256
736f154a5d5c3b8f0d77cd58fa0644ca97ad94c7a6b46af98d394c1dc1578f9a- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
8c4364ab598241ddbe4adb16453ba2e12b0917a1d1852caedb07884d1a54570b
external-gold-expansion-v2-batch3 immutable identities
- case_freeze_sha256
c1595743f48fe8ae6b3f70f2ea401731052b3bd7614297feabaa8172ce190ae5- final_freeze_receipt_commit
a309bda7e8dd2867ccc7894dc7d42efbfc48395e- freeze_input_commit
8450ed889756784a12ca4ebb8d84974d73b31778- immutable_measurement_receipt_commit
9d8290160849a5124d1896fc318d53470585ef41- measurement_artifact_sha256
7dcab9d4b71e7de59094c07cbca830ebf674b601ac6de024db0d3e677e7c0067- measurement_commit
b81cc816158f7c0fbe7f8ac0a6fcf2b3f38f4c86- model_lock_sha256
48b5ed9c98f16cc2b3b76bc0da2e7ec8072ecd89ba1914683181b1c20074a3a8- policy_evaluation_sha256
3ea5b62044b4da6e92812831ebb00205970849accffe763a270248a24a0adef4- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
c5cfe765f6b72489220cc73f689acb1c1c2c9284ffd8d093de0e597b645f6fbf