← Back to Reliability Explorer
Measured selection evidence
NixOS
nixos-mcp
Measured historical tool-selection reliability from frozen benchmark evidence.
Multi-batchS2-backedObserved missesVersion-bound
Version-bound source snapshot. Historical evidence does not establish current endpoint availability or current production behavior.
3independent benchmark batches
21measured frozen cases
S2evidence level(s)
1observed semantic misses
First measurement: 2026-09-10 18:47:01.455700 UTC · Last measurement: 2026-09-11 00:07:39.419700 UTC · Observed strict accuracy range: 83.3%–100.0%
Batch history
Each immutable benchmark batch is shown separately. No pooled accuracy is calculated here.
| Benchmark / batch | Type | Frozen cases | Strict score | Wilson 95% CI | Provider-valid coverage | Infrastructure failures | Evidence / measured |
|---|---|---|---|---|---|---|---|
external-gold-expansion-v2-batch4external-gold-expansion-v2-batch4 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 6 | 5/6 83.3% |
43.6%–97.0% | 6 / 6 (100.0%) | 0 | S2 2026-09-10 18:47:01.455700 UTC |
external-gold-expansion-v2-batch5external-gold-expansion-v2-batch5 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 7 | 7/7 100.0% |
64.6%–100.0% | 7 / 7 (100.0%) | 0 | S2 2026-09-10 19:13:49.842728 UTC |
external-gold-expansion-v2-batch6external-gold-expansion-v2-batch6 |
EXTERNAL_INDEPENDENT_GOLD_BENCHMARK | 8 | 8/8 100.0% |
67.6%–100.0% | 8 / 8 (100.0%) | 0 | S2 2026-09-11 00:07:39.419700 UTC |
Observed semantic misses
These are preserved adjudicated historical observations, attributable to a benchmark, batch, and case.
| Benchmark / batch | Case ID | Expected tool | Selected tool | Boundary family | Adjudication |
|---|---|---|---|---|---|
external-gold-expansion-v2-batch4external-gold-expansion-v2-batch4 |
b4-nx-001 | home_manager_search |
home_manager_info | HOME_MANAGER_VS_DARWIN_SEARCH | VALID_GOLD_RETAINED |
Evidence provenance
Technical identities are secondary detail for auditability.
NixOS
- Canonical subject ID
nixos-mcp- Provenance type
- EXTERNAL_INDEPENDENT_GOLD_BENCHMARK
- Identity mode
- VERSION_BOUND_SNAPSHOT
- Identity source
VERSION_BOUND_RECONSTRUCTED_SOURCE_SNAPSHOT
external-gold-expansion-v2-batch4 immutable identities
- case_freeze_sha256
a9513c1cb9b800b75d822d75fa8c6e47041e551d86a75699f1858b6c3785204d- final_freeze_receipt_commit
224aeab80db539383018d77cc7adb849976f1db5- freeze_input_commit
003985017fbd8c1e207fb146329b97b38582e606- immutable_measurement_receipt_sha256
7d6d0a1ac4b1d501aa4c5d0044b4a4e50bb319b272bd33eb783aec6499d594c6- measurement_artifact_commit
9b10dc0c99c4eed15a3d370743e8f46439302bae- measurement_artifact_sha256
7d9a5f57e0897f2ca8b2734ac0f38dc5afd099b2edf2495dbfa25e41f088cee9- model_lock_sha256
ffc1ec351c1a62c3692cdaedba0c5a6ec2c6285923eb065e0cd31953f978e3a1- policy_evaluation_sha256
0e9b11360db41731ce139584060b3f1417ce50b31dcf86ef88180e391e56e0d5- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
869812a69814f58e73a1c6c3f94a28b6dabc2cdb729285d3a532dd528ad36c89- scanner_freeze_sha256
e31caf3131b4e03f54ff411a386bb72ac70c35fbe9027e3383b7026019c22ae1
external-gold-expansion-v2-batch5 immutable identities
- case_freeze_sha256
40847d041af701492505676ae027f824248a94dba8dad19c4c764baec224a2bc- final_freeze_receipt_commit
4c11daed10b52027a8a31130703335e4835873ec- freeze_input_commit
935e16c4cc073d72b1c64c48206196feaf28c170- immutable_measurement_receipt_commit
b064dea6a0ea488826d11fc0fc88c5b4de3d5c26- measurement_artifact_commit
87a78aba4797f79f3c99c22e202eddd1da70f034- measurement_artifact_sha256
9b5f0c7f94783f2b25d28d8ecdaebe3a5a11b9db00dd84f677f95ac008f32eeb- model_lock_sha256
60c5653e461180e44e9621a54bb77f3f474e6889ddc29d1ff30d3cb9398013fa- policy_evaluation_sha256
2ef31f2a44a722fb862425dc11755b118c3ffb712672471d10d4cd0bbcee9af1- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
612c995839a9ad1a32d87320055ada3201c511b626a0f3e8778a8bbe5201bc68- scanner_freeze_sha256
80b3b0d60a5ec2e3b5019d2c45838dd06517453ce64c9a32ed0dbda6be718795- validation_criteria_sha256
0cd5821683b9f85ae37140bfc73d25064012d69aabbbbb4363e56b058cb240a6
external-gold-expansion-v2-batch6 immutable identities
- case_freeze_sha256
4e7500bb9cc1bb4c2b1bbdc757814a7eb2dd5eac48f131665328c1d886f6960b- final_freeze_receipt_commit
6b1957bf7d556dbf023ac24cf22e50bcd64b0136- freeze_input_commit
95d1a1017ab4b0fde8dfb3dbab3140997867a07b- immutable_measurement_receipt_binding_commit
df7c3c0a04b91e4d113e445f1ed4d78a212fe6dd- measurement_artifact_commit
8854ae3a9e69abfc09cd63fb0daaf9fe5b34fc63- measurement_artifact_sha256
e662d13721d11587fb102fe4699b16906c566d1342286a1c946424653aacd912- model_lock_sha256
c020241636261d34903e5155cb6177402ce658c0f6693f3efaf75bd3946829ee- policy_evaluation_sha256
94bbf82474a26461f303406b30a671ef776c31cadbd229e627e114c538acffd1- promotion_policy_sha256
6714ca504b2c5679c0f1ba0c7da2e5ee52f61b70b0e814ecaf9071f080f9f61d- raw_predictions_sha256
ed1eac7db87bf2be0dbc9f94acf79a2f72cc9a5dc9dbb33e5935a0a5c8971c51- scanner_freeze_sha256
b21d5f9693cecf1d29554e4064de814c9b3d64a2c41ea9b446430cd0e68a0a67- scanner_validation_result_sha256
c182332ae5ec6829b68054f1575066c35bf89b8d088b65322d5798e1d8042192- validation_criteria_sha256
6c081fe020106093c25f85b40eb2960faee1bd88337ca3688f231573f06b36b5