IBIB：转向服务路径而非模型标识的企业级 AI 评测协议

My Analysis of a Complex Evaluation Protocol

Alright, let me distill these thoughts. It's crucial to understand that when enterprises deploy AI systems, they're deploying the entire system, not just some individual model checkpoint. Usable capability isn't just about the model weights; it’s a confluence of factors: the weights, how we route requests through the serving infrastructure, the precision, the output contract, and the evaluation harness. Yet, current benchmarking, even the comprehensive ones, merely score based on advertised model identifiers. We've identified this discrepancy as a significant source of measurement error, and we've developed a protocol to make this error observable and reportable.

This protocol, which we've termed IB2, is structured into three key parts. First, we have a "gold-blind capability-binding preflight" step, which, before any real task is introduced, makes sure the selected serving route can actually execute the evaluation contract – which in practice amounts to making sure the serving system is able to respond to queries that meet the evaluation's specific criteria. Second, a "reliability-inclusive first-pass scoring rule" keeps track of failures in the final score, but also excludes capabilities that the system isn't designed to support. And third, and critically, the "adjudication" stage of the evaluation is, in its structure, completely blind to the scores, to remove any bias in how we handle discrepancies.

We're open-sourcing the algorithms, classification tables, request contract, and manifest schemas associated with IB2. The actual reference instantiation, comprising 128 locked tasks and 987 assertions across diverse workflows, including document, spreadsheet, chart, tool, and database operations, remains sealed. The procedure is the artifact, not the test data itself.

In our experiments, across eleven systems, we observed four distinct results. We found that capability availability is measurable. Two complete runs with identical model weights, but different routes, later failed distinct predicates of the finalized binding gate (essentially the requirements we set for a model to be considered “valid”), while a third run passed that gate before the next test. The advertised identifier exposed neither of these limits.

Also, discrimination isn’t uniform across the evaluated systems. Four out of seven test suites saturate within a six-system bandwidth, and this variation is almost entirely driven by the intricacies of regulated database tasks and those multi-tab joins within spreadsheets. Consequently, we're reporting our results as interval-backed resolution groups rather than simple rankings, which would hide this uneven discrimination. Also, two of the nominal five-label output's four cut-points failed to meet multiple test adjustment criteria, raising questions about how the data is grouped.

The choice of service branch (serving-arm) had a concrete effect: it moved one declared revision and precision from 77.38 to 82.54, with a paired interval of [0.11, 10.60], even though the service branches differ in access mode, the generation of the harness (the evaluation framework), and the tool-call parser that’s used. Moreover, the generation of the harness is a property of our evaluator, not something determined by any endpoint.

Finally, excluding failed responses from the denominators changes how we order the data points – it impacts the point ordering. So, reliability inclusion – accounting for whether responses are successful or failed – modifies the conclusion itself, not just how we describe it.

企业部署的是系统，而非检查点。可用能力共同取决于权重、服务路由、精度、输出契约以及测试框架，然而受审计的所有 18 个基准测试评估的却只是对外公布的模型标识符。我们将此视为测量误差，并提出了一套使其具备可报告性的协议。该协议包含三个部分：真实答案盲测的能力绑定预检在任务送达前验证路由是否能执行评估契约；纳许可靠性的首轮评分规则在将失败计入评分的同时排除不受支持的能力；裁决流程在结构上对分数保持盲审。我们将该协议命名为 IB2，并开源其算法、分类表、请求契约和清单模式规范。其参考实现涵盖针对文档、电子表格、图表、工具和数据库操作的 128 个锁定任务和 987 项断言，且保持封存状态：评测流程本身即产物，而非语料库。

在 11 个系统上的评估得出四项结果。其一，能力可用性是可测量的：在相同权重下进行的两次完整单路由运行，随后均未能通过定型绑定门禁的不同判定条件，而第三次运行则在全新测试前通过了该门禁，对外公布的标识符并未体现出上述任何一种局限。其二，区分度并不均匀：七个套件中有四个在六系统区间内达到饱和，区分度几乎完全源自受监管的数据库操作和多工作表连接，因此我们报告的是基于置信区间的辨析度分组而非排名；名义上的五标签输出的四个切分点中，有两个未能通过多重性调整。其三，服务分支的选择将某个声明的修订版本和精度从 77.38 改变至 82.54（配对区间为 [0.11, 10.60]），尽管各分支在访问模式、测试框架生成和推理端工具调用解析器上存在差异，且测试框架生成属于我们评估器的属性，而非任何端点。其四，从分母中排除失败响应会改变点估计排序，因此将可靠性纳入评估改变的是结论本身，而不仅是其表述方式。