IBIB: A Protocol for Measuring Enterprise AI Systems by Serving Route, Not Model Identifier
Enterprises deploy operational systems rather than raw checkpoints, yet prevailing benchmark suites almost exclusively evaluate advertised model identifiers. The authors introduce IB2, a measurement protocol emphasizing serving routes, precision, and tool-call parsers alongside weights. In experiments spanning eleven enterprise systems, differing serving arms shifted scores for the identical model revision from 77.38 to 82.54. By enforcing capability-binding preflight checks and retaining execution failures in scoring denominators, the protocol exposes real-world reliability gaps obscured by checkpoint-centric leaderboards.
Why it's worth reading
It exposes a critical blind spot in enterprise evaluation, demonstrating that inference routes and execution harnesses sway benchmark scores as much as underlying model weights.