JarvisGUI：面向多设备与动态任务组合的 GUI 智能体基准

My Thoughts on JarvisGUI

Okay, here's what I'm thinking about this work. The core issue is that real-world GUI interactions are inherently complex. They span multiple devices and platforms, necessitating the seamless transfer of intermediate results, shared state maintenance, and coordination across diverse environments. Yet, current GUI benchmarks are severely limited. They predominantly focus on single-device, statically defined tasks, leaving crucial cross-device capabilities largely unexamined. This leads to an overly optimistic assessment of agent readiness for real-world applications.

So, we need a better benchmark. We're introducing JarvisGUI, a dynamic benchmark specifically designed to evaluate GUI agents within cross-device workflows. These workflows require coordinated interaction across heterogeneous platforms, including Android, Windows, and Ubuntu. We've formalized GUI tasks as input-output transformations under a lightweight type system. This allows us to automatically compose multi-step, cross-device workflows and dynamically assess agent performance within a unified framework.

By evaluating these agents in virtual environments that encompass multiple operating systems, JarvisGUI uncovers a critical shortcoming. It reveals that even state-of-the-art open-source GUI agents struggle to effectively handle the state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management that are essential for real-world workflows. This is a crucial capability gap that existing benchmarks have failed to expose.

现实场景中的 GUI 使用往往涉及跨越多种设备与平台的工作流，需要传输中间结果、维护共享状态并在异构环境间进行协同。然而，现有的 GUI 基准测试绝大多数仅在单设备、静态定义的任务上评估智能体，导致此类跨设备能力在很大程度上未得到检验，从而对智能体在真实场景中的应用就绪度给出了过于乐观的评估。我们推出了 JarvisGUI，这是一个动态基准测试，用于评估 GUI 智能体在需要跨异构平台（包括 Android、Windows 和 Ubuntu）协同交互的跨设备工作流中的表现。具体而言，JarvisGUI 在轻量级类型系统下将 GUI 任务形式化为输入-输出转换，使我们能够自动构建多步骤的跨设备工作流，并在统一框架内动态评估智能体的性能。通过在跨越多个操作系统的虚拟环境中评估智能体，JarvisGUI 揭示出当前最先进的开源 GUI 智能体难以应对真实工作流所需的状态传递感知、跨平台上下文推理以及长程依赖管理，暴露出在现有基准测试中未能显现的关键能力差距。