DUET-DINO：基于跨视角世界模型的机械臂全自由度潜在规划

DUET-DINO: A Novel Approach to Action-Conditioned World Modeling

Okay, so here's the deal: Action-conditioned latent world models are great; they predict future visual representations, which is key for zero-shot goal-conditioned robot planning and control. But, and this is a big but, their predictions fall apart when you get down to the nitty-gritty of fine-grained spatial and rotational actions. They just aren't reliable enough for full 7-DoF end-effector control, and that's a problem I need to address. To close this gap, I’m introducing DUET-DINO. It's a simultaneous cross-view latent world model. It jointly learns action-conditioned predictions from static side- and wrist-camera observations using cross-view conditioning. I can exploit the complementary information from the global scene and the gripper’s perspective. This allows for latent planning over the complete 7-DoF action space. I've tested it, and it outperforms single-view and independent dual-view baselines across a range of tasks: spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks. Specifically, it achieves 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. I've trained DUET-DINO from scratch on the DROID and RoboArena datasets, and I can say that it generalizes robustly, even when the visual distribution shifts. Furthermore, I’ve found that while wrist-view predictions from V-JEPA 2 tend to underestimate the visual dynamics caused by fine-grained actions, DINOv3's predictions are much better at capturing these action-conditioned scene changes. This leads to much stronger downstream planning capabilities. Finally, the code and model checkpoints are going to be open-sourced. You can find more information on the project page: https://utn-air.github.io/DUET-DINO

动作条件潜空间世界模型能够预测未来的视觉表征，从而实现零样本目标条件机器人规划与控制。然而，其对细粒度空间和旋转动作的预测不够可靠，难以满足完整的 7-DoF 末端执行器控制需求。针对这一不足，我们提出了 DUET-DINO，这是一种同步跨视角潜空间世界模型，通过跨视角条件机制联合学习来自静态侧视相机和腕部相机观测的动作条件预测。通过充分利用互补的全局场景与以夹爪为中心的信息，DUET-DINO 实现了在完整 7-DoF 动作空间上的潜空间规划。在空间多样化的触达（reach）、强朝向依赖的角度触达（angled-reach）以及多目标抓取与提升（grasp-and-lift）任务中，DUET-DINO 的表现始终优于单视角和独立双视角基准模型，在触达、角度触达和提升任务中分别取得了 92%、72.5% 和 60.0% 的成功率。DUET-DINO 在 DROID 和 RoboArena 数据集上从零开始训练，并在视觉分布偏移下展现出强大的泛化能力。我们还进一步表明，虽然 V-JEPA 2 在腕部视角的预测中低估了细粒度动作所引发的视觉动态，但 DINOv3 的预测能够更好地捕捉动作条件下的场景变化，从而带来更强大的下游规划性能。代码和模型权重将会开源。项目主页：https://utn-air.github.io/DUET-DINO