What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Visuomotor imitation policies such as ACT frequently stumble when visually similar distractor objects or receptacles appear, despite executing mechanically sound trajectories. Treating this as a breakdown in conditional visual grounding across manipulation phases, the authors localize failure points to target selection during picking and placing. By pairing phase-dependent attention regularization with distractor augmentation and appearance prompting, the proposed interventions restore robust target selection on physical UR3e arms and state-conditioned vision-language-action policies.