Tactile: Giving Computer-Using Agents Hands and Feet

Tactile is an open-source tool layer for computer-use agents that unifies accessibility semantics, OCR-grounded text, and visual fallback regions into action-grounded interface states. Each candidate includes provenance, role or text, state, geometry, executable affordances, and verification cues. Agents use an observe-ground-act-verify loop, preferring native semantic actions and falling back to OCR-based coordinates when necessary. On macOSWorld-style tasks, Tactile raises Codex Success@100 from 41.1% to 50.0% overall and from 45.2% to 55.3% on accessibility-adapted tasks. A 96-task cross-agent subset reportedly shows gains for Codex, Claude Code, OpenCode, and Goose.