DreamTraj: Generating 6-DoF Object Trajectories from Unrendered Video Diffusion Latents
Original title:DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents
AI Summary
DreamTraj predicts relative 6-DoF object trajectories from a single RGB image and a task instruction, without requiring video, depth, or CAD models at inference. Instead of generating a video and recovering motion afterward, it reads motion-related information from intermediate representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states. The paper also introduces MOVE, a dataset of 5,038 object-centric egocentric trajectories paired with fine-grained natural-language instructions rather than coarse verb-noun labels.
Why it's worth reading
It is timely because it reframes video diffusion as a motion representation source for robotics, while MOVE addresses the shortage of fine-grained language-to-motion supervision in manipulation datasets.
Deep Read
1. What happened
Original facts: The paper introduces DreamTraj and the MOVE dataset. DreamTraj predicts relative 6-DoF object trajectories from one RGB image and a task instruction. At inference, it does not require video, depth, or CAD models. MOVE contains 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction.
2. Core technology
Original facts: DreamTraj uses a frozen image-to-video diffusion model and reads its intermediate representations at an early denoising step instead of rendering a complete video. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. Analysis: The central hypothesis is that these intermediate representations encode task-relevant motion structure before full video generation.
3. Key evidence and numbers
Original facts: MOVE has 5,038 trajectories. The stated input is a single RGB image plus a natural-language instruction, while video, depth, and CAD models are excluded at inference. Unverified inference: The supplied abstract does not report trajectory errors, success rates, latency, or ablation results, so the magnitude of any advantage over prior methods cannot be established here.
4. Why it matters
Analysis: Existing approaches may use privileged inputs or generate full videos before recovering motion through an additional perception pipeline. DreamTraj attempts to use the generative model's internal state directly, potentially reducing generation and post-processing cost. MOVE also targets finer language-to-motion supervision than coarse verb-noun labels.
5. Practical impact
Analysis: If the reported results remain stable in physical manipulation settings, the approach could support monocular, language-conditioned grasping and manipulation while reducing dependence on depth sensors and CAD assets. Deployment still requires coordinate calibration, temporal execution, closed-loop correction, and safety constraints; the abstract does not establish that these issues are solved.
6. Limitations and uncertainty
Original facts: The abstract identifies limited fine-grained language-to-motion annotations and the privileged inputs used by some existing predictors as major challenges. Uncertainty: The supplied material does not specify MOVE's scene diversity, object categories, trajectory lengths, train-test splits, diffusion backbone, Reader size, or cross-domain generalization. Whether early denoising representations consistently preserve decodable motion information requires ablations and independent reproduction.
7. Original sources
- arXiv:2608.00486
- Source: Hugging Face Papers (hf-papers)
- Supplied publication timestamp: 2026-07-31T20:00:00.000Z