利用测试期部分观测引导单图到3D生成

My Thoughts on Enhancing Image-to-3D Model Fidelity

Okay, so here's what I'm thinking: Image-to-3D models are great at churning out visually compelling 3D assets from a single RGB image. But, as we know, their geometry tends to be loosely constrained by the available data. This limits their practical application, especially when we need that crucial geometric fidelity. In many real-world scenarios, however, we actually do have access to partial geometric observations of the object, particularly during the testing phase.

The challenge is how to best leverage this additional information. My idea is a training-free framework, so we can incorporate this extra information into pre-trained image-to-3D models without the hassle of retraining or even fine-tuning them.

The core of this approach lies in using a ray-consistent observation likelihood, which is defined over the model's occupancy representation. By cleverly combining surface occupancy and free-space evidence, we can guide the generation process.

We tested this approach on SAM 3D and its multi-view extension. The results are quite promising – significant improvements in geometric fidelity across varying levels of observability, along with better visual quality.

Ultimately, this demonstrates that pre-trained image-to-3D models can effectively integrate partial geometric observations using explicit test-time guidance. This allows us to refine their learned generative priors without needing to modify the underlying model itself.

图像到3D模型可以从单张RGB图像生成视觉效果出色的3D资产，但其几何结构往往仅受到现有观测的弱约束，从而限制了其在需要几何保真度的实际应用中的使用。然而，在许多现实场景中，测试时通常可以获取物体的部分几何观测。我们提出了一种免训练框架，无需重新训练或微调即可将此类证据融入预训练的图像到3D生成模型中。为此，我们在模型的占用表示上定义了光线一致的观测似然，结合表面占用与自由空间证据来引导生成。应用于 SAM 3D 及其多视角扩展后，我们的方法在不同的可观测性水平下均显著提升了几何保真度以及视觉质量。研究结果表明，预训练的图像到3D模型能够通过显式的测试时引导有效整合部分几何观测，在不修改底层模型的前提下对其学到的生成先验形成有力补充。