2D3D-DiffMatch: Detector-Free Image and Point Cloud Matching via Diffusion-Guided Cross-Modal Consistency

Abstract

In the realms of computer vision and remote sensing, the matching of images and point clouds poses significant challenges due to modality discrepancies. This study introduces a cross-modal consistency network, detector-free image and point cloud matching via diffusion-guided cross-modal consistency, 2D3D-DiffMatch, leveraging diffusion prior information to enhance feature extraction consistency and alignment across modalities. To strengthen cross-modal consistency in complex scenes, diffusion priors generated by a pre-trained diffusion model are used to guide the feature extraction process toward semantically and geometrically consistent representations. These representations are further refined through a hierarchical fusion process, in which the most consistent diffusion features are adaptively selected using centered kernel alignment(CKA) and integrated with multi-scale backbone features, thereby mitigating the impact of modality gaps. Furthermore, to address feature-space misalignment between images and point clouds, we propose a cross-modal feature consistency loss that adaptively constrains correspondences, separates positive and negative pairs, and optimizes the agreement of positive pairs, enabling high-quality, detector-free matching. Experimental results on the 7Scenes and RGB-D Scenes V2 Datasets demonstrate superior registration recall rates of 81.2% and 61.0%, respectively, outperforming state-of-the-art methods and exhibiting robustness in challenging scenarios.This research advances the collaborative processing of multi-modal data, offering a robust solution for image–point cloud matching in challenging scenarios.

Keywords

image and point cloud matching / cross-modal consistent features / diffusion model / diffusion prior information