Improving Misaligned Multi-Modality Image Fusion With One-Stage Progressive Dense Registration

Abstract

Misalignments between multi-modality images pose challenges in image fusion, manifesting as structural distortions and edge ghosts. Existing efforts commonly resort to registering first and fusing later, typically employing two separate stages for registration, i.e., coarse registration and fine registration. Both stages directly estimate the respective target deformation fields. This paper contends that the separate two-stage registration lacks compactness, and the direct estimation of their target deformation fields falls short in accuracy. To tackle these challenges, we introduce IMF, a framework for improving misaligned multi-modality image fusion. Central to IMF is a One-stage Progressive Dense Registration (OPDR) scheme, which accomplishes the coarse-to-fine registration through only a one-stage optimization. Specifically, two pivotal components are involved in OPDR, a dense Deformation Field Fusion (DFF) module and a Progressive Feature Fine (PFF) module. The DFF aggregates the predicted multi-scale deformation sub-fields at the current scale, while the PFF progressively refines the remaining misaligned features. Together, they effectively and accurately estimate the final deformation fields. In addition, we develop a Transformer-Conv-based Fusion (TCF) subnetwork that considers local and long-range feature dependencies, allowing us to capture more informative features from the registered infrared and visible images for the generation of high-quality fused images. Extensive experimental analysis demonstrates the superiority of the proposed method in the fusion of misaligned cross-modality images. The code will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wdhudiekou/IMF</uri>.

Ähnliche Arbeiten