image fusion
199677411
IFCNN: A general image fusion framework based on convolutional neural network
Abstract In this paper, we propose a general image fusion framework based on the convolutional neural network, named as IFCNN. Inspired by the transform domain image fusion algorithms, we firstly utilize two convolutional layers to extract the salient image features from multiple input images. Afterwards, the convolutional features of multiple input images are fused by an appropriate fusion rule (elementwise max, elementwise min or elementwise mean) which is selected according to the type of input images. Finally, the fused features are reconstructed by two convolutional layers to produce the informative fusion image. The proposed model is fully convolutional, so it could be trained in the end to end manner without any post processing procedures. In order to fully train the model, we have generated a large scale multi focus image dataset based on the large scale RGB D dataset (i.e. NYU D2) which owns ground truth fusion images and contains more diverse and larger images than the existing datasets for image fusion. Without finetuning on other types of image datasets, the experimental results show that the proposed model demonstrates better generalization ability than the existing image fusion models for fusing various types of images, such as multi focus, infrared visual, multi modal medical and multi exposure images. Moreover, the results also verify that our model has achieved comparable or even better results compared to the state of the art image fusion algorithms on four types of image datasets.
Inf. Fusion
Yanshun Zhang Ying Liu Peng Sun Han Yan Xiaolin Zhao Li Zhang
199688641
Infrared and visible image fusion via detail preserving adversarial learning
Abstract TargefTablets can be detected easily from the background of infrared images due to their significantly discriminative thermal radiations, while visible images contain textural details with high spatial resolution which are beneficial to the enhancement of target recognition. Therefore, fused images with abundant detail information and effective target areas are desirable. In this paper, we propose an end to end model for infrared and visible image fusion based on detail preserving adversarial learning. It is able to overcome the limitations of the manual and complicated design of activity level measurement and fusion rules in traditional fusion methods. Considering the specific information of infrared and visible images, we design two loss functions including the detail loss and target edge enhancement loss to improve the quality of detail information and sharpen the edge of infrared targets under the framework of generative adversarial network. Our approach enables the fused image to simultaneously retain the thermal radiation with sharpening infrared target boundaries in the infrared image and the abundant textural details in the visible image. Experiments conducted on publicly available datasets demonstrate the superiority of our strategy over the state of the art methods in both objective metrics and visual impressions. In particular, our results look like enhanced infrared images with clearly highlighted and edge sharpened targets as well as abundant detail information.
Inf. Fusion
Jiayi Ma Pengwei Liang Wei Yu Chen Chen Xiaojie Guo Jia Wu Junjun Jiang
211677495
MDLatLRR: A Novel Decomposition Method for Infrared and Visible Image Fusion
Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi level image decomposition method based on latent low rank representation(LatLRR) which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We build a novel image fusion framework based on MDLatLRR which is used to decompose source images into detail parts(salient features) and base parts. A nuclear norm based fusion strategy is used to fuse the detail parts and the base parts are fused by an averaging strategy. Compared with other state of the art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation.
IEEE Transactions on Image Processing
Hui Li Xiaojun Wu Josef Kittler
71142966
FusionGAN: A generative adversarial network for infrared and visible image fusion
Abstract Infrared images can distinguish targets from their backgrounds on the basis of difference in thermal radiation, which works well at all day/night time and under all weather conditions. By contrast, visible images can provide texture details with high spatial resolution and definition in a manner consistent with the human visual system. This paper proposes a novel method to fuse these two types of information using a generative adversarial network, termed as FusionGAN. Our method establishes an adversarial game between a generator and a discriminator, where the generator aims to generate a fused image with major infrared intensities together with additional visible gradients, and the discriminator aims to force the fused image to have more details existing in visible images. This enables that the final fused image simultaneously keeps the thermal radiation in an infrared image and the textures in a visible image. In addition, our FusionGAN is an end to end model, avoiding manually designing complicated activity level measurements and fusion rules as in traditional methods. Experiments on public datasets demonstrate the superiority of our strategy over state of the arts, where our results look like sharpened infrared images with clear highlighted targets and abundant details. Moreover, we also generalize our FusionGAN to fuse images with different resolutions, say a low resolution infrared image and a high resolution visible image. Extensive results demonstrate that our strategy can generate clear and clean fused images which do not suffer from noise caused by upsampling of infrared information.
Inf. Fusion
Jiayi Ma Wei Yu Pengwei Liang Chang Li Junjun Jiang
52131893
Super Resolution for Hyperspectral and Multispectral Image Fusion Accounting for Seasonal Spectral Variability
Image fusion combines data from different heterogeneous sources to obtain more precise information about an underlying scene. Hyperspectral multispectral (HS MS) image fusion is currently attracting great interest in remote sensing since it allows the generation of high spatial resolution HS images and circumventing the main limitation of this imaging modality. Existing HS MS fusion algorithms, however, neglect the spectral variability often existing between images acquired at different time instants. This time difference causes variations in spectral signatures of the underlying constituent materials due to the different acquisition and seasonal conditions. This paper introduces a novel HS MS image fusion strategy that combines an unmixing based formulation with an explicit parametric model for typical spectral variability between the two images. Simulations with synthetic and real data show that the proposed strategy leads to a significant performance improvement under spectral variability and state of the art performance otherwise.
IEEE Transactions on Image Processing
Ricardo Augusto Borsoi Tales Imbiriba Jose Carlos Moreira Bermudez
54462396
Medical Image Fusion With Parameter Adaptive Pulse Coupled Neural Network in Nonsubsampled Shearlet Transform Domain
As an effective way to integrate the information contained in multiple medical images with different modalities, medical image fusion has emerged as a powerful technique in various clinical applications such as disease diagnosis and treatment planning. In this paper, a new multimodal medical image fusion method in nonsubsampled shearlet transform (NSST) domain is proposed. In the proposed method, the NSST decomposition is first performed on the source images to obtain their multiscale and multidirection representations. The high frequency bands are fused by a parameter adaptive pulse coupled neural network (PA PCNN) model, in which all the PCNN parameters can be adaptively estimated by the input band. The low frequency bands are merged by a novel strategy that simultaneously addresses two crucial issues in medical image fusion, namely, energy preservation and detail extraction. Finally, the fused image is reconstructed by performing inverse NSST on the fused high frequency and low frequency bands. The effectiveness of the proposed method is verified by four different categories of medical image fusion problems [computed tomography (CT) and magnetic resonance (MR) MR T1 and MR T2, MR and positron emission tomography, and MR and single photon emission CT] with more than 80 pairs of source images in total. Experimental results demonstrate that the proposed method can obtain more competitive performance in comparison to nine representative medical image fusion methods, leading to state of the art results on both visual quality and objective assessment.
IEEE Transactions on Instrumentation and Measurement
Ming Yin Xiaoning Liu Yu Liu Xun Chen
52073194
Infrared and visible image fusion methods and applications: A survey
Abstract Infrared images can distinguish targets from their backgrounds based on the radiation difference, which works well in all weather and all day/night conditions. By contrast, visible images can provide texture details with high spatial resolution and definition in a manner consistent with the human visual system. Therefore, it is desirable to fuse these two types of images, which can combine the advantages of thermal radiation information in infrared images and detailed texture information in visible images. In this work, we comprehensively survey the existing methods and applications for the fusion of infrared and visible images. First, infrared and visible image fusion methods are reviewed in detail. Meanwhile, image registration, as a prerequisite of image fusion, is briefly introduced. Second, we provide an overview of the main applications of infrared and visible image fusion. Third, the evaluation metrics of fusion performance are discussed and summarized. Fourth, we select eighteen representative methods and nine assessment metrics to conduct qualitative and quantitative experiments, which can provide an objective performance reference for different fusion methods and thus support relative engineering with credible and solid evidence. Finally, we conclude with the current status of infrared and visible image fusion and deliver insightful discussions and prospects for future work. This survey can serve as a reference for researchers in infrared and visible image fusion and related fields.
Inf. Fusion
Jiayi Ma Yong Ma Chang Li
11925688
Multi focus image fusion with a deep convolutional neural network
Introduces Convolutional neural networks (CNNs) into the field of image fusion.Discusses the feasibility and superiority of CNNs used for image fusion.Proposes a state of the art CNN based multi focus image fusion method.Exhibits the potential of CNNs for other type image fusion issues.Puts forward some suggestions on the future study of CNN based image fusion. As is well known, activity level measurement and fusion rule are two crucial factors in image fusion. For most existing fusion methods, either in spatial domain or in a transform domain like wavelet, the activity level measurement is essentially implemented by designing local filters to extract high frequency details, and the calculated clarity information of different source images are then compared using some elaborately designed rules to obtain a clarity/focus map. Consequently, the focus map contains the integrated clarity information, which is of great significance to various image fusion issues, such as multi focus image fusion, multi modal image fusion, etc. However, in order to achieve a satisfactory fusion performance, these two tasks are usually difficult to finish. In this study, we address this problem with a deep learning approach, aiming to learn a direct mapping between source images and focus map. To this end, a deep convolutional neural network (CNN) trained by high quality image patches and their blurred versions is adopted to encode the mapping. The main novelty of this idea is that the activity level measurement and fusion rule can be jointly generated through learning a CNN model, which overcomes the difficulty faced by the existing fusion methods. Based on the above idea, a new multi focus image fusion method is primarily proposed in this paper. Experimental results demonstrate that the proposed method can obtain state of the art fusion performance in terms of both visual quality and objective assessment. The computational speed of the proposed method using parallel computing is fast enough for practical usage. The potential of the learned CNN model for some other type image fusion issues is also briefly exhibited in the experiments.
Inf. Fusion
Yu Liu Xun Chen Hu Peng Zengfu Wang
67872646
A Phase Congruency and Local Laplacian Energy Based Multi Modality Medical Image Fusion Method in NSCT Domain
Multi modality image fusion provides more comprehensive and sophisticated information in modern medical diagnosis, remote sensing, video surveillance, and so on. This paper presents a novel multi modality medical image fusion method based on phase congruency and local Laplacian energy. In the proposed method, the non subsampled contourlet transform is performed on medical image pairs to decompose the source images into high pass and low pass subbands. The high pass subbands are integrated by a phase congruency based fusion rule that can enhance the detailed features of the fused image for medical diagnosis. A local Laplacian energy based fusion rule is proposed for low pass subbands. The local Laplacian energy consists of weighted local energy and the weighted sum of Laplacian coefficients that describe the structured information and the detailed features of source image pairs, respectively. Thus, the proposed fusion rule can simultaneously integrate two key components for the fusion of low pass subbands. The fused high pass and low pass subbands are inversely transformed to obtain the fused image. In the comparative experiments, three categories of multi modality medical image pairs are used to verify the effectiveness of the proposed method. The experiment results show that the proposed method achieves competitive performance in both the image quantity and computational costs.
IEEE Access
Zhiqin Zhu Mingyao Zheng Guanqiu Qi Di Wang Yan Xiang
126786100
A survey on region based image fusion methods
Abstract Image fusion has been emerging as an important area of research. It has attracted many applications such as surveillance, photography, medical diagnosis, etc. Image fusion techniques are developed at three levels: pixel, feature and decision. Region based image fusion is one of the methods of feature level. It possesses certain advantages less sensitive to noise, more robust and avoids misregistration. This paper presents a review of region based fusion approaches. A first hand classification of region based fusion methods is carried out. A comprehensive list of objective fusion evaluation metrics is highlighted to compare the existing methods. A detailed analysis is carried out and results are presented in tabular form. This may attract researchers to further explore the research in this direction.
Inf. Fusion
Bikash Meher Sanjay Agrawal Rutuparna Panda Ajith Abraham