Multimodal medical image fusion based on mask optimization and parallel attention mechanism

Abstract

Medical image fusion technology is crucial for improving the detection accuracy and treatment efficiency of diseases, but existing fusion methods have problems such as blurred texture details, low contrast, and inability to fully extract fused image information. Therefore, a multimodal medical image fusion method based on mask optimization and parallel attention mechanism was proposed to address the aforementioned issues. Firstly, it converted the entire image into a binary mask, and constructed a contour feature map to maximize the contour feature information of the image and a triple path network for image texture detail feature extraction and optimization. Secondly, a contrast enhancement module and a detail preservation module were proposed to enhance the overall brightness and texture details of the image. Afterwards, a parallel attention mechanism was constructed using channel features and spatial feature changes to fuse images and enhance the salient information of the fused images. Finally, a decoupling network composed of residual networks was set up to optimize the information between the fused image and the source image so as to reduce information loss in the fused image. Compared with nine high-level methods proposed in recent years, the seven objective evaluation indicators of our method have improved by 6%-31%, indicating that this method can obtain fusion results with clearer texture details, higher contrast, and smaller pixel differences between the fused image and the source image. It is superior to other comparison algorithms in both subjective and objective indicators.

Keywords

multimodal medical image fusion / binary mask / contrast enhancement module / parallel attention mechanism / decoupling network

References

[1] CHAO Z, DUAN X G, JIA S F, et al. Medical image fusion via discrete stationary wavelet transform and an enhanced radial basis function neural network. Applied Soft Computing, 2022, 118: 108542.

[2] YU N N, LI J J, HUA Z. Decolorization algorithm based on contrast pyramid transform fusion. Multimedia Tools and Applications, 2022, 81(11): 15017-15039.

[3] ZHAO Z X, XU S, ZHANG C X, et al. Bayesian fusion for infrared and visible images. Signal Processing, 2020, 177: 107734.

[4] SATHISH KUMAR G A E, DEVANNA H. Computational effective multimodal medical image fusion in NSCT domain. IOP Conference Series: Materials Science and Engineering, 2021, 1042(1): 012003.

[5] ZHANG H, XU H, TIAN X, et al. Image fusion meets deep learning: A survey and perspective. Information Fusion, 2021, 76: 323-336.

[6] ZHOU Q, YE S Z, WEN M W, et al. Multi-modal medical image fusion based on densely-connected high-resolution CNN and hybrid transformer. Neural Computing and Applications, 2022, 34(24): 21741-21761.

[7] GUO K, LI X F, HU X H, et al. Hahn-PCNN-CNN: an end-to-end multi-modal brain medical image fusion framework useful for clinical diagnosis. BMC Medical Imaging, 2021, 21(1): 111.

[8] SHI B B, CHEN Y N, ZHANG P, et al. Nonlinear feature transformation and deep fusion for Alzheimer’s Disease staging analysis. Pattern Recognition, 2017, 63: 487-498.

[9] LIU S W, YANG L H. BPDGAN: a GAN-based unsupervised back project dense network for multi-modal medical image fusion. Entropy, 2022, 24(12): 1823.

[10] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.

[11] ZHANG H, MA J Y. SDNet: a versatile squeeze-and-decomposition network for real-time image fusion. International Journal of Computer Vision, 2021, 129(10): 2761-2785.

[12] LIU J Y, LIN R J, WU G Y, et al. CoCoNet: coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusion. International Journal of Computer Vision, 2023, 132(5): 1748-1775.

[13] LIU X W, WANG R H, HUO H T, et al. An attention-guided and wavelet-constrained generative adversarial network for infrared and visible image fusion. Infrared Physics & Technology, 2023, 129: 104570.

[14] TANG L F, XIANG X Y, ZHANG H, et al. DIVFusion: Darkness-free infrared and visible image fusion. Information Fusion, 2023, 91: 477-493.

[15] VAN AARDT J. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2008, 2(1): 1-28.

[16] LI S T, YANG B. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing, 2008, 26(7): 971-979.

[17] MANJUSHA D, BHOSALE U. Image fusion and image quality assessment of fused images. International Journal of Image Processing, 2010, 4(5): 484-508.

[18] QU G H, ZHANG D L, YAN P F. Information measure for performance of image fusion. Electronics Letters, 2002, 38(7): 313.

[19] ASLANTAS V, BENDES E. A new image quality metric for image fusion: The sum of the correlations of differences. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890-1896.

[20] ZHANG Y, XIANG W H, ZHANG S L, et al. Local extreme map guided multi-modal brain image fusion. Frontiers in Neuroscience, 2022, 16: 1055451.

[21] LI B, PENG H, LUO X H, et al. Medical image fusion method based on coupled neural P systems in nonsubsampled shearlet transform domain. International Journal of Neural Systems, 2021, 31(1): 2050050.

[22] LI B, PENG H, WANG J. A novel fusion method based on dynamic threshold neural P systems and nonsubsampled contourlet transform for multi-modality medical images. Signal Processing, 2021, 178: 107793.

[23] TANG W, HE F Z, LIU Y, et al. MATR: Multimodal medical image fusion via multiscale adaptive transformer. IEEE Transactions on Image Processing, 2022, 31: 5134-5149.

[24] SONG X, WU X J, LI H. A Medical Image fusion method based on MDLatLRRv2. arXiv preprint arXiv:2206.15179, 2022.

[25] LIU Y, CHEN X, CHENG J, et al. A medical image fusion method based on convolutional neural networks//2017 20th International Conference on Information Fusion, July 10-13, 2017, Xi’an, China. New York: IEEE, 2017: 1-7.

[26] TAN W, THITØN W, XIANG P, et al. Multi-modal brain image fusion based on multi-level edge-preserving filtering. Biomedical Signal Processing and Control, 2021, 64: 102280.

[27] MA J Y, TANG L F, FAN F, et al. SwinFusion: cross-domain long-range learning for general image fusion via swin transformer. IEEE/CAA Journal of Automatica Sinica, 2022, 9(7): 1200-1217.