Image Denoising and Segmentation Using Piecewise Smooth Relaxed Total Generalized Variation

Abstract

In the paper, we introduce a novel image denoising and segmentation technique using the piecewise smooth relaxed total generalized variation. The relaxed total generalized variational model (RTGV) is a combination of gradient operator and weighted divergence operator, which can improve the computational performance. For the denoising method, the RTGV regularization term and a smooth term are introduced to further improve the restored results. For the image segmentation method, we introduce the RTGV regularized piecewise smooth Mumford–Shah segmentation model instead of total variation regularized, which can deal with image with noise and inhomogeneous intensity effectively. In addition, the augmented Lagrangian algorithm is used to iteratively solve the proposed RTGV method, which produced several closed form solutions. Our algorithm is discussed from several aspects, including theoretical analysis, influence of parameters, and comparisons with several state-of-the-art methods. Numerical experimental results show that our algorithms can yield competitive results both visually and numerical errors with reasonable computational cost. Our image denoising algorithm has the advantage in preserving image features and can effectively prevent the staircase artifacts when compared to several other existing methods. Meanwhile, our segmentation algorithm can obtain results with boundaries more consistent with the human visual system and is shown to be able to achieve multi-region segmentation.

Keywords

Relaxed total generalized variation / Mumford–Shah model / Image denoising / Image segmentation / 94A08 / 68U10

References

[1] Blomgren, P., Chan, T.F., Mulet, P., Wong, C.-K.: Total variation image restoration: numerical methods and extensions. In: International Conference on Image Processing, vol. 3, pp. 384–387 (1997)

[2] Boykov Y, Funka-Lea G. Graph cuts and efficient nd image segmentation. Int. J. Comput. Vision, 2006, 70(2): 109-131

[3] Bredies, K., Valkonen, T.: Inverse problems with second-order total generalized variation constraints. arXiv preprint arXiv:2005.09725 (2020)

[4] Bredies K, Sun HP. Preconditioned Douglas–Rachford algorithms for TV-and TGV-regularized variational imaging problems. J. Math. Imaging and Vis., 2015, 52(3): 317-344

[5] Bredies K, Kunisch K, Pock T. Total generalized variation. SIAM J. Imag. Sci., 2010, 3(3): 492-526

[6] Brox T, Cremers D. On local region models and a statistical interpretation of the piecewise smooth Mumford–Shah functional. Int. J. Comput. Vision, 2009, 84(2): 184-193

[7] Buades, A., Coll, B., Morel, J.-M.: A non-local algorithm for image denoising. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 60–65 (2005)

[8] Buades A, Coll B, Morel J-M. A review of image denoising algorithms, with a new one. Multiscale Model. Simul., 2005, 4(2): 490-530

[9] Buades A, Coll B, Morel J-M. Image denoising methods: a new nonlocal principle. SIAM Rev., 2010, 52(1): 113-147

[10] Caselles V, Kimmel R, Sapiro G. Geodesic active contours. Int. J. Comput. Vision, 1997, 22(1): 61-79

[11] Chan, T., Vese, L.: An active contour model without edges. In: International Conference on Scale-space Theories in Computer Vision, pp. 141–151 (1999)

[12] Chan T, Marquina A, Mulet P. High-order total variation-based image restoration. SIAM J. Sci. Comput., 2000, 22(2): 503-516

[13] Chan TF, Esedoglu S, Nikolova M. Algorithms for finding global minimizers of image segmentation and denoising models. SIAM J. Appl. Math., 2006, 66(5): 1632-1648

[14] Chen T, Ma K-K, Chen L-H. Tri-state median filter for image denoising. IEEE Trans. Image Process., 1999, 8(12): 1834-1838

[15] Deng L-J, Glowinski R, Tai X-C. A new operator splitting method for the Euler–Elastica model for image smoothing. SIAM J. Imag. Sci., 2019, 12(2): 1190-1230

[16] Ekeland I, Temam R. Convex Analysis and Variational Problems, 1999, Philadelphia, SIAM

[17] Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process., 2006, 15(12): 3736-3745

[18] Fan L, Zhang F, Fan H, Zhang C. Brief review of image denoising techniques. Visual Comput. Ind. Biomed. Art, 2019, 2(1): 1-12

[19] Ghosh S, Das N, Das I, Maulik U. Understanding deep learning techniques for image segmentation. ACM Comput. Surv. (CSUR), 2019, 52(4): 1-35

[20] Glowinski, R., Le Tallec, P.: Augmented Lagrangian and operator-splitting methods in nonlinear mechanics 9 (1989)

[21] Grady L. Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 2006, 28(11): 1768-1783

[22] Grady L, Alvino CV. The piecewise smooth Mumford–Shah functional on an arbitrary graph. IEEE Trans. Image Process., 2009, 18(11): 2547-2561

[23] Gu Y, Xiong W, Wang L-L, Cheng J. Generalizing Mumford–Shah model for multiphase piecewise smooth image segmentation. IEEE Trans. Image Process., 2016, 26(2): 942-952

[24] Guo X, Xue Y, Wu C. Effective two-stage image segmentation: a new non-Lipschitz decomposition approach with convergent algorithm. J. Math. Imaging Vision, 2021, 63(3): 356-379

[25] He, X., Zhu, W., Tai, X.-C.: Segmentation by Elastica energy with l1 and l2 curvatures: a performance comparison. Numer. Math.: Theory Methods Appl. 12(06) (2019)

[26] Hesamian MH, Jia W, He X, Kennedy P. Deep learning techniques for medical image segmentation: achievements and challenges. J. Digit. Imaging, 2019, 32(4): 582-596

[27] Huang C, Ng MK, Wu T, Zeng T. Quaternion-based dictionary learning and saturation-value total variation regularization for color image restoration. IEEE Trans. Multimedia, 2021, 24: 3769-3781

[28] Jia Z, Ng MK, Wang W. Color image restoration by saturation-value total variation. SIAM J. Imag. Sci., 2019, 12(2): 972-1000

[29] Jung M. Piecewise-smooth image segmentation models with l1\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$l_1$$\end{document}data-fidelity terms. J. Sci. Comput., 2017, 70(3): 1229-1261

[30] Kang M, Kang M, Jung M. Total generalized variation based denoising models for ultrasound images. J. Sci. Comput., 2017, 72(1): 172-197

[31] Kass M, Witkin A, Terzopoulos D. Snakes: active contour models. Int. J. Comput. Vision, 1988, 1(4): 321-331

[32] Kim B, Ye JC. Mumford–Shah loss functional for image segmentation with deep learning. IEEE Trans. Image Process., 2019, 29: 1856-1866

[33] Kimmel R, Elad M, Shaked D, Keshet R, Sobel I. A variational framework for retinex. Int. J. Comput. Vision, 2003, 52: 7-23

[34] Knoll F, Bredies K, Pock T, Stollberger R. Second order total generalized variation (TGV) for MRI. Magn. Reson. Med., 2011, 65(2): 480-491

[35] Kumar BKS. Image denoising based on gaussian/bilateral filter and its method noise thresholding. Signal Image Video Process., 2013, 7(6): 1159-1172

[36] Laine, S., Karras, T., Lehtinen, J., Aila, T.: High-quality self-supervised deep image denoising, vol. 32 (2019)

[37] Le, T., Vese, L.: Additive and multiplicative piecewise-smooth segmentation models in a variational level set approach. Contemp. Math. 445 (2007)

[38] Lebrun M, Buades A, Morel J-M. A nonlocal Bayesian image denoising algorithm. SIAM J. Imag. Sci., 2013, 6(3): 1665-1688

[39] Lefkimmiatis, S.: Non-local color image denoising with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3587–3596 (2017)

[40] Lellmann, J., Kappes, J., Yuan, J., Becker, F., Schnörr, C.: Convex multi-class image labeling by simplex-constrained total variation. In: International Conference on Scale Space and Variational Methods in Computer Vision, pp. 150–162 (2009)

[41] Lellmann J, Schnörr C. Continuous multiclass labeling approaches and algorithms. SIAM J. Imag. Sci., 2011, 4(4): 1049-1096

[42] Li F, Ng MK, Li C. Variational fuzzy Mumford–Shah model for image segmentation. SIAM J. Appl. Math., 2010, 70(7): 2750-2770

[43] Li C, Huang R, Ding Z, Gatenby JC, Metaxas DN, Gore JC. A level set method for image segmentation in the presence of intensity inhomogeneities with application to MRI. IEEE Trans. Image Process., 2011, 20(7): 2007-2016

[44] Li Y, Wu C, Duan Y. The TVp regularized Mumford–Shah model for image labeling and segmentation. IEEE Trans. Image Process., 2020, 29: 7061-7075

[45] Li X, Xiao J, Zhou Y, Ye Y, Lv N, Wang X, Wang S, Gao S. Detail retaining convolutional neural network for image denoising. J. Vis. Commun. Image Represent., 2020, 71: 102774

[46] Lie J, Lysaker M, Tai X-C. A binary level set model and some applications to Mumford–Shah image segmentation. IEEE Trans. Image Process., 2006, 15(5): 1171-1181

[47] Liu X. Augmented Lagrangian method for total generalized variation based Poissonian image restoration. Comput. Math. Appl., 2016, 71(8): 1694-1705

[48] Lysaker M, Tai X-C. Iterative image restoration combining total variation minimization and a second-order functional. Int. J. Comput. Vision, 2006, 66(1): 5-18

[49] Michelot C. A finite algorithm for finding the projection of a point onto the canonical simplex of RN. J. Optim. Theory Appl., 1986, 50(1): 195-200

[50] Minaee S, Boykov YY, Porikli F, Plaza AJ, Kehtarnavaz N, Terzopoulos D. Image segmentation using deep learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell., 2021, 44(7): 3523-3542

[51] Mumford DB, Shah J. Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math., 1989, 42(5): 577-685

[52] Ng MK, Wang W. A total variation model for retinex. SIAM J. Imag. Sci., 2011, 4(1): 345-365

[53] Papafitsoros K, Schnlieb CB. A combined first and second order variational approach for image reconstruction. J. Math. Imaging Vis., 2014, 48(2): 308-338

[54] Rudin LI, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms. Physica D, 1992, 60(1): 259-268

[55] Shi J, Malik J. Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 2000, 22(8): 888-905

[56] Tai X-C, Hahn J, Chung GJ. A fast algorithm for Euler’s Elastica model using augmented Lagrangian method. SIAM J. Imag. Sci., 2011, 4(1): 313-344

[57] Tian C, Fei L, Zheng W, Xu Y, Zuo W, Lin C-W. Deep learning on image denoising: an overview. Neural Netw., 2020, 131: 251-275

[58] Tian C, Xu Y, Li Z, Zuo W, Liu H. Attention-guided CNN for image denoising. Neural Netw., 2020, 124: 117-129

[59] Wang, W., Wen, S., Wu, C., Deng, J.: Denoising piecewise constant images with selective averaging and outlier removal. numerical mathematics: theory, methods and applications 12(2) (2019)

[60] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 2004, 13(4): 600-612

[61] Wu C, Tai X-C. Augmented Uagrangian method, dual methods, and split Bregman iteration for ROF\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\rm ROF}$$\end{document}, vectorial TV\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\rm TV}$$\end{document}, and high order models. SIAM J. Imag. Sci., 2010, 3(3): 300-339

[62] Wu C, Zhang J, Tai X-C. Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Problems Imaging, 2011, 5(1): 237-261

[63] Wu T, Gu X, Wang Y, Zeng T. Adaptive total variation based image segmentation with semi-proximal alternating minimization. Signal Process., 2021, 183: 108017

[64] Yashtini M, Kang SH. A fast relaxed normal two split method and an effective weighted tv approach for Eulers¨\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\ddot{{\rm s}}$$\end{document} Elastica image inpainting. SIAM J. Imag. Sci., 2016, 9(4): 1552-1581

[65] Yi, F., Moon, I.: Image segmentation: a survey of graph-cut methods. In: 2012 International Conference on Systems and Informatics (ICSAI2012), pp. 1936–1941 (2012)

[66] Zhang J, Chen K. A total fractional-order variation model for image restoration with nonhomogeneous boundary conditions and its numerical solution. SIAM J. Imag. Sci., 2015, 8(4): 2487-2518

[67] Zhang M, Gunturk BK. Multiresolution bilateral filtering for image denoising. IEEE Trans. Image Process., 2008, 17(12): 2324-2333

[68] Zhang H, Peng Z. Total generalized variation for triangulated surface data. J. Sci. Comput., 2022, 93(3): 87

[69] Zhang J, Lai R, Kuo C-CJ. Adaptive directional total-variation model for latent fingerprint segmentation. IEEE Trans. Inf. Forens. Secur., 2013, 8(8): 1261-1273

[70] Zhang H, Wu C, Zhang J, Deng J. Variational mesh denoising using total variation and piecewise constant function space. IEEE Trans. Vis. Comput. Graph., 2015, 21(7): 873-886

[71] Zhang H, He Z, Wang X. A novel mesh denoising method based on relaxed second-order total generalized variation. SIAM J. Imag. Sci., 2022, 15(1): 1-22

[72] Zhong, Q., Li, Y., Yang, Y., Duan, Y.: Minimizing discrete total curvature for image processing. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

[73] Zhu L, Kang X, Ye L, Ming A. Explored normalized cut with random walk refining term for image segmentation. IEEE Trans. Image Process., 2022, 31: 2893-2906

[74] Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. In: International Conference on Computer Vision, pp. 479–486 (2011)

Rights & permissions

School of Mathematical Sciences, University of Science and Technology of China and Springer-Verlag GmbH Germany, part of Springer Nature

Just Accepted

This article has successfully passed peer review and final editorial review, and will soon enter typesetting, proofreading and other publishing processes. The currently displayed version is the accepted final manuscript. The officially published version will be updated with format, DOI and citation information upon launch. We recommend that you pay attention to subsequent journal notifications and preferentially cite the officially published version. Thank you for your support and cooperation.