Joint fuzzy background and adaptive foreground model for moving target detection

Abstract

Moving target detection is one of the most basic tasks in computer vision. In conventional wisdom, the problem is solved by iterative optimization under either Matrix Decomposition (MD) or Matrix Factorization (MF) framework. MD utilizes foreground information to facilitate background recovery. MF uses noise-based weights to fine-tune the background. So both noise and foreground information contribute to the recovery of the background. To jointly exploit their advantages, inspired by two framework complementary characteristics, we propose to simultaneously exploit the advantages of these two optimizing approaches in a unified framework called Joint Matrix Decomposition and Factorization (JMDF). To improve background extraction, a fuzzy factorization is designed. The fuzzy membership of the background/foreground association is calculated during the factorization process to distinguish their contributions of both to background estimation. To describe the spatio-temporal continuity of foreground more accurately, we propose to incorporate the first order temporal difference into the group sparsity constraint adaptively. The temporal constraint is adjusted adaptively. Both foreground and the background are jointly estimated through an effective alternate optimization process, and the noise can be modeled with the specific probability distribution. The experimental results of vast real videos illustrate the effectiveness of our method. Compared with the current state-of-the-art technology, our method can usually form the clearer background and extract the more accurate foreground. Anti-noise experiments show the noise robustness of our method.

Graphical abstract

Keywords

matrix decomposition / matrix factorization / generalized sparsity / noise modeling

Dawei Zhang received the BS degree from Tiangong University, China in 2019. He is currently a Master Candidate of Hebei University of Technology, China. His research interests include computer vision and machine learning

Peng Wang received the Bachelor of Engineering degree in Computer Science and Technology, the Master of Engineering degree in Computer Science and Technology, and the PhD degree in Microelectronics and Solid State Electronics from Hebei University, China in 2001, 2004, and 2011, respectively. Now, he is an Associate Professor with the School of Artificial Intelligent, Hebei University of Technology, China. His current research interests include computer application technology, industrial vision, remote sensing image processing

Yongfeng Dong received the BS and MS degrees in computer science and technology and the PhD degree in electrical theory and new technology from the Hebei University of Technology, China in 2000, 2003, and 2008, respectively. Since 2016, he has been a Professor with the School of Artificial Intelligence, Hebei University of Technology, China. His research interests include intelligent information processing, big data technology, robot and intelligent control, and software engineering. He is a Senior Member of the China Computer Federation, a member of the Education Committee of the China Computer Federation, the Standing Director of the Hebei Computer Federation, and the Director of the Hebei Institute of Electronics

Linhao Li received the BS degree in applied mathematics, the MS degree in computational mathematics, and the PhD degree in computer science from Tianjin University, China in 2012, 2014, and 2019, respectively. He is currently an Associate Professor with the School of Artificial Intelligent, Hebei University of Technology, China. His current research interests include background modeling and foreground detection, sparse signal recovery, Incremental learning and knowledge tracing

Xin Li received the BS degree with highest honors in electronic engineering and information science from University of Science and Technology of China, China in 1996, and the PhD degree in electrical engineering from Princeton University, USA in 2000. He was a Member of Technical Staff with Sharp Laboratories of America, Camas, WA from Aug. 2000 to Dec. 2002. Since Jan. 2003, he has been a faculty member in Lane Department of Computer Science and Electrical Engineering. Dr. Li was elected a Fellow of IEEE in 2017 for his contributions to image interpolation, restoration and compression

References

[1] Garcia-Garcia B, Bouwmans T, Silva A J R . Background subtraction in real applications: challenges, current models and future directions. Computer Science Review, 2020, 35: 100204

[2] Stauffer C, Grimson W E L. Adaptive background mixture models for real-time tracking. In: Proceedings of 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 1999, 246−252

[3] Weinland D, Boyer E. Action recognition using exemplar-based embedding. In: Proceedings of 2008 IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1−7

[4] Lin Z, Zhang H. Low-Rank Models in Visual Analysis: Theories, Algorithms, and Applications. Sea Harbor Drive Orlando: Academic Press, 2017

[5] Vaswani N, Bouwmans T, Javed S, Narayanamurthy P . Robust subspace learning: robust PCA, robust subspace tracking, and robust subspace recovery. IEEE Signal Processing Magazine, 2018, 35( 4): 32–55

[6] Yong H, Meng D, Zuo W, Zhang L . Robust online matrix factorization for dynamic background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40( 7): 1726–1740

[7] Zivkovic Z. Improved adaptive Gaussian mixture model for background subtraction. In: Proceedings of the 17th International Conference on Pattern Recognition. 2004, 28−31

[8] Elgammal A, Harwood D, Davis L. Non-parametric model for background subtraction. In: Proceedings of the 6th European Conference on Computer Vision. 2000, 751−767

[9] J Candès E J, Li X, Ma Y, Wright J . Robust principal component analysis?. Journal of the ACM, 2011, 58( 3): 11

[10] Chen J, Yang J . Robust subspace segmentation via low-rank representation. IEEE Transactions on Cybernetics, 2014, 44( 8): 1432–1445

[11] Liu X, Zhao G, Yao J, Qi C . Background subtraction based on low-rank and structured sparse decomposition. IEEE Transactions on Image Processing, 2015, 24( 8): 2502–2514

[12] Guo X, Lin Z . Low-rank matrix recovery via robust outlier estimation. IEEE Transactions on Image Processing, 2018, 27( 11): 5316–5327

[13] Javed S, Mahmood A, Al-Maadeed S, Bouwmans T, Jung S K . Moving object detection in complex scene using spatiotemporal structured-sparse RPCA. IEEE Transactions on Image Processing, 2019, 28( 2): 1007–1022

[14] Boyd S, Parikh N, Chu E, Peleato B, Eckstein J . Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine Learning, 2011, 3( 1): 1–122

[15] Zhou X, Yang C, Yu W . Moving object detection by detecting contiguous outliers in the low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35( 3): 597–610

[16] Cao X, Yang L, Guo X . Total variation regularized RPCA for irregularly moving object detection under dynamic background. IEEE Transactions on Cybernetics, 2016, 46( 4): 1014–1027

[17] Xie H B, Li C, Xu R Y D, Mengersen K. Robust kernelized Bayesian matrix factorization for video background/foreground separation. In: Proceedings of the 5th International Conference on Machine Learning, Optimization, and Data Science. 2019, 484−495

[18] Meng D, De La Torre F. Robust matrix factorization with unknown noise. In: Proceedings of 2013 IEEE International Conference on Computer Vision. 2013, 1337−1344

[19] Cao X, Chen Y, Zhao Q, Meng D, Wang Y, Wang D, Xu Z. Low-rank matrix factorization under general mixture noise distributions. In: Proceedings of 2015 IEEE International Conference on Computer Vision. 2015, 1493−1501

[20] Chu Y, Wu X, Liu T, Liu J. A basis-background subtraction method using non-negative matrix factorization. In: Proceedings of SPIE 7546, Second International Conference on Digital Image Processing. 2010, 75461A

[21] Li L, Hu Q, Li X . Moving object detection in video via hierarchical modeling and alternating optimization. IEEE Transactions on Image Processing, 2019, 28( 4): 2021–2036

[22] Li L, Huang W, Gu I Y H, Tian Q . Statistical modeling of complex backgrounds for foreground object detection. IEEE Transactions on Image Processing, 2004, 13( 11): 1459–1472

[23] Brutzer S, Höferlin B, Heidemann G. Evaluation of background subtraction techniques for video surveillance. In: Proceedings of CVPR 2011. 2011, 1937−1944

[24] Bouwmans T . Recent advanced statistical background modeling for foreground detection - a systematic survey. Recent Patents on Computer Science, 2011, 4( 3): 147–176

[25] Nebili W, Farou B, Seridi H . Using resources competition and memory cell development to select the best GMM for background subtraction. International Journal of Strategic Information Technology and Applications, 2019, 10( 2): 21–43

[26] Chen M, Wei X, Yang Q, Li Q, Wang G, Yang M H . Spatiotemporal GMM for background subtraction with superpixel hierarchy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40( 6): 1518–1525

[27] Elguebaly T, Bouguila N . Background subtraction using finite mixtures of asymmetric Gaussian distributions and shadow detection. Machine Vision and Applications, 2014, 25( 5): 1145–1162

[28] Yang Y, Han D, Ding J, Yang Y. An improved ViBe for video moving object detection based on evidential reasoning. In: Proceedings of 2016 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems. 2016, 26−31

[29] Ramírez-Alonso G, Chacón-Murguía M I . Auto-adaptive parallel SOM architecture with a modular analysis for dynamic object segmentation in videos. Neurocomputing, 2016, 175: 990–1000

[30] Wang H C, Lai Y C, Cheng W H, Cheng C Y, Hua K L . Background extraction based on joint gaussian conditional random fields. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28( 11): 3127–3140

[31] Xie Y, Gu S, Liu Y, Zuo W, Zhang W, Zhang L . Weighted schatten p-norm minimization for image denoising and background subtraction. IEEE Transactions on Image Processing, 2016, 25( 10): 4842–4857

[32] Hu W, Yang Y, Zhang W, Xie Y . Moving object detection using tensor-based low-rank and saliently fused-sparse decomposition. IEEE Transactions on Image Processing, 2017, 26( 2): 724–737

[33] Yin L, Parekh A, Selesnick I . Stable principal component pursuit via convex analysis. IEEE Transactions on Signal Processing, 2019, 67( 10): 2595–2607

[34] Feng J, Xu H, Yan S. Online robust PCA via stochastic optimization. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 404−412

[35] Guo K, Liu L, Xu X, Xu D, Tao D . GoDec+: fast and robust low-rank matrix decomposition based on maximum correntropy. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29( 6): 2323–2336

[36] Cao H, Shang X, Wang Y, Song M, Chen S, Chang C I. GO decomposition (GoDec) approach to finding low rank and sparsity matrices for hyperspectral target detection. In: Proceedings of 2020 IEEE International Geoscience and Remote Sensing Symposium. 2020, 2807−2810

[37] Shakeri M, Zhang H . COROLA: a sequential solution to moving object detection using low-rank approximation. Computer Vision and Image Understanding, 2016, 146: 27–39

[38] Javed S, Oh S H, Sobral A, Bouwmans T, Jung S K. OR-PCA with MRF for robust foreground detection in highly dynamic backgrounds. In: Proceedings of the 12th Asian Conference on Computer Vision. 2014, 284−299

[39] Javed S, Oh S H, Bouwmans T, Jung S K . Robust background subtraction to global illumination changes via multiple features-based online robust principal components analysis with Markov random field. Journal of Electronic Imaging, 2015, 24( 4): 043011

[40] Li C, Wang X, Zhang L, Tang J, Wu H, Lin L . Weighted low-rank decomposition for robust grayscale-thermal foreground detection. IEEE Transactions on Circuits and Systems for Video Technology, 2017, 27( 4): 725–738

[41] Zhu L, Hao Y, Song Y . L_1/2 norm and spatial continuity regularized low-rank approximation for moving object detection in dynamic background. IEEE Signal Processing Letters, 2018, 25( 1): 15–19

[42] Xu Y, Wu Z, Chanussot J, Mura M D, Bertozzi A L, Wei Z . Low-rank decomposition and total variation regularization of hyperspectral video sequences. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56( 3): 1680–1694

[43] Zhang J, Jia X, Hu J . Error bounded foreground and background modeling for moving object detection in satellite videos. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58( 4): 2659–2669

[44] Wu M, Sun Y, Hang R, Liu Q, Liu G . Multi-component group sparse RPCA model for motion object detection under complex dynamic background. Neurocomputing, 2018, 314: 120–131

[45] Liu X, Zhao G . Background subtraction using multi-channel fused lasso. Electronic Imaging, 2019, 2019( 11): 269

[46] Xin B, Tian Y, Wang Y, Gao W. Background subtraction via generalized fused lasso foreground modeling. In: Proceedings of 2015 IEEE conference on Computer Vision and Pattern Recognition. 2015, 4676−4684

[47] Javed S, Oh S H, Sobral A, Bouwmans T, Jung S K. Background subtraction via superpixel-based online matrix decomposition with structured foreground constraints. In: Proceedings of 2015 IEEE International Conference on Computer Vision Workshop. 2015, 930−938

[48] Ye X, Yang J, Sun X, Li K, Hou C, Wang Y . Foreground-background separation from video clips via motion-assisted matrix restoration. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25( 11): 1721–1734

[49] Cao X, Zhao Q, Meng D, Chen Y, Xu Z . Robust low-rank matrix factorization under general mixture noise distributions. IEEE Transactions on Image Processing, 2016, 25( 10): 4677–4690

[50] Liu Q, Li X . Efficient low-rank matrix factorization based on ℓ_1,ε-norm for online background subtraction. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32( 7): 4900–4904

[51] Zhu Z, Meng Y, Kong D, Zhang X, Guo Y, Zhao Y . To see in the dark: N2DGAN for background modeling in nighttime scene. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31( 2): 492–502

[52] Patil P W, Murala S . MSFgNet: a novel compact end-to-end deep network for moving object detection. IEEE Transactions on Intelligent Transportation Systems, 2019, 20( 11): 4066–4077

[53] Zhao C, Basu A . Dynamic deep pixel distribution learning for background subtraction. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 11): 4192–4206

[54] Hou B, Liu Y, Ling N. A super-fast deep network for moving object detection. In: Proceedings of 2020 IEEE International Symposium on Circuits and Systems. 2020, 1−5

[55] Patil P W, Murala S, Dhall A, Chaudhary S. MsEDNet: multi-scale deep saliency learning for moving object detection. In: Proceedings of 2018 IEEE International Conference on Systems, Man, and Cybernetics. 2018, 1670−1675

[56] Yan L F, Tu X Y. Background modeling based on chebyshev approximation. Journal of System Simulation, 2008, 20(4): 944−946, 1001

[57] Stagliano A, Noceti N, Verri A, Odone F . Online space-variant background modeling with sparse coding. IEEE Transactions on Image Processing, 2015, 24( 8): 2415–2428

[58] Messelodi S, Modena M C, Segata N, Zanin M. A Kalman filter based background updating algorithm robust to sharp illumination changes. In: Proceedings of the 13th International Conference on Image Analysis and Processing. 2005, 163−170

[59] Giraldo J H, Javed S, Sultana M, Jung S K, Bouwmans T. The emerging field of graph signal processing for moving object segmentation. In: Proceedings of the 27th International Workshop on Frontiers of Computer Vision. 2021, 31−45

[60] Pólya G. Isoperimetric Inequalities in Mathematical Physics. Princeton: Princeton University Press, 1951

[61] Tekalp A M. Digital Video Processing. 2nd ed. Upper Saddle River: Prentice Hall Press, 2015

[62] Li L, Wang Z, Hu Q, Dong Y . Adaptive nonconvex sparsity based background subtraction for intelligent video surveillance. IEEE Transactions on Industrial Informatics, 2021, 17( 6): 4168–4178

[63] Javed S, Narayanamurthy P, Bouwmans T, Vaswani N. Robust PCA and robust subspace tracking: a comparative evaluation. In: Proceedings of 2018 IEEE Statistical Signal Processing Workshop. 2018, 836−840

[64] Bouwmans T, Aybat N S, Zahzah E H. Handbook of Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing. Boca Raton: CRC Press, 2016

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant No. 61902106) and in part by the Natural Science Foundation of Hebei Province (No. F2020202028).

RIGHTS & PERMISSIONS

2024 Higher Education Press