Robust visual tracking using temporal regularization correlation filter with high-confidence strategy

Abstract

Target tracking is an essential task in contemporary computer vision applications. However, its effectiveness is susceptible to model drift, due to the different appearances of targets, which often compromises tracking robustness and precision. In this paper, a universally applicable method based on correlation filters is introduced to mitigate model drift in complex scenarios. It employs temporal-confidence samples as a priori to guide the model update process and ensure its precision and consistency over a long period. An improved update mechanism based on the peak side-lobe to peak correlation energy (PSPCE) criterion is proposed, which selects high-confidence samples along the temporal dimension to update temporal-confidence samples. Extensive experiments on various benchmarks demonstrate that the proposed method achieves a competitive performance compared with the state-of-the-art methods. Especially when the target appearance changes significantly, our method is more robust and can achieve a balance between precision and speed. Specifically, on the object tracking benchmark (OTB-100) dataset, compared to the baseline, the tracking precision of our model improves by 8.8%, 8.8%, 5.1%, 5.6%, and 6.9% for background clutter, deformation, occlusion, rotation, and illumination variation, respectively. The results indicate that this proposed method can significantly enhance the robustness and precision of target tracking in dynamic and challenging environments, offering a reliable solution for applications such as real-time monitoring, autonomous driving, and precision guidance.

Keywords

Appearance changes / Correlation filter / High-confidence strategy / Temporal regularization / Visual tracking

CRediT authorship contribution statement

Xiao-Gang Dong: Writing – original draft, Supervision, Resources, Methodology, Investigation, Data curation, Conceptualization, Project administration. Ke-Xuan Li: Writing – review & editing, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Hong-Xia Mao: Resources, Investigation, Data curation, Visualization, Funding acquisition. Chen Hu: Writing – review & editing, Validation, Formal analysis, Investigation, Data curation. Tian Pu: Supervision, Project administration, Investigation, Funding acquisition, Software.

Declaration of competing interest

The authors declare that they have no conflicts of interest in the research presented in this manuscript.

Acknowledgements

This work was supported by the Natural Science Foundation of Sichuan Province of China under Grant No. 2025ZNSFSC0522 and partially supported by the National Natural Science Foundation of China under Grants No. 61775030 and No. 61571096.

References

[1] P.-X. Li, D. Wang, L.-J. Wang, H.-C. Lu, Deep visual tracking: review and experimental comparison, Pattern Recogn. 76 (2018) 323-338.

[2] S.-H. Ding, Q. Zhai, Y. Li, J.-D. Zhu, Y.-F. Zheng, D. Xuan, Simultaneous body part and motion identification for human-following robots, Pattern Recogn. 50 (2016) 118-130.

[3] B.-Y. Wu, B.-G. Hu, Q. Ji, A coupled hidden Markov random field model for simultaneous face clustering and tracking in videos, Pattern Recogn. 64 (2017) 361-373.

[4] M.-H. Li, L.-B. Peng, Y.-P. Chen, S.-Q. Huang, F.-Y. Qin, Z.-M. Peng, Mask sparse representation based on semantic features for thermal infrared target tracking, Remote Sens. 11 (17) (2019) 1967.

[5] M.-H. Li, L.-B. Peng, T.-F. Wu, Z.-M. Peng, A bottom-up and top-down integration framework for online object tracking, IEEE T. Multimedia 23 (2020) 105-119.

[6] M.-N. Dai, G. Xiao, S.-Y. Cheng, D.-D. Wang, X.-J. He, Structural correlation filters combined with a Gaussian particle filter for hierarchical visual tracking, Neurocomputing 398 (2020) 235-246.

[7] V. Arulalan, V. Premanand, D. Kumar, Object detection and tracking using TSM-EFFICIENTDET and JS-KM in adverse weather conditions, J. Intell. Fuzzy Syst. 46 (1) (2024) 2399-2413.

[8] J.-M. Zhang, X.-K. Jin, J. Sun, J. Wang, K.-Q. Li, Dual model learning combined with multiple feature selection for accurate visual tracking, IEEE Access 7 (2019) 43956-43969.

[9] Z.-Y. Huang, C.-H. Fu, Y.-M. Li, F.-L. Lin, P. Lu, Learning aberrance repressed correlation filters for real-time UAV tracking, in: Proc. of the IEEE/CVF Intl. Conf. on Computer Vision, Seoul, Republic of Korea, (2019), pp. 2891-2900.

[10] D.S. Bolme, J.R. Beveridge, B.A. Draper, Y.-M. Lui, Visual object tracking using adaptive correlation filters, in: Proc. of IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, San Francisco, USA, (2010), pp. 2544-2550.

[11] J.F. Henriques, R. Caseiro, P. Martins, J. Batista, Exploiting the circulant structure of tracking-by-detection with kernels, in: Proc. of the 12th European Conf. on Computer Vision, Florence, Italy, (2012), pp. 702-715.

[12] J.F. Henriques, R. Caseiro, P. Martins, J. Batista, High-speed tracking with kernelized correlation filters, IEEE T. Pattern Anal. 37 (3) (2015) 583-596.

[13] M. Danelljan, G. Häger, F.S. Khan, M. Felsberg, Learning spatially regularized correlation filters for visual tracking, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Santiago, Chile, (2015), pp. 4310-4318.

[14] H.K. Galoogahi, A. Fagg, S. Lucey, Learning background-aware correlation filters for visual tracking, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Venice, Italy, (2017), pp. 1144-1152.

[15] H.K. Galoogahi, T. Sim, S. Lucey, Correlation filters with limited boundaries, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Boston, USA, (2015), pp. 4603-4638.

[16] F. Li, C. Tian, W.-M. Zuo, L. Zhang, M.-H. Yang, Learning spatial-temporal regularized correlation filters for visual tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, (2018), pp. 4904-4913.

[17] M. Danelljan, A. Robinson, F.S. Khan, M. Felsberg, Beyond correlation filters: learning continuous convolution operators for visual tracking, in: Proc. of the 14th European Conf. on Computer Vision, Amsterdam, The Netherlands, (2016), pp. 472-488.

[18] Y. Li, J.-K. Zhu, A scale adaptive kernel correlation filter tracker with feature integration, in: Proc. of the European Conf. on Computer Vision, Zurich, Switzerland, (2015), pp. 254-265.

[19] L. Bertinetto, J. Valmadre, S. Golodetz, O. Miksik, P.H.S. Torr, Staple: complementary learners for real-time tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, (2016), pp. 1401-1409.

[20] M. Danelljan, G. Häger, F.S. Khan, M. Felsberg, Accurate scale estimation for robust visual tracking, in: Proc. of the British Machine Vision Conf., Nottingham, UK, (2014), pp. 1-11.

[21] A. Lukezic, T. Vojir, L.C. Zajc, J. Matas, M. Kristan, Discriminative correlation filter with channel and spatial reliability, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, USA, (2017), pp. 4847-4856.

[22] C. Ma, J.-B. Huang, X.-K. Yang, M.-H. Yang, Hierarchical convolutional features for visual tracking, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Santiago, Chile, (2015), pp. 3074-3082.

[23] J.-R. Cai, M.-Z. Xu, W. Li, et al., MeMOT: multi-object tracking with memory, in: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, New Orleans, USA, (2022), pp. 8090-8100.

[24] Z. Qin, S.-P. Zhou, L. Wang, J.-H. Duan, G. Hua, W. Tang, MotionTrack: learning robust short-term and long-term motions for multi-object tracking, in: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Vancouver, Canada, (2023), pp. 17939-17948.

[25] B. Van Hoorick, P. Tokmakov, S. Stent, J. Li, C. Vondrick, Tracking through containers and occluders in the wild, in: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Vancouver, Canada, (2023), pp. 13802-13812.

[26] H. Ren, S.-D. Han, H.-L. Ding, Z.-W. Zhang, H.-W. Wang, F.-Q. Wang, Focus on details: online multi-object tracking with diverse fine-grained representation, in: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Vancouver, Canada, (2023), pp. 11289-11298.

[27] C. Ma, X.-K. Yang, C.-Y. Zhang, M.-H. Yang, Long-term correlation tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Boston, USA, (2015), pp. 5388-5396.

[28] K. Crammer, O. Dekel, J. Keshet, S. Shalev-Shwartz, Y. Singer, M.K. Warmuth, Online passive-aggressive algorithms, J. Mach. Learn. Res. 7 (19) (2006) 551-585.

[29] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Found. Trends Mach. Le. 3 (1) (2011) 1-122.

[30] J. Sherman, W.J. Morrison, Adjustment of an inverse matrix corresponding to a change in one element of a given matrix, Ann. Math. Stat. 21 (1) (1950) 124-127.

[31] M.-M. Wang, Y. Liu, Z.-Y. Huang, Large margin object tracking with circulant feature maps, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, USA, (2017), pp. 4800-4808.

[32] P. Liang, E. Blasch, H. Ling, Encoding color information for visual tracking: algorithms and benchmark, IEEE T. Image Process. 24 (12) (2015) 5630-5644.

[33] U.T. Benchmark, A benchmark and simulator for UAV tracking, in: Proc. of the European Conf. on Computer Vision, Amsterdam, The Netherlands, (2016), pp. 1-14.

[34] M. Kristan, J. Matas, A. Leonardis, et al., A novel performance evaluation methodology for single-target trackers, IEEE T. Pattern Anal. 38 (11) (2016) 2137-2155.

[35] H. Fan, L.-T. Lin, F. Yang, et al., LaSOT: a high-quality benchmark for large-scale single object tracking, in: Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, Long Beach, USA, (2019), pp. 5369-5378.

[36] Y. Wu, J. Lim, M.-H. Yang, Online object tracking: a benchmark, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Portland, USA, (2013), pp. 2411-2418.

[37] Y.-K. Qi, S.-P. Zhang, L. Qin, et al., Hedged deep tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, (2016), pp. 4303-4311.

[38] M. Danelljan, G. Bhat, F.S. Khan, M. Felsberg, ECO: efficient convolution operators for tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Honolulu, USA, (2017), pp. 6931-6939.

[39] L. Bertinetto, J. Valmadre, J.F. Henriques, A. Vedaldi, P.H.S. Torr, Fully-convolutional Siamese networks for object tracking, in: Proc. of the European Conf. on Computer Vision, Amsterdam, The Netherlands, (2016), pp. 850-965.

[40] B. Li, J.-J. Yan, W. Wu, Z. Zhu, X.-L. Hu, High performance visual tracking with Siamese region proposal network, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, (2018), pp. 8971-8980.

[41] J.-M. Zhang, S.-G. Ma, S. Sclaroff, MEEM: robust tracking via multiple experts using entropy minimization, in: Proc. of the 13th European Conf. on Computer Vision, Zurich, Switzerland, (2014), pp. 188-203.

[42] N. Wang, W. Zhou, Q. Tian, et al., Multi-cue correlation filters for robust visual tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, (2018), pp. 4844-4853.

[43] J. Karoliny, B. Etzlinger, A. Springer, Mixture density networks for WSN localization, in: Proc. of the IEEE Intl. Conf. on Communications Workshops, Dublin, Ireland, (2020), pp. 1-5.

[44] Y. Song, C. Ma, X. Wu, et al., VITAL: visual tracking via adversarial learning, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, (2018), pp. 8990-8999.

[45] J. Zhu, D. Wang, H. Lu, Visual tracking by learning spatiotemporal consistency in correlation filters, Sci. China Inf. Sci. 50 (1) (2020) 128-150.

[46] Y. Zhang, L. Wang, J. Qi, et al., Structured Siamese network for real-time visual tracking, in: Proc. of the European Conf. on Computer Vision, Munich, Germany, (2018), pp. 351-366.

[47] D. Held, S. Thrun, S. Savarese, Learning to track at 100 fps with deep regression networks, in: Proc. of the 14th European Conf. on Computer Vision, Amsterdam, The Netherlands, (2016), pp. 749-765.

[48] J. Choi, H.J. Chang, T. Fischer, et al., Context-aware deep feature compression for high-speed visual tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Salt Lake City, USA, (2018), pp. 479-488.

[49] G. Zhang, Z. Li, J. Li, et al., CEDNet: a cascade encoder-decoder network for dense prediction [Online]. Available, https://arxiv.org/abs/2302.06052, Feburary 2023.

[50] H. Fan, H. Ling, Parallel tracking and verifying: a framework for real-time and high accuracy visual tracking, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Venice, Italy, (2017), pp. 5486-5494.

[51] C. Ma, J.B. Huang, X. Yang, et al., Hierarchical convolutional features for visual tracking, in: Proc. of the IEEE Intl. Conf. on Computer Vision, Santiago, Chile, (2015), pp. 3074-3082.

[52] Z. Kalal, K. Mikolajczyk, J. Matas, Tracking-learning-detection, IEEE T. Pattern Anal. Mach. Intell. 34 (7) (2012) 1409-1422.

[53] J. Choi, H.J. Chang, J. Jeong, et al., Visual tracking using attention-modulated disintegration and integration, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Las Vegas, USA, (2016), pp. 4321-4330.

[54] M. Danelljan, F.S. Khan, M. Felsberg, et al., Adaptive color attributes for real-time visual tracking, in: Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, Columbus, USA, (2014), pp. 1090-1097.

[55] H. Nam, M. Baek, B. Han, Modeling and propagating CNNs in a tree structure for visual tracking [Online]. Available, https://arxiv.org/abs/1608.07242, August 2016.