Proposals from binary tree and spatio-temporal tunnel for temporal segmentation of rough videos

Abstract

Existing temporal segmentation methods suffer from the problems of high computational complexity and complicated steps. To address this issue, we present a method that combines the binary tree and spatio-temporal tunnel (STT) for temporal segmentation of rough videos. First, we compute initial cumulative spatio-temporal flow to determine flow overflow of sub-video which is divided from a rough video. Second, the decision tree is generated by combining binary tree and balance factor to dynamically adjust the sampling line of the STT. Finally, pixels on the sampling line are extracted to generate an adaptive STT for temporal proposals. Experimental results show that the computational complexity of the proposed method is significantly better than that of the comparison methods while ensuring accuracy.

References

[1] PENGJ L, ZHAOY L, WANGL M. Research on video abnormal behavior detection based on deep learning[J]. Laser & optoelectronics progress, 2021, 58(06):51-61

[2] ZHANGZ, NIEY, SUNH, et al.. Multi-view video synopsis via simultaneous object-shifting and view-switching optimization[J]. IEEE transactions on image processing, 2020, 29: 971-985

[3] LIT Y, BINGB, WUX X. Boundary discrimination and proposal evaluation for temporal action proposal generation[J]. Multimedia tools and applications, 2021, 80(02):2123-2139

[4] ANP, LIANGJ X, MAJ. LiDAR-camera-system-based 3D object detection with proposal selection and grid attention pooling[J]. Applied optics, 2022, 61(11):2998-3007

[5] MURTAZAF, YOUSAFM H, VELASTINS A. PMHI: proposals from motion history images for temporal segmentation of long uncut videos[J]. IEEE signal processing letters, 2018, 25(02):179-183

[6] QUJ J, XINY H. Combined continuous frame difference with background difference method for moving object detection[J]. Acta photonica sinica, 2014, 43(07):219-226

[7] NAWAZM, YANH. Saliency detection using deep features and affinity-based robust background subtraction[J]. IEEE transactions on multimedia, 2021, 23(01):2902-2916

[8] GUOF, WANGW G, SHENZ Y, et al.. Motion-aware rapid video saliency detection[J]. IEEE transactions on circuits and systems for video technology, 2020, 30(12):4887-4898

[9] CONGR, LEIJ, FUH, et al.. Video saliency detection via sparsity-based reconstruction and propagation[J]. IEEE transactions on image processing, 2019, 28(10):4819-4831

[10] HEILBRONF C, NIEBLESJ C, GHANEMB. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 27–30, 2016, Las Vegas, NV, USA, 2016, New York, IEEE: 1914-1923

[11] WUQ, QUOH, WUX, et al.. Fast action localization based on spatio-temporal path search[C]//Proceeding of 2017 IEEE International Conference on Image Processing (ICIP), September 18–20, 2017, Beijing, China, 2017, New York, IEEE: 3350-3354

[12] QIUJ, WANGL, WANGY, et al.. Efficient proposals: scale estimation for object proposals in pedestrian detection tasks[J]. IEEE signal processing letters, 2020, 27(01):855-859

[13] PENGW, SHIJ, ZHAOG. Spatial temporal graph deconvolutional network for skeleton-based human sction recognition[J]. IEEE signal processing letters, 2021, 28(01):244-248

[14] KUEHNEH, RICHARDA, GALLJ. A hybrid RNN-HMM approach for weakly supervised temporal action segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42(04):765-779

[15] LIUZ, WAQASM, YANGJ, et al.. A multi-task CNN for maritime target detection[J]. IEEE signal processing letters, 2021, 28(01):434-438

[16] YUG, YUANJ. Fast action proposals for human action detection and search[C]//Proceeding of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 8–10, 2015, Boston, USA, 2015, New York, IEEE: 1302-1311

[17] CHENK, WANGJ, YANGS, et al.. Optimizing video object detection via a scale-time lattice[C]//Proceeding of 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 18–21, 2018, Salt Lake City, UT, USA, 2018, New York, IEEE: 7814-7823

[18] SHENZ, LIUZ, LIJ, et al.. Object detection from scratch with deep supervision[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42(02):398-412

[19] QUS, ZHANGH, WUW, et al.. Symmetric pyramid attention convolutional neural network for moving object detection[J]. Signal, image and video processing, 2021, 15(08):1747-1755

[20] ZHANGY Z, LIW X, YANGP L. Surveillance video motion segmentation based on the progressive spatio-temporal tunnel flow model[J]. Electronics letters, 2021, 57(13):505-507

[21] ZHUANGX T. Research on deep learning networks for small object detection based on multi-level feature fusion[D], 2021, Nanjing, Nanjing University of Posts and Telecommunications

[22] VEZZANIR, CUCCHIARAR. Video surveillance online repository (VISOR): an integrated framework[J]. Multimedia tools and applications, 2010, 50(01):359-380