Accelerating temporal action proposal generation via high performance computing

Abstract

Temporal action proposal generation aims to output the starting and ending times of each potential action for long videos and often suffers from high computation cost. To address the issue, we propose a new temporal convolution network called Multipath Temporal ConvNet (MTCN). In our work, one novel high performance ring parallel architecture based is further introduced into temporal action proposal generation in order to respond to the requirements of large memory occupation and a large number of videos. Remarkably, the total data transmission is reduced by adding a connection between multiplecomputing load in the newly developed architecture. Compared to the traditional Parameter Server architecture, our parallel architecture has higher efficiency on temporal action detection tasks with multiple GPUs. We conduct experiments on ActivityNet-1.3 and THUMOS14, where our method outperformsother state-of-art temporal action detection methods with high recall and high temporal precision. In addition, a time metric is further proposed here to evaluate the speed performancein the distributed training process.

Graphical abstract

Keywords

temporal convolution / temporal action proposal generation / deep learning

References

[1] Muhammad K , Hamza R , Ahmad J , Lloret J , Wang H , Baik S . Secure surveillance framework for IoT systems using probabilistic image encryption. IEEE Transactions on Industrial Informatics, 2018, 14( 8): 3679– 3689

[2] Sajjad M , Haq I U , Lloret J , Ding W , Muhammad K . Robust image hashing based efficient authentication for smart industrial environment. IEEE Transactions on Industrial Informatics, 2019, 15( 12): 6541– 6550

[3] Wang T , Qiao M , Lin Z , Li C , Snoussi H , Liu Z , Choi C . Generative neural networks for anomaly detection in crowded scenes. IEEE Transactions on Information Forensics and Security, 2018, 14( 5): 1390– 1399

[4] Muhammad K , Khan S , Palade V , Mehmood I , De Albuquerque V H . Edge intelligence-assisted smoke detection in foggy surveillance environments. IEEE Transactions on Industrial Informatics, 2019, 16( 2): 1067– 1075

[5] Wang T , Miao Z , Chen Y , Zhou Y , Shan G , Snoussi H . Aed-net: an abnormal event detection network. Engineering, 2019, 5( 5): 930– 939

[6] Caba Heilbron F, Escorcia V, Ghanem B, Carlos Niebles J. Activitynet: a large-scale video benchmark for human activity understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 961−970

[7] Jiang Y G, Liu J, Zamir A. R, Toderici G, Laptev I, Shah M, Sukthankar R. Thumos challenge: action recognition with a large number of classes. 2014

[8] Lin T, Zhao X, Su H, Wang C, Yang M. BSN: boundary sensitive network for temporal action proposal generation. In: Proceedings of the European Conference on Computer Vision. 2018, 3−19

[9] Buch S, Escorcia V, Shen C, Ghanem B, Carlos Niebles J. SST: singlestream temporal action proposals. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 2911−2920

[10] Caba Heilbron F, Carlos Niebles J, Ghanem B. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1914−1923

[11] Escorcia V, Heilbron F. C, Niebles J. C, Ghanem B. Daps: deep action proposals for action understanding. In: Proceedings of the European Conference on Computer Vision. 2016, 768– 784

[12] Shou Z, Wang D, Chang SF. Temporal action localization in untrimmed videos via multi-stage cnns. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1049−1058

[13] Dean J, Corrado G, Monga R, Chen K, Devin M, Mao M, Senior A, Tucker P, Yang K, Le Q V, et al. Large scale distributed deep networks. In: Proceedings of the Advances in Neural Information Processing Systems. 2012, 1223–1231

[14] Karaman S, Seidenari L, Del Bimbo A. Fast saliency based pooling of fisher encoded dense trajectories. In: Proceedings of the European Conference on Computer Vision THUMOS Workshop. 2014

[15] Wang L , Qiao Y , Tang X . Action recognition and detection by combining motion and appearance features. THUMOS14 Action Recognition Challenge, 2014, 1( 2): 2–

[16] Wang T, Chen Y, Lin Z, Zhu A, Li Y, Snoussi H, Wang H. Recapnet: action proposal generation mimicking human cognitive process. IEEE Transactions on Cybernetics, 2020, DOI: 10.1109/TCYB.2020.2965196

[17] Gao J, Yang Z, Chen K, Sun C, Nevatia R. Turn tap: temporal unit regression network for temporal action proposals. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 3628−3636

[18] Zhao Y, Xiong Y, Wang L, Wu Z, Tang X, Lin D. Temporal action detection with structured segment networks. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 2914−2923

[19] Jian M , Lam K M , Dong J , Shen L . Visual-patch-attention-aware saliency detection. IEEE Transactions on Cybernetics, 2014, 45( 8): 1575– 1586

[20] Jian M , Qi Q , Dong J , Yin Y , Lam K M . Integrating qdwd with pattern distinctness and local contrast for underwater saliency detection. Journal of Visual Communication and Image Representation, 2018, 53 : 31– 41

[21] Jian M , Qi Q , Yu H , Dong J , Cui C , Nie X , Zhang H , Yin Y , Lam K M . The extended marine underwater environment database and baseline evaluations. Applied Soft Computing, 2019, 80 : 425– 437

[22] Wang T , Chen Y , Lv H , Teng J , Snoussi H , Tao F . Online detection of action start via soft computing for smart city. IEEE Transactions on Industrial Informatics, 2020, 17( 1): 524– 533

[23] Wang H, Kläser A, Schmid C, Liu C L. Action recognition by dense trajectories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 3169−3176

[24] Feichtenhofer C, Pinz A, Zisserman A. Convolutional two-stream network fusion for video action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1933−1941

[25] Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, Van Gool L. Temporal segment networks: towards good practices for deep action recognition. In: Proceedings of the European Conference on Computer Vision. 2016, 20– 36

[26] Tran D, Bourdev L, Fergus R, Torresani L, Paluri M. Learning spatiotemporal features with 3d convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 4489−4497

[27] Dean J , Ghemawat S . Mapreduce: simplified data processing on large clusters. Communications of the ACM, 2008, 51( 1): 107– 113

[28] Low Y , Bickson D , Gonzalez J , Guestrin C , Kyrola A , Hellerstein J M . Distributed graphlab: a framework for machine learning and data mining in the cloud. Proceedings of the VLDB Endowment, 2012, 5( 8): 716– 727

[29] Hu J, Shen L, Sun G. Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 7132−7141

[30] Simonyan K, Zisserman A. Two-stream convolutional networks for action recognition in videos. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 568−576

[31] Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G S, Davis A, Dean J, Devin M, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. 2016, arXiv preprint arXiv: 1603.04467

[32] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. 2015, arXiv preprint arXiv: 1502.03167

[33] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770−778

[34] Dai X, Singh B, Zhang G, Davis L. S, Qiu Chen Y. Temporal context network for activity localization in videos. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 5793– 5802

[35] Ghanem B, Niebles J C, Snoek C, Heilbron F C, Alwassel H, Khrisna R, Escorcia V, Hata K, Buch S. Activitynet challenge 2017 summary. 2017, arXiv preprint arXiv: 1710.08011

Acknowledgements

This work was partially supported by the National Key Research and Development Program of China (2016YFE0204200), the National Natural Science Foundation of China (Grant Nos. 61972016, 62032016), Beijing Natural Science Foundation (L191007), the Fundamental Research Funds for the Central Universities (YWF-21-BJ-J-313 and YWF-20-BJ-J-612), Open Research Fund of Digital Fujian Environment Monitoring Internet of Things Laboratory Foundation (202004). The experimental platform is provided by Marc Casas at the Barcelona Supercomputing Center (BSC).

RIGHTS & PERMISSIONS

2022 Higher Education Press