Memory-boosting RNN with dynamic graph for event-based action recognition

Abstract

Existing action recognition methods based on event cameras have not fully exploited the advantages of event cameras, such as compressing event streams into frames for subsequent calculation, which greatly sacrifices the time information of event streams. Meanwhile, the conventional PointCloud-based methods suffer from large computational complexity while processing event data, which make it difficult to handle long-term actions. To tackle the above problems, we propose a dynamic graph memory-boosting recurrent neural network (DG-MBRNN). The proposed DG-MBRNN splits the event stream into sequential graph data for preserving structural information, then uses the recurrent neural network (RNN) with boosting spatiotemporal memory to handle long-term sequences of actions. In addition, the proposed method introduces a dynamic reorganization mechanism for the graph based on the distances of features, which can effectively increase the ability to extract local features. In order to cope with the situation that the existing datasets have too simple actions and too limited categories, we propose a new event-based dataset containing 36 complex actions. This dataset will greatly promote the development of event-based action recognition research. Experimental results show the effectiveness of the proposed method in completing the event-based action recognition task.

References

[1] ROHAN G. Spatiotemporal filtering for event-based action recognition[EB/OL]. (2019-03-19) [2023-02-20]. https://doi.org/10.48550/arXiv.1903.07067.

[2] HUANG C X. Event-based action recognition using timestamp image encoding network[EB/OL]. (2020-09-28) [2023-02-20]. https://doi.org/10.48550/arXiv.2009.13049.

[3] LiuC, QiX, LamE Y, et al.. Fast classification and action recognition with event-based imaging[J]. IEEE access, 2022, 10: 55638-55649

[4] MaroJ M, IengS H, BenosmanR. Event-based gesture recognition with dynamic background suppression using smartphone computational capabilities[J]. Frontiers in neuroscience, 2020, 14: 275

[5] InnocentiS U, BecattiniF, PerniciF, et al.. Temporal binary representation for event-based action recognition[C], 2021, New York, IEEE: 10426-10432

[6] WangQ, ZhangY, YuanJ, et al.. Space-time event clouds for gesture recognition: from RGB cameras to event cameras[C], 2019, New York, IEEE: 1826-1835

[7] YangJ, ZhangQ, NiB, et al.. Modeling point clouds with self-attention and gumbel subset sampling[C], 2019, New York, IEEE: 3323-3332

[8] WangQ, ZhangY, YuanJ, et al.. ST-EVNet: hierarchical spatial and temporal feature learning on space-time event clouds[C], 2020, New York, IEEE

[9] ChenJ, MengJ, WangX, et al.. Dynamic graph CNN for event-camera based gesture recognition[C], 2020, New York, IEEE: 1-5

[10] BiY, ChadhaA, AbbasA, et al.. Graph-based spatio-temporal feature learning for neuromorphic vision sensing[J]. IEEE transactions on image processing, 2020, 29: 9084-9098

[11] SongY, LiuG, WangG, et al.. SDN traffic prediction based on graph convolutional network[J]. Computer science, 2021, 48(6A):392-397

[12] WANG Y, LONG M, WANG J, et al. PredRNN recurrent neural networks for predictive learning using spatiotemporal LSTMs[J]. Advances in neural information processing systems, 2017, 30.

[13] ScarpelliniG, MorerioP, Del BueA. Lifting monocular events to 3D human poses[C], 2021, New York, IEEE: 1358-1368Online

[14] MiaoS, ChenG, NingX, et al.. Neuromorphic vision datasets for pedestrian detection, action recognition, and fall detection[J]. Frontiers in neurorobotics, 2019, 13: 38

[15] AmirA, TabaB, BergD, et al.. A low power, fully event-based gesture recognition system[C], 2017, New York, IEEE: 7243-7252

[16] LeeK H, WooH, SukT. Point data reduction using 3D grids[J]. The international journal of advanced manufacturing technology, 2001, 18: 201-210