Video-Based Deception Detection with Non-Contact Heart Rate Monitoring and Multi-Modal Feature Selection

Abstract

Deception detection plays a crucial role in criminal investigation. Videos contain a wealth of information regarding apparent and physiological changes in individuals, and thus can serve as an effective means of deception detection. In this paper, we investigate video-based deception detection considering both apparent visual features such as eye gaze, head pose and facial action unit (AU), and non-contact heart rate detected by remote photoplethysmography (rPPG) technique. Multiple wrapper-based feature selection methods combined with the K-nearest neighbor (KNN) and support vector machine (SVM) classifiers are employed to screen the most effective features for deception detection. We evaluate the performance of the proposed method on both a self-collected physiological-assisted visual deception detection (PV3D) dataset and a public bag-of-lies (BOL) dataset. Experimental results demonstrate that the SVM classifier with symbiotic organisms search (SOS) feature selection yields the best overall performance, with an area under the curve (AUC) of 83.27% and accuracy (ACC) of 83.33% for PV3D, and an AUC of 71.18% and ACC of 70.33% for BOL. This demonstrates the stability and effectiveness of the proposed method in video-based deception detection tasks.

Keywords

deception detection / apparent visual features / remote photoplethysmography / non-contact heart rate / feature selection