Multimodal emotional analysis through hierarchical video summarization and face tracking

Title

Multimodal emotional analysis through hierarchical video summarization and face tracking

Creator

Thiruthuvanathan M.M.; Krishnan B.

Description

The era of video data has fascinated users into creating, processing, and manipulating videos for various applications. Voluminous video data requires higher computation power and processing time. In this work, a model is developed that can precisely acquire keyframes through hierarchical summarization and use the keyframes to detect faces and assess the emotional intent of the user. The key-frames are used to detect faces using recursive Viola-Jones algorithm and an emotional analysis for the faces extracted is conducted using an underlying architecture developed based on Deep Neural Networks (DNN). This work has significantly contributed in improving the accuracy of face detection and emotional analysis in non-redundant frames. The number of frames selected after summarization was less than 30% using the local minima extraction. The recursive routine introduced for face detection reduced false positives in all the video frames to lesser than 2%. The accuracy of emotional prediction on the faces acquired through the summarized frames, on Indian faces achieved a 90%. The computational requirement scaled down to 40% due to the hierarchical summarization that removed redundant frames and recursive face detection removed false localization of faces. The proposed model intends to emphasize the importance of keyframe detection and use them for facial emotional recognition. 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.

Source

Multimedia Tools and Applications, Vol-81, No. 25, pp. 35535-35554.

Date

2022-01-01

Publisher

Springer

Subject

Deep neural networks; Emotional intelligence; Entropy; Face detection; Hierarchical summarization; Video summarization; Viola Jones

Coverage

Thiruthuvanathan M.M., Department of Computer Science and Engineering, School of Engineering and Technology, CHRIST (Deemed to be University), Bangalore, India; Krishnan B., Department of Computer Science and Engineering, School of Engineering and Technology, CHRIST (Deemed to be University), Bangalore, India

Rights

Restricted Access

Relation

ISSN: 13807501; CODEN: MTAPF

Format

Online

Language

English

Type

Article

Identifier

https://doi.org/10.1007/s11042-021-11010-y

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85106472136&doi=10.1007%2fs11042-021-11010-y&partnerID=40&md5=88b39e18f88ebd344befd233d85493e9