High-Movement Human Segmentation in Video Using Adaptive N-Frames Ensemble

Title

High-Movement Human Segmentation in Video Using Adaptive N-Frames Ensemble

Creator

Kim Y.-W.; Byun Y.-C.; Han D.S.; Dominic D.; Cyriac S.

Description

A wide range of camera apps and online video conferencing services support the feature of changing the background in real-time for aesthetic, privacy, and security reasons. Numerous studies show that the Deep-Learning (DL) is a suitable option for human segmentation, and the ensemble of multiple DL-based segmentation models can improve the segmentation result. However, these approaches are not as effective when directly applied to the image segmentation in a video. This paper proposes an Adaptive N-Frames Ensemble (AFE) approach for high-movement human segmentation in a video using an ensemble of multiple DL models. In contrast to an ensemble, which executes multiple DL models simultaneously for every single video frame, the proposed AFE approach executes only a single DL model upon a current video frame. It combines the segmentation outputs of previous frames for the final segmentation output when the frame difference is less than a particular threshold. Our method employs the idea of the N-Frames Ensemble (NFE) method, which uses the ensemble of the image segmentation of a current video frame and previous video frames. However, NFE is not suitable for the segmentation of fast-moving objects in a video nor a video with low frame rates. The proposed AFE approach addresses the limitations of the NFE method. Our experiment uses three human segmentation models, namely Fully Convolutional Network (FCN), DeepLabv3, and Mediapipe. We evaluated our approach using 1711 videos of the TikTok50f dataset with a single-person view. The TikTok50f dataset is a reconstructed version of the publicly available TikTok dataset by cropping, resizing and dividing it into videos having 50 frames each. This paper compares the proposed AFE with single models and the Two-Models Ensemble, as well as the NFE models. The experiment results show that the proposed AFE is suitable for low-movement as well as high-movement human segmentation in a video. 2022 Tech Science Press. All rights reserved.

Source

Computers, Materials and Continua, Vol-73, No. 3, pp. 4743-4762.

Date

2022-01-01

Publisher

Tech Science Press

Subject

artificial intelligence; deep learning; ensemble; High movement; human segmentation; video instance segmentation

Coverage

Kim Y.-W., Centre for Digital Innovation, CHRIST (Deemed to be University), Bangalore, 560029, India; Byun Y.-C., Department of Computer Engineering, Jeju National University, Jeju, 63243, South Korea; Han D.S., School of Electronics Engineering, Kyungpook National University, Daegu, 41566, South Korea; Dominic D., Centre for Digital Innovation, CHRIST (Deemed to be University), Bangalore, 560029, India; Cyriac S., Centre for Digital Innovation, CHRIST (Deemed to be University), Bangalore, 560029, India

Rights

All Open Access; Gold Open Access

Relation

ISSN: 15462218

Format

Online

Language

English

Type

Article

Identifier

https://doi.org/10.32604/cmc.2022.028632

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85135060531&doi=10.32604%2fcmc.2022.028632&partnerID=40&md5=5f84861c79f1e8a86b03c0abbc4200c0