LiST: A Lightweight Framework for Continuous Indian Sign Language Translation

Title

LiST: A Lightweight Framework for Continuous Indian Sign Language Translation

Creator

K A.; P P.; Poonia R.C.

Description

Sign language is a natural, structured, and complete form of communication to exchange information. Non-verbal communicators, also referred to as hearing impaired and hard of hearing (HI&HH), consider sign language an elemental mode of communication to convey information. As this language is less familiar among a large percentage of the human population, an automatic sign language translator that can act as an interpreter and remove the language barrier is mandatory. The advent of deep learning has resulted in the availability of several sign language translation (SLT) models. However, SLT models are complex, resulting in increased latency in language translation. Furthermore, SLT models consider only hand gestures for further processing, which might lead to the misinterpretation of ambiguous sign language words. In this paper, we propose a lightweight SLT framework, LiST (Lightweight Sign language Translation), that simultaneously considers multiple modalities, such as hand gestures, facial expressions, and hand orientation, from an Indian sign video. The Inception V3 architecture handles the features associated with different signer modalities, resulting in the generation of a feature map, which is processed by a two-layered (long short-term memory) (LSTM) architecture. This sequence helps in sentence-by-sentence recognition and in the translation of sign language into text and audio. The model was tested with continuous Indian Sign Language (ISL) sentences taken from the INCLUDE dataset. The experimental results show that the LiST framework achieved a high translation accuracy of 91.2% and a prediction accuracy of 95.9% while maintaining a low word-level translation error compared to other existing models. 2023 by the authors.

Source

Information (Switzerland), Vol-14, No. 2

Date

2023-01-01

Publisher

MDPI

Subject

artificial intelligence; humancomputer interface; Inception V3; INCLUDE dataset; Indian Sign Language; LSTM; natural language processing

Coverage

K A., Department of Computer Science, Christ (Deemed to be University), Karnataka, Bengaluru, 560029, India; P P., Department of Computer Science, Christ (Deemed to be University), Karnataka, Bengaluru, 560029, India; Poonia R.C., Department of Computer Science, Christ (Deemed to be University), Karnataka, Bengaluru, 560029, India

Rights

All Open Access; Gold Open Access

Relation

ISSN: 20782489

Format

Online

Language

English

Type

Article

Identifier

https://doi.org/10.3390/info14020079

https://www.scopus.com/inward/record.uri?eid=2-s2.0-85148686683&doi=10.3390%2finfo14020079&partnerID=40&md5=7cf8f827e002f7927eb6e24309a53e07