Future trends in multimodal learning: from theory to practical applications

Title

Future trends in multimodal learning: from theory to practical applications

Creator

Dsouza, Salu; Panen, Jos

Description

The human ability to seamlessly integrate information from various sensory channelssight, sound, touchhas long inspired researchers in artificial intelligence. This ability to learn from and reason with multimodal datatext, speech, vision, and moreforms the core of multimodal learning. This abstract delves into the theoretical foundations of multimodal learning, explores its cutting-edge advancements, and critically examines the path toward practical applications in diverse fields. At its heart, multimodal learning seeks to exploit the inherent complementarity between different data modalities. Text, for instance, provides rich semantic meaning, while visual data offers valuable context. Speech captures the nuances of emotion and prosody often absent in text. By learning from these combined modalities, models can achieve a more comprehensive understanding of the world around them. Recent years have witnessed significant progress in multimodal learning architectures. Deep learning approaches, particularly convolutional neural networks and recurrent neural networks, have proven adept at capturing complex relationships within individual modalities. New architectures like multimodal transformers further bridge the gap by allowing models to learn joint representations across different modalities. These advancements pave the way for a paradigm shift in areas like computer vision, natural language processing, and robotics. In computer vision, multimodal learning allows models to not only recognize objects in images but also understand the context and actions depicted. By incorporating textual descriptions or speech narratives alongside visual data, models can achieve better scene understanding, image captioning, and action recognition. This has applications in autonomous vehicles, where understanding traffic signs, pedestrians, and road conditions is crucial, and in video surveillance systems, where interpreting visual cues alongside spoken dialogue improves anomaly detection. 2026 Elsevier Inc. All rights reserved.

Source

Multimodal Learning Using Heterogeneous Data;pp.247-263

Date

01-01-2025

Publisher

Elsevier

Subject

artificial intelligence; cognitive process; computer in society; Computer vision, robotics, explainability, fairness, deep learning; human- interaction; information management; information systems; machine learning

Coverage

Dsouza S., School of Law, Christ University, Karnataka, Bengaluru, India; Panen J., Department of Commercial Sciences, Business Management & Informatics, Vives University of Applied Sciences, Bruges, Belgium

Rights

Restricted Access; Hardcopy may be available in the library

Relation

ISBN: 978-044327528-9; 978-044327529-6;

Format

online

Language

English

Type

Book chapter

Identifier

https://doi.org/10.1016/B978-0-443-27528-9.00009-7

https://www.scopus.com/pages/publications/105032985842?origin=resultslist