A review of hybrid EEG-based multimodal human–computer interfaces using deep learning: applications, advances, and challenges

초록

Human–computer interaction (HCI) focuses on designing efficient and intuitive interactions between humans and computer systems. Recent advancements have utilized multimodal approaches, such as electroencephalography (EEG)-based systems combined with other biosignals, along with deep learning to enhance performance and reliability. However, no systematic review has consolidated findings on EEG-based multimodal HCI systems. This review examined 124 studies published from 2016 to 2024, retrieved from the Web of Science database, focusing on hybrid EEG-based multimodal HCI systems employing deep learning. The keywords used for evaluation were as follows: ‘Deep Learning’ AND ‘EEG’ AND (‘fNIRS’ OR ‘NIRS’ OR ‘MEG’ OR ‘fMRI’ OR ‘EOG’ OR ‘EMG’ OR ‘ECG’ OR ‘PPG’ OR ‘GSR’). Main topics explored are: (1) types of biosignals used with EEG, (2) neural network architectures, (3) fusion strategies, (4) system performance, and (5) target applications. Frequently paired signals, such as EOG, EMG, and fNIRS, effectively complement EEG by addressing its limitations. Convolutional neural networks are extensively employed for spatio-temporal-spectral feature extraction, with early and intermediate fusion strategies being the most commonly used. Applications, such as sleep stage classification, emotion recognition, and mental state decoding, have shown notable performance improvements. Despite these advancements, challenges remain, including the lack of real-time online systems, difficulties in signal synchronization, limited data availability, and insufficient explainable AI (XAI) methods to interpret signal interactions. Emerging solutions, such as portable systems, lightweight deep learning models, and data augmentation techniques, offer promising pathways to address these issues. This review highlights the potential of EEG-based multimodal HCI systems and emphasizes the need for advancements in real-time interaction, fusion algorithms, and XAI to enhance their adaptability, interpretability, and reliability. © Korean Society of Medical and Biological Engineering 2025.

키워드