A unified framework for multimodal emotion recognition across homogeneous and heterogeneous modalities with adaptive fusion

Document Type

Article

Publication Date

Spring 5-2-2026

Abstract

Amid the growing demand for emotionally intelligent systems, Multimodal Emotion Recognition (MER) has emerged as a critical frontier in affective computing. However, achieving reliable generalization across heterogeneous data sources and ensuring semantic alignment across diverse modalities remain unresolved challenges. So, this research presents a novel and unified framework for MER that unfolds in five coordinated stages: modality-specific cross-dataset pretraining, diffusion-based generative data augmentation, reinforcement learning-driven hyperparameter optimization, latent space alignment, and task-aware multimodal fusion with fine-tuning. Each modality-text, audio, video, and motion-is initially pretrained using large-scale, emotion-labeled corpora to extract domain-invariant affective features. A generative augmentation stage that uses diffusion models increases sample diversity and improves class balance. Hyperparameter scheduling is governed by a Proximal Policy Optimization (PPO) agent that dynamically adjusts learning parameters during both pretraining and fine-tuning phases. Latent space alignment is achieved through a combination of domain-adversarial objectives, statistical regularization (e.g., MMD, CCA), and prototypical contrastive learning. The fusion strategy integrates Cross-Attentional Modality Interaction (CAMI), Bidirectional Alignment Networks (BAN), Gaussian Mixture Interaction Modules (GMIM), and Neural Variational Mixture-of-Experts (NV-MoE) to support context-aware and uncertainty-resilient emotion inference. Empirical evaluations on MELD, IEMOCAP, and SAVEE demonstrate exceptional performance. Test accuracies reached 99.91 %, 99.87 %, and 99.52 % respectively, with minimal losses ( ≤  0.000056) and inference latencies between 0.02-0.07 ms. Post-alignment diagnostics across 100 runs revealed highly stable latent embeddings (Silhouette: 0.960-0.980, CKA: 0.970-0.990), confirming strong cross-modal coherence. Zero-shot testing on external unseen datasets (GoEmotions, CREMA-D, EmotiW, HUMAINE) yielded accuracies above 99.90 %, demonstrating robust generalization without fine-tuning. Even though the model is trained on batch data, the deployment through ONNX ensures adaptability for real-time emotion recognition in resource-constrained environments. These findings establish the proposed system as a highly performant and deployable solution for multimodal affect analysis

Link to Full Text

Share

COinS