Dual-Teacher Multimodal Fusion for Depression Detection

Existing machine learning approaches for analyzing behavioral data—including electronic health records, wearable devices, and social media—remain constrained by unimodal paradigms. Language models often ignore paralinguistic biomarkers such as speech prosody, while audio-based methods overlook semantic context, leaving critical cross-modal interactions unexploited.

Here, we introduce a Teacher-Student Architecture-based Multimodal Fusion Network (TSA-MFN) that fundamentally advances depression classification. Dual specialized teacher models extract high-confidence text semantics and audio biomarkers, which are integrated by a student network through multi-head attention mechanisms. Fusion weights are dynamically computed via learnable similarity matrices, and a hybrid loss function balances knowledge transfer with classification optimization, enabling phased teacher-student synergy during training.