首页|期刊导航|计算机应用研究|基于统一对齐与多阶段融合机制的多模态情感分析模型

基于统一对齐与多阶段融合机制的多模态情感分析模型OA

Multimodal sentiment analysis model based on unified alignment and multi-stage fusion

中文摘要英文摘要

针对多模态情感分析中模态异构、贡献动态与语义抽象不足等问题,提出一种三阶段闭环融合模型MIFA,路径包含"统一对齐-动态融合调控-高阶语义抽象".方法上,首先以统一语义对齐实现异构模态在共享空间的一致表达;继而通过上下文门控与通道调制联合估计模态/通道权重;最终以分层残差语义增强实现高阶抽象与判别强化.在CMU-MOSI与CMU-MOSEI数据集上的实验表明,二分类Acc2与F1分别达到86.43%/86.03%和 86.42%/85.81%,七分类 Acc7 为 45.04%/50.41%,回归任务中 MAE 为 0.689/0.532,总体优于主流模型.验证了该方法能够稳定对齐并自适应调控信息流,提升情感分类与强度回归性能,具备在复杂跨模态场景中的应用潜力.

To address the modality heterogeneity,time-varying contributions,and limited semantic abstraction in multimodal sentiment analysis,this paper proposed a three-stage closed-loop fusion model(MIFA).Specifically,it followed an align-ment-regulation-abstraction fusion path that comprised unified semantic alignment,dynamic fusion regulation,and hierarchical semantic abstraction.Firstly,this model used unified semantic alignment to express heterogeneous modalities in a shared space.Next,it used contextual gating and channel modulation to estimate modality/channel weights and refine representa-tions.Finally,it used a hierarchical residual abstraction to enhance discriminative features,forming an end-to-end closed loop.Experiments on CMU-MOSI and CMU-MOSEI show that MIFA achieves Acc2 of 86.43%/86.03%and F,of 86.42%/85.81%,Acc7 reaches 45.04%/50.41%,and MAE is 0.689/0.532,outperforming mainstream baselines.These results indicate that MIFA stabilizes cross-modal alignment and adaptively regulates information flow,improves sentiment classification and intensity regression,and offers practical potential for complex multimodal scenarios.

冯广;刘馨婷;林忆宝;赵志文;肖俊鸿;周科栋;黄俊辉

广东工业大学自动化学院,广州 510006广东工业大学自动化学院,广州 510006广东工业大学计算机学院,广州 510006广东工业大学自动化学院,广州 510006广东工业大学自动化学院,广州 510006广东工业大学自动化学院,广州 510006广东工业大学计算机学院,广州 510006

信息技术与安全科学

多模态情感分析跨模态特征融合统一语义对齐动态融合调控分层残差机制跨模态鲁棒性

multimodal sentiment analysiscross-modal feature fusionunified semantic alignmentdynamic fusion regula-tionhierarchical residual mechanismcross-modal robustness

《计算机应用研究》 2026 (2)

342-352,11

国家自然科学基金重点资助项目(62237001)广东省哲学社会科学青年项目(GD23YJY08)

10.19734/j.issn.1001-3695.2025.06.0214

评论