基于超图学习与成对跨模态融合的多模态对话情绪识别OA

To address issues such as insufficient utilization of interaction information between modalities and multivariate dia-logue relations in current multimodal emotion recognition in conversation models,this paper proposed a multimodal emotion recognition in conversation model based on hypergraph learning and pairwise cross-modal fusion.In the hypergraph learning module of the model,it took discourse representations as nodes,and designed two different types of hyperedges containing multimodal and temporal information to form a hypergraph.It used hypergraph convolution to capture multivariate dialogue re-lations between speakers.Meanwhile,this paper proposed a dual-stream gated attention network to dynamically adjust node features and reduce information redundancy.In the pairwise cross-modal fusion module,it used each modality as a baseline feature.And based on the cross-modal attention mechanism,it was repeatedly reinforced with other modal features to excavate deep interaction information between pairwise modalities and enhance cross-modal feature representation.Experimental results show that on the IEMOCAP and CMU-MOSEI datasets,the accuracy and weighted average F1 score of the proposed model are better than those of multiple comparison models,fully verifying the effectiveness of the model.