跨模态特征一致性对齐下的多模态情感分析OA

Current multimodal data-based sentiment analysis methods have made significant progress in feature modeling and modal interaction,but there are still problems such as insufficient feature extraction and insufficient inter-modal semantic alignment,which can easily lead to the increase of redundant information and limited cross-modal interaction.This paper proposes a multimodal sentiment analysis method based on cross-modal feature consistency alignment.Firstly,based on the characteristics of each modal data,a three-branch feature extraction sub-network is designed to model the deep-level representation of textual,visual and auditory modalities respectively.Secondly,in order to mitigate the impact of modal inconsistency on the fusion effect,a cross-modal feature consistency alignment module is constructed,and an interme-diate layer supervisory mechanism is introduced to improve the alignment quality.Finally,the cross-modal Transformer fusion module is proposed to realize deep-level semantic interaction and emotion fusion modeling among multimodalities.Seven classification experiments are carried out on CMU-MOSI and CMU-MOSEI datasets,and the accuracies are improved by 3.7 and 6.4 percentage points,respectively,which verifies the effectiveness and robustness of the proposed model in multi-modal emotion recognition tasks.