融合音频文本的多模态音乐情感联合学习识别OA

Aiming at the problem of multi-modal emotion recognition in music,a joint learning framework considering the characteristics of multi-modal data was proposed.The multi-modal emotion recognition was taken as the main task,and the multi-modal emotion category recognition was taken as the auxiliary task.Through the emotion category recognition task assisting emotion recognition task,the performance of music emotion and emotion category recognition was improved together.Firstly,the audio and text information in the main task were encoded respectively through the private layer network,and the internal emotional characteristics of single audio and text modal information were learned.Then,through the shared network layer,the emotion information in the main task and the emotion classification information in the auxiliary task were learned,and then combined with the single mode independent features in the main task.The single mode information on the main task was obtained.Finally,multi-modal feature information was captured by self-attention mechanism to obtain the final emotion and emotion category recognition.Experiments showed that the proposed method can improve the performance of emotion recognition through the framework of joint learning,and the performance of emotion category recognition can also be improved to some extent.