基于细粒度特征增强的多模态视觉问答研究OA

Existing multimodal Visual Question Answering(VQA)models often overlook the fine-grained interac-tion between local salient regions in images and local basic words in texts,thus the semantic relevance between ima-ges and texts needs to be improved.Here,we propose a novel multimodal VQA framework based on fine-grained fea-ture enhancement.First,a fine-grained feature extraction approach is added to both the visual and textual modalities to capture more detailed semantic features from images and questions.Then,to utilize cross-modal alignment cues,an alignment-guided self-attention module is proposed to align the correspondence between fine-grained features and global semantic features within each single modality(visual or text),and fuse unimodal information at different lev-els in a unified way.Finally,experiments are conducted on VQA v2.0 and VQA-CP v2 datasets.The results show that the proposed method outperforms existing models across multiple evaluation metrics.