用于知识视觉问答的问题增强知识检索网络OA

Knowledge-Based Visual Question Answering(KB-VQA)requires answering questions with external knowledge in addition to the content of images.Nowadays,many works transform everything into the textual space for retrieving knowledge by textual space paradigm retriever,but there are two major limitations in textual space paradigm retriever for KB-VQA:1)The query obtained via image-to-text transformation can be inaccurate and redundant due to the absence of the question;2)Rel-evance between queries and supporting knowledge are computed with their semantic similarity,which can be insufficient to ques-tion answering.To this end,this paper proposes a Question-augmented Knowledge Retrieval Network(QKRN)for knowledge-based visual question answering,which consists of Question-augmented Query Construction(QQC)and Reverse Inference-based Re-ranking Retriever(RIR)modules.More specifically,the QQC module utilizes the cross-attention mechanism to local-ize question-related visual regions and construct question-augmented queries.Furthermore,the RIR module re-ranks the re-trieved knowledge by computing the likelihood of question generation conditioned on the knowledge.Extensive experiments con-ducted on OK-VQA and FVQA datasets verify the outperformance of the proposed QKRN.