开放环境下无监督跨模态概念自动提取OA

As machine learning models become increasingly complex,the demand for interpretability in their decision-making processes continues to grow.Concept learning has become increasingly important in the field of machine learning as a means to enhance model transparency and comprehensibility.Consequently,the use of concepts to explain the reason-ing processes of black-box models,such as large language models,has also developed.The ability to accurately and auto-matically extract concepts is a crucial step in this interpretability process.To address the issues in existing concept extrac-tion methods,such as reliance on manual annotation,inconsistent granularity,and poor scalability,this paper proposes an automatic cross-modal visual concept extraction framework.This framework employs a multimodal large model-based inquiry method to achieve automatic extraction of target objects in images.The integration of the CLIP model and object detection models establishes a region-matching mechanism to enhance extraction accuracy.Additionally,a large language model based on text segmentation is utilized to extract attributes and relationships of target objects,ensuring consistency between concepts and data.Furthermore,ConceptNet is incorporated to enrich the semantic information of the extracted concepts,improving the flexibility and applicability of framework.Through the implementation of these technologies,this paper demonstrates the practical application of this framework in three different scenarios,showcasing its potential in im-proving the efficiency,accuracy,and scalability of concept extraction.