中国大语言模型知识实体抽取能力评测OACHSSCD

[Purpose/Significance]The evaluation of Large Language Models(LLMs)for extracting knowledge entities from scientific texts is a growing area of international research.However,systematic and multi-dimensional assessments focusing on the capabilities of domestic LLMs within specialized academic fields remain relatively underexplored.This study aims to provide a comprehensive empirical evaluation of mainstream Chinese LLMs,using the Information Systems(IS)domain as a case study.The objective is to deliver a substantive and detailed benchmark to inform model selection and practical application in domain-specific text mining tasks.[Method/Process]To conduct this rigorous evaluation,the research first constructed a bilingual benchmark dataset tailored to the IS domain.This dataset comprises 250 Chinese research articles sourced from the Journal of Information System(spanning 2007-2025)and 865 English articles from MIS Quarterly(2008-2025).The research defined a taxonomy of six categories of fine-grained knowledge entities critical to IS research.A structured prompt engineering framework was then meticulously designed to guide eight prominent domestic LLMs—DeepSeek,GLM-4.6,Qwen3,Spark X1.5,Doubao-1.6,Hunyuan-T1,Kimi-K2,and ERNIE-X1—through the entity extraction task.It assessed model performance using a multi-dimensional analytical framework.This framework quantitatively analyzed key performance metrics(Macro Precision,Recall,and F1-score),conducted a fine-grained quali-tative error analysis(incorporating span comparison scores and entity type confusion matrices),and evaluated practical cost-effectiveness based on API pricing and average processing time per document.[Result/Conclusion]The evaluation reveals significant and noteworthy performance variation among the tested Chinese LLMs.DeepSeek consistently achieved the highest overall scores across both Chinese and English contexts,demonstrating superior extraction capability.Qwen3 presented a more balanced profile,offering competitive performance with favorable cost efficiency.All models exhibited strong competency in identifying well-structured,categorically clear entities such as research perspectives.Conversely,they faced consistent and pronounced difficulties in determining precise textual boundaries for fine-grained entities and showed a marked performance decline in cross-lingual settings.Error analysis further pinpointed systematic challenges in disambiguating semantically similar entity types.This study contributes a detailed,multi-dimensional benchmark that clarifies the current landscape of Chinese LLMs for specialized knowledge extraction.It affirms their utility as effective tools for preliminary knowledge entity recognition in scenarios with scarce annotated data.Simultaneously,it delineates persistent challenges—particularly in semantic precision,boundary detection,and cross-lingual generalization—that must inform future model development and application design.The proposed assessment framework offers an adaptable founda-tion for future comparative research across other academic domains and evolving model ecosystems.