从中医皮肤病病历结构化提取角度评估五个大语言模型的效能OA

Objective To evaluate the performance of five large language models(LLMs)in the structured extraction of Traditional Chinese Medicine(TCM)dermatology medical records,clarifying their capabilities in extracting information from TCM clinical texts.Methods A total of 240 electronic medical records from a TCM dermatology specialty outpatient clinic were retrospectively collected.A structured template was designed,containing fields for basic information,four-diagnosis information,diagnoses,and integrated TCM-Western treatment plans.Two TCM medical students and two TCM dermatology specialists performed manual annotation separately.Under a unified prompt,five general-purpose LLMs were tasked with performing structured extraction from TCM dermatology medical records.A 0～3 point scale was used to evaluate extraction quality,and the time required per record was recorded.First,within-group consistency was compared between the medical student group and the physician group.If no statistically significant difference was found within these two groups(P>0.05),one annotator was then randomly selected from each group.These selected annotators were subsequently used for between-group comparative tests with the LLMs,with a P-value of<0.05 considered statistically significant.Results In basic information extraction,LLMs overall performed comparably to specialists and significantly outperformed medical students(P<0.05).For TCM diagnostic or therapeutic information and overall record extraction quality(P<0.05),LLMs generally performed better than students but were inferior to specialists(P<0.05)only individual models approached specialist-level performance after prompt optimization(P>0.05).The annotation efficiency of LLMs was significantly higher than that of both students and specialists.Conclusion In the task of structured extraction of TCM dermatology records,general-purpose LLMs have demonstrated the potential to replace junior human annotators and substantially improve efficiency.However,they still struggle to substitute experienced TCM dermatologists in understanding and expressing professional TCM diagnostic and therapeutic information.They can serve as auxiliary tools in the stages of TCM clinical data governance and research preparation.