大语言模型驱动的文献摘要筛选工具应用方法学构建和验证的诊断准确性研究OA

Background In secondary literature research,screening thousands to tens of thousands of articles through article-by-article review not only gives rise to inconsistency across multiple reviewers,but also entails substantial labor and time costs.Objective To establish an artificial intelligence(AI)-assisted screening workflow through a"human-AI-human collaborative evidence-based medicine practice"model and to provide a reliable technical solution for large-scale literature screening.Design Diagnostic accuracy study.Methods An AI-assisted literature abstract screening tool comprising 4 modules(PICO semantic parsing,evidence-chain semantic reasoning,decision mechanism,and prompt engineering with rule constraints)was developed based on high-performance large language models(LLMs)and built with GPT-5.With the negative predictive value(NPV)as the screening quality-control indicator,the tool was validated using a staged workflow from sampled to full-record literature screening.Main Outcome Measures The NPV of AI screening on both sampled and full-record literature sets were consistently 100%,and the number of records retained was generally in line with researchers'clinical expectations.Results Six disease-specific data sets were included,covering 4 registered guideline categories and 1 scoping review,with a total of 98 754 articles.These data sets spanned multiple disciplines,including neonatology,surgical oncology,rare diseases,and nursing,etc.Multiple approaches were attempted to develop pre-specified eligibility criteria,including criteria provided by guideline working groups and formulated on the basis of sampled articles by physicians,nurses,and graduate students.For 5 data sets,up to 3 rounds of AI screening based on the pre-specified eligibility criteria were conducted before proceeding to AI screening of the full-record literature sets;for 1 data set,AI screening was directly applied to the full-record literature set due to the small number of included studies.In full-record AI screening,2 data sets achieved an NPV of 100%,and the number of retained records met researchers'expectations.For the remaining 4 data sets,the pre-specified eligibility criteria were further refined,after which both sampled and full-record AI screening achieved NPV of 100%.Among these,the number of retained records of 3 data sets consistent with researchers'expectations,whereas 1 retained 7 201 studies,exceeding researchers'clinical expectations.After further refinement of the eligibility criteria,both sampled and full-record AI screening again achieved NPV of 100%,and the number of retained records was then in line with researchers'expectations.Taking the nursing data set(18 793 articles)as an example for time statistics,a screening day was defined as 480 minutes.The time shared by AI screening and by screening of 1 group of 2 researchers was 4 660 min;the AI-specific screening time was 2 776 min,including the shared undertaken time,the total amounts to 15.5 screening days;the researcher-specific screening time of 1 group of 2 researchers was 83 629 min,including the time shared jointly,the total is 88 289 minutes,which equals 184 screening days.AI-assisted screening achieved a screening-day saving rate of(184-15.5)/184×100%=91.6%,with an absolute saving of 168.5 screening days.Conclusion The large language models(LLMs)-driven abstract screening tool and a corresponding screening workflow developed by this study are adaptable to abstract screening for clinical guidelines and systematic reviews,and capable of substantially reducing the number of screening days required on the basis of a false-exclusion rate of zero.