基于REALM与LoRA融合的审计专业问答大模型研究OA

To address the issues of general large language models in audit applications—such as deviations in understanding professional terminology,insufficient dynamic adaptation to regulatory standards,and high fine-tuning costs—this paper proposes a construction scheme for an audit professional question-answering large model integrating Retrieval-Augmented Language Model Pre-training(REALM)and Low-Rank Adaptation(LoRA)fine-tuning.By integrating industry data including audit reports,laws and regulations,and professional standards,a dedicated dataset containing 10,000 high-quality audit question-answering dialogues is constructed.The REALM retrieval-augmented pre-training mechanism enables modular storage and accurate retrieval of audit knowledge during the pre-training phase.Qwen-7B-Chat serves as the base model,with LoRA technology employed for parameter fine-tuning;performance differences among full LoRA,QLoRA-8bit,and QLoRA-4bit quantization schemes are also evaluated.Experimental results show that after REALM pre-training and LoRA fine-tuning,Qwen-7B-Chat achieves 26.34,48.79,29.87,and 42.51 on the BLEU-4,ROUGE-1,ROUGE-2,and ROUGE-L metrics,respectively,improving by 9.5％,6.9％,8.9％,and 8.3％over the single LoRA fine-tuned model.The audit qualitative accuracy reaches 94.2％,the precision of regulatory citation increases to 93.7％,and the training computing power demand is reduced by more than 78％.This model demonstrates good accuracy and professionalism in audit professional knowledge question-answering,providing a lightweight and cost-effective technical approach for assisting auditors in decision-making,professional training,intelligent compliance review,and other scenarios.