视觉—语言协同微调LLaVA的遥感图像目标检测方法OA

To address the challenge that remote sensing image object detection heavily depends on ex-tensive annotated data and that existing efficient fine-tuning methods for general parameters exhibit in-sufficient spatial perception when adapted to remote sensing tasks,a vision-language collaborative effi-cient fine-tuning method for multimodal large models is proposed.Firstly,the existing dataset is recon-structed to construct instruction data suitable for large language and vision assistant(LLaVA)fine-tun-ing.Secondly,combining the multi-cognitive visual adapter(Mona)and the quantized low-rank adapta-tion(QLoRA)method,the LLaVA model is efficiently fine-tuned on the instruction fine-tuning dataset.Finally,the target category and coordinate information are obtained in the inference stage.Experiments on the NWPU VHR-10 dataset show that compared with the pre-trained LLaVA model,the collabora-tive fine-tuning model has greatly improved in F1 score,accuracy,and mIoU.Its mAPs on both NWPU VHR-10 and DIOR datasets are comparable to those of traditional detection models.This method still performs well in low-annotation scenarios,providing a new technical approach for the efficient applica-tion of multimodal large models in remote sensing vision tasks.