追踪提示结合视觉大模型微调的猪只实例分割方法OA

Aiming to address the challenges of lack of coordination between tracking and segmentation tasks and reliance on costly manual annotation in current mainstream pig perception algorithms,a novel end-to-end framework was proposed.Specifically,the proposed method ingeniously leveraged the bounding boxes output by the multi-object tracking algorithm as dynamic spatial prompts for the segment anything model(SAM).This integration facilitated the generation of temporally coherent and identity-consistent individual mask sequences across video frames,bridging the gap between localization and pixel-level segmentation.To adapt the vision foundation model to the specific agricultural domain,the low-rank adaptation(LoRA)was combined with a custom-designed pig multi-scale feature adapter(PF Adapter).The model required only a minimal number of annotated samples to achieve robust feature extraction,ultimately surpassing the performance of traditional fully supervised learning algorithms and effectively overcoming the bottleneck of expensive data annotation.Comprehensive experimental results demonstrated the superiority of the proposed framework.Evaluated across two distinct datasets,the fine-tuned SAM achieved impressive performance metrics,with pixel accuracy(PA)of 91.24%,an intersection over union(IoU)of 85.06%,and a dice similarity coefficient of 86.18%.Remarkably,this significant performance enhancement was achieved with merely a 3%increase in number of parameters compared with the baseline SAM architecture.Furthermore,the pig body posture and movement trajectory information output by the improved model can be used for pig evaluation and sow farrowing warning,providing technical support for smart farming.