融合跨视角特征和注意力机制的医学影像报告生成方法OA

Objective To explore a cross-perspective strategy for aligning medical images and medical reports to optimize the quality of medical image reports automatically generated by deep learning models.Methods A generative model combining cross-view features and attention mechanism was designed.Firstly,different angles were taken by the medical image and the pathological description features of the report were encoded based on the pre-trained model.Then,multiple attention mechanisms were utilized to complete the fusion calculation of features.Finally,the joint features were decoded into case reports using the decoder.Results It could be known from multiple rounds of tests on the authoritative and publicly available chest X-Ray dataset IU X-ray that the average performance of this model in the evaluation metrics of BLEU-1,BLEU-2,BLEU-3,BLEU-4,METEOR,ROUGE_L and P_MEAN was higher than that of the previously proposed methods by 8.34%,14.20%,10.90%,6.14%,1.70%and 5.50%respectively,and the comprehensive performance was improved by 7.79%.Conclusion This model performs well in terms of the accuracy and fluency of report generation,indicating that this fusion strategy can better capture the potential connection between images and reports,and improve the performance of the model in generating reports.