图引句增强的科技文献图像语义理解研究OACHSSCD

[Purpose/Significance]Figures are the central medium for conveying complex scientific knowledge in sci-entific literature.Current mainstream vision-language models rely on figure captions as the primary supervision signal for training visual representations.However,Captions are essentially concise identifiers of image content,they rarely capture the causal logic that figures carry within scientific arguments.Confined to captions alone,models remain stuck at shallow visual matching,unable to reach the knowledge core of scientific reasoning.This semantic gap is precisely the problem this paper sets out to solve.To address this problem,this paper constructs MSLD,a multimodal scientific literature dataset,and proposes CE-SciVL,an image citing sentence-augmented scientific vision-language framework.By taking image citing sentences from the main text as the core supervision signal,the framework aims to strengthen the model's capacity for perceiving and reasoning about complex scientific image-text content.Experimental results show that the proposed method offers a viable new path for cross-modal semantic understanding in scientific literature.[Method/Process]Conventional PDF-based image-text extraction schemes suffer from inherent problems such as content misalignment and information omission.To solve this problem,this paper bypassed PDF parsing and instead built a complete data extraction and cleaning pipeline directly on LaTeX source files from arXiv papers,and on this basis constructed the MSLD dataset.Exploiting the structured markup of LaTeX sources,the pipeline achieved full-precision alignment between figures and their corres-ponding texts,guaranteeing data accuracy and reliability at the source.The CE-SciVL framework adopted a dual-stream archi-tecture consisting of SciBERT and Vision Transformer,which encoded textual semantics and visual features separately.On this basis,this paper designed a structured multi-signal layered input mechanism to decouple the semantic features of cap-tions,image citing sentences and their contextual paragraphs,guiding the model toward differentiated cross-modal associa-tions.During training,this paper adopted a single language modeling loss.Masked prediction tasks drove the model to extract key visual semantics and filter out irrelevant noise.The framework achieved stable semantic learning on complex sci-entific image-text pairs.[Result/Conclusion]Based on extensive experiments and analyses,this paper draws the following conclusions.First,image citing sentences carry richer logical information than captions:models trained on MSLD move beyond shallow image-text matching and are able to mine the internal logical rules connecting figures and texts in scientific literature,narrowing the semantic gap between visual features and logical reasoning.Second,in complex scientific image-text understanding tasks,explicit alignment losses exert negative interference on model optimization and hinder the accu-rate capture of cross-modal semantic correlations.The implicit alignment mechanism formed by the language modeling loss sidesteps gradient conflicts in multi-task learning,guides the model toward deeper image-text associations and improves its cross-modal semantic learning capacity.Third,the Semantic Signal-to-Noise Ratio(SSNR)of textual semantics is a critical factor governing model performance:redundant contextual information disturbs the model's ability to localize and learn core image-text content.Although the proposed framework already shows the ability to extract useful information from noisy texts,its capacity for information screening and noise reduction under very long textual contexts still leaves room for improvement.It is also a direction that warrants further investigation.