双向特征调制与文本引导渐进融合的细粒度指代表达分割方法OA

Current referring expression segmentation methods had key limitations.They showed spatial re-dundancy in visual features.Language descriptions had referential ambiguity.Lexical-level semantics were often ignored.Fusion modules failed to fully exploit text information.This paper proposed fine-grained referring expression segmentation via bidirectional feature modulation and text-guided progres-sive fusion(BFMPF).It used a pre-trained BEIT3 as an image-text encoder.It designed a bidirectional feature modulation module(BFMM).BFMM used global features of one modality to modulate local fea-tures of the other.This decoupled strategy suppressed noise before cross-modal interaction.Specifically,text-guided visual modulation filtered visual spatial redundancy.Vision-guided text modulation reduced the weights of irrelevant words.Then a text-guided progressive fusion module(TPFM)was introduced.TPFM embedded convolutions to compensate for the transformer's local modeling.It re-injected text se-mantics at multiple abstraction levels.It used multi-layer cross-modal attention to progressively fuse text features with high-level image features.A multilayer perceptron finally produced the segmentation mask.Experiments are conducted on three classic datasets.On RefCOCO,the method achieves 77.12％,79.46％and 73.45％on val,testA and testB.On RefCOCO+,it reaches 69.72％,72.86％and 62.42％on val,testA and testB.On RefCOCOg,it reaches 69.12％on val-u and 69.58％on test-u.The proposed BFMPF has strong generalization.It enables fine-grained cross-modal alignment.It per-forms robust referring expression segmentation.