基于关键点感知和窗口-时序注意力Transformer的模糊手势重建方法OA

Reconstructing hand gestures from ambiguous images caused by rapid hand motion is of great significance.In order to avoid the ambiguity caused by blurred gesture images,this paper proposes an improved approach based on key point perception and window-temporal attention.First,the method utilizes residual networks and feature pyramids to extract key point features of the hand from dynamic fuzzy gesture images;then,with the proposed window-temporal attention Transformer,the spatial information within a single frame is captured by using the shifted window-based multi-head self-attention(SW-MSA),and a novel inter-frame temporal attention(FTA)mechanism is introduced to explicitly model the correlation of multi-frames across the time-steps to effectively fuse the spatial and temporal information so as to solve the ambiguity of the hand motion.In addition,considering that different hand joints have different motion sensitivities,a weighted strategy based on motion analysis of key points is introduced in the training to better constrain the reconstruction process.Results on the BlurHand dataset show that the proposed method can significantly improve the accuracy of re-constructing 3D hand sequences from a single blurred image compared to existing methods.