Transformer-Driven Multimodal for Human-Object Detection and Recognition for Intelligent Robotic SurveillanceOA
Human object detection and recognition is essential for elderly monitoring and assisted living however,models relying solely on pose or scene context often struggle in cluttered or visually ambiguous settings.To address this,we present SCENET-3D,a transformer-drivenmultimodal framework that unifies human-centric skeleton features with scene-object semantics for intelligent robotic vision through a three-stage pipeline.In the first stage,scene analysis,rich geometric and texture descriptors are extracted from RGB frames,including surface-normal histograms,angles between neighboring normals,Zernike moments,directional standard deviation,and Gabor-filter responses.In the second stage,scene-object analysis,non-human objects are segmented and represented using local feature descriptors and complementary surface-normal information.In the third stage,human-pose estimation,silhouettes are processed through an enhanced MoveNet to obtain 2D anatomical keypoints,which are fused with depth information and converted into RGB-based point clouds to construct pseudo-3D skeletons.Features from all three stages are fused and fed in a transformer encoder with multi-head attention to resolve visually similar activities.Experiments on UCLA(95.8%),ETRI-Activity3D(89.4%),andCAD-120(91.2%)demonstrate that combining pseudo-3D skeletonswith rich scene-object fusion significantly improves generalizable activity recognition,enabling safer elderly care,natural human–robot interaction,and robust context-aware robotic perception in real-world environments.
Aman Aman Ullah;Yanfeng Wu;Shaheryar Najam;Nouf Abdullah Almujally;Ahmad Jalal;Hui Liu
Guodian Nanjing Automation Co.,Ltd.,Nanjing,210003,China Department of Biomedical Engineering,Riphah International University,I-14,Islamabad,44000,PakistanGuodian Nanjing Automation Co.,Ltd.,Nanjing,210003,ChinaDepartment of Electrical Engineering,Bahria University,H-11,Islamabad,44000,PakistanDepartment of Information Systems,College of Computer and Information Sciences,Princess Nourah bint Abdulrahman University,Riyadh,11671,Saudi ArabiaDepartment of Computer Science,Air University,E-9,Islamabad,44000,Pakistan Department of Computer Science and Engineering,College of Informatics,Korea University,Seoul,02841,Republic of KoreaGuodian Nanjing Automation Co.,Ltd.,Nanjing,210003,China Jiangsu Key Laboratory of Intelligent Medical Image Computing,School of Artificial Intelligence(School of Future Technology),Nanjing University of Information Science and Technology,Nanjing,210003,China Cognitive Systems Lab,University of Bremen,Bremen,28359,Germany
信息技术与安全科学
Human object detectionelderly careRGB-based pose estimationscene context analysisobject recognition Gabor featurespoint cloud reconstruction
《Computers, Materials & Continua》 2026 (4)
P.1364-1383,20
funded by Princess Nourah bint Abdulrahman University Researchers Supporting Project number(PNURSP2025R410),Princess Nourah bint Abdulrahman University,Riyadh,Saudi Arabia.
评论