基于多模态大语言模型的Web自动化智能体设计OA

Traditional Web automation methods rely on scripted operation strategies,which suffer from poor robustness and high maintenance costs,making them difficult to adapt to the dynamic and complexity of modern Web systems.Multi-modal large language models offer advantages such as visual understanding,intent recognition and complex reasoning,providing a workable technical path for building Web agent with autonomous perception and execution capabilities.Therefore,this paper proposes a Web automation method based on multimodal large language models and designs a Web agent to automate the processes of interface understanding,instruction reasoning,and autonomous interaction.The main contributions are threefold.By introducing multimodal semantic fusion and instruction-wise context awareness mechanisms,the Web agent's interface understanding and operation decision accuracy are enhanced.Based on this,this paper designs a prototype system of Web-oriented multimodal agent to achieve an end-to-end automated interaction process from natural language to operation execution.Finally,this paper constructs an experimental dataset to verify the feasibility and effectiveness of the method,and empirically demonstrates its adaptability and application potential under constrained model conditions.Experimental results show that the method outperforms baseline methods in terms of task success rate.In addition,multimodal semantic fusion and the instruction-wise context awareness mechanisms effectively improve the task success rate and the element localization accuracy in complex interfaces.The findings provide practical methods and experimental basis for the integrated application of Web automation technology and multimodal large language models.