大模型辅助的强化学习奖励设计方法研究综述OA

Reward design stands as a core challenge in reinforcement learning.Traditional methods,which rely on experts' manual design,are plagued by several limitations:strong subjectivity,difficulty in expressing complex objectives,high susceptibility to reward hacking,and the sparse rewards problem.Leveraging their robust capabilities in semantic under-standing,task decomposition,and code generation,large models have enabled end-to-end automated mapping from"human intentions"to"executable reward functions".This significantly lowers the barrier to reward design and overcomes the bot-tlenecks of traditional methods,which rely heavily on expert knowledge and suffer from ambiguities in goal expression.This paper systematically reviews the research progress of LLM-assisted reward design in reinforcement learning.It ana-lyzes the limitations of traditional reward design methods.It proposes a classification framework based on the roles of LLMs in reward design,namely reward function initializer,"human-in-the-loop"iterator,and"human-out-of-the-loop"iterator;it also elaborates on the representative works,underlying mechanisms,advantages,and disadvantages of each cat-egory of methods.It expounds on the fundamental framework of reinforcement learning and the challenges of reward design,analyzing the intrinsic mechanisms and value of large model assistance.It reviews the limitations of traditional reward design methods,constructs a"Reward Initializer-Reward Iterator"classification framework,and details the repre-sentative work,mechanisms,and applicable scenarios of various methods.Furthermore,it summarizes the key challenges faced by current research at the"Model-Methodology-System"level in terms of model reliability,interaction efficiency,and engineering safety,and proposes corresponding technical pathways.