Moonlight:基于数据处理器的大模型训练在网检查点结构OA

Large models have a vast number of parameters,and training them requires thousands of neural network accel-erators to work together for days or even months.Due to the frequent data synchronization required by parallel train-ing algorithms,an error in a single training node can lead to the loss of training progress and significant waste of computing resources.Existing training clusters typically use checkpoint to help recover from errors.However,cur-rent checkpoint strategies bring severe host central processing unit(CPU)usage and memory overhead,and the ef-ficiency of checkpoint operations is limited by the low bandwidth of remote storage.To address these issues,this paper proposes a new in-network checkpoint strategy called Moonlight,based on the data processing unit(DPU).Moonlight offloads the checkpoint operation workload from the host to the network devices,provides checkpoint management and control functions through hardware structure designs,and implements hierarchical checkpoint stor-age using network device storage,offering an efficient checkpoint structure for large model training.Experimental results show that:(1)Moonlight can effectively offload the checkpoint workload from hosts,with no host CPU us-age at the checkpoint data plane and negligible host memory overhead.(2)Moonlight can provide efficient check-point saving functions,with the backend storage bandwidth being 4.10 times that of existing commercial solutions and the sending efficiency of checkpoint data packets being 1.96 times that of the baseline solution.