深度学习模型训练过程检查点访问性能优化方法OA

As LLMs become more widely used and their scale continues to expand,currently LLMs training faces issues such as high error rate and poor performance of checkpoint accessing.This paper reviews the strengths and weaknesses of existing methods for optimizing checkpoint accessing performance and introduces a novel method for optimizing checkpoint accessing performance.Based on the observation of data patterns in checkpoints,where the model weights change between adjacent checkpoints are minimal,making them suitable for delta compression.The proposed method implements delta compression across multiple interconnected training nodes and conducts experimental tests using real checkpoints generated during deep learning model training.The results demonstrate that,during the model training,delta compression has good compression effect for most checkpoints.Furthermore,the paper introduces dynamic intervals in delta compression to balance compression ratio and storage overhead,while also analyzing the characteristics of momentum datas.The analysis of existing methods and the optimization of checkpoint accessing performance offer insights for accelerating LLMs training.