text
The Universal Checkpointing paper is out! 
The Universal Checkpointing paper is out! 
link
https://arxiv.org/abs/2406.18820
https://arxiv.org/abs/2406.18820
new_line
new_line
text
If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.
new_line
new_line
text
Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.
new_line
new_line
text
The blog post is here: 
The blog post is here: 
link
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
The Universal Checkpointing paper is out! https://arxiv.org/abs/2406.18820

If you remember the Bigscience BLOOM-176B training, Tunji Ruwase and I co-invented this technology for Megatron-Deepspeed in order to enable to quickly scale up and down node topology while continuing training.

Since then the DeepSpeed team continued improving on that and it has now been fully integrated into Deepspeed.

The blog post is here: https://github.com/microsoft/DeepSpeed/blob/master/blogs/deepspeed-ucp/README.md
👀
alielfilali01 John6666