

TITLE
Saving LLAMA 13B checkpoint with FSDP finetuning results in disk full error

COMMENTS
8

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### System Info

transformers - installed from source
accelerate - installed from source
torch 2.0.1


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Finetune LLAMA 13B with accelerate launcher. Saving strategy "epoch". FSDP based training.

### Expected behavior

I would be able to save the checkpoint. Now getting disk full error. Note that the disk initially had space. My code was working with transformers-4.28, accelerate 0.18 and torch 1.13. This error started after I moved to accelerate based launcher and upgraded the packages.