Can we use Gradient Checkpointing and Gradient Accumulation at Once?
Hi I’m trying to train large batch size for my model,So can I use Gradient Checkpointing and Gradient Accumulation at once?I’m not sure that gradient would safely added when checkpointing is doneP.S : would it be okay to use multi-GPU + Gradient Checkpointing + Gradient Accumulation at Once?
2021-09-14T00:41:37Z
Yes, those two techniques can be used together, and also with distributed training as well.