

Maximizing the throughput (samples/second) leads to lower training cost. This is generally achieved by utilizing the GPU 
as much as possible and thus filling GPU memory to its limit. If the desired batch size exceeds the limits of the GPU memory, 
the memory optimization techniques, such as gradient accumulation, can help.
However, if the preferred batch size fits into memory, there's no reason to apply memory-optimizing techniques because they can 
slow down the training. Just because one can use a large batch size, does not necessarily mean they should. As part of 
hyperparameter tuning, you should determine which batch size yields the best results and then optimize resources accordingly.
The methods and tools covered in this guide can be classified based on the effect they have on the training process:
| Method/tool                                                | Improves training speed | Optimizes memory utilization |
|:-----------------------------------------------------------|:------------------------|:-----------------------------|
| Batch size choice                    | Yes                     | Yes                          |
| Gradient accumulation            | No                      | Yes                          |
| Gradient checkpointing          | No                      | Yes                          |
| Mixed precision training      | Yes                     | (No)                         |
| Optimizer choice                      | Yes                     | Yes                          |
| Data preloading                        | Yes                     | No                           |
| DeepSpeed Zero                          | No                      | Yes                          |
| torch.compile                       | Yes                     | No                           |
| Parameter-Efficient Fine Tuning (PEFT)            | No                      | Yes                          |