

You can combine the above methods to get a cumulative effect. These techniques are available to you whether you are 
training your model with [Trainer] or writing a pure PyTorch loop, in which case you can configure these optimizations 
with 🤗 Accelerate.
If these methods do not result in sufficient gains, you can explore the following options: 
* Look into building your own custom Docker container with efficient softare prebuilds
* Consider a model that uses Mixture of Experts (MoE)
* Convert your model to BetterTransformer to leverage PyTorch native attention
Finally, if all of the above is still not enough, even after switching to a server-grade GPU like A100, consider moving 
to a multi-GPU setup. All these approaches are still valid in a multi-GPU setup, plus you can leverage additional parallelism 
techniques outlined in the multi-GPU section. 
Batch size choice
To achieve optimal performance, start by identifying the appropriate batch size. It is recommended to use batch sizes and 
input/output neuron counts that are of size 2^N. Often it's a multiple of 8, but it can be 
higher depending on the hardware being used and the model's dtype.
For reference, check out NVIDIA's recommendation for input/output neuron counts and 
batch size for 
fully connected layers (which are involved in GEMMs (General Matrix Multiplications)).
Tensor Core Requirements 
define the multiplier based on the dtype and the hardware. For instance, for fp16 data type a multiple of 8 is recommended, unless 
it's an A100 GPU, in which case use multiples of 64.
For parameters that are small, consider also Dimension Quantization Effects. 
This is where tiling happens and the right multiplier can have a significant speedup.
Gradient Accumulation
The gradient accumulation method aims to calculate gradients in smaller increments instead of computing them for the 
entire batch at once. This approach involves iteratively calculating gradients in smaller batches by performing forward 
and backward passes through the model and accumulating the gradients during the process. Once a sufficient number of 
gradients have been accumulated, the model's optimization step is executed. By employing gradient accumulation, it 
becomes possible to increase the effective batch size beyond the limitations imposed by the GPU's memory capacity. 
However, it is important to note that the additional forward and backward passes introduced by gradient accumulation can 
slow down the training process.
You can enable gradient accumulation by adding the gradient_accumulation_steps argument to  [TrainingArguments]: 
py
training_args = TrainingArguments(per_device_train_batch_size=1, gradient_accumulation_steps=4, **default_args)
In the above example, your effective batch size becomes 4. 
Alternatively, use 🤗 Accelerate to gain full control over the training loop. Find the 🤗 Accelerate example 
further down in this guide.
While it is advised to max out GPU usage as much as possible, a high number of gradient accumulation steps can 
result in a more pronounced training slowdown. Consider the following example. Let's say, the per_device_train_batch_size=4 
without gradient accumulation hits the GPU's limit. If you would like to train with batches of size 64, do not set the 
per_device_train_batch_size to 1 and gradient_accumulation_steps to 64. Instead, keep per_device_train_batch_size=4 
and set gradient_accumulation_steps=16. This results in the same effective batch size while making better use of 
the available GPU resources.
For additional information, please refer to batch size and gradient accumulation benchmarks for RTX-3090
and A100.
Gradient Checkpointing
Some large models may still face memory issues even when the batch size is set to 1 and gradient accumulation is used. 
This is because there are other components that also require memory storage.
Saving all activations from the forward pass in order to compute the gradients during the backward pass can result in 
significant memory overhead. The alternative approach of discarding the activations and recalculating them when needed 
during the backward pass, would introduce a considerable computational overhead and slow down the training process.
Gradient checkpointing offers a compromise between these two approaches and saves strategically selected activations 
throughout the computational graph so only a fraction of the activations need to be re-computed for the gradients. For 
an in-depth explanation of gradient checkpointing, refer to this great article.
To enable gradient checkpointing in the [Trainer], pass the corresponding a flag to [TrainingArguments]:
py
training_args = TrainingArguments(
    per_device_train_batch_size=1, gradient_accumulation_steps=4, gradient_checkpointing=True, **default_args
)
Alternatively, use 🤗 Accelerate - find the 🤗 Accelerate example further in this guide. 