

Before diving deeper into the specifics of each technique, let's go over the rough decision process when training 
large models on a large infrastructure.
Scalability strategy
Begin by estimating how much vRAM is required to train your model. For models hosted on the 🤗 Hub, use our 
Model Memory Calculator, which gives you 
accurate calculations within a few percent margin.  
Parallelization strategy for a single Node / multi-GPU setup
When training a model on a single node with multiple GPUs, your choice of parallelization strategy can significantly 
impact performance. Here's a breakdown of your options:
Case 1: Your model fits onto a single GPU
If your model can comfortably fit onto a single GPU, you have two primary options: