Custom hardware for training
The hardware you use to run model training and inference can have a big effect on performance. For a deep dive into GPUs make sure to check out Tim Dettmer's excellent blog post.
Let's have a look at some practical advice for GPU setups.
GPU
When you train bigger models you have essentially three options:

bigger GPUs
more GPUs
more CPU and NVMe (offloaded to by DeepSpeed-Infinity)