If you have access to a machine with multiple GPUs, these approaches are still valid, plus you can leverage additional methods outlined in the multi-GPU section.

When training large models, there are two aspects that should be considered at the same time: 

Data throughput/training time
Model performance