

The significance of this framework is that it takes resources like (1) GPU/TPU/CPU vs. (2) RAM/DRAM vs. (3) 
fast-intra-connect/slow-inter-connect and it automatically optimizes all these algorithmically deciding which 
parallelisation to use where.
One very important aspect is that FlexFlow is designed for optimizing DNN parallelizations for models with static and 
fixed workloads, since models with dynamic behavior may prefer different parallelization strategies across iterations.
So the promise is very attractive - it runs a 30min simulation on the cluster of choice and it comes up with the best 
strategy to utilise this specific environment. If you add/remove/replace any parts it'll run and re-optimize the plan 
for that. And then you can train. A different setup will have its own custom optimization.
🤗 Transformers status: Transformers models are FX-trace-able via transformers.utils.fx, 
which is a prerequisite for FlexFlow, however, changes are required on the FlexFlow side to make it work with Transformers models.
GPU selection
When training on multiple GPUs, you can specify the number of GPUs to use and in what order. This can be useful for instance when you have GPUs with different computing power and want to use the faster GPU first. The selection process works for both DistributedDataParallel and DataParallel to use only a subset of the available GPUs, and you don't need Accelerate or the DeepSpeed integration.
Number of GPUs
For example, if you have 4 GPUs and you only want to use the first 2: