

Finally, pass the custom optimizer as an argument to the Trainer:
py
trainer = Trainer(model=model, args=training_args, train_dataset=ds, optimizers=(adam_bnb_optim, None))
Combined with other approaches (gradient accumulation, gradient checkpointing, and mixed precision training), 
you can expect to get about a 3x memory improvement and even slightly higher throughput as using Adafactor. 
multi_tensor
pytorch-nightly introduced torch.optim._multi_tensor which should significantly speed up the optimizers for situations 
with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner, take a look at this GitHub issue.
Data preloading
One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it 
can handle. By default, everything happens in the main process, and it might not be able to read the data from disk fast 
enough, and thus create a bottleneck, leading to GPU under-utilization. Configure the following arguments to reduce the bottleneck: