

Using torch.compile
PyTorch 2.0 introduced a new compile function that doesn't require any modification to existing PyTorch code but can 
optimize your code by adding a single line of code: model = torch.compile(model).
If using [Trainer], you only need to pass the torch_compile option in the [TrainingArguments]: 
python
training_args = TrainingArguments(torch_compile=True, **default_args)
torch.compile uses Python's frame evaluation API to automatically create a graph from existing PyTorch programs. After 
capturing the graph, different backends can be deployed to lower the graph to an optimized engine. 
You can find more details and benchmarks in PyTorch documentation.
torch.compile has a growing list of backends, which can be found in by calling torchdynamo.list_backends(), each of which with its optional dependencies.
Choose which backend to use by specifying it via torch_compile_backend in the [TrainingArguments].  Some of the most commonly used backends are:
Debugging backends:
* dynamo.optimize("eager") - Uses PyTorch to run the extracted GraphModule. This is quite useful in debugging TorchDynamo issues.
* dynamo.optimize("aot_eager") - Uses AotAutograd with no compiler, i.e, just using PyTorch eager for the AotAutograd's extracted forward and backward graphs. This is useful for debugging, and unlikely to give speedups.
Training & inference backends:
* dynamo.optimize("inductor") - Uses TorchInductor backend with AotAutograd and cudagraphs by leveraging codegened Triton kernels  Read more
* dynamo.optimize("nvfuser") -  nvFuser with TorchScript. Read more
* dynamo.optimize("aot_nvfuser") -  nvFuser with AotAutograd. Read more
* dynamo.optimize("aot_cudagraphs") - cudagraphs with AotAutograd. Read more
Inference-only backends:
* dynamo.optimize("ofi") -  Uses Torchscript optimize_for_inference.  Read more
* dynamo.optimize("fx2trt") -  Uses NVIDIA TensorRT for inference optimizations.  Read more
* dynamo.optimize("onnxrt") -  Uses ONNXRT for inference on CPU/GPU.  Read more
* dynamo.optimize("ipex") -  Uses IPEX for inference on CPU.  Read more
For an example of using torch.compile with 🤗 Transformers, check out this blog post on fine-tuning a BERT model for Text Classification using the newest PyTorch 2.0 features
Using 🤗 PEFT
Parameter-Efficient Fine Tuning (PEFT) methods freeze the pretrained model parameters during fine-tuning and add a small number of trainable parameters (the adapters) on top of it.
As a result the memory associated to the optimizer states and gradients are greatly reduced.
For example with a vanilla AdamW, the memory requirement for the optimizer state would be:
* fp32 copy of parameters: 4 bytes/param
* Momentum: 4 bytes/param
* Variance: 4 bytes/param
Suppose a model with 7B parameters and 200 millions parameters injected with Low Rank Adapters.
The memory requirement for the optimizer state of the plain model would be 12 * 7 = 84 GB (assuming 7B trainable parameters).
Adding Lora increases slightly the memory associated to the model weights and substantially decreases memory requirement for the optimizer state to 12 * 0.2 = 2.4GB.
Read more about PEFT and its detailed usage in the PEFT documentation or PEFT repository.
Using 🤗 Accelerate
With 🤗 Accelerate you can use the above methods while gaining full 
control over the training loop and can essentially write the loop in pure PyTorch with some minor modifications. 
Suppose you have combined the methods in the [TrainingArguments] like so:
py
training_args = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    fp16=True,
    **default_args,
)
The full example training loop with 🤗 Accelerate is only a handful of lines of code long: