Memory utilization at vanilla training
Let's use the [Trainer] and train the model without using any GPU performance optimization techniques and a batch size of 4:

from transformers import TrainingArguments, Trainer, logging
logging.set_verbosity_error()
training_args = TrainingArguments(per_device_train_batch_size=4, **default_args)
trainer = Trainer(model=model, args=training_args, train_dataset=ds)
result = trainer.train()
print_summary(result)