FP16 doesn't reduce Trainer Training time
Hi,I’m using this SageMaker HF samplenotebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHubadjusted with train_batch_size = 128, tested on both 1 p3.16xlarge and 1 p4d.24xlarge. For each instance I’m doing a job with fp16=True and a job without the flag. GPU usage is a erratic (sawtooth oscillating between 50 and 750%)The impact of fp16=True is only a 1% training time reduction, on each instance. Is it because:Not specifying fp16 in the trainer already uses fp16? (seems to befalseby default though)There is a lot of CPU work & I/O in that demo that will not leverage float16?Transformers don’t benefit from fp16 training?
2021-07-20T13:29:36Z
Hey@OlivierCR,can you please share all of yourhyperparametersyou used?tran_batch_size = 128seems pretty high to me to work.Are you usingdistributedtraining?The training time reduction should be way higher more around 20-40%.When using the example have you adjustedtrain.pyto accept fp16 as hyperparameter or have you defined it directly in the script?
2021-07-21T07:46:16Z
batch was 32 on 1 V100 so it’s ok to scale 4x on 4*V100 right? my understanding is thattran_batch_sizeis the cluster-level batch, not the per-GPU batch right? here is my config. it ran fine both on p3.16x and p4d.24xl, and GPU mem is not 800%from sagemaker.huggingface import HuggingFace

# hyperparameters, which are passed into the training job
hyperparameters={'epochs': 3,
                 'train_batch_size': 128,
                 'model_name':'distilbert-base-uncased',
                 'fp16': True}

huggingface_estimator = HuggingFace(entry_point='train.py',
                            source_dir='./scripts',
                            instance_type='ml.p4d.24xlarge',
                            instance_count=1,
                            role=role,
                            transformers_version='4.6',
                            pytorch_version='1.7',
                            py_version='py36',
                            hyperparameters = hyperparameters)train_batch_size = 256also works fine, it is as fast as 128 on p3.16x and 7% faster than 128 on p4d.24xlarge (both still fp16)
2021-07-21T11:21:50Z
No thetrain_batch_sizeis the batch_size per device. Seenotebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHubor in the documentation hereTrainer — transformers 4.7.0 documentationThis speakingtrain_batch_size = 256would have resulted in 256*8 = 2048 which is not possible.Have you modifiedtrain.pyotherwisefp16:Truewill not be used. Since it is not parsed herenotebooks/train.py at 6b8367544a374daf187854c6ba2274640fa0d7b4 · huggingface/notebooks · GitHub
2021-07-21T13:21:36Z
Well running train_batch_size = 256 on 1*p3.16xlarge for me worked
2021-07-22T08:50:50Z
Could you share the exact code/notebook you have used? I would like to reproduce it.
2021-07-27T13:58:25Z
github.comnotebooks/sagemaker/01_getting_started_pytorch at master · huggingface/notebooksmaster/sagemaker/01_getting_started_pytorchNotebooks using the Hugging Face libraries 🤗. Contribute to huggingface/notebooks development by creating an account on GitHub.
2021-07-28T06:47:04Z
So you didn’t change anything?If sofp16couldn’t work because it is not parsed intrain.py. Can you share the logs of your script? There you should be able to see which batch size was used by theTrainer
2021-07-28T08:30:02Z
@philschmidI didn’t change anything apart from the config above in the estimator (batch size, fp16, instance type). Did you try and see that batch 256 works fine on p3.16x?
2021-07-28T11:26:36Z
Hey,found the issue. thetrain.pyexpected the hyperparamter to betrain-batch-sizeand we always passed intrain_batch_sizeso it used the default of32and not the values passed in. I fixed all of the examples and tested256and it failed.If you want to testfp16you would need to change thetrain.py
2023-06-29T09:13:59Z
Hi,So how would I change the code to enable fp16 computation in this case?