Explicitly set number of training steps using Trainer
I’m usingTrainerto handle finetuning a GPT2 model. I see inTrainingArgumentsthere is amax_stepsthat overridesnum_train_epochs.For a batch size of 32, is settingmax_steps=1000000the equivalent of settingnum_train_epochs=31250?Also, what happens if I have a batch size of 6 but want to setmax_steps=1000000? DoesTrainerstop training after the nearest divisible whole number or does it change the batch size at the end?Thanks in advance!!
2020-09-14T21:00:14Z
The number of steps is the number of update steps. It’s not the number of training examples to be seen.
2020-09-14T21:03:51Z
Ok. Is it then the case that for a batch size of 32,  settingmax_steps=1000000is the same as  settingnum_train_epochs=31250?
2020-09-15T12:01:41Z
No (unless your dataset has 32 batches so is of length 32*32).num_train_epochs = max_steps / len(train_dataloader)in general.
2020-09-15T16:45:05Z
Apologies for my confusion. Is thenmax_stepsthe maximum number of times a batch makes a forward and backward pass through the network?
2020-09-16T01:06:13Z
Not exactly it is the number of update steps (so if you’re using gradient accumulation, it’s a bit different than just doing forward and backward).