TrainingArguments changing the GPU by iteslf
I have 4 gpus available, out of which i have selected the second one usingif torch.cuda.is_available():
    torch.cuda.set_device(2)However when i compute theTrainingArgument()command :training_args = TrainingArguments('mydirectory')thentorch.cuda.current_device(). is giving 0Any idea why this is happening?
2021-07-07T12:19:13Z
Yes, the training argument set the GPU corresponding to itslocal_rankvalue (for distributed training), so you have to make sure to pass alonglocal_rank=2when you instantiate them.Though to execute a script on a given GPU, you would be better off setting the global env variableCUDA_VISIBLE_DEVICES.