Multi GPU fintuning BART
Hi,I am trying to fine-tune the BART model checkpoints on a large dataset(around 1M data points). Since the dataset is large, I want to utilize a multi-GPU setup but I see that because of thislineit’s not currently possible to train in a multi-gpu setting. Any work arounds for it?@sshleiferTagging you here since you’ve worked with BART and summarization in particular a lot on the repo.
2020-07-11T14:29:55Z
Which task are you finetuning on?For sequence to sequence tasks, like summarization,examples/seq2seq/finetune.pysupports multigpu for training only. There is a caveat: you have to run the final eval yourself on one GPU.For language modeling tasks,  multi-gpu is supported through theTrainerclass.
2020-07-11T15:23:51Z
Thanks for the reply. It’s a seq2seq task but wouldn’t the assert condition fail during training if I specify multiple GPUs in the training command? Do you mean I can comment out that part and then run the script?I am okay with the caveat.
2020-07-11T16:53:32Z
hi@dakshvar22you won’t need to comment that line, just setsortish_samplerargument toFalse, anyway, it’sFalseby default so you won’t need to change anything.