What arguments need to be changed when using deepeed in trainer?
I understand that we will need to change a regular call function likepython run_trainer.py ...todeepspeed --hostfile <hostfile> run_trainer.py ... --deepspeed <deepspeed_config_file>.Besides thedeepspeedargument, is there anything else I should change, for example,sharded_ddp,ddp_find_unused_parameters,skip_memory_metrics, etc.?
2021-07-02T08:11:23Z
You should find everything that you need over here:DeepSpeed Integration — transformers 4.7.0 documentation (huggingface.co)
2021-07-03T02:47:16Z
Thanks! I already looked at this page. It’s actually still unclear to me what the purpose is forsharded_ddp, given it seems to be related todeepspeed. Could you explain a bit more on the usage ofsharded_ddp? Thanks!