How to run trainer.py with megatron_lm_plugin
src/transformers/trainer.py, as I understand it, supports deepspeed but not megatron-lm. Is it right?When I tried to make it support megatron-lm, I encountered some problems：When and where to load megatron-lm ckpt?A. function_inner_training_loopB. prepare_model inaccelerate/utils/megatron_lm.pyC. other good wayOnly the loss of last_pp_rank is valid in Megatron-LM, while the Trainer takes the mean of ranks. When and where to deal with loss of megatron?A. function _nested_gather intrainer.pyB.GitHub - NVIDIA/Megatron-LM: Ongoing research training transformer models at scaleC. other good waysTrainer calls accelerate.prepare for model and dataloader respectively, which will initialize megatron-lm repeatedly. How to avoid this problemA. Modify initialization in Megatron-LM/megatron/global_vars.pyB. Modify function initialize inaccelerate/utils/megatron_lm.pyC. other good waysIn order to support the given dataset with different seq-length for each batch, when and where to padding it to meet requirements of megatron. (All batches have same seq-length)A. AccelerateB. Megatron-LMC. other good ways