What should I do if I want to use model from DeepSpeed
I am trying the language model pre-trained by using run_mlm.py. I want to add the mixture of expert (MoE) integrated with the original Bert-base model. Specifically, I reuse theMoELayerimplemented by DeepSpeed and add it to theBertForMaskedLM. From the document of DeepSpeed, I find the training of DeepSpeed requires to call some functions like this:model_engine, optimizer, _, _ = deepspeed.initialize(args=cmd_args,
                                                     model=net,
                                                     model_parameters=net.parameters())However, I just want to reuse the MoE model implemented by DeepSpeed and maintain the training behaviors of Huggingface. Currently, I ignore calling this function and directly pass the language model (with DeepSpeed model) into theTrainer(). Although it runs successfully, my question is that does this incurs some potential dangers?
2021-09-24T13:52:52Z
Hi ezio98.I can’t answer your question, but I’m a bit confused.   From what I have read about the MoE layer, the point of it is to facilitate the use (Mixture) of many different models (Experts) concurrently, but you say you are using the MoE layer on top of a single BertForMaskedLM model.What are you hoping the MoE layer will do for you?   Does it have some other advantages?
2021-09-25T00:37:39Z
Hi, rgwatwormhill. Sorry For I didn’t mention it clearly. Actually, I combine the MoE with the FFN module, following the design of Switch Transformer. This paper claimed this design can accelerate the training efficiency. Even though I haven’t observed in my experiments…
2022-09-07T06:16:21Z
Hi,I also plan to extend the huggingface transformer with deepspeed MoE. Do you have any successful experience in doing so? Can the models run and train successfully by simply adding the MoE layer to the FFN layer?
2022-09-08T05:59:51Z
Not yet. But at this stage those libraries are quite different. You can try this now.
2024-04-06T11:29:10Z
Hey, were you able to succeed in doing so?