How to accelerate.pepare() two different models based on different accelerate configs?
I want to fine-tune an LM using DeepSpeed with some ZeRO stage. However, the setup involves another model which evaluates the LM-in-training and only does inference. Preparing the latter without an associated optimizer throws an error about this ZeRO stage requiring an optimizer. However, for the inference model, I only want to parallelize it across devices without any fancy ZeRO partitioning at all. Is that possible?I was hoping for being able to pass a custom config dict toacelerate.prepare(), but there’s no such option. Any advice?
2022-11-21T16:21:10Z
Hello@paulbricman, given that you only want to do inference with second model (eval model), just don’t pass it to theaccelerator.prepare(). That way, eval model has parallel copies on each GPU with same params which can be used for inference without any issues. Let us know if this solves your query. Thank you.
2022-11-21T16:58:33Z
Thanks for the quick reply.As in, consistently assign the model (I’m using pipelines) to accelerator.device, and the accelerate wrapper will place each in the right place while the other is “prepared”?
2022-11-22T04:59:22Z
Yes, for eval model you would have to do the followingeval_model = eval_model.to(accelerator.device)