Conceptual question: Early loading of the model defeats the purpose of deepspeed!
From my understanding, one of the uses ofdeepspeedis to allocate the model across different GPUs, meaning avoid to load the model on a single GPU RAM + CPU RAM and distribute it instead.Lets take a case where a single machine cannot load the model (including GPU and CPU RAMs) but 2 or more can load the model in combination:Since,deepspeedis integrated into the pipelineafterthe model is loaded. i.e. after runningAutoModelForCausalLM.from_pretrainedleading to an OOM error, doesnt it defeat the purpose of usingdeepspeedin this case?Is there an alternative way to load the model in such cases (except downgrading precision)?