Infer_auto_device_map returns empty
Hi,Following the instructions inthis postto load the same opt 13b. I have access to 8 Nvidea A100 80GB machines.model = AutoConfig.from_pretrained("facebook/opt-13b")runs successfully and the model card is available to use. Example of the modelOPTForCausalLM(
  (model): OPTModel(
    (decoder): OPTDecoder(
      (embed_tokens): Embedding(50272, 5120, padding_idx=1)
      (embed_positions): OPTLearnedPositionalEmbedding(2050, 5120)
      (final_layer_norm): LayerNorm((5120,), eps=1e-05, elementwise_affine=True)
      (layers): ModuleList(
...
...However,device_map = infer_auto_device_map(model)returns{'': 0}despite 8 GPUs being available.The code is exactly the same as the blog postfrom accelerate import infer_auto_device_map, init_empty_weights
from transformers import AutoConfig, AutoModelForCausalLM

config = AutoConfig.from_pretrained("facebook/opt-13b")
with init_empty_weights():
    model = AutoModelForCausalLM.from_config(config)

device_map = infer_auto_device_map(model)Not sure why this is the case.Any suggestions/help appreciated. Thanks!
2022-10-11T19:06:17Z
Your model fits on the whole GPU, soinfer_auto_device_mapjust returns that ({'': 0"}means put everything on GPU 0). You can pass along amax_memoryargument if you want to limit how much memory to use on each GPU.
2023-03-15T17:43:07Z
I found that Auto classes sometimes cause this{'': 0}issue. In my case, I’m usingT5-largefor inference, and I foundinfer_auto_device_map()would return the device map properly if I load the model withT5ForConditionalGenerationclass but{'': 0}if loaded withAutoModelForSeq2SeqLM. However, Auto classes are not always like that. For example,infer_auto_device_map()works fine withgoogle/flan_ul2loaded withAutoModelForSeq2SeqLM