Enabling load_in_8bit makes inference much slower
I loaded the 7b llama on an A100 this way:quantization_config = transformers.BitsAndBytesConfig(load_in_8bit=False, llm_int8_threshold=0.0)
model = LlamaForCausalLM.from_pretrained(
   "decapoda-research/llama-7b-hf",
    device_map="auto",
    torch_dtype=torch.float16,
    quantization_config=quantization_config,
)Whenload_in_8bitisFalse, it generates 16.7 tokens/sec whereasload_in_8bit=Truegenerates only 6.7 tokens/sec. It seems I probably set up configurations incorrectly forload_in_8bit=True.My transformers version is4.29.0.dev0. Did I miss anything? Thanks.
2023-08-30T13:09:58Z
I am facing similar issue, did you find a solution?
2023-09-09T07:48:03Z
me too，and training is slower too
2024-02-13T07:05:40Z
same here, not sure if anyone found the root cause.