Does autogpt-q require float16?
Hi! I’ve been excited about the recent quantization integration of auto-gptq and appreciate the work folks did to make this happen. I was trying to play around a bit, and saw this linegithub.comhuggingface/transformers/blob/50573c648ae953dcc1b94d663651f07fb02268f4/src/transformers/modeling_utils.py#L2579-L2583logger.info(f"Overriding torch_dtype={torch_dtype} with `torch_dtype=torch.float16` due to ""requirements of `auto-gptq` to enable model quantization ")torch_dtype = torch.float16which seems to imply that auto gptq only supports float16 vs. bfloat16… I uncommented this line and things seem to be working fine — do folks have thoughts about this? TheBloke seems to sometimes run quantization with bfloat16 ([BUG]CUDA OUT OF MEMORY · Issue #179 · PanQiWei/AutoGPTQ · GitHub) so I am not sure why this line is here, but could be missing something (e.g., maybe some of the CUDA kernels are float16 only)?