Error saving quantized model
I get the following error trying to save a quantized model. Can anyone help? Thanks.quantize modelquantized_model = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)quantized_model.save_pretrained(quantized_dir)AttributeError                            Traceback (most recent call last)in ()3     model, {torch.nn.Linear}, dtype=torch.qint84 )----> 5 quantized_model.save_pretrained(quantized_dir)1 frames/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in shard_checkpoint(state_dict, max_shard_size)291292     for key, weight in state_dict.items():→ 293         weight_size = weight.numel() * dtype_byte_size(weight.dtype)294295         # If this weight is going to tip up over the maximal size, we split.AttributeError: ‘torch.dtype’ object has no attribute ‘numel’
2022-06-30T21:59:45Z
I found the solution herePegasus Model Weights Compression/PruningModels@lewtunI chose dynamic quantization approach. And I dont think from_pretrained support loading quantised models. 
I’d be happy if you can take a look. Can we hop on a google meet call and you can help me out. 
Hopefully. 
Thanks
2022-11-24T09:59:56Z
How did this issue got resolved? Am still getting the same error.
2023-02-14T03:31:54Z
Im getting the same error. Im trying to save a quantized model using the following line torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.float16).When running quantized_model.save_pretrained(path_dir), I’m running into the error AttributeError: ‘torch.dtype’ object has no attribute ‘numel’.Anyone have a solution?
2023-02-16T18:10:00Z
I was actually able to resolve the error from the Pegasus Model Weights Compression/Pruning post above. Specifically, I was using Pytorch version 1.11.0 and HuggingFace version 4.20.1.