text
Question: Quantization through GPTQ 
Question: Quantization through GPTQ 
new_line
new_line
text
Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 
Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 
new_line
new_line
text
quantization_config = GPTQConfig(
quantization_config = GPTQConfig(
new_line
text
     bits=4,
     bits=4,
new_line
text
     group_size=128,
     group_size=128,
new_line
text
     dataset="wikitext2",
     dataset="wikitext2",
new_line
text
     batch_size=16,
     batch_size=16,
new_line
text
     desc_act=False
     desc_act=False
new_line
new_line
text
)
)
new_line
new_line
text
1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
new_line
text
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)
new_line
new_line
text
I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 
I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 
new_line
new_line
text
 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.
 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.
new_line
new_line
link
https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51
https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51
text
new_line
new_line
text
Could you kindly suggest?  Appreciate your kind support. 
Could you kindly suggest?  Appreciate your kind support. 
new_line
text
Thanks
Thanks
new_line
new_line
Question: Quantization through GPTQ 

Hi Team, I’m trying to quantize a 13b model using the below configuration on A100. I tried the below options 

quantization_config = GPTQConfig(
     bits=4,
     group_size=128,
     dataset="wikitext2",
     batch_size=16,
     desc_act=False

)

1. Enforce batch_size = 16  or batch_size = 2 at the quant configurations
2. Set tokenizer.pad_token_id = tokenizer.eos_token_id (which is 2)

I observed that even if we explicitly enforce the batch size and set the pad_token_id  value other than None. It is not being considered 

 Can’t we set the batch_size and pad_token_id to some other value is this expected behavior with GPTQ . What is the reason behind this? Please suggest if there is any way to override the batch size config.

https://github.com/huggingface/optimum/blob/main/optimum/gptq/data.py#L51  

Could you kindly suggest?  Appreciate your kind support. 
Thanks


👀
victor
/avatars/a8ae31b23226e85b2f3d8e1c173da5f3.svg
KR
KashyapR
user