Configuring Group Size for BitsAndBytesConfig
I am trying to find a way to configure the group size (a.k.a. block size) of NF4 quantization.According to my understanding, NF4 quantization can be done byBitsAndBytesConfig.Looking at this documentation page:huggingface.coQuantizationWe’re on a journey to advance and democratize artificial intelligence through open source and open science.Some configs likeAwqConfigandGPTQConfigdo havegroup_sizearg but `BitsAndBytesConfig| doesn’t.How can I know what is the group_size ofBitsAndBytesConfigand how can I modify it?