Number of parameters reduced after loading in 4bit
image897×429 53.5 KBWhy is there a decrease in no of parameters in 4bit?
2023-09-04T16:10:37Z
When we load in 4 bit, the linear layers are replaced with linear 4bit layers. These layers have half the number of parameters. But still I am also not clear how number of parameters become half.image1813×616 42 KBimage1276×392 23.5 KB
2023-11-09T17:53:29Z
In the source code, when picking 4-bit, the parameter count is divided by 2.I don’t know why, you can check the codegithub.comartidoro/qlora/blob/7f4e95a68dc076bea9b3a413d2b512eca6d004e5/qlora.py#L408C1-L423C6def print_trainable_parameters(args, model):"""Prints the number of trainable parameters in the model."""trainable_params = 0all_param = 0for _, param in model.named_parameters():all_param += param.numel()if param.requires_grad:trainable_params += param.numel()if args.bits == 4: trainable_params /= 2print(f"trainable params: {trainable_params} || "f"all params: {all_param} || "f"trainable: {100 * trainable_params / all_param}")
2023-11-29T07:58:15Z
I still don’t have  a clear answer for this and I would love to know, bumping for visibility.
2023-12-10T05:59:59Z
I too has the same doubt.@sgugger@sayakpaulsorry to tag you guys… I am thinking I could use your help here.
2024-06-03T06:21:18Z
Hi everyone. Have you figured this out? Is there a way to get access to the full weight with 4bit quantization?
2024-06-03T06:56:54Z
cc@ybelkada
2024-06-28T11:10:54Z
This is normal sincetorch.int4data dtype is not supported in PyTorch. Instead, we pack the 4bit data intotorch.int8tensor, hence the number of parameters is divided by 2 when we quantize in 4 bit !From@marcsun13