Bitsandbytes `has_fp16_weights` issue
When loading a LLM using int8 quantization as specified in LLM.int8(), how arefp16_weightshandled?from transformers import (
    AutoModelForCausalLM,
    BitsAndBytesConfig,
)
from bitsandbytes.nn.modules import Linear8bitLt

quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_has_fp16_weight=True,
    llm_int8_threshold=0.0000000001,
)
print(quantization_config)

model = AutoModelForCausalLM.from_pretrained(
    "cache/bigscience/bloom-1b7",
    quantization_config=quantization_config,
)

for k, v in model.named_modules():
    if isinstance(v, Linear8bitLt):
        print("===")
        print(k)
        print(v.weight.has_fp16_weights)Examples of output are:transformer.h.23.self_attention.dense
False
===
transformer.h.23.mlp.dense_h_to_4h
False
===
transformer.h.23.mlp.dense_4h_to_h
FalsEven though I specifiedllm_int8_has_fp16_weightas True, allLinear8bitLtmodule weight’shas_fp16_weightsare printed as False.Does it internally holds fp16 weights (for outliers)? How can I access to these values.I’m so confused.
2024-08-15T18:15:27Z
I have the same doubt too - where are detected outliers stored? Because LLM.int8() performs operations on these outliers using fp16 compared to others in int8, it becomes important to store their positions/values. Moreover, when I check the model weight dtype, all of them are int8. I don’t understand what’s going on here?Capture1304×794 35.4 KBAlso, do note that the config parameterllm_int8_has_fp16_weighthas a different use case. It’s used for finetuning the model whereallmultiplications are carried out in fp16, although they’re stored in int8.