Which weights does QLoRA train by default?
The closest answer I got to this question ishere. Anyone know if this means that LoRA does Q and V matrices, while QLoRA only does linear layers? Or does it mean that QLoRA also does Q and V on top of linear layers?
2024-04-01T08:33:02Z
Hi,Q and V are linear layers, so those are included when you train LoRa or QLoRa. QLoRa is the same as LoRa except that you also quantize the base model (turn it into 4 bits rather than 16 per parameter).