PEFT LORA for Text Classification?
I am working on a project for text classification. I started out with BERT and AutoModelForSequenceClassification and now i want to move up the food chain and try some larger models. to do that on a single A100 i was hoping to use PEFT, LORA and bitsandbytes or accelerate for example. Bit the examples i have found all use AutoModelForCausalLM.I tried to adapt one of the tutorials to use BloomForSequenceClassification but thePEFT tutorial suggested " Finally, we need to apply some post-processing on the 8-bit model to enable training, let’s freeze all our layers, and cast the layer-norm infloat32for stability. We also cast the output of the last layer infloat32for the same reasons." and had codemodel.lm_head = CastOutputToFloat(model.lm_head)Which is fine for AutoModelForCausalLM but does not work for BloomForSequenceClassification because it does not have an lm_head layer.Which leads me to ask, can one use PEFT and LORA with a AutoModelForSequenceClassification?Alternatively, can one use a AutoModelForCausalLM for text classification?cheers
2023-12-08T06:12:24Z
I have exactly the same problem. If removing CastOutputToFloat line entirely, another error is triggered. Is there a solution for this?