Using Trainer class + 4/8 bit quantised model for prediction
Hi all,I would like to runtrainer.predictfor generation with e.g.mistralai/Mistral-7B-v0.1.The trainer is initialised liketrainer = Seq2SeqTrainer(model,
                      args=training_args
                      ...
                      )However, when I want to initialise the trainer with a model in 4 or 8 bit, I get following error:ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see:Load adapters with 🤗 PEFTfor more detailsWell, I don’t want to perform fine-tuning, just prediction.Any smart way to walk around this error?Thanks for your help!Cheers,Stephan