How to finetune BLOOM for classification?
I’m trying to finetune BLOOM-350m for a simple classification task. It seems right now this is impossible? Here is the error I’m getting:BlockquoteFile “/usr/local/google/home/windmaple/miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py”, line 448, in from_pretrainedf"Unrecognized configuration class {config.class} for this kind of AutoModel: {cls.name}.\n"ValueError: Unrecognized configuration class <class ‘transformers.models.bloom.configuration_bloom.BloomConfig’> for this kind of AutoModel: TFAutoModelForSequenceClassification.Model type should be one of AlbertConfig, BertConfig, CamembertConfig, ConvBertConfig, CTRLConfig, DebertaConfig, DebertaV2Config, DistilBertConfig, ElectraConfig, FlaubertConfig, FunnelConfig, GPT2Config, GPTJConfig, LayoutLMConfig, LongformerConfig, MobileBertConfig, MPNetConfig, OpenAIGPTConfig, RemBertConfig, RobertaConfig, RoFormerConfig, TapasConfig, TransfoXLConfig, XLMConfig, XLMRobertaConfig, XLNetConfig.Given that BLOOM is so new, I’m curious if this is being worked on?
2022-08-06T01:04:18Z
I am also trying to fine-tune Bloom. The below works for me:tokenizer = AutoTokenizer.from_pretrained(“bigscience/bloom-350m”)model = AutoModel.from_pretrained(“bigscience/bloom-350m”, num_labels=1)However, I get the below error in the forward function of my model when trying to use the transformer:transformer_out = self.transformer(ids, mask)667         past_key_values_length = 0668         if past_key_values[0] is not None:→ 669             past_key_values_length = past_key_values[0][0].shape[1]670             current_sequence_length += past_key_values_length671IndexError: tuple index out of rangeI am unable to figure out if I am doing something wrong, but the code works for other transformers like Deberta and Roberta-Large.If you are able to get yours to work please post again.
2022-08-06T09:17:36Z
Did you update the version to the latest? I can run inference just fine.