How to Finetune Deberta Model on SQUAD dataset?
Hi All,I am trying to finetune DeBerta Model on SQUAD. If I try existing notebooks likethisIt uses Trainer and Fast Tokenizer.DeBerta doesn’t have support for Fast Tokenizer Yet. How can I finetune it on SQUAD?I am also willing to implement Fast Tokenizer for the Deberta model, Can anyone help me with resources so I can get started with that?Here is my notebook for trainingDeberta(I am facing issues)
2021-01-26T13:01:49Z
Hi@bhadresh-savani, as far as I can tell the problem seems to lie with yourfind_sublist_indicesfunction, not on the availability of a fast tokenizer.One simple thing to try: can you pass a slice of examples to yourconvert_to_featuresfunction, e.g.convert_to_features(train_dataset[:3])I’m not sure whether this will solve the problem, but perhaps yourfind_sublist_indicesis expected a list of lists which is what you’ll get from the slice.I also noticed that yourconvert_to_featuresfunction is quite different to theprepare_train_featuresin the tutorial - what happens if you try the latter with your tokenizer?If that doesn’t work, then you might be able to use the oldrun_qa.pyscript that doesn’t rely on fast tokenizers:transformers/examples/legacy/question-answering at master · huggingface/transformers · GitHubLewis
2021-01-27T10:20:15Z
Hi@lewtunThanks for your answerI tried to run old versionv3.5.1by keeping latest version ofmodeling_deberta.pyfile with few changes (i neededQuestionAnsweringModelOutputclass for SQUAD kind of training)I was getting below errorTraceback (most recent call last):
  File "run_squad.py", line 820, in <module>
    main()
  File "run_squad.py", line 734, in main
    model = AutoModelForQuestionAnswering.from_pretrained(
  File "/media/data2/anaconda/envs/transformers-hugginface/lib/python3.8/site-packages/transformers/modeling_auto.py", line 1330, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.configuration_deberta.DebertaConfig'> for this kind of AutoModel: AutoModelForQuestionAnswering.
Model type should be one of DistilBertConfig, AlbertConfig, CamembertConfig, BartConfig, LongformerConfig, XLMRobertaConfig, RobertaConfig, SqueezeBertConfig, BertConfig, XLNetConfig, FlaubertConfig, MobileBertConfig, XLMConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig.find_sublist_indicesi created by taking ref of thisnotebookwhich uses a fast tokenizer, I am trying to do the same without fast tokenizerFast tokenizer has method calledchar_to_tokeni am trying to implement the same on Python based tokenizer.Hi@valhalla,Can you tell me how can i use the samenotebookwithout fast tokenizer since it was created by you?