Missing `model_type` key in config.json of TinyBERT
Hello,I’m trying to use one of the TinyBERT models produced by HUAWEI (link) and it seems there is a field missing in theconfig.jsonfile:>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huawei-noah/TinyBERT_General_4L_312D")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lewtun/git/transformers/src/transformers/models/auto/tokenization_auto.py", line 345, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/lewtun/git/transformers/src/transformers/models/auto/configuration_auto.py", line 360, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in huawei-noah/TinyBERT_General_4L_312D. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, mt5, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, mpnet, bart, blenderbot, reformer, longformer, roberta, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag, tapasLooking at the config.json file (link) it seems like it should be an easy enough fix to add something like"model_type": "tinybert"so my question is how does one go about patching a fix in a community provided model? Do I raise an issue on theTransformers repo or somewhere else?
2020-12-16T23:50:06Z
Oh that field is missing indeed. From what I know though, it can’t be filled with “tinybert” as it needs to be a model type implemented in the library, so I think it should be “bert”. As for adding it, I think it needs to be done on our side (in the future we’ll have the ability to open PRs on model repos, which would help in that case!)Looking at that config, there are a few things that are a bit weird, so it may be that they have an internal class for those models not compatible with Transformers.
2020-12-17T14:33:12Z
Thanks for the info Sylvain!I agree that the config is a bit weird, so I’ve posted a question on the FastFormers repo (link) whose experiments I am trying to reproduce and who somehow found a way to make TinyBERT work withTransformers.Failing that, I’ll ask the TinyBERT authors directly and report back here
2021-03-17T08:21:29Z
I’m experiencing the same issue with the BlenderBot models.After executingconfig.save_pretrained(model_dir), themodel_type:'blenderbot'attribute is missing from theconfig.jsonfile
2021-03-17T10:00:09Z
@lewtun- Regarding TinyBERT, have you checked Albert joint model fromGitHub - legacyai/tf-transformers: State of the art faster Natural Language Processing in Tensorflow 2.0 ..Glue score on Albert base 14M and 6 layer seems to have 81, which is better than Tinybert, Mobilebert, distillbert, which has 60M parameter.