Best practice for upgrading models?
Hi everybody. I’m looking for best practices for upgrading models which have been trained with Transformers. We have a model which is trained with Transformers 4.0.x, which breaks with the latest version (4.1.1). We had the same issue with models trained with Transformers 3 when Transformers 4 came out.Retraining models every time the Transformers package is updated is expensive and the new model might give different output. Pinning the version is of course possible, but we would prefer to keep the models up-to-date.Are there other options than fully retraining the model or pinning the version that we could consider? We save our models and tokenizers with thesave_pretrainedmethod.
2021-01-04T10:55:04Z
How does your model “breaks”? Can you give more details?
2021-01-04T10:55:32Z
Normally you should not need to retrain models indeed.
2021-01-04T11:47:17Z
When I try to load the model, which was trained with an older version of Transformers, now in my env with Transformers 4.1, I get the following error:'BertEmbeddings' object has no attribute 'position_embedding_type'I had similar issues when we upgraded from Transformers 3 to 4, attributes that were missing.
2021-01-04T13:17:15Z
Hi@RobinvdM, sorry you’re experiencing this issue, it should not happen. Do you mind sharing the code you used to reload the model, as well as the stacktrace?
2021-01-04T15:58:42Z
This is the code used to reload the model.if torch.cuda.is_available():
            device = torch.device("cuda")
        else:
            device = torch.device("cpu")
        model = torch.load(model_path, map_location=device)
        if torch.cuda.is_available():
            model.cuda()
        model.eval()
        input_ids = dataset[0].to(device)
        input_masks = dataset[1].to(device)
        with torch.no_grad():
            logits = model(
                input_ids,
                token_type_ids=None,
                attention_mask=input_masks,
                return_dict=False,
                labels=None
            )[0]The error occurs after the model() call. Stack trace:/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:1375: in forward
    return_dict=return_dict,
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:862: in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py:722: in _call_impl
    result = self.forward(*input, **kwargs)
/usr/local/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:202: in forward
    if self.position_embedding_type == "absolute":
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = BertEmbeddings(
  (word_embeddings): Embedding(30000, 768, padding_idx=3)
  (position_embeddings): Embedding(512, 768)... 768)
  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)
name = 'position_embedding_type'

    def __getattr__(self, name: str) -> Union[Tensor, 'Module']:
        if '_parameters' in self.__dict__:
            _parameters = self.__dict__['_parameters']
            if name in _parameters:
                return _parameters[name]
        if '_buffers' in self.__dict__:
            _buffers = self.__dict__['_buffers']
            if name in _buffers:
                return _buffers[name]
        if '_modules' in self.__dict__:
            modules = self.__dict__['_modules']
            if name in modules:
                return modules[name]
        raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
>           type(self).__name__, name))
E       torch.nn.modules.module.ModuleAttributeError: 'BertEmbeddings' object has no attribute 'position_embedding_type'Thank you for your help
2021-01-05T09:24:40Z
In your code I’m not seeing how your model is built, which is probably where the issue lies. Is it a model you created yourself, which does not inherit fromPretrainedModel, hence why you’re usingtorch.loadinstead offrom_pretrained?If so, could you show me your model creation so that I can try and replicate on my side? Thank you.
2021-01-06T11:52:47Z
Also@RobinvdMyou can consider uploading your model to theHuggingface.comodel hub if you want us to take a look.You can upload it privately if needed.
2021-01-06T12:37:39Z
Does that mean that you (as maintainer of the storage) have access to users’ private models? I guess so, but I am not sure if that is desired for some people who expectrealprivate storage for their models. (If this is disclaimed somewhere already when taking a subscription, that is perfect. It should be clear to subscribers what “private” means.)