Load Custom Model
I tried look into this similar issue:Custom GPT2 Model won't load after trainingbut still…class CustomClass(PreTrainedModel):
    def __init__(self, config, num_labels):
        super().__init__(config, num_labels)
        self.distilbert = DistilBertModel.from_pretrained('distilbert-base-uncased')
        self.pre_classifier = torch.nn.Linear(768, 768)
        self.dropout = torch.nn.Dropout(0.1)
        self.classifier = torch.nn.Linear(768, num_labels)

    def forward(self, input_ids, attention_mask):
        distilbert_output = self.distilbert(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = distilbert_output[0]
        pooled_output = hidden_state[:, 0]
        pooled_output = self.pre_classifier(pooled_output)
        pooled_output = torch.nn.Tanh()(pooled_output) 
        pooled_output = self.dropout(pooled_output)  
        output = self.classifier(pooled_output)  
        return outputI was able to fine tune with a linear classifier for a classification job.config = PretrainedConfig(name_or_path='own-model', num_labels=100, output_hidden_states=True)

model = CustomClass(config, 100)I can also save it withmodel.save_pretrained(PATH)But when I try to load it withnew_model=PreTrainedModel.from_pretrained('./PATH/')i got'NoneType' object has no attribute 'from_pretrained'which is really strangeAlternatively, with a config file, this doesnt throw error though the model weights wont loadnew_config=PretrainedConfig.from_pretrained('./PATH/') new_model=PreTrainedModel.from_pretrained('./PATH/', config=new_config)please help. I am running out of idea. Oh and I am at v4.6.0Thank you!
2021-11-21T14:13:43Z
You should useCustomClass.from_pretrained.PreTrainedModel.from_pretrainedwon’t work directly.
2021-11-21T15:16:41Z
daboliang:CustomClass(config, 100)CustomClass.from_pretraineddidnt work for me, and still getting the same error. Not sure whyfrom_pretrainedisn’t inherited as a class method.But this “seems” to work. Strange it didnt needfrom_pretrainedbut just instantiate a model. I read it somewhere loading with config wont load weights, so I suspect it is still NOT done.cfg = PretrainedConfig.from_pretrained('./PATH/')
model = CustomClass(cfg, num_labels=100)
2021-11-21T15:31:03Z
Oh you need to add some class variables to your custom model, specifically:config_class = CustomConfigand have your config be an instance of aCustomConfig(it might work withconfig_class = PretrainedConfigbut I’m not 100% sure).
2021-11-21T16:40:42Z
class CustomClass(PreTrainedModel):

    class_config = PretrainedConfig   # class attribute like this here?

    def __init__(self, config, num_labels):
        super().__init__(config, num_labels)
        self.distilbert = DistilBertModel.from_pretrained('distilbert-base-uncased')
        self.pre_classifier = torch.nn.Linear(768, 768)
        self.dropout = torch.nn.Dropout(0.1)
        self.classifier = torch.nn.Linear(768, num_labels)

    def forward(self, input_ids, attention_mask):
        distilbert_output = self.distilbert(input_ids=input_ids, attention_mask=attention_mask)
        hidden_state = distilbert_output[0]
        pooled_output = hidden_state[:, 0]
        pooled_output = self.pre_classifier(pooled_output)
        pooled_output = torch.nn.Tanh()(pooled_output) 
        pooled_output = self.dropout(pooled_output)  
        output = self.classifier(pooled_output)  
        return outputI added it like above but still no luck to load it withnew_config=PretrainedConfig.from_pretrained('./PATH/') 

new_model=CusomClass.from_pretrained('./PATH/', config=new_config)Any notebook that I can look at for reference. This is a very common pattern I suppose.
2021-11-21T16:56:42Z
Looks like this is the way to load it withfrom_pretrainednew_config=PretrainedConfig.from_pretrained('./PATH/') 
new_model=CustomClass.from_pretrained(pretrained_model_name_or_path='./PATH/', config=new_config, num_labels=100)I have to verify the weights are also loaded successfully.
2021-11-22T17:00:47Z
By doing above, it is still not picking the weights properly. Can you share any reference where examples of inheritingPreTrainedModelis used to create custom model? And how to re-load it withfrom_pretrained?
2022-11-19T02:28:22Z
@daboliangdid you fix it? I am running into the same problem.
2022-11-21T18:56:59Z
@LearnToGroware still struggling? It’s working for me. I can save my my custom model with my custom head and are able to load it afterwards with the correct weights.Important is that you use keyword arguments. So you need to write:new_model=CustomClass.from_pretrained(pretrained_model_name_or_path='./PATH/', config=new_config)instead of only:new_model=CustomClass.from_pretrained('./PATH/', new_config)And I use a custom config which I assume will be necessary for all custom models.https://huggingface.co/docs/transformers/custom_modelsHope that helps!