Loading finetuned model to generate text
This is a basic question. Suppose I finetune a pretrained GPT2 model on my particular corpus. How do I go about loading that new model and generating some text from that new model?
2020-08-28T15:31:08Z
Greattutorialongeneratemethod which lets you do greedy decoding, sampling, beam search. It has all the details you are looking for
2020-08-29T20:51:27Z
@valhallaYes that tutorial is fantastic. Thank you for linking it.My finetuned model inherits fromGPT2LMHeadModel. I wanted to check if the best way to load my finetuned model was viafrom_pretrained()and if my model will be able to generate text from the new corpus it eas finetuned on.
2020-08-30T11:34:54Z
If your finetuned model is just tuned (and not extended with extra layers) then save_pretrained and from_pretrained should work, and may be the easiest.If you have altered your model (with extra layers) then I think you need to use torch.save and torch.load, and redefine your model configuration before loading, egtorch.save(model.state_dict(),‘filepath’)model = ATBertClass()                      # redefine using your custom model definitionmodel.load_state_dict(torch.load(‘filepath’))
2020-09-18T07:17:51Z
@rgwatwormhill,  I just fine tuned the model and did not alter it.If the name of that model class wasMyGPT2Modeland I saved it usingTrainer, to load it would simply be the lineMyGPT2Model.from_pretrained('/path/to/model/')?And since it inherits fromGPT2LMHeadModelI can call thegeneratemethod to generate text like:import MyGPT2Model
from transformers import GPT2Tokenizer

prompt = 'Huggingface Transformers is fantastic!'
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
encoded_results = tokenizer(prompt, return_tensors='pt')
tok_ids = encoded_results['input_ids']

my_model = MyGPT2Model.from_pretrained('/path/to/model/')
gen_ids = my_model.generate(tok_ids)
decoded_str = tokenizer.decode(gen_ids)Is this correct?
2020-09-18T10:04:28Z
Hi aclifton314,Short answer: I’ve no idea - what happens if you try that?I’ve not tried using Trainer or GPT2 (I’ve been using Bert with native pytorch commands), so I could easily be wrong here.   Please ignore anything where you know better.In the Trainer documentation, it looks like from_pretrained is the correct command to use.Did you set output_dir in your Trainer training arguments?  Have you checked that your trained model exists in your /path/to/model/ directory?I don’t think you can import your custom model class (I think you will need to include your code to define it).If you are not altering the shape of the model, do you need a custom model class?I think you might need to import GPT2LMHeadModel, whether you use it as-is or by inheriting.I think you might need to use tokenizer.tokenize() or tokenizer.encode_plus() or something like that, rather than just tokenizer()
2020-09-21T18:30:51Z
Short answer: I’ve no idea - what happens if you try that?It looks like it’s working!Did you set output_dir in your Trainer training arguments? Have you checked that your trained model exists in your /path/to/model/ directory?Yes I did. The model does exist in the output directory set in the training arguments.
2020-10-17T04:45:48Z
thank you for your answer, but i have a question for youI validate the model as I train it, and save the model with the highest scores on the validation set usingtorch.save(model.state_dict(), output_model_file). As shown in the figure belowThen I trained again and loaded the previously saved model instead of training from scratch, but it didn’t work well, which made me feel like it wasn’t saved or loaded successfully?model.load_state_dict(torch.load(output_model_file))My model code ishere
2020-10-19T09:03:45Z
Hi@mathor,what learning-rate did you use for the second lot of training?Were you using similar training data or very different training data?Did you save the optimizer state_dict as well as the model state_dict?This post might help:Checkpoint vs model weightIf you load your saved (fine-tuned) model, and do a validation check before you start any more training, what kind of validation accuracy do you get?
2023-07-25T13:18:58Z
Hey aclifton314, I am a beginner I was trying to fine-tune model to generate text i will be the great help if you will be able to provide your notebook to me
2023-07-25T16:45:00Z
@Pranavagrlwelcome to the HF Forums!I’m happy to help out in any way I can. If you wouldn’t mind providing some more details, it would help me out to walk you through finetuning.Are you wanting to use a model to generate text over a particular corpus that you have?Which model would you prefer to use?Have you any previous experience (either with coding or learning) with machine learning and/or NLP?
2023-07-26T07:29:37Z
Hello@aclitfton314Yeah, I have a basic understanding of that.Are you wanting to use a model to generate text over a particular corpus that you have? ==> NoWhich model would you prefer to use? ==> Any Model Using This Architecture: LlamaForCausalLM
2023-08-07T17:27:11Z
Hi@Pranavagrl,Do you mind describing in more detail what you’re interested in doing?