Incrementally finetuning a HF model in SageMaker
If I have an already pretrained model, and want to train it on some additional data, how do I go about that in SageMaker?In this case I’ve trained a model loaded with AutoModelForSequenceClassification using “distilbert-base-uncased”.
2022-05-03T06:19:15Z
You can provide your trained model as model_id instead of the base model this would further train it. But you need to make sure that you have the same labels etc.
2022-05-03T08:24:00Z
MaximusDecimusMeridi:In this case I’ve trained a model loaded with AutoModelForSequenceClassification using “distilbert-base-uncased”.Hello!Consider using huggingface’s Trainer and TrainingArguments. Please check this page:Trainer.If you have any questions on how to do it, let me know and I will forward some other links or help you do it.
2022-05-03T15:39:35Z
petarulev:If you have any questions on how to do it, let me know and I will forward some other links or help you do it.Thank you. Yes if you can please share any other links with additional context/backgrouynd that would be great
2022-05-03T20:45:10Z
philschmid:You can provide your trained model as model_idCan I do this without using the hub? Would I just unzip the model into a local directory and load this and train with data and use same labels?Thanks
2022-05-04T02:30:14Z
@MaximusDecimusMeridi- if you want to do this without using the model hub then, yes, downloading the model from S3, unzipping it into a local folder, and usingmodel = AutoModelForSequenceClassification.from_pretrained('path/to/model/folder')should work!CheersHeiko
2022-05-04T06:05:45Z
You can also provide the model in a similar fashion you do with your training in the estimator, e.g.huggingface_estimator.fit({"train":s3_train_uri,"model":s3_model_uri})which should download the s3_model_uri into /opt/ml/input/model or rather speaking stores the path in the env varSM_CHANNEL_MODEL