Hub and Versioning, how to save a new version?
I’m currently using the model.push_to_hub() from a notebook and it saves the model correctly.I have also seen an example on how to read using a version (which is just a branch?)My problem is - how do I save a model to a new version using the APIs?Help would be greatly appreciated, thanks
2022-03-29T16:54:20Z
Hello revuze, welcome!Since models are stored on the Hub using a git repository, every time you usepush_to_hubit will automatically create a new commit on the model repo, which is effectively a new version. In this model,revuze/my-awesome-model-2-v1, for example, you can see that in thecommitsthere are two commits that say “add model”, probably from runningpush_to_hubtwice.When you runpush_to_hubyou can choose to set a commit message (Models), among other options. And when you want to download your model, you specify a specific version of the model to download with the particular Git commit hash (e.g.5f22418800ced4f10a5f0ef1ba67748d50d3661bfor the first version of the model in your repo above), branch, or Git tag. You can see more about that here:How to download files from the Hub
2022-03-29T20:11:37Z
As an addition to@NimaBoscarinofantastic answer, all theAutoclasses, such asAutoModel, have arevisionparameter you can use to specify the Git commit hash.Here is an examplemodel = AutoModel.from_pretrained(
    "julien-c/EsperBERTo-small", revision="v2.0.1"  # tag name, or branch name, or commit hash
)fromShare a model
2022-03-30T09:08:19Z
I understand now, thank you.So as a follow up question - how do I set up a tag whilst pushing from the API? (or have the commit hash of the current push) ?I’m tryin to implement an automated thing, without needing to look at the UI for the older versions.Thanks
2022-03-30T10:17:49Z
There are many approaches for this. Withhuggingface_hublibrary, you can easily add a tag as followsrepo = Repository(
            "hub,
            clone_from=f"{USER}/{REPO_NAME}",
            revision="main",
        )

        repo.add_tag("v4.5.0", message="This is an annotated tag", remote="origin")with the con-side that it requires to clone the repo locally.If you just want the last commit hash, you can usemodel_info. I don’t think we have any support to automatically retrieve older commits though.from huggingface_hub import model_info
model_info("gpt2")
>>ModelInfo: {
>>	  modelId: gpt2
>>	sha: 6c0e6080953db56375760c0471a8c5f2929baf11
>>	...
>>}Any ideas or feedback are more than appreciated both inGitHub - huggingface/huggingface_hub: All the open source things related to the Hugging Face Hub.andGitHub - huggingface/transformers: 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.cc@adrin@lysandre
2023-08-31T04:40:54Z
HI, Ill revive this one:I understand the setup, but I have now updated a model (changed both tokenizer and model files, and called push_to_hub on both model and tokenizer. However, after running this, the tokenizer is still old and some model checkpoints are still in the repo. I changed from a model with 3 checkpoints files to 9, and since all of them have different names they just added the nine files. however, for the tokenizer it should have recognized that the tokenizer-files were completely changed?What is the correct procedure for this setup?