AutoModel resolution outside of HF ecosystem
Hey guys,In the current API, unless I am doing something wrong, I have to specify each remote file (config, PyTorch model) explicitly in order to load a custom model outside of HF repo ecosystem.For example, to load custom modelprivate_model_namefrom a hypothetical remote repoexample.comI need to do the following:config = AutoConfig.from_pretrained(“https://example.com/models/private_model_name/config.json”)model = AutoModel.from_pretrained(“https://example.com/models/private_model_name/pytorch_model.bin”, config=config)Why not add naming resolution capabilities that currently exist to all remote repos so we can do:model = AutoModel.from_pretrained(“https://example.com/models/private_model_name/”)Can we add the same name resolution assumption for non-HF model repos?
2021-01-31T21:54:02Z
I think the last command you type is supposed to work, as long as you passuse_auth_token=Trueto use your Hugging Face token (you need to be logged in viatransformers-cliwith an account having permission to access the private model though).cc@julien-c
2021-01-31T21:59:33Z
Sylvain, but in my case, I want to load/store the model to an internal S3 bucket (totally unrelated to the HF ecosystem).
2021-02-01T13:18:11Z
What I would do in that case, is to sync a portion or directory of your bucket locally (assuming different models are in ≠ “subfolders”) and then just load your model from disk withAutoModel.from_pretrained(path)