S3 uploading advice
@staswas about to write this to you but decided to send it here!I have done everything withpip install awsclibut that may require more privs.  (you probably still want awscli to runls)Each model should be its own subdir understas/with the full complement of files:something likeaws s3 ls s3://models.huggingface.co/bert/Helsinki-NLP/opus-mt-ru-en/2020-08-21 10:42:49       1148 README.md
2020-08-18 20:43:55       1133 config.json
2020-04-29 09:40:09  306991893 pytorch_model.bin
2020-05-27 11:23:35  563074700 rust_model.ot
2020-04-29 09:40:21    1080169 source.spm
2020-04-29 09:40:21     802781 target.spm
2020-08-18 20:43:55         42 tokenizer_config.json
2020-08-18 20:43:55    2601758 vocab.jsonIf you are missing files, you can sometimes get them withworking_model.save_pretrained('en_de') # will mkdir en_de
tokenizer.save_pretrained('en_de')That’s all that comes to mind at the moment, but lmk if you need anything!
2020-08-28T00:05:28Z
another thing that bothered me earlier thattransformers-clihandlesyou have to duplicate all tokenizer files across every model subdir. S3 cant do symlinks.
2020-08-28T02:13:51Z
Thank you very much,@sshleifer- that’s super useful - I will usetransformer-clito start with and then we will update with symlinks.
2020-08-31T17:02:54Z
Don’t forget to also add model cards!