New Model sharing and uploading is extremely slow
Hello,The new “Model sharing and uploading” using git is a great idea.However, I tried to update one of my models “t5-3b” which is “11G”, and it is taking too much time for “git add --all” command.I had to cancel it at the end.This is because the nature of git for large files while calculating the diff.Is there any solution for this problem or workaround ?
2020-11-16T09:42:53Z
To have an idea of which steps take time, you can add the following env variables to your git commands:GIT_CURL_VERBOSE=1 GIT_TRACE=1Also make sure you actually have git-lfs activated:git lfs install. (We’re in the process of adding it everywhere in the documentation as it was easy to miss).On the hashing overhead part: for me hashing (w/ sha256) a 42GB file takes 2’30sec (on a beefy computer) so the hashing overhead is not too huge, and of course gives the benefit of security/reproducibility.And finally in your precise use case, uploading files larger than 5GB is currently not supported out of the box, though we’ll support it in the next 2-3 weeks. Issue to follow is athttps://github.com/huggingface/transformers/issues/8480#issuecomment-726731046
2020-11-16T13:14:27Z
Thanks@julien-c, the issei’s was indeed on “git lsf”. After reinstalling it, it worked much faster.