Uploading files larger than 5GB to model hub
I want to uploadctrlto model hub. I have followed the instructions from the documentation and it seems that they are applicable for smaller models (<5GB). Issues have been raisedhereandherebut it still seems unresolved.I followed the one shared by@julien-c(aws s3 cp …) but got the errorUnable to locate credentialsindicating that they are for HF staff.I am getting the errorcould not push some refsas 5GB limit is crossed. Can you please suggest a workaround if that exists ?@patrickvonplaten
2021-03-02T17:24:53Z
Did you install git-lfs?git lfs installYou also have to install our custom transfer agent for files >5GB:transformers-cli lfs-enable-largefilesLet me know if this helps!
2021-03-02T18:44:23Z
Thanks for replying. I havegit lfsand I tried custom transfer agent earlier. I tried it again just now, still gives me:EOFoading LFS objects:  50% (1/2), 5.0 GB | 0 B/s                                                                                                                                                            
error: failed to push some refs to 'https://huggingface.co/prajjwal1/ctrl_discovery_1'
2021-03-02T20:53:01Z
I manged to do it on the server. For some reason, it won’t work on local machine.
2021-03-02T22:30:36Z
are you able to upload files to S3 from your local machine? Might be a proxy-related issue.If you want more info, you can tryGIT_TRACE=1 GIT_CURL_VERBOSE=1 git push
2021-03-03T02:32:00Z
I have loaded many models with earlier version oftransformerswhich used to not rely on lfs. I’m not sure whats the issue on local machine, but its working on the server. May use the flag if I would need a trace.
2021-03-03T09:02:40Z
@prajjwal1git-lfs does its own networking (upload) calls with libcurl so the behavior when behind a proxy might be different frompython-requests.Let me know if you get more info in the future, this is interesting!
2021-05-02T14:23:01Z
Hello@julien-c,I am facing similar issues, attempting to upload a large file (just above 5GB) using git-lfs. I have a dual boot on my machine, and unfortunately did not manage to upload neither from Windows or Ubuntu 18.04.I have git-lfs installed with the environment variableGIT_LFS_SKIP_SMUDGE=1and ran(transformers) λ transformers-cli lfs-enable-largefiles .
Local repo set up for largefilesInterestingly, both the Windows and Ubuntu version return the following warning after adding the file:E:\path\to\gpt-neo-1.3B (main -> origin)
(transformers) λ git add rust_model.ot
Encountered 1 file(s) that may not have been copied correctly on Windows:
        rust_model.ot

See: `git lfs help smudge` for more details.Attempting topushreturns a similar warning:(transformers) λ git push
EOFoading LFS objects: 100% (1/1), 5.3 GB | 0 B/s
error: failed to push some refs to 'https://huggingface.co/EleutherAI/gpt-neo-1.3B'I read that git-lfs has issues for files larger than 4GB on windows - but I am surprised to face issues on my Ubuntu partition. Am I missing something?
2021-05-02T18:22:06Z
Just chiming in to say I’m getting the same errors on Arch. Git LFS is installed and enabled, adding a 5GB model gives the same warning about Windows, and then trying push results in the samefailed to push some refserror.EDIT: Trying again a few hours later and it worked, guess it was just a blip.
2021-05-03T11:52:41Z
cc’ing@pierricon this.If you’re not on Windows, the Windows-related warning should be innocuous (I get them too)@guillaume-becan you try again in case it was a temporary hiccup?
2021-05-03T16:05:17Z
Thank you for the quick feedback!I just tried again from Ubuntu and got the same error message:(transformers) guillaume@guillaume-MS-7B78:~/gpt-neo-1.3B$ transformers-cli lfs-enable-largefiles .
Local repo set up for largefiles
(transformers) guillaume@guillaume-MS-7B78:~/gpt-neo-1.3B$ git lfs track rust_model.ot
Tracking "rust_model.ot"
(transformers) guillaume@guillaume-MS-7B78:~/gpt-neo-1.3B$ git add rust_model.ot 
Encountered 1 file(s) that may not have been copied correctly on Windows:
	rust_model.ot
See: `git lfs help smudge` for more details.
(transformers) guillaume@guillaume-MS-7B78:~/gpt-neo-1.3B$ git commit -m "Addition of Rust model"
[main 094438a] Addition of Rust model
 1 file changed, 3 insertions(+)
 create mode 100755 rust_model.ot
((transformers) guillaume@guillaume-MS-7B78:~/gpt-neo-1.3B$ git push
Username for 'https://huggingface.co': guillaume-be
Password for 'https://guillaume-be@huggingface.co': 
Username for 'https://huggingface.co': guillaume-be                                                    
Password for 'https://guillaume-be@huggingface.co': 
EOFoading LFS objects:   0% (0/1), 5.0 GB | 0 B/s                                                      
error: failed to push some refs to 'https://huggingface.co/EleutherAI/gpt-neo-1.3B'I tried with and without the stepgit lfs track rust_model.ot, getting the same error both times. Unfortunately troubleshooting is time-consuming given the upload takes ~1 hour.edit: I just tried the upload again withGIT_TRACE=1 GIT_CURL_VERBOSE=1. Coincidentally, I am trying to upload ~5.3GB which should take just over an hour to complete. The upload crashes towards the end of the upload - but I believe this could very well be due to a timeout. Looks like the git worker below sets a"expires_in": 3600that exactly matches 1 hour.Is there a setting on server side (or something I should change on my side) to allow for uploads taking longer than an hour?18:06:53.358398 trace git-lfs: xfer: Custom adapter worker 0 sending message: 
{
  "event": "upload",
  "oid": "[...]",
  "size": 5312743379,
  "path": "/home/guillaume/gpt-neo-1.3B/.git/lfs/objects/cc/d3/[...]",
  "action": {
    "href": "https://huggingface.co/EleutherAI/gpt-neo-1.3B.git/info/lfs/objects/complete_multipart?uploadId=[...]",
    "header": {
      "00001": "h[...]",
      "00002": "[...]",
      "chunk_size": "5000000000"
    },
    "expires_at": "0001-01-01T00:00:00Z",
    "expires_in": 3600
  }
}
2021-05-03T16:50:52Z
Hi@guillaume-be, thank you for the debugging.The problem you face is indeed probably related to the expiration time defined for multipart uploads (1 hour), which you are exceeding.I just changed it to 3 hours, let me know how it goes!
2021-05-04T15:38:59Z
@pierricworks like a charm thank you!I am probably going to attempt an upload of Rust weights for GPT-Neo 2.7B tomorrow, hopefully I don’t exceed the 3 hours upload time
2021-05-04T19:15:06Z
@guillaume-beIf your upload speed is too slow a workaround might be to convert/upload from a server.
2021-05-04T19:59:39Z
Absolutely, this will definitely be the way to go for large models!
2021-09-03T18:19:51Z
I’m facing the same problem here can you tell me how can I change the expiration time, please?
2021-09-21T10:34:39Z
Hello,We just deployed a 24h expiration time for uploads.Hope this will fix the slow upload bandwidth issue.Regards.