Uploading large files (>5GB) to HF Spaces
Hi everyone,I need to use an 8.5 GB faiss index for my HF Spaces app. Although I’ve successfully uploaded large files to datasets and model hub before, I am now getting an error when pushing to HF Spaces repo. I read all theissues and troubleshootingothers had with git lfs, but this one I really couldn’t solve after trying for some time.Is this the recommended approach for large faiss files (pushing with git lfs)? I am using a cloud server so upload speed shouldn’t be an issue. It seems like the first 5 GB are transferred ok, and then towards the end of the transfer, I notice “400 Bad request” error logs. Here are my full gitlogs.TIA, Vladimir
2021-11-18T17:41:51Z
i don’t remember, did we implement lfs-largefiles for Spaces or no?@cbensimon@pierric
2021-11-22T12:45:46Z
Yes, it should be implemented. It might be buggy though sometimes to upload very large files … not completely sure about this. Are you still facing this issue@vblagoje?
2021-11-22T20:25:12Z
Yes@cbensimonno luck. I still get:(hf) paperspace@psnwsc5dm:~/lfqa$ git pushUploading LFS objects:   0% (0/1), 5.0 GB | 13 MB/s                                                                                                                                                       EOFoading LFS objects:   0% (0/1), 10 GB | 23 MB/serror: failed to push some refs to ‘https://huggingface.co/spaces/vblagoje/lfqa’
2021-11-25T17:10:00Z
Hey guys, even an alternative approach of getting the faiss file using wget fails with: “No space left on device”. Perhaps there is a size limitation on the container running the HF Spaces app but then how is the runtime related to git? Any thoughts? Can anything be done here@cbensimon?Thanks,Vladimir
2022-08-10T22:15:33Z
@vblagojeI am facing the exact same issue@cbensimon, please help us!
2022-08-16T10:56:24Z
Hey there, I usedhuggingface huband it worked! LMK if that worked for you!