LImit of 100000 files
Hello!I am trying to upload a dataset with 231685 audio files but apparently there is a limit in the number of files.remote: Your push was rejected because it contains too many files.
remote: Your git repo would contain 231687 files after this push, over
remote: the limit of 100000 files.What is the best practice for such datasets? Should I create an archive?
2022-11-21T15:28:23Z
Hi@ccoreilly,Grouping files in archives could be a solution indeed, but there might be solutions more adapted here - pinging@lhoestq@mariosasko@albertvillanovaas datasets experts
2022-11-21T15:43:42Z
I’d highly recommend to use archives indeed. Many filesystems wouldn’t support having so many files in a directory very well. Feel free to use TAR archives for examples (ZIP uses compression which is not ideal for already-compressed audio files).And if you have many archives, it also allows people to download a subset of the dataset easily, and it’s also a nice bonus for parallel downloading and processing