Huggingface_hub list_datasets call
Hi everyone,I’m working with the huggingface_hub client library, which works so smooth!The reason to create this post is that I noticed that when calling the functionlist_datasetswith the parameterfull=True, thesiblingsfield (which has the names of the repository files) is alwaysNone.However, when calling the functionlist_repo_fileswith the parameterrepo_typeset to “dataset” we can retrieve all the files in the repository.siblingsattribute are of the classModelFile. Is there an ongoing implementation for theDatasetFileor another way to retrieve the filenames of a dataset repository with thelist_datasetscall?Thank you in advance
2022-07-19T11:12:34Z
Hello there,list_datasetsfunction is used tofilter all datasets on Hub with a given filter, meanwhilelist_repo_filesiterates over files of a given repository and thus you get siblings. Since they serve different purposes on different scopes, I’d suggest you to uselist_repo_filestolist siblings in a given repository.
2022-07-19T11:35:53Z
Hi@merve,I understand the idea here, but I was wondering is why is there a field inlist_datasetswhile never gives back the file’s list.
2022-10-21T15:43:39Z
Hi! I agree we should either fetch this info withfull=Trueor remove the field. cc@Wauplin
2022-10-31T16:00:55Z
The idea is that we have inhuggingface_hubaModelInfoobject to describe a model. Depending on the use case, not all information is fetched from the server (especially listing all files from each repo when listing all repos). If an information is not fetched, is it set toNone. Maybe not optimal but not sure we want to change this anytime soon. A solution to change this would be to have different classesModelInfoXXXdepending on the context but not sure it will be easier to use from a user perspective.