Dataset list error - returns partial information
Hi,We tried to use HfApi to get a list of the datasets.When we run api.list_datasets() - we got exactly 10000 DatasetInfos , although the web count 19,351.Is it possible that the api uses pagination? If it is so, can you please explain how to use it? If not, can you please explain what are we missing?
2023-01-24T17:12:31Z
Yes, theHub APInow returns a paginated response for the lists. The huggingface_hub library takes care of the pagination and returns the complete list, but you’ll have to update it to 0.11.0 or more recent (Release Extended HfApi, pagination, simplified login and more · huggingface/huggingface_hub · GitHub).