Introduction to Pagination
Hello there,I am using thelist_modelsfunction from the Hub client library. I have noticed that the return is a generator where formerly was a list, and that it will be added pagination on version 0.14.Therefore, I cast the output as recommended (list(iter(list_models))). However, sometimes it returns just 10k repos and sometimes it returns the whole list of model repos. If I use thesortanddirectionparametres it will always return 10k repos, however if I don’t use them it returns the full length of repos.Thank you,Adem
2023-02-10T14:01:26Z
Hi Adem, thank you for reporting use this issue. We have indeed implemented pagination as we are getting more and more models uploaded to the Hub. I tried to investigate your issue but couldn’t reproduce the error. Could you please provide more information about:the version ofhuggingface_hubyou are using (runhuggingface-cli envin your terminal and copy-paste the output)the exact commands that are causing you to retrieve only 10k modelsThank you very much in advance.I made aquick demo Spacehere to check if I get the same issues as you but it doesn’t seem to be the case:len(list(iter(list_models())))=>133871modelslen(list(iter(list_models(sort="lastModified", direction=1))))=>133871modelslen(list(iter(list_models(sort="lastModified", direction=-1))))=>133871modelslen(list(iter(list_models(sort="downloads", direction=1))))=>133871modelslen(list(iter(list_models(sort="downloads", direction=-1))))=>133871models
2023-02-13T16:31:53Z
Hi@Wauplin, the exact command is:hub_models = list(iter(api.list_models(full=True, cardData=True, fetch_config=True,sort="lastModified", direction=-1, use_auth_token=access_token))), and then it retrieves just 10k models. But if I removesortanddirectionparametres, it retrieves all the models (133544).I have also checked that the version is 0.12:
2023-02-14T09:05:39Z
@ademaitThank you for your feedback. I am able to reproduce your issue and it’s indeed a bug. I opened aninternal ticket(internal link) to get it fixed server-side. I’ll keep you posted if I have any updates.
2023-02-23T17:42:37Z
Hi@ademait. I’m getting back to you about this issue. It is now fixed server-side. If you tried again with the same code, you should not be truncated to 10k models anymore. Please let me know if are having any further issues
2023-02-24T10:41:17Z
ademait:ist(iter(api.list_models(full=True, cardData=True, fetch_config=True,sort="lastModified", direction=-1, use_auth_token=access_token)))Hi@Wauplin, I have just tried it. It is solved.Thank you for the fast response.