Chapter 4 questions
Use this topic for any question aboutChapter 4of the course.
2021-06-30T16:04:31Z
Can someone take a look at the chapter 4 notebook. The.push_to_hub()method isn’t working, at first the error was about ‘git-lfs’ and even downloading that doesn’t seem to work…Thanks
2021-06-30T16:16:40Z
Could you tell us more? Which notebook are you running? When does it fail?
2021-06-30T17:14:19Z
This section’s colab notebook:Sharing models and tokenizers - Hugging Face CourseWhen I executemodel.push_to_hub("dummy-model")it throws error, although I’ve tried to solve it bysudo apt-get install git-lfsbut it still doesn’t work.By the way, I’m running all my codes in Google Colab, so I think you could reproduce the error by  running the code from colabThanks
2021-06-30T17:46:06Z
@khalidsaifullaahI see the following errorValueError: If not specifyingclone_from, you need to pass Repository a valid git clone.Do you see the same? If yes, for the time being, settinguse_temp_dir=Truein thepush_to_hubparams solved the issue for me.
2021-06-30T17:51:23Z
Yeah, I saw this one as well. Thanks for the solution, I’ll try using it now…But I was actually wondering,@sguggerdidn’t face these errors when he ran these same codes in the “Push To Hub” video (maybe it’s something to do with colab’s dependencies? in the video he used jupyter notebook)
2021-06-30T18:12:55Z
I have updated the install instructions in the notebooks to reflect all the necessary steps. Could you try again and tell me if it is working (on the latest version of the colab).
2021-06-30T18:55:42Z
thanks@sgugger!just checked the notebook, it’s working fine now!
2021-07-05T06:16:40Z
I’m getting this even though git lfs is installedimage1534×559 71 KB
2021-07-05T10:45:36Z
hey@khalidsaifullaahcould you please share a minimal example so i can try to reproduce the error on my side?
2021-07-05T13:02:28Z
thanks for the quick response@lewtun!I was actually trying to pretrain roberta model on GCP’s TPU using HF’s Roberta Flax trining script. I’ve followed the following steps to do it -transformers/examples/flax/language-modeling at master · huggingface/transformers (github.com)For the time being, I sidetracked the error by removing --push_to_hub flag when running the training script.
2021-07-05T13:16:41Z
ok thanks for the info! since the flax integration is quite new intransformersit’s possible there are some rough edges when it comes to integration with the hub.i’ll try to reproduce the error and report backps. you should be able to push the model to the hub using plain old git-lfs if you really need it
2021-07-05T16:57:41Z
Thanks@lewtun! Really appreciate your support.Just wanted to be clear on one thing-As I’ve removed thepush_to_hubflag, after everyepochthemodel.save_pretrained()method saving the checkpoints in my local directory. Now, when my training will be done, should I just do the following to upload everything to my model hub directory?git add .
git commit -m "model trained"
git push origin mainAre any other commands necessary (likegit lfs)? If so, in which order should it go, could you maybe give some suggestions on it?Thanks
2021-07-05T17:13:39Z
khalidsaifullaah:Are any other commands necessary (likegit lfs)? If so, in which order should it go, could you maybe give some suggestions on it?yes, for files larger than 10MB you’ll need to rungit lfs trackbeforegit add, e.g.git lfs track some_large_file.huge
git add .gitattributes
git add some_large_file.huge
git commit -m "add model files"hth!
2021-07-05T17:54:42Z
Thanks a lot!
2021-07-06T16:48:20Z
Getting this when tried to push in the hub. I didgit lfs track flax_model.msgpackandgit lfs *tfevents*before commit and push…image779×341 46.4 KB
2021-07-06T18:14:17Z
lewtun:.gitattributesfinally was able to push with the help of this -Failed to push model repo · Issue #8504 · huggingface/transformers (github.com)
2021-07-08T18:21:00Z
Hi team,Going through the last part got me thinking on some questions regarding quota & limits:Is there any limit number of repos a user can have private and public?Is there any limit to the size an individual dataset/model repo can have? Or a per-account limit?  (eg: On Kaggle, each user gets a certain fixed GBs to host and sum total should remain within limit.)If I do a 1000 commits of a 1GB model, is 1TB going to be ‘always-accesible’, or we have some stack limitations wrt git history?Is there any limit on number of downloads per model (specifically a privately uploaded model)?Questions are not necessarily as to what’s supported right now, but with some near-future perspective as well. eg: If I upload a public/private model (hypothetically for both commercial/non-commercial use) and not do the inference-api (just storage), will there be any threat to the 1)stability 2)scalability of such a pipeline?
2021-07-09T15:29:28Z
hey@dk-crazydivin the near-to-mid future, there are no limits
2021-07-09T22:25:08Z
Thank you@lewtun, but I am imagining myself crossing the gartners hype cycle and identifying the plateau on which the offering lands. Even though the fan in me would love to see this possible, but practically it raises many concerns. Could you please elaborate a bit?No limits onnumber of private repos on modelhub/datasethubnumber of public repossize of the reposnumber of commits on those reposnumber of downloads from those reposspeed cap on downloads of those repos.And also if someone subjectively “abuses” the policies and takes “unfair” advantage, does HF hold the right to ban? If yes, then it becomes even more concerning, as it is very very subjective.Before going deep into specific use cases, it will be great if you could point me to policies and terms of use, as it probably will clear up many of the cases I am thinking of.
No date available
No reply text available