Some issues when training model on Sagemaker
Hello world,I’m getting two issues when I fine-tuning on my model using thissagemaker notebook.No GUI login prompt out when running notebook_login(), instead I’m getting this:error2991×230 16.6 KBAs a workaround, I’m using hardcoded tokenHit ResourceLimitExceeded Error when running huggingface_estimater.fit(…):error984×607 36.8 KBFor item 2, 1. I have opened an issue on AWS support to request for increasing the limit but I will expect a slow reply from them. Is there any other way to get around this while getting GPU boost from Sagemaker?FYI, I’m using my own AWS account (Free Tier account but having some credits).Thanks.
2021-11-23T15:51:23Z
Hello@ivanlau,thanks for opening the thread.To 1. where are you running the sagemaker notebook?To 2. I think you can go with theml.g4dn.xlargeit also has 1 GPU and shouldn’t need a limit increase for that.
2021-11-23T16:00:04Z
@philschmidTo 1:I’m running it on ml.t3.medium instance. Open it using JupyterLab environement (conda_pytorchp36)To 2:I changed it to your suggested instance but still same error:image1161×171 9.82 KB
2021-11-23T16:01:53Z
settings:image896×637 35.3 KB
2021-11-23T16:06:58Z
ivanlau:I’m running it on ml.t3.medium instance. Open it using JupyterLab environement (conda_pytorchp36)Can you test the Jupyter Noteboook?To 2. Okay then i guess to need to open a Support ticket. you can do this from AWS Console with service quota
2021-11-24T11:59:09Z
@philschmidHi,I have tested on Jupyter notebook. and ya, It’s working fine over there. It seems that JupyterLab ipywidgets is disabled by default or outdated? I not sure. but anyways, I can continue working on.For 2, Yes I already opened ticket and following up with them.Anyways, thanks for the help and your notebook too.This is my first time doing ML on the cloud. Learnt a lot.