Error in finetuning File "/app/env/bin/accelerate"
image2652×895 330 KBI was trying to finetune a llm model but while loading the shards i am encountring the above error and all the process of downloading the model starts again.File “/app/env/bin/accelerate” , line 8, in sys.exit (main ())File “/app/env/lib/python3.10/site-packages/accelerate/commands/accelerate_clipy”,line 47, inmain args. func (args)File “/app/env/lib/python3.10/site-packages/accelerate/commands/launch.py”, line 1017, inlaunch_command simple_launcher (args)File “/app/env/lib/python3.10/site-packages/accelerate/commands/launch.py”, line 637, insimple_launcher raise subprocess. CalledProcessError (returncode=process.returncode,cmd=cmd) subprocess. CalledProcessError: Command ‘[’/app/env/bin/python’, "-m’.‘autotrain.trainers.cIm’, ‘–training_config’, ‘/tmp/model/training_params.json’]’ died with <Signals.SIGKILL:
2023-12-21T19:52:32Z
memory error. need bigger gpu