Training fails but no error message
Hi All,I have been following this tutorialNon-engineers guide: Train a LLaMA 2 chatbotto fine tune an LLM (meta/opt125m) with my own data.The training starts well, but always fails between 3-10%. I cannot see any error message that I could use to find the issue an correct it. The Space with the model training goes into “Paused” mode.Could you help me what is thue issue?Any points would be greatly appreciated, many thanks!I paste the logs from 2 trainings here. This is the end of the logfile, the instances are Paused by the platform.1st one:300/11676 [01:01<38:19,  4.95it/s]
  3%|▎         | 301/11676 [01:02<38:17,  4.95it/s]
  3%|▎         | 302/11676 [01:02<38:17,  4.95it/s]
  3%|▎         | 303/11676 [01:02<38:17,  4.95it/s]
  3%|▎         | 304/11676 [01:02<38:18,  4.95it/s]
  3%|▎         | 305/11676 [01:02<38:17,  4.95it/s]
  3%|▎         | 306/11676 [01:03<38:15,  4.95it/s]
  3%|▎         | 307/11676 [01:03<38:14,  4.95it/s]
  3%|▎         | 308/11676 [01:03<38:14,  4.95it/s]
  3%|▎         | 309/11676 [01:03<38:15,  4.95it/s]
  3%|▎         | 310/11676 [01:03<38:16,  4.95it/s]
  3%|▎         | 311/11676 [0
 4.95it/s]
  3%|▎         | 312/11676 [01:04<38:19,  4.94it/s]
  3%|▎         | 313/11676 [01:04<38:21,  4.94it/s]
  3%|▎         | 314/11676 [01:04<38:19,  4.94it/s]
  3%|▎         | 315/11676 [01:04<38:17,  4.94it/s]
  3%|▎         | 316/11676 [01:05<38:15,  4.95it/s]
  3%|▎         | 317/11676 [01:05<38:17,  4.94it/s]
  3%|▎         | 318/11676 [01:05<38:17,  4.94it/s]
  3%|▎         | 319/11676 [01:05<38:16,  4.95it/s]
  3%|▎         | 320/11676 [01:05<38:14,  4.95it/s]
  3%|▎         | 321/11676 [01:06<38:13,  4.95it/s]
  3%|▎         | 322/11676 [01:06<38:14,  4.95it/s]
  3%|▎         | 323/11676 [01:06<38:16,  4.94it/s]
  3%|▎         | 324/11676 [01:06<38:15,  4.95it/s]
  3%|▎         | 325/11676 [01:06<38:13,  4.95it/s]
  3%|▎         | 326/11676 [01:07<38:11,  4.95it/s]
  3%|▎         | 327/11676 [01:07<38:12,  4.95it/s]
  3%|▎         | 328/11676 [01:07<38:14,  4.95it/s]
  3%|▎         | 329/11676 [01:07<38:45,  4.88it/s]
  3%|▎         | 330/11676 [01:07<33:28,  5.65it/s]/app/env/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
INFO:     10.16.38.82:50124 - "GET /?logs=container&__sign=eyJhbGciOiJFZERTQSJ9.eyJpYXQiOjE2OTY2ODQxODIsInN1YiI6IlJlZDVSZWQ1L2F1dG90cmFpbi1kYXRhcGFydDA5LW9wdDEyNS0wIiwiZXhwIjoxNjk2NzcwNTgyLCJpc3MiOiJodHRwczovL2h1Z2dpbmdmYWNlLmNvIn0.2acH2ViyMIqWcwNHjg2frZ5xntfUEbPyEma7rcD1iUwAnO4oacx1o022SVrek3bTbZWkW0qyMJgrjSGOQJjuBw HTTP/1.1" 200 OK2nd one[02:06<37:07,  4.96it/s]
  5%|▌         | 620/11676 [02:07<37:06,  4.97it/s]
  5%|▌         | 621/11676 [02:07<37:06,  4.97it/s]
  5%|▌         | 622/11676 [02:07<37:06,  4.96it/s]
  5%|▌         | 623/11676 [02:07<37:07,  4.96it/s]
  5%|▌         | 624/11676 [02:07<37:06,  4.96it/s]
  5%|▌         | 625/11676 [02:08<37:06,  4.96it/s]
  5%|▌         | 626/11676 [02:08<37:07,  4.96it/s]
  5%|▌         | 627/11676 [02:08<37:07,  4.96it/s]
  5%|▌         | 628/11676 [02:08<37:07,  4.96it/s]
  5%|▌         | 629/11676 [02:08<37:06,  4.96it/s]
  5%|▌         | 630/11676 [02:09<37:06,  4.96it/s]
  5%|▌         | 631/11676 [02:09<37:06,  4.96it/s]
  5%|▌         | 632/11676 [02:09<37:05,  4.96it/s]
  5%|▌         | 633/11676 [02:09<37:05,  4.96it/s]
  5%|▌         | 634/11676 [02:09<37:04,  4.96it/s]
  5%|▌         | 635/11676 [02:10<37:06,  4.96it/s]
  5%|▌         | 636/11676 [02:10<37:05,  4.96it/s]
  5%|▌         | 637/11676 [02:10<37:05,  4.96it/s]
  5%|▌         | 638/11676 [02:10<37:04,  4.96it/s]
  5%|▌         | 639/11676 [02:10<37:03,  4.96it/s]
  5%|▌         | 640/11676 [02:11<37:06,  4.96it/s]
  5%|▌         | 641/11676 [02:11<37:06,  4.96it/s]
  5%|▌         | 642/11676 [02:11<37:06,  4.96it/s]
  6%|▌         | 643/11676 [02:11<37:05,  4.96it/s]
  6%|▌       
 | 644/11676 [02:11<37:04,  4.96it/s]
  6%|▌         | 645/11676 [02:12<37:04,  4.96it/s]
  6%|▌         | 646/11676 [02:12<37:04,  4.96it/s]
  6%|▌         | 647/11676 [02:12<37:03,  4.96it/s]
  6%|▌         | 648/11676 [02:12<37:02,  4.96it/s]
  6%|▌         | 649/11676 [02:12<37:01,  4.96it/s]
  6%|▌         | 650/11676 [02:13<37:01,  4.96it/s]
  6%|▌         | 651/11676 [02:13<37:01,  4.96it/s]
  6%|▌         | 652/11676 [02:13<37:00,  4.96it/s]
  6%|▌         | 653/11676 [02:13<37:00,  4.96it/s]
  6%|▌         | 654/11676 [02:13<37:00,  4.96it/s]
  6%|▌         | 655/11676 [02:14<37:00,  4.96it/s]
  6%|▌         | 656/11676 [02:14<37:00,  4.96it/s]
  6%|▌         | 657/11676 [02:14<37:00,  4.96it/s]
  6%|▌         | 658/11676 [02:14<37:00,  4.96it/s]
  6%|▌         | 659/11676 [02:14<37:00,  4.96it/s]
  6%|▌         | 660/11676 [02:15<37:00,  4.96it/s]
  6%|▌         | 661/11676 [02:15<37:01,  4.96it/s]
  6%|▌         | 662/11676 [02:15<37:01,  4.96it/s]
  6%|▌         | 663/11676 [02:15<36:59,  4.96it/s]
  6%|▌         | 664/11676 [02:15<36:59,  4.96it/s]
  6%|▌         | 665/11676 [02:16<36:58,  4.96it/s]
  6%|▌         | 666/11676 [02:16<36:59,  4.96it/s]
  6%|▌         | 667/11676 [02:16<36:59,  4.96it/s]
  6%|▌         | 668/11676 [02:16<36:58,  4.96it/s]
  6%|▌         | 669/11676 [02:16<36:57,  4.96it/s]
  6%|▌         | 670/11676 [02:17<36:57,  4.96it/s]
  6%|▌         | 671/11676 [02:17<36:56,  4.96it/s]
  6%|▌         | 672/11676 [02:17<36:56,  4.97it/s]
  6%|▌         | 673/11676 [02:17<36:55,  4.97it/s]
  6%|▌         | 674/11676 [02:17<36:55,  4.97it/s]
  6%|▌         | 675/11676 [02:18<36:55,  4.97it/s]
  6%|▌         | 676/11676 [02:18<36:54,  4.97it/s]
  6%|▌         | 677/11676 [02:18<36:54,  4.97it/s]
  6%|▌         | 678/11676 [02:18<36:54,  4.97it/s]
  6%|▌         | 679/11676 [02:18<36:53,  4.97it/s]
  6%|▌         | 680/11676 [02:19<37:23,  4.90it/s]/app/env/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
2023-10-07T14:33:08Z
Seems to be going out of memory. Which machine did you use?
2023-10-07T14:47:19Z
A10G Large in all casesI tried to use the A100, but I always get an error that my account is not eligible.Is it possible to get the A100 enabled? Or would you recommend using a smaller model?The meta/opt125m is anyway one of the smaller options in the selectionMy own dataset is about 3-4mb, so not much at all
2023-10-07T14:50:11Z
thats a small model. are you sure the trained model is not in your profile? the training space pauses on its own when the training is done
2023-10-07T14:55:59Z
I am positive, the model did not complete the training.I have kept the browser with open with the logs scrolling as the training went on, and it always suddenly stops always between 3-10% completion. I tried multiple models, multiple training datasets. Always leaving all options on default, as suggested by the tutorial above.
2023-10-07T15:07:23Z
If you are using SFT, the progress bar is not representative. The only way to confirm if it finished is to check your hf account for new model repos.
2023-10-07T18:16:58Z
I have the same issue.  I’ve wasted $30 at this point and gotten nowhere.  I get a model repo with a folder called “checkpoint - xxx” which doesn’t seem complete and will not work with a chat ui as outlined in the tutorial.  I’ve followed the tutorial provided by huggingface word for word and it doesn’t work.
2023-10-07T18:27:39Z
thanks so much! that’s amazingly helpful. The models were acutaly there on under my HF account, but I didn’t understand that they were actually readyNow I want to do the ChatUI part. I launched the ChatUI docker instance, based on a model I successfully trainedBut now I get this error:OSError: Red5Red5/arc-ai-v01-0 does not appear to have a file named config.json. Checkout ‘https://huggingface.co/Red5Red5/arc-ai-v01-0/main’ for available files.There is indeed no config.json file. The model is based on “meta-llama/Llama-2-7b-chat-hf”.How can I generate a config.json file?Thank you again for the help!
2023-10-07T18:29:59Z
Try clicking your HF profile on the top-right corner, click on your own username.Then scroll below the “Spaces” section.Under the Models section do you see anything? That’s where I found my precious models that I trained for good $$$Kind of wish there was a little notification that the AutoTraining was successful and you can find your model in the Models section of your account
2023-10-07T18:36:07Z
Hi Red, Yes, but I don’t think the model completed which is why we’re getting the config.json error.  I’m assuming the checkpoint folder in the model repo is where the model training “paused” before completion.
2023-10-08T23:33:03Z
@abhishek(or anyone) can you confirm if an A10G large is actually powerful enough to autotrain Llama 7b on a 52k row dataset as described in your tutorial?Non-engineers guide: Train a LLaMA 2 chatbot
2023-10-10T08:33:17Z
@mortsnnot finding config.json is not an issue. the model was trained successfully, but since its an adapter model, we need to merge the adapter. seems like this step is missing from the blog post. ill update it.
2023-10-10T17:53:44Z
Thanks you!
2023-10-16T18:28:09Z
@abhishekare you still planning to update the blog post?  I’d love to continue working with autotrain, but am unsure how to merge the adapter.  Thanks.
2023-10-16T18:47:44Z
If you use --merge-adapter param during training, it will be merged when training finishes. ill update blog post tomorrow
2023-10-20T17:50:18Z
Hi@abhishekI don’t believe there is an option to set the --merge-adapter parameter in the autotrain tool.  Are you still planning to update the blog post?
2023-10-21T09:47:11Z
Sorry for my late response. You can use this space to merge adapter for models trained using autotrain ui:Llm Merge Adapter - a Hugging Face Space by autotrain-projects. Please let me know if you have any questions.
2023-10-21T18:02:58Z
Thank you@abhishek!  It seems like I was able to merge the model successfully but now the Chat UI step seems broken. It gets stuck in a loop of “Failed to connect to 127.0.0.1 port 8080: Connection refused” when building.There are multiple threads reporting this issue and it looks like your own space is having the issue. Is there an issue with Chat UI or does this mean something is wrong with the model?thread 1thread 2spaceThanks
2023-10-21T18:14:19Z
Are you attaching gpu of appropriate size for the model?
2023-10-21T19:42:19Z
@abhishekyes I get this error whether I use the recommended for ChatUi A10 small or try to go higher to the A10 large.It does give this error before the failed to connect: ValueError: Expected target PEFT class: PeftModelForCausalLM, but you have asked for: PeftModelForSeq2SeqLM make sure that you are loading the correct model for your task type.Could that be the issue?  Is there a way to specify to use “PeftModelForCausalLM”?
No date available
No reply text available