AutoTrain models performance (mainly F1 score)
Where can I see the performance of my models (mainly the F1 score), as shown on the homepage of AutoNLP/AutoTrain?I have trained 5 models using my own data but am only able to see the Loss of each one under the metrics tab. Seeing as I’m researching ways to generate synthetic data for question answering models, I would like to allocate my time towards the generation of the context, question, answer triplets and not the evaluation. AutoNLP/AutoTrain seems like a good solution for this, however, not if I’m unable to get F1 or EM scores like in the screenshot below.Screenshot from 2022-04-12 15-09-13979×595 102 KBAbout my research (for context)I have generated QA data in (1) English based on Wiki texts, (2) Dutch based on Wiki texts and (3) Dutch based on technical/domain specific documents. I have used 8 different methods of generating the data, resulting in 8 x 3 = 24 datasets that I would like to evaluate using a QA model. If I were to use AutoNLP/AutoTrain for this, I would like to be sure that the results are something I can use.
2022-04-13T09:35:27Z
pinging@abhishek@sbrandeison this!
2022-04-13T10:24:34Z
Unfortunately, currently, other metrics are not available for QA tasks. I have made an internal issue to keep track of this and will update you when other metrics are available. You can, for now, calculate other metrics by generating predictions on your own. In the meantime, i would also like to know if there are any specific metrics would you want to see for question answering?
2022-12-31T13:33:27Z
Please provide any update on this issue.
2022-12-31T13:51:51Z
If you are asking about the QA metrics, they are available in the UI now.
2023-01-02T10:55:17Z
Hi abhishek, i still see only loss metrics for models finetuned for qa tasks, not seeing f1 score or em.
2023-01-02T12:13:26Z
when was the model trained? could you please provide the project name and username?
2023-01-03T14:13:07Z
Hi abhishek, i just now trained a new model. Project name: testing-finetune-autotrainusername: gokul427model used: bert-base-uncaseddata: lmqg/qa_squadshiftsplease help.