text
On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis  based on data are:
On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis  based on data are:
new_line
new_line
text
 - mmlu is hard to improve when fine tuning a base model on a different language 
 - mmlu is hard to improve when fine tuning a base model on a different language 
new_line
text
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
new_line
text
- fine tuning can specialize well but at cost of loosing some foundational  knowledge. 
- fine tuning can specialize well but at cost of loosing some foundational  knowledge. 
new_line
new_line
text
Here the data 
Here the data 
link
https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
new_line
text
Here the colab 
Here the colab 
link
https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
new_line
text
Here an article with some considerations 
Here an article with some considerations 
link
https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1
new_line
new_line
On evaluating fine tuned 7B Italian open source LLMs I have collected many data points and I created a super simple explorative analyses. My hypothesis  based on data are:

 - mmlu is hard to improve when fine tuning a base model on a different language 
- fine tuning also on single GPUs can improve by 5% to 10% the base model on common tasks but a lot more on specific cases with the right training time and data
- fine tuning can specialize well but at cost of loosing some foundational  knowledge. 

Here the data https://docs.google.com/spreadsheets/d/1MBcxy1loK8eIycZG4DN84Q2ejZ0jSjxUBgoShHDR6IY/edit?usp=sharing
Here the colab https://colab.research.google.com/drive/1ra4_skG5QYWSYOzvagOoIoj4bibQD8Gw?usp=sharing
Here an article with some considerations https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1


🚀
Osd111 beekay24 lunarflu nikgr giux78