MMLU-PRO-ITA a new eval for Italian LLMs

<p>

In a previous  <a href="https://medium.com/@giuxale/an-analyses-on-italian-llms-models-evaluations-51bffe1d44d1" rel="nofollow">post</a>, we as  <a href="https://mii-lab.it/" rel="nofollow"><strong>mii-llm</strong></a>  lab, described an analysis on evaluating Italian LLMs on different common used benchmarks and launched the redesign of the  <a href="https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard">Italian Leaderboard</a>. In this post we will present a new evaluation benchmark  <a href="https://huggingface.co/datasets/efederici/MMLU-Pro-ita"><strong>mmlu-pro-ita</strong></a>  that has an open pull request on  <a href="https://github.com/EleutherAI/lm-evaluation-harness/pull/1860" rel="nofollow">lm-evaluation-harness</a>  and some results. If you want to see all data open the “Eval Aggiuntive” tab in the  <a href="https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard">Italian Leaderboard</a>.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*YtloA5vJvwDCNcgEbnB3zw.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*YtloA5vJvwDCNcgEbnB3zw.png"/></a></p>
<p><a href="https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro">MMLU-PRO</a> is an evolution of MMLU designed to evaluate language understanding models across broader and more challenging tasks. Building on the Massive Multitask Language Understanding (MMLU) dataset.  <a href="https://huggingface.co/datasets/efederici/MMLU-Pro-ita">MMLU-Pro-ita</a>  is a curated translation of the original dataset. Claude Opus from Anthropic with a prompt engineer draft and refine technique has been used as translator.</p>
<pre><code>You are a professional translation system that accurately translates multiple-choice exercises from English to Italian. Follow these steps to ensure high-quality translations:  
  
1. Provide an initial translation within &lt;traduzione&gt;&lt;/traduzione&gt; tags.  
2. Propose corrections, if necessary, within &lt;correzioni&gt;&lt;/correzioni&gt; tags, always re-reading the input problem.  
3. Write the final, polished translation within &lt;traduzione-finale&gt;&lt;/traduzione-finale&gt; tags.  
  
Adhere to the following requirements:  
1. Deliver top-notch, professional translations in Italian.  
2. Ensure the translated text is fluent, grammatically perfect, and uses standard Italian without regional bias.  
3. Accurately translate mathematical terms, notations, and equations, preserving their original meaning and structure.  
4. Focus solely on translating content without providing explanations, adding extra information, or copying the source text verbatim.  
  
Always use the following output format:  
  
&lt;traduzione&gt;  
    &lt;domanda&gt;[write the translated question here]&lt;/domanda&gt;  
    &lt;opzioni&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        ...  
    &lt;/opzioni&gt;  
&lt;/traduzione&gt;  
&lt;correzioni&gt;  
[write your corrections here, analyzing the translation quality, errors, and providing suggestions regarding the exercise and given options]  
&lt;/correzioni&gt;  
&lt;traduzione-finale&gt;  
    &lt;domanda&gt;[write the translated question here]&lt;/domanda&gt;  
    &lt;opzioni&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        &lt;opzione&gt;[write the translated option here]&lt;/opzione&gt;  
        ...  
    &lt;/opzioni&gt;  
&lt;/traduzione-finale&gt;  
  
From now on, only write in Italian and translate all incoming messages. Ensure the best translation possible.
</code></pre>
<p>The final result is a high quality translated dataset from the original MMLU-Pro. If you are interested in the technique, have a look at the  <a href="https://huggingface.co/datasets/efederici/MMLU-Pro-ita">dataset card</a>.</p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#results" id="results" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Results
	</span>
</h1>
<p>In the chart below the rank on MMLU-Pro-ita. Surprisingly:</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*TbnFA2RvG7ohPiINdtx9KQ.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*TbnFA2RvG7ohPiINdtx9KQ.png"/></a></p>
<ol>
<li><p><a href="https://huggingface.co/microsoft/Phi-3-medium-4k-instruct">microsoft/Phi-3medium-4k-instruct</a>  is the best performer. The Phi-3 family of models are trained on synthetic data, most likely in English, and from our experience they don’t speak very well Italian and are not easy to fine tune.Despite this it is in first position.</p>
</li>
<li><p>At the second position we found a fine tuned of Phi-3  <a href="https://huggingface.co/SeacomSrl/SeaPhi3-medium">seacom/SeaPhi3-medium</a>.</p>
</li>
<li><p>From the llama3 family at the third place there is the fine tune  <a href="https://huggingface.co/DeepMount00/Llama-3-8b-Ita">DeepMount00/Llama-3b-Ita</a>,</p>
</li>
<li><p>Interesting also the merged model  <a href="https://huggingface.co/anakin87/Llama-3-8b-ita-slerp">anakin87/Llama-3–8b-ita-slerp</a>  in the fourth place.</p>
</li>
<li><p>In the fifth place  <a href="https://huggingface.co/swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA">swap-uniba/LLaMAntino-3-ANITA-8B-Inst-DPO-ITA</a>.</p>
</li>
<li><p>And the best from the mistral family in sixth place  <a href="https://huggingface.co/mii-llm/maestrale-chat-v0.4-beta">mii-llm/maestrale-chat-v0.4-beta</a>  one of our favorite model.</p>
</li>
</ol>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#fine-tuned-models" id="fine-tuned-models" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Fine tuned models
	</span>
</h1>
<p>For fine-tuned models, another important metric is the percentage gain compared to their baseline models. This is useful for assessing the success of the training and also serves as a good indicator of the quality and quantity of data used in the fine-tuning steps. The chart below shows that  <a href="https://huggingface.co/DeepMount00/Llama-3-8b-Ita">DeepMount00/Llama-3b-Ita</a>  and  <a href="https://huggingface.co/mii-llm/maestrale-chat-v0.4-beta">mii-llm/maestrale-chat-v0.4-beta</a>  are the best models in terms of improvement from their base models, suggesting that they have used the most complete and best datasets for the mmlu-pro-it task.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*FLXVG7ZMiO2tZJt7UtEylg.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*FLXVG7ZMiO2tZJt7UtEylg.png"/></a></p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#maestrale-series" id="maestrale-series" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Maestrale series
	</span>
</h1>
<p>A deep dive analysis on Maestrale, that we know well, shows, for example, that continual pre-training tends to erode a small amount of specific knowledge, decreasing MMLU performance, while instilling more language specific knowledge. Instead the next releases based on SFT and KTO tends to increase specific knowledge on MMLU subjects demonstrating the quality of datasets for this particular set of tasks.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*YB7Rdf8scXisWNsMvl_Mbg.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*YB7Rdf8scXisWNsMvl_Mbg.png"/></a></p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#on-evals-and-benchmark" id="on-evals-and-benchmark" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		On evals and benchmark
	</span>
</h1>
<p>Evaluations and benchmarks tend to test only a particular or a small set of abilities of a model and they are subject to cultural knowledge and bias. This assumes that they can be only indicators of the model performance but cannot be used to judge a model for a particular use case. I suspect that we will see a fast domain specialisation of models and moreover a domain specialisation of evaluation and benchmarks. For example MMLU family of benchmarks test the knowledge on subjects evaluating how much knowledge has been able to be absorbed from a huge amount of text.</p>
<p>But that implies that if in the model training datasets such knowledge is not present or is not well represented your model will not perform very well, This is the reason why foundational models like  <a href="https://huggingface.co/collections/sapienzanlp/minerva-llms-661e6011828fe67de4fe7961">sapienzanlp/minerva-llms</a>  and  <a href="https://huggingface.co/iGeniusAI/Italia-9B-Instruct-v0.1">i</a><a href="https://huggingface.co/iGeniusAI/Italia-9B-Instruct-v0.1">iGeniusAI/Italia-9B-Instruct-v0.1</a>  trained mainly on Italian data do not perform so well in MMLU challenges. I’m a big fan of foundational models and we think that there will be space also for specific foundational SLM. Something we are cooking about.</p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#igenius-italia-vs-minerva" id="igenius-italia-vs-minerva" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Igenius Italia vs Minerva
	</span>
</h1>
<p>As in the chart below  <a href="https://huggingface.co/iGeniusAI/Italia-9B-Instruct-v0.1">iGeniusAI/Italia-9B-Instruct-v0.1</a>  a 9 billion model performs better than  <a href="https://huggingface.co/collections/sapienzanlp/minerva-llms-661e6011828fe67de4fe7961">sapienzanlp/minerva-llms</a>  but the biggest is only 3 billion parameters. Another curiousity is that the 350 M model is better than the 3B, may be it is something interesting to deepen.</p>
<p><a href="https://miro.medium.com/v2/resize:fit:1400/1*_L1WC-6zxo9jEijAVIlnqg.png" rel="nofollow"><img alt="" src="https://miro.medium.com/v2/resize:fit:1400/1*_L1WC-6zxo9jEijAVIlnqg.png"/></a></p>
<h1 class="relative group flex items-center">
<a class="block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full" href="#conclusions" id="conclusions" rel="nofollow">
<span class="header-link"><svg aria-hidden="true" class="text-gray-500 hover:text-black dark:hover:text-gray-200 w-4" height="1em" preserveaspectratio="xMidYMid meet" role="img" viewbox="0 0 256 256" width="1em" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span>
</a>
<span>
		Conclusions
	</span>
</h1>
<p>The evaluations and benchmarks fields in LLMs is very dynamic and changing fast, our vision is that in the next months and years will born hyper specialized LLMs and hence, many new benchmarks specific for various domains. We are also ready to release a new Italian based evaluation benchmark. Stay tuned and join our  <a href="https://mii-llm.ai/" rel="nofollow">community research lab</a>.</p>
