--------------------------------------------------------------------------------------------------

<body>
<span class="vertical-text" style="background-color:lightgreen;border-radius: 3px;padding: 3px;"> </span>
<br>
<span class="vertical-text" style="background-color:orange;border-radius: 3px;padding: 3px;">  </span>
<br>
<span class="vertical-text" style="background-color:lightblue;border-radius: 3px;padding: 3px;">    Model: BLAZE 🔥</span>
<br>
<span class="vertical-text" style="background-color:tomato;border-radius: 3px;padding: 3px;">    Lang: IT</span>
<br>
<span class="vertical-text" style="background-color:lightgrey;border-radius: 3px;padding: 3px;">  </span>
<br>
<span class="vertical-text" style="background-color:#CF9FFF;border-radius: 3px;padding: 3px;"> </span>
</body>

--------------------------------------------------------------------------------------------------

<h3>Introduction</h3>

This model is a <b>lightweight</b> and uncased version of <b>BERT</b> <b>[1]</b> for the <b>Italian</b> language. Its <b>55M parameters</b> and <b>220MB</b> size make it
<b>50% lighter</b> than a typical mono-lingual BERT model. It is ideal when memory consumption and execution speed are critical while maintaining high-quality results.


<h3>AILC CLiC-IT 2023 Proceedings</h3>

"Blaze-IT: a lightweight BERT model for the Italian language" has been accepted at AILC CLiC-IT 2023 and published in the conference proceedings.
<br>
You can find the proceedings here: https://clic2023.ilc.cnr.it/proceedings/
<br>
And the published paper here: https://ceur-ws.org/Vol-3596/paper43.pdf


<h3>Model description</h3>

The model builds on the multilingual <b>DistilBERT</b> <b>[2]</b> model (from the HuggingFace team: [distilbert-base-multilingual-cased](https://huggingface.co/distilbert-base-multilingual-cased)) as a starting point, 
focusing it on the Italian language while at the same time turning it into an uncased model by modifying the embedding layer 
(as in <b>[3]</b>, but computing document-level frequencies over the <b>Wikipedia</b> dataset and setting a frequency threshold of 0.1%), which brings a considerable
reduction in the number of parameters.

To compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words previously capitalized, 
the model has been further pre-trained on the Italian split of the [Wikipedia](https://huggingface.co/datasets/wikipedia) dataset, using the <b>whole word masking [4]</b> technique to make it more robust 
to the new uncased representations.

The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it <b>50% lighter</b> than a typical mono-lingual BERT model and
20% lighter than a standard mono-lingual DistilBERT model.


<h3>Training procedure</h3>

The model has been trained for <b>masked language modeling</b> on the Italian <b>Wikipedia</b> (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512 
(obtained through 128 gradient accumulation steps),
a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using <b>dynamic masking</b> between epochs and
exploiting the <b>whole word masking</b> technique.


<h3>Performances</h3>

The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the <b>UD Italian ISDT</b> and <b>WikiNER</b> datasets, respectively. 
The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation

| Task | Recall | Precision | F1 |
| ------ | ------ | ------ |  ------ |
| Part of Speech Tagging | 97.48  | 97.29 | 97.37 |
| Named Entity Recognition | 89.29  | 89.84 | 89.53 |

The metrics have been computed at the token level and macro-averaged over the classes.


<h3>Demo</h3>

You can try the model online (fine-tuned on named entity recognition) using this web app: https://huggingface.co/spaces/osiria/blaze-it-demo

<h3>Quick usage</h3>

```python
from transformers import AutoTokenizer, DistilBertForMaskedLM
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("osiria/blaze-it")
model = DistilBertForMaskedLM.from_pretrained("osiria/blaze-it")
pipeline_mlm = pipeline(task="fill-mask", model=model, tokenizer=tokenizer)
```


<h3>Limitations</h3>

This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text 
from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). However, it may show limitations when it comes to chaotic text, containing errors and slang expressions
(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content).

<h3>References</h3>

[1] https://arxiv.org/abs/1810.04805

[2] https://arxiv.org/abs/1910.01108

[3] https://arxiv.org/abs/2010.05609

[4] https://arxiv.org/abs/1906.08101

<h3>License</h3>

The model is released under <b>Apache-2.0</b> license
{"language": ["it"], "license": "apache-2.0", "datasets": ["wikipedia"], "widget": [{"text": "milano \u00e8 una [MASK] dell'italia", "example_title": "Example 1"}, {"text": "il sole \u00e8 una [MASK] della via lattea", "example_title": "Example 2"}, {"text": "l'italia \u00e8 una [MASK] dell'unione europea", "example_title": "Example 3"}]}
transformers pytorch safetensors distilbert fill-mask it dataset:wikipedia arxiv:1810.04805 arxiv:1910.01108 arxiv:2010.05609 arxiv:1906.08101 license:apache-2.0 autotrain_compatible endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #it #dataset-wikipedia #arxiv-1810.04805 #arxiv-1910.01108 #arxiv-2010.05609 #arxiv-1906.08101 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 



---



 
  

  
  

    Model: BLAZE 
  

    Lang: IT
  

  
  

 



---


### Introduction


This model is a **lightweight** and uncased version of **BERT** **[1]** for the **Italian** language. Its **55M parameters** and **220MB** size make it
**50% lighter** than a typical mono-lingual BERT model. It is ideal when memory consumption and execution speed are critical while maintaining high-quality results.


### AILC CLiC-IT 2023 Proceedings


"Blaze-IT: a lightweight BERT model for the Italian language" has been accepted at AILC CLiC-IT 2023 and published in the conference proceedings.
  

You can find the proceedings here: URL
  

And the published paper here: URL


### Model description


The model builds on the multilingual **DistilBERT** **[2]** model (from the HuggingFace team: distilbert-base-multilingual-cased) as a starting point,
focusing it on the Italian language while at the same time turning it into an uncased model by modifying the embedding layer
(as in **[3]**, but computing document-level frequencies over the **Wikipedia** dataset and setting a frequency threshold of 0.1%), which brings a considerable
reduction in the number of parameters.


To compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words previously capitalized,
the model has been further pre-trained on the Italian split of the Wikipedia dataset, using the **whole word masking [4]** technique to make it more robust
to the new uncased representations.


The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it **50% lighter** than a typical mono-lingual BERT model and
20% lighter than a standard mono-lingual DistilBERT model.


### Training procedure


The model has been trained for **masked language modeling** on the Italian **Wikipedia** (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512
(obtained through 128 gradient accumulation steps),
a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using **dynamic masking** between epochs and
exploiting the **whole word masking** technique.


### Performances


The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the **UD Italian ISDT** and **WikiNER** datasets, respectively.
The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation



The metrics have been computed at the token level and macro-averaged over the classes.


### Demo


You can try the model online (fine-tuned on named entity recognition) using this web app: URL


### Quick usage


### Limitations


This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text
from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). However, it may show limitations when it comes to chaotic text, containing errors and slang expressions
(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content).


### References


[1] URL


[2] URL


[3] URL


[4] URL


### License


The model is released under **Apache-2.0** license



### Introduction


This model is a **lightweight** and uncased version of **BERT** **[1]** for the **Italian** language. Its **55M parameters** and **220MB** size make it
**50% lighter** than a typical mono-lingual BERT model. It is ideal when memory consumption and execution speed are critical while maintaining high-quality results. ### AILC CLiC-IT 2023 Proceedings


"Blaze-IT: a lightweight BERT model for the Italian language" has been accepted at AILC CLiC-IT 2023 and published in the conference proceedings.
  

You can find the proceedings here: URL
  

And the published paper here: URL ### Model description


The model builds on the multilingual **DistilBERT** **[2]** model (from the HuggingFace team: distilbert-base-multilingual-cased) as a starting point,
focusing it on the Italian language while at the same time turning it into an uncased model by modifying the embedding layer
(as in **[3]**, but computing document-level frequencies over the **Wikipedia** dataset and setting a frequency threshold of 0.1%), which brings a considerable
reduction in the number of parameters.


To compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words previously capitalized,
the model has been further pre-trained on the Italian split of the Wikipedia dataset, using the **whole word masking [4]** technique to make it more robust
to the new uncased representations.


The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it **50% lighter** than a typical mono-lingual BERT model and
20% lighter than a standard mono-lingual DistilBERT model. ### Training procedure


The model has been trained for **masked language modeling** on the Italian **Wikipedia** (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512
(obtained through 128 gradient accumulation steps),
a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using **dynamic masking** between epochs and
exploiting the **whole word masking** technique. ### Performances


The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the **UD Italian ISDT** and **WikiNER** datasets, respectively.
The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation



The metrics have been computed at the token level and macro-averaged over the classes. ### Demo


You can try the model online (fine-tuned on named entity recognition) using this web app: URL ### Quick usage ### Limitations


This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text
from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). However, it may show limitations when it comes to chaotic text, containing errors and slang expressions
(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content). ### References


[1] URL


[2] URL


[3] URL


[4] URL ### License


The model is released under **Apache-2.0** license
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #it #dataset-wikipedia #arxiv-1810.04805 #arxiv-1910.01108 #arxiv-2010.05609 #arxiv-1906.08101 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 
 ### Introduction


This model is a **lightweight** and uncased version of **BERT** **[1]** for the **Italian** language. Its **55M parameters** and **220MB** size make it
**50% lighter** than a typical mono-lingual BERT model. It is ideal when memory consumption and execution speed are critical while maintaining high-quality results. ### AILC CLiC-IT 2023 Proceedings


"Blaze-IT: a lightweight BERT model for the Italian language" has been accepted at AILC CLiC-IT 2023 and published in the conference proceedings.
  

You can find the proceedings here: URL
  

And the published paper here: URL ### Model description


The model builds on the multilingual **DistilBERT** **[2]** model (from the HuggingFace team: distilbert-base-multilingual-cased) as a starting point,
focusing it on the Italian language while at the same time turning it into an uncased model by modifying the embedding layer
(as in **[3]**, but computing document-level frequencies over the **Wikipedia** dataset and setting a frequency threshold of 0.1%), which brings a considerable
reduction in the number of parameters.


To compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words previously capitalized,
the model has been further pre-trained on the Italian split of the Wikipedia dataset, using the **whole word masking [4]** technique to make it more robust
to the new uncased representations.


The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it **50% lighter** than a typical mono-lingual BERT model and
20% lighter than a standard mono-lingual DistilBERT model. ### Training procedure


The model has been trained for **masked language modeling** on the Italian **Wikipedia** (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512
(obtained through 128 gradient accumulation steps),
a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using **dynamic masking** between epochs and
exploiting the **whole word masking** technique. ### Performances


The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the **UD Italian ISDT** and **WikiNER** datasets, respectively.
The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation



The metrics have been computed at the token level and macro-averaged over the classes. ### Demo


You can try the model online (fine-tuned on named entity recognition) using this web app: URL ### Quick usage ### Limitations


This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text
from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). However, it may show limitations when it comes to chaotic text, containing errors and slang expressions
(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content). ### References


[1] URL


[2] URL


[3] URL


[4] URL ### License


The model is released under **Apache-2.0** license
TAGS
#transformers #pytorch #safetensors #distilbert #fill-mask #it #dataset-wikipedia #arxiv-1810.04805 #arxiv-1910.01108 #arxiv-2010.05609 #arxiv-1906.08101 #license-apache-2.0 #autotrain_compatible #endpoints_compatible #has_space #region-us 
### Introduction


This model is a **lightweight** and uncased version of **BERT** **[1]** for the **Italian** language. Its **55M parameters** and **220MB** size make it
**50% lighter** than a typical mono-lingual BERT model. It is ideal when memory consumption and execution speed are critical while maintaining high-quality results.### AILC CLiC-IT 2023 Proceedings


"Blaze-IT: a lightweight BERT model for the Italian language" has been accepted at AILC CLiC-IT 2023 and published in the conference proceedings.
  

You can find the proceedings here: URL
  

And the published paper here: URL### Model description


The model builds on the multilingual **DistilBERT** **[2]** model (from the HuggingFace team: distilbert-base-multilingual-cased) as a starting point,
focusing it on the Italian language while at the same time turning it into an uncased model by modifying the embedding layer
(as in **[3]**, but computing document-level frequencies over the **Wikipedia** dataset and setting a frequency threshold of 0.1%), which brings a considerable
reduction in the number of parameters.


To compensate for the deletion of cased tokens, which now forces the model to exploit lowercase representations of words previously capitalized,
the model has been further pre-trained on the Italian split of the Wikipedia dataset, using the **whole word masking [4]** technique to make it more robust
to the new uncased representations.


The resulting model has 55M parameters, a vocabulary of 13.832 tokens, and a size of 220MB, which makes it **50% lighter** than a typical mono-lingual BERT model and
20% lighter than a standard mono-lingual DistilBERT model.### Training procedure


The model has been trained for **masked language modeling** on the Italian **Wikipedia** (~3GB) dataset for 10K steps, using the AdamW optimizer, with a batch size of 512
(obtained through 128 gradient accumulation steps),
a sequence length of 512, and a linearly decaying learning rate starting from 5e-5. The training has been performed using **dynamic masking** between epochs and
exploiting the **whole word masking** technique.### Performances


The following metrics have been computed on the Part of Speech Tagging and Named Entity Recognition tasks, using the **UD Italian ISDT** and **WikiNER** datasets, respectively.
The PoST model has been trained for 5 epochs, and the NER model for 3 epochs, both with a constant learning rate, fixed at 1e-5. For Part of Speech Tagging, the metrics have been computed on the default test set
provided with the dataset, while for Named Entity Recognition the metrics have been computed with a 5-fold cross-validation



The metrics have been computed at the token level and macro-averaged over the classes.### Demo


You can try the model online (fine-tuned on named entity recognition) using this web app: URL### Quick usage### Limitations


This lightweight model is mainly trained on Wikipedia, so it's particularly suitable as an agile analyzer for large volumes of natively digital text
from the world wide web, written in a correct and fluent form (like wikis, web pages, news, etc.). However, it may show limitations when it comes to chaotic text, containing errors and slang expressions
(like social media posts) or when it comes to domain-specific text (like medical, financial or legal content).### References


[1] URL


[2] URL


[3] URL


[4] URL### License


The model is released under **Apache-2.0** license