
# XLM-ROBERTA-BASE-XNLI

## Model description 
This model takes the XLM-Roberta-base model which has been continued to pre-traine on a large corpus of Twitter in multiple languages.   
It was developed following a similar strategy as introduced as part of the [Tweet Eval](https://github.com/cardiffnlp/tweeteval) framework.   
The model is further finetuned on all of the languages of the XNLI train set

## Intended Usage

This model was developed to do Zero-Shot Text Classification in the realm of Hate Speech Detection. It is finetuned on the whole xnli train set containing 15 different languages like:  
**ar, bg ,de , en, el , es, fr, hi, ru, sw, th, tr, ur, vi, zh**  
Since the base model was pre-trained on 100 different languages it has shown some effectiveness in other languages. Please refer to the list of languages in the [XLM Roberta paper](https://arxiv.org/abs/1911.02116)

### Usage with Zero-Shot Classification pipeline
```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                      model="morit/XLM-T-full-xnli")
```

## Training 
This model was pre-trained on a set of 100 languages and follwed further training on 198M multilingual tweets as described in the original [paper](https://arxiv.org/abs/2104.12250). Further it was trained on the full train set of XNLI dataset which is a machine translated version of the MNLI dataset. It was trained on 5 epochs of the XNLI train set and evaluated on the XNLI eval dataset at the end of every epoch to find the best performing model. The model which had the highest accuracy on the eval set was chosen at the end.

![Training Charts from wandb](screen_wandb.png)
- learning rate:    2e-5
- batch size:     32
- max sequence:   length 128

using a GPU (NVIDIA GeForce RTX 3090)

# Evaluation 
The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies: 

|  ar |  bg  | de | el |  en | es |  fr |  hi|  ru | sw | th | tr |ur | vi | zh |
|-----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|
| 0.749 | 0.787 | 0.774 | 0.774 | 0.831 | 0.796 | 0.785 | 0.734 | 0.761 | 0.701 | 0.757 | 0.758 | 0.704 | 0.778 | 0.774 |


{"language": ["de"], "license": "mit", "datasets": ["xnli"], "metrics": ["accuracy"], "pipeline_tag": "zero-shot-classification"}
transformers pytorch xlm-roberta text-classification zero-shot-classification de dataset:xnli arxiv:1911.02116 arxiv:2104.12250 license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #xlm-roberta #text-classification #zero-shot-classification #de #dataset-xnli #arxiv-1911.02116 #arxiv-2104.12250 #license-mit #autotrain_compatible #endpoints_compatible #region-us 

XLM-ROBERTA-BASE-XNLI
=====================


Model description
-----------------


This model takes the XLM-Roberta-base model which has been continued to pre-traine on a large corpus of Twitter in multiple languages.  

It was developed following a similar strategy as introduced as part of the Tweet Eval framework.  

The model is further finetuned on all of the languages of the XNLI train set


Intended Usage
--------------


This model was developed to do Zero-Shot Text Classification in the realm of Hate Speech Detection. It is finetuned on the whole xnli train set containing 15 different languages like:  

ar, bg ,de , en, el , es, fr, hi, ru, sw, th, tr, ur, vi, zh  

Since the base model was pre-trained on 100 different languages it has shown some effectiveness in other languages. Please refer to the list of languages in the XLM Roberta paper


### Usage with Zero-Shot Classification pipeline


Training
--------


This model was pre-trained on a set of 100 languages and follwed further training on 198M multilingual tweets as described in the original paper. Further it was trained on the full train set of XNLI dataset which is a machine translated version of the MNLI dataset. It was trained on 5 epochs of the XNLI train set and evaluated on the XNLI eval dataset at the end of every epoch to find the best performing model. The model which had the highest accuracy on the eval set was chosen at the end.


!Training Charts from wandb


* learning rate: 2e-5
* batch size: 32
* max sequence: length 128


using a GPU (NVIDIA GeForce RTX 3090)


Evaluation
==========


The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies:




### Usage with Zero-Shot Classification pipeline


Training
--------


This model was pre-trained on a set of 100 languages and follwed further training on 198M multilingual tweets as described in the original paper. Further it was trained on the full train set of XNLI dataset which is a machine translated version of the MNLI dataset. It was trained on 5 epochs of the XNLI train set and evaluated on the XNLI eval dataset at the end of every epoch to find the best performing model. The model which had the highest accuracy on the eval set was chosen at the end.


!Training Charts from wandb


* learning rate: 2e-5
* batch size: 32
* max sequence: length 128


using a GPU (NVIDIA GeForce RTX 3090)


Evaluation
==========


The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies:
TAGS
#transformers #pytorch #xlm-roberta #text-classification #zero-shot-classification #de #dataset-xnli #arxiv-1911.02116 #arxiv-2104.12250 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 ### Usage with Zero-Shot Classification pipeline


Training
--------


This model was pre-trained on a set of 100 languages and follwed further training on 198M multilingual tweets as described in the original paper. Further it was trained on the full train set of XNLI dataset which is a machine translated version of the MNLI dataset. It was trained on 5 epochs of the XNLI train set and evaluated on the XNLI eval dataset at the end of every epoch to find the best performing model. The model which had the highest accuracy on the eval set was chosen at the end.


!Training Charts from wandb


* learning rate: 2e-5
* batch size: 32
* max sequence: length 128


using a GPU (NVIDIA GeForce RTX 3090)


Evaluation
==========


The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies:
TAGS
#transformers #pytorch #xlm-roberta #text-classification #zero-shot-classification #de #dataset-xnli #arxiv-1911.02116 #arxiv-2104.12250 #license-mit #autotrain_compatible #endpoints_compatible #region-us 
### Usage with Zero-Shot Classification pipeline


Training
--------


This model was pre-trained on a set of 100 languages and follwed further training on 198M multilingual tweets as described in the original paper. Further it was trained on the full train set of XNLI dataset which is a machine translated version of the MNLI dataset. It was trained on 5 epochs of the XNLI train set and evaluated on the XNLI eval dataset at the end of every epoch to find the best performing model. The model which had the highest accuracy on the eval set was chosen at the end.


!Training Charts from wandb


* learning rate: 2e-5
* batch size: 32
* max sequence: length 128


using a GPU (NVIDIA GeForce RTX 3090)


Evaluation
==========


The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies: