
# DebV3: A French language model based on DeBERTa V3

**this a temporary model (paper in submission)**

DebV3, a French language model based on DeBERTa V3, which is a DeBerta V2 with ELECTRA style pretraining using the Replaced Token Detection (RTD) objective.
RTD uses a generator model, trained using the MLM objective, to replace masked tokens with plausible candidates, and a discriminator model trained to detect which tokens were replaced by the generator.
Usually the generator and discriminator share the same embedding matrix, but the authors of DeBERTa V3 propose a new technique to disentagle the gradients of the shared embedding between the generator and discriminator called gradient-disentangled embedding sharing (GDES)

*This the first publicly available implementation of DeBERTa V3, and the first publicly DeBERTaV3 model outside of the original Microsoft release.*

## How to use DebV3
Our pretrained weights are available on the HuggingFace model hub, you can load them using the following code:

```python
from transformers import AutoTokenizer, AutoModel, AutoModelForMaskedLM

debv3 = AutoModel.from_pretrained("anondeb/debv3-base")
tokenizer = AutoTokenizer.from_pretrained("anondeb/debv3-base")

debv3_gen = AutoModelForMaskedLM.from_pretrained("anondeb/debv3-base-generator")
tokenizer_gen = AutoTokenizer.from_pretrained("anondeb/debv3-base-generator")
```

We also include the TF2 weights including the weights for the model's RTD head for the discriminator, and the MLM head for the generator.
debv3 is compatible with most finetuning scripts from the transformers library.

## Pretraining Setup

Code: TODO

The model was trained on the French subset of the CCNet corpus (the same subset used in CamemBERT and PaGNOL) and is available on the HuggingFace model hub: debv3 and debv3 Generator.
To speed up the pre-training experiments, the pre-training was split into two phases;
in phase 1, the model is trained with a maximum sequence length of 128 tokens for 10,000 steps with 2,000 warm-up steps and a very large batch size of 67,584.
In phase 2, maximum sequence length is increased to the full model capacity of 512 tokens for 3,300 steps with 200 warm-up steps and a batch size of 27,648.
The model would have seen 133B tokens compared to 419B tokens for CamemBERT-CCNet which was trained for 100K steps, this represents roughly 30% of CamemBERT’s full training.
To have a fair comparison, we trained a RoBERTa model, CamemBERT30%, using the same exact pretraining setup but with the MLM objective.

## Pretraining Loss Curves
check the tensorboard logs and plots

## Fine-tuning results

Datasets: POS tagging and Dependency Parsing (GSD, Rhapsodie, Sequoia, FSMB), NER (FTB), the FLUE benchmark (XNLI, CLS, PAWS-X), and the French Question Answering Dataset (FQuAD)

| Model             | UPOS      | LAS       | NER       | CLS       | PAWS-X    | XNLI      | F1 (FQuAD) | EM (FQuAD) |
|-------------------|-----------|-----------|-----------|-----------|-----------|-----------|------------|------------|
| CamemBERT (CCNet) | **97.59** | **88.69** | 89.97     | 94.62     | 91.36     | 81.95     | 80.98      | **62.51**  |
| CamemBERT (30%)   | 97.53     | 87.98     | **91.04** | 93.28     | 88.94     | 79.89     | 75.14      | 56.19      |
| debv3        | 97.57     | 88.55     | 90.33     | **94.92** | **91.67** | **82.00** | **81.15**  | 62.01      |

The following table compares debv3's performance on XNLI against other models under different training setups, which demonstrates the data efficiency of debv3.


| Model             | XNLI (Acc.) | Training Steps | Tokens seen in pre-training | Dataset Size in Tokens |
|-------------------|-------------|----------------|-----------------------------|------------------------|
| mDeBERTa          | 84.4        | 500k           | 2T                          | 2.5T                   |
| debv3        | 82.0        | 33k            | 0.139T                      | 0.319T                 |
| XLM-R             | 81.4        | 1.5M           | 6T                          | 2.5T                   |
| CamemBERT - CCNet | 81.95       | 100k           | 0.419T                      | 0.319T                 |

*Note: The debv3 training steps was adjusted for a batch size of 8192.*

## License

The public model weights are licensed under MIT License.
This code is licensed under the Apache License 2.0.

## Citation

Paper under review, will update this section when the paper is published.
{"license": "mit"}
transformers pytorch tensorboard deberta-v2 feature-extraction license:mit endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #deberta-v2 #feature-extraction #license-mit #endpoints_compatible #region-us 

DebV3: A French language model based on DeBERTa V3
==================================================


this a temporary model (paper in submission)


DebV3, a French language model based on DeBERTa V3, which is a DeBerta V2 with ELECTRA style pretraining using the Replaced Token Detection (RTD) objective.
RTD uses a generator model, trained using the MLM objective, to replace masked tokens with plausible candidates, and a discriminator model trained to detect which tokens were replaced by the generator.
Usually the generator and discriminator share the same embedding matrix, but the authors of DeBERTa V3 propose a new technique to disentagle the gradients of the shared embedding between the generator and discriminator called gradient-disentangled embedding sharing (GDES)


*This the first publicly available implementation of DeBERTa V3, and the first publicly DeBERTaV3 model outside of the original Microsoft release.*


How to use DebV3
----------------


Our pretrained weights are available on the HuggingFace model hub, you can load them using the following code:


We also include the TF2 weights including the weights for the model's RTD head for the discriminator, and the MLM head for the generator.
debv3 is compatible with most finetuning scripts from the transformers library.


Pretraining Setup
-----------------


Code: TODO


The model was trained on the French subset of the CCNet corpus (the same subset used in CamemBERT and PaGNOL) and is available on the HuggingFace model hub: debv3 and debv3 Generator.
To speed up the pre-training experiments, the pre-training was split into two phases;
in phase 1, the model is trained with a maximum sequence length of 128 tokens for 10,000 steps with 2,000 warm-up steps and a very large batch size of 67,584.
In phase 2, maximum sequence length is increased to the full model capacity of 512 tokens for 3,300 steps with 200 warm-up steps and a batch size of 27,648.
The model would have seen 133B tokens compared to 419B tokens for CamemBERT-CCNet which was trained for 100K steps, this represents roughly 30% of CamemBERT’s full training.
To have a fair comparison, we trained a RoBERTa model, CamemBERT30%, using the same exact pretraining setup but with the MLM objective.


Pretraining Loss Curves
-----------------------


check the tensorboard logs and plots


Fine-tuning results
-------------------


Datasets: POS tagging and Dependency Parsing (GSD, Rhapsodie, Sequoia, FSMB), NER (FTB), the FLUE benchmark (XNLI, CLS, PAWS-X), and the French Question Answering Dataset (FQuAD)



The following table compares debv3's performance on XNLI against other models under different training setups, which demonstrates the data efficiency of debv3.



*Note: The debv3 training steps was adjusted for a batch size of 8192.*


License
-------


The public model weights are licensed under MIT License.
This code is licensed under the Apache License 2.0.


Paper under review, will update this section when the paper is published.



TAGS
#transformers #pytorch #tensorboard #deberta-v2 #feature-extraction #license-mit #endpoints_compatible #region-us 

TAGS
#transformers #pytorch #tensorboard #deberta-v2 #feature-extraction #license-mit #endpoints_compatible #region-us 
