# Vietnamese DebertaV3 Large (vi-deberta-v3-large)
### Todo
```
 [x] Corpora collection
 [x] Tokenizer training
 [x] Model pretraining
 [ ] Model finetuning
 [ ] Experimental results, comparision, and conclusion
```

## Model Info
```
LAYER NAME                      #PARAMS      RATIO       MEM(MB)
--model:                    851,542,017    100.00%       3248.38
  --generator:              284,459,008     33.41%       1085.12
    --deberta:              283,279,360     33.27%       1080.62
    --lm_predictions:         1,179,648      0.14%          4.50
  --discriminator:          567,083,009     66.59%       2163.25
    --deberta:              566,030,336     66.47%       2159.23
    --mask_predictions:       1,052,673      0.12%          4.02
```

## Model Perfomance

| Metric       | Value                      |
|--------------|----------------------------|
| accuracy     | 0.7113977778702509         |
| eval_loss    | 1.3216993808746338         |
| eval_metric  | 0.7113977778702509         |
| eval_samples | 240310                     |
| perplexity   | 3.749788284301758          |
| best_metric  | 0.7113977778702509@2200000 |
| train_steps  | 2200000                    |
| train_loss   | 1.1960319969044906          |


## TL;DR

| Aspect      | Sub-Aspect   | Description                                      |
|-------------|--------------|--------------------------------------------------|
| Corpus      | Language     | Vietnames                                        |
|             | Source       | Wiki 2023 (1GB), News 2023 (17.2GB), News (64GB) |
|             | Size         | 1GB, 18GB, 64GB                                  |
|             | Preprocesing | None                                             |
| Tokenizer   | Lib          | SentencePiece                                    |
|             | Algorithm    | BPE                                              |
|             | Type         | spm                                              |
|             | Vocab        | 128000                                           |
|             | Ref          | https://github.com/google/sentencepiece          |
| Model       | Type         | DeBERTaV3                                        |
|             | Ref          | https://openreview.net/forum?id=sE7-XhLxHA       |
|             | Code         | https://github.com/microsoft/DeBERTa             |
| Pretraining | Task         | RTD                                              |
|             | Config       | model_config.json                                |
|             | Args         | default                                          |
|             | Hardware     | 5x Nvidia A100-SXM4-80G, 2x Nvidia 4090-PCI-24GB |
|             | Phases       | Init, Refining, Enlarging            |
|             | Status       | Training on hold step 2200000             |
| Finetuning  | Status       | Not started (need help)                          |


## Repo
```
📁vi-deberta-v3-large
  |---🗎config.json
  |---🗎pytorch_model.bin
  |---🗎spm.model
  |---tl;dr.pdf
  |---📁discriminator
  |---📁generator
  |---📁tokenizer
  |---📁metrics
  |---📁logs
```

### Pretraining

<!-- START-OF PHASE 0 -->
<details>
  <summary>Phase 0: Init</summary>

### Info
  - Goal: Init
  - Progress: 30.00%  ▓▓▓▓░░░░░░
  - Status: training interrupped, step 1000000
  - Loss: <a href="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/raw/main/logs/loss.html" download>**init loss**</a>

### Metrics
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/train_eval_losses.png">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/losses.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/eval_losses.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/accuracies.png"  width="648">  
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grads.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grad_exp_xscale_log.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/perplexities.png"  width="648">

</details>
<!-- END-OF PHASE 0 -->

<!-- START-OF PHASE 1 -->
<details>
  <summary>Phase 1: Refining</summary>

### Info
  - Goal: refining
  - Changes: smaller learning rate (100µ -> 2µ)
  - Progress: 100.00%  ▓▓▓▓▓▓▓▓▓▓
  - Status: training finished, step 1500000
  - Loss: <a href="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/raw/main/logs/loss_refining.html" download>**refining loss**</a>
  
### Metrics

| Metric       | Value                      |
|--------------|----------------------------|
| accuracy     | 0.7515653334245732         |
| eval_loss    | 1.0692176818847656          |
| eval_metric  | 0.7515653334245732         |
| eval_samples | 29227                      |
| perplexity   | 2.913099527359009         |
| best_metric  | 0.7522154172511719@1450000 |
| train_steps  | 1500000 |
| train_loss  | 1.1779516744723688 |

<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/train_eval_losses_refining.png">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/losses_refining.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/eval_losses_refining.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/accuracies_refining.png"  width="648">  
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grads_refining.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grad_exp_xscale_log_refining.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/perplexities_refining.png"  width="648">

</details>
<!-- END-OF PHASE 1 -->

<!-- START-OF PHASE 2 -->
<details>
  <summary>Phase 2: Enlarging</summary>

### Info

  - Goal: enlarging, augmenting, expanding data
  - Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
  - Progress: 20.00%  ▓▓░░░░░░░░
  - Status: training in progress, step 2000000
  - Loss: <a href="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/raw/main/logs/loss_enlarging.html" download>**enlarging loss**</a>
    
### Metrics
| Metric       | Value                      |
|--------------|----------------------------|
| accuracy     | 0.7084723898298032         |
| eval_loss    | 1.3221531009674072          |
| eval_metric  | 0.7084723898298032         |
| eval_samples | 240310                      |
| perplexity   | 3.7141621112823486         |
| best_metric  | 0.7084723898298032@2000000 |
| train_steps  | 2000000 |
| train_loss  | 1.2167873119241372 |

<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/train_eval_losses_enlarging.png">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/losses_enlarging.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/eval_losses_enlarging.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/accuracies_enlarging.png"  width="648">  
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grads_enlarging.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grad_exp_xscale_log_enlarging.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/perplexities_enlarging.png"  width="648">

</details>
<!-- END-OF PHASE 2 -->


#### Phase 2: Enlarging (resume, on hold)
  - Goal: enlarging, augmenting, expanding data
  - Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
  - Progress: 20.00%  ▓▓░░░░░░░░
  - Status: training phase 2 interrupted intentionally, step 2200000
  - Loss: <a href="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/raw/main/logs/loss_enlarging_2.html" download>**enlarging loss**</a>
  - Metrics:
    
| Metric       | Value                      |
|--------------|----------------------------|
| accuracy     | 0.7113977778702509         |
| eval_loss    | 1.3216993808746338          |
| eval_metric  | 0.7113977778702509         |
| eval_samples | 240310                      |
| perplexity   | 3.749788284301758         |
| best_metric  | 0.7113977778702509@2200000 |
| train_steps  | 2200000 |
| train_loss  | 1.1960319969044906 |

<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/train_eval_losses_enlarging_2.png">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/losses_enlarging_2.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/eval_losses_enlarging_2.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/accuracies_enlarging_2.png"  width="648">  
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grads_enlarging_2.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/grad_exp_xscale_log_enlarging_2.png"  width="648">
<img src="https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/metrics/perplexities_enlarging_2.png"  width="648">


### Finetuning **NEED HELP**
  - Token Classification: tasks, datasets
  - Sequence Classification: tasks, datasets

## Experimental Results and Comparision
  - Not started
    
## Usage

![test](https://huggingface.co/anhdungitvn/vi-deberta-v3-large/resolve/main/test.gif)

#### Method 1: Load pretrained vi-deberta-v3-large with Transformers AutoClass
  - Install ClickAI
    ```
    # Why need to install clickai?
    # HF Transformers has not yet supported DebertaV3.
    # ClickAI: locally register DeBertaV3 with HF Transformers.
    ```
          
    ```bash
    pip install git+https://gitlab.com/anhdungvo/clickai.git    
  
    ```
    
    
  - Tokenizer, Config, Model
    ```python
    import clickai  
    from transformers import AutoTokenizer, AutoConfig, AutoModel
    
    config = AutoConfig.from_pretrained("anhdungitvn/vi-deberta-v3-large")
    tokenizer = AutoTokenizer.from_pretrained("anhdungitvn/vi-deberta-v3-large")
    model = AutoModel.from_pretrained("anhdungitvn/vi-deberta-v3-large")

    tokenizer("Xử lý ngôn ngữ tiếng Việt", return_tensors='pt')
    ```

  - Transfer Pretrained Model Weights
    ```python
    your_model.load_state_dict(model.GET_NEEDED_MODULE_WEIGHTS())
    ```
    
<!-- START-OF MODEL STAGE DICT KEYS -->
<details>
  <summary>Model state dict  keys</summary>

  
  ```python
  model.state_dict().keys()
  ```
  
  ```python
  [
	generator.deberta.embeddings.word_embeddings.weight
	generator.deberta.embeddings.position_embeddings.weight
	generator.deberta.embeddings.LayerNorm.weight
	generator.deberta.embeddings.LayerNorm.bias
	generator.deberta.encoder.layer.0.attention.self.query_proj.weight
	generator.deberta.encoder.layer.0.attention.self.query_proj.bias
	generator.deberta.encoder.layer.0.attention.self.key_proj.weight
	generator.deberta.encoder.layer.0.attention.self.key_proj.bias
	generator.deberta.encoder.layer.0.attention.self.value_proj.weight
	generator.deberta.encoder.layer.0.attention.self.value_proj.bias
	generator.deberta.encoder.layer.0.attention.output.dense.weight
	generator.deberta.encoder.layer.0.attention.output.dense.bias
	generator.deberta.encoder.layer.0.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.0.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.0.intermediate.dense.weight
	generator.deberta.encoder.layer.0.intermediate.dense.bias
	generator.deberta.encoder.layer.0.output.dense.weight
	generator.deberta.encoder.layer.0.output.dense.bias
	generator.deberta.encoder.layer.0.output.LayerNorm.weight
	generator.deberta.encoder.layer.0.output.LayerNorm.bias
	generator.deberta.encoder.layer.1.attention.self.query_proj.weight
	generator.deberta.encoder.layer.1.attention.self.query_proj.bias
	generator.deberta.encoder.layer.1.attention.self.key_proj.weight
	generator.deberta.encoder.layer.1.attention.self.key_proj.bias
	generator.deberta.encoder.layer.1.attention.self.value_proj.weight
	generator.deberta.encoder.layer.1.attention.self.value_proj.bias
	generator.deberta.encoder.layer.1.attention.output.dense.weight
	generator.deberta.encoder.layer.1.attention.output.dense.bias
	generator.deberta.encoder.layer.1.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.1.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.1.intermediate.dense.weight
	generator.deberta.encoder.layer.1.intermediate.dense.bias
	generator.deberta.encoder.layer.1.output.dense.weight
	generator.deberta.encoder.layer.1.output.dense.bias
	generator.deberta.encoder.layer.1.output.LayerNorm.weight
	generator.deberta.encoder.layer.1.output.LayerNorm.bias
	generator.deberta.encoder.layer.2.attention.self.query_proj.weight
	generator.deberta.encoder.layer.2.attention.self.query_proj.bias
	generator.deberta.encoder.layer.2.attention.self.key_proj.weight
	generator.deberta.encoder.layer.2.attention.self.key_proj.bias
	generator.deberta.encoder.layer.2.attention.self.value_proj.weight
	generator.deberta.encoder.layer.2.attention.self.value_proj.bias
	generator.deberta.encoder.layer.2.attention.output.dense.weight
	generator.deberta.encoder.layer.2.attention.output.dense.bias
	generator.deberta.encoder.layer.2.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.2.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.2.intermediate.dense.weight
	generator.deberta.encoder.layer.2.intermediate.dense.bias
	generator.deberta.encoder.layer.2.output.dense.weight
	generator.deberta.encoder.layer.2.output.dense.bias
	generator.deberta.encoder.layer.2.output.LayerNorm.weight
	generator.deberta.encoder.layer.2.output.LayerNorm.bias
	generator.deberta.encoder.layer.3.attention.self.query_proj.weight
	generator.deberta.encoder.layer.3.attention.self.query_proj.bias
	generator.deberta.encoder.layer.3.attention.self.key_proj.weight
	generator.deberta.encoder.layer.3.attention.self.key_proj.bias
	generator.deberta.encoder.layer.3.attention.self.value_proj.weight
	generator.deberta.encoder.layer.3.attention.self.value_proj.bias
	generator.deberta.encoder.layer.3.attention.output.dense.weight
	generator.deberta.encoder.layer.3.attention.output.dense.bias
	generator.deberta.encoder.layer.3.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.3.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.3.intermediate.dense.weight
	generator.deberta.encoder.layer.3.intermediate.dense.bias
	generator.deberta.encoder.layer.3.output.dense.weight
	generator.deberta.encoder.layer.3.output.dense.bias
	generator.deberta.encoder.layer.3.output.LayerNorm.weight
	generator.deberta.encoder.layer.3.output.LayerNorm.bias
	generator.deberta.encoder.layer.4.attention.self.query_proj.weight
	generator.deberta.encoder.layer.4.attention.self.query_proj.bias
	generator.deberta.encoder.layer.4.attention.self.key_proj.weight
	generator.deberta.encoder.layer.4.attention.self.key_proj.bias
	generator.deberta.encoder.layer.4.attention.self.value_proj.weight
	generator.deberta.encoder.layer.4.attention.self.value_proj.bias
	generator.deberta.encoder.layer.4.attention.output.dense.weight
	generator.deberta.encoder.layer.4.attention.output.dense.bias
	generator.deberta.encoder.layer.4.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.4.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.4.intermediate.dense.weight
	generator.deberta.encoder.layer.4.intermediate.dense.bias
	generator.deberta.encoder.layer.4.output.dense.weight
	generator.deberta.encoder.layer.4.output.dense.bias
	generator.deberta.encoder.layer.4.output.LayerNorm.weight
	generator.deberta.encoder.layer.4.output.LayerNorm.bias
	generator.deberta.encoder.layer.5.attention.self.query_proj.weight
	generator.deberta.encoder.layer.5.attention.self.query_proj.bias
	generator.deberta.encoder.layer.5.attention.self.key_proj.weight
	generator.deberta.encoder.layer.5.attention.self.key_proj.bias
	generator.deberta.encoder.layer.5.attention.self.value_proj.weight
	generator.deberta.encoder.layer.5.attention.self.value_proj.bias
	generator.deberta.encoder.layer.5.attention.output.dense.weight
	generator.deberta.encoder.layer.5.attention.output.dense.bias
	generator.deberta.encoder.layer.5.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.5.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.5.intermediate.dense.weight
	generator.deberta.encoder.layer.5.intermediate.dense.bias
	generator.deberta.encoder.layer.5.output.dense.weight
	generator.deberta.encoder.layer.5.output.dense.bias
	generator.deberta.encoder.layer.5.output.LayerNorm.weight
	generator.deberta.encoder.layer.5.output.LayerNorm.bias
	generator.deberta.encoder.layer.6.attention.self.query_proj.weight
	generator.deberta.encoder.layer.6.attention.self.query_proj.bias
	generator.deberta.encoder.layer.6.attention.self.key_proj.weight
	generator.deberta.encoder.layer.6.attention.self.key_proj.bias
	generator.deberta.encoder.layer.6.attention.self.value_proj.weight
	generator.deberta.encoder.layer.6.attention.self.value_proj.bias
	generator.deberta.encoder.layer.6.attention.output.dense.weight
	generator.deberta.encoder.layer.6.attention.output.dense.bias
	generator.deberta.encoder.layer.6.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.6.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.6.intermediate.dense.weight
	generator.deberta.encoder.layer.6.intermediate.dense.bias
	generator.deberta.encoder.layer.6.output.dense.weight
	generator.deberta.encoder.layer.6.output.dense.bias
	generator.deberta.encoder.layer.6.output.LayerNorm.weight
	generator.deberta.encoder.layer.6.output.LayerNorm.bias
	generator.deberta.encoder.layer.7.attention.self.query_proj.weight
	generator.deberta.encoder.layer.7.attention.self.query_proj.bias
	generator.deberta.encoder.layer.7.attention.self.key_proj.weight
	generator.deberta.encoder.layer.7.attention.self.key_proj.bias
	generator.deberta.encoder.layer.7.attention.self.value_proj.weight
	generator.deberta.encoder.layer.7.attention.self.value_proj.bias
	generator.deberta.encoder.layer.7.attention.output.dense.weight
	generator.deberta.encoder.layer.7.attention.output.dense.bias
	generator.deberta.encoder.layer.7.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.7.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.7.intermediate.dense.weight
	generator.deberta.encoder.layer.7.intermediate.dense.bias
	generator.deberta.encoder.layer.7.output.dense.weight
	generator.deberta.encoder.layer.7.output.dense.bias
	generator.deberta.encoder.layer.7.output.LayerNorm.weight
	generator.deberta.encoder.layer.7.output.LayerNorm.bias
	generator.deberta.encoder.layer.8.attention.self.query_proj.weight
	generator.deberta.encoder.layer.8.attention.self.query_proj.bias
	generator.deberta.encoder.layer.8.attention.self.key_proj.weight
	generator.deberta.encoder.layer.8.attention.self.key_proj.bias
	generator.deberta.encoder.layer.8.attention.self.value_proj.weight
	generator.deberta.encoder.layer.8.attention.self.value_proj.bias
	generator.deberta.encoder.layer.8.attention.output.dense.weight
	generator.deberta.encoder.layer.8.attention.output.dense.bias
	generator.deberta.encoder.layer.8.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.8.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.8.intermediate.dense.weight
	generator.deberta.encoder.layer.8.intermediate.dense.bias
	generator.deberta.encoder.layer.8.output.dense.weight
	generator.deberta.encoder.layer.8.output.dense.bias
	generator.deberta.encoder.layer.8.output.LayerNorm.weight
	generator.deberta.encoder.layer.8.output.LayerNorm.bias
	generator.deberta.encoder.layer.9.attention.self.query_proj.weight
	generator.deberta.encoder.layer.9.attention.self.query_proj.bias
	generator.deberta.encoder.layer.9.attention.self.key_proj.weight
	generator.deberta.encoder.layer.9.attention.self.key_proj.bias
	generator.deberta.encoder.layer.9.attention.self.value_proj.weight
	generator.deberta.encoder.layer.9.attention.self.value_proj.bias
	generator.deberta.encoder.layer.9.attention.output.dense.weight
	generator.deberta.encoder.layer.9.attention.output.dense.bias
	generator.deberta.encoder.layer.9.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.9.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.9.intermediate.dense.weight
	generator.deberta.encoder.layer.9.intermediate.dense.bias
	generator.deberta.encoder.layer.9.output.dense.weight
	generator.deberta.encoder.layer.9.output.dense.bias
	generator.deberta.encoder.layer.9.output.LayerNorm.weight
	generator.deberta.encoder.layer.9.output.LayerNorm.bias
	generator.deberta.encoder.layer.10.attention.self.query_proj.weight
	generator.deberta.encoder.layer.10.attention.self.query_proj.bias
	generator.deberta.encoder.layer.10.attention.self.key_proj.weight
	generator.deberta.encoder.layer.10.attention.self.key_proj.bias
	generator.deberta.encoder.layer.10.attention.self.value_proj.weight
	generator.deberta.encoder.layer.10.attention.self.value_proj.bias
	generator.deberta.encoder.layer.10.attention.output.dense.weight
	generator.deberta.encoder.layer.10.attention.output.dense.bias
	generator.deberta.encoder.layer.10.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.10.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.10.intermediate.dense.weight
	generator.deberta.encoder.layer.10.intermediate.dense.bias
	generator.deberta.encoder.layer.10.output.dense.weight
	generator.deberta.encoder.layer.10.output.dense.bias
	generator.deberta.encoder.layer.10.output.LayerNorm.weight
	generator.deberta.encoder.layer.10.output.LayerNorm.bias
	generator.deberta.encoder.layer.11.attention.self.query_proj.weight
	generator.deberta.encoder.layer.11.attention.self.query_proj.bias
	generator.deberta.encoder.layer.11.attention.self.key_proj.weight
	generator.deberta.encoder.layer.11.attention.self.key_proj.bias
	generator.deberta.encoder.layer.11.attention.self.value_proj.weight
	generator.deberta.encoder.layer.11.attention.self.value_proj.bias
	generator.deberta.encoder.layer.11.attention.output.dense.weight
	generator.deberta.encoder.layer.11.attention.output.dense.bias
	generator.deberta.encoder.layer.11.attention.output.LayerNorm.weight
	generator.deberta.encoder.layer.11.attention.output.LayerNorm.bias
	generator.deberta.encoder.layer.11.intermediate.dense.weight
	generator.deberta.encoder.layer.11.intermediate.dense.bias
	generator.deberta.encoder.layer.11.output.dense.weight
	generator.deberta.encoder.layer.11.output.dense.bias
	generator.deberta.encoder.layer.11.output.LayerNorm.weight
	generator.deberta.encoder.layer.11.output.LayerNorm.bias
	generator.deberta.encoder.rel_embeddings.weight
	generator.deberta.encoder.LayerNorm.weight
	generator.deberta.encoder.LayerNorm.bias
	generator.lm_predictions.lm_head.bias
	generator.lm_predictions.lm_head.dense.weight
	generator.lm_predictions.lm_head.dense.bias
	generator.lm_predictions.lm_head.LayerNorm.weight
	generator.lm_predictions.lm_head.LayerNorm.bias
	discriminator.deberta.embeddings.word_embeddings.weight
	discriminator.deberta.embeddings.word_embeddings._weight
	discriminator.deberta.embeddings.position_embeddings.weight
	discriminator.deberta.embeddings.position_embeddings._weight
	discriminator.deberta.embeddings.LayerNorm.weight
	discriminator.deberta.embeddings.LayerNorm.bias
	discriminator.deberta.encoder.layer.0.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.0.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.0.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.0.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.0.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.0.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.0.attention.output.dense.weight
	discriminator.deberta.encoder.layer.0.attention.output.dense.bias
	discriminator.deberta.encoder.layer.0.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.0.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.0.intermediate.dense.weight
	discriminator.deberta.encoder.layer.0.intermediate.dense.bias
	discriminator.deberta.encoder.layer.0.output.dense.weight
	discriminator.deberta.encoder.layer.0.output.dense.bias
	discriminator.deberta.encoder.layer.0.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.0.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.1.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.1.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.1.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.1.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.1.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.1.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.1.attention.output.dense.weight
	discriminator.deberta.encoder.layer.1.attention.output.dense.bias
	discriminator.deberta.encoder.layer.1.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.1.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.1.intermediate.dense.weight
	discriminator.deberta.encoder.layer.1.intermediate.dense.bias
	discriminator.deberta.encoder.layer.1.output.dense.weight
	discriminator.deberta.encoder.layer.1.output.dense.bias
	discriminator.deberta.encoder.layer.1.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.1.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.2.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.2.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.2.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.2.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.2.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.2.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.2.attention.output.dense.weight
	discriminator.deberta.encoder.layer.2.attention.output.dense.bias
	discriminator.deberta.encoder.layer.2.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.2.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.2.intermediate.dense.weight
	discriminator.deberta.encoder.layer.2.intermediate.dense.bias
	discriminator.deberta.encoder.layer.2.output.dense.weight
	discriminator.deberta.encoder.layer.2.output.dense.bias
	discriminator.deberta.encoder.layer.2.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.2.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.3.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.3.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.3.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.3.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.3.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.3.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.3.attention.output.dense.weight
	discriminator.deberta.encoder.layer.3.attention.output.dense.bias
	discriminator.deberta.encoder.layer.3.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.3.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.3.intermediate.dense.weight
	discriminator.deberta.encoder.layer.3.intermediate.dense.bias
	discriminator.deberta.encoder.layer.3.output.dense.weight
	discriminator.deberta.encoder.layer.3.output.dense.bias
	discriminator.deberta.encoder.layer.3.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.3.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.4.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.4.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.4.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.4.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.4.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.4.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.4.attention.output.dense.weight
	discriminator.deberta.encoder.layer.4.attention.output.dense.bias
	discriminator.deberta.encoder.layer.4.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.4.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.4.intermediate.dense.weight
	discriminator.deberta.encoder.layer.4.intermediate.dense.bias
	discriminator.deberta.encoder.layer.4.output.dense.weight
	discriminator.deberta.encoder.layer.4.output.dense.bias
	discriminator.deberta.encoder.layer.4.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.4.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.5.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.5.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.5.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.5.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.5.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.5.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.5.attention.output.dense.weight
	discriminator.deberta.encoder.layer.5.attention.output.dense.bias
	discriminator.deberta.encoder.layer.5.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.5.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.5.intermediate.dense.weight
	discriminator.deberta.encoder.layer.5.intermediate.dense.bias
	discriminator.deberta.encoder.layer.5.output.dense.weight
	discriminator.deberta.encoder.layer.5.output.dense.bias
	discriminator.deberta.encoder.layer.5.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.5.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.6.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.6.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.6.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.6.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.6.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.6.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.6.attention.output.dense.weight
	discriminator.deberta.encoder.layer.6.attention.output.dense.bias
	discriminator.deberta.encoder.layer.6.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.6.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.6.intermediate.dense.weight
	discriminator.deberta.encoder.layer.6.intermediate.dense.bias
	discriminator.deberta.encoder.layer.6.output.dense.weight
	discriminator.deberta.encoder.layer.6.output.dense.bias
	discriminator.deberta.encoder.layer.6.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.6.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.7.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.7.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.7.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.7.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.7.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.7.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.7.attention.output.dense.weight
	discriminator.deberta.encoder.layer.7.attention.output.dense.bias
	discriminator.deberta.encoder.layer.7.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.7.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.7.intermediate.dense.weight
	discriminator.deberta.encoder.layer.7.intermediate.dense.bias
	discriminator.deberta.encoder.layer.7.output.dense.weight
	discriminator.deberta.encoder.layer.7.output.dense.bias
	discriminator.deberta.encoder.layer.7.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.7.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.8.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.8.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.8.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.8.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.8.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.8.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.8.attention.output.dense.weight
	discriminator.deberta.encoder.layer.8.attention.output.dense.bias
	discriminator.deberta.encoder.layer.8.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.8.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.8.intermediate.dense.weight
	discriminator.deberta.encoder.layer.8.intermediate.dense.bias
	discriminator.deberta.encoder.layer.8.output.dense.weight
	discriminator.deberta.encoder.layer.8.output.dense.bias
	discriminator.deberta.encoder.layer.8.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.8.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.9.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.9.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.9.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.9.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.9.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.9.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.9.attention.output.dense.weight
	discriminator.deberta.encoder.layer.9.attention.output.dense.bias
	discriminator.deberta.encoder.layer.9.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.9.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.9.intermediate.dense.weight
	discriminator.deberta.encoder.layer.9.intermediate.dense.bias
	discriminator.deberta.encoder.layer.9.output.dense.weight
	discriminator.deberta.encoder.layer.9.output.dense.bias
	discriminator.deberta.encoder.layer.9.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.9.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.10.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.10.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.10.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.10.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.10.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.10.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.10.attention.output.dense.weight
	discriminator.deberta.encoder.layer.10.attention.output.dense.bias
	discriminator.deberta.encoder.layer.10.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.10.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.10.intermediate.dense.weight
	discriminator.deberta.encoder.layer.10.intermediate.dense.bias
	discriminator.deberta.encoder.layer.10.output.dense.weight
	discriminator.deberta.encoder.layer.10.output.dense.bias
	discriminator.deberta.encoder.layer.10.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.10.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.11.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.11.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.11.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.11.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.11.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.11.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.11.attention.output.dense.weight
	discriminator.deberta.encoder.layer.11.attention.output.dense.bias
	discriminator.deberta.encoder.layer.11.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.11.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.11.intermediate.dense.weight
	discriminator.deberta.encoder.layer.11.intermediate.dense.bias
	discriminator.deberta.encoder.layer.11.output.dense.weight
	discriminator.deberta.encoder.layer.11.output.dense.bias
	discriminator.deberta.encoder.layer.11.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.11.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.12.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.12.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.12.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.12.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.12.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.12.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.12.attention.output.dense.weight
	discriminator.deberta.encoder.layer.12.attention.output.dense.bias
	discriminator.deberta.encoder.layer.12.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.12.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.12.intermediate.dense.weight
	discriminator.deberta.encoder.layer.12.intermediate.dense.bias
	discriminator.deberta.encoder.layer.12.output.dense.weight
	discriminator.deberta.encoder.layer.12.output.dense.bias
	discriminator.deberta.encoder.layer.12.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.12.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.13.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.13.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.13.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.13.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.13.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.13.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.13.attention.output.dense.weight
	discriminator.deberta.encoder.layer.13.attention.output.dense.bias
	discriminator.deberta.encoder.layer.13.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.13.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.13.intermediate.dense.weight
	discriminator.deberta.encoder.layer.13.intermediate.dense.bias
	discriminator.deberta.encoder.layer.13.output.dense.weight
	discriminator.deberta.encoder.layer.13.output.dense.bias
	discriminator.deberta.encoder.layer.13.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.13.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.14.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.14.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.14.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.14.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.14.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.14.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.14.attention.output.dense.weight
	discriminator.deberta.encoder.layer.14.attention.output.dense.bias
	discriminator.deberta.encoder.layer.14.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.14.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.14.intermediate.dense.weight
	discriminator.deberta.encoder.layer.14.intermediate.dense.bias
	discriminator.deberta.encoder.layer.14.output.dense.weight
	discriminator.deberta.encoder.layer.14.output.dense.bias
	discriminator.deberta.encoder.layer.14.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.14.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.15.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.15.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.15.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.15.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.15.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.15.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.15.attention.output.dense.weight
	discriminator.deberta.encoder.layer.15.attention.output.dense.bias
	discriminator.deberta.encoder.layer.15.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.15.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.15.intermediate.dense.weight
	discriminator.deberta.encoder.layer.15.intermediate.dense.bias
	discriminator.deberta.encoder.layer.15.output.dense.weight
	discriminator.deberta.encoder.layer.15.output.dense.bias
	discriminator.deberta.encoder.layer.15.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.15.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.16.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.16.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.16.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.16.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.16.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.16.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.16.attention.output.dense.weight
	discriminator.deberta.encoder.layer.16.attention.output.dense.bias
	discriminator.deberta.encoder.layer.16.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.16.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.16.intermediate.dense.weight
	discriminator.deberta.encoder.layer.16.intermediate.dense.bias
	discriminator.deberta.encoder.layer.16.output.dense.weight
	discriminator.deberta.encoder.layer.16.output.dense.bias
	discriminator.deberta.encoder.layer.16.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.16.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.17.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.17.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.17.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.17.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.17.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.17.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.17.attention.output.dense.weight
	discriminator.deberta.encoder.layer.17.attention.output.dense.bias
	discriminator.deberta.encoder.layer.17.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.17.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.17.intermediate.dense.weight
	discriminator.deberta.encoder.layer.17.intermediate.dense.bias
	discriminator.deberta.encoder.layer.17.output.dense.weight
	discriminator.deberta.encoder.layer.17.output.dense.bias
	discriminator.deberta.encoder.layer.17.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.17.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.18.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.18.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.18.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.18.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.18.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.18.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.18.attention.output.dense.weight
	discriminator.deberta.encoder.layer.18.attention.output.dense.bias
	discriminator.deberta.encoder.layer.18.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.18.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.18.intermediate.dense.weight
	discriminator.deberta.encoder.layer.18.intermediate.dense.bias
	discriminator.deberta.encoder.layer.18.output.dense.weight
	discriminator.deberta.encoder.layer.18.output.dense.bias
	discriminator.deberta.encoder.layer.18.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.18.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.19.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.19.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.19.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.19.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.19.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.19.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.19.attention.output.dense.weight
	discriminator.deberta.encoder.layer.19.attention.output.dense.bias
	discriminator.deberta.encoder.layer.19.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.19.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.19.intermediate.dense.weight
	discriminator.deberta.encoder.layer.19.intermediate.dense.bias
	discriminator.deberta.encoder.layer.19.output.dense.weight
	discriminator.deberta.encoder.layer.19.output.dense.bias
	discriminator.deberta.encoder.layer.19.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.19.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.20.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.20.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.20.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.20.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.20.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.20.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.20.attention.output.dense.weight
	discriminator.deberta.encoder.layer.20.attention.output.dense.bias
	discriminator.deberta.encoder.layer.20.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.20.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.20.intermediate.dense.weight
	discriminator.deberta.encoder.layer.20.intermediate.dense.bias
	discriminator.deberta.encoder.layer.20.output.dense.weight
	discriminator.deberta.encoder.layer.20.output.dense.bias
	discriminator.deberta.encoder.layer.20.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.20.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.21.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.21.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.21.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.21.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.21.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.21.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.21.attention.output.dense.weight
	discriminator.deberta.encoder.layer.21.attention.output.dense.bias
	discriminator.deberta.encoder.layer.21.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.21.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.21.intermediate.dense.weight
	discriminator.deberta.encoder.layer.21.intermediate.dense.bias
	discriminator.deberta.encoder.layer.21.output.dense.weight
	discriminator.deberta.encoder.layer.21.output.dense.bias
	discriminator.deberta.encoder.layer.21.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.21.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.22.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.22.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.22.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.22.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.22.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.22.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.22.attention.output.dense.weight
	discriminator.deberta.encoder.layer.22.attention.output.dense.bias
	discriminator.deberta.encoder.layer.22.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.22.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.22.intermediate.dense.weight
	discriminator.deberta.encoder.layer.22.intermediate.dense.bias
	discriminator.deberta.encoder.layer.22.output.dense.weight
	discriminator.deberta.encoder.layer.22.output.dense.bias
	discriminator.deberta.encoder.layer.22.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.22.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.23.attention.self.query_proj.weight
	discriminator.deberta.encoder.layer.23.attention.self.query_proj.bias
	discriminator.deberta.encoder.layer.23.attention.self.key_proj.weight
	discriminator.deberta.encoder.layer.23.attention.self.key_proj.bias
	discriminator.deberta.encoder.layer.23.attention.self.value_proj.weight
	discriminator.deberta.encoder.layer.23.attention.self.value_proj.bias
	discriminator.deberta.encoder.layer.23.attention.output.dense.weight
	discriminator.deberta.encoder.layer.23.attention.output.dense.bias
	discriminator.deberta.encoder.layer.23.attention.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.23.attention.output.LayerNorm.bias
	discriminator.deberta.encoder.layer.23.intermediate.dense.weight
	discriminator.deberta.encoder.layer.23.intermediate.dense.bias
	discriminator.deberta.encoder.layer.23.output.dense.weight
	discriminator.deberta.encoder.layer.23.output.dense.bias
	discriminator.deberta.encoder.layer.23.output.LayerNorm.weight
	discriminator.deberta.encoder.layer.23.output.LayerNorm.bias
	discriminator.deberta.encoder.rel_embeddings.weight
	discriminator.deberta.encoder.LayerNorm.weight
	discriminator.deberta.encoder.LayerNorm.bias
	discriminator.mask_predictions.dense.weight
	discriminator.mask_predictions.dense.bias
	discriminator.mask_predictions.LayerNorm.weight
	discriminator.mask_predictions.LayerNorm.bias
	discriminator.mask_predictions.classifier.weight
	discriminator.mask_predictions.classifier.bias
]
  
  ```

</details>
<!-- END-OF MODEL STAGE DICT KEYS -->


#### Method 2: Manually load pretrained vi-deberta-v3-large 
   
  - Dev YourTokenizer, Your Model
    ```python
    class YourTokenizer:
        @classmethod
        def from_pretrained(model_name_or_path, **kwargs):
            # https://huggingface.co/anhdungitvn/vi-deberta-v3-large/blob/main/tokenizer/spm.model
            pass

    class YourModel:
        @classmethod
        def from_pretrained(model_name_or_path, **kwargs):
            # Discriminator: https://huggingface.co/anhdungitvn/vi-deberta-v3-large/tree/main/discriminator
            # Generator: https://huggingface.co/anhdungitvn/vi-deberta-v3-large/tree/main/generator
            pass      
    ```
  - Use
    ```python
    tokenizer = YourTokenizer.from_pretrained("anhdungitvn/vi-deberta-v3-large")
    tokenizer = YourModel.from_pretrained("anhdungitvn/vi-deberta-v3-large")
    ```





### Log

<!-- START-OF LOG -->
<details>
  <summary>Details</summary>

Logs:
  - 2023-03-29: init, todolist
  - 2023-03-30: data preparation
    - vi_wiki_23: lastest
    - vi_news_17g: available
  - 2023-03-30: tokenizer training
    - algorithm: unigram, bpe
    - size: 8k, 16k, 32k, 64k, 128k, 256k
  - 2023-03-31: training trials
    - config: base, large
    - tokenizer: unigram_16k, bpe_16k, bpe_128k
    - args: default changed batch_size, grad_acc
    - optimizer: default, customized optimizer
  - 2023-03-31: training phase 0 started
    - config: large
    - tokenizer: bpe_128k
    - args: default
    - GPU: 5x A100-SXM4-80G
  - 2023-04-04: training interrupted unintentionally, optimizer checkpoint none, step 300000 
  - 2023-04-05: training resumed from lastcheckpoint 300000, learning_rate addjusted 100µ -> 50µ
  - 2023-04-10: sweet spot detected, step 800000
  - 2023-04-11: training in progress, step 900000, loss increases, accuracy increases, regularization being working well, overfitting problem under monitoring
  - 2023-04-12: training interrupted intentionally, step 1000000 
  - 2023-04-12: training phase 1 started, resumed intentionally, refining, learning_rate -> 2µ (diverging)
  - 2023-04-24: training phase 1 finished, refining, step 1500000
  - 2023-04-24: training phase 1 finished, refining, step 1500000
  - 2023-04-26: training phase 2 started, resumed intentionally, step 1500000
  - 2023-05-02: training phase 2 interrupted unintentionally, step 1980000
  - 2023-05-05: training phase 2 resumed intentionally, step 2000000
  - 2023-05-09: training in progress, step 2200000

</details>
<!-- END-OF LOG -->
{"language": ["vi"], "license": "other", "datasets": ["anhdungitvn/vi-general-64g"], "metrics": ["accuracy", "perplexity"]}
transformers pytorch deberta-v3 vi dataset:anhdungitvn/vi-general-64g license:other endpoints_compatible region:us
TAGS
#transformers #pytorch #deberta-v3 #vi #dataset-anhdungitvn/vi-general-64g #license-other #endpoints_compatible #region-us 

Vietnamese DebertaV3 Large (vi-deberta-v3-large)
================================================


### Todo


Model Info
----------


Model Perfomance
----------------



TL;DR
-----


Aspect: Corpus, Sub-Aspect: Language, Description: Vietnames
Aspect: , Sub-Aspect: Source, Description: Wiki 2023 (1GB), News 2023 (17.2GB), News (64GB)
Aspect: , Sub-Aspect: Size, Description: 1GB, 18GB, 64GB
Aspect: , Sub-Aspect: Preprocesing, Description: None
Aspect: Tokenizer, Sub-Aspect: Lib, Description: SentencePiece
Aspect: , Sub-Aspect: Algorithm, Description: BPE
Aspect: , Sub-Aspect: Type, Description: spm
Aspect: , Sub-Aspect: Vocab, Description: 128000
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: Model, Sub-Aspect: Type, Description: DeBERTaV3
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: , Sub-Aspect: Code, Description: URL
Aspect: Pretraining, Sub-Aspect: Task, Description: RTD
Aspect: , Sub-Aspect: Config, Description: model\_config.json
Aspect: , Sub-Aspect: Args, Description: default
Aspect: , Sub-Aspect: Hardware, Description: 5x Nvidia A100-SXM4-80G, 2x Nvidia 4090-PCI-24GB
Aspect: , Sub-Aspect: Phases, Description: Init, Refining, Enlarging
Aspect: , Sub-Aspect: Status, Description: Training on hold step 2200000
Aspect: Finetuning, Sub-Aspect: Status, Description: Not started (need help)


Repo
----


### Pretraining



Phase 0: Init
### Info


* Goal: Init
* Progress: 30.00% ▓▓▓▓░░░░░░
* Status: training interrupped, step 1000000
* Loss: <a href="URL download>init loss


### Metrics


<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 1: Refining
### Info


* Goal: refining
* Changes: smaller learning rate (100µ -> 2µ)
* Progress: 100.00% ▓▓▓▓▓▓▓▓▓▓
* Status: training finished, step 1500000
* Loss: <a href="URL download>refining loss


### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 2: Enlarging
### Info


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training in progress, step 2000000
* Loss: <a href="URL download>enlarging loss


### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">



#### Phase 2: Enlarging (resume, on hold)


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training phase 2 interrupted intentionally, step 2200000
* Loss: <a href="URL download>enlarging loss
* Metrics:



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">


### Finetuning NEED HELP


* Token Classification: tasks, datasets
* Sequence Classification: tasks, datasets


Experimental Results and Comparision
------------------------------------


* Not started


Usage
-----


!test


#### Method 1: Load pretrained vi-deberta-v3-large with Transformers AutoClass


* Install ClickAI
* Tokenizer, Config, Model
* Transfer Pretrained Model Weights



Model state dict keys

#### Method 2: Manually load pretrained vi-deberta-v3-large


* Dev YourTokenizer, Your Model
* Use


### Log



Details
Logs:


* 2023-03-29: init, todolist
* 2023-03-30: data preparation
	+ vi\_wiki\_23: lastest
	+ vi\_news\_17g: available
* 2023-03-30: tokenizer training
	+ algorithm: unigram, bpe
	+ size: 8k, 16k, 32k, 64k, 128k, 256k
* 2023-03-31: training trials
	+ config: base, large
	+ tokenizer: unigram\_16k, bpe\_16k, bpe\_128k
	+ args: default changed batch\_size, grad\_acc
	+ optimizer: default, customized optimizer
* 2023-03-31: training phase 0 started
	+ config: large
	+ tokenizer: bpe\_128k
	+ args: default
	+ GPU: 5x A100-SXM4-80G
* 2023-04-04: training interrupted unintentionally, optimizer checkpoint none, step 300000
* 2023-04-05: training resumed from lastcheckpoint 300000, learning\_rate addjusted 100µ -> 50µ
* 2023-04-10: sweet spot detected, step 800000
* 2023-04-11: training in progress, step 900000, loss increases, accuracy increases, regularization being working well, overfitting problem under monitoring
* 2023-04-12: training interrupted intentionally, step 1000000
* 2023-04-12: training phase 1 started, resumed intentionally, refining, learning\_rate -> 2µ (diverging)
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-26: training phase 2 started, resumed intentionally, step 1500000
* 2023-05-02: training phase 2 interrupted unintentionally, step 1980000
* 2023-05-05: training phase 2 resumed intentionally, step 2000000
* 2023-05-09: training in progress, step 2200000




### Todo


Model Info
----------


Model Perfomance
----------------



TL;DR
-----


Aspect: Corpus, Sub-Aspect: Language, Description: Vietnames
Aspect: , Sub-Aspect: Source, Description: Wiki 2023 (1GB), News 2023 (17.2GB), News (64GB)
Aspect: , Sub-Aspect: Size, Description: 1GB, 18GB, 64GB
Aspect: , Sub-Aspect: Preprocesing, Description: None
Aspect: Tokenizer, Sub-Aspect: Lib, Description: SentencePiece
Aspect: , Sub-Aspect: Algorithm, Description: BPE
Aspect: , Sub-Aspect: Type, Description: spm
Aspect: , Sub-Aspect: Vocab, Description: 128000
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: Model, Sub-Aspect: Type, Description: DeBERTaV3
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: , Sub-Aspect: Code, Description: URL
Aspect: Pretraining, Sub-Aspect: Task, Description: RTD
Aspect: , Sub-Aspect: Config, Description: model\_config.json
Aspect: , Sub-Aspect: Args, Description: default
Aspect: , Sub-Aspect: Hardware, Description: 5x Nvidia A100-SXM4-80G, 2x Nvidia 4090-PCI-24GB
Aspect: , Sub-Aspect: Phases, Description: Init, Refining, Enlarging
Aspect: , Sub-Aspect: Status, Description: Training on hold step 2200000
Aspect: Finetuning, Sub-Aspect: Status, Description: Not started (need help)


Repo
---- ### Pretraining



Phase 0: Init ### Info


* Goal: Init
* Progress: 30.00% ▓▓▓▓░░░░░░
* Status: training interrupped, step 1000000
* Loss: <a href="URL download>init loss ### Metrics


<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 1: Refining ### Info


* Goal: refining
* Changes: smaller learning rate (100µ -> 2µ)
* Progress: 100.00% ▓▓▓▓▓▓▓▓▓▓
* Status: training finished, step 1500000
* Loss: <a href="URL download>refining loss ### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 2: Enlarging ### Info


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training in progress, step 2000000
* Loss: <a href="URL download>enlarging loss ### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648"> #### Phase 2: Enlarging (resume, on hold)


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training phase 2 interrupted intentionally, step 2200000
* Loss: <a href="URL download>enlarging loss
* Metrics:



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648"> ### Finetuning NEED HELP


* Token Classification: tasks, datasets
* Sequence Classification: tasks, datasets


Experimental Results and Comparision
------------------------------------


* Not started


Usage
-----


!test #### Method 1: Load pretrained vi-deberta-v3-large with Transformers AutoClass


* Install ClickAI
* Tokenizer, Config, Model
* Transfer Pretrained Model Weights



Model state dict keys #### Method 2: Manually load pretrained vi-deberta-v3-large


* Dev YourTokenizer, Your Model
* Use ### Log



Details
Logs:


* 2023-03-29: init, todolist
* 2023-03-30: data preparation
	+ vi\_wiki\_23: lastest
	+ vi\_news\_17g: available
* 2023-03-30: tokenizer training
	+ algorithm: unigram, bpe
	+ size: 8k, 16k, 32k, 64k, 128k, 256k
* 2023-03-31: training trials
	+ config: base, large
	+ tokenizer: unigram\_16k, bpe\_16k, bpe\_128k
	+ args: default changed batch\_size, grad\_acc
	+ optimizer: default, customized optimizer
* 2023-03-31: training phase 0 started
	+ config: large
	+ tokenizer: bpe\_128k
	+ args: default
	+ GPU: 5x A100-SXM4-80G
* 2023-04-04: training interrupted unintentionally, optimizer checkpoint none, step 300000
* 2023-04-05: training resumed from lastcheckpoint 300000, learning\_rate addjusted 100µ -> 50µ
* 2023-04-10: sweet spot detected, step 800000
* 2023-04-11: training in progress, step 900000, loss increases, accuracy increases, regularization being working well, overfitting problem under monitoring
* 2023-04-12: training interrupted intentionally, step 1000000
* 2023-04-12: training phase 1 started, resumed intentionally, refining, learning\_rate -> 2µ (diverging)
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-26: training phase 2 started, resumed intentionally, step 1500000
* 2023-05-02: training phase 2 interrupted unintentionally, step 1980000
* 2023-05-05: training phase 2 resumed intentionally, step 2000000
* 2023-05-09: training in progress, step 2200000
TAGS
#transformers #pytorch #deberta-v3 #vi #dataset-anhdungitvn/vi-general-64g #license-other #endpoints_compatible #region-us 
 ### Todo


Model Info
----------


Model Perfomance
----------------



TL;DR
-----


Aspect: Corpus, Sub-Aspect: Language, Description: Vietnames
Aspect: , Sub-Aspect: Source, Description: Wiki 2023 (1GB), News 2023 (17.2GB), News (64GB)
Aspect: , Sub-Aspect: Size, Description: 1GB, 18GB, 64GB
Aspect: , Sub-Aspect: Preprocesing, Description: None
Aspect: Tokenizer, Sub-Aspect: Lib, Description: SentencePiece
Aspect: , Sub-Aspect: Algorithm, Description: BPE
Aspect: , Sub-Aspect: Type, Description: spm
Aspect: , Sub-Aspect: Vocab, Description: 128000
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: Model, Sub-Aspect: Type, Description: DeBERTaV3
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: , Sub-Aspect: Code, Description: URL
Aspect: Pretraining, Sub-Aspect: Task, Description: RTD
Aspect: , Sub-Aspect: Config, Description: model\_config.json
Aspect: , Sub-Aspect: Args, Description: default
Aspect: , Sub-Aspect: Hardware, Description: 5x Nvidia A100-SXM4-80G, 2x Nvidia 4090-PCI-24GB
Aspect: , Sub-Aspect: Phases, Description: Init, Refining, Enlarging
Aspect: , Sub-Aspect: Status, Description: Training on hold step 2200000
Aspect: Finetuning, Sub-Aspect: Status, Description: Not started (need help)


Repo
---- ### Pretraining



Phase 0: Init ### Info


* Goal: Init
* Progress: 30.00% ▓▓▓▓░░░░░░
* Status: training interrupped, step 1000000
* Loss: <a href="URL download>init loss ### Metrics


<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 1: Refining ### Info


* Goal: refining
* Changes: smaller learning rate (100µ -> 2µ)
* Progress: 100.00% ▓▓▓▓▓▓▓▓▓▓
* Status: training finished, step 1500000
* Loss: <a href="URL download>refining loss ### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 2: Enlarging ### Info


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training in progress, step 2000000
* Loss: <a href="URL download>enlarging loss ### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648"> #### Phase 2: Enlarging (resume, on hold)


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training phase 2 interrupted intentionally, step 2200000
* Loss: <a href="URL download>enlarging loss
* Metrics:



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648"> ### Finetuning NEED HELP


* Token Classification: tasks, datasets
* Sequence Classification: tasks, datasets


Experimental Results and Comparision
------------------------------------


* Not started


Usage
-----


!test #### Method 1: Load pretrained vi-deberta-v3-large with Transformers AutoClass


* Install ClickAI
* Tokenizer, Config, Model
* Transfer Pretrained Model Weights



Model state dict keys #### Method 2: Manually load pretrained vi-deberta-v3-large


* Dev YourTokenizer, Your Model
* Use ### Log



Details
Logs:


* 2023-03-29: init, todolist
* 2023-03-30: data preparation
	+ vi\_wiki\_23: lastest
	+ vi\_news\_17g: available
* 2023-03-30: tokenizer training
	+ algorithm: unigram, bpe
	+ size: 8k, 16k, 32k, 64k, 128k, 256k
* 2023-03-31: training trials
	+ config: base, large
	+ tokenizer: unigram\_16k, bpe\_16k, bpe\_128k
	+ args: default changed batch\_size, grad\_acc
	+ optimizer: default, customized optimizer
* 2023-03-31: training phase 0 started
	+ config: large
	+ tokenizer: bpe\_128k
	+ args: default
	+ GPU: 5x A100-SXM4-80G
* 2023-04-04: training interrupted unintentionally, optimizer checkpoint none, step 300000
* 2023-04-05: training resumed from lastcheckpoint 300000, learning\_rate addjusted 100µ -> 50µ
* 2023-04-10: sweet spot detected, step 800000
* 2023-04-11: training in progress, step 900000, loss increases, accuracy increases, regularization being working well, overfitting problem under monitoring
* 2023-04-12: training interrupted intentionally, step 1000000
* 2023-04-12: training phase 1 started, resumed intentionally, refining, learning\_rate -> 2µ (diverging)
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-26: training phase 2 started, resumed intentionally, step 1500000
* 2023-05-02: training phase 2 interrupted unintentionally, step 1980000
* 2023-05-05: training phase 2 resumed intentionally, step 2000000
* 2023-05-09: training in progress, step 2200000
TAGS
#transformers #pytorch #deberta-v3 #vi #dataset-anhdungitvn/vi-general-64g #license-other #endpoints_compatible #region-us 
### Todo


Model Info
----------


Model Perfomance
----------------



TL;DR
-----


Aspect: Corpus, Sub-Aspect: Language, Description: Vietnames
Aspect: , Sub-Aspect: Source, Description: Wiki 2023 (1GB), News 2023 (17.2GB), News (64GB)
Aspect: , Sub-Aspect: Size, Description: 1GB, 18GB, 64GB
Aspect: , Sub-Aspect: Preprocesing, Description: None
Aspect: Tokenizer, Sub-Aspect: Lib, Description: SentencePiece
Aspect: , Sub-Aspect: Algorithm, Description: BPE
Aspect: , Sub-Aspect: Type, Description: spm
Aspect: , Sub-Aspect: Vocab, Description: 128000
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: Model, Sub-Aspect: Type, Description: DeBERTaV3
Aspect: , Sub-Aspect: Ref, Description: URL
Aspect: , Sub-Aspect: Code, Description: URL
Aspect: Pretraining, Sub-Aspect: Task, Description: RTD
Aspect: , Sub-Aspect: Config, Description: model\_config.json
Aspect: , Sub-Aspect: Args, Description: default
Aspect: , Sub-Aspect: Hardware, Description: 5x Nvidia A100-SXM4-80G, 2x Nvidia 4090-PCI-24GB
Aspect: , Sub-Aspect: Phases, Description: Init, Refining, Enlarging
Aspect: , Sub-Aspect: Status, Description: Training on hold step 2200000
Aspect: Finetuning, Sub-Aspect: Status, Description: Not started (need help)


Repo
----### Pretraining



Phase 0: Init### Info


* Goal: Init
* Progress: 30.00% ▓▓▓▓░░░░░░
* Status: training interrupped, step 1000000
* Loss: <a href="URL download>init loss### Metrics


<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 1: Refining### Info


* Goal: refining
* Changes: smaller learning rate (100µ -> 2µ)
* Progress: 100.00% ▓▓▓▓▓▓▓▓▓▓
* Status: training finished, step 1500000
* Loss: <a href="URL download>refining loss### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">




Phase 2: Enlarging### Info


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training in progress, step 2000000
* Loss: <a href="URL download>enlarging loss### Metrics



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">#### Phase 2: Enlarging (resume, on hold)


* Goal: enlarging, augmenting, expanding data
* Changes: smaller learning rate (2µ -> 1µ), larger corpus (18G -> 64GB), eval samples (wiki 29227 -> news 240310)
* Progress: 20.00% ▓▓░░░░░░░░
* Status: training phase 2 interrupted intentionally, step 2200000
* Loss: <a href="URL download>enlarging loss
* Metrics:



<img src="URL
<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">  

<img src="URL width="648">
<img src="URL width="648">
<img src="URL width="648">### Finetuning NEED HELP


* Token Classification: tasks, datasets
* Sequence Classification: tasks, datasets


Experimental Results and Comparision
------------------------------------


* Not started


Usage
-----


!test#### Method 1: Load pretrained vi-deberta-v3-large with Transformers AutoClass


* Install ClickAI
* Tokenizer, Config, Model
* Transfer Pretrained Model Weights



Model state dict keys#### Method 2: Manually load pretrained vi-deberta-v3-large


* Dev YourTokenizer, Your Model
* Use### Log



Details
Logs:


* 2023-03-29: init, todolist
* 2023-03-30: data preparation
	+ vi\_wiki\_23: lastest
	+ vi\_news\_17g: available
* 2023-03-30: tokenizer training
	+ algorithm: unigram, bpe
	+ size: 8k, 16k, 32k, 64k, 128k, 256k
* 2023-03-31: training trials
	+ config: base, large
	+ tokenizer: unigram\_16k, bpe\_16k, bpe\_128k
	+ args: default changed batch\_size, grad\_acc
	+ optimizer: default, customized optimizer
* 2023-03-31: training phase 0 started
	+ config: large
	+ tokenizer: bpe\_128k
	+ args: default
	+ GPU: 5x A100-SXM4-80G
* 2023-04-04: training interrupted unintentionally, optimizer checkpoint none, step 300000
* 2023-04-05: training resumed from lastcheckpoint 300000, learning\_rate addjusted 100µ -> 50µ
* 2023-04-10: sweet spot detected, step 800000
* 2023-04-11: training in progress, step 900000, loss increases, accuracy increases, regularization being working well, overfitting problem under monitoring
* 2023-04-12: training interrupted intentionally, step 1000000
* 2023-04-12: training phase 1 started, resumed intentionally, refining, learning\_rate -> 2µ (diverging)
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-24: training phase 1 finished, refining, step 1500000
* 2023-04-26: training phase 2 started, resumed intentionally, step 1500000
* 2023-05-02: training phase 2 interrupted unintentionally, step 1980000
* 2023-05-05: training phase 2 resumed intentionally, step 2000000
* 2023-05-09: training in progress, step 2200000