
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# roberta-tiny-2l-10M

This model was trained from scratch on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.1695
- Accuracy: 0.4534

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0004
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- gradient_accumulation_steps: 32
- total_train_batch_size: 512
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 50
- num_epochs: 100.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 7.7619        | 1.04  | 50   | 7.2338          | 0.0748   |
| 7.0524        | 2.08  | 100  | 6.6252          | 0.1331   |
| 6.8423        | 3.12  | 150  | 6.4622          | 0.1463   |
| 6.7298        | 4.16  | 200  | 6.3971          | 0.1488   |
| 6.669         | 5.21  | 250  | 6.3628          | 0.1519   |
| 6.2038        | 6.25  | 300  | 6.3371          | 0.1518   |
| 6.1783        | 7.29  | 350  | 6.3115          | 0.1532   |
| 6.1459        | 8.33  | 400  | 6.2922          | 0.1530   |
| 6.1096        | 9.37  | 450  | 6.2696          | 0.1536   |
| 6.0745        | 10.41 | 500  | 6.2545          | 0.1541   |
| 6.0689        | 11.45 | 550  | 6.2496          | 0.1533   |
| 6.0562        | 12.49 | 600  | 6.2313          | 0.1542   |
| 6.0324        | 13.53 | 650  | 6.2248          | 0.1536   |
| 5.9907        | 14.58 | 700  | 6.2179          | 0.1544   |
| 5.9683        | 15.62 | 750  | 6.1832          | 0.1545   |
| 5.9236        | 16.66 | 800  | 6.1413          | 0.1550   |
| 5.8808        | 17.7  | 850  | 6.0900          | 0.1558   |
| 5.8392        | 18.74 | 900  | 6.0543          | 0.1566   |
| 5.7962        | 19.78 | 950  | 6.0222          | 0.1575   |
| 5.7473        | 20.82 | 1000 | 5.9471          | 0.1617   |
| 5.5787        | 21.86 | 1050 | 5.7038          | 0.1891   |
| 5.2316        | 22.9  | 1100 | 5.2708          | 0.2382   |
| 4.6613        | 23.95 | 1150 | 4.7075          | 0.2975   |
| 4.3006        | 24.99 | 1200 | 4.4180          | 0.3222   |
| 4.3754        | 26.04 | 1250 | 4.2383          | 0.3385   |
| 4.2531        | 27.08 | 1300 | 4.1157          | 0.3491   |
| 4.0987        | 28.12 | 1350 | 4.0197          | 0.3578   |
| 4.0045        | 29.16 | 1400 | 3.9504          | 0.3656   |
| 3.9145        | 30.21 | 1450 | 3.8819          | 0.3718   |
| 3.5808        | 31.25 | 1500 | 3.8279          | 0.3781   |
| 3.5354        | 32.29 | 1550 | 3.7830          | 0.3826   |
| 3.4788        | 33.33 | 1600 | 3.7400          | 0.3872   |
| 3.4315        | 34.37 | 1650 | 3.7028          | 0.3911   |
| 3.3906        | 35.41 | 1700 | 3.6629          | 0.3956   |
| 3.3508        | 36.45 | 1750 | 3.6344          | 0.3984   |
| 3.288         | 37.49 | 1800 | 3.6046          | 0.4019   |
| 3.2678        | 38.53 | 1850 | 3.5799          | 0.4053   |
| 3.2382        | 39.58 | 1900 | 3.5549          | 0.4074   |
| 3.2151        | 40.62 | 1950 | 3.5285          | 0.4103   |
| 3.1777        | 41.66 | 2000 | 3.5069          | 0.4132   |
| 3.1499        | 42.7  | 2050 | 3.4917          | 0.4150   |
| 3.131         | 43.74 | 2100 | 3.4701          | 0.4168   |
| 3.0942        | 44.78 | 2150 | 3.4530          | 0.4189   |
| 3.0683        | 45.82 | 2200 | 3.4320          | 0.4212   |
| 3.0363        | 46.86 | 2250 | 3.4195          | 0.4227   |
| 3.0264        | 47.9  | 2300 | 3.4046          | 0.4249   |
| 3.0079        | 48.95 | 2350 | 3.3874          | 0.4267   |
| 2.9869        | 49.99 | 2400 | 3.3792          | 0.4277   |
| 3.1592        | 51.04 | 2450 | 3.3655          | 0.4289   |
| 3.1353        | 52.08 | 2500 | 3.3548          | 0.4310   |
| 3.1257        | 53.12 | 2550 | 3.3489          | 0.4308   |
| 3.0822        | 54.16 | 2600 | 3.3353          | 0.4327   |
| 3.0771        | 55.21 | 2650 | 3.3220          | 0.4341   |
| 2.8639        | 56.25 | 2700 | 3.3119          | 0.4354   |
| 2.8477        | 57.29 | 2750 | 3.3104          | 0.4360   |
| 2.8373        | 58.33 | 2800 | 3.2954          | 0.4378   |
| 2.818         | 59.37 | 2850 | 3.2935          | 0.4381   |
| 2.8137        | 60.41 | 2900 | 3.2786          | 0.4394   |
| 2.7985        | 61.45 | 2950 | 3.2747          | 0.4401   |
| 2.7936        | 62.49 | 3000 | 3.2668          | 0.4411   |
| 2.7764        | 63.53 | 3050 | 3.2569          | 0.4419   |
| 2.7819        | 64.58 | 3100 | 3.2492          | 0.4434   |
| 2.7672        | 65.62 | 3150 | 3.2494          | 0.4433   |
| 2.7629        | 66.66 | 3200 | 3.2410          | 0.4443   |
| 2.747         | 67.7  | 3250 | 3.2368          | 0.4446   |
| 2.7303        | 68.74 | 3300 | 3.2246          | 0.4460   |
| 2.7461        | 69.78 | 3350 | 3.2212          | 0.4462   |
| 2.7179        | 70.82 | 3400 | 3.2217          | 0.4470   |
| 2.7184        | 71.86 | 3450 | 3.2132          | 0.4479   |
| 2.7077        | 72.9  | 3500 | 3.2086          | 0.4487   |
| 2.6916        | 73.95 | 3550 | 3.2057          | 0.4482   |
| 2.6934        | 74.99 | 3600 | 3.2010          | 0.4495   |
| 2.8585        | 76.04 | 3650 | 3.1980          | 0.4497   |
| 2.8559        | 77.08 | 3700 | 3.1940          | 0.4503   |
| 2.8519        | 78.12 | 3750 | 3.1940          | 0.4506   |
| 2.8391        | 79.16 | 3800 | 3.1897          | 0.4509   |
| 2.845         | 80.21 | 3850 | 3.1858          | 0.4510   |
| 2.6636        | 81.25 | 3900 | 3.1819          | 0.4518   |
| 2.6569        | 82.29 | 3950 | 3.1834          | 0.4517   |
| 2.647         | 83.33 | 4000 | 3.1798          | 0.4517   |
| 2.6665        | 84.37 | 4050 | 3.1786          | 0.4525   |
| 2.6382        | 85.41 | 4100 | 3.1733          | 0.4525   |
| 2.6346        | 86.45 | 4150 | 3.1700          | 0.4532   |
| 2.6457        | 87.49 | 4200 | 3.1714          | 0.4529   |
| 2.6328        | 88.53 | 4250 | 3.1686          | 0.4537   |
| 2.6429        | 89.58 | 4300 | 3.1715          | 0.4534   |
| 2.6369        | 90.62 | 4350 | 3.1687          | 0.4538   |
| 2.628         | 91.66 | 4400 | 3.1651          | 0.4539   |
| 2.6373        | 92.7  | 4450 | 3.1660          | 0.4539   |
| 2.6357        | 93.74 | 4500 | 3.1662          | 0.4537   |
| 2.6302        | 94.78 | 4550 | 3.1695          | 0.4533   |


### Framework versions

- Transformers 4.24.0
- Pytorch 1.11.0+cu113
- Datasets 2.6.1
- Tokenizers 0.12.1

{"tags": ["generated_from_trainer"], "metrics": ["accuracy"], "model-index": [{"name": "roberta-tiny-2l-10M", "results": []}]}
transformers pytorch tensorboard roberta fill-mask generated_from_trainer autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #generated_from_trainer #autotrain_compatible #endpoints_compatible #region-us 

roberta-tiny-2l-10M
===================


This model was trained from scratch on an unknown dataset.
It achieves the following results on the evaluation set:


* Loss: 3.1695
* Accuracy: 0.4534


Model description
-----------------


More information needed


Intended uses & limitations
---------------------------


More information needed


Training and evaluation data
----------------------------


More information needed


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0004
* train\_batch\_size: 16
* eval\_batch\_size: 32
* seed: 42
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 512
* optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_steps: 50
* num\_epochs: 100.0
* mixed\_precision\_training: Native AMP


### Training results



### Framework versions


* Transformers 4.24.0
* Pytorch 1.11.0+cu113
* Datasets 2.6.1
* Tokenizers 0.12.1



### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0004
* train\_batch\_size: 16
* eval\_batch\_size: 32
* seed: 42
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 512
* optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_steps: 50
* num\_epochs: 100.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.24.0
* Pytorch 1.11.0+cu113
* Datasets 2.6.1
* Tokenizers 0.12.1
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #generated_from_trainer #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0004
* train\_batch\_size: 16
* eval\_batch\_size: 32
* seed: 42
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 512
* optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_steps: 50
* num\_epochs: 100.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.24.0
* Pytorch 1.11.0+cu113
* Datasets 2.6.1
* Tokenizers 0.12.1
TAGS
#transformers #pytorch #tensorboard #roberta #fill-mask #generated_from_trainer #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0004
* train\_batch\_size: 16
* eval\_batch\_size: 32
* seed: 42
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 512
* optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_steps: 50
* num\_epochs: 100.0
* mixed\_precision\_training: Native AMP### Training results### Framework versions


* Transformers 4.24.0
* Pytorch 1.11.0+cu113
* Datasets 2.6.1
* Tokenizers 0.12.1