
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert_add_pre-training-complete

This model is a fine-tuned version of [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) on the wikitext wikitext-103-raw-v1 dataset.
It achieves the following results on the evaluation set:
- Loss: 5.0239
- Accuracy: 0.2307

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 10
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 100
- training_steps: 300000
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:--------:|
| 6.295         | 1.0   | 3573   | 6.0701          | 0.1522   |
| 6.0482        | 2.0   | 7146   | 5.9533          | 0.1565   |
| 5.9799        | 3.0   | 10719  | 5.9008          | 0.1584   |
| 5.9378        | 4.0   | 14292  | 5.8997          | 0.1545   |
| 5.9057        | 5.0   | 17865  | 5.8905          | 0.1536   |
| 5.8811        | 6.0   | 21438  | 5.8646          | 0.1550   |
| 5.8617        | 7.0   | 25011  | 5.8322          | 0.1534   |
| 5.844         | 8.0   | 28584  | 5.8563          | 0.1523   |
| 5.8297        | 9.0   | 32157  | 5.8352          | 0.1548   |
| 5.8175        | 10.0  | 35730  | 5.8136          | 0.1558   |
| 5.8056        | 11.0  | 39303  | 5.8147          | 0.1526   |
| 5.7921        | 12.0  | 42876  | 5.8020          | 0.1548   |
| 5.7777        | 13.0  | 46449  | 5.7891          | 0.1545   |
| 5.7596        | 14.0  | 50022  | 5.7370          | 0.1587   |
| 5.7414        | 15.0  | 53595  | 5.7396          | 0.1604   |
| 5.7243        | 16.0  | 57168  | 5.7490          | 0.1564   |
| 5.6997        | 17.0  | 60741  | 5.7135          | 0.1561   |
| 5.6698        | 18.0  | 64314  | 5.6858          | 0.1620   |
| 5.6398        | 19.0  | 67887  | 5.6735          | 0.1644   |
| 5.6135        | 20.0  | 71460  | 5.6174          | 0.1681   |
| 5.5899        | 21.0  | 75033  | 5.6191          | 0.1684   |
| 5.5699        | 22.0  | 78606  | 5.5977          | 0.1669   |
| 5.5487        | 23.0  | 82179  | 5.6139          | 0.1669   |
| 5.529         | 24.0  | 85752  | 5.5272          | 0.1741   |
| 5.512         | 25.0  | 89325  | 5.5271          | 0.1727   |
| 5.4939        | 26.0  | 92898  | 5.5190          | 0.1721   |
| 5.4765        | 27.0  | 96471  | 5.4824          | 0.1770   |
| 5.4604        | 28.0  | 100044 | 5.5159          | 0.1747   |
| 5.4422        | 29.0  | 103617 | 5.4577          | 0.1807   |
| 5.4243        | 30.0  | 107190 | 5.4546          | 0.1772   |
| 5.408         | 31.0  | 110763 | 5.4297          | 0.1837   |
| 5.3915        | 32.0  | 114336 | 5.4089          | 0.1866   |
| 5.3766        | 33.0  | 117909 | 5.3996          | 0.1848   |
| 5.3594        | 34.0  | 121482 | 5.3974          | 0.1841   |
| 5.3451        | 35.0  | 125055 | 5.3718          | 0.1908   |
| 5.3294        | 36.0  | 128628 | 5.3706          | 0.1878   |
| 5.3155        | 37.0  | 132201 | 5.3677          | 0.1903   |
| 5.2996        | 38.0  | 135774 | 5.2970          | 0.1994   |
| 5.287         | 39.0  | 139347 | 5.3127          | 0.1977   |
| 5.2735        | 40.0  | 142920 | 5.3145          | 0.1955   |
| 5.26          | 41.0  | 146493 | 5.2985          | 0.2017   |
| 5.2487        | 42.0  | 150066 | 5.2661          | 0.2025   |
| 5.2362        | 43.0  | 153639 | 5.2712          | 0.2031   |
| 5.2248        | 44.0  | 157212 | 5.2452          | 0.2049   |
| 5.2115        | 45.0  | 160785 | 5.2325          | 0.2054   |
| 5.1998        | 46.0  | 164358 | 5.2233          | 0.2075   |
| 5.188         | 47.0  | 167931 | 5.1994          | 0.2118   |
| 5.1779        | 48.0  | 171504 | 5.2436          | 0.2069   |
| 5.1664        | 49.0  | 175077 | 5.2203          | 0.2129   |
| 5.1546        | 50.0  | 178650 | 5.1820          | 0.2134   |
| 5.1431        | 51.0  | 182223 | 5.2029          | 0.2122   |
| 5.133         | 52.0  | 185796 | 5.1458          | 0.2140   |
| 5.1226        | 53.0  | 189369 | 5.1757          | 0.2163   |
| 5.1138        | 54.0  | 192942 | 5.1380          | 0.2193   |
| 5.1046        | 55.0  | 196515 | 5.1498          | 0.2178   |
| 5.0984        | 56.0  | 200088 | 5.1094          | 0.2194   |
| 5.0907        | 57.0  | 203661 | 5.1354          | 0.2202   |
| 5.0812        | 58.0  | 207234 | 5.0662          | 0.2256   |
| 5.0748        | 59.0  | 210807 | 5.1163          | 0.2181   |
| 5.067         | 60.0  | 214380 | 5.1193          | 0.2199   |
| 5.0609        | 61.0  | 217953 | 5.0919          | 0.2224   |
| 5.0536        | 62.0  | 221526 | 5.0899          | 0.2239   |
| 5.0491        | 63.0  | 225099 | 5.1125          | 0.2224   |
| 5.0433        | 64.0  | 228672 | 5.0892          | 0.2226   |
| 5.0373        | 65.0  | 232245 | 5.0644          | 0.2260   |
| 5.032         | 66.0  | 235818 | 5.0623          | 0.2253   |
| 5.0283        | 67.0  | 239391 | 5.1004          | 0.2213   |
| 5.0223        | 68.0  | 242964 | 5.0573          | 0.2279   |
| 5.0184        | 69.0  | 246537 | 5.0488          | 0.2271   |
| 5.014         | 70.0  | 250110 | 5.0482          | 0.2280   |
| 5.0102        | 71.0  | 253683 | 5.0600          | 0.2269   |
| 5.0079        | 72.0  | 257256 | 5.0271          | 0.2279   |
| 5.0029        | 73.0  | 260829 | 5.0629          | 0.2267   |
| 4.9994        | 74.0  | 264402 | 5.0304          | 0.2297   |
| 4.9978        | 75.0  | 267975 | 5.0485          | 0.2269   |
| 4.9945        | 76.0  | 271548 | 5.0380          | 0.2306   |
| 4.9917        | 77.0  | 275121 | 5.0590          | 0.2265   |
| 4.9913        | 78.0  | 278694 | 5.0585          | 0.2262   |
| 4.987         | 79.0  | 282267 | 5.0339          | 0.2278   |
| 4.9862        | 80.0  | 285840 | 5.0214          | 0.2305   |
| 4.9841        | 81.0  | 289413 | 5.0393          | 0.2271   |
| 4.983         | 82.0  | 292986 | 5.0200          | 0.2298   |
| 4.9816        | 83.0  | 296559 | 5.0289          | 0.2300   |
| 4.9801        | 83.96 | 300000 | 4.9972          | 0.2332   |


### Framework versions

- Transformers 4.26.0
- Pytorch 1.14.0a0+410ce96
- Datasets 2.9.0
- Tokenizers 0.13.2

{"license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": ["wikitext"], "metrics": ["accuracy"], "model-index": [{"name": "distilbert_add_pre-training-complete", "results": [{"task": {"type": "fill-mask", "name": "Masked Language Modeling"}, "dataset": {"name": "wikitext wikitext-103-raw-v1", "type": "wikitext", "config": "wikitext-103-raw-v1", "split": "validation", "args": "wikitext-103-raw-v1"}, "metrics": [{"type": "accuracy", "value": 0.23073914743840437, "name": "Accuracy"}]}]}]}
transformers pytorch tensorboard distilbert fill-mask generated_from_trainer dataset:wikitext license:apache-2.0 model-index autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #tensorboard #distilbert #fill-mask #generated_from_trainer #dataset-wikitext #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #region-us 

distilbert\_add\_pre-training-complete
======================================


This model is a fine-tuned version of distilbert-base-uncased on the wikitext wikitext-103-raw-v1 dataset.
It achieves the following results on the evaluation set:


* Loss: 5.0239
* Accuracy: 0.2307


Model description
-----------------


More information needed


Intended uses & limitations
---------------------------


More information needed


Training and evaluation data
----------------------------


More information needed


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 64
* eval\_batch\_size: 64
* seed: 10
* distributed\_type: multi-GPU
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 100
* training\_steps: 300000
* mixed\_precision\_training: Native AMP


### Training results



### Framework versions


* Transformers 4.26.0
* Pytorch 1.14.0a0+410ce96
* Datasets 2.9.0
* Tokenizers 0.13.2



### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 64
* eval\_batch\_size: 64
* seed: 10
* distributed\_type: multi-GPU
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 100
* training\_steps: 300000
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.26.0
* Pytorch 1.14.0a0+410ce96
* Datasets 2.9.0
* Tokenizers 0.13.2
TAGS
#transformers #pytorch #tensorboard #distilbert #fill-mask #generated_from_trainer #dataset-wikitext #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 64
* eval\_batch\_size: 64
* seed: 10
* distributed\_type: multi-GPU
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 100
* training\_steps: 300000
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.26.0
* Pytorch 1.14.0a0+410ce96
* Datasets 2.9.0
* Tokenizers 0.13.2
TAGS
#transformers #pytorch #tensorboard #distilbert #fill-mask #generated_from_trainer #dataset-wikitext #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 5e-05
* train\_batch\_size: 64
* eval\_batch\_size: 64
* seed: 10
* distributed\_type: multi-GPU
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: linear
* lr\_scheduler\_warmup\_steps: 100
* training\_steps: 300000
* mixed\_precision\_training: Native AMP### Training results### Framework versions


* Transformers 4.26.0
* Pytorch 1.14.0a0+410ce96
* Datasets 2.9.0
* Tokenizers 0.13.2