transformers pytorch bert fill-mask generated_from_trainer autotrain_compatible endpoints_compatible region:us
---
tags:
- generated_from_trainer
model-index:
- name: mlm_bert-steps27053-bs4096-0.0003-8-8-512-0.1
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# mlm_bert-steps27053-bs4096-0.0003-8-8-512-0.1

This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.8228

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 512
- eval_batch_size: 512
- seed: 42
- distributed_type: multi-GPU
- num_devices: 2
- gradient_accumulation_steps: 8
- total_train_batch_size: 8192
- total_eval_batch_size: 1024
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- training_steps: 27053

### Training results

| Training Loss | Epoch | Step  | Validation Loss |
|:-------------:|:-----:|:-----:|:---------------:|
| 9.2672        | 0.15  | 100   | 9.1858          |
| 8.166         | 0.3   | 200   | 8.0656          |
| 6.9338        | 0.44  | 300   | 6.8679          |
| 6.4326        | 0.59  | 400   | 6.3892          |
| 6.2326        | 0.74  | 500   | 6.1984          |
| 6.1182        | 0.89  | 600   | 6.0812          |
| 6.0395        | 1.03  | 700   | 6.0034          |
| 5.9768        | 1.18  | 800   | 5.9629          |
| 5.9271        | 1.33  | 900   | 5.9083          |
| 5.8918        | 1.48  | 1000  | 5.8730          |
| 5.8617        | 1.63  | 1100  | 5.8512          |
| 5.84          | 1.77  | 1200  | 5.8212          |
| 5.8115        | 1.92  | 1300  | 5.8011          |
| 5.7889        | 2.07  | 1400  | 5.7758          |
| 5.7719        | 2.22  | 1500  | 5.7586          |
| 5.7512        | 2.37  | 1600  | 5.7396          |
| 5.7314        | 2.51  | 1700  | 5.7243          |
| 5.7207        | 2.66  | 1800  | 5.7065          |
| 5.7027        | 2.81  | 1900  | 5.6940          |
| 5.6889        | 2.96  | 2000  | 5.6751          |
| 5.6756        | 3.1   | 2100  | 5.6593          |
| 5.6648        | 3.25  | 2200  | 5.6497          |
| 5.6471        | 3.4   | 2300  | 5.6359          |
| 5.6326        | 3.55  | 2400  | 5.6251          |
| 5.6297        | 3.7   | 2500  | 5.6168          |
| 5.6125        | 3.84  | 2600  | 5.6039          |
| 5.6059        | 3.99  | 2700  | 5.5956          |
| 5.5906        | 4.14  | 2800  | 5.5832          |
| 5.5898        | 4.29  | 2900  | 5.5857          |
| 5.5816        | 4.44  | 3000  | 5.5710          |
| 5.5691        | 4.58  | 3100  | 5.5611          |
| 5.4637        | 4.73  | 3200  | 5.4215          |
| 5.3117        | 4.88  | 3300  | 5.2148          |
| 5.1268        | 5.03  | 3400  | 4.9871          |
| 4.9629        | 5.17  | 3500  | 4.7933          |
| 4.8129        | 5.32  | 3600  | 4.5922          |
| 4.4889        | 5.47  | 3700  | 4.2237          |
| 4.0294        | 5.62  | 3800  | 3.7873          |
| 3.6563        | 5.77  | 3900  | 3.4882          |
| 3.4771        | 5.91  | 4000  | 3.3127          |
| 3.3476        | 6.06  | 4100  | 3.1826          |
| 3.2367        | 6.21  | 4200  | 3.0838          |
| 3.146         | 6.36  | 4300  | 2.9982          |
| 3.08          | 6.51  | 4400  | 2.9275          |
| 3.0095        | 6.65  | 4500  | 2.8609          |
| 2.946         | 6.8   | 4600  | 2.8042          |
| 2.8998        | 6.95  | 4700  | 2.7545          |
| 2.8479        | 7.1   | 4800  | 2.7136          |
| 2.811         | 7.24  | 4900  | 2.6697          |
| 2.7711        | 7.39  | 5000  | 2.6365          |
| 2.7317        | 7.54  | 5100  | 2.5999          |
| 2.6977        | 7.69  | 5200  | 2.5754          |
| 2.6692        | 7.84  | 5300  | 2.5444          |
| 2.64          | 7.98  | 5400  | 2.5163          |
| 2.6158        | 8.13  | 5500  | 2.4947          |
| 2.5959        | 8.28  | 5600  | 2.4712          |
| 2.5721        | 8.43  | 5700  | 2.4514          |
| 2.5509        | 8.58  | 5800  | 2.4347          |
| 2.5319        | 8.72  | 5900  | 2.4133          |
| 2.5184        | 8.87  | 6000  | 2.3941          |
| 2.5021        | 9.02  | 6100  | 2.3781          |
| 2.4867        | 9.17  | 6200  | 2.3639          |
| 2.4694        | 9.31  | 6300  | 2.3511          |
| 2.4585        | 9.46  | 6400  | 2.3360          |
| 2.4411        | 9.61  | 6500  | 2.3206          |
| 2.43          | 9.76  | 6600  | 2.3104          |
| 2.4194        | 9.91  | 6700  | 2.2952          |
| 2.4079        | 10.05 | 6800  | 2.2863          |
| 2.3947        | 10.2  | 6900  | 2.2767          |
| 2.3809        | 10.35 | 7000  | 2.2628          |
| 2.3741        | 10.5  | 7100  | 2.2527          |
| 2.361         | 10.64 | 7200  | 2.2429          |
| 2.3544        | 10.79 | 7300  | 2.2363          |
| 2.3429        | 10.94 | 7400  | 2.2265          |
| 2.3354        | 11.09 | 7500  | 2.2156          |
| 2.3232        | 11.24 | 7600  | 2.2077          |
| 2.3187        | 11.38 | 7700  | 2.2002          |
| 2.3104        | 11.53 | 7800  | 2.1944          |
| 2.3009        | 11.68 | 7900  | 2.1853          |
| 2.2959        | 11.83 | 8000  | 2.1755          |
| 2.285         | 11.98 | 8100  | 2.1701          |
| 2.2847        | 12.12 | 8200  | 2.1632          |
| 2.2711        | 12.27 | 8300  | 2.1556          |
| 2.2617        | 12.42 | 8400  | 2.1455          |
| 2.2552        | 12.57 | 8500  | 2.1415          |
| 2.2466        | 12.71 | 8600  | 2.1311          |
| 2.2437        | 12.86 | 8700  | 2.1235          |
| 2.2372        | 13.01 | 8800  | 2.1178          |
| 2.2271        | 13.16 | 8900  | 2.1109          |
| 2.2232        | 13.31 | 9000  | 2.1061          |
| 2.2188        | 13.45 | 9100  | 2.1014          |
| 2.2111        | 13.6  | 9200  | 2.0953          |
| 2.2014        | 13.75 | 9300  | 2.0884          |
| 2.2005        | 13.9  | 9400  | 2.0823          |
| 2.1918        | 14.05 | 9500  | 2.0775          |
| 2.1874        | 14.19 | 9600  | 2.0713          |
| 2.1828        | 14.34 | 9700  | 2.0672          |
| 2.1802        | 14.49 | 9800  | 2.0624          |
| 2.1688        | 14.64 | 9900  | 2.0577          |
| 2.1686        | 14.78 | 10000 | 2.0535          |
| 2.1642        | 14.93 | 10100 | 2.0471          |
| 2.1598        | 15.08 | 10200 | 2.0424          |
| 2.1561        | 15.23 | 10300 | 2.0390          |
| 2.1528        | 15.38 | 10400 | 2.0353          |
| 2.1491        | 15.52 | 10500 | 2.0346          |
| 2.1437        | 15.67 | 10600 | 2.0272          |
| 2.1418        | 15.82 | 10700 | 2.0239          |
| 2.1371        | 15.97 | 10800 | 2.0206          |
| 2.1323        | 16.12 | 10900 | 2.0180          |
| 2.1309        | 16.26 | 11000 | 2.0133          |
| 2.1274        | 16.41 | 11100 | 2.0117          |
| 2.1225        | 16.56 | 11200 | 2.0069          |
| 2.1187        | 16.71 | 11300 | 2.0035          |
| 2.1173        | 16.85 | 11400 | 2.0018          |
| 2.1152        | 17.0  | 11500 | 1.9992          |
| 2.1059        | 17.15 | 11600 | 1.9942          |
| 2.1081        | 17.3  | 11700 | 1.9920          |
| 2.106         | 17.45 | 11800 | 1.9886          |
| 2.1013        | 17.59 | 11900 | 1.9868          |
| 2.0974        | 17.74 | 12000 | 1.9850          |
| 2.0956        | 17.89 | 12100 | 1.9795          |
| 2.0916        | 18.04 | 12200 | 1.9789          |
| 2.089         | 18.19 | 12300 | 1.9773          |
| 2.0876        | 18.33 | 12400 | 1.9740          |
| 2.0871        | 18.48 | 12500 | 1.9692          |
| 2.0772        | 18.63 | 12600 | 1.9686          |
| 2.0823        | 18.78 | 12700 | 1.9661          |
| 2.0756        | 18.92 | 12800 | 1.9621          |
| 2.0748        | 19.07 | 12900 | 1.9625          |
| 2.0708        | 19.22 | 13000 | 1.9572          |
| 2.0736        | 19.37 | 13100 | 1.9544          |
| 2.0681        | 19.52 | 13200 | 1.9531          |
| 2.0639        | 19.66 | 13300 | 1.9522          |
| 2.0603        | 19.81 | 13400 | 1.9493          |
| 2.0666        | 19.96 | 13500 | 1.9465          |
| 2.0609        | 20.11 | 13600 | 1.9461          |
| 2.0552        | 20.26 | 13700 | 1.9417          |
| 2.0596        | 20.4  | 13800 | 1.9420          |
| 2.0569        | 20.55 | 13900 | 1.9376          |
| 2.0546        | 20.7  | 14000 | 1.9385          |
| 2.0518        | 20.85 | 14100 | 1.9342          |
| 2.0519        | 20.99 | 14200 | 1.9310          |
| 2.0503        | 21.14 | 14300 | 1.9322          |
| 2.0471        | 21.29 | 14400 | 1.9303          |
| 2.0447        | 21.44 | 14500 | 1.9282          |
| 2.044         | 21.59 | 14600 | 1.9236          |
| 2.041         | 21.73 | 14700 | 1.9224          |
| 2.0342        | 21.88 | 14800 | 1.9227          |
| 2.0333        | 22.03 | 14900 | 1.9222          |
| 2.0374        | 22.18 | 15000 | 1.9174          |
| 2.0319        | 22.32 | 15100 | 1.9173          |
| 2.0317        | 22.47 | 15200 | 1.9159          |
| 2.0304        | 22.62 | 15300 | 1.9135          |
| 2.0271        | 22.77 | 15400 | 1.9128          |
| 2.0283        | 22.92 | 15500 | 1.9123          |
| 2.0209        | 23.06 | 15600 | 1.9092          |
| 2.0227        | 23.21 | 15700 | 1.9069          |
| 2.0211        | 23.36 | 15800 | 1.9050          |
| 2.0195        | 23.51 | 15900 | 1.9062          |
| 2.0156        | 23.66 | 16000 | 1.9029          |
| 2.0155        | 23.8  | 16100 | 1.9015          |
| 2.0156        | 23.95 | 16200 | 1.9004          |
| 2.0133        | 24.1  | 16300 | 1.8988          |
| 2.0092        | 24.25 | 16400 | 1.8965          |
| 2.0091        | 24.39 | 16500 | 1.8974          |
| 2.01          | 24.54 | 16600 | 1.8964          |
| 2.0119        | 24.69 | 16700 | 1.8923          |
| 2.0055        | 24.84 | 16800 | 1.8925          |
| 2.0063        | 24.99 | 16900 | 1.8894          |
| 2.0046        | 25.13 | 17000 | 1.8895          |
| 2.0034        | 25.28 | 17100 | 1.8881          |
| 2.0014        | 25.43 | 17200 | 1.8872          |
| 2.0042        | 25.58 | 17300 | 1.8853          |
| 2.0014        | 25.73 | 17400 | 1.8840          |
| 1.9992        | 25.87 | 17500 | 1.8842          |
| 1.9948        | 26.02 | 17600 | 1.8826          |
| 1.9992        | 26.17 | 17700 | 1.8815          |
| 1.997         | 26.32 | 17800 | 1.8792          |
| 1.9942        | 26.46 | 17900 | 1.8801          |
| 1.9916        | 26.61 | 18000 | 1.8765          |
| 1.9965        | 26.76 | 18100 | 1.8764          |
| 1.9921        | 26.91 | 18200 | 1.8751          |
| 1.9907        | 27.06 | 18300 | 1.8745          |
| 1.9918        | 27.2  | 18400 | 1.8733          |
| 1.9864        | 27.35 | 18500 | 1.8727          |
| 1.9865        | 27.5  | 18600 | 1.8715          |
| 1.9881        | 27.65 | 18700 | 1.8699          |
| 1.9834        | 27.8  | 18800 | 1.8689          |
| 1.9835        | 27.94 | 18900 | 1.8677          |
| 1.9738        | 28.09 | 19000 | 1.8675          |
| 1.9807        | 28.24 | 19100 | 1.8679          |
| 1.9828        | 28.39 | 19200 | 1.8661          |
| 1.9813        | 28.53 | 19300 | 1.8645          |
| 1.9772        | 28.68 | 19400 | 1.8642          |
| 1.9766        | 28.83 | 19500 | 1.8643          |
| 1.9805        | 28.98 | 19600 | 1.8615          |
| 1.9746        | 29.13 | 19700 | 1.8627          |
| 1.9767        | 29.27 | 19800 | 1.8617          |
| 1.9751        | 29.42 | 19900 | 1.8605          |
| 1.9742        | 29.57 | 20000 | 1.8597          |
| 1.9734        | 29.72 | 20100 | 1.8574          |
| 1.9695        | 29.87 | 20200 | 1.8551          |
| 1.9695        | 30.01 | 20300 | 1.8560          |
| 1.9734        | 30.16 | 20400 | 1.8555          |
| 1.9726        | 30.31 | 20500 | 1.8527          |
| 1.9707        | 30.46 | 20600 | 1.8543          |
| 1.9683        | 30.6  | 20700 | 1.8534          |
| 1.9675        | 30.75 | 20800 | 1.8522          |
| 1.9668        | 30.9  | 20900 | 1.8496          |
| 1.9687        | 31.05 | 21000 | 1.8500          |
| 1.9678        | 31.2  | 21100 | 1.8493          |
| 1.9659        | 31.34 | 21200 | 1.8497          |
| 1.962         | 31.49 | 21300 | 1.8483          |
| 1.9637        | 31.64 | 21400 | 1.8483          |
| 1.9612        | 31.79 | 21500 | 1.8468          |
| 1.9642        | 31.93 | 21600 | 1.8463          |
| 1.9585        | 32.08 | 21700 | 1.8457          |
| 1.9616        | 32.23 | 21800 | 1.8458          |
| 1.9593        | 32.38 | 21900 | 1.8429          |
| 1.9579        | 32.53 | 22000 | 1.8431          |
| 1.9576        | 32.67 | 22100 | 1.8439          |
| 1.9552        | 32.82 | 22200 | 1.8409          |
| 1.9586        | 32.97 | 22300 | 1.8414          |
| 1.9582        | 33.12 | 22400 | 1.8415          |
| 1.9542        | 33.27 | 22500 | 1.8406          |
| 1.9557        | 33.41 | 22600 | 1.8406          |
| 1.9528        | 33.56 | 22700 | 1.8397          |
| 1.9542        | 33.71 | 22800 | 1.8374          |
| 1.9584        | 33.86 | 22900 | 1.8380          |
| 1.9549        | 34.0  | 23000 | 1.8366          |
| 1.9549        | 34.15 | 23100 | 1.8365          |
| 1.9551        | 34.3  | 23200 | 1.8368          |
| 1.9505        | 34.45 | 23300 | 1.8367          |
| 1.953         | 34.6  | 23400 | 1.8351          |
| 1.9508        | 34.74 | 23500 | 1.8345          |
| 1.9473        | 34.89 | 23600 | 1.8336          |
| 1.9507        | 35.04 | 23700 | 1.8339          |
| 1.9499        | 35.19 | 23800 | 1.8329          |
| 1.9486        | 35.34 | 23900 | 1.8342          |
| 1.946         | 35.48 | 24000 | 1.8313          |
| 1.9423        | 35.63 | 24100 | 1.8319          |
| 1.9439        | 35.78 | 24200 | 1.8315          |
| 1.9425        | 35.93 | 24300 | 1.8308          |
| 1.944         | 36.07 | 24400 | 1.8304          |
| 1.9462        | 36.22 | 24500 | 1.8301          |
| 1.9424        | 36.37 | 24600 | 1.8295          |
| 1.9443        | 36.52 | 24700 | 1.8294          |
| 1.9463        | 36.67 | 24800 | 1.8282          |
| 1.9448        | 36.81 | 24900 | 1.8290          |
| 1.9424        | 36.96 | 25000 | 1.8271          |
| 1.9428        | 37.11 | 25100 | 1.8285          |
| 1.9503        | 37.26 | 25200 | 1.8263          |
| 1.9474        | 37.41 | 25300 | 1.8277          |
| 1.9406        | 37.55 | 25400 | 1.8260          |
| 1.9407        | 37.7  | 25500 | 1.8267          |
| 1.946         | 37.85 | 25600 | 1.8257          |
| 1.9395        | 38.0  | 25700 | 1.8254          |
| 1.9412        | 38.14 | 25800 | 1.8255          |
| 1.9421        | 38.29 | 25900 | 1.8252          |
| 1.9398        | 38.44 | 26000 | 1.8257          |
| 1.9373        | 38.59 | 26100 | 1.8239          |
| 1.938         | 38.74 | 26200 | 1.8234          |
| 1.9399        | 38.88 | 26300 | 1.8230          |
| 1.9382        | 39.03 | 26400 | 1.8235          |
| 1.9377        | 39.18 | 26500 | 1.8216          |
| 1.9358        | 39.33 | 26600 | 1.8217          |
| 1.9366        | 39.48 | 26700 | 1.8227          |
| 1.9375        | 39.62 | 26800 | 1.8224          |
| 1.9385        | 39.77 | 26900 | 1.8233          |
| 1.9373        | 39.92 | 27000 | 1.8228          |


### Framework versions

- Transformers 4.34.1
- Pytorch 2.1.0+cu121
- Datasets 2.14.6
- Tokenizers 0.14.1
