transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q1-75p-filtered-random
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q1-75p-filtered-random

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9103

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.07  | 8000    | 2.2397          |
| 2.4342        | 0.13  | 16000   | 2.1511          |
| 2.4342        | 0.2   | 24000   | 2.1109          |
| 2.2417        | 0.26  | 32000   | 2.0789          |
| 2.2417        | 0.33  | 40000   | 2.0657          |
| 2.1852        | 0.39  | 48000   | 2.0397          |
| 2.1852        | 0.46  | 56000   | 2.0303          |
| 2.1511        | 0.52  | 64000   | 2.0248          |
| 2.1511        | 0.59  | 72000   | 2.0221          |
| 2.1261        | 0.65  | 80000   | 2.0128          |
| 2.1261        | 0.72  | 88000   | 2.0067          |
| 2.1179        | 0.78  | 96000   | 2.0039          |
| 2.1179        | 0.85  | 104000  | 1.9973          |
| 2.1097        | 0.91  | 112000  | 1.9835          |
| 2.1097        | 0.98  | 120000  | 1.9983          |
| 2.1031        | 1.04  | 128000  | 1.9899          |
| 2.1031        | 1.11  | 136000  | 1.9755          |
| 2.0977        | 1.17  | 144000  | 1.9855          |
| 2.0977        | 1.24  | 152000  | 1.9721          |
| 2.0892        | 1.3   | 160000  | 1.9813          |
| 2.0892        | 1.37  | 168000  | 1.9828          |
| 2.0882        | 1.44  | 176000  | 1.9704          |
| 2.0882        | 1.5   | 184000  | 1.9729          |
| 2.0884        | 1.57  | 192000  | 1.9721          |
| 2.0884        | 1.63  | 200000  | 1.9663          |
| 2.0814        | 1.7   | 208000  | 1.9612          |
| 2.0814        | 1.76  | 216000  | 1.9717          |
| 2.0806        | 1.83  | 224000  | 1.9594          |
| 2.0806        | 1.89  | 232000  | 1.9605          |
| 2.0838        | 1.96  | 240000  | 1.9556          |
| 2.0838        | 2.02  | 248000  | 1.9552          |
| 2.0711        | 2.09  | 256000  | 1.9653          |
| 2.0711        | 2.15  | 264000  | 1.9581          |
| 2.065         | 2.22  | 272000  | 1.9559          |
| 2.065         | 2.28  | 280000  | 1.9615          |
| 2.0769        | 2.35  | 288000  | 1.9494          |
| 2.0769        | 2.41  | 296000  | 1.9487          |
| 2.0733        | 2.48  | 304000  | 1.9546          |
| 2.0733        | 2.54  | 312000  | 1.9445          |
| 2.0675        | 2.61  | 320000  | 1.9535          |
| 2.0675        | 2.67  | 328000  | 1.9581          |
| 2.0599        | 2.74  | 336000  | 1.9472          |
| 2.0599        | 2.8   | 344000  | 1.9545          |
| 2.0675        | 2.87  | 352000  | 1.9552          |
| 2.0675        | 2.94  | 360000  | 1.9397          |
| 2.0711        | 3.0   | 368000  | 1.9475          |
| 2.0711        | 3.07  | 376000  | 1.9387          |
| 2.0663        | 3.13  | 384000  | 1.9484          |
| 2.0663        | 3.2   | 392000  | 1.9424          |
| 2.0628        | 3.26  | 400000  | 1.9411          |
| 2.0628        | 3.33  | 408000  | 1.9409          |
| 2.0651        | 3.39  | 416000  | 1.9447          |
| 2.0651        | 3.46  | 424000  | 1.9402          |
| 2.0598        | 3.52  | 432000  | 1.9504          |
| 2.0598        | 3.59  | 440000  | 1.9414          |
| 2.0612        | 3.65  | 448000  | 1.9330          |
| 2.0612        | 3.72  | 456000  | 1.9424          |
| 2.0653        | 3.78  | 464000  | 1.9310          |
| 2.0653        | 3.85  | 472000  | 1.9364          |
| 2.0585        | 3.91  | 480000  | 1.9507          |
| 2.0585        | 3.98  | 488000  | 1.9320          |
| 2.0593        | 4.04  | 496000  | 1.9416          |
| 2.0593        | 4.11  | 504000  | 1.9347          |
| 2.0671        | 4.17  | 512000  | 1.9391          |
| 2.0671        | 4.24  | 520000  | 1.9454          |
| 2.0552        | 4.31  | 528000  | 1.9501          |
| 2.0552        | 4.37  | 536000  | 1.9355          |
| 2.0626        | 4.44  | 544000  | 1.9240          |
| 2.0626        | 4.5   | 552000  | 1.9399          |
| 2.0592        | 4.57  | 560000  | 1.9360          |
| 2.0592        | 4.63  | 568000  | 1.9378          |
| 2.0584        | 4.7   | 576000  | 1.9293          |
| 2.0584        | 4.76  | 584000  | 1.9431          |
| 2.0515        | 4.83  | 592000  | 1.9325          |
| 2.0515        | 4.89  | 600000  | 1.9266          |
| 2.0545        | 4.96  | 608000  | 1.9215          |
| 2.0545        | 5.02  | 616000  | 1.9245          |
| 2.0525        | 5.09  | 624000  | 1.9373          |
| 2.0525        | 5.15  | 632000  | 1.9341          |
| 2.0556        | 5.22  | 640000  | 1.9313          |
| 2.0556        | 5.28  | 648000  | 1.9230          |
| 2.0567        | 5.35  | 656000  | 1.9300          |
| 2.0567        | 5.41  | 664000  | 1.9337          |
| 2.0506        | 5.48  | 672000  | 1.9317          |
| 2.0506        | 5.54  | 680000  | 1.9275          |
| 2.0561        | 5.61  | 688000  | 1.9376          |
| 2.0561        | 5.68  | 696000  | 1.9461          |
| 2.0496        | 5.74  | 704000  | 1.9239          |
| 2.0496        | 5.81  | 712000  | 1.9251          |
| 2.045         | 5.87  | 720000  | 1.9309          |
| 2.045         | 5.94  | 728000  | 1.9259          |
| 2.0512        | 6.0   | 736000  | 1.9237          |
| 2.0512        | 6.07  | 744000  | 1.9148          |
| 2.0512        | 6.13  | 752000  | 1.9220          |
| 2.0512        | 6.2   | 760000  | 1.9397          |
| 2.0445        | 6.26  | 768000  | 1.9241          |
| 2.0445        | 6.33  | 776000  | 1.9330          |
| 2.0481        | 6.39  | 784000  | 1.9124          |
| 2.0481        | 6.46  | 792000  | 1.9268          |
| 2.048         | 6.52  | 800000  | 1.9211          |
| 2.048         | 6.59  | 808000  | 1.9279          |
| 2.0555        | 6.65  | 816000  | 1.9169          |
| 2.0555        | 6.72  | 824000  | 1.9229          |
| 2.052         | 6.78  | 832000  | 1.9253          |
| 2.052         | 6.85  | 840000  | 1.9244          |
| 2.0475        | 6.91  | 848000  | 1.9192          |
| 2.0475        | 6.98  | 856000  | 1.9167          |
| 2.0521        | 7.05  | 864000  | 1.9202          |
| 2.0521        | 7.11  | 872000  | 1.9240          |
| 2.0516        | 7.18  | 880000  | 1.9231          |
| 2.0516        | 7.24  | 888000  | 1.9246          |
| 2.0526        | 7.31  | 896000  | 1.9174          |
| 2.0526        | 7.37  | 904000  | 1.9256          |
| 2.044         | 7.44  | 912000  | 1.9234          |
| 2.044         | 7.5   | 920000  | 1.9208          |
| 2.0493        | 7.57  | 928000  | 1.9233          |
| 2.0493        | 7.63  | 936000  | 1.9180          |
| 2.0535        | 7.7   | 944000  | 1.9200          |
| 2.0535        | 7.76  | 952000  | 1.9152          |
| 2.0454        | 7.83  | 960000  | 1.9268          |
| 2.0454        | 7.89  | 968000  | 1.9206          |
| 2.0428        | 7.96  | 976000  | 1.9170          |
| 2.0428        | 8.02  | 984000  | 1.9240          |
| 2.052         | 8.09  | 992000  | 1.9306          |
| 2.052         | 8.15  | 1000000 | 1.9192          |
| 2.0472        | 8.22  | 1008000 | 1.9313          |
| 2.0472        | 8.28  | 1016000 | 1.9238          |
| 2.0454        | 8.35  | 1024000 | 1.9162          |
| 2.0454        | 8.41  | 1032000 | 1.9130          |
| 2.0503        | 8.48  | 1040000 | 1.9260          |
| 2.0503        | 8.55  | 1048000 | 1.9212          |
| 2.0511        | 8.61  | 1056000 | 1.9115          |
| 2.0511        | 8.68  | 1064000 | 1.9123          |
| 2.049         | 8.74  | 1072000 | 1.9259          |
| 2.049         | 8.81  | 1080000 | 1.9321          |
| 2.0463        | 8.87  | 1088000 | 1.9148          |
| 2.0463        | 8.94  | 1096000 | 1.9145          |
| 2.0494        | 9.0   | 1104000 | 1.9097          |
| 2.0494        | 9.07  | 1112000 | 1.9135          |
| 2.0467        | 9.13  | 1120000 | 1.9164          |
| 2.0467        | 9.2   | 1128000 | 1.9224          |
| 2.0483        | 9.26  | 1136000 | 1.9135          |
| 2.0483        | 9.33  | 1144000 | 1.9199          |
| 2.0437        | 9.39  | 1152000 | 1.9213          |
| 2.0437        | 9.46  | 1160000 | 1.9161          |
| 2.0526        | 9.52  | 1168000 | 1.9148          |
| 2.0526        | 9.59  | 1176000 | 1.9183          |
| 2.0408        | 9.65  | 1184000 | 1.9079          |
| 2.0408        | 9.72  | 1192000 | 1.9186          |
| 2.0488        | 9.78  | 1200000 | 1.9141          |
| 2.0488        | 9.85  | 1208000 | 1.9079          |
| 2.0441        | 9.92  | 1216000 | 1.9251          |
| 2.0441        | 9.98  | 1224000 | 1.9255          |
| 2.0483        | 10.05 | 1232000 | 1.9108          |
| 2.0483        | 10.11 | 1240000 | 1.9045          |
| 2.0503        | 10.18 | 1248000 | 1.9170          |
| 2.0503        | 10.24 | 1256000 | 1.9025          |
| 2.0334        | 10.31 | 1264000 | 1.9199          |
| 2.0334        | 10.37 | 1272000 | 1.9187          |
| 2.0388        | 10.44 | 1280000 | 1.9030          |
| 2.0388        | 10.5  | 1288000 | 1.9231          |
| 2.0489        | 10.57 | 1296000 | 1.9084          |
| 2.0489        | 10.63 | 1304000 | 1.9184          |
| 2.0476        | 10.7  | 1312000 | 1.9160          |
| 2.0476        | 10.76 | 1320000 | 1.9276          |
| 2.037         | 10.83 | 1328000 | 1.9041          |
| 2.037         | 10.89 | 1336000 | 1.9228          |
| 2.0447        | 10.96 | 1344000 | 1.9151          |
| 2.0447        | 11.02 | 1352000 | 1.9069          |
| 2.039         | 11.09 | 1360000 | 1.9275          |
| 2.039         | 11.15 | 1368000 | 1.9067          |
| 2.0434        | 11.22 | 1376000 | 1.9087          |
| 2.0434        | 11.29 | 1384000 | 1.9041          |
| 2.0501        | 11.35 | 1392000 | 1.9033          |
| 2.0501        | 11.42 | 1400000 | 1.9153          |
| 2.0455        | 11.48 | 1408000 | 1.9174          |
| 2.0455        | 11.55 | 1416000 | 1.9174          |
| 2.0466        | 11.61 | 1424000 | 1.9261          |
| 2.0466        | 11.68 | 1432000 | 1.9181          |
| 2.0424        | 11.74 | 1440000 | 1.9141          |
| 2.0424        | 11.81 | 1448000 | 1.9004          |
| 2.0441        | 11.87 | 1456000 | 1.9197          |
| 2.0441        | 11.94 | 1464000 | 1.9075          |
| 2.04          | 12.0  | 1472000 | 1.9121          |
| 2.04          | 12.07 | 1480000 | 1.9211          |
| 2.0375        | 12.13 | 1488000 | 1.9111          |
| 2.0375        | 12.2  | 1496000 | 1.9188          |
| 2.0482        | 12.26 | 1504000 | 1.9099          |
| 2.0482        | 12.33 | 1512000 | 1.9161          |
| 2.0432        | 12.39 | 1520000 | 1.9198          |
| 2.0432        | 12.46 | 1528000 | 1.9154          |
| 2.0514        | 12.52 | 1536000 | 1.9059          |
| 2.0514        | 12.59 | 1544000 | 1.9204          |
| 2.0397        | 12.65 | 1552000 | 1.9055          |
| 2.0397        | 12.72 | 1560000 | 1.8962          |
| 2.0454        | 12.79 | 1568000 | 1.9040          |
| 2.0454        | 12.85 | 1576000 | 1.9168          |
| 2.0391        | 12.92 | 1584000 | 1.9037          |
| 2.0391        | 12.98 | 1592000 | 1.9186          |
| 2.0414        | 13.05 | 1600000 | 1.9122          |
| 2.0414        | 13.11 | 1608000 | 1.9116          |
| 2.0431        | 13.18 | 1616000 | 1.9057          |
| 2.0431        | 13.24 | 1624000 | 1.9115          |
| 2.0368        | 13.31 | 1632000 | 1.9120          |
| 2.0368        | 13.37 | 1640000 | 1.9114          |
| 2.0427        | 13.44 | 1648000 | 1.9128          |
| 2.0427        | 13.5  | 1656000 | 1.9201          |
| 2.0366        | 13.57 | 1664000 | 1.9053          |
| 2.0366        | 13.63 | 1672000 | 1.9077          |
| 2.0423        | 13.7  | 1680000 | 1.9155          |
| 2.0423        | 13.76 | 1688000 | 1.9025          |
| 2.0345        | 13.83 | 1696000 | 1.9117          |
| 2.0345        | 13.89 | 1704000 | 1.9146          |
| 2.0523        | 13.96 | 1712000 | 1.9094          |
| 2.0523        | 14.02 | 1720000 | 1.9028          |
| 2.0405        | 14.09 | 1728000 | 1.9034          |
| 2.0405        | 14.16 | 1736000 | 1.9033          |
| 2.0416        | 14.22 | 1744000 | 1.8958          |
| 2.0416        | 14.29 | 1752000 | 1.9072          |
| 2.0453        | 14.35 | 1760000 | 1.9067          |
| 2.0453        | 14.42 | 1768000 | 1.9113          |
| 2.0425        | 14.48 | 1776000 | 1.9104          |
| 2.0425        | 14.55 | 1784000 | 1.9110          |
| 2.0404        | 14.61 | 1792000 | 1.9037          |
| 2.0404        | 14.68 | 1800000 | 1.9003          |
| 2.0427        | 14.74 | 1808000 | 1.9116          |
| 2.0427        | 14.81 | 1816000 | 1.9106          |
| 2.0368        | 14.87 | 1824000 | 1.9095          |
| 2.0368        | 14.94 | 1832000 | 1.8980          |
| 2.0441        | 15.0  | 1840000 | 1.9186          |
| 2.0441        | 15.07 | 1848000 | 1.9040          |
| 2.0313        | 15.13 | 1856000 | 1.9186          |
| 2.0313        | 15.2  | 1864000 | 1.9016          |
| 2.0488        | 15.26 | 1872000 | 1.9048          |
| 2.0488        | 15.33 | 1880000 | 1.8995          |
| 2.0361        | 15.39 | 1888000 | 1.9120          |
| 2.0361        | 15.46 | 1896000 | 1.9079          |
| 2.0449        | 15.53 | 1904000 | 1.9110          |
| 2.0449        | 15.59 | 1912000 | 1.9091          |
| 2.043         | 15.66 | 1920000 | 1.9062          |
| 2.043         | 15.72 | 1928000 | 1.9070          |
| 2.0414        | 15.79 | 1936000 | 1.9134          |
| 2.0414        | 15.85 | 1944000 | 1.9079          |
| 2.0419        | 15.92 | 1952000 | 1.9061          |
| 2.0419        | 15.98 | 1960000 | 1.9058          |
| 2.0384        | 16.05 | 1968000 | 1.9114          |
| 2.0384        | 16.11 | 1976000 | 1.9040          |
| 2.0391        | 16.18 | 1984000 | 1.9095          |
| 2.0391        | 16.24 | 1992000 | 1.9182          |
| 2.0405        | 16.31 | 2000000 | 1.9111          |
| 2.0405        | 16.37 | 2008000 | 1.9056          |
| 2.0404        | 16.44 | 2016000 | 1.9134          |
| 2.0404        | 16.5  | 2024000 | 1.9070          |
| 2.0414        | 16.57 | 2032000 | 1.9085          |
| 2.0414        | 16.63 | 2040000 | 1.9063          |
| 2.0483        | 16.7  | 2048000 | 1.9187          |
| 2.0483        | 16.76 | 2056000 | 1.9105          |
| 2.0452        | 16.83 | 2064000 | 1.9118          |
| 2.0452        | 16.89 | 2072000 | 1.9092          |
| 2.0401        | 16.96 | 2080000 | 1.9114          |
| 2.0401        | 17.03 | 2088000 | 1.9098          |
| 2.0353        | 17.09 | 2096000 | 1.9069          |
| 2.0353        | 17.16 | 2104000 | 1.9027          |
| 2.0468        | 17.22 | 2112000 | 1.9102          |
| 2.0468        | 17.29 | 2120000 | 1.9046          |
| 2.0448        | 17.35 | 2128000 | 1.9024          |
| 2.0448        | 17.42 | 2136000 | 1.9108          |
| 2.0435        | 17.48 | 2144000 | 1.9122          |
| 2.0435        | 17.55 | 2152000 | 1.9044          |
| 2.0421        | 17.61 | 2160000 | 1.9069          |
| 2.0421        | 17.68 | 2168000 | 1.9020          |
| 2.0366        | 17.74 | 2176000 | 1.9153          |
| 2.0366        | 17.81 | 2184000 | 1.9072          |
| 2.034         | 17.87 | 2192000 | 1.9182          |
| 2.034         | 17.94 | 2200000 | 1.9086          |
| 2.0397        | 18.0  | 2208000 | 1.9071          |
| 2.0397        | 18.07 | 2216000 | 1.9147          |
| 2.0374        | 18.13 | 2224000 | 1.9200          |
| 2.0374        | 18.2  | 2232000 | 1.9178          |
| 2.0413        | 18.26 | 2240000 | 1.9090          |
| 2.0413        | 18.33 | 2248000 | 1.9037          |
| 2.047         | 18.4  | 2256000 | 1.9126          |
| 2.047         | 18.46 | 2264000 | 1.9117          |
| 2.0395        | 18.53 | 2272000 | 1.9110          |
| 2.0395        | 18.59 | 2280000 | 1.9158          |
| 2.0447        | 18.66 | 2288000 | 1.9017          |
| 2.0447        | 18.72 | 2296000 | 1.9072          |
| 2.0377        | 18.79 | 2304000 | 1.9136          |
| 2.0377        | 18.85 | 2312000 | 1.9084          |
| 2.0312        | 18.92 | 2320000 | 1.9092          |
| 2.0312        | 18.98 | 2328000 | 1.9103          |
| 2.0387        | 19.05 | 2336000 | 1.9023          |
| 2.0387        | 19.11 | 2344000 | 1.9035          |
| 2.0358        | 19.18 | 2352000 | 1.9131          |
| 2.0358        | 19.24 | 2360000 | 1.9066          |
| 2.0402        | 19.31 | 2368000 | 1.9083          |
| 2.0402        | 19.37 | 2376000 | 1.9068          |
| 2.0319        | 19.44 | 2384000 | 1.9012          |
| 2.0319        | 19.5  | 2392000 | 1.9273          |
| 2.0436        | 19.57 | 2400000 | 1.9059          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
