transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q3-50p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q3-50p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.6316

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.03  | 8000    | 2.9380          |
| 3.1308        | 0.07  | 16000   | 2.8593          |
| 3.1308        | 0.1   | 24000   | 2.8063          |
| 2.9519        | 0.14  | 32000   | 2.7832          |
| 2.9519        | 0.17  | 40000   | 2.7521          |
| 2.889         | 0.2   | 48000   | 2.7322          |
| 2.889         | 0.24  | 56000   | 2.7259          |
| 2.8592        | 0.27  | 64000   | 2.7218          |
| 2.8592        | 0.3   | 72000   | 2.7106          |
| 2.8345        | 0.34  | 80000   | 2.7071          |
| 2.8345        | 0.37  | 88000   | 2.6903          |
| 2.8303        | 0.41  | 96000   | 2.7001          |
| 2.8303        | 0.44  | 104000  | 2.6929          |
| 2.824         | 0.47  | 112000  | 2.6907          |
| 2.824         | 0.51  | 120000  | 2.6852          |
| 2.8223        | 0.54  | 128000  | 2.6804          |
| 2.8223        | 0.57  | 136000  | 2.6727          |
| 2.8141        | 0.61  | 144000  | 2.6784          |
| 2.8141        | 0.64  | 152000  | 2.6775          |
| 2.8124        | 0.68  | 160000  | 2.6723          |
| 2.8124        | 0.71  | 168000  | 2.6683          |
| 2.8042        | 0.74  | 176000  | 2.6712          |
| 2.8042        | 0.78  | 184000  | 2.6661          |
| 2.8051        | 0.81  | 192000  | 2.6783          |
| 2.8051        | 0.85  | 200000  | 2.6683          |
| 2.798         | 0.88  | 208000  | 2.6656          |
| 2.798         | 0.91  | 216000  | 2.6659          |
| 2.8043        | 0.95  | 224000  | 2.6700          |
| 2.8043        | 0.98  | 232000  | 2.6680          |
| 2.8055        | 1.01  | 240000  | 2.6597          |
| 2.8055        | 1.05  | 248000  | 2.6597          |
| 2.8048        | 1.08  | 256000  | 2.6569          |
| 2.8048        | 1.12  | 264000  | 2.6502          |
| 2.806         | 1.15  | 272000  | 2.6593          |
| 2.806         | 1.18  | 280000  | 2.6597          |
| 2.8012        | 1.22  | 288000  | 2.6604          |
| 2.8012        | 1.25  | 296000  | 2.6545          |
| 2.8029        | 1.28  | 304000  | 2.6571          |
| 2.8029        | 1.32  | 312000  | 2.6534          |
| 2.7991        | 1.35  | 320000  | 2.6650          |
| 2.7991        | 1.39  | 328000  | 2.6680          |
| 2.7949        | 1.42  | 336000  | 2.6544          |
| 2.7949        | 1.45  | 344000  | 2.6460          |
| 2.7972        | 1.49  | 352000  | 2.6553          |
| 2.7972        | 1.52  | 360000  | 2.6428          |
| 2.7924        | 1.56  | 368000  | 2.6536          |
| 2.7924        | 1.59  | 376000  | 2.6550          |
| 2.805         | 1.62  | 384000  | 2.6524          |
| 2.805         | 1.66  | 392000  | 2.6524          |
| 2.7972        | 1.69  | 400000  | 2.6579          |
| 2.7972        | 1.72  | 408000  | 2.6500          |
| 2.8003        | 1.76  | 416000  | 2.6526          |
| 2.8003        | 1.79  | 424000  | 2.6444          |
| 2.8005        | 1.83  | 432000  | 2.6463          |
| 2.8005        | 1.86  | 440000  | 2.6549          |
| 2.7957        | 1.89  | 448000  | 2.6530          |
| 2.7957        | 1.93  | 456000  | 2.6504          |
| 2.7949        | 1.96  | 464000  | 2.6480          |
| 2.7949        | 1.99  | 472000  | 2.6497          |
| 2.7978        | 2.03  | 480000  | 2.6490          |
| 2.7978        | 2.06  | 488000  | 2.6505          |
| 2.8041        | 2.1   | 496000  | 2.6388          |
| 2.8041        | 2.13  | 504000  | 2.6460          |
| 2.7935        | 2.16  | 512000  | 2.6519          |
| 2.7935        | 2.2   | 520000  | 2.6494          |
| 2.7982        | 2.23  | 528000  | 2.6550          |
| 2.7982        | 2.27  | 536000  | 2.6460          |
| 2.7949        | 2.3   | 544000  | 2.6497          |
| 2.7949        | 2.33  | 552000  | 2.6478          |
| 2.7953        | 2.37  | 560000  | 2.6487          |
| 2.7953        | 2.4   | 568000  | 2.6400          |
| 2.7942        | 2.43  | 576000  | 2.6440          |
| 2.7942        | 2.47  | 584000  | nan             |
| 2.803         | 2.5   | 592000  | 2.6455          |
| 2.803         | 2.54  | 600000  | 2.6401          |
| 2.7961        | 2.57  | 608000  | 2.6511          |
| 2.7961        | 2.6   | 616000  | 2.6401          |
| 2.7975        | 2.64  | 624000  | 2.6437          |
| 2.7975        | 2.67  | 632000  | 2.6432          |
| 2.7946        | 2.7   | 640000  | 2.6461          |
| 2.7946        | 2.74  | 648000  | 2.6491          |
| 2.7963        | 2.77  | 656000  | 2.6442          |
| 2.7963        | 2.81  | 664000  | 2.6416          |
| 2.7924        | 2.84  | 672000  | 2.6403          |
| 2.7924        | 2.87  | 680000  | 2.6466          |
| 2.8004        | 2.91  | 688000  | 2.6436          |
| 2.8004        | 2.94  | 696000  | 2.6447          |
| 2.8039        | 2.98  | 704000  | 2.6412          |
| 2.8039        | 3.01  | 712000  | 2.6400          |
| 2.7958        | 3.04  | 720000  | 2.6419          |
| 2.7958        | 3.08  | 728000  | 2.6413          |
| 2.7967        | 3.11  | 736000  | nan             |
| 2.7967        | 3.14  | 744000  | 2.6399          |
| 2.7934        | 3.18  | 752000  | 2.6405          |
| 2.7934        | 3.21  | 760000  | 2.6387          |
| 2.7988        | 3.25  | 768000  | 2.6463          |
| 2.7988        | 3.28  | 776000  | 2.6308          |
| 2.793         | 3.31  | 784000  | 2.6343          |
| 2.793         | 3.35  | 792000  | 2.6358          |
| 2.797         | 3.38  | 800000  | 2.6397          |
| 2.797         | 3.41  | 808000  | 2.6341          |
| 2.7832        | 3.45  | 816000  | 2.6394          |
| 2.7832        | 3.48  | 824000  | 2.6341          |
| 2.792         | 3.52  | 832000  | 2.6424          |
| 2.792         | 3.55  | 840000  | 2.6380          |
| 2.7945        | 3.58  | 848000  | 2.6373          |
| 2.7945        | 3.62  | 856000  | 2.6366          |
| 2.7876        | 3.65  | 864000  | 2.6409          |
| 2.7876        | 3.69  | 872000  | 2.6382          |
| 2.7975        | 3.72  | 880000  | 2.6259          |
| 2.7975        | 3.75  | 888000  | 2.6443          |
| 2.7965        | 3.79  | 896000  | 2.6248          |
| 2.7965        | 3.82  | 904000  | 2.6395          |
| 2.7991        | 3.85  | 912000  | 2.6325          |
| 2.7991        | 3.89  | 920000  | 2.6354          |
| 2.7947        | 3.92  | 928000  | 2.6342          |
| 2.7947        | 3.96  | 936000  | 2.6290          |
| 2.7977        | 3.99  | 944000  | 2.6315          |
| 2.7977        | 4.02  | 952000  | 2.6347          |
| 2.8           | 4.06  | 960000  | 2.6318          |
| 2.8           | 4.09  | 968000  | 2.6328          |
| 2.7945        | 4.12  | 976000  | 2.6315          |
| 2.7945        | 4.16  | 984000  | 2.6297          |
| 2.7946        | 4.19  | 992000  | 2.6378          |
| 2.7946        | 4.23  | 1000000 | 2.6328          |
| 2.7962        | 4.26  | 1008000 | 2.6296          |
| 2.7962        | 4.29  | 1016000 | 2.6347          |
| 2.7932        | 4.33  | 1024000 | 2.6355          |
| 2.7932        | 4.36  | 1032000 | 2.6364          |
| 2.7992        | 4.4   | 1040000 | 2.6327          |
| 2.7992        | 4.43  | 1048000 | 2.6273          |
| 2.7922        | 4.46  | 1056000 | 2.6301          |
| 2.7922        | 4.5   | 1064000 | 2.6350          |
| 2.7939        | 4.53  | 1072000 | 2.6358          |
| 2.7939        | 4.56  | 1080000 | nan             |
| 2.789         | 4.6   | 1088000 | 2.6288          |
| 2.789         | 4.63  | 1096000 | 2.6267          |
| 2.7965        | 4.67  | 1104000 | 2.6229          |
| 2.7965        | 4.7   | 1112000 | 2.6331          |
| 2.7963        | 4.73  | 1120000 | 2.6368          |
| 2.7963        | 4.77  | 1128000 | 2.6436          |
| 2.7993        | 4.8   | 1136000 | 2.6363          |
| 2.7993        | 4.83  | 1144000 | 2.6288          |
| 2.7952        | 4.87  | 1152000 | 2.6294          |
| 2.7952        | 4.9   | 1160000 | 2.6337          |
| 2.7972        | 4.94  | 1168000 | 2.6235          |
| 2.7972        | 4.97  | 1176000 | 2.6405          |
| 2.7988        | 5.0   | 1184000 | 2.6266          |
| 2.7988        | 5.04  | 1192000 | 2.6328          |
| 2.7901        | 5.07  | 1200000 | 2.6335          |
| 2.7901        | 5.11  | 1208000 | 2.6405          |
| 2.7975        | 5.14  | 1216000 | 2.6246          |
| 2.7975        | 5.17  | 1224000 | 2.6315          |
| 2.7974        | 5.21  | 1232000 | 2.6390          |
| 2.7974        | 5.24  | 1240000 | 2.6318          |
| 2.7909        | 5.27  | 1248000 | 2.6237          |
| 2.7909        | 5.31  | 1256000 | 2.6343          |
| 2.7899        | 5.34  | 1264000 | 2.6288          |
| 2.7899        | 5.38  | 1272000 | 2.6297          |
| 2.7937        | 5.41  | 1280000 | 2.6343          |
| 2.7937        | 5.44  | 1288000 | 2.6306          |
| 2.7916        | 5.48  | 1296000 | 2.6268          |
| 2.7916        | 5.51  | 1304000 | 2.6317          |
| 2.7874        | 5.54  | 1312000 | 2.6380          |
| 2.7874        | 5.58  | 1320000 | 2.6281          |
| 2.7967        | 5.61  | 1328000 | 2.6334          |
| 2.7967        | 5.65  | 1336000 | 2.6273          |
| 2.791         | 5.68  | 1344000 | 2.6339          |
| 2.791         | 5.71  | 1352000 | 2.6276          |
| 2.791         | 5.75  | 1360000 | 2.6247          |
| 2.791         | 5.78  | 1368000 | 2.6303          |
| 2.7909        | 5.82  | 1376000 | 2.6355          |
| 2.7909        | 5.85  | 1384000 | 2.6352          |
| 2.7833        | 5.88  | 1392000 | 2.6321          |
| 2.7833        | 5.92  | 1400000 | 2.6336          |
| 2.7944        | 5.95  | 1408000 | 2.6312          |
| 2.7944        | 5.98  | 1416000 | 2.6223          |
| 2.8001        | 6.02  | 1424000 | 2.6369          |
| 2.8001        | 6.05  | 1432000 | 2.6299          |
| 2.7954        | 6.09  | 1440000 | 2.6373          |
| 2.7954        | 6.12  | 1448000 | 2.6223          |
| 2.7914        | 6.15  | 1456000 | 2.6225          |
| 2.7914        | 6.19  | 1464000 | 2.6277          |
| 2.7896        | 6.22  | 1472000 | 2.6334          |
| 2.7896        | 6.26  | 1480000 | 2.6260          |
| 2.7925        | 6.29  | 1488000 | 2.6312          |
| 2.7925        | 6.32  | 1496000 | 2.6336          |
| 2.7976        | 6.36  | 1504000 | 2.6270          |
| 2.7976        | 6.39  | 1512000 | 2.6286          |
| 2.8025        | 6.42  | 1520000 | 2.6320          |
| 2.8025        | 6.46  | 1528000 | 2.6252          |
| 2.7953        | 6.49  | 1536000 | 2.6319          |
| 2.7953        | 6.53  | 1544000 | 2.6223          |
| 2.7994        | 6.56  | 1552000 | 2.6358          |
| 2.7994        | 6.59  | 1560000 | 2.6296          |
| 2.7966        | 6.63  | 1568000 | 2.6360          |
| 2.7966        | 6.66  | 1576000 | 2.6327          |
| 2.7883        | 6.69  | 1584000 | 2.6365          |
| 2.7883        | 6.73  | 1592000 | 2.6258          |
| 2.7963        | 6.76  | 1600000 | 2.6401          |
| 2.7963        | 6.8   | 1608000 | 2.6318          |
| 2.7923        | 6.83  | 1616000 | 2.6330          |
| 2.7923        | 6.86  | 1624000 | 2.6372          |
| 2.789         | 6.9   | 1632000 | 2.6363          |
| 2.789         | 6.93  | 1640000 | 2.6346          |
| 2.7883        | 6.97  | 1648000 | 2.6292          |
| 2.7883        | 7.0   | 1656000 | 2.6284          |
| 2.7965        | 7.03  | 1664000 | 2.6408          |
| 2.7965        | 7.07  | 1672000 | 2.6296          |
| 2.7963        | 7.1   | 1680000 | 2.6331          |
| 2.7963        | 7.13  | 1688000 | 2.6339          |
| 2.7911        | 7.17  | 1696000 | 2.6206          |
| 2.7911        | 7.2   | 1704000 | 2.6268          |
| 2.794         | 7.24  | 1712000 | 2.6278          |
| 2.794         | 7.27  | 1720000 | 2.6242          |
| 2.7893        | 7.3   | 1728000 | 2.6329          |
| 2.7893        | 7.34  | 1736000 | 2.6342          |
| 2.7935        | 7.37  | 1744000 | 2.6329          |
| 2.7935        | 7.4   | 1752000 | 2.6294          |
| 2.7936        | 7.44  | 1760000 | 2.6301          |
| 2.7936        | 7.47  | 1768000 | 2.6295          |
| 2.7922        | 7.51  | 1776000 | 2.6261          |
| 2.7922        | 7.54  | 1784000 | 2.6370          |
| 2.7911        | 7.57  | 1792000 | 2.6364          |
| 2.7911        | 7.61  | 1800000 | 2.6232          |
| 2.795         | 7.64  | 1808000 | 2.6201          |
| 2.795         | 7.68  | 1816000 | 2.6329          |
| 2.7898        | 7.71  | 1824000 | 2.6249          |
| 2.7898        | 7.74  | 1832000 | 2.6249          |
| 2.7931        | 7.78  | 1840000 | 2.6361          |
| 2.7931        | 7.81  | 1848000 | nan             |
| 2.7919        | 7.84  | 1856000 | 2.6270          |
| 2.7919        | 7.88  | 1864000 | 2.6362          |
| 2.7833        | 7.91  | 1872000 | 2.6278          |
| 2.7833        | 7.95  | 1880000 | 2.6232          |
| 2.8067        | 7.98  | 1888000 | 2.6260          |
| 2.8067        | 8.01  | 1896000 | 2.6262          |
| 2.7953        | 8.05  | 1904000 | 2.6271          |
| 2.7953        | 8.08  | 1912000 | 2.6270          |
| 2.7953        | 8.11  | 1920000 | 2.6305          |
| 2.7953        | 8.15  | 1928000 | 2.6254          |
| 2.7881        | 8.18  | 1936000 | 2.6297          |
| 2.7881        | 8.22  | 1944000 | 2.6271          |
| 2.7928        | 8.25  | 1952000 | 2.6254          |
| 2.7928        | 8.28  | 1960000 | 2.6286          |
| 2.8003        | 8.32  | 1968000 | 2.6330          |
| 2.8003        | 8.35  | 1976000 | 2.6286          |
| 2.7935        | 8.39  | 1984000 | 2.6408          |
| 2.7935        | 8.42  | 1992000 | 2.6275          |
| 2.7925        | 8.45  | 2000000 | 2.6259          |
| 2.7925        | 8.49  | 2008000 | 2.6302          |
| 2.7924        | 8.52  | 2016000 | 2.6320          |
| 2.7924        | 8.55  | 2024000 | 2.6295          |
| 2.799         | 8.59  | 2032000 | 2.6259          |
| 2.799         | 8.62  | 2040000 | 2.6246          |
| 2.7983        | 8.66  | 2048000 | 2.6295          |
| 2.7983        | 8.69  | 2056000 | 2.6194          |
| 2.7901        | 8.72  | 2064000 | 2.6258          |
| 2.7901        | 8.76  | 2072000 | 2.6334          |
| 2.7956        | 8.79  | 2080000 | 2.6361          |
| 2.7956        | 8.82  | 2088000 | 2.6177          |
| 2.8008        | 8.86  | 2096000 | 2.6322          |
| 2.8008        | 8.89  | 2104000 | 2.6281          |
| 2.791         | 8.93  | 2112000 | 2.6249          |
| 2.791         | 8.96  | 2120000 | 2.6284          |
| 2.7933        | 8.99  | 2128000 | 2.6270          |
| 2.7933        | 9.03  | 2136000 | 2.6241          |
| 2.7825        | 9.06  | 2144000 | 2.6254          |
| 2.7825        | 9.1   | 2152000 | 2.6283          |
| 2.7854        | 9.13  | 2160000 | 2.6343          |
| 2.7854        | 9.16  | 2168000 | 2.6208          |
| 2.7949        | 9.2   | 2176000 | 2.6293          |
| 2.7949        | 9.23  | 2184000 | 2.6266          |
| 2.7938        | 9.26  | 2192000 | 2.6270          |
| 2.7938        | 9.3   | 2200000 | 2.6238          |
| 2.7905        | 9.33  | 2208000 | 2.6282          |
| 2.7905        | 9.37  | 2216000 | 2.6246          |
| 2.8004        | 9.4   | 2224000 | 2.6274          |
| 2.8004        | 9.43  | 2232000 | 2.6252          |
| 2.7921        | 9.47  | 2240000 | 2.6343          |
| 2.7921        | 9.5   | 2248000 | 2.6328          |
| 2.7964        | 9.53  | 2256000 | 2.6206          |
| 2.7964        | 9.57  | 2264000 | 2.6235          |
| 2.7954        | 9.6   | 2272000 | 2.6288          |
| 2.7954        | 9.64  | 2280000 | 2.6204          |
| 2.7902        | 9.67  | 2288000 | 2.6232          |
| 2.7902        | 9.7   | 2296000 | 2.6239          |
| 2.8046        | 9.74  | 2304000 | 2.6241          |
| 2.8046        | 9.77  | 2312000 | 2.6259          |
| 2.793         | 9.81  | 2320000 | 2.6275          |
| 2.793         | 9.84  | 2328000 | 2.6264          |
| 2.7893        | 9.87  | 2336000 | 2.6332          |
| 2.7893        | 9.91  | 2344000 | 2.6214          |
| 2.7898        | 9.94  | 2352000 | 2.6318          |
| 2.7898        | 9.97  | 2360000 | 2.6239          |
| 2.7906        | 10.01 | 2368000 | 2.6215          |
| 2.7906        | 10.04 | 2376000 | 2.6336          |
| 2.7942        | 10.08 | 2384000 | 2.6218          |
| 2.7942        | 10.11 | 2392000 | 2.6299          |
| 2.7997        | 10.14 | 2400000 | 2.6303          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
