transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q2-50p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q2-50p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.5608

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1400
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.03  | 8000    | 2.6645          |
| 2.8656        | 0.07  | 16000   | 2.6465          |
| 2.8656        | 0.1   | 24000   | 2.6186          |
| 2.7946        | 0.13  | 32000   | 2.6235          |
| 2.7946        | 0.17  | 40000   | 2.6151          |
| 2.7911        | 0.2   | 48000   | 2.6128          |
| 2.7911        | 0.24  | 56000   | 2.6010          |
| 2.7898        | 0.27  | 64000   | 2.6144          |
| 2.7898        | 0.3   | 72000   | 2.5976          |
| 2.7791        | 0.34  | 80000   | 2.6006          |
| 2.7791        | 0.37  | 88000   | 2.5889          |
| 2.7776        | 0.4   | 96000   | 2.5888          |
| 2.7776        | 0.44  | 104000  | 2.5842          |
| 2.7702        | 0.47  | 112000  | 2.5760          |
| 2.7702        | 0.51  | 120000  | 2.5720          |
| 2.7661        | 0.54  | 128000  | 2.5710          |
| 2.7661        | 0.57  | 136000  | 2.5673          |
| 2.7609        | 0.61  | 144000  | 2.5693          |
| 2.7609        | 0.64  | 152000  | 2.5623          |
| 2.7557        | 0.67  | 160000  | 2.5559          |
| 2.7557        | 0.71  | 168000  | 2.5650          |
| 2.7584        | 0.74  | 176000  | 2.5584          |
| 2.7584        | 0.77  | 184000  | 2.5591          |
| 2.7619        | 0.81  | 192000  | 2.5597          |
| 2.7619        | 0.84  | 200000  | 2.5650          |
| 2.7678        | 0.88  | 208000  | 2.5728          |
| 2.7678        | 0.91  | 216000  | 2.5712          |
| 2.7735        | 0.94  | 224000  | 2.5729          |
| 2.7735        | 0.98  | 232000  | 2.5755          |
| 2.777         | 1.01  | 240000  | 2.5715          |
| 2.777         | 1.04  | 248000  | 2.5747          |
| 2.7692        | 1.08  | 256000  | 2.5782          |
| 2.7692        | 1.11  | 264000  | 2.5841          |
| 2.7826        | 1.15  | 272000  | 2.5731          |
| 2.7826        | 1.18  | 280000  | 2.5836          |
| 2.7845        | 1.21  | 288000  | 2.5841          |
| 2.7845        | 1.25  | 296000  | 2.5811          |
| 2.7909        | 1.28  | 304000  | 2.5928          |
| 2.7909        | 1.31  | 312000  | 2.5977          |
| 2.7993        | 1.35  | 320000  | 2.6025          |
| 2.7993        | 1.38  | 328000  | 2.6072          |
| 2.8107        | 1.41  | 336000  | 2.6110          |
| 2.8107        | 1.45  | 344000  | 2.6020          |
| 2.8102        | 1.48  | 352000  | 2.6065          |
| 2.8102        | 1.52  | 360000  | 2.6207          |
| 2.8247        | 1.55  | 368000  | 2.6192          |
| 2.8247        | 1.58  | 376000  | 2.6224          |
| 2.8271        | 1.62  | 384000  | 2.6205          |
| 2.8271        | 1.65  | 392000  | 2.6292          |
| 2.8415        | 1.68  | 400000  | 2.6348          |
| 2.8415        | 1.72  | 408000  | 2.6518          |
| 2.842         | 1.75  | 416000  | 2.6465          |
| 2.842         | 1.79  | 424000  | 2.6434          |
| 2.8431        | 1.82  | 432000  | 2.6414          |
| 2.8431        | 1.85  | 440000  | 2.6532          |
| 2.8599        | 1.89  | 448000  | 2.6645          |
| 2.8599        | 1.92  | 456000  | 2.6651          |
| 2.8567        | 1.95  | 464000  | 2.6694          |
| 2.8567        | 1.99  | 472000  | 2.6610          |
| 2.8682        | 2.02  | 480000  | 2.6877          |
| 2.8682        | 2.05  | 488000  | 2.6724          |
| 2.8693        | 2.09  | 496000  | 2.6839          |
| 2.8693        | 2.12  | 504000  | 2.6923          |
| 2.8881        | 2.16  | 512000  | 2.6964          |
| 2.8881        | 2.19  | 520000  | 2.6982          |
| 2.8874        | 2.22  | 528000  | 2.6961          |
| 2.8874        | 2.26  | 536000  | 2.6884          |
| 2.8899        | 2.29  | 544000  | 2.7055          |
| 2.8899        | 2.32  | 552000  | 2.6988          |
| 2.8966        | 2.36  | 560000  | 2.7103          |
| 2.8966        | 2.39  | 568000  | 2.7100          |
| 2.9           | 2.43  | 576000  | 2.7169          |
| 2.9           | 2.46  | 584000  | 2.7180          |
| 2.9237        | 2.49  | 592000  | 2.7270          |
| 2.9237        | 2.53  | 600000  | 2.7265          |
| 2.9236        | 2.56  | 608000  | 2.7323          |
| 2.9236        | 2.59  | 616000  | 2.7350          |
| 2.9276        | 2.63  | 624000  | 2.7333          |
| 2.9276        | 2.66  | 632000  | 2.7345          |
| 2.9252        | 2.69  | 640000  | 2.7497          |
| 2.9252        | 2.73  | 648000  | 2.7428          |
| 2.9364        | 2.76  | 656000  | 2.7392          |
| 2.9364        | 2.8   | 664000  | 2.7505          |
| 2.9366        | 2.83  | 672000  | 2.7393          |
| 2.9366        | 2.86  | 680000  | 2.7372          |
| 2.9437        | 2.9   | 688000  | 2.7451          |
| 2.9437        | 2.93  | 696000  | 2.7488          |
| 2.9483        | 2.96  | 704000  | 2.7586          |
| 2.9483        | 3.0   | 712000  | 2.7613          |
| 2.9588        | 3.03  | 720000  | 2.7619          |
| 2.9588        | 3.07  | 728000  | 2.7680          |
| 2.9422        | 3.1   | 736000  | 2.7546          |
| 2.9422        | 3.13  | 744000  | 2.7629          |
| 2.965         | 3.17  | 752000  | 2.7595          |
| 2.965         | 3.2   | 760000  | 2.7763          |
| 2.959         | 3.23  | 768000  | 2.7739          |
| 2.959         | 3.27  | 776000  | 2.7839          |
| 2.9604        | 3.3   | 784000  | 2.7681          |
| 2.9604        | 3.33  | 792000  | 2.7816          |
| 2.9638        | 3.37  | 800000  | 2.7812          |
| 2.9638        | 3.4   | 808000  | 2.7846          |
| 2.9704        | 3.44  | 816000  | 2.7766          |
| 2.9704        | 3.47  | 824000  | 2.7869          |
| 2.9684        | 3.5   | 832000  | 2.7741          |
| 2.9684        | 3.54  | 840000  | 2.7735          |
| 2.9723        | 3.57  | 848000  | 2.7701          |
| 2.9723        | 3.6   | 856000  | 2.7780          |
| 2.9734        | 3.64  | 864000  | 2.7833          |
| 2.9734        | 3.67  | 872000  | 2.7910          |
| 2.9806        | 3.71  | 880000  | 2.7941          |
| 2.9806        | 3.74  | 888000  | 2.7997          |
| 2.9808        | 3.77  | 896000  | 2.8027          |
| 2.9808        | 3.81  | 904000  | 2.7972          |
| 3.0008        | 3.84  | 912000  | 2.8026          |
| 3.0008        | 3.87  | 920000  | 2.7975          |
| 2.9934        | 3.91  | 928000  | 2.7971          |
| 2.9934        | 3.94  | 936000  | 2.8030          |
| 2.9927        | 3.97  | 944000  | 2.8082          |
| 2.9927        | 4.01  | 952000  | 2.8208          |
| 3.0013        | 4.04  | 960000  | 2.8129          |
| 3.0013        | 4.08  | 968000  | 2.8236          |
| 2.9996        | 4.11  | 976000  | 2.8226          |
| 2.9996        | 4.14  | 984000  | 2.8273          |
| 3.0125        | 4.18  | 992000  | 2.8161          |
| 3.0125        | 4.21  | 1000000 | 2.8249          |
| 3.0086        | 4.24  | 1008000 | 2.8320          |
| 3.0086        | 4.28  | 1016000 | 2.8313          |
| 3.0077        | 4.31  | 1024000 | 2.8321          |
| 3.0077        | 4.35  | 1032000 | 2.8332          |
| 3.0186        | 4.38  | 1040000 | 2.8288          |
| 3.0186        | 4.41  | 1048000 | 2.8392          |
| 3.0311        | 4.45  | 1056000 | 2.8243          |
| 3.0311        | 4.48  | 1064000 | 2.8524          |
| 3.0199        | 4.51  | 1072000 | 2.8347          |
| 3.0199        | 4.55  | 1080000 | 2.8438          |
| 3.0198        | 4.58  | 1088000 | 2.8415          |
| 3.0198        | 4.61  | 1096000 | 2.8460          |
| 3.0279        | 4.65  | 1104000 | 2.8551          |
| 3.0279        | 4.68  | 1112000 | 2.8528          |
| 3.0319        | 4.72  | 1120000 | 2.8601          |
| 3.0319        | 4.75  | 1128000 | 2.8544          |
| 3.0371        | 4.78  | 1136000 | 2.8553          |
| 3.0371        | 4.82  | 1144000 | 2.8597          |
| 3.038         | 4.85  | 1152000 | 2.8653          |
| 3.038         | 4.88  | 1160000 | 2.8560          |
| 3.0318        | 4.92  | 1168000 | 2.8602          |
| 3.0318        | 4.95  | 1176000 | 2.8484          |
| 3.0449        | 4.99  | 1184000 | 2.8612          |
| 3.0449        | 5.02  | 1192000 | 2.8598          |
| 3.0384        | 5.05  | 1200000 | 2.8581          |
| 3.0384        | 5.09  | 1208000 | 2.8481          |
| 3.0243        | 5.12  | 1216000 | 2.8458          |
| 3.0243        | 5.15  | 1224000 | 2.8494          |
| 3.0345        | 5.19  | 1232000 | 2.8544          |
| 3.0345        | 5.22  | 1240000 | 2.8488          |
| 3.0251        | 5.25  | 1248000 | 2.8453          |
| 3.0251        | 5.29  | 1256000 | 2.8464          |
| 3.0234        | 5.32  | 1264000 | 2.8486          |
| 3.0234        | 5.36  | 1272000 | 2.8436          |
| 3.0205        | 5.39  | 1280000 | 2.8476          |
| 3.0205        | 5.42  | 1288000 | 2.8327          |
| 3.0228        | 5.46  | 1296000 | 2.8452          |
| 3.0228        | 5.49  | 1304000 | 2.8372          |
| 3.0063        | 5.52  | 1312000 | 2.8306          |
| 3.0063        | 5.56  | 1320000 | 2.8411          |
| 3.0068        | 5.59  | 1328000 | 2.8273          |
| 3.0068        | 5.63  | 1336000 | 2.8343          |
| 3.0109        | 5.66  | 1344000 | 2.8328          |
| 3.0109        | 5.69  | 1352000 | 2.8431          |
| 3.0068        | 5.73  | 1360000 | 2.8332          |
| 3.0068        | 5.76  | 1368000 | 2.8275          |
| 3.002         | 5.79  | 1376000 | 2.8314          |
| 3.002         | 5.83  | 1384000 | 2.8324          |
| 3.0037        | 5.86  | 1392000 | 2.8394          |
| 3.0037        | 5.89  | 1400000 | 2.8338          |
| 3.0086        | 5.93  | 1408000 | 2.8448          |
| 3.0086        | 5.96  | 1416000 | 2.8326          |
| 2.9977        | 6.0   | 1424000 | 2.8311          |
| 2.9977        | 6.03  | 1432000 | 2.8410          |
| 2.9984        | 6.06  | 1440000 | 2.8359          |
| 2.9984        | 6.1   | 1448000 | 2.8393          |
| 3.0095        | 6.13  | 1456000 | 2.8388          |
| 3.0095        | 6.16  | 1464000 | 2.8448          |
| 3.0051        | 6.2   | 1472000 | 2.8472          |
| 3.0051        | 6.23  | 1480000 | 2.8421          |
| 3.0142        | 6.27  | 1488000 | 2.8424          |
| 3.0142        | 6.3   | 1496000 | 2.8477          |
| 3.0149        | 6.33  | 1504000 | 2.8428          |
| 3.0149        | 6.37  | 1512000 | 2.8529          |
| 3.0147        | 6.4   | 1520000 | 2.8541          |
| 3.0147        | 6.43  | 1528000 | 2.8519          |
| 3.0205        | 6.47  | 1536000 | 2.8527          |
| 3.0205        | 6.5   | 1544000 | 2.8471          |
| 3.029         | 6.53  | 1552000 | 2.8583          |
| 3.029         | 6.57  | 1560000 | 2.8497          |
| 3.024         | 6.6   | 1568000 | 2.8653          |
| 3.024         | 6.64  | 1576000 | 2.8553          |
| 3.0371        | 6.67  | 1584000 | 2.8653          |
| 3.0371        | 6.7   | 1592000 | 2.8604          |
| 3.0319        | 6.74  | 1600000 | 2.8624          |
| 3.0319        | 6.77  | 1608000 | 2.8657          |
| 3.0369        | 6.8   | 1616000 | 2.8616          |
| 3.0369        | 6.84  | 1624000 | 2.8667          |
| 3.0357        | 6.87  | 1632000 | 2.8660          |
| 3.0357        | 6.91  | 1640000 | 2.8682          |
| 3.0342        | 6.94  | 1648000 | 2.8676          |
| 3.0342        | 6.97  | 1656000 | 2.8815          |
| 3.0375        | 7.01  | 1664000 | 2.8667          |
| 3.0375        | 7.04  | 1672000 | 2.8735          |
| 3.0419        | 7.07  | 1680000 | 2.8788          |
| 3.0419        | 7.11  | 1688000 | 2.8767          |
| 3.0403        | 7.14  | 1696000 | 2.8812          |
| 3.0403        | 7.17  | 1704000 | 2.8795          |
| 3.0482        | 7.21  | 1712000 | 2.8805          |
| 3.0482        | 7.24  | 1720000 | 2.8794          |
| 3.0533        | 7.28  | 1728000 | 2.8788          |
| 3.0533        | 7.31  | 1736000 | 2.8844          |
| 3.0453        | 7.34  | 1744000 | 2.8709          |
| 3.0453        | 7.38  | 1752000 | 2.8835          |
| 3.0562        | 7.41  | 1760000 | 2.8891          |
| 3.0562        | 7.44  | 1768000 | 2.8903          |
| 3.0617        | 7.48  | 1776000 | 2.8849          |
| 3.0617        | 7.51  | 1784000 | 2.8766          |
| 3.0539        | 7.55  | 1792000 | 2.8872          |
| 3.0539        | 7.58  | 1800000 | 2.8981          |
| 3.0561        | 7.61  | 1808000 | 2.8862          |
| 3.0561        | 7.65  | 1816000 | 2.8940          |
| 3.0529        | 7.68  | 1824000 | 2.8874          |
| 3.0529        | 7.71  | 1832000 | 2.8839          |
| 3.0484        | 7.75  | 1840000 | 2.8838          |
| 3.0484        | 7.78  | 1848000 | 2.8856          |
| 3.0562        | 7.81  | 1856000 | 2.8984          |
| 3.0562        | 7.85  | 1864000 | 2.8844          |
| 3.0578        | 7.88  | 1872000 | 2.8874          |
| 3.0578        | 7.92  | 1880000 | 2.8887          |
| 3.0553        | 7.95  | 1888000 | 2.8798          |
| 3.0553        | 7.98  | 1896000 | 2.8789          |
| 3.0623        | 8.02  | 1904000 | 2.8968          |
| 3.0623        | 8.05  | 1912000 | 2.8834          |
| 3.0652        | 8.08  | 1920000 | 2.8902          |
| 3.0652        | 8.12  | 1928000 | 2.8822          |
| 3.0487        | 8.15  | 1936000 | 2.8844          |
| 3.0487        | 8.19  | 1944000 | 2.8909          |
| 3.0546        | 8.22  | 1952000 | 2.8915          |
| 3.0546        | 8.25  | 1960000 | 2.8870          |
| 3.0524        | 8.29  | 1968000 | 2.8828          |
| 3.0524        | 8.32  | 1976000 | 2.8781          |
| 3.0491        | 8.35  | 1984000 | 2.8948          |
| 3.0491        | 8.39  | 1992000 | 2.8904          |
| 3.0534        | 8.42  | 2000000 | 2.8839          |
| 3.0534        | 8.45  | 2008000 | 2.8918          |
| 3.0547        | 8.49  | 2016000 | 2.8739          |
| 3.0547        | 8.52  | 2024000 | 2.8684          |
| 3.0544        | 8.56  | 2032000 | 2.8740          |
| 3.0544        | 8.59  | 2040000 | 2.8784          |
| 3.0448        | 8.62  | 2048000 | 2.8758          |
| 3.0448        | 8.66  | 2056000 | 2.8801          |
| 3.0499        | 8.69  | 2064000 | 2.8793          |
| 3.0499        | 8.72  | 2072000 | 2.8707          |
| 3.0368        | 8.76  | 2080000 | 2.8722          |
| 3.0368        | 8.79  | 2088000 | 2.8752          |
| 3.0548        | 8.83  | 2096000 | 2.8880          |
| 3.0548        | 8.86  | 2104000 | 2.8781          |
| 3.0457        | 8.89  | 2112000 | 2.8825          |
| 3.0457        | 8.93  | 2120000 | 2.8827          |
| 3.0377        | 8.96  | 2128000 | 2.8810          |
| 3.0377        | 8.99  | 2136000 | 2.8727          |
| 3.0341        | 9.03  | 2144000 | 2.8750          |
| 3.0341        | 9.06  | 2152000 | 2.8638          |
| 3.0275        | 9.09  | 2160000 | 2.8690          |
| 3.0275        | 9.13  | 2168000 | 2.8660          |
| 3.0413        | 9.16  | 2176000 | 2.8578          |
| 3.0413        | 9.2   | 2184000 | 2.8692          |
| 3.0272        | 9.23  | 2192000 | 2.8702          |
| 3.0272        | 9.26  | 2200000 | 2.8707          |
| 3.034         | 9.3   | 2208000 | 2.8666          |
| 3.034         | 9.33  | 2216000 | 2.8734          |
| 3.0346        | 9.36  | 2224000 | 2.8685          |
| 3.0346        | 9.4   | 2232000 | 2.8675          |
| 3.0234        | 9.43  | 2240000 | 2.8662          |
| 3.0234        | 9.47  | 2248000 | 2.8670          |
| 3.0256        | 9.5   | 2256000 | 2.8764          |
| 3.0256        | 9.53  | 2264000 | 2.8664          |
| 3.0232        | 9.57  | 2272000 | 2.8625          |
| 3.0232        | 9.6   | 2280000 | 2.8647          |
| 3.0309        | 9.63  | 2288000 | 2.8561          |
| 3.0309        | 9.67  | 2296000 | 2.8657          |
| 3.0254        | 9.7   | 2304000 | 2.8667          |
| 3.0254        | 9.73  | 2312000 | 2.8618          |
| 3.0198        | 9.77  | 2320000 | 2.8650          |
| 3.0198        | 9.8   | 2328000 | 2.8630          |
| 3.0109        | 9.84  | 2336000 | 2.8533          |
| 3.0109        | 9.87  | 2344000 | 2.8656          |
| 3.0316        | 9.9   | 2352000 | 2.8607          |
| 3.0316        | 9.94  | 2360000 | 2.8572          |
| 3.0225        | 9.97  | 2368000 | 2.8617          |
| 3.0225        | 10.0  | 2376000 | 2.8604          |
| 3.0132        | 10.04 | 2384000 | 2.8577          |
| 3.0132        | 10.07 | 2392000 | 2.8535          |
| 3.0202        | 10.11 | 2400000 | 2.8566          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
