transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q1-90p-filtered-random
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q1-90p-filtered-random

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9012

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.16  | 8000    | 2.2376          |
| 2.4394        | 0.33  | 16000   | 2.1557          |
| 2.4394        | 0.49  | 24000   | 2.0965          |
| 2.2403        | 0.66  | 32000   | 2.0637          |
| 2.2403        | 0.82  | 40000   | 2.0620          |
| 2.1859        | 0.98  | 48000   | 2.0427          |
| 2.1859        | 1.15  | 56000   | 2.0440          |
| 2.1472        | 1.31  | 64000   | 2.0177          |
| 2.1472        | 1.48  | 72000   | 1.9980          |
| 2.1334        | 1.64  | 80000   | 2.0021          |
| 2.1334        | 1.8   | 88000   | 1.9963          |
| 2.1271        | 1.97  | 96000   | 1.9918          |
| 2.1271        | 2.13  | 104000  | 1.9889          |
| 2.1065        | 2.29  | 112000  | 1.9689          |
| 2.1065        | 2.46  | 120000  | 1.9919          |
| 2.105         | 2.62  | 128000  | 1.9706          |
| 2.105         | 2.79  | 136000  | 1.9725          |
| 2.1033        | 2.95  | 144000  | 2.0009          |
| 2.1033        | 3.11  | 152000  | 1.9661          |
| 2.0934        | 3.28  | 160000  | 1.9641          |
| 2.0934        | 3.44  | 168000  | 1.9733          |
| 2.0899        | 3.61  | 176000  | 1.9747          |
| 2.0899        | 3.77  | 184000  | 1.9442          |
| 2.0903        | 3.93  | 192000  | 1.9586          |
| 2.0903        | 4.1   | 200000  | 1.9586          |
| 2.0842        | 4.26  | 208000  | 1.9402          |
| 2.0842        | 4.43  | 216000  | 1.9483          |
| 2.0761        | 4.59  | 224000  | 1.9532          |
| 2.0761        | 4.75  | 232000  | 1.9456          |
| 2.0799        | 4.92  | 240000  | 1.9322          |
| 2.0799        | 5.08  | 248000  | 1.9460          |
| 2.0704        | 5.24  | 256000  | 1.9478          |
| 2.0704        | 5.41  | 264000  | 1.9435          |
| 2.0727        | 5.57  | 272000  | 1.9356          |
| 2.0727        | 5.74  | 280000  | 1.9543          |
| 2.073         | 5.9   | 288000  | 1.9542          |
| 2.073         | 6.06  | 296000  | 1.9503          |
| 2.0647        | 6.23  | 304000  | 1.9437          |
| 2.0647        | 6.39  | 312000  | 1.9450          |
| 2.0668        | 6.56  | 320000  | 1.9221          |
| 2.0668        | 6.72  | 328000  | 1.9277          |
| 2.0695        | 6.88  | 336000  | 1.9357          |
| 2.0695        | 7.05  | 344000  | 1.9244          |
| 2.0652        | 7.21  | 352000  | 1.9387          |
| 2.0652        | 7.38  | 360000  | 1.9355          |
| 2.0607        | 7.54  | 368000  | 1.9390          |
| 2.0607        | 7.7   | 376000  | 1.9359          |
| 2.0707        | 7.87  | 384000  | 1.9393          |
| 2.0707        | 8.03  | 392000  | 1.9217          |
| 2.0621        | 8.2   | 400000  | 1.9284          |
| 2.0621        | 8.36  | 408000  | 1.9383          |
| 2.0643        | 8.52  | 416000  | 1.9224          |
| 2.0643        | 8.69  | 424000  | 1.9284          |
| 2.0498        | 8.85  | 432000  | 1.9399          |
| 2.0498        | 9.01  | 440000  | 1.9427          |
| 2.0689        | 9.18  | 448000  | 1.9351          |
| 2.0689        | 9.34  | 456000  | 1.9313          |
| 2.0511        | 9.51  | 464000  | 1.9392          |
| 2.0511        | 9.67  | 472000  | 1.9278          |
| 2.0664        | 9.83  | 480000  | 1.9385          |
| 2.0664        | 10.0  | 488000  | 1.9472          |
| 2.0565        | 10.16 | 496000  | 1.9377          |
| 2.0565        | 10.33 | 504000  | 1.9481          |
| 2.0566        | 10.49 | 512000  | 1.9454          |
| 2.0566        | 10.65 | 520000  | 1.9244          |
| 2.0523        | 10.82 | 528000  | 1.9358          |
| 2.0523        | 10.98 | 536000  | 1.9176          |
| 2.0554        | 11.15 | 544000  | 1.9284          |
| 2.0554        | 11.31 | 552000  | 1.9287          |
| 2.0485        | 11.47 | 560000  | 1.9237          |
| 2.0485        | 11.64 | 568000  | 1.9209          |
| 2.0485        | 11.8  | 576000  | 1.9262          |
| 2.0485        | 11.97 | 584000  | 1.9208          |
| 2.0542        | 12.13 | 592000  | 1.9320          |
| 2.0542        | 12.29 | 600000  | 1.9077          |
| 2.0527        | 12.46 | 608000  | 1.9249          |
| 2.0527        | 12.62 | 616000  | 1.9192          |
| 2.0606        | 12.78 | 624000  | 1.9152          |
| 2.0606        | 12.95 | 632000  | 1.9194          |
| 2.0542        | 13.11 | 640000  | 1.9198          |
| 2.0542        | 13.28 | 648000  | 1.9135          |
| 2.0593        | 13.44 | 656000  | 1.9192          |
| 2.0593        | 13.6  | 664000  | 1.9257          |
| 2.0467        | 13.77 | 672000  | 1.9135          |
| 2.0467        | 13.93 | 680000  | 1.8995          |
| 2.0535        | 14.1  | 688000  | 1.9306          |
| 2.0535        | 14.26 | 696000  | 1.9278          |
| 2.0559        | 14.42 | 704000  | 1.9137          |
| 2.0559        | 14.59 | 712000  | 1.9165          |
| 2.0544        | 14.75 | 720000  | 1.9219          |
| 2.0544        | 14.92 | 728000  | 1.9200          |
| 2.0493        | 15.08 | 736000  | 1.9242          |
| 2.0493        | 15.24 | 744000  | 1.9264          |
| 2.0538        | 15.41 | 752000  | 1.9333          |
| 2.0538        | 15.57 | 760000  | 1.9126          |
| 2.0457        | 15.73 | 768000  | 1.9023          |
| 2.0457        | 15.9  | 776000  | 1.9154          |
| 2.0436        | 16.06 | 784000  | 1.8947          |
| 2.0436        | 16.23 | 792000  | 1.9198          |
| 2.0527        | 16.39 | 800000  | 1.9125          |
| 2.0527        | 16.55 | 808000  | 1.9129          |
| 2.0484        | 16.72 | 816000  | 1.9302          |
| 2.0484        | 16.88 | 824000  | 1.9152          |
| 2.0535        | 17.05 | 832000  | 1.9223          |
| 2.0535        | 17.21 | 840000  | 1.9195          |
| 2.0516        | 17.37 | 848000  | 1.9072          |
| 2.0516        | 17.54 | 856000  | 1.9210          |
| 2.0546        | 17.7  | 864000  | 1.9014          |
| 2.0546        | 17.87 | 872000  | 1.9130          |
| 2.0439        | 18.03 | 880000  | 1.9064          |
| 2.0439        | 18.19 | 888000  | 1.9166          |
| 2.0391        | 18.36 | 896000  | 1.9310          |
| 2.0391        | 18.52 | 904000  | 1.9064          |
| 2.0568        | 18.69 | 912000  | 1.9107          |
| 2.0568        | 18.85 | 920000  | 1.9317          |
| 2.047         | 19.01 | 928000  | 1.9235          |
| 2.047         | 19.18 | 936000  | 1.9173          |
| 2.0431        | 19.34 | 944000  | 1.8946          |
| 2.0431        | 19.5  | 952000  | 1.9247          |
| 2.0444        | 19.67 | 960000  | 1.9106          |
| 2.0444        | 19.83 | 968000  | 1.9023          |
| 2.0465        | 20.0  | 976000  | 1.9196          |
| 2.0465        | 20.16 | 984000  | 1.9112          |
| 2.0454        | 20.32 | 992000  | 1.9046          |
| 2.0454        | 20.49 | 1000000 | 1.9032          |
| 2.04          | 20.65 | 1008000 | 1.9130          |
| 2.04          | 20.82 | 1016000 | 1.9223          |
| 2.0406        | 20.98 | 1024000 | 1.9261          |
| 2.0406        | 21.14 | 1032000 | 1.9014          |
| 2.0401        | 21.31 | 1040000 | 1.9052          |
| 2.0401        | 21.47 | 1048000 | 1.9005          |
| 2.044         | 21.64 | 1056000 | 1.9044          |
| 2.044         | 21.8  | 1064000 | 1.9170          |
| 2.0401        | 21.96 | 1072000 | 1.9103          |
| 2.0401        | 22.13 | 1080000 | 1.8971          |
| 2.0458        | 22.29 | 1088000 | 1.9257          |
| 2.0458        | 22.45 | 1096000 | 1.9029          |
| 2.0414        | 22.62 | 1104000 | 1.9150          |
| 2.0414        | 22.78 | 1112000 | 1.9124          |
| 2.0419        | 22.95 | 1120000 | 1.9030          |
| 2.0419        | 23.11 | 1128000 | 1.9145          |
| 2.0415        | 23.27 | 1136000 | 1.9132          |
| 2.0415        | 23.44 | 1144000 | 1.9054          |
| 2.0394        | 23.6  | 1152000 | 1.9155          |
| 2.0394        | 23.77 | 1160000 | 1.9147          |
| 2.0414        | 23.93 | 1168000 | 1.9130          |
| 2.0414        | 24.09 | 1176000 | 1.9002          |
| 2.036         | 24.26 | 1184000 | 1.8991          |
| 2.036         | 24.42 | 1192000 | 1.9203          |
| 2.0393        | 24.59 | 1200000 | 1.9327          |
| 2.0393        | 24.75 | 1208000 | 1.9099          |
| 2.0375        | 24.91 | 1216000 | 1.9097          |
| 2.0375        | 25.08 | 1224000 | 1.9006          |
| 2.0384        | 25.24 | 1232000 | 1.9063          |
| 2.0384        | 25.41 | 1240000 | 1.9057          |
| 2.0392        | 25.57 | 1248000 | 1.9037          |
| 2.0392        | 25.73 | 1256000 | 1.9013          |
| 2.0503        | 25.9  | 1264000 | 1.9037          |
| 2.0503        | 26.06 | 1272000 | 1.9042          |
| 2.0418        | 26.22 | 1280000 | 1.8966          |
| 2.0418        | 26.39 | 1288000 | 1.9187          |
| 2.0416        | 26.55 | 1296000 | 1.9098          |
| 2.0416        | 26.72 | 1304000 | 1.9153          |
| 2.0396        | 26.88 | 1312000 | 1.9164          |
| 2.0396        | 27.04 | 1320000 | 1.8867          |
| 2.0397        | 27.21 | 1328000 | 1.8969          |
| 2.0397        | 27.37 | 1336000 | 1.9155          |
| 2.0442        | 27.54 | 1344000 | 1.9004          |
| 2.0442        | 27.7  | 1352000 | 1.9027          |
| 2.0332        | 27.86 | 1360000 | 1.9095          |
| 2.0332        | 28.03 | 1368000 | 1.9134          |
| 2.0398        | 28.19 | 1376000 | 1.9083          |
| 2.0398        | 28.36 | 1384000 | 1.9041          |
| 2.0387        | 28.52 | 1392000 | 1.8980          |
| 2.0387        | 28.68 | 1400000 | 1.9209          |
| 2.0378        | 28.85 | 1408000 | 1.8962          |
| 2.0378        | 29.01 | 1416000 | 1.8981          |
| 2.0359        | 29.17 | 1424000 | 1.9078          |
| 2.0359        | 29.34 | 1432000 | 1.8962          |
| 2.0357        | 29.5  | 1440000 | 1.8843          |
| 2.0357        | 29.67 | 1448000 | 1.9157          |
| 2.0367        | 29.83 | 1456000 | 1.9278          |
| 2.0367        | 29.99 | 1464000 | 1.9009          |
| 2.0442        | 30.16 | 1472000 | 1.8969          |
| 2.0442        | 30.32 | 1480000 | 1.9086          |
| 2.0401        | 30.49 | 1488000 | 1.9059          |
| 2.0401        | 30.65 | 1496000 | 1.8997          |
| 2.0293        | 30.81 | 1504000 | 1.9014          |
| 2.0293        | 30.98 | 1512000 | 1.8971          |
| 2.035         | 31.14 | 1520000 | 1.9114          |
| 2.035         | 31.31 | 1528000 | 1.9108          |
| 2.0389        | 31.47 | 1536000 | 1.8971          |
| 2.0389        | 31.63 | 1544000 | 1.9082          |
| 2.0346        | 31.8  | 1552000 | 1.9209          |
| 2.0346        | 31.96 | 1560000 | 1.9018          |
| 2.0428        | 32.13 | 1568000 | 1.8988          |
| 2.0428        | 32.29 | 1576000 | 1.9089          |
| 2.0286        | 32.45 | 1584000 | 1.8983          |
| 2.0286        | 32.62 | 1592000 | 1.9030          |
| 2.037         | 32.78 | 1600000 | 1.9042          |
| 2.037         | 32.94 | 1608000 | 1.9022          |
| 2.0348        | 33.11 | 1616000 | 1.8988          |
| 2.0348        | 33.27 | 1624000 | 1.9159          |
| 2.042         | 33.44 | 1632000 | 1.8934          |
| 2.042         | 33.6  | 1640000 | 1.8908          |
| 2.0426        | 33.76 | 1648000 | 1.8878          |
| 2.0426        | 33.93 | 1656000 | 1.8882          |
| 2.0293        | 34.09 | 1664000 | 1.9031          |
| 2.0293        | 34.26 | 1672000 | 1.9006          |
| 2.0401        | 34.42 | 1680000 | 1.9066          |
| 2.0401        | 34.58 | 1688000 | 1.8970          |
| 2.0315        | 34.75 | 1696000 | 1.8953          |
| 2.0315        | 34.91 | 1704000 | 1.9030          |
| 2.0393        | 35.08 | 1712000 | 1.9089          |
| 2.0393        | 35.24 | 1720000 | 1.9037          |
| 2.0422        | 35.4  | 1728000 | 1.8991          |
| 2.0422        | 35.57 | 1736000 | 1.8826          |
| 2.0307        | 35.73 | 1744000 | 1.9027          |
| 2.0307        | 35.9  | 1752000 | 1.9173          |
| 2.0414        | 36.06 | 1760000 | 1.9065          |
| 2.0414        | 36.22 | 1768000 | 1.8985          |
| 2.0393        | 36.39 | 1776000 | 1.8849          |
| 2.0393        | 36.55 | 1784000 | 1.8945          |
| 2.0302        | 36.71 | 1792000 | 1.8962          |
| 2.0302        | 36.88 | 1800000 | 1.9015          |
| 2.0344        | 37.04 | 1808000 | 1.8924          |
| 2.0344        | 37.21 | 1816000 | 1.8804          |
| 2.0303        | 37.37 | 1824000 | 1.8947          |
| 2.0303        | 37.53 | 1832000 | 1.8917          |
| 2.0318        | 37.7  | 1840000 | 1.8992          |
| 2.0318        | 37.86 | 1848000 | 1.9018          |
| 2.039         | 38.03 | 1856000 | 1.8903          |
| 2.039         | 38.19 | 1864000 | 1.9089          |
| 2.0319        | 38.35 | 1872000 | 1.9093          |
| 2.0319        | 38.52 | 1880000 | 1.8951          |
| 2.0359        | 38.68 | 1888000 | 1.8970          |
| 2.0359        | 38.85 | 1896000 | 1.8995          |
| 2.0353        | 39.01 | 1904000 | 1.8888          |
| 2.0353        | 39.17 | 1912000 | 1.9149          |
| 2.0343        | 39.34 | 1920000 | 1.8881          |
| 2.0343        | 39.5  | 1928000 | 1.8935          |
| 2.0395        | 39.66 | 1936000 | 1.8938          |
| 2.0395        | 39.83 | 1944000 | 1.8929          |
| 2.0316        | 39.99 | 1952000 | 1.9186          |
| 2.0316        | 40.16 | 1960000 | 1.9189          |
| 2.0302        | 40.32 | 1968000 | 1.9125          |
| 2.0302        | 40.48 | 1976000 | 1.9078          |
| 2.0355        | 40.65 | 1984000 | 1.8975          |
| 2.0355        | 40.81 | 1992000 | 1.8929          |
| 2.0332        | 40.98 | 2000000 | 1.8899          |
| 2.0332        | 41.14 | 2008000 | 1.9043          |
| 2.0327        | 41.3  | 2016000 | 1.9086          |
| 2.0327        | 41.47 | 2024000 | 1.8944          |
| 2.0414        | 41.63 | 2032000 | 1.9029          |
| 2.0414        | 41.8  | 2040000 | 1.8990          |
| 2.0327        | 41.96 | 2048000 | 1.9175          |
| 2.0327        | 42.12 | 2056000 | 1.8888          |
| 2.0428        | 42.29 | 2064000 | 1.8971          |
| 2.0428        | 42.45 | 2072000 | 1.9071          |
| 2.0337        | 42.62 | 2080000 | 1.8968          |
| 2.0337        | 42.78 | 2088000 | 1.9060          |
| 2.0394        | 42.94 | 2096000 | 1.9028          |
| 2.0394        | 43.11 | 2104000 | 1.8917          |
| 2.0314        | 43.27 | 2112000 | 1.8868          |
| 2.0314        | 43.43 | 2120000 | 1.9087          |
| 2.0368        | 43.6  | 2128000 | 1.8951          |
| 2.0368        | 43.76 | 2136000 | 1.8938          |
| 2.0298        | 43.93 | 2144000 | 1.8874          |
| 2.0298        | 44.09 | 2152000 | 1.9065          |
| 2.0353        | 44.25 | 2160000 | 1.9097          |
| 2.0353        | 44.42 | 2168000 | 1.8985          |
| 2.0324        | 44.58 | 2176000 | 1.9160          |
| 2.0324        | 44.75 | 2184000 | 1.9060          |
| 2.0316        | 44.91 | 2192000 | 1.8912          |
| 2.0316        | 45.07 | 2200000 | 1.9014          |
| 2.0322        | 45.24 | 2208000 | 1.9031          |
| 2.0322        | 45.4  | 2216000 | 1.9086          |
| 2.035         | 45.57 | 2224000 | 1.9147          |
| 2.035         | 45.73 | 2232000 | 1.9004          |
| 2.0431        | 45.89 | 2240000 | 1.9027          |
| 2.0431        | 46.06 | 2248000 | 1.8916          |
| 2.0347        | 46.22 | 2256000 | 1.9049          |
| 2.0347        | 46.38 | 2264000 | 1.8973          |
| 2.0353        | 46.55 | 2272000 | 1.8932          |
| 2.0353        | 46.71 | 2280000 | 1.9121          |
| 2.0309        | 46.88 | 2288000 | 1.8973          |
| 2.0309        | 47.04 | 2296000 | 1.8986          |
| 2.0359        | 47.2  | 2304000 | 1.9202          |
| 2.0359        | 47.37 | 2312000 | 1.8778          |
| 2.037         | 47.53 | 2320000 | 1.9137          |
| 2.037         | 47.7  | 2328000 | 1.9095          |
| 2.0306        | 47.86 | 2336000 | 1.9040          |
| 2.0306        | 48.02 | 2344000 | 1.8932          |
| 2.0342        | 48.19 | 2352000 | 1.9152          |
| 2.0342        | 48.35 | 2360000 | 1.9059          |
| 2.0457        | 48.52 | 2368000 | 1.8845          |
| 2.0457        | 48.68 | 2376000 | 1.9040          |
| 2.0349        | 48.84 | 2384000 | 1.9000          |
| 2.0349        | 49.01 | 2392000 | 1.9059          |
| 2.0322        | 49.17 | 2400000 | 1.8926          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
