transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q2-90p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q2-90p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.5439

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.17  | 8000    | 4.0640          |
| 4.2654        | 0.34  | 16000   | 3.9414          |
| 4.2654        | 0.51  | 24000   | 3.8956          |
| 4.0459        | 0.67  | 32000   | 3.8527          |
| 4.0459        | 0.84  | 40000   | 3.8232          |
| 3.9781        | 1.01  | 48000   | 3.7806          |
| 3.9781        | 1.18  | 56000   | 3.7861          |
| 3.9323        | 1.35  | 64000   | 3.7930          |
| 3.9323        | 1.52  | 72000   | 3.7814          |
| 3.9224        | 1.68  | 80000   | 3.7815          |
| 3.9224        | 1.85  | 88000   | 3.7403          |
| 3.8924        | 2.02  | 96000   | 3.7468          |
| 3.8924        | 2.19  | 104000  | 3.7400          |
| 3.879         | 2.36  | 112000  | 3.7283          |
| 3.879         | 2.53  | 120000  | 3.7381          |
| 3.8806        | 2.69  | 128000  | 3.7073          |
| 3.8806        | 2.86  | 136000  | 3.7083          |
| 3.8659        | 3.03  | 144000  | 3.6992          |
| 3.8659        | 3.2   | 152000  | 3.6956          |
| 3.8634        | 3.37  | 160000  | 3.6745          |
| 3.8634        | 3.54  | 168000  | 3.7017          |
| 3.8632        | 3.71  | 176000  | 3.6960          |
| 3.8632        | 3.87  | 184000  | 3.7202          |
| 3.8416        | 4.04  | 192000  | 3.7109          |
| 3.8416        | 4.21  | 200000  | 3.6942          |
| 3.8368        | 4.38  | 208000  | 3.6944          |
| 3.8368        | 4.55  | 216000  | 3.6751          |
| 3.8359        | 4.72  | 224000  | 3.6815          |
| 3.8359        | 4.88  | 232000  | 3.6915          |
| 3.8411        | 5.05  | 240000  | 3.6796          |
| 3.8411        | 5.22  | 248000  | 3.6847          |
| 3.8359        | 5.39  | 256000  | 3.6988          |
| 3.8359        | 5.56  | 264000  | 3.6799          |
| 3.8268        | 5.73  | 272000  | 3.6810          |
| 3.8268        | 5.89  | 280000  | 3.6639          |
| 3.8172        | 6.06  | 288000  | 3.6663          |
| 3.8172        | 6.23  | 296000  | 3.6838          |
| 3.8263        | 6.4   | 304000  | 3.6756          |
| 3.8263        | 6.57  | 312000  | 3.6507          |
| 3.8215        | 6.74  | 320000  | 3.6409          |
| 3.8215        | 6.91  | 328000  | 3.6790          |
| 3.8189        | 7.07  | 336000  | 3.6679          |
| 3.8189        | 7.24  | 344000  | 3.6443          |
| 3.8155        | 7.41  | 352000  | 3.6588          |
| 3.8155        | 7.58  | 360000  | 3.6448          |
| 3.8075        | 7.75  | 368000  | 3.6520          |
| 3.8075        | 7.92  | 376000  | 3.6541          |
| 3.8064        | 8.08  | 384000  | 3.6569          |
| 3.8064        | 8.25  | 392000  | 3.6586          |
| 3.8092        | 8.42  | 400000  | 3.6701          |
| 3.8092        | 8.59  | 408000  | 3.6544          |
| 3.8032        | 8.76  | 416000  | 3.6668          |
| 3.8032        | 8.93  | 424000  | 3.6631          |
| 3.8062        | 9.09  | 432000  | 3.6481          |
| 3.8062        | 9.26  | 440000  | 3.6392          |
| 3.7987        | 9.43  | 448000  | 3.6482          |
| 3.7987        | 9.6   | 456000  | 3.6357          |
| 3.7954        | 9.77  | 464000  | 3.6333          |
| 3.7954        | 9.94  | 472000  | 3.6653          |
| 3.7938        | 10.11 | 480000  | 3.6267          |
| 3.7938        | 10.27 | 488000  | 3.6490          |
| 3.7901        | 10.44 | 496000  | 3.6417          |
| 3.7901        | 10.61 | 504000  | 3.6263          |
| 3.7935        | 10.78 | 512000  | 3.6523          |
| 3.7935        | 10.95 | 520000  | 3.6444          |
| 3.7951        | 11.12 | 528000  | 3.6226          |
| 3.7951        | 11.28 | 536000  | 3.6347          |
| 3.7861        | 11.45 | 544000  | 3.6372          |
| 3.7861        | 11.62 | 552000  | 3.6163          |
| 3.7846        | 11.79 | 560000  | 3.6299          |
| 3.7846        | 11.96 | 568000  | 3.6330          |
| 3.7778        | 12.13 | 576000  | 3.6371          |
| 3.7778        | 12.29 | 584000  | 3.6343          |
| 3.777         | 12.46 | 592000  | 3.6242          |
| 3.777         | 12.63 | 600000  | 3.6119          |
| 3.778         | 12.8  | 608000  | 3.6167          |
| 3.778         | 12.97 | 616000  | 3.6191          |
| 3.7795        | 13.14 | 624000  | 3.6225          |
| 3.7795        | 13.3  | 632000  | 3.6056          |
| 3.7766        | 13.47 | 640000  | 3.6135          |
| 3.7766        | 13.64 | 648000  | 3.6169          |
| 3.7729        | 13.81 | 656000  | 3.6035          |
| 3.7729        | 13.98 | 664000  | 3.6109          |
| 3.7846        | 14.15 | 672000  | 3.6180          |
| 3.7846        | 14.32 | 680000  | 3.6171          |
| 3.7726        | 14.48 | 688000  | 3.6182          |
| 3.7726        | 14.65 | 696000  | 3.6086          |
| 3.7717        | 14.82 | 704000  | 3.5852          |
| 3.7717        | 14.99 | 712000  | 3.5883          |
| 3.7713        | 15.16 | 720000  | 3.6056          |
| 3.7713        | 15.33 | 728000  | 3.6004          |
| 3.7745        | 15.49 | 736000  | 3.6059          |
| 3.7745        | 15.66 | 744000  | 3.6156          |
| 3.7557        | 15.83 | 752000  | 3.6029          |
| 3.7557        | 16.0  | 760000  | 3.6099          |
| 3.7628        | 16.17 | 768000  | 3.6016          |
| 3.7628        | 16.34 | 776000  | 3.6008          |
| 3.7717        | 16.5  | 784000  | 3.5972          |
| 3.7717        | 16.67 | 792000  | 3.5838          |
| 3.7616        | 16.84 | 800000  | 3.5868          |
| 3.7616        | 17.01 | 808000  | 3.5834          |
| 3.7608        | 17.18 | 816000  | 3.6066          |
| 3.7608        | 17.35 | 824000  | 3.5911          |
| 3.7625        | 17.52 | 832000  | 3.5997          |
| 3.7625        | 17.68 | 840000  | 3.5855          |
| 3.7634        | 17.85 | 848000  | 3.5861          |
| 3.7634        | 18.02 | 856000  | 3.6021          |
| 3.75          | 18.19 | 864000  | 3.5966          |
| 3.75          | 18.36 | 872000  | 3.5761          |
| 3.7492        | 18.53 | 880000  | 3.5757          |
| 3.7492        | 18.69 | 888000  | 3.6123          |
| 3.7522        | 18.86 | 896000  | 3.5841          |
| 3.7522        | 19.03 | 904000  | 3.5831          |
| 3.7482        | 19.2  | 912000  | 3.5860          |
| 3.7482        | 19.37 | 920000  | 3.5804          |
| 3.75          | 19.54 | 928000  | 3.5730          |
| 3.75          | 19.7  | 936000  | 3.5955          |
| 3.755         | 19.87 | 944000  | 3.5868          |
| 3.755         | 20.04 | 952000  | 3.5992          |
| 3.7549        | 20.21 | 960000  | 3.5657          |
| 3.7549        | 20.38 | 968000  | 3.5780          |
| 3.743         | 20.55 | 976000  | 3.5828          |
| 3.743         | 20.72 | 984000  | 3.5676          |
| 3.75          | 20.88 | 992000  | 3.5724          |
| 3.75          | 21.05 | 1000000 | 3.5850          |
| 3.7483        | 21.22 | 1008000 | 3.5873          |
| 3.7483        | 21.39 | 1016000 | 3.5799          |
| 3.7523        | 21.56 | 1024000 | 3.5974          |
| 3.7523        | 21.73 | 1032000 | 3.5790          |
| 3.7458        | 21.89 | 1040000 | 3.5884          |
| 3.7458        | 22.06 | 1048000 | 3.5904          |
| 3.7498        | 22.23 | 1056000 | 3.5851          |
| 3.7498        | 22.4  | 1064000 | 3.5776          |
| 3.7496        | 22.57 | 1072000 | 3.5685          |
| 3.7496        | 22.74 | 1080000 | 3.5731          |
| 3.7395        | 22.9  | 1088000 | 3.5858          |
| 3.7395        | 23.07 | 1096000 | 3.5931          |
| 3.7466        | 23.24 | 1104000 | 3.5614          |
| 3.7466        | 23.41 | 1112000 | 3.5456          |
| 3.7503        | 23.58 | 1120000 | 3.5895          |
| 3.7503        | 23.75 | 1128000 | 3.5608          |
| 3.7484        | 23.92 | 1136000 | 3.5696          |
| 3.7484        | 24.08 | 1144000 | 3.5653          |
| 3.7435        | 24.25 | 1152000 | 3.5721          |
| 3.7435        | 24.42 | 1160000 | 3.5510          |
| 3.7348        | 24.59 | 1168000 | 3.5631          |
| 3.7348        | 24.76 | 1176000 | 3.5727          |
| 3.7341        | 24.93 | 1184000 | 3.5835          |
| 3.7341        | 25.09 | 1192000 | 3.5766          |
| 3.7435        | 25.26 | 1200000 | 3.5606          |
| 3.7435        | 25.43 | 1208000 | 3.5497          |
| 3.732         | 25.6  | 1216000 | 3.5433          |
| 3.732         | 25.77 | 1224000 | 3.5420          |
| 3.7343        | 25.94 | 1232000 | 3.5987          |
| 3.7343        | 26.1  | 1240000 | 3.5956          |
| 3.7336        | 26.27 | 1248000 | 3.5673          |
| 3.7336        | 26.44 | 1256000 | 3.5643          |
| 3.7444        | 26.61 | 1264000 | 3.5848          |
| 3.7444        | 26.78 | 1272000 | 3.5693          |
| 3.7395        | 26.95 | 1280000 | 3.5745          |
| 3.7395        | 27.12 | 1288000 | 3.5758          |
| 3.7389        | 27.28 | 1296000 | 3.5685          |
| 3.7389        | 27.45 | 1304000 | 3.5712          |
| 3.7416        | 27.62 | 1312000 | 3.5693          |
| 3.7416        | 27.79 | 1320000 | 3.5740          |
| 3.7305        | 27.96 | 1328000 | 3.5803          |
| 3.7305        | 28.13 | 1336000 | 3.5682          |
| 3.7268        | 28.29 | 1344000 | 3.5928          |
| 3.7268        | 28.46 | 1352000 | 3.5608          |
| 3.7363        | 28.63 | 1360000 | 3.5587          |
| 3.7363        | 28.8  | 1368000 | 3.5603          |
| 3.7325        | 28.97 | 1376000 | 3.5711          |
| 3.7325        | 29.14 | 1384000 | 3.5828          |
| 3.7337        | 29.3  | 1392000 | 3.5790          |
| 3.7337        | 29.47 | 1400000 | 3.5795          |
| 3.7367        | 29.64 | 1408000 | 3.5528          |
| 3.7367        | 29.81 | 1416000 | 3.5766          |
| 3.7313        | 29.98 | 1424000 | 3.5610          |
| 3.7313        | 30.15 | 1432000 | 3.5834          |
| 3.7277        | 30.32 | 1440000 | 3.5546          |
| 3.7277        | 30.48 | 1448000 | 3.5534          |
| 3.7296        | 30.65 | 1456000 | 3.5646          |
| 3.7296        | 30.82 | 1464000 | 3.5436          |
| 3.7411        | 30.99 | 1472000 | 3.5778          |
| 3.7411        | 31.16 | 1480000 | 3.5541          |
| 3.7233        | 31.33 | 1488000 | 3.5720          |
| 3.7233        | 31.49 | 1496000 | 3.5567          |
| 3.7291        | 31.66 | 1504000 | 3.5477          |
| 3.7291        | 31.83 | 1512000 | 3.5557          |
| 3.7265        | 32.0  | 1520000 | 3.5643          |
| 3.7265        | 32.17 | 1528000 | 3.5739          |
| 3.7352        | 32.34 | 1536000 | 3.5628          |
| 3.7352        | 32.5  | 1544000 | 3.5542          |
| 3.7353        | 32.67 | 1552000 | 3.5496          |
| 3.7353        | 32.84 | 1560000 | 3.5737          |
| 3.7243        | 33.01 | 1568000 | 3.5788          |
| 3.7243        | 33.18 | 1576000 | 3.5631          |
| 3.7192        | 33.35 | 1584000 | 3.5438          |
| 3.7192        | 33.52 | 1592000 | 3.5554          |
| 3.7266        | 33.68 | 1600000 | 3.5748          |
| 3.7266        | 33.85 | 1608000 | 3.5620          |
| 3.73          | 34.02 | 1616000 | 3.5464          |
| 3.73          | 34.19 | 1624000 | 3.5670          |
| 3.7264        | 34.36 | 1632000 | 3.5626          |
| 3.7264        | 34.53 | 1640000 | 3.5640          |
| 3.7317        | 34.69 | 1648000 | 3.5650          |
| 3.7317        | 34.86 | 1656000 | 3.5458          |
| 3.7332        | 35.03 | 1664000 | 3.5567          |
| 3.7332        | 35.2  | 1672000 | 3.5610          |
| 3.7248        | 35.37 | 1680000 | 3.5650          |
| 3.7248        | 35.54 | 1688000 | 3.5580          |
| 3.7232        | 35.7  | 1696000 | 3.5829          |
| 3.7232        | 35.87 | 1704000 | 3.5532          |
| 3.729         | 36.04 | 1712000 | 3.5723          |
| 3.729         | 36.21 | 1720000 | 3.5454          |
| 3.7273        | 36.38 | 1728000 | 3.5623          |
| 3.7273        | 36.55 | 1736000 | 3.5462          |
| 3.7261        | 36.72 | 1744000 | 3.5743          |
| 3.7261        | 36.88 | 1752000 | 3.5638          |
| 3.7208        | 37.05 | 1760000 | 3.5519          |
| 3.7208        | 37.22 | 1768000 | 3.5584          |
| 3.7183        | 37.39 | 1776000 | 3.5308          |
| 3.7183        | 37.56 | 1784000 | 3.5549          |
| 3.7193        | 37.73 | 1792000 | 3.5409          |
| 3.7193        | 37.89 | 1800000 | 3.5396          |
| 3.7271        | 38.06 | 1808000 | 3.5536          |
| 3.7271        | 38.23 | 1816000 | 3.5452          |
| 3.7284        | 38.4  | 1824000 | 3.5582          |
| 3.7284        | 38.57 | 1832000 | 3.5668          |
| 3.714         | 38.74 | 1840000 | 3.5673          |
| 3.714         | 38.9  | 1848000 | 3.5477          |
| 3.7105        | 39.07 | 1856000 | 3.5662          |
| 3.7105        | 39.24 | 1864000 | 3.5498          |
| 3.7189        | 39.41 | 1872000 | 3.5493          |
| 3.7189        | 39.58 | 1880000 | 3.5676          |
| 3.7203        | 39.75 | 1888000 | 3.5640          |
| 3.7203        | 39.91 | 1896000 | 3.5747          |
| 3.7271        | 40.08 | 1904000 | 3.5592          |
| 3.7271        | 40.25 | 1912000 | 3.5515          |
| 3.7237        | 40.42 | 1920000 | 3.5704          |
| 3.7237        | 40.59 | 1928000 | 3.5642          |
| 3.723         | 40.76 | 1936000 | 3.5300          |
| 3.723         | 40.93 | 1944000 | 3.5482          |
| 3.7224        | 41.09 | 1952000 | 3.5586          |
| 3.7224        | 41.26 | 1960000 | 3.5463          |
| 3.715         | 41.43 | 1968000 | 3.5323          |
| 3.715         | 41.6  | 1976000 | 3.5426          |
| 3.7209        | 41.77 | 1984000 | 3.5513          |
| 3.7209        | 41.94 | 1992000 | 3.5614          |
| 3.7183        | 42.1  | 2000000 | 3.5678          |
| 3.7183        | 42.27 | 2008000 | 3.5304          |
| 3.7161        | 42.44 | 2016000 | 3.5631          |
| 3.7161        | 42.61 | 2024000 | 3.5589          |
| 3.7215        | 42.78 | 2032000 | 3.5639          |
| 3.7215        | 42.95 | 2040000 | 3.5376          |
| 3.7205        | 43.11 | 2048000 | 3.5478          |
| 3.7205        | 43.28 | 2056000 | 3.5511          |
| 3.7178        | 43.45 | 2064000 | 3.5285          |
| 3.7178        | 43.62 | 2072000 | 3.5428          |
| 3.7232        | 43.79 | 2080000 | 3.5347          |
| 3.7232        | 43.96 | 2088000 | 3.5501          |
| 3.7167        | 44.13 | 2096000 | 3.5422          |
| 3.7167        | 44.29 | 2104000 | 3.5487          |
| 3.7253        | 44.46 | 2112000 | 3.5540          |
| 3.7253        | 44.63 | 2120000 | 3.5432          |
| 3.7139        | 44.8  | 2128000 | 3.5502          |
| 3.7139        | 44.97 | 2136000 | 3.5450          |
| 3.7194        | 45.14 | 2144000 | 3.5564          |
| 3.7194        | 45.3  | 2152000 | 3.5441          |
| 3.7167        | 45.47 | 2160000 | 3.5549          |
| 3.7167        | 45.64 | 2168000 | 3.5429          |
| 3.7202        | 45.81 | 2176000 | 3.5613          |
| 3.7202        | 45.98 | 2184000 | 3.5469          |
| 3.7193        | 46.15 | 2192000 | 3.5467          |
| 3.7193        | 46.31 | 2200000 | 3.5493          |
| 3.717         | 46.48 | 2208000 | 3.5652          |
| 3.717         | 46.65 | 2216000 | 3.5669          |
| 3.7164        | 46.82 | 2224000 | 3.5755          |
| 3.7164        | 46.99 | 2232000 | 3.5580          |
| 3.715         | 47.16 | 2240000 | 3.5403          |
| 3.715         | 47.33 | 2248000 | 3.5521          |
| 3.7091        | 47.49 | 2256000 | 3.5604          |
| 3.7091        | 47.66 | 2264000 | 3.5401          |
| 3.7199        | 47.83 | 2272000 | 3.5408          |
| 3.7199        | 48.0  | 2280000 | 3.5509          |
| 3.7238        | 48.17 | 2288000 | 3.5348          |
| 3.7238        | 48.34 | 2296000 | 3.5530          |
| 3.7193        | 48.5  | 2304000 | 3.5447          |
| 3.7193        | 48.67 | 2312000 | 3.5453          |
| 3.7195        | 48.84 | 2320000 | 3.5487          |
| 3.7195        | 49.01 | 2328000 | 3.5357          |
| 3.7187        | 49.18 | 2336000 | 3.5404          |
| 3.7187        | 49.35 | 2344000 | 3.5247          |
| 3.7157        | 49.51 | 2352000 | 3.5557          |
| 3.7157        | 49.68 | 2360000 | 3.5532          |
| 3.7144        | 49.85 | 2368000 | 3.5453          |
| 3.7144        | 50.02 | 2376000 | 3.5421          |
| 3.715         | 50.19 | 2384000 | 3.5183          |
| 3.715         | 50.36 | 2392000 | 3.5473          |
| 3.7208        | 50.53 | 2400000 | 3.5386          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
