transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q3-90p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q3-90p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.7327

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.17  | 8000    | 4.1206          |
| 4.294         | 0.34  | 16000   | 4.0057          |
| 4.294         | 0.51  | 24000   | 3.9651          |
| 4.0928        | 0.68  | 32000   | 3.8968          |
| 4.0928        | 0.85  | 40000   | 3.8837          |
| 4.0249        | 1.01  | 48000   | 3.8693          |
| 4.0249        | 1.18  | 56000   | 3.8535          |
| 3.9984        | 1.35  | 64000   | 3.8320          |
| 3.9984        | 1.52  | 72000   | 3.8221          |
| 3.9768        | 1.69  | 80000   | 3.8312          |
| 3.9768        | 1.86  | 88000   | 3.8663          |
| 3.9664        | 2.03  | 96000   | 3.8159          |
| 3.9664        | 2.2   | 104000  | 3.8216          |
| 3.9582        | 2.37  | 112000  | 3.8313          |
| 3.9582        | 2.54  | 120000  | 3.8215          |
| 3.9513        | 2.7   | 128000  | 3.7974          |
| 3.9513        | 2.87  | 136000  | 3.8077          |
| 3.9604        | 3.04  | 144000  | 3.7820          |
| 3.9604        | 3.21  | 152000  | 3.7946          |
| 3.9564        | 3.38  | 160000  | 3.8025          |
| 3.9564        | 3.55  | 168000  | 3.8236          |
| 3.9533        | 3.72  | 176000  | 3.8194          |
| 3.9533        | 3.89  | 184000  | 3.7996          |
| 3.9554        | 4.06  | 192000  | 3.7775          |
| 3.9554        | 4.23  | 200000  | 3.7879          |
| 3.9471        | 4.4   | 208000  | 3.8141          |
| 3.9471        | 4.56  | 216000  | 3.8025          |
| 3.9478        | 4.73  | 224000  | 3.7839          |
| 3.9478        | 4.9   | 232000  | 3.8026          |
| 3.9336        | 5.07  | 240000  | 3.8102          |
| 3.9336        | 5.24  | 248000  | 3.7985          |
| 3.9384        | 5.41  | 256000  | 3.8082          |
| 3.9384        | 5.58  | 264000  | 3.7966          |
| 3.9467        | 5.75  | 272000  | 3.7826          |
| 3.9467        | 5.92  | 280000  | 3.7730          |
| 3.936         | 6.09  | 288000  | 3.7910          |
| 3.936         | 6.26  | 296000  | 3.7904          |
| 3.9438        | 6.42  | 304000  | 3.7918          |
| 3.9438        | 6.59  | 312000  | 3.7773          |
| 3.9386        | 6.76  | 320000  | 3.7665          |
| 3.9386        | 6.93  | 328000  | 3.7803          |
| 3.9365        | 7.1   | 336000  | 3.7851          |
| 3.9365        | 7.27  | 344000  | 3.7787          |
| 3.946         | 7.44  | 352000  | 3.7683          |
| 3.946         | 7.61  | 360000  | 3.7781          |
| 3.9323        | 7.78  | 368000  | 3.7725          |
| 3.9323        | 7.95  | 376000  | 3.7714          |
| 3.9355        | 8.11  | 384000  | 3.7898          |
| 3.9355        | 8.28  | 392000  | 3.7781          |
| 3.9296        | 8.45  | 400000  | 3.7859          |
| 3.9296        | 8.62  | 408000  | 3.7685          |
| 3.9272        | 8.79  | 416000  | 3.7745          |
| 3.9272        | 8.96  | 424000  | 3.7547          |
| 3.9377        | 9.13  | 432000  | 3.7588          |
| 3.9377        | 9.3   | 440000  | 3.7635          |
| 3.9279        | 9.47  | 448000  | 3.7588          |
| 3.9279        | 9.64  | 456000  | 3.7531          |
| 3.9267        | 9.81  | 464000  | 3.7778          |
| 3.9267        | 9.97  | 472000  | 3.7847          |
| 3.9168        | 10.14 | 480000  | 3.7633          |
| 3.9168        | 10.31 | 488000  | 3.7564          |
| 3.9304        | 10.48 | 496000  | 3.7547          |
| 3.9304        | 10.65 | 504000  | 3.7544          |
| 3.9218        | 10.82 | 512000  | 3.7564          |
| 3.9218        | 10.99 | 520000  | 3.7775          |
| 3.9247        | 11.16 | 528000  | 3.7621          |
| 3.9247        | 11.33 | 536000  | 3.7585          |
| 3.9148        | 11.5  | 544000  | 3.7779          |
| 3.9148        | 11.66 | 552000  | 3.7580          |
| 3.9224        | 11.83 | 560000  | 3.7564          |
| 3.9224        | 12.0  | 568000  | 3.7553          |
| 3.9196        | 12.17 | 576000  | 3.7440          |
| 3.9196        | 12.34 | 584000  | 3.7556          |
| 3.9228        | 12.51 | 592000  | 3.7632          |
| 3.9228        | 12.68 | 600000  | 3.7566          |
| 3.926         | 12.85 | 608000  | 3.7429          |
| 3.926         | 13.02 | 616000  | 3.7447          |
| 3.922         | 13.19 | 624000  | 3.7573          |
| 3.922         | 13.36 | 632000  | 3.7698          |
| 3.9269        | 13.52 | 640000  | 3.7592          |
| 3.9269        | 13.69 | 648000  | 3.7665          |
| 3.918         | 13.86 | 656000  | 3.7588          |
| 3.918         | 14.03 | 664000  | 3.7394          |
| 3.9173        | 14.2  | 672000  | 3.7605          |
| 3.9173        | 14.37 | 680000  | 3.7484          |
| 3.9115        | 14.54 | 688000  | 3.7686          |
| 3.9115        | 14.71 | 696000  | 3.7632          |
| 3.9111        | 14.88 | 704000  | 3.7682          |
| 3.9111        | 15.05 | 712000  | 3.7497          |
| 3.9131        | 15.21 | 720000  | 3.7470          |
| 3.9131        | 15.38 | 728000  | 3.7567          |
| 3.9211        | 15.55 | 736000  | 3.7546          |
| 3.9211        | 15.72 | 744000  | 3.7390          |
| 3.9059        | 15.89 | 752000  | 3.7530          |
| 3.9059        | 16.06 | 760000  | 3.7728          |
| 3.9051        | 16.23 | 768000  | 3.7345          |
| 3.9051        | 16.4  | 776000  | 3.7468          |
| 3.9103        | 16.57 | 784000  | 3.7357          |
| 3.9103        | 16.74 | 792000  | 3.7434          |
| 3.9058        | 16.91 | 800000  | 3.7456          |
| 3.9058        | 17.07 | 808000  | 3.7634          |
| 3.9076        | 17.24 | 816000  | 3.7374          |
| 3.9076        | 17.41 | 824000  | 3.7534          |
| 3.9071        | 17.58 | 832000  | 3.7576          |
| 3.9071        | 17.75 | 840000  | 3.7682          |
| 3.8976        | 17.92 | 848000  | 3.7599          |
| 3.8976        | 18.09 | 856000  | 3.7436          |
| 3.9071        | 18.26 | 864000  | 3.7564          |
| 3.9071        | 18.43 | 872000  | 3.7284          |
| 3.9067        | 18.6  | 880000  | 3.7458          |
| 3.9067        | 18.77 | 888000  | 3.7448          |
| 3.914         | 18.93 | 896000  | 3.7490          |
| 3.914         | 19.1  | 904000  | 3.7290          |
| 3.9086        | 19.27 | 912000  | 3.7238          |
| 3.9086        | 19.44 | 920000  | 3.7402          |
| 3.9115        | 19.61 | 928000  | 3.7399          |
| 3.9115        | 19.78 | 936000  | 3.7504          |
| 3.905         | 19.95 | 944000  | 3.7206          |
| 3.905         | 20.12 | 952000  | 3.7351          |
| 3.9012        | 20.29 | 960000  | 3.7409          |
| 3.9012        | 20.46 | 968000  | 3.7426          |
| 3.9043        | 20.62 | 976000  | 3.7349          |
| 3.9043        | 20.79 | 984000  | 3.7376          |
| 3.9104        | 20.96 | 992000  | 3.7325          |
| 3.9104        | 21.13 | 1000000 | 3.7488          |
| 3.9025        | 21.3  | 1008000 | 3.7192          |
| 3.9025        | 21.47 | 1016000 | 3.7437          |
| 3.9131        | 21.64 | 1024000 | 3.7483          |
| 3.9131        | 21.81 | 1032000 | 3.7290          |
| 3.9048        | 21.98 | 1040000 | 3.7547          |
| 3.9048        | 22.15 | 1048000 | 3.7387          |
| 3.9011        | 22.32 | 1056000 | 3.7373          |
| 3.9011        | 22.48 | 1064000 | 3.7222          |
| 3.8988        | 22.65 | 1072000 | 3.7362          |
| 3.8988        | 22.82 | 1080000 | 3.7068          |
| 3.9022        | 22.99 | 1088000 | 3.7358          |
| 3.9022        | 23.16 | 1096000 | 3.7317          |
| 3.8916        | 23.33 | 1104000 | 3.7264          |
| 3.8916        | 23.5  | 1112000 | 3.7487          |
| 3.9079        | 23.67 | 1120000 | 3.7486          |
| 3.9079        | 23.84 | 1128000 | 3.7403          |
| 3.9037        | 24.01 | 1136000 | 3.7345          |
| 3.9037        | 24.17 | 1144000 | 3.7221          |
| 3.8932        | 24.34 | 1152000 | 3.7286          |
| 3.8932        | 24.51 | 1160000 | 3.7309          |
| 3.9055        | 24.68 | 1168000 | 3.7464          |
| 3.9055        | 24.85 | 1176000 | 3.7289          |
| 3.901         | 25.02 | 1184000 | 3.7245          |
| 3.901         | 25.19 | 1192000 | 3.7169          |
| 3.8934        | 25.36 | 1200000 | 3.7176          |
| 3.8934        | 25.53 | 1208000 | 3.7250          |
| 3.9063        | 25.7  | 1216000 | 3.7347          |
| 3.9063        | 25.87 | 1224000 | 3.7445          |
| 3.8937        | 26.03 | 1232000 | 3.7318          |
| 3.8937        | 26.2  | 1240000 | 3.7202          |
| 3.9056        | 26.37 | 1248000 | 3.7427          |
| 3.9056        | 26.54 | 1256000 | 3.7317          |
| 3.8966        | 26.71 | 1264000 | 3.7210          |
| 3.8966        | 26.88 | 1272000 | 3.7238          |
| 3.8888        | 27.05 | 1280000 | 3.7164          |
| 3.8888        | 27.22 | 1288000 | 3.7068          |
| 3.8924        | 27.39 | 1296000 | 3.7498          |
| 3.8924        | 27.56 | 1304000 | 3.7183          |
| 3.8992        | 27.72 | 1312000 | 3.7284          |
| 3.8992        | 27.89 | 1320000 | 3.7397          |
| 3.891         | 28.06 | 1328000 | 3.7212          |
| 3.891         | 28.23 | 1336000 | 3.7249          |
| 3.8947        | 28.4  | 1344000 | 3.7143          |
| 3.8947        | 28.57 | 1352000 | 3.7273          |
| 3.8938        | 28.74 | 1360000 | 3.7317          |
| 3.8938        | 28.91 | 1368000 | 3.7182          |
| 3.8848        | 29.08 | 1376000 | 3.7199          |
| 3.8848        | 29.25 | 1384000 | 3.7309          |
| 3.8907        | 29.42 | 1392000 | 3.7221          |
| 3.8907        | 29.58 | 1400000 | 3.7299          |
| 3.8956        | 29.75 | 1408000 | 3.7257          |
| 3.8956        | 29.92 | 1416000 | 3.7313          |
| 3.8915        | 30.09 | 1424000 | 3.7374          |
| 3.8915        | 30.26 | 1432000 | 3.7146          |
| 3.901         | 30.43 | 1440000 | 3.7190          |
| 3.901         | 30.6  | 1448000 | 3.7211          |
| 3.8901        | 30.77 | 1456000 | 3.7215          |
| 3.8901        | 30.94 | 1464000 | 3.7189          |
| 3.89          | 31.11 | 1472000 | 3.6938          |
| 3.89          | 31.28 | 1480000 | 3.7315          |
| 3.8981        | 31.44 | 1488000 | 3.7328          |
| 3.8981        | 31.61 | 1496000 | 3.6945          |
| 3.8866        | 31.78 | 1504000 | 3.7125          |
| 3.8866        | 31.95 | 1512000 | 3.7229          |
| 3.8943        | 32.12 | 1520000 | 3.7248          |
| 3.8943        | 32.29 | 1528000 | 3.7076          |
| 3.8978        | 32.46 | 1536000 | 3.7276          |
| 3.8978        | 32.63 | 1544000 | 3.7146          |
| 3.8917        | 32.8  | 1552000 | 3.7323          |
| 3.8917        | 32.97 | 1560000 | 3.7308          |
| 3.902         | 33.13 | 1568000 | 3.7290          |
| 3.902         | 33.3  | 1576000 | 3.7153          |
| 3.8942        | 33.47 | 1584000 | 3.7271          |
| 3.8942        | 33.64 | 1592000 | 3.7061          |
| 3.885         | 33.81 | 1600000 | 3.7335          |
| 3.885         | 33.98 | 1608000 | 3.7009          |
| 3.8916        | 34.15 | 1616000 | 3.7184          |
| 3.8916        | 34.32 | 1624000 | 3.7282          |
| 3.8941        | 34.49 | 1632000 | 3.7347          |
| 3.8941        | 34.66 | 1640000 | 3.7183          |
| 3.8923        | 34.83 | 1648000 | 3.7179          |
| 3.8923        | 34.99 | 1656000 | 3.7358          |
| 3.8959        | 35.16 | 1664000 | 3.7375          |
| 3.8959        | 35.33 | 1672000 | 3.7214          |
| 3.8864        | 35.5  | 1680000 | 3.6894          |
| 3.8864        | 35.67 | 1688000 | 3.7273          |
| 3.8882        | 35.84 | 1696000 | 3.7326          |
| 3.8882        | 36.01 | 1704000 | 3.7454          |
| 3.8897        | 36.18 | 1712000 | 3.7255          |
| 3.8897        | 36.35 | 1720000 | 3.7154          |
| 3.8846        | 36.52 | 1728000 | 3.7161          |
| 3.8846        | 36.68 | 1736000 | 3.7116          |
| 3.8768        | 36.85 | 1744000 | 3.6959          |
| 3.8768        | 37.02 | 1752000 | 3.7220          |
| 3.8972        | 37.19 | 1760000 | 3.7378          |
| 3.8972        | 37.36 | 1768000 | 3.7325          |
| 3.8849        | 37.53 | 1776000 | 3.7259          |
| 3.8849        | 37.7  | 1784000 | 3.7256          |
| 3.892         | 37.87 | 1792000 | 3.7198          |
| 3.892         | 38.04 | 1800000 | 3.7202          |
| 3.8947        | 38.21 | 1808000 | 3.7185          |
| 3.8947        | 38.38 | 1816000 | 3.7097          |
| 3.8904        | 38.54 | 1824000 | 3.7317          |
| 3.8904        | 38.71 | 1832000 | 3.7153          |
| 3.8884        | 38.88 | 1840000 | 3.7399          |
| 3.8884        | 39.05 | 1848000 | 3.7260          |
| 3.8815        | 39.22 | 1856000 | 3.7156          |
| 3.8815        | 39.39 | 1864000 | 3.7196          |
| 3.8915        | 39.56 | 1872000 | 3.7313          |
| 3.8915        | 39.73 | 1880000 | 3.7223          |
| 3.8795        | 39.9  | 1888000 | 3.7163          |
| 3.8795        | 40.07 | 1896000 | 3.7089          |
| 3.8865        | 40.23 | 1904000 | 3.7227          |
| 3.8865        | 40.4  | 1912000 | 3.7250          |
| 3.8987        | 40.57 | 1920000 | 3.6921          |
| 3.8987        | 40.74 | 1928000 | 3.7254          |
| 3.8826        | 40.91 | 1936000 | 3.7100          |
| 3.8826        | 41.08 | 1944000 | 3.7356          |
| 3.8911        | 41.25 | 1952000 | 3.7179          |
| 3.8911        | 41.42 | 1960000 | 3.7277          |
| 3.8752        | 41.59 | 1968000 | 3.7068          |
| 3.8752        | 41.76 | 1976000 | 3.7191          |
| 3.8818        | 41.93 | 1984000 | 3.7262          |
| 3.8818        | 42.09 | 1992000 | 3.7295          |
| 3.8774        | 42.26 | 2000000 | 3.7066          |
| 3.8774        | 42.43 | 2008000 | 3.6993          |
| 3.883         | 42.6  | 2016000 | 3.7086          |
| 3.883         | 42.77 | 2024000 | 3.7209          |
| 3.8885        | 42.94 | 2032000 | 3.7046          |
| 3.8885        | 43.11 | 2040000 | 3.6927          |
| 3.8909        | 43.28 | 2048000 | 3.7372          |
| 3.8909        | 43.45 | 2056000 | 3.7064          |
| 3.8904        | 43.62 | 2064000 | 3.7271          |
| 3.8904        | 43.79 | 2072000 | 3.7223          |
| 3.8877        | 43.95 | 2080000 | 3.7141          |
| 3.8877        | 44.12 | 2088000 | 3.6969          |
| 3.8886        | 44.29 | 2096000 | 3.7125          |
| 3.8886        | 44.46 | 2104000 | 3.7268          |
| 3.8832        | 44.63 | 2112000 | 3.7046          |
| 3.8832        | 44.8  | 2120000 | 3.7419          |
| 3.8812        | 44.97 | 2128000 | 3.7163          |
| 3.8812        | 45.14 | 2136000 | 3.7046          |
| 3.8852        | 45.31 | 2144000 | 3.7006          |
| 3.8852        | 45.48 | 2152000 | 3.7212          |
| 3.8864        | 45.64 | 2160000 | 3.7030          |
| 3.8864        | 45.81 | 2168000 | 3.7296          |
| 3.8867        | 45.98 | 2176000 | 3.7194          |
| 3.8867        | 46.15 | 2184000 | 3.7131          |
| 3.8856        | 46.32 | 2192000 | 3.7299          |
| 3.8856        | 46.49 | 2200000 | 3.6991          |
| 3.8843        | 46.66 | 2208000 | 3.7246          |
| 3.8843        | 46.83 | 2216000 | 3.7112          |
| 3.8923        | 47.0  | 2224000 | 3.7293          |
| 3.8923        | 47.17 | 2232000 | 3.7117          |
| 3.8905        | 47.34 | 2240000 | 3.7194          |
| 3.8905        | 47.5  | 2248000 | 3.6994          |
| 3.8818        | 47.67 | 2256000 | 3.7122          |
| 3.8818        | 47.84 | 2264000 | 3.7011          |
| 3.888         | 48.01 | 2272000 | 3.7052          |
| 3.888         | 48.18 | 2280000 | 3.7164          |
| 3.8801        | 48.35 | 2288000 | 3.7082          |
| 3.8801        | 48.52 | 2296000 | 3.7080          |
| 3.8818        | 48.69 | 2304000 | 3.7073          |
| 3.8818        | 48.86 | 2312000 | 3.7132          |
| 3.8889        | 49.03 | 2320000 | 3.7126          |
| 3.8889        | 49.19 | 2328000 | 3.7306          |
| 3.8868        | 49.36 | 2336000 | 3.7130          |
| 3.8868        | 49.53 | 2344000 | 3.7146          |
| 3.8849        | 49.7  | 2352000 | 3.7087          |
| 3.8849        | 49.87 | 2360000 | 3.7074          |
| 3.8807        | 50.04 | 2368000 | 3.7131          |
| 3.8807        | 50.21 | 2376000 | 3.7047          |
| 3.8847        | 50.38 | 2384000 | 3.7149          |
| 3.8847        | 50.55 | 2392000 | 3.7103          |
| 3.8927        | 50.72 | 2400000 | 3.7306          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
