transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q1-90p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q1-90p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.2574

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1400
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.16  | 8000    | 3.4495          |
| 3.5684        | 0.33  | 16000   | 3.4166          |
| 3.5684        | 0.49  | 24000   | 3.3847          |
| 3.3755        | 0.66  | 32000   | 3.3665          |
| 3.3755        | 0.82  | 40000   | 3.3654          |
| 3.3533        | 0.98  | 48000   | 3.3654          |
| 3.3533        | 1.15  | 56000   | 3.3328          |
| 3.3014        | 1.31  | 64000   | 3.3210          |
| 3.3014        | 1.48  | 72000   | 3.3492          |
| 3.2888        | 1.64  | 80000   | 3.3213          |
| 3.2888        | 1.8   | 88000   | 3.2708          |
| 3.2609        | 1.97  | 96000   | 3.2908          |
| 3.2609        | 2.13  | 104000  | 3.2767          |
| 3.2159        | 2.29  | 112000  | 3.2592          |
| 3.2159        | 2.46  | 120000  | 3.2411          |
| 3.2167        | 2.62  | 128000  | 3.2377          |
| 3.2167        | 2.79  | 136000  | 3.2485          |
| 3.199         | 2.95  | 144000  | 3.2609          |
| 3.199         | 3.11  | 152000  | 3.2553          |
| 3.1905        | 3.28  | 160000  | 3.2425          |
| 3.1905        | 3.44  | 168000  | 3.2422          |
| 3.1822        | 3.61  | 176000  | 3.2624          |
| 3.1822        | 3.77  | 184000  | 3.2507          |
| 3.1852        | 3.93  | 192000  | 3.2483          |
| 3.1852        | 4.1   | 200000  | 3.2514          |
| 3.1767        | 4.26  | 208000  | 3.2426          |
| 3.1767        | 4.43  | 216000  | 3.2348          |
| 3.1767        | 4.59  | 224000  | 3.2735          |
| 3.1767        | 4.75  | 232000  | 3.2472          |
| 3.1973        | 4.92  | 240000  | 3.2596          |
| 3.1973        | 5.08  | 248000  | 3.2606          |
| 3.1781        | 5.24  | 256000  | 3.2815          |
| 3.1781        | 5.41  | 264000  | 3.2734          |
| 3.1803        | 5.57  | 272000  | 3.2739          |
| 3.1803        | 5.74  | 280000  | 3.2712          |
| 3.1989        | 5.9   | 288000  | 3.2734          |
| 3.1989        | 6.06  | 296000  | 3.2939          |
| 3.1929        | 6.23  | 304000  | 3.2880          |
| 3.1929        | 6.39  | 312000  | 3.2894          |
| 3.2083        | 6.56  | 320000  | 3.3086          |
| 3.2083        | 6.72  | 328000  | 3.3067          |
| 3.2013        | 6.88  | 336000  | 3.2787          |
| 3.2013        | 7.05  | 344000  | 3.3153          |
| 3.2111        | 7.21  | 352000  | 3.3246          |
| 3.2111        | 7.38  | 360000  | 3.3323          |
| 3.2186        | 7.54  | 368000  | 3.2938          |
| 3.2186        | 7.7   | 376000  | 3.3500          |
| 3.2268        | 7.87  | 384000  | 3.3180          |
| 3.2268        | 8.03  | 392000  | 3.3171          |
| 3.233         | 8.2   | 400000  | 3.3462          |
| 3.233         | 8.36  | 408000  | 3.3413          |
| 3.2432        | 8.52  | 416000  | 3.3281          |
| 3.2432        | 8.69  | 424000  | 3.3420          |
| 3.2586        | 8.85  | 432000  | 3.3609          |
| 3.2586        | 9.01  | 440000  | 3.3527          |
| 3.2567        | 9.18  | 448000  | 3.3594          |
| 3.2567        | 9.34  | 456000  | 3.3497          |
| 3.2592        | 9.51  | 464000  | 3.3607          |
| 3.2592        | 9.67  | 472000  | 3.3840          |
| 3.2793        | 9.83  | 480000  | 3.3668          |
| 3.2793        | 10.0  | 488000  | 3.3609          |
| 3.257         | 10.16 | 496000  | 3.3682          |
| 3.257         | 10.33 | 504000  | 3.4006          |
| 3.2656        | 10.49 | 512000  | 3.3588          |
| 3.2656        | 10.65 | 520000  | 3.3799          |
| 3.2727        | 10.82 | 528000  | 3.3833          |
| 3.2727        | 10.98 | 536000  | 3.3566          |
| 3.2705        | 11.15 | 544000  | 3.3794          |
| 3.2705        | 11.31 | 552000  | 3.3838          |
| 3.2676        | 11.47 | 560000  | 3.3660          |
| 3.2676        | 11.64 | 568000  | 3.3938          |
| 3.258         | 11.8  | 576000  | 3.3661          |
| 3.258         | 11.97 | 584000  | 3.3490          |
| 3.2646        | 12.13 | 592000  | 3.3716          |
| 3.2646        | 12.29 | 600000  | 3.3877          |
| 3.2578        | 12.46 | 608000  | 3.3930          |
| 3.2578        | 12.62 | 616000  | 3.3921          |
| 3.2719        | 12.78 | 624000  | 3.3957          |
| 3.2719        | 12.95 | 632000  | 3.4196          |
| 3.2828        | 13.11 | 640000  | 3.4078          |
| 3.2828        | 13.28 | 648000  | 3.4203          |
| 3.2805        | 13.44 | 656000  | 3.3900          |
| 3.2805        | 13.6  | 664000  | 3.4038          |
| 3.2975        | 13.77 | 672000  | 3.4056          |
| 3.2975        | 13.93 | 680000  | 3.4284          |
| 3.2965        | 14.1  | 688000  | 3.4180          |
| 3.2965        | 14.26 | 696000  | 3.4196          |
| 3.3069        | 14.42 | 704000  | 3.4257          |
| 3.3069        | 14.59 | 712000  | 3.4299          |
| 3.3152        | 14.75 | 720000  | 3.4788          |
| 3.3152        | 14.92 | 728000  | 3.4425          |
| 3.3125        | 15.08 | 736000  | 3.4301          |
| 3.3125        | 15.24 | 744000  | 3.4441          |
| 3.3174        | 15.41 | 752000  | 3.4396          |
| 3.3174        | 15.57 | 760000  | 3.4639          |
| 3.3242        | 15.73 | 768000  | 3.4524          |
| 3.3242        | 15.9  | 776000  | 3.4560          |
| 3.3385        | 16.06 | 784000  | 3.4780          |
| 3.3385        | 16.23 | 792000  | 3.4774          |
| 3.3371        | 16.39 | 800000  | 3.4772          |
| 3.3371        | 16.55 | 808000  | 3.4955          |
| 3.3633        | 16.72 | 816000  | 3.4861          |
| 3.3633        | 16.88 | 824000  | 3.5063          |
| 3.3678        | 17.05 | 832000  | 3.5044          |
| 3.3678        | 17.21 | 840000  | 3.5202          |
| 3.3634        | 17.37 | 848000  | 3.4941          |
| 3.3634        | 17.54 | 856000  | 3.5223          |
| 3.3797        | 17.7  | 864000  | 3.5028          |
| 3.3797        | 17.87 | 872000  | 3.5264          |
| 3.3802        | 18.03 | 880000  | 3.5313          |
| 3.3802        | 18.19 | 888000  | 3.4963          |
| 3.357         | 18.36 | 896000  | 3.5171          |
| 3.357         | 18.52 | 904000  | 3.5307          |
| 3.3866        | 18.69 | 912000  | 3.5222          |
| 3.3866        | 18.85 | 920000  | 3.5319          |
| 3.3818        | 19.01 | 928000  | 3.5326          |
| 3.3818        | 19.18 | 936000  | 3.5116          |
| 3.3754        | 19.34 | 944000  | 3.5229          |
| 3.3754        | 19.5  | 952000  | 3.5383          |
| 3.3893        | 19.67 | 960000  | 3.5445          |
| 3.3893        | 19.83 | 968000  | 3.5231          |
| 3.3899        | 20.0  | 976000  | 3.5310          |
| 3.3899        | 20.16 | 984000  | 3.5329          |
| 3.3918        | 20.32 | 992000  | 3.5159          |
| 3.3918        | 20.49 | 1000000 | 3.5628          |
| 3.3786        | 20.65 | 1008000 | 3.5291          |
| 3.3786        | 20.82 | 1016000 | 3.5163          |
| 3.3862        | 20.98 | 1024000 | 3.5312          |
| 3.3862        | 21.14 | 1032000 | 3.5140          |
| 3.3855        | 21.31 | 1040000 | 3.5617          |
| 3.3855        | 21.47 | 1048000 | 3.5375          |
| 3.3872        | 21.64 | 1056000 | 3.5328          |
| 3.3872        | 21.8  | 1064000 | 3.5616          |
| 3.3931        | 21.96 | 1072000 | 3.5648          |
| 3.3931        | 22.13 | 1080000 | 3.5443          |
| 3.3708        | 22.29 | 1088000 | 3.5401          |
| 3.3708        | 22.45 | 1096000 | 3.5529          |
| 3.4099        | 22.62 | 1104000 | 3.5334          |
| 3.4099        | 22.78 | 1112000 | 3.5325          |
| 3.4027        | 22.95 | 1120000 | 3.5819          |
| 3.4027        | 23.11 | 1128000 | 3.5471          |
| 3.4035        | 23.27 | 1136000 | 3.5486          |
| 3.4035        | 23.44 | 1144000 | 3.5470          |
| 3.3964        | 23.6  | 1152000 | 3.5722          |
| 3.3964        | 23.77 | 1160000 | 3.5510          |
| 3.4115        | 23.93 | 1168000 | 3.5610          |
| 3.4115        | 24.09 | 1176000 | 3.5757          |
| 3.4173        | 24.26 | 1184000 | 3.5541          |
| 3.4173        | 24.42 | 1192000 | 3.5777          |
| 3.4169        | 24.59 | 1200000 | 3.5638          |
| 3.4169        | 24.75 | 1208000 | 3.5463          |
| 3.4031        | 24.91 | 1216000 | 3.5300          |
| 3.4031        | 25.08 | 1224000 | 3.5584          |
| 3.4094        | 25.24 | 1232000 | 3.5682          |
| 3.4094        | 25.41 | 1240000 | 3.5558          |
| 3.4116        | 25.57 | 1248000 | 3.5629          |
| 3.4116        | 25.73 | 1256000 | 3.5490          |
| 3.4199        | 25.9  | 1264000 | 3.5679          |
| 3.4199        | 26.06 | 1272000 | 3.5885          |
| 3.412         | 26.22 | 1280000 | 3.5579          |
| 3.412         | 26.39 | 1288000 | 3.5465          |
| 3.4123        | 26.55 | 1296000 | 3.5726          |
| 3.4123        | 26.72 | 1304000 | 3.5775          |
| 3.4132        | 26.88 | 1312000 | 3.5478          |
| 3.4132        | 27.04 | 1320000 | 3.5589          |
| 3.4161        | 27.21 | 1328000 | 3.5662          |
| 3.4161        | 27.37 | 1336000 | 3.5895          |
| 3.4097        | 27.54 | 1344000 | 3.5941          |
| 3.4097        | 27.7  | 1352000 | 3.5912          |
| 3.415         | 27.86 | 1360000 | 3.5658          |
| 3.415         | 28.03 | 1368000 | 3.5554          |
| 3.4193        | 28.19 | 1376000 | 3.5899          |
| 3.4193        | 28.36 | 1384000 | 3.5652          |
| 3.4136        | 28.52 | 1392000 | 3.5832          |
| 3.4136        | 28.68 | 1400000 | 3.5885          |
| 3.4294        | 28.85 | 1408000 | 3.5832          |
| 3.4294        | 29.01 | 1416000 | 3.6025          |
| 3.4243        | 29.17 | 1424000 | 3.6040          |
| 3.4243        | 29.34 | 1432000 | 3.5890          |
| 3.4427        | 29.5  | 1440000 | 3.5835          |
| 3.4427        | 29.67 | 1448000 | 3.6185          |
| 3.4293        | 29.83 | 1456000 | 3.6029          |
| 3.4293        | 29.99 | 1464000 | 3.6162          |
| 3.4363        | 30.16 | 1472000 | 3.6258          |
| 3.4363        | 30.32 | 1480000 | 3.6038          |
| 3.4532        | 30.49 | 1488000 | 3.6039          |
| 3.4532        | 30.65 | 1496000 | 3.6054          |
| 3.4401        | 30.81 | 1504000 | 3.6269          |
| 3.4401        | 30.98 | 1512000 | 3.6004          |
| 3.4491        | 31.14 | 1520000 | 3.6096          |
| 3.4491        | 31.31 | 1528000 | 3.6217          |
| 3.4438        | 31.47 | 1536000 | 3.6081          |
| 3.4438        | 31.63 | 1544000 | 3.6190          |
| 3.4337        | 31.8  | 1552000 | 3.6120          |
| 3.4337        | 31.96 | 1560000 | 3.5861          |
| 3.4475        | 32.13 | 1568000 | 3.6209          |
| 3.4475        | 32.29 | 1576000 | 3.6302          |
| 3.4406        | 32.45 | 1584000 | 3.6053          |
| 3.4406        | 32.62 | 1592000 | 3.5934          |
| 3.4392        | 32.78 | 1600000 | 3.5942          |
| 3.4392        | 32.94 | 1608000 | 3.6013          |
| 3.4514        | 33.11 | 1616000 | 3.6506          |
| 3.4514        | 33.27 | 1624000 | 3.6049          |
| 3.4406        | 33.44 | 1632000 | 3.6285          |
| 3.4406        | 33.6  | 1640000 | 3.6107          |
| 3.4522        | 33.76 | 1648000 | 3.6081          |
| 3.4522        | 33.93 | 1656000 | 3.6121          |
| 3.4592        | 34.09 | 1664000 | 3.6396          |
| 3.4592        | 34.26 | 1672000 | 3.6284          |
| 3.4587        | 34.42 | 1680000 | 3.6195          |
| 3.4587        | 34.58 | 1688000 | 3.6168          |
| 3.4589        | 34.75 | 1696000 | 3.6315          |
| 3.4589        | 34.91 | 1704000 | 3.6045          |
| 3.4703        | 35.08 | 1712000 | 3.6251          |
| 3.4703        | 35.24 | 1720000 | 3.6252          |
| 3.4565        | 35.4  | 1728000 | 3.6254          |
| 3.4565        | 35.57 | 1736000 | 3.6544          |
| 3.4634        | 35.73 | 1744000 | 3.6290          |
| 3.4634        | 35.9  | 1752000 | 3.6124          |
| 3.4625        | 36.06 | 1760000 | 3.6262          |
| 3.4625        | 36.22 | 1768000 | 3.6318          |
| 3.457         | 36.39 | 1776000 | 3.6408          |
| 3.457         | 36.55 | 1784000 | 3.6433          |
| 3.4618        | 36.71 | 1792000 | 3.6276          |
| 3.4618        | 36.88 | 1800000 | 3.6314          |
| 3.4611        | 37.04 | 1808000 | 3.6416          |
| 3.4611        | 37.21 | 1816000 | 3.6658          |
| 3.4651        | 37.37 | 1824000 | 3.6382          |
| 3.4651        | 37.53 | 1832000 | 3.6562          |
| 3.4625        | 37.7  | 1840000 | 3.6376          |
| 3.4625        | 37.86 | 1848000 | 3.6520          |
| 3.4561        | 38.03 | 1856000 | 3.6301          |
| 3.4561        | 38.19 | 1864000 | 3.6195          |
| 3.4655        | 38.35 | 1872000 | 3.6279          |
| 3.4655        | 38.52 | 1880000 | 3.6365          |
| 3.4637        | 38.68 | 1888000 | 3.6386          |
| 3.4637        | 38.85 | 1896000 | 3.6434          |
| 3.458         | 39.01 | 1904000 | 3.6519          |
| 3.458         | 39.17 | 1912000 | 3.6438          |
| 3.4523        | 39.34 | 1920000 | 3.6408          |
| 3.4523        | 39.5  | 1928000 | 3.6513          |
| 3.4743        | 39.66 | 1936000 | 3.6178          |
| 3.4743        | 39.83 | 1944000 | 3.6399          |
| 3.4626        | 39.99 | 1952000 | 3.6243          |
| 3.4626        | 40.16 | 1960000 | 3.6326          |
| 3.4692        | 40.32 | 1968000 | 3.6723          |
| 3.4692        | 40.48 | 1976000 | 3.6456          |
| 3.4765        | 40.65 | 1984000 | 3.6437          |
| 3.4765        | 40.81 | 1992000 | 3.6477          |
| 3.4747        | 40.98 | 2000000 | 3.6384          |
| 3.4747        | 41.14 | 2008000 | 3.6370          |
| 3.4683        | 41.3  | 2016000 | 3.6625          |
| 3.4683        | 41.47 | 2024000 | 3.6453          |
| 3.4599        | 41.63 | 2032000 | 3.6489          |
| 3.4599        | 41.8  | 2040000 | 3.6311          |
| 3.4713        | 41.96 | 2048000 | 3.6192          |
| 3.4713        | 42.12 | 2056000 | 3.6511          |
| 3.4677        | 42.29 | 2064000 | 3.6426          |
| 3.4677        | 42.45 | 2072000 | 3.6363          |
| 3.4689        | 42.62 | 2080000 | 3.6378          |
| 3.4689        | 42.78 | 2088000 | 3.6450          |
| 3.4598        | 42.94 | 2096000 | 3.6481          |
| 3.4598        | 43.11 | 2104000 | 3.6675          |
| 3.4487        | 43.27 | 2112000 | 3.6558          |
| 3.4487        | 43.43 | 2120000 | 3.6451          |
| 3.4555        | 43.6  | 2128000 | 3.6431          |
| 3.4555        | 43.76 | 2136000 | 3.6470          |
| 3.4727        | 43.93 | 2144000 | 3.6265          |
| 3.4727        | 44.09 | 2152000 | 3.6335          |
| 3.4626        | 44.25 | 2160000 | 3.6396          |
| 3.4626        | 44.42 | 2168000 | 3.6537          |
| 3.4724        | 44.58 | 2176000 | 3.6168          |
| 3.4724        | 44.75 | 2184000 | 3.6444          |
| 3.4545        | 44.91 | 2192000 | 3.6440          |
| 3.4545        | 45.07 | 2200000 | 3.6327          |
| 3.461         | 45.24 | 2208000 | 3.6363          |
| 3.461         | 45.4  | 2216000 | 3.6537          |
| 3.4702        | 45.57 | 2224000 | 3.6123          |
| 3.4702        | 45.73 | 2232000 | 3.6554          |
| 3.4565        | 45.89 | 2240000 | 3.6523          |
| 3.4565        | 46.06 | 2248000 | 3.6340          |
| 3.4517        | 46.22 | 2256000 | 3.6459          |
| 3.4517        | 46.38 | 2264000 | 3.6561          |
| 3.4631        | 46.55 | 2272000 | 3.6548          |
| 3.4631        | 46.71 | 2280000 | 3.6229          |
| 3.4518        | 46.88 | 2288000 | 3.6350          |
| 3.4518        | 47.04 | 2296000 | 3.6483          |
| 3.4592        | 47.2  | 2304000 | 3.6263          |
| 3.4592        | 47.37 | 2312000 | 3.6339          |
| 3.4569        | 47.53 | 2320000 | 3.6594          |
| 3.4569        | 47.7  | 2328000 | 3.6385          |
| 3.4524        | 47.86 | 2336000 | 3.6434          |
| 3.4524        | 48.02 | 2344000 | 3.6502          |
| 3.4644        | 48.19 | 2352000 | 3.6176          |
| 3.4644        | 48.35 | 2360000 | 3.6293          |
| 3.4586        | 48.52 | 2368000 | 3.6304          |
| 3.4586        | 48.68 | 2376000 | 3.6343          |
| 3.4439        | 48.84 | 2384000 | 3.6090          |
| 3.4439        | 49.01 | 2392000 | 3.6414          |
| 3.4474        | 49.17 | 2400000 | 3.6208          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
