transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q1-50p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q1-50p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.4514

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.03  | 8000    | 2.8937          |
| 3.073         | 0.07  | 16000   | 2.7660          |
| 3.073         | 0.1   | 24000   | 2.7233          |
| 2.8244        | 0.13  | 32000   | 2.6878          |
| 2.8244        | 0.16  | 40000   | 2.6520          |
| 2.7542        | 0.2   | 48000   | 2.6300          |
| 2.7542        | 0.23  | 56000   | 2.6135          |
| 2.7083        | 0.26  | 64000   | 2.6068          |
| 2.7083        | 0.3   | 72000   | 2.5854          |
| 2.6752        | 0.33  | 80000   | 2.5755          |
| 2.6752        | 0.36  | 88000   | 2.5721          |
| 2.6657        | 0.39  | 96000   | 2.5709          |
| 2.6657        | 0.43  | 104000  | 2.5656          |
| 2.6534        | 0.46  | 112000  | 2.5558          |
| 2.6534        | 0.49  | 120000  | 2.5496          |
| 2.646         | 0.52  | 128000  | 2.5471          |
| 2.646         | 0.56  | 136000  | 2.5408          |
| 2.625         | 0.59  | 144000  | 2.5315          |
| 2.625         | 0.62  | 152000  | 2.5365          |
| 2.6222        | 0.66  | 160000  | 2.5372          |
| 2.6222        | 0.69  | 168000  | 2.5342          |
| 2.6256        | 0.72  | 176000  | 2.5308          |
| 2.6256        | 0.75  | 184000  | 2.5312          |
| 2.6074        | 0.79  | 192000  | 2.5228          |
| 2.6074        | 0.82  | 200000  | 2.5292          |
| 2.6071        | 0.85  | 208000  | 2.5295          |
| 2.6071        | 0.89  | 216000  | 2.5235          |
| 2.5955        | 0.92  | 224000  | 2.5219          |
| 2.5955        | 0.95  | 232000  | 2.5191          |
| 2.6036        | 0.98  | 240000  | 2.5171          |
| 2.6036        | 1.02  | 248000  | 2.5102          |
| 2.6046        | 1.05  | 256000  | 2.5070          |
| 2.6046        | 1.08  | 264000  | 2.5109          |
| 2.5892        | 1.11  | 272000  | 2.5105          |
| 2.5892        | 1.15  | 280000  | 2.5087          |
| 2.5929        | 1.18  | 288000  | 2.5094          |
| 2.5929        | 1.21  | 296000  | 2.5086          |
| 2.5857        | 1.25  | 304000  | 2.4991          |
| 2.5857        | 1.28  | 312000  | 2.5089          |
| 2.5828        | 1.31  | 320000  | 2.5017          |
| 2.5828        | 1.34  | 328000  | 2.5039          |
| 2.5812        | 1.38  | 336000  | 2.5065          |
| 2.5812        | 1.41  | 344000  | 2.5083          |
| 2.5775        | 1.44  | 352000  | 2.5099          |
| 2.5775        | 1.48  | 360000  | 2.5079          |
| 2.5711        | 1.51  | 368000  | 2.4922          |
| 2.5711        | 1.54  | 376000  | 2.5012          |
| 2.5797        | 1.57  | 384000  | 2.4999          |
| 2.5797        | 1.61  | 392000  | 2.4881          |
| 2.5718        | 1.64  | 400000  | 2.4960          |
| 2.5718        | 1.67  | 408000  | 2.4908          |
| 2.5627        | 1.7   | 416000  | 2.4971          |
| 2.5627        | 1.74  | 424000  | 2.4916          |
| 2.5641        | 1.77  | 432000  | 2.4971          |
| 2.5641        | 1.8   | 440000  | 2.4954          |
| 2.5633        | 1.84  | 448000  | 2.4860          |
| 2.5633        | 1.87  | 456000  | 2.4894          |
| 2.5676        | 1.9   | 464000  | 2.4893          |
| 2.5676        | 1.93  | 472000  | 2.4884          |
| 2.5687        | 1.97  | 480000  | 2.4921          |
| 2.5687        | 2.0   | 488000  | 2.4873          |
| 2.5633        | 2.03  | 496000  | 2.4919          |
| 2.5633        | 2.07  | 504000  | 2.4821          |
| 2.5547        | 2.1   | 512000  | 2.4909          |
| 2.5547        | 2.13  | 520000  | 2.4818          |
| 2.5617        | 2.16  | 528000  | 2.4855          |
| 2.5617        | 2.2   | 536000  | 2.4850          |
| 2.5569        | 2.23  | 544000  | 2.4803          |
| 2.5569        | 2.26  | 552000  | 2.4776          |
| 2.5535        | 2.29  | 560000  | 2.4824          |
| 2.5535        | 2.33  | 568000  | 2.4822          |
| 2.5534        | 2.36  | 576000  | 2.4763          |
| 2.5534        | 2.39  | 584000  | 2.4797          |
| 2.5583        | 2.43  | 592000  | 2.4872          |
| 2.5583        | 2.46  | 600000  | 2.4812          |
| 2.5545        | 2.49  | 608000  | 2.4748          |
| 2.5545        | 2.52  | 616000  | 2.4736          |
| 2.5561        | 2.56  | 624000  | 2.4714          |
| 2.5561        | 2.59  | 632000  | 2.4858          |
| 2.5384        | 2.62  | 640000  | 2.4829          |
| 2.5384        | 2.66  | 648000  | 2.4766          |
| 2.541         | 2.69  | 656000  | 2.4836          |
| 2.541         | 2.72  | 664000  | 2.4651          |
| 2.5439        | 2.75  | 672000  | 2.4797          |
| 2.5439        | 2.79  | 680000  | 2.4702          |
| 2.5597        | 2.82  | 688000  | 2.4751          |
| 2.5597        | 2.85  | 696000  | 2.4744          |
| 2.5491        | 2.88  | 704000  | 2.4756          |
| 2.5491        | 2.92  | 712000  | 2.4731          |
| 2.5505        | 2.95  | 720000  | 2.4756          |
| 2.5505        | 2.98  | 728000  | 2.4704          |
| 2.5432        | 3.02  | 736000  | 2.4763          |
| 2.5432        | 3.05  | 744000  | 2.4743          |
| 2.5485        | 3.08  | 752000  | 2.4627          |
| 2.5485        | 3.11  | 760000  | 2.4714          |
| 2.5482        | 3.15  | 768000  | 2.4685          |
| 2.5482        | 3.18  | 776000  | 2.4673          |
| 2.5411        | 3.21  | 784000  | 2.4726          |
| 2.5411        | 3.25  | 792000  | 2.4761          |
| 2.5407        | 3.28  | 800000  | 2.4612          |
| 2.5407        | 3.31  | 808000  | 2.4743          |
| 2.5307        | 3.34  | 816000  | 2.4699          |
| 2.5307        | 3.38  | 824000  | 2.4721          |
| 2.5391        | 3.41  | 832000  | 2.4614          |
| 2.5391        | 3.44  | 840000  | 2.4641          |
| 2.5378        | 3.47  | 848000  | 2.4652          |
| 2.5378        | 3.51  | 856000  | 2.4641          |
| 2.5399        | 3.54  | 864000  | 2.4691          |
| 2.5399        | 3.57  | 872000  | 2.4612          |
| 2.5412        | 3.61  | 880000  | 2.4696          |
| 2.5412        | 3.64  | 888000  | 2.4638          |
| 2.5389        | 3.67  | 896000  | 2.4658          |
| 2.5389        | 3.7   | 904000  | 2.4725          |
| 2.5325        | 3.74  | 912000  | 2.4642          |
| 2.5325        | 3.77  | 920000  | 2.4599          |
| 2.5351        | 3.8   | 928000  | 2.4617          |
| 2.5351        | 3.84  | 936000  | 2.4646          |
| 2.522         | 3.87  | 944000  | 2.4665          |
| 2.522         | 3.9   | 952000  | 2.4762          |
| 2.5331        | 3.93  | 960000  | 2.4669          |
| 2.5331        | 3.97  | 968000  | 2.4550          |
| 2.5276        | 4.0   | 976000  | 2.4662          |
| 2.5276        | 4.03  | 984000  | 2.4645          |
| 2.5206        | 4.06  | 992000  | 2.4587          |
| 2.5206        | 4.1   | 1000000 | 2.4725          |
| 2.5294        | 4.13  | 1008000 | 2.4588          |
| 2.5294        | 4.16  | 1016000 | 2.4591          |
| 2.5312        | 4.2   | 1024000 | 2.4681          |
| 2.5312        | 4.23  | 1032000 | 2.4625          |
| 2.525         | 4.26  | 1040000 | 2.4659          |
| 2.525         | 4.29  | 1048000 | 2.4609          |
| 2.5318        | 4.33  | 1056000 | 2.4571          |
| 2.5318        | 4.36  | 1064000 | 2.4582          |
| 2.5332        | 4.39  | 1072000 | 2.4566          |
| 2.5332        | 4.43  | 1080000 | 2.4588          |
| 2.5168        | 4.46  | 1088000 | 2.4606          |
| 2.5168        | 4.49  | 1096000 | 2.4598          |
| 2.5181        | 4.52  | 1104000 | 2.4543          |
| 2.5181        | 4.56  | 1112000 | 2.4620          |
| 2.5246        | 4.59  | 1120000 | 2.4639          |
| 2.5246        | 4.62  | 1128000 | 2.4556          |
| 2.5318        | 4.65  | 1136000 | 2.4571          |
| 2.5318        | 4.69  | 1144000 | 2.4636          |
| 2.512         | 4.72  | 1152000 | 2.4568          |
| 2.512         | 4.75  | 1160000 | 2.4644          |
| 2.5174        | 4.79  | 1168000 | 2.4529          |
| 2.5174        | 4.82  | 1176000 | 2.4614          |
| 2.5196        | 4.85  | 1184000 | 2.4638          |
| 2.5196        | 4.88  | 1192000 | 2.4534          |
| 2.5248        | 4.92  | 1200000 | 2.4553          |
| 2.5248        | 4.95  | 1208000 | 2.4537          |
| 2.5201        | 4.98  | 1216000 | 2.4579          |
| 2.5201        | 5.02  | 1224000 | 2.4525          |
| 2.5164        | 5.05  | 1232000 | 2.4645          |
| 2.5164        | 5.08  | 1240000 | 2.4480          |
| 2.5186        | 5.11  | 1248000 | 2.4606          |
| 2.5186        | 5.15  | 1256000 | 2.4623          |
| 2.5123        | 5.18  | 1264000 | 2.4566          |
| 2.5123        | 5.21  | 1272000 | 2.4644          |
| 2.5233        | 5.24  | 1280000 | 2.4576          |
| 2.5233        | 5.28  | 1288000 | 2.4519          |
| 2.513         | 5.31  | 1296000 | 2.4570          |
| 2.513         | 5.34  | 1304000 | 2.4627          |
| 2.5226        | 5.38  | 1312000 | 2.4500          |
| 2.5226        | 5.41  | 1320000 | 2.4563          |
| 2.5222        | 5.44  | 1328000 | 2.4521          |
| 2.5222        | 5.47  | 1336000 | 2.4591          |
| 2.5191        | 5.51  | 1344000 | 2.4509          |
| 2.5191        | 5.54  | 1352000 | 2.4559          |
| 2.5243        | 5.57  | 1360000 | 2.4502          |
| 2.5243        | 5.61  | 1368000 | 2.4515          |
| 2.5157        | 5.64  | 1376000 | 2.4563          |
| 2.5157        | 5.67  | 1384000 | 2.4526          |
| 2.5162        | 5.7   | 1392000 | 2.4586          |
| 2.5162        | 5.74  | 1400000 | 2.4584          |
| 2.5169        | 5.77  | 1408000 | 2.4542          |
| 2.5169        | 5.8   | 1416000 | 2.4602          |
| 2.5127        | 5.84  | 1424000 | 2.4587          |
| 2.5127        | 5.87  | 1432000 | 2.4529          |
| 2.5144        | 5.9   | 1440000 | 2.4620          |
| 2.5144        | 5.93  | 1448000 | 2.4509          |
| 2.5175        | 5.97  | 1456000 | 2.4503          |
| 2.5175        | 6.0   | 1464000 | 2.4545          |
| 2.5147        | 6.03  | 1472000 | 2.4440          |
| 2.5147        | 6.06  | 1480000 | 2.4577          |
| 2.5128        | 6.1   | 1488000 | 2.4566          |
| 2.5128        | 6.13  | 1496000 | 2.4499          |
| 2.5168        | 6.16  | 1504000 | 2.4480          |
| 2.5168        | 6.2   | 1512000 | 2.4436          |
| 2.5225        | 6.23  | 1520000 | 2.4467          |
| 2.5225        | 6.26  | 1528000 | 2.4520          |
| 2.5135        | 6.29  | 1536000 | 2.4535          |
| 2.5135        | 6.33  | 1544000 | 2.4463          |
| 2.5161        | 6.36  | 1552000 | 2.4556          |
| 2.5161        | 6.39  | 1560000 | 2.4605          |
| 2.5144        | 6.43  | 1568000 | 2.4516          |
| 2.5144        | 6.46  | 1576000 | 2.4488          |
| 2.5209        | 6.49  | 1584000 | 2.4525          |
| 2.5209        | 6.52  | 1592000 | 2.4502          |
| 2.5102        | 6.56  | 1600000 | 2.4538          |
| 2.5102        | 6.59  | 1608000 | 2.4491          |
| 2.5176        | 6.62  | 1616000 | 2.4528          |
| 2.5176        | 6.65  | 1624000 | 2.4460          |
| 2.5208        | 6.69  | 1632000 | 2.4485          |
| 2.5208        | 6.72  | 1640000 | 2.4513          |
| 2.5064        | 6.75  | 1648000 | 2.4519          |
| 2.5064        | 6.79  | 1656000 | 2.4493          |
| 2.5111        | 6.82  | 1664000 | 2.4505          |
| 2.5111        | 6.85  | 1672000 | 2.4502          |
| 2.5141        | 6.88  | 1680000 | 2.4560          |
| 2.5141        | 6.92  | 1688000 | 2.4500          |
| 2.5089        | 6.95  | 1696000 | 2.4513          |
| 2.5089        | 6.98  | 1704000 | 2.4418          |
| 2.5174        | 7.02  | 1712000 | 2.4477          |
| 2.5174        | 7.05  | 1720000 | 2.4508          |
| 2.5198        | 7.08  | 1728000 | 2.4486          |
| 2.5198        | 7.11  | 1736000 | 2.4577          |
| 2.4974        | 7.15  | 1744000 | 2.4416          |
| 2.4974        | 7.18  | 1752000 | 2.4549          |
| 2.5016        | 7.21  | 1760000 | 2.4557          |
| 2.5016        | 7.24  | 1768000 | 2.4532          |
| 2.5112        | 7.28  | 1776000 | 2.4451          |
| 2.5112        | 7.31  | 1784000 | 2.4607          |
| 2.5172        | 7.34  | 1792000 | 2.4452          |
| 2.5172        | 7.38  | 1800000 | 2.4427          |
| 2.5089        | 7.41  | 1808000 | 2.4511          |
| 2.5089        | 7.44  | 1816000 | 2.4441          |
| 2.5136        | 7.47  | 1824000 | 2.4492          |
| 2.5136        | 7.51  | 1832000 | 2.4524          |
| 2.509         | 7.54  | 1840000 | 2.4512          |
| 2.509         | 7.57  | 1848000 | 2.4528          |
| 2.5157        | 7.61  | 1856000 | 2.4440          |
| 2.5157        | 7.64  | 1864000 | 2.4402          |
| 2.5181        | 7.67  | 1872000 | 2.4538          |
| 2.5181        | 7.7   | 1880000 | 2.4481          |
| 2.5145        | 7.74  | 1888000 | 2.4417          |
| 2.5145        | 7.77  | 1896000 | 2.4512          |
| 2.5013        | 7.8   | 1904000 | 2.4560          |
| 2.5013        | 7.83  | 1912000 | 2.4509          |
| 2.5064        | 7.87  | 1920000 | 2.4473          |
| 2.5064        | 7.9   | 1928000 | 2.4576          |
| 2.5068        | 7.93  | 1936000 | 2.4461          |
| 2.5068        | 7.97  | 1944000 | 2.4451          |
| 2.5152        | 8.0   | 1952000 | 2.4421          |
| 2.5152        | 8.03  | 1960000 | 2.4458          |
| 2.5025        | 8.06  | 1968000 | 2.4532          |
| 2.5025        | 8.1   | 1976000 | 2.4541          |
| 2.5151        | 8.13  | 1984000 | 2.4499          |
| 2.5151        | 8.16  | 1992000 | 2.4501          |
| 2.5138        | 8.2   | 2000000 | 2.4448          |
| 2.5138        | 8.23  | 2008000 | 2.4562          |
| 2.5039        | 8.26  | 2016000 | 2.4613          |
| 2.5039        | 8.29  | 2024000 | 2.4471          |
| 2.5055        | 8.33  | 2032000 | 2.4450          |
| 2.5055        | 8.36  | 2040000 | 2.4493          |
| 2.5085        | 8.39  | 2048000 | 2.4482          |
| 2.5085        | 8.42  | 2056000 | 2.4572          |
| 2.5114        | 8.46  | 2064000 | 2.4443          |
| 2.5114        | 8.49  | 2072000 | 2.4456          |
| 2.5132        | 8.52  | 2080000 | 2.4528          |
| 2.5132        | 8.56  | 2088000 | 2.4497          |
| 2.5072        | 8.59  | 2096000 | 2.4548          |
| 2.5072        | 8.62  | 2104000 | 2.4548          |
| 2.504         | 8.65  | 2112000 | 2.4443          |
| 2.504         | 8.69  | 2120000 | 2.4452          |
| 2.5128        | 8.72  | 2128000 | 2.4510          |
| 2.5128        | 8.75  | 2136000 | 2.4480          |
| 2.5133        | 8.79  | 2144000 | 2.4470          |
| 2.5133        | 8.82  | 2152000 | 2.4437          |
| 2.5067        | 8.85  | 2160000 | 2.4447          |
| 2.5067        | 8.88  | 2168000 | 2.4531          |
| 2.4996        | 8.92  | 2176000 | 2.4475          |
| 2.4996        | 8.95  | 2184000 | 2.4438          |
| 2.5123        | 8.98  | 2192000 | 2.4552          |
| 2.5123        | 9.01  | 2200000 | 2.4441          |
| 2.5044        | 9.05  | 2208000 | 2.4438          |
| 2.5044        | 9.08  | 2216000 | 2.4534          |
| 2.5068        | 9.11  | 2224000 | 2.4497          |
| 2.5068        | 9.15  | 2232000 | 2.4440          |
| 2.5165        | 9.18  | 2240000 | 2.4577          |
| 2.5165        | 9.21  | 2248000 | 2.4507          |
| 2.5087        | 9.24  | 2256000 | 2.4494          |
| 2.5087        | 9.28  | 2264000 | 2.4393          |
| 2.5036        | 9.31  | 2272000 | 2.4487          |
| 2.5036        | 9.34  | 2280000 | 2.4423          |
| 2.5086        | 9.38  | 2288000 | 2.4456          |
| 2.5086        | 9.41  | 2296000 | 2.4496          |
| 2.5034        | 9.44  | 2304000 | 2.4499          |
| 2.5034        | 9.47  | 2312000 | 2.4433          |
| 2.5099        | 9.51  | 2320000 | 2.4534          |
| 2.5099        | 9.54  | 2328000 | 2.4495          |
| 2.5065        | 9.57  | 2336000 | 2.4510          |
| 2.5065        | 9.6   | 2344000 | 2.4513          |
| 2.502         | 9.64  | 2352000 | 2.4512          |
| 2.502         | 9.67  | 2360000 | 2.4469          |
| 2.5043        | 9.7   | 2368000 | 2.4544          |
| 2.5043        | 9.74  | 2376000 | 2.4493          |
| 2.5068        | 9.77  | 2384000 | 2.4537          |
| 2.5068        | 9.8   | 2392000 | 2.4387          |
| 2.5118        | 9.83  | 2400000 | 2.4494          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
