transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q2-75p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q2-75p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.0187

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1400
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.07  | 8000    | 3.1520          |
| 3.3704        | 0.13  | 16000   | 3.1286          |
| 3.3704        | 0.2   | 24000   | 3.1079          |
| 3.2908        | 0.27  | 32000   | 3.0845          |
| 3.2908        | 0.34  | 40000   | 3.0868          |
| 3.2742        | 0.4   | 48000   | 3.0768          |
| 3.2742        | 0.47  | 56000   | 3.0706          |
| 3.2579        | 0.54  | 64000   | 3.0621          |
| 3.2579        | 0.61  | 72000   | 3.0659          |
| 3.2448        | 0.67  | 80000   | 3.0457          |
| 3.2448        | 0.74  | 88000   | 3.0554          |
| 3.2416        | 0.81  | 96000   | 3.0335          |
| 3.2416        | 0.88  | 104000  | 3.0321          |
| 3.23          | 0.94  | 112000  | 3.0137          |
| 3.23          | 1.01  | 120000  | 3.0061          |
| 3.2084        | 1.08  | 128000  | 3.0251          |
| 3.2084        | 1.15  | 136000  | 3.0092          |
| 3.2055        | 1.21  | 144000  | 3.0043          |
| 3.2055        | 1.28  | 152000  | 3.0055          |
| 3.2026        | 1.35  | 160000  | 3.0066          |
| 3.2026        | 1.41  | 168000  | 3.0125          |
| 3.2069        | 1.48  | 176000  | 3.0032          |
| 3.2069        | 1.55  | 184000  | 2.9959          |
| 3.1904        | 1.62  | 192000  | 2.9960          |
| 3.1904        | 1.68  | 200000  | 3.0038          |
| 3.1989        | 1.75  | 208000  | 3.0016          |
| 3.1989        | 1.82  | 216000  | 3.0049          |
| 3.2113        | 1.89  | 224000  | 3.0086          |
| 3.2113        | 1.95  | 232000  | 3.0104          |
| 3.217         | 2.02  | 240000  | 3.0166          |
| 3.217         | 2.09  | 248000  | 3.0139          |
| 3.2029        | 2.16  | 256000  | 3.0217          |
| 3.2029        | 2.22  | 264000  | 3.0238          |
| 3.2226        | 2.29  | 272000  | 3.0234          |
| 3.2226        | 2.36  | 280000  | 3.0216          |
| 3.2199        | 2.43  | 288000  | 3.0175          |
| 3.2199        | 2.49  | 296000  | 3.0365          |
| 3.2254        | 2.56  | 304000  | 3.0282          |
| 3.2254        | 2.63  | 312000  | 3.0228          |
| 3.2349        | 2.7   | 320000  | 3.0205          |
| 3.2349        | 2.76  | 328000  | 3.0406          |
| 3.2424        | 2.83  | 336000  | 3.0307          |
| 3.2424        | 2.9   | 344000  | 3.0413          |
| 3.2347        | 2.96  | 352000  | 3.0401          |
| 3.2347        | 3.03  | 360000  | 3.0520          |
| 3.2476        | 3.1   | 368000  | 3.0489          |
| 3.2476        | 3.17  | 376000  | 3.0521          |
| 3.2506        | 3.23  | 384000  | 3.0685          |
| 3.2506        | 3.3   | 392000  | 3.0546          |
| 3.2547        | 3.37  | 400000  | 3.0542          |
| 3.2547        | 3.44  | 408000  | 3.0537          |
| 3.2519        | 3.5   | 416000  | 3.0588          |
| 3.2519        | 3.57  | 424000  | 3.0729          |
| 3.2679        | 3.64  | 432000  | 3.0842          |
| 3.2679        | 3.71  | 440000  | 3.0685          |
| 3.2656        | 3.77  | 448000  | 3.0942          |
| 3.2656        | 3.84  | 456000  | 3.0942          |
| 3.2908        | 3.91  | 464000  | 3.0918          |
| 3.2908        | 3.98  | 472000  | 3.0922          |
| 3.2944        | 4.04  | 480000  | 3.1093          |
| 3.2944        | 4.11  | 488000  | 3.1158          |
| 3.2917        | 4.18  | 496000  | 3.0997          |
| 3.2917        | 4.24  | 504000  | 3.1111          |
| 3.2916        | 4.31  | 512000  | 3.1133          |
| 3.2916        | 4.38  | 520000  | 3.1129          |
| 3.2836        | 4.45  | 528000  | 3.1134          |
| 3.2836        | 4.51  | 536000  | 3.1058          |
| 3.3068        | 4.58  | 544000  | 3.1211          |
| 3.3068        | 4.65  | 552000  | 3.0946          |
| 3.3026        | 4.72  | 560000  | 3.1079          |
| 3.3026        | 4.78  | 568000  | 3.1202          |
| 3.3078        | 4.85  | 576000  | 3.1155          |
| 3.3078        | 4.92  | 584000  | 3.1254          |
| 3.3168        | 4.99  | 592000  | 3.1279          |
| 3.3168        | 5.05  | 600000  | 3.1179          |
| 3.3113        | 5.12  | 608000  | 3.1277          |
| 3.3113        | 5.19  | 616000  | 3.1334          |
| 3.3102        | 5.26  | 624000  | 3.1233          |
| 3.3102        | 5.32  | 632000  | 3.1274          |
| 3.3235        | 5.39  | 640000  | 3.1434          |
| 3.3235        | 5.46  | 648000  | 3.1368          |
| 3.331         | 5.53  | 656000  | 3.1591          |
| 3.331         | 5.59  | 664000  | 3.1546          |
| 3.3308        | 5.66  | 672000  | 3.1663          |
| 3.3308        | 5.73  | 680000  | 3.1535          |
| 3.3396        | 5.79  | 688000  | 3.1558          |
| 3.3396        | 5.86  | 696000  | 3.1698          |
| 3.3558        | 5.93  | 704000  | 3.1651          |
| 3.3558        | 6.0   | 712000  | 3.1706          |
| 3.3474        | 6.06  | 720000  | 3.1942          |
| 3.3474        | 6.13  | 728000  | 3.1705          |
| 3.3513        | 6.2   | 736000  | 3.1834          |
| 3.3513        | 6.27  | 744000  | 3.1810          |
| 3.362         | 6.33  | 752000  | 3.1723          |
| 3.362         | 6.4   | 760000  | 3.1827          |
| 3.3694        | 6.47  | 768000  | 3.1937          |
| 3.3694        | 6.54  | 776000  | 3.2004          |
| 3.378         | 6.6   | 784000  | 3.2023          |
| 3.378         | 6.67  | 792000  | 3.1936          |
| 3.3703        | 6.74  | 800000  | 3.1948          |
| 3.3703        | 6.81  | 808000  | 3.2082          |
| 3.3838        | 6.87  | 816000  | 3.1974          |
| 3.3838        | 6.94  | 824000  | 3.2029          |
| 3.3871        | 7.01  | 832000  | 3.2160          |
| 3.3871        | 7.07  | 840000  | 3.2198          |
| 3.3839        | 7.14  | 848000  | 3.2190          |
| 3.3839        | 7.21  | 856000  | 3.2204          |
| 3.389         | 7.28  | 864000  | 3.2188          |
| 3.389         | 7.34  | 872000  | 3.2246          |
| 3.398         | 7.41  | 880000  | 3.2333          |
| 3.398         | 7.48  | 888000  | 3.2168          |
| 3.4001        | 7.55  | 896000  | 3.2311          |
| 3.4001        | 7.61  | 904000  | 3.2390          |
| 3.4255        | 7.68  | 912000  | 3.2447          |
| 3.4255        | 7.75  | 920000  | 3.2546          |
| 3.4218        | 7.82  | 928000  | 3.2510          |
| 3.4218        | 7.88  | 936000  | 3.2433          |
| 3.4326        | 7.95  | 944000  | 3.2509          |
| 3.4326        | 8.02  | 952000  | 3.2573          |
| 3.4268        | 8.09  | 960000  | 3.2499          |
| 3.4268        | 8.15  | 968000  | 3.2704          |
| 3.4165        | 8.22  | 976000  | 3.2579          |
| 3.4165        | 8.29  | 984000  | 3.2669          |
| 3.4425        | 8.36  | 992000  | 3.2723          |
| 3.4425        | 8.42  | 1000000 | 3.2718          |
| 3.4433        | 8.49  | 1008000 | 3.2655          |
| 3.4433        | 8.56  | 1016000 | 3.2794          |
| 3.4437        | 8.62  | 1024000 | 3.2808          |
| 3.4437        | 8.69  | 1032000 | 3.2731          |
| 3.4499        | 8.76  | 1040000 | 3.2785          |
| 3.4499        | 8.83  | 1048000 | 3.2823          |
| 3.4593        | 8.89  | 1056000 | 3.2844          |
| 3.4593        | 8.96  | 1064000 | 3.2877          |
| 3.4481        | 9.03  | 1072000 | 3.2969          |
| 3.4481        | 9.1   | 1080000 | 3.2870          |
| 3.4542        | 9.16  | 1088000 | 3.2946          |
| 3.4542        | 9.23  | 1096000 | 3.2901          |
| 3.4547        | 9.3   | 1104000 | 3.2813          |
| 3.4547        | 9.37  | 1112000 | 3.2910          |
| 3.4618        | 9.43  | 1120000 | 3.2978          |
| 3.4618        | 9.5   | 1128000 | 3.3055          |
| 3.46          | 9.57  | 1136000 | 3.2885          |
| 3.46          | 9.64  | 1144000 | 3.2871          |
| 3.4572        | 9.7   | 1152000 | 3.2905          |
| 3.4572        | 9.77  | 1160000 | 3.3006          |
| 3.4597        | 9.84  | 1168000 | 3.3081          |
| 3.4597        | 9.9   | 1176000 | 3.3031          |
| 3.4651        | 9.97  | 1184000 | 3.2883          |
| 3.4651        | 10.04 | 1192000 | 3.3189          |
| 3.4571        | 10.11 | 1200000 | 3.2978          |
| 3.4571        | 10.17 | 1208000 | 3.3091          |
| 3.4567        | 10.24 | 1216000 | 3.2755          |
| 3.4567        | 10.31 | 1224000 | 3.2968          |
| 3.4584        | 10.38 | 1232000 | 3.2991          |
| 3.4584        | 10.44 | 1240000 | 3.2818          |
| 3.4459        | 10.51 | 1248000 | 3.2823          |
| 3.4459        | 10.58 | 1256000 | 3.2800          |
| 3.4474        | 10.65 | 1264000 | 3.2856          |
| 3.4474        | 10.71 | 1272000 | 3.2845          |
| 3.4383        | 10.78 | 1280000 | 3.2804          |
| 3.4383        | 10.85 | 1288000 | 3.2707          |
| 3.4496        | 10.92 | 1296000 | 3.2824          |
| 3.4496        | 10.98 | 1304000 | 3.2765          |
| 3.4411        | 11.05 | 1312000 | 3.2838          |
| 3.4411        | 11.12 | 1320000 | 3.2839          |
| 3.4305        | 11.19 | 1328000 | 3.2748          |
| 3.4305        | 11.25 | 1336000 | 3.2821          |
| 3.4258        | 11.32 | 1344000 | 3.2746          |
| 3.4258        | 11.39 | 1352000 | 3.2861          |
| 3.4227        | 11.45 | 1360000 | 3.2710          |
| 3.4227        | 11.52 | 1368000 | 3.2788          |
| 3.4319        | 11.59 | 1376000 | 3.2794          |
| 3.4319        | 11.66 | 1384000 | 3.2766          |
| 3.436         | 11.72 | 1392000 | 3.2924          |
| 3.436         | 11.79 | 1400000 | 3.2812          |
| 3.4368        | 11.86 | 1408000 | 3.2851          |
| 3.4368        | 11.93 | 1416000 | 3.2822          |
| 3.4346        | 11.99 | 1424000 | 3.2657          |
| 3.4346        | 12.06 | 1432000 | 3.2748          |
| 3.4265        | 12.13 | 1440000 | 3.2685          |
| 3.4265        | 12.2  | 1448000 | 3.2947          |
| 3.4306        | 12.26 | 1456000 | 3.2841          |
| 3.4306        | 12.33 | 1464000 | 3.2748          |
| 3.4254        | 12.4  | 1472000 | 3.2794          |
| 3.4254        | 12.47 | 1480000 | 3.2774          |
| 3.4353        | 12.53 | 1488000 | 3.2726          |
| 3.4353        | 12.6  | 1496000 | 3.2763          |
| 3.4358        | 12.67 | 1504000 | 3.2659          |
| 3.4358        | 12.73 | 1512000 | 3.2710          |
| 3.4182        | 12.8  | 1520000 | 3.2777          |
| 3.4182        | 12.87 | 1528000 | 3.2824          |
| 3.4384        | 12.94 | 1536000 | 3.2887          |
| 3.4384        | 13.0  | 1544000 | 3.2667          |
| 3.4287        | 13.07 | 1552000 | 3.2713          |
| 3.4287        | 13.14 | 1560000 | 3.2640          |
| 3.4181        | 13.21 | 1568000 | 3.2607          |
| 3.4181        | 13.27 | 1576000 | 3.2643          |
| 3.4173        | 13.34 | 1584000 | 3.2630          |
| 3.4173        | 13.41 | 1592000 | 3.2572          |
| 3.4214        | 13.48 | 1600000 | 3.2728          |
| 3.4214        | 13.54 | 1608000 | 3.2822          |
| 3.4223        | 13.61 | 1616000 | 3.2704          |
| 3.4223        | 13.68 | 1624000 | 3.2634          |
| 3.417         | 13.75 | 1632000 | 3.2691          |
| 3.417         | 13.81 | 1640000 | 3.2550          |
| 3.4146        | 13.88 | 1648000 | 3.2529          |
| 3.4146        | 13.95 | 1656000 | 3.2713          |
| 3.4186        | 14.02 | 1664000 | 3.2672          |
| 3.4186        | 14.08 | 1672000 | 3.2542          |
| 3.4082        | 14.15 | 1680000 | 3.2576          |
| 3.4082        | 14.22 | 1688000 | 3.2680          |
| 3.4186        | 14.28 | 1696000 | 3.2667          |
| 3.4186        | 14.35 | 1704000 | 3.2694          |
| 3.4131        | 14.42 | 1712000 | 3.2606          |
| 3.4131        | 14.49 | 1720000 | 3.2622          |
| 3.4239        | 14.55 | 1728000 | 3.2678          |
| 3.4239        | 14.62 | 1736000 | 3.2708          |
| 3.4197        | 14.69 | 1744000 | 3.2622          |
| 3.4197        | 14.76 | 1752000 | 3.2605          |
| 3.4073        | 14.82 | 1760000 | 3.2647          |
| 3.4073        | 14.89 | 1768000 | 3.2619          |
| 3.4167        | 14.96 | 1776000 | 3.2816          |
| 3.4167        | 15.03 | 1784000 | 3.2603          |
| 3.413         | 15.09 | 1792000 | 3.2661          |
| 3.413         | 15.16 | 1800000 | 3.2589          |
| 3.4117        | 15.23 | 1808000 | 3.2688          |
| 3.4117        | 15.3  | 1816000 | 3.2678          |
| 3.4103        | 15.36 | 1824000 | 3.2661          |
| 3.4103        | 15.43 | 1832000 | 3.2705          |
| 3.4074        | 15.5  | 1840000 | 3.2670          |
| 3.4074        | 15.56 | 1848000 | 3.2619          |
| 3.4167        | 15.63 | 1856000 | 3.2624          |
| 3.4167        | 15.7  | 1864000 | 3.2552          |
| 3.4195        | 15.77 | 1872000 | 3.2503          |
| 3.4195        | 15.83 | 1880000 | 3.2606          |
| 3.4091        | 15.9  | 1888000 | 3.2812          |
| 3.4091        | 15.97 | 1896000 | 3.2837          |
| 3.4116        | 16.04 | 1904000 | 3.2658          |
| 3.4116        | 16.1  | 1912000 | 3.2676          |
| 3.4183        | 16.17 | 1920000 | 3.2770          |
| 3.4183        | 16.24 | 1928000 | 3.2756          |
| 3.4177        | 16.31 | 1936000 | 3.2876          |
| 3.4177        | 16.37 | 1944000 | 3.2612          |
| 3.4226        | 16.44 | 1952000 | 3.2748          |
| 3.4226        | 16.51 | 1960000 | 3.2679          |
| 3.4154        | 16.58 | 1968000 | 3.2659          |
| 3.4154        | 16.64 | 1976000 | 3.2689          |
| 3.4199        | 16.71 | 1984000 | 3.2701          |
| 3.4199        | 16.78 | 1992000 | 3.2564          |
| 3.4166        | 16.85 | 2000000 | 3.2714          |
| 3.4166        | 16.91 | 2008000 | 3.2738          |
| 3.4054        | 16.98 | 2016000 | 3.2633          |
| 3.4054        | 17.05 | 2024000 | 3.2574          |
| 3.4022        | 17.11 | 2032000 | 3.2637          |
| 3.4022        | 17.18 | 2040000 | 3.2688          |
| 3.408         | 17.25 | 2048000 | 3.2667          |
| 3.408         | 17.32 | 2056000 | 3.2578          |
| 3.4065        | 17.38 | 2064000 | 3.2605          |
| 3.4065        | 17.45 | 2072000 | 3.2768          |
| 3.4105        | 17.52 | 2080000 | 3.2569          |
| 3.4105        | 17.59 | 2088000 | 3.2519          |
| 3.4011        | 17.65 | 2096000 | 3.2555          |
| 3.4011        | 17.72 | 2104000 | 3.2488          |
| 3.4078        | 17.79 | 2112000 | 3.2516          |
| 3.4078        | 17.86 | 2120000 | 3.2527          |
| 3.4105        | 17.92 | 2128000 | 3.2561          |
| 3.4105        | 17.99 | 2136000 | 3.2580          |
| 3.4054        | 18.06 | 2144000 | 3.2453          |
| 3.4054        | 18.13 | 2152000 | 3.2426          |
| 3.3937        | 18.19 | 2160000 | 3.2517          |
| 3.3937        | 18.26 | 2168000 | 3.2446          |
| 3.4001        | 18.33 | 2176000 | 3.2449          |
| 3.4001        | 18.39 | 2184000 | 3.2527          |
| 3.413         | 18.46 | 2192000 | 3.2557          |
| 3.413         | 18.53 | 2200000 | 3.2483          |
| 3.3882        | 18.6  | 2208000 | 3.2520          |
| 3.3882        | 18.66 | 2216000 | 3.2354          |
| 3.3974        | 18.73 | 2224000 | 3.2540          |
| 3.3974        | 18.8  | 2232000 | 3.2426          |
| 3.3864        | 18.87 | 2240000 | 3.2341          |
| 3.3864        | 18.93 | 2248000 | 3.2408          |
| 3.3896        | 19.0  | 2256000 | 3.2342          |
| 3.3896        | 19.07 | 2264000 | 3.2415          |
| 3.3845        | 19.14 | 2272000 | 3.2445          |
| 3.3845        | 19.2  | 2280000 | 3.2422          |
| 3.3916        | 19.27 | 2288000 | 3.2379          |
| 3.3916        | 19.34 | 2296000 | 3.2411          |
| 3.3919        | 19.41 | 2304000 | 3.2429          |
| 3.3919        | 19.47 | 2312000 | 3.2372          |
| 3.39          | 19.54 | 2320000 | 3.2380          |
| 3.39          | 19.61 | 2328000 | 3.2353          |
| 3.3905        | 19.68 | 2336000 | 3.2327          |
| 3.3905        | 19.74 | 2344000 | 3.2494          |
| 3.3826        | 19.81 | 2352000 | 3.2369          |
| 3.3826        | 19.88 | 2360000 | 3.2390          |
| 3.3935        | 19.94 | 2368000 | 3.2415          |
| 3.3935        | 20.01 | 2376000 | 3.2486          |
| 3.3846        | 20.08 | 2384000 | 3.2354          |
| 3.3846        | 20.15 | 2392000 | 3.2466          |
| 3.3875        | 20.21 | 2400000 | 3.2425          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
