transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q3-25p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q3-25p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.2870

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.02  | 8000    | 2.5705          |
| 2.7559        | 0.05  | 16000   | 2.4932          |
| 2.7559        | 0.07  | 24000   | 2.4516          |
| 2.5786        | 0.09  | 32000   | 2.4174          |
| 2.5786        | 0.11  | 40000   | 2.4071          |
| 2.5316        | 0.14  | 48000   | 2.3903          |
| 2.5316        | 0.16  | 56000   | 2.3744          |
| 2.5006        | 0.18  | 64000   | 2.3650          |
| 2.5006        | 0.2   | 72000   | 2.3600          |
| 2.483         | 0.23  | 80000   | 2.3548          |
| 2.483         | 0.25  | 88000   | 2.3485          |
| 2.4703        | 0.27  | 96000   | 2.3475          |
| 2.4703        | 0.29  | 104000  | 2.3384          |
| 2.47          | 0.32  | 112000  | 2.3330          |
| 2.47          | 0.34  | 120000  | 2.3354          |
| 2.4601        | 0.36  | 128000  | 2.3343          |
| 2.4601        | 0.38  | 136000  | 2.3282          |
| 2.4486        | 0.41  | 144000  | 2.3316          |
| 2.4486        | 0.43  | 152000  | 2.3180          |
| 2.4536        | 0.45  | 160000  | 2.3257          |
| 2.4536        | 0.47  | 168000  | 2.3222          |
| 2.4523        | 0.5   | 176000  | 2.3208          |
| 2.4523        | 0.52  | 184000  | 2.3218          |
| 2.4489        | 0.54  | 192000  | 2.3184          |
| 2.4489        | 0.56  | 200000  | 2.3225          |
| 2.4448        | 0.59  | 208000  | 2.3185          |
| 2.4448        | 0.61  | 216000  | 2.3139          |
| 2.4412        | 0.63  | 224000  | 2.3235          |
| 2.4412        | 0.65  | 232000  | 2.3148          |
| 2.442         | 0.68  | 240000  | 2.3146          |
| 2.442         | 0.7   | 248000  | 2.3145          |
| 2.4408        | 0.72  | 256000  | 2.3083          |
| 2.4408        | 0.74  | 264000  | 2.3068          |
| 2.4336        | 0.77  | 272000  | 2.3104          |
| 2.4336        | 0.79  | 280000  | 2.3147          |
| 2.4394        | 0.81  | 288000  | 2.3105          |
| 2.4394        | 0.83  | 296000  | 2.3135          |
| 2.4363        | 0.86  | 304000  | 2.3057          |
| 2.4363        | 0.88  | 312000  | 2.3050          |
| 2.4403        | 0.9   | 320000  | 2.3066          |
| 2.4403        | 0.92  | 328000  | 2.3076          |
| 2.4409        | 0.95  | 336000  | 2.3026          |
| 2.4409        | 0.97  | 344000  | 2.3045          |
| 2.4434        | 0.99  | 352000  | 2.3047          |
| 2.4434        | 1.01  | 360000  | 2.3080          |
| 2.4372        | 1.04  | 368000  | 2.3143          |
| 2.4372        | 1.06  | 376000  | 2.3049          |
| 2.4329        | 1.08  | 384000  | 2.3066          |
| 2.4329        | 1.1   | 392000  | 2.3050          |
| 2.437         | 1.13  | 400000  | 2.3012          |
| 2.437         | 1.15  | 408000  | 2.3033          |
| 2.4378        | 1.17  | 416000  | 2.3064          |
| 2.4378        | 1.19  | 424000  | 2.2984          |
| 2.4386        | 1.22  | 432000  | 2.3057          |
| 2.4386        | 1.24  | 440000  | 2.3035          |
| 2.4411        | 1.26  | 448000  | 2.2969          |
| 2.4411        | 1.28  | 456000  | 2.2930          |
| 2.4466        | 1.31  | 464000  | 2.3005          |
| 2.4466        | 1.33  | 472000  | 2.2975          |
| 2.4451        | 1.35  | 480000  | 2.3042          |
| 2.4451        | 1.37  | 488000  | 2.3061          |
| 2.4399        | 1.4   | 496000  | 2.2987          |
| 2.4399        | 1.42  | 504000  | 2.2967          |
| 2.4397        | 1.44  | 512000  | 2.3010          |
| 2.4397        | 1.47  | 520000  | 2.3019          |
| 2.4483        | 1.49  | 528000  | 2.3009          |
| 2.4483        | 1.51  | 536000  | 2.3048          |
| 2.4436        | 1.53  | 544000  | 2.3029          |
| 2.4436        | 1.56  | 552000  | 2.3026          |
| 2.4407        | 1.58  | 560000  | 2.3027          |
| 2.4407        | 1.6   | 568000  | 2.3061          |
| 2.4364        | 1.62  | 576000  | 2.2972          |
| 2.4364        | 1.65  | 584000  | 2.2967          |
| 2.4406        | 1.67  | 592000  | 2.2965          |
| 2.4406        | 1.69  | 600000  | 2.2966          |
| 2.4393        | 1.71  | 608000  | 2.2982          |
| 2.4393        | 1.74  | 616000  | 2.2993          |
| 2.4352        | 1.76  | 624000  | 2.2916          |
| 2.4352        | 1.78  | 632000  | 2.2931          |
| 2.4366        | 1.8   | 640000  | 2.3016          |
| 2.4366        | 1.83  | 648000  | 2.2984          |
| 2.4361        | 1.85  | 656000  | 2.2877          |
| 2.4361        | 1.87  | 664000  | 2.2983          |
| 2.437         | 1.89  | 672000  | 2.3033          |
| 2.437         | 1.92  | 680000  | 2.2928          |
| 2.4488        | 1.94  | 688000  | 2.2953          |
| 2.4488        | 1.96  | 696000  | 2.2945          |
| 2.4459        | 1.98  | 704000  | 2.2961          |
| 2.4459        | 2.01  | 712000  | 2.2899          |
| 2.4334        | 2.03  | 720000  | 2.2964          |
| 2.4334        | 2.05  | 728000  | 2.2896          |
| 2.4343        | 2.07  | 736000  | 2.2954          |
| 2.4343        | 2.1   | 744000  | 2.3004          |
| 2.4345        | 2.12  | 752000  | 2.2892          |
| 2.4345        | 2.14  | 760000  | 2.2996          |
| 2.4386        | 2.16  | 768000  | 2.2886          |
| 2.4386        | 2.19  | 776000  | 2.2974          |
| 2.434         | 2.21  | 784000  | 2.2882          |
| 2.434         | 2.23  | 792000  | 2.2965          |
| 2.4379        | 2.25  | 800000  | 2.2899          |
| 2.4379        | 2.28  | 808000  | 2.2938          |
| 2.4356        | 2.3   | 816000  | 2.2997          |
| 2.4356        | 2.32  | 824000  | 2.2942          |
| 2.4399        | 2.34  | 832000  | 2.2916          |
| 2.4399        | 2.37  | 840000  | 2.2934          |
| 2.437         | 2.39  | 848000  | 2.2978          |
| 2.437         | 2.41  | 856000  | 2.2834          |
| 2.4311        | 2.43  | 864000  | 2.2872          |
| 2.4311        | 2.46  | 872000  | 2.2928          |
| 2.4453        | 2.48  | 880000  | 2.2888          |
| 2.4453        | 2.5   | 888000  | 2.2933          |
| 2.4434        | 2.52  | 896000  | 2.2911          |
| 2.4434        | 2.55  | 904000  | 2.2929          |
| 2.443         | 2.57  | 912000  | 2.2926          |
| 2.443         | 2.59  | 920000  | 2.2908          |
| 2.4361        | 2.61  | 928000  | 2.2914          |
| 2.4361        | 2.64  | 936000  | 2.2878          |
| 2.44          | 2.66  | 944000  | 2.2872          |
| 2.44          | 2.68  | 952000  | 2.2857          |
| 2.4447        | 2.7   | 960000  | 2.2932          |
| 2.4447        | 2.73  | 968000  | 2.2918          |
| 2.4362        | 2.75  | 976000  | 2.2875          |
| 2.4362        | 2.77  | 984000  | 2.2900          |
| 2.4457        | 2.8   | 992000  | 2.2913          |
| 2.4457        | 2.82  | 1000000 | 2.2871          |
| 2.4474        | 2.84  | 1008000 | 2.2875          |
| 2.4474        | 2.86  | 1016000 | 2.2902          |
| 2.444         | 2.89  | 1024000 | 2.2878          |
| 2.444         | 2.91  | 1032000 | 2.2856          |
| 2.4316        | 2.93  | 1040000 | 2.2908          |
| 2.4316        | 2.95  | 1048000 | 2.2889          |
| 2.4388        | 2.98  | 1056000 | 2.2922          |
| 2.4388        | 3.0   | 1064000 | 2.2867          |
| 2.442         | 3.02  | 1072000 | 2.2912          |
| 2.442         | 3.04  | 1080000 | 2.2891          |
| 2.4388        | 3.07  | 1088000 | 2.2855          |
| 2.4388        | 3.09  | 1096000 | 2.2949          |
| 2.4296        | 3.11  | 1104000 | 2.2853          |
| 2.4296        | 3.13  | 1112000 | 2.2854          |
| 2.4411        | 3.16  | 1120000 | 2.2902          |
| 2.4411        | 3.18  | 1128000 | 2.2902          |
| 2.4354        | 3.2   | 1136000 | 2.2873          |
| 2.4354        | 3.22  | 1144000 | 2.2931          |
| 2.4436        | 3.25  | 1152000 | 2.2906          |
| 2.4436        | 3.27  | 1160000 | 2.2945          |
| 2.4372        | 3.29  | 1168000 | 2.2899          |
| 2.4372        | 3.31  | 1176000 | 2.2869          |
| 2.4327        | 3.34  | 1184000 | 2.2891          |
| 2.4327        | 3.36  | 1192000 | 2.2933          |
| 2.4387        | 3.38  | 1200000 | 2.2849          |
| 2.4387        | 3.4   | 1208000 | 2.2934          |
| 2.4433        | 3.43  | 1216000 | 2.2876          |
| 2.4433        | 3.45  | 1224000 | 2.2860          |
| 2.4396        | 3.47  | 1232000 | 2.2898          |
| 2.4396        | 3.49  | 1240000 | 2.2830          |
| 2.4332        | 3.52  | 1248000 | 2.2855          |
| 2.4332        | 3.54  | 1256000 | 2.2925          |
| 2.4332        | 3.56  | 1264000 | 2.2832          |
| 2.4332        | 3.58  | 1272000 | 2.2851          |
| 2.4307        | 3.61  | 1280000 | 2.2912          |
| 2.4307        | 3.63  | 1288000 | 2.2924          |
| 2.4432        | 3.65  | 1296000 | 2.2916          |
| 2.4432        | 3.67  | 1304000 | 2.2892          |
| 2.4319        | 3.7   | 1312000 | 2.2908          |
| 2.4319        | 3.72  | 1320000 | 2.2898          |
| 2.4394        | 3.74  | 1328000 | 2.2860          |
| 2.4394        | 3.76  | 1336000 | 2.2879          |
| 2.4462        | 3.79  | 1344000 | 2.2865          |
| 2.4462        | 3.81  | 1352000 | 2.2844          |
| 2.4373        | 3.83  | 1360000 | 2.2933          |
| 2.4373        | 3.85  | 1368000 | 2.2877          |
| 2.4436        | 3.88  | 1376000 | 2.2937          |
| 2.4436        | 3.9   | 1384000 | 2.2902          |
| 2.4387        | 3.92  | 1392000 | 2.2870          |
| 2.4387        | 3.94  | 1400000 | 2.2823          |
| 2.4384        | 3.97  | 1408000 | 2.2899          |
| 2.4384        | 3.99  | 1416000 | 2.2865          |
| 2.4389        | 4.01  | 1424000 | 2.2856          |
| 2.4389        | 4.03  | 1432000 | 2.2911          |
| 2.4408        | 4.06  | 1440000 | 2.2906          |
| 2.4408        | 4.08  | 1448000 | 2.2860          |
| 2.4424        | 4.1   | 1456000 | 2.2816          |
| 2.4424        | 4.12  | 1464000 | 2.2850          |
| 2.4446        | 4.15  | 1472000 | 2.2936          |
| 2.4446        | 4.17  | 1480000 | 2.2829          |
| 2.4419        | 4.19  | 1488000 | 2.2871          |
| 2.4419        | 4.22  | 1496000 | 2.2892          |
| 2.4327        | 4.24  | 1504000 | 2.2822          |
| 2.4327        | 4.26  | 1512000 | 2.2900          |
| 2.4346        | 4.28  | 1520000 | 2.2906          |
| 2.4346        | 4.31  | 1528000 | 2.2837          |
| 2.4342        | 4.33  | 1536000 | 2.2846          |
| 2.4342        | 4.35  | 1544000 | 2.2863          |
| 2.4381        | 4.37  | 1552000 | 2.2940          |
| 2.4381        | 4.4   | 1560000 | 2.2900          |
| 2.4445        | 4.42  | 1568000 | 2.2887          |
| 2.4445        | 4.44  | 1576000 | 2.2901          |
| 2.4306        | 4.46  | 1584000 | 2.2832          |
| 2.4306        | 4.49  | 1592000 | 2.2862          |
| 2.4348        | 4.51  | 1600000 | 2.2877          |
| 2.4348        | 4.53  | 1608000 | 2.2834          |
| 2.4446        | 4.55  | 1616000 | 2.2892          |
| 2.4446        | 4.58  | 1624000 | 2.2800          |
| 2.444         | 4.6   | 1632000 | 2.2891          |
| 2.444         | 4.62  | 1640000 | 2.2839          |
| 2.4335        | 4.64  | 1648000 | 2.2787          |
| 2.4335        | 4.67  | 1656000 | 2.2856          |
| 2.4369        | 4.69  | 1664000 | 2.2889          |
| 2.4369        | 4.71  | 1672000 | 2.2900          |
| 2.4446        | 4.73  | 1680000 | 2.2891          |
| 2.4446        | 4.76  | 1688000 | 2.2835          |
| 2.4334        | 4.78  | 1696000 | 2.2841          |
| 2.4334        | 4.8   | 1704000 | 2.2895          |
| 2.4426        | 4.82  | 1712000 | 2.2832          |
| 2.4426        | 4.85  | 1720000 | 2.2870          |
| 2.4434        | 4.87  | 1728000 | 2.2819          |
| 2.4434        | 4.89  | 1736000 | 2.2896          |
| 2.4382        | 4.91  | 1744000 | 2.2869          |
| 2.4382        | 4.94  | 1752000 | 2.2844          |
| 2.4405        | 4.96  | 1760000 | 2.2820          |
| 2.4405        | 4.98  | 1768000 | 2.2922          |
| 2.4507        | 5.0   | 1776000 | 2.2808          |
| 2.4507        | 5.03  | 1784000 | 2.2868          |
| 2.4437        | 5.05  | 1792000 | 2.2815          |
| 2.4437        | 5.07  | 1800000 | 2.2889          |
| 2.4373        | 5.09  | 1808000 | 2.2797          |
| 2.4373        | 5.12  | 1816000 | 2.2882          |
| 2.4368        | 5.14  | 1824000 | 2.2879          |
| 2.4368        | 5.16  | 1832000 | 2.2829          |
| 2.4398        | 5.18  | 1840000 | 2.2867          |
| 2.4398        | 5.21  | 1848000 | 2.2829          |
| 2.4469        | 5.23  | 1856000 | 2.2846          |
| 2.4469        | 5.25  | 1864000 | 2.2839          |
| 2.4457        | 5.27  | 1872000 | 2.2880          |
| 2.4457        | 5.3   | 1880000 | 2.2849          |
| 2.4444        | 5.32  | 1888000 | 2.2838          |
| 2.4444        | 5.34  | 1896000 | 2.2800          |
| 2.437         | 5.36  | 1904000 | 2.2915          |
| 2.437         | 5.39  | 1912000 | 2.2813          |
| 2.4415        | 5.41  | 1920000 | 2.2893          |
| 2.4415        | 5.43  | 1928000 | 2.2848          |
| 2.4472        | 5.45  | 1936000 | 2.2920          |
| 2.4472        | 5.48  | 1944000 | 2.2759          |
| 2.4418        | 5.5   | 1952000 | 2.2837          |
| 2.4418        | 5.52  | 1960000 | 2.2860          |
| 2.4406        | 5.54  | 1968000 | 2.2825          |
| 2.4406        | 5.57  | 1976000 | 2.2794          |
| 2.4359        | 5.59  | 1984000 | 2.2773          |
| 2.4359        | 5.61  | 1992000 | 2.2876          |
| 2.4416        | 5.64  | 2000000 | 2.2793          |
| 2.4416        | 5.66  | 2008000 | 2.2814          |
| 2.4327        | 5.68  | 2016000 | 2.2865          |
| 2.4327        | 5.7   | 2024000 | 2.2903          |
| 2.4395        | 5.73  | 2032000 | 2.2850          |
| 2.4395        | 5.75  | 2040000 | 2.2835          |
| 2.4379        | 5.77  | 2048000 | 2.2837          |
| 2.4379        | 5.79  | 2056000 | 2.2833          |
| 2.4471        | 5.82  | 2064000 | 2.2857          |
| 2.4471        | 5.84  | 2072000 | 2.2863          |
| 2.4443        | 5.86  | 2080000 | 2.2882          |
| 2.4443        | 5.88  | 2088000 | 2.2849          |
| 2.4406        | 5.91  | 2096000 | 2.2885          |
| 2.4406        | 5.93  | 2104000 | 2.2852          |
| 2.4502        | 5.95  | 2112000 | 2.2898          |
| 2.4502        | 5.97  | 2120000 | 2.2924          |
| 2.4356        | 6.0   | 2128000 | 2.2886          |
| 2.4356        | 6.02  | 2136000 | 2.2883          |
| 2.4431        | 6.04  | 2144000 | 2.2935          |
| 2.4431        | 6.06  | 2152000 | 2.2918          |
| 2.4379        | 6.09  | 2160000 | 2.2824          |
| 2.4379        | 6.11  | 2168000 | 2.2850          |
| 2.4504        | 6.13  | 2176000 | 2.2842          |
| 2.4504        | 6.15  | 2184000 | 2.2891          |
| 2.4352        | 6.18  | 2192000 | 2.2834          |
| 2.4352        | 6.2   | 2200000 | 2.2877          |
| 2.4385        | 6.22  | 2208000 | 2.2836          |
| 2.4385        | 6.24  | 2216000 | 2.2923          |
| 2.4401        | 6.27  | 2224000 | 2.2884          |
| 2.4401        | 6.29  | 2232000 | 2.2876          |
| 2.4396        | 6.31  | 2240000 | 2.2955          |
| 2.4396        | 6.33  | 2248000 | 2.2843          |
| 2.4384        | 6.36  | 2256000 | 2.2884          |
| 2.4384        | 6.38  | 2264000 | 2.2903          |
| 2.4365        | 6.4   | 2272000 | 2.2850          |
| 2.4365        | 6.42  | 2280000 | 2.2877          |
| 2.4361        | 6.45  | 2288000 | 2.2887          |
| 2.4361        | 6.47  | 2296000 | 2.2872          |
| 2.4409        | 6.49  | 2304000 | 2.2851          |
| 2.4409        | 6.51  | 2312000 | 2.2847          |
| 2.4423        | 6.54  | 2320000 | 2.2845          |
| 2.4423        | 6.56  | 2328000 | 2.2849          |
| 2.4409        | 6.58  | 2336000 | 2.2865          |
| 2.4409        | 6.6   | 2344000 | 2.2856          |
| 2.4468        | 6.63  | 2352000 | 2.2842          |
| 2.4468        | 6.65  | 2360000 | 2.2870          |
| 2.4461        | 6.67  | 2368000 | 2.2858          |
| 2.4461        | 6.69  | 2376000 | 2.2852          |
| 2.4469        | 6.72  | 2384000 | 2.2871          |
| 2.4469        | 6.74  | 2392000 | 2.2895          |
| 2.4413        | 6.76  | 2400000 | 2.2823          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
