transformers pytorch roberta fill-mask generated_from_trainer base_model:DouglasPontes/2020-Q3-50p-filtered-random license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: DouglasPontes/2020-Q3-50p-filtered-random
tags:
- generated_from_trainer
model-index:
- name: 2020-Q4-50p-filtered-random-prog_from_Q3
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q4-50p-filtered-random-prog_from_Q3

This model is a fine-tuned version of [DouglasPontes/2020-Q3-50p-filtered-random](https://huggingface.co/DouglasPontes/2020-Q3-50p-filtered-random) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.2730

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.03  | 8000    | 2.3816          |
| 2.5089        | 0.07  | 16000   | 2.3647          |
| 2.5089        | 0.1   | 24000   | 2.3552          |
| 2.4989        | 0.13  | 32000   | 2.3528          |
| 2.4989        | 0.17  | 40000   | 2.3486          |
| 2.4836        | 0.2   | 48000   | 2.3463          |
| 2.4836        | 0.24  | 56000   | 2.3411          |
| 2.4904        | 0.27  | 64000   | 2.3394          |
| 2.4904        | 0.3   | 72000   | 2.3350          |
| 2.4733        | 0.34  | 80000   | 2.3309          |
| 2.4733        | 0.37  | 88000   | 2.3289          |
| 2.4675        | 0.4   | 96000   | 2.3381          |
| 2.4675        | 0.44  | 104000  | 2.3317          |
| 2.4762        | 0.47  | 112000  | 2.3218          |
| 2.4762        | 0.51  | 120000  | 2.3267          |
| 2.4616        | 0.54  | 128000  | 2.3241          |
| 2.4616        | 0.57  | 136000  | 2.3281          |
| 2.4601        | 0.61  | 144000  | 2.3152          |
| 2.4601        | 0.64  | 152000  | 2.3243          |
| 2.4563        | 0.67  | 160000  | 2.3202          |
| 2.4563        | 0.71  | 168000  | 2.3193          |
| 2.459         | 0.74  | 176000  | 2.3182          |
| 2.459         | 0.77  | 184000  | 2.3247          |
| 2.4639        | 0.81  | 192000  | 2.3201          |
| 2.4639        | 0.84  | 200000  | 2.3243          |
| 2.4561        | 0.88  | 208000  | 2.3218          |
| 2.4561        | 0.91  | 216000  | 2.3137          |
| 2.4556        | 0.94  | 224000  | 2.3180          |
| 2.4556        | 0.98  | 232000  | 2.3147          |
| 2.4573        | 1.01  | 240000  | 2.3100          |
| 2.4573        | 1.04  | 248000  | 2.3118          |
| 2.4516        | 1.08  | 256000  | 2.3158          |
| 2.4516        | 1.11  | 264000  | 2.3133          |
| 2.4561        | 1.15  | 272000  | 2.3065          |
| 2.4561        | 1.18  | 280000  | 2.3183          |
| 2.4476        | 1.21  | 288000  | 2.3106          |
| 2.4476        | 1.25  | 296000  | 2.3131          |
| 2.4503        | 1.28  | 304000  | 2.3104          |
| 2.4503        | 1.31  | 312000  | 2.3101          |
| 2.4495        | 1.35  | 320000  | 2.3086          |
| 2.4495        | 1.38  | 328000  | 2.3057          |
| 2.4534        | 1.41  | 336000  | 2.3086          |
| 2.4534        | 1.45  | 344000  | 2.3093          |
| 2.4486        | 1.48  | 352000  | 2.3018          |
| 2.4486        | 1.52  | 360000  | 2.3060          |
| 2.4457        | 1.55  | 368000  | 2.3083          |
| 2.4457        | 1.58  | 376000  | 2.3110          |
| 2.4443        | 1.62  | 384000  | 2.2975          |
| 2.4443        | 1.65  | 392000  | 2.3009          |
| 2.4405        | 1.68  | 400000  | 2.3067          |
| 2.4405        | 1.72  | 408000  | 2.3027          |
| 2.4531        | 1.75  | 416000  | 2.3050          |
| 2.4531        | 1.79  | 424000  | 2.3026          |
| 2.4539        | 1.82  | 432000  | 2.2929          |
| 2.4539        | 1.85  | 440000  | 2.3051          |
| 2.4499        | 1.89  | 448000  | 2.3035          |
| 2.4499        | 1.92  | 456000  | 2.3011          |
| 2.4401        | 1.95  | 464000  | 2.2920          |
| 2.4401        | 1.99  | 472000  | 2.2999          |
| 2.4401        | 2.02  | 480000  | 2.3034          |
| 2.4401        | 2.05  | 488000  | 2.3021          |
| 2.4433        | 2.09  | 496000  | 2.3102          |
| 2.4433        | 2.12  | 504000  | 2.2985          |
| 2.4445        | 2.16  | 512000  | 2.3018          |
| 2.4445        | 2.19  | 520000  | 2.2996          |
| 2.4379        | 2.22  | 528000  | 2.3006          |
| 2.4379        | 2.26  | 536000  | 2.2970          |
| 2.4454        | 2.29  | 544000  | 2.3014          |
| 2.4454        | 2.32  | 552000  | 2.2992          |
| 2.4457        | 2.36  | 560000  | 2.2962          |
| 2.4457        | 2.39  | 568000  | 2.3009          |
| 2.4354        | 2.43  | 576000  | 2.2960          |
| 2.4354        | 2.46  | 584000  | 2.3008          |
| 2.4361        | 2.49  | 592000  | 2.2898          |
| 2.4361        | 2.53  | 600000  | 2.3060          |
| 2.4377        | 2.56  | 608000  | 2.2990          |
| 2.4377        | 2.59  | 616000  | 2.2989          |
| 2.4416        | 2.63  | 624000  | 2.2969          |
| 2.4416        | 2.66  | 632000  | 2.2933          |
| 2.434         | 2.69  | 640000  | 2.2998          |
| 2.434         | 2.73  | 648000  | 2.2948          |
| 2.433         | 2.76  | 656000  | 2.2892          |
| 2.433         | 2.8   | 664000  | 2.2929          |
| 2.44          | 2.83  | 672000  | 2.2973          |
| 2.44          | 2.86  | 680000  | 2.2926          |
| 2.4291        | 2.9   | 688000  | 2.2990          |
| 2.4291        | 2.93  | 696000  | 2.2937          |
| 2.4336        | 2.96  | 704000  | 2.2894          |
| 2.4336        | 3.0   | 712000  | 2.2958          |
| 2.439         | 3.03  | 720000  | 2.2956          |
| 2.439         | 3.07  | 728000  | 2.2928          |
| 2.4405        | 3.1   | 736000  | 2.2956          |
| 2.4405        | 3.13  | 744000  | 2.2905          |
| 2.4332        | 3.17  | 752000  | 2.2921          |
| 2.4332        | 3.2   | 760000  | 2.2907          |
| 2.4353        | 3.23  | 768000  | 2.2879          |
| 2.4353        | 3.27  | 776000  | 2.2929          |
| 2.4273        | 3.3   | 784000  | 2.2953          |
| 2.4273        | 3.33  | 792000  | 2.2917          |
| 2.4233        | 3.37  | 800000  | 2.2947          |
| 2.4233        | 3.4   | 808000  | 2.2943          |
| 2.4324        | 3.44  | 816000  | 2.2940          |
| 2.4324        | 3.47  | 824000  | 2.2911          |
| 2.4461        | 3.5   | 832000  | 2.2920          |
| 2.4461        | 3.54  | 840000  | 2.2911          |
| 2.4267        | 3.57  | 848000  | 2.2940          |
| 2.4267        | 3.6   | 856000  | 2.2890          |
| 2.4313        | 3.64  | 864000  | 2.2913          |
| 2.4313        | 3.67  | 872000  | 2.2967          |
| 2.4388        | 3.71  | 880000  | 2.2907          |
| 2.4388        | 3.74  | 888000  | 2.2952          |
| 2.4326        | 3.77  | 896000  | 2.2873          |
| 2.4326        | 3.81  | 904000  | 2.2871          |
| 2.4312        | 3.84  | 912000  | 2.2880          |
| 2.4312        | 3.87  | 920000  | 2.2941          |
| 2.4398        | 3.91  | 928000  | 2.2925          |
| 2.4398        | 3.94  | 936000  | 2.2965          |
| 2.441         | 3.97  | 944000  | 2.2890          |
| 2.441         | 4.01  | 952000  | 2.2946          |
| 2.4345        | 4.04  | 960000  | 2.2910          |
| 2.4345        | 4.08  | 968000  | 2.2792          |
| 2.4332        | 4.11  | 976000  | 2.2856          |
| 2.4332        | 4.14  | 984000  | 2.2879          |
| 2.4375        | 4.18  | 992000  | 2.2861          |
| 2.4375        | 4.21  | 1000000 | 2.2892          |
| 2.4282        | 4.24  | 1008000 | 2.2894          |
| 2.4282        | 4.28  | 1016000 | 2.2902          |
| 2.4231        | 4.31  | 1024000 | 2.2830          |
| 2.4231        | 4.35  | 1032000 | 2.2948          |
| 2.4299        | 4.38  | 1040000 | 2.2915          |
| 2.4299        | 4.41  | 1048000 | 2.2922          |
| 2.4353        | 4.45  | 1056000 | 2.2876          |
| 2.4353        | 4.48  | 1064000 | 2.2893          |
| 2.4308        | 4.51  | 1072000 | 2.2920          |
| 2.4308        | 4.55  | 1080000 | 2.2860          |
| 2.4358        | 4.58  | 1088000 | 2.2907          |
| 2.4358        | 4.61  | 1096000 | 2.2808          |
| 2.4341        | 4.65  | 1104000 | 2.2902          |
| 2.4341        | 4.68  | 1112000 | 2.2815          |
| 2.4315        | 4.72  | 1120000 | 2.2961          |
| 2.4315        | 4.75  | 1128000 | 2.2885          |
| 2.434         | 4.78  | 1136000 | 2.2917          |
| 2.434         | 4.82  | 1144000 | 2.2851          |
| 2.4324        | 4.85  | 1152000 | 2.2837          |
| 2.4324        | 4.88  | 1160000 | 2.2883          |
| 2.4297        | 4.92  | 1168000 | 2.2824          |
| 2.4297        | 4.95  | 1176000 | 2.2832          |
| 2.436         | 4.99  | 1184000 | 2.2865          |
| 2.436         | 5.02  | 1192000 | 2.2816          |
| 2.4329        | 5.05  | 1200000 | 2.2862          |
| 2.4329        | 5.09  | 1208000 | 2.2847          |
| 2.4276        | 5.12  | 1216000 | 2.2951          |
| 2.4276        | 5.15  | 1224000 | 2.2980          |
| 2.4362        | 5.19  | 1232000 | 2.2889          |
| 2.4362        | 5.22  | 1240000 | 2.2914          |
| 2.4309        | 5.25  | 1248000 | 2.2915          |
| 2.4309        | 5.29  | 1256000 | 2.2822          |
| 2.4414        | 5.32  | 1264000 | 2.2871          |
| 2.4414        | 5.36  | 1272000 | 2.2890          |
| 2.4241        | 5.39  | 1280000 | 2.2844          |
| 2.4241        | 5.42  | 1288000 | 2.2812          |
| 2.4251        | 5.46  | 1296000 | 2.2874          |
| 2.4251        | 5.49  | 1304000 | 2.2846          |
| 2.4318        | 5.52  | 1312000 | 2.2831          |
| 2.4318        | 5.56  | 1320000 | 2.2895          |
| 2.4247        | 5.59  | 1328000 | 2.2796          |
| 2.4247        | 5.63  | 1336000 | 2.2834          |
| 2.4305        | 5.66  | 1344000 | 2.2811          |
| 2.4305        | 5.69  | 1352000 | 2.2922          |
| 2.4336        | 5.73  | 1360000 | 2.2830          |
| 2.4336        | 5.76  | 1368000 | 2.2904          |
| 2.428         | 5.79  | 1376000 | 2.2843          |
| 2.428         | 5.83  | 1384000 | 2.2804          |
| 2.4254        | 5.86  | 1392000 | 2.2852          |
| 2.4254        | 5.89  | 1400000 | 2.2858          |
| 2.4287        | 5.93  | 1408000 | 2.2922          |
| 2.4287        | 5.96  | 1416000 | 2.2847          |
| 2.4291        | 6.0   | 1424000 | 2.2856          |
| 2.4291        | 6.03  | 1432000 | 2.2876          |
| 2.4289        | 6.06  | 1440000 | 2.2822          |
| 2.4289        | 6.1   | 1448000 | 2.2787          |
| 2.4272        | 6.13  | 1456000 | 2.2811          |
| 2.4272        | 6.16  | 1464000 | 2.2853          |
| 2.4267        | 6.2   | 1472000 | 2.2818          |
| 2.4267        | 6.23  | 1480000 | 2.2765          |
| 2.4237        | 6.27  | 1488000 | 2.2791          |
| 2.4237        | 6.3   | 1496000 | 2.2768          |
| 2.4277        | 6.33  | 1504000 | 2.2866          |
| 2.4277        | 6.37  | 1512000 | 2.2821          |
| 2.4316        | 6.4   | 1520000 | 2.2856          |
| 2.4316        | 6.43  | 1528000 | 2.2820          |
| 2.4222        | 6.47  | 1536000 | 2.2891          |
| 2.4222        | 6.5   | 1544000 | 2.2803          |
| 2.426         | 6.53  | 1552000 | 2.2797          |
| 2.426         | 6.57  | 1560000 | 2.2844          |
| 2.422         | 6.6   | 1568000 | 2.2872          |
| 2.422         | 6.64  | 1576000 | 2.2904          |
| 2.4323        | 6.67  | 1584000 | 2.2797          |
| 2.4323        | 6.7   | 1592000 | 2.2757          |
| 2.4315        | 6.74  | 1600000 | 2.2874          |
| 2.4315        | 6.77  | 1608000 | 2.2763          |
| 2.421         | 6.8   | 1616000 | 2.2857          |
| 2.421         | 6.84  | 1624000 | 2.2804          |
| 2.4299        | 6.87  | 1632000 | 2.2825          |
| 2.4299        | 6.91  | 1640000 | 2.2819          |
| 2.4289        | 6.94  | 1648000 | 2.2824          |
| 2.4289        | 6.97  | 1656000 | 2.2821          |
| 2.4257        | 7.01  | 1664000 | 2.2802          |
| 2.4257        | 7.04  | 1672000 | 2.2760          |
| 2.4227        | 7.07  | 1680000 | 2.2810          |
| 2.4227        | 7.11  | 1688000 | 2.2777          |
| 2.4287        | 7.14  | 1696000 | 2.2772          |
| 2.4287        | 7.17  | 1704000 | 2.2786          |
| 2.4227        | 7.21  | 1712000 | 2.2859          |
| 2.4227        | 7.24  | 1720000 | 2.2862          |
| 2.4262        | 7.28  | 1728000 | 2.2789          |
| 2.4262        | 7.31  | 1736000 | 2.2848          |
| 2.4263        | 7.34  | 1744000 | 2.2754          |
| 2.4263        | 7.38  | 1752000 | 2.2778          |
| 2.4246        | 7.41  | 1760000 | 2.2735          |
| 2.4246        | 7.44  | 1768000 | 2.2827          |
| 2.4147        | 7.48  | 1776000 | 2.2850          |
| 2.4147        | 7.51  | 1784000 | 2.2821          |
| 2.4288        | 7.55  | 1792000 | 2.2803          |
| 2.4288        | 7.58  | 1800000 | 2.2760          |
| 2.4231        | 7.61  | 1808000 | 2.2749          |
| 2.4231        | 7.65  | 1816000 | 2.2749          |
| 2.4243        | 7.68  | 1824000 | 2.2743          |
| 2.4243        | 7.71  | 1832000 | 2.2792          |
| 2.4215        | 7.75  | 1840000 | 2.2752          |
| 2.4215        | 7.78  | 1848000 | 2.2770          |
| 2.4213        | 7.81  | 1856000 | 2.2802          |
| 2.4213        | 7.85  | 1864000 | 2.2796          |
| 2.4236        | 7.88  | 1872000 | 2.2883          |
| 2.4236        | 7.92  | 1880000 | 2.2792          |
| 2.4237        | 7.95  | 1888000 | 2.2726          |
| 2.4237        | 7.98  | 1896000 | 2.2816          |
| 2.4183        | 8.02  | 1904000 | 2.2790          |
| 2.4183        | 8.05  | 1912000 | 2.2815          |
| 2.4215        | 8.08  | 1920000 | 2.2774          |
| 2.4215        | 8.12  | 1928000 | 2.2700          |
| 2.4258        | 8.15  | 1936000 | 2.2763          |
| 2.4258        | 8.19  | 1944000 | 2.2786          |
| 2.4209        | 8.22  | 1952000 | 2.2763          |
| 2.4209        | 8.25  | 1960000 | 2.2789          |
| 2.4217        | 8.29  | 1968000 | 2.2784          |
| 2.4217        | 8.32  | 1976000 | 2.2773          |
| 2.4279        | 8.35  | 1984000 | 2.2861          |
| 2.4279        | 8.39  | 1992000 | 2.2728          |
| 2.4268        | 8.42  | 2000000 | 2.2762          |
| 2.4268        | 8.45  | 2008000 | 2.2789          |
| 2.4177        | 8.49  | 2016000 | 2.2822          |
| 2.4177        | 8.52  | 2024000 | 2.2759          |
| 2.4166        | 8.56  | 2032000 | 2.2792          |
| 2.4166        | 8.59  | 2040000 | 2.2721          |
| 2.4223        | 8.62  | 2048000 | 2.2768          |
| 2.4223        | 8.66  | 2056000 | 2.2726          |
| 2.4139        | 8.69  | 2064000 | 2.2825          |
| 2.4139        | 8.72  | 2072000 | 2.2739          |
| 2.4236        | 8.76  | 2080000 | 2.2834          |
| 2.4236        | 8.79  | 2088000 | 2.2750          |
| 2.4235        | 8.83  | 2096000 | 2.2752          |
| 2.4235        | 8.86  | 2104000 | 2.2803          |
| 2.4193        | 8.89  | 2112000 | 2.2763          |
| 2.4193        | 8.93  | 2120000 | 2.2755          |
| 2.4179        | 8.96  | 2128000 | 2.2794          |
| 2.4179        | 8.99  | 2136000 | 2.2711          |
| 2.4181        | 9.03  | 2144000 | 2.2792          |
| 2.4181        | 9.06  | 2152000 | 2.2752          |
| 2.4173        | 9.09  | 2160000 | 2.2775          |
| 2.4173        | 9.13  | 2168000 | 2.2752          |
| 2.4242        | 9.16  | 2176000 | 2.2729          |
| 2.4242        | 9.2   | 2184000 | 2.2793          |
| 2.4166        | 9.23  | 2192000 | 2.2719          |
| 2.4166        | 9.26  | 2200000 | 2.2820          |
| 2.4181        | 9.3   | 2208000 | 2.2716          |
| 2.4181        | 9.33  | 2216000 | 2.2855          |
| 2.4245        | 9.36  | 2224000 | 2.2805          |
| 2.4245        | 9.4   | 2232000 | 2.2721          |
| 2.4204        | 9.43  | 2240000 | 2.2707          |
| 2.4204        | 9.47  | 2248000 | 2.2767          |
| 2.4255        | 9.5   | 2256000 | 2.2710          |
| 2.4255        | 9.53  | 2264000 | 2.2814          |
| 2.4254        | 9.57  | 2272000 | 2.2746          |
| 2.4254        | 9.6   | 2280000 | 2.2766          |
| 2.4232        | 9.63  | 2288000 | 2.2725          |
| 2.4232        | 9.67  | 2296000 | 2.2765          |
| 2.4189        | 9.7   | 2304000 | 2.2756          |
| 2.4189        | 9.73  | 2312000 | 2.2768          |
| 2.4105        | 9.77  | 2320000 | 2.2804          |
| 2.4105        | 9.8   | 2328000 | 2.2873          |
| 2.415         | 9.84  | 2336000 | 2.2783          |
| 2.415         | 9.87  | 2344000 | 2.2737          |
| 2.4174        | 9.9   | 2352000 | 2.2786          |
| 2.4174        | 9.94  | 2360000 | 2.2730          |
| 2.4199        | 9.97  | 2368000 | 2.2794          |
| 2.4199        | 10.0  | 2376000 | 2.2848          |
| 2.4224        | 10.04 | 2384000 | 2.2811          |
| 2.4224        | 10.07 | 2392000 | 2.2818          |
| 2.4226        | 10.11 | 2400000 | 2.2798          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
