transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q3-90p-filtered-random
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q3-90p-filtered-random

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.2598

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.17  | 8000    | 2.5349          |
| 2.7955        | 0.34  | 16000   | 2.4448          |
| 2.7955        | 0.51  | 24000   | 2.3949          |
| 2.5335        | 0.67  | 32000   | 2.3699          |
| 2.5335        | 0.84  | 40000   | 2.3544          |
| 2.4757        | 1.01  | 48000   | 2.3477          |
| 2.4757        | 1.18  | 56000   | 2.3281          |
| 2.446         | 1.35  | 64000   | 2.3171          |
| 2.446         | 1.52  | 72000   | 2.3082          |
| 2.4291        | 1.68  | 80000   | 2.3170          |
| 2.4291        | 1.85  | 88000   | 2.2962          |
| 2.4275        | 2.02  | 96000   | 2.3044          |
| 2.4275        | 2.19  | 104000  | 2.2738          |
| 2.4148        | 2.36  | 112000  | 2.2927          |
| 2.4148        | 2.53  | 120000  | 2.2684          |
| 2.4062        | 2.69  | 128000  | 2.2891          |
| 2.4062        | 2.86  | 136000  | 2.2789          |
| 2.4022        | 3.03  | 144000  | 2.2659          |
| 2.4022        | 3.2   | 152000  | 2.2824          |
| 2.3943        | 3.37  | 160000  | 2.2684          |
| 2.3943        | 3.54  | 168000  | 2.2683          |
| 2.3957        | 3.71  | 176000  | 2.2737          |
| 2.3957        | 3.87  | 184000  | 2.2779          |
| 2.3976        | 4.04  | 192000  | 2.2710          |
| 2.3976        | 4.21  | 200000  | 2.2495          |
| 2.3933        | 4.38  | 208000  | 2.2660          |
| 2.3933        | 4.55  | 216000  | 2.2687          |
| 2.4039        | 4.72  | 224000  | 2.2581          |
| 2.4039        | 4.88  | 232000  | 2.2656          |
| 2.3966        | 5.05  | 240000  | 2.2543          |
| 2.3966        | 5.22  | 248000  | 2.2768          |
| 2.3902        | 5.39  | 256000  | 2.2551          |
| 2.3902        | 5.56  | 264000  | 2.2782          |
| 2.3906        | 5.73  | 272000  | 2.2639          |
| 2.3906        | 5.89  | 280000  | 2.2585          |
| 2.3849        | 6.06  | 288000  | 2.2540          |
| 2.3849        | 6.23  | 296000  | 2.2749          |
| 2.3805        | 6.4   | 304000  | 2.2503          |
| 2.3805        | 6.57  | 312000  | 2.2739          |
| 2.3873        | 6.74  | 320000  | 2.2541          |
| 2.3873        | 6.91  | 328000  | 2.2512          |
| 2.3942        | 7.07  | 336000  | 2.2595          |
| 2.3942        | 7.24  | 344000  | 2.2603          |
| 2.386         | 7.41  | 352000  | 2.2575          |
| 2.386         | 7.58  | 360000  | 2.2789          |
| 2.3806        | 7.75  | 368000  | 2.2650          |
| 2.3806        | 7.92  | 376000  | 2.2706          |
| 2.3883        | 8.08  | 384000  | 2.2652          |
| 2.3883        | 8.25  | 392000  | 2.2540          |
| 2.3922        | 8.42  | 400000  | 2.2683          |
| 2.3922        | 8.59  | 408000  | 2.2638          |
| 2.3887        | 8.76  | 416000  | 2.2535          |
| 2.3887        | 8.93  | 424000  | 2.2529          |
| 2.3818        | 9.09  | 432000  | 2.2483          |
| 2.3818        | 9.26  | 440000  | 2.2574          |
| 2.387         | 9.43  | 448000  | 2.2624          |
| 2.387         | 9.6   | 456000  | 2.2664          |
| 2.3839        | 9.77  | 464000  | 2.2572          |
| 2.3839        | 9.94  | 472000  | 2.2524          |
| 2.3901        | 10.11 | 480000  | 2.2533          |
| 2.3901        | 10.27 | 488000  | 2.2501          |
| 2.382         | 10.44 | 496000  | 2.2669          |
| 2.382         | 10.61 | 504000  | 2.2596          |
| 2.3829        | 10.78 | 512000  | 2.2705          |
| 2.3829        | 10.95 | 520000  | 2.2553          |
| 2.3963        | 11.12 | 528000  | 2.2741          |
| 2.3963        | 11.28 | 536000  | 2.2664          |
| 2.3843        | 11.45 | 544000  | 2.2532          |
| 2.3843        | 11.62 | 552000  | 2.2720          |
| 2.3853        | 11.79 | 560000  | 2.2532          |
| 2.3853        | 11.96 | 568000  | 2.2700          |
| 2.3907        | 12.13 | 576000  | 2.2571          |
| 2.3907        | 12.29 | 584000  | 2.2523          |
| 2.3865        | 12.46 | 592000  | 2.2458          |
| 2.3865        | 12.63 | 600000  | 2.2647          |
| 2.3827        | 12.8  | 608000  | 2.2490          |
| 2.3827        | 12.97 | 616000  | 2.2624          |
| 2.3869        | 13.14 | 624000  | 2.2538          |
| 2.3869        | 13.3  | 632000  | 2.2357          |
| 2.3958        | 13.47 | 640000  | 2.2509          |
| 2.3958        | 13.64 | 648000  | 2.2690          |
| 2.3852        | 13.81 | 656000  | 2.2476          |
| 2.3852        | 13.98 | 664000  | 2.2721          |
| 2.3889        | 14.15 | 672000  | 2.2537          |
| 2.3889        | 14.32 | 680000  | 2.2723          |
| 2.3839        | 14.48 | 688000  | 2.2664          |
| 2.3839        | 14.65 | 696000  | 2.2726          |
| 2.3884        | 14.82 | 704000  | 2.2652          |
| 2.3884        | 14.99 | 712000  | 2.2633          |
| 2.3827        | 15.16 | 720000  | 2.2681          |
| 2.3827        | 15.33 | 728000  | 2.2543          |
| 2.3861        | 15.49 | 736000  | 2.2634          |
| 2.3861        | 15.66 | 744000  | 2.2707          |
| 2.3812        | 15.83 | 752000  | 2.2575          |
| 2.3812        | 16.0  | 760000  | 2.2549          |
| 2.3862        | 16.17 | 768000  | 2.2446          |
| 2.3862        | 16.34 | 776000  | 2.2617          |
| 2.3859        | 16.5  | 784000  | 2.2505          |
| 2.3859        | 16.67 | 792000  | 2.2720          |
| 2.3873        | 16.84 | 800000  | 2.2521          |
| 2.3873        | 17.01 | 808000  | 2.2543          |
| 2.381         | 17.18 | 816000  | 2.2675          |
| 2.381         | 17.35 | 824000  | 2.2545          |
| 2.3851        | 17.52 | 832000  | 2.2489          |
| 2.3851        | 17.68 | 840000  | 2.2606          |
| 2.3878        | 17.85 | 848000  | 2.2580          |
| 2.3878        | 18.02 | 856000  | 2.2604          |
| 2.3812        | 18.19 | 864000  | 2.2631          |
| 2.3812        | 18.36 | 872000  | 2.2505          |
| 2.3849        | 18.53 | 880000  | 2.2658          |
| 2.3849        | 18.69 | 888000  | 2.2567          |
| 2.3833        | 18.86 | 896000  | 2.2533          |
| 2.3833        | 19.03 | 904000  | 2.2456          |
| 2.3847        | 19.2  | 912000  | 2.2533          |
| 2.3847        | 19.37 | 920000  | 2.2575          |
| 2.3869        | 19.54 | 928000  | 2.2668          |
| 2.3869        | 19.7  | 936000  | 2.2599          |
| 2.3867        | 19.87 | 944000  | 2.2680          |
| 2.3867        | 20.04 | 952000  | 2.2669          |
| 2.3942        | 20.21 | 960000  | 2.2483          |
| 2.3942        | 20.38 | 968000  | 2.2734          |
| 2.3863        | 20.55 | 976000  | 2.2623          |
| 2.3863        | 20.72 | 984000  | 2.2650          |
| 2.3924        | 20.88 | 992000  | 2.2603          |
| 2.3924        | 21.05 | 1000000 | 2.2708          |
| 2.3871        | 21.22 | 1008000 | 2.2512          |
| 2.3871        | 21.39 | 1016000 | 2.2568          |
| 2.3827        | 21.56 | 1024000 | 2.2676          |
| 2.3827        | 21.73 | 1032000 | 2.2710          |
| 2.3799        | 21.89 | 1040000 | 2.2804          |
| 2.3799        | 22.06 | 1048000 | 2.2499          |
| 2.3863        | 22.23 | 1056000 | 2.2557          |
| 2.3863        | 22.4  | 1064000 | 2.2604          |
| 2.3858        | 22.57 | 1072000 | 2.2832          |
| 2.3858        | 22.74 | 1080000 | 2.2443          |
| 2.3859        | 22.9  | 1088000 | 2.2604          |
| 2.3859        | 23.07 | 1096000 | 2.2631          |
| 2.3846        | 23.24 | 1104000 | 2.2690          |
| 2.3846        | 23.41 | 1112000 | 2.2595          |
| 2.3887        | 23.58 | 1120000 | 2.2501          |
| 2.3887        | 23.75 | 1128000 | 2.2533          |
| 2.3856        | 23.92 | 1136000 | 2.2529          |
| 2.3856        | 24.08 | 1144000 | 2.2456          |
| 2.3856        | 24.25 | 1152000 | 2.2544          |
| 2.3856        | 24.42 | 1160000 | 2.2554          |
| 2.3867        | 24.59 | 1168000 | 2.2596          |
| 2.3867        | 24.76 | 1176000 | 2.2522          |
| 2.3795        | 24.93 | 1184000 | 2.2493          |
| 2.3795        | 25.09 | 1192000 | 2.2609          |
| 2.3926        | 25.26 | 1200000 | 2.2658          |
| 2.3926        | 25.43 | 1208000 | 2.2593          |
| 2.3887        | 25.6  | 1216000 | 2.2704          |
| 2.3887        | 25.77 | 1224000 | 2.2632          |
| 2.3926        | 25.94 | 1232000 | 2.2628          |
| 2.3926        | 26.1  | 1240000 | 2.2657          |
| 2.3809        | 26.27 | 1248000 | 2.2546          |
| 2.3809        | 26.44 | 1256000 | 2.2596          |
| 2.3878        | 26.61 | 1264000 | 2.2545          |
| 2.3878        | 26.78 | 1272000 | 2.2668          |
| 2.3861        | 26.95 | 1280000 | 2.2534          |
| 2.3861        | 27.12 | 1288000 | 2.2612          |
| 2.3815        | 27.28 | 1296000 | 2.2441          |
| 2.3815        | 27.45 | 1304000 | 2.2714          |
| 2.3861        | 27.62 | 1312000 | 2.2604          |
| 2.3861        | 27.79 | 1320000 | 2.2535          |
| 2.388         | 27.96 | 1328000 | 2.2466          |
| 2.388         | 28.13 | 1336000 | 2.2581          |
| 2.3864        | 28.29 | 1344000 | 2.2572          |
| 2.3864        | 28.46 | 1352000 | 2.2381          |
| 2.39          | 28.63 | 1360000 | 2.2398          |
| 2.39          | 28.8  | 1368000 | 2.2695          |
| 2.39          | 28.97 | 1376000 | 2.2628          |
| 2.39          | 29.14 | 1384000 | 2.2599          |
| 2.3804        | 29.3  | 1392000 | 2.2628          |
| 2.3804        | 29.47 | 1400000 | 2.2722          |
| 2.3858        | 29.64 | 1408000 | 2.2490          |
| 2.3858        | 29.81 | 1416000 | 2.2627          |
| 2.3804        | 29.98 | 1424000 | 2.2623          |
| 2.3804        | 30.15 | 1432000 | 2.2522          |
| 2.3834        | 30.32 | 1440000 | 2.2633          |
| 2.3834        | 30.48 | 1448000 | 2.2553          |
| 2.3853        | 30.65 | 1456000 | 2.2391          |
| 2.3853        | 30.82 | 1464000 | 2.2616          |
| 2.3946        | 30.99 | 1472000 | 2.2631          |
| 2.3946        | 31.16 | 1480000 | 2.2639          |
| 2.385         | 31.33 | 1488000 | 2.2736          |
| 2.385         | 31.49 | 1496000 | 2.2715          |
| 2.387         | 31.66 | 1504000 | 2.2557          |
| 2.387         | 31.83 | 1512000 | 2.2583          |
| 2.3831        | 32.0  | 1520000 | 2.2544          |
| 2.3831        | 32.17 | 1528000 | 2.2756          |
| 2.3835        | 32.34 | 1536000 | 2.2794          |
| 2.3835        | 32.5  | 1544000 | 2.2648          |
| 2.3857        | 32.67 | 1552000 | 2.2563          |
| 2.3857        | 32.84 | 1560000 | 2.2537          |
| 2.3856        | 33.01 | 1568000 | 2.2610          |
| 2.3856        | 33.18 | 1576000 | 2.2646          |
| 2.3902        | 33.35 | 1584000 | 2.2545          |
| 2.3902        | 33.52 | 1592000 | 2.2710          |
| 2.3897        | 33.68 | 1600000 | 2.2601          |
| 2.3897        | 33.85 | 1608000 | 2.2543          |
| 2.3866        | 34.02 | 1616000 | 2.2526          |
| 2.3866        | 34.19 | 1624000 | 2.2629          |
| 2.3823        | 34.36 | 1632000 | 2.2617          |
| 2.3823        | 34.53 | 1640000 | 2.2520          |
| 2.3874        | 34.69 | 1648000 | 2.2612          |
| 2.3874        | 34.86 | 1656000 | 2.2569          |
| 2.3895        | 35.03 | 1664000 | 2.2633          |
| 2.3895        | 35.2  | 1672000 | 2.2593          |
| 2.3857        | 35.37 | 1680000 | 2.2651          |
| 2.3857        | 35.54 | 1688000 | 2.2567          |
| 2.3811        | 35.7  | 1696000 | 2.2534          |
| 2.3811        | 35.87 | 1704000 | 2.2633          |
| 2.3944        | 36.04 | 1712000 | 2.2504          |
| 2.3944        | 36.21 | 1720000 | 2.2519          |
| 2.3883        | 36.38 | 1728000 | 2.2572          |
| 2.3883        | 36.55 | 1736000 | 2.2576          |
| 2.3859        | 36.72 | 1744000 | 2.2719          |
| 2.3859        | 36.88 | 1752000 | 2.2668          |
| 2.3914        | 37.05 | 1760000 | 2.2509          |
| 2.3914        | 37.22 | 1768000 | 2.2601          |
| 2.3848        | 37.39 | 1776000 | 2.2687          |
| 2.3848        | 37.56 | 1784000 | 2.2513          |
| 2.3903        | 37.73 | 1792000 | 2.2519          |
| 2.3903        | 37.89 | 1800000 | 2.2594          |
| 2.3822        | 38.06 | 1808000 | 2.2565          |
| 2.3822        | 38.23 | 1816000 | 2.2812          |
| 2.383         | 38.4  | 1824000 | 2.2589          |
| 2.383         | 38.57 | 1832000 | 2.2560          |
| 2.3868        | 38.74 | 1840000 | 2.2648          |
| 2.3868        | 38.9  | 1848000 | 2.2507          |
| 2.3775        | 39.07 | 1856000 | 2.2570          |
| 2.3775        | 39.24 | 1864000 | 2.2549          |
| 2.3818        | 39.41 | 1872000 | 2.2583          |
| 2.3818        | 39.58 | 1880000 | 2.2610          |
| 2.3887        | 39.75 | 1888000 | 2.2629          |
| 2.3887        | 39.91 | 1896000 | 2.2739          |
| 2.3893        | 40.08 | 1904000 | 2.2657          |
| 2.3893        | 40.25 | 1912000 | 2.2507          |
| 2.3826        | 40.42 | 1920000 | 2.2506          |
| 2.3826        | 40.59 | 1928000 | 2.2630          |
| 2.3842        | 40.76 | 1936000 | 2.2716          |
| 2.3842        | 40.93 | 1944000 | 2.2642          |
| 2.3866        | 41.09 | 1952000 | 2.2451          |
| 2.3866        | 41.26 | 1960000 | 2.2521          |
| 2.3857        | 41.43 | 1968000 | 2.2457          |
| 2.3857        | 41.6  | 1976000 | 2.2575          |
| 2.3943        | 41.77 | 1984000 | 2.2659          |
| 2.3943        | 41.94 | 1992000 | 2.2608          |
| 2.387         | 42.1  | 2000000 | 2.2687          |
| 2.387         | 42.27 | 2008000 | 2.2718          |
| 2.387         | 42.44 | 2016000 | 2.2629          |
| 2.387         | 42.61 | 2024000 | 2.2283          |
| 2.3804        | 42.78 | 2032000 | 2.2422          |
| 2.3804        | 42.95 | 2040000 | 2.2431          |
| 2.3842        | 43.11 | 2048000 | 2.2689          |
| 2.3842        | 43.28 | 2056000 | 2.2586          |
| 2.3856        | 43.45 | 2064000 | 2.2590          |
| 2.3856        | 43.62 | 2072000 | 2.2602          |
| 2.3843        | 43.79 | 2080000 | 2.2557          |
| 2.3843        | 43.96 | 2088000 | 2.2776          |
| 2.3891        | 44.13 | 2096000 | 2.2554          |
| 2.3891        | 44.29 | 2104000 | 2.2615          |
| 2.3811        | 44.46 | 2112000 | 2.2591          |
| 2.3811        | 44.63 | 2120000 | 2.2600          |
| 2.3874        | 44.8  | 2128000 | 2.2595          |
| 2.3874        | 44.97 | 2136000 | 2.2762          |
| 2.3822        | 45.14 | 2144000 | 2.2516          |
| 2.3822        | 45.3  | 2152000 | 2.2530          |
| 2.3933        | 45.47 | 2160000 | 2.2652          |
| 2.3933        | 45.64 | 2168000 | 2.2480          |
| 2.3853        | 45.81 | 2176000 | 2.2717          |
| 2.3853        | 45.98 | 2184000 | 2.2569          |
| 2.3917        | 46.15 | 2192000 | 2.2564          |
| 2.3917        | 46.31 | 2200000 | 2.2512          |
| 2.3859        | 46.48 | 2208000 | 2.2612          |
| 2.3859        | 46.65 | 2216000 | 2.2609          |
| 2.3879        | 46.82 | 2224000 | 2.2552          |
| 2.3879        | 46.99 | 2232000 | 2.2568          |
| 2.3823        | 47.16 | 2240000 | 2.2507          |
| 2.3823        | 47.33 | 2248000 | 2.2762          |
| 2.388         | 47.49 | 2256000 | 2.2522          |
| 2.388         | 47.66 | 2264000 | 2.2532          |
| 2.3773        | 47.83 | 2272000 | 2.2490          |
| 2.3773        | 48.0  | 2280000 | 2.2648          |
| 2.3828        | 48.17 | 2288000 | 2.2500          |
| 2.3828        | 48.34 | 2296000 | 2.2534          |
| 2.3816        | 48.5  | 2304000 | 2.2515          |
| 2.3816        | 48.67 | 2312000 | 2.2702          |
| 2.3784        | 48.84 | 2320000 | 2.2584          |
| 2.3784        | 49.01 | 2328000 | 2.2382          |
| 2.3863        | 49.18 | 2336000 | 2.2604          |
| 2.3863        | 49.35 | 2344000 | 2.2607          |
| 2.3863        | 49.51 | 2352000 | 2.2646          |
| 2.3863        | 49.68 | 2360000 | 2.2534          |
| 2.3873        | 49.85 | 2368000 | 2.2742          |
| 2.3873        | 50.02 | 2376000 | 2.2687          |
| 2.39          | 50.19 | 2384000 | 2.2581          |
| 2.39          | 50.36 | 2392000 | 2.2460          |
| 2.3937        | 50.53 | 2400000 | 2.2642          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
