transformers pytorch roberta fill-mask generated_from_trainer base_model:cardiffnlp/twitter-roberta-base-2019-90m license:mit autotrain_compatible endpoints_compatible region:us
---
license: mit
base_model: cardiffnlp/twitter-roberta-base-2019-90m
tags:
- generated_from_trainer
model-index:
- name: 2020-Q3-75p-filtered
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# 2020-Q3-75p-filtered

This model is a fine-tuned version of [cardiffnlp/twitter-roberta-base-2019-90m](https://huggingface.co/cardiffnlp/twitter-roberta-base-2019-90m) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.1267

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 4.1e-07
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-08
- lr_scheduler_type: linear
- training_steps: 2400000

### Training results

| Training Loss | Epoch | Step    | Validation Loss |
|:-------------:|:-----:|:-------:|:---------------:|
| No log        | 0.07  | 8000    | 3.4513          |
| 3.6645        | 0.14  | 16000   | 3.3790          |
| 3.6645        | 0.2   | 24000   | 3.3226          |
| 3.4678        | 0.27  | 32000   | 3.2901          |
| 3.4678        | 0.34  | 40000   | 3.2590          |
| 3.4118        | 0.41  | 48000   | 3.2534          |
| 3.4118        | 0.47  | 56000   | 3.2357          |
| 3.3843        | 0.54  | 64000   | 3.2228          |
| 3.3843        | 0.61  | 72000   | 3.2331          |
| 3.3633        | 0.68  | 80000   | 3.2047          |
| 3.3633        | 0.74  | 88000   | 3.2138          |
| 3.3474        | 0.81  | 96000   | 3.2050          |
| 3.3474        | 0.88  | 104000  | 3.2051          |
| 3.3414        | 0.95  | 112000  | 3.1930          |
| 3.3414        | 1.01  | 120000  | 3.2002          |
| 3.335         | 1.08  | 128000  | 3.1920          |
| 3.335         | 1.15  | 136000  | 3.1914          |
| 3.3283        | 1.22  | 144000  | 3.1853          |
| 3.3283        | 1.28  | 152000  | 3.1825          |
| 3.3276        | 1.35  | 160000  | 3.1827          |
| 3.3276        | 1.42  | 168000  | 3.1756          |
| 3.323         | 1.49  | 176000  | 3.1865          |
| 3.323         | 1.56  | 184000  | 3.1749          |
| 3.3275        | 1.62  | 192000  | 3.1782          |
| 3.3275        | 1.69  | 200000  | 3.1676          |
| 3.3309        | 1.76  | 208000  | 3.1832          |
| 3.3309        | 1.83  | 216000  | 3.1744          |
| 3.3166        | 1.89  | 224000  | 3.1645          |
| 3.3166        | 1.96  | 232000  | 3.1770          |
| 3.3206        | 2.03  | 240000  | 3.1656          |
| 3.3206        | 2.1   | 248000  | 3.1561          |
| 3.3228        | 2.16  | 256000  | 3.1665          |
| 3.3228        | 2.23  | 264000  | 3.1657          |
| 3.3208        | 2.3   | 272000  | 3.1693          |
| 3.3208        | 2.37  | 280000  | 3.1778          |
| 3.3106        | 2.43  | 288000  | 3.1760          |
| 3.3106        | 2.5   | 296000  | 3.1664          |
| 3.3189        | 2.57  | 304000  | 3.1677          |
| 3.3189        | 2.64  | 312000  | 3.1600          |
| 3.319         | 2.7   | 320000  | 3.1570          |
| 3.319         | 2.77  | 328000  | 3.1699          |
| 3.3236        | 2.84  | 336000  | 3.1578          |
| 3.3236        | 2.91  | 344000  | 3.1665          |
| 3.3205        | 2.98  | 352000  | 3.1558          |
| 3.3205        | 3.04  | 360000  | 3.1678          |
| 3.3114        | 3.11  | 368000  | 3.1597          |
| 3.3114        | 3.18  | 376000  | 3.1618          |
| 3.3067        | 3.25  | 384000  | 3.1584          |
| 3.3067        | 3.31  | 392000  | 3.1597          |
| 3.314         | 3.38  | 400000  | 3.1565          |
| 3.314         | 3.45  | 408000  | 3.1612          |
| 3.3183        | 3.52  | 416000  | 3.1637          |
| 3.3183        | 3.58  | 424000  | 3.1569          |
| 3.318         | 3.65  | 432000  | 3.1576          |
| 3.318         | 3.72  | 440000  | 3.1639          |
| 3.3114        | 3.79  | 448000  | 3.1460          |
| 3.3114        | 3.85  | 456000  | 3.1611          |
| 3.3068        | 3.92  | 464000  | 3.1587          |
| 3.3068        | 3.99  | 472000  | 3.1542          |
| 3.3166        | 4.06  | 480000  | 3.1422          |
| 3.3166        | 4.12  | 488000  | 3.1604          |
| 3.3057        | 4.19  | 496000  | 3.1587          |
| 3.3057        | 4.26  | 504000  | 3.1576          |
| 3.3176        | 4.33  | 512000  | 3.1602          |
| 3.3176        | 4.4   | 520000  | 3.1544          |
| 3.3126        | 4.46  | 528000  | 3.1478          |
| 3.3126        | 4.53  | 536000  | 3.1520          |
| 3.3044        | 4.6   | 544000  | 3.1581          |
| 3.3044        | 4.67  | 552000  | 3.1625          |
| 3.3118        | 4.73  | 560000  | 3.1510          |
| 3.3118        | 4.8   | 568000  | 3.1548          |
| 3.3085        | 4.87  | 576000  | 3.1539          |
| 3.3085        | 4.94  | 584000  | 3.1503          |
| 3.3014        | 5.0   | 592000  | 3.1504          |
| 3.3014        | 5.07  | 600000  | 3.1534          |
| 3.3115        | 5.14  | 608000  | 3.1551          |
| 3.3115        | 5.21  | 616000  | 3.1493          |
| 3.3079        | 5.27  | 624000  | 3.1427          |
| 3.3079        | 5.34  | 632000  | 3.1500          |
| 3.3138        | 5.41  | 640000  | 3.1546          |
| 3.3138        | 5.48  | 648000  | 3.1482          |
| 3.3096        | 5.54  | 656000  | 3.1346          |
| 3.3096        | 5.61  | 664000  | 3.1328          |
| 3.3121        | 5.68  | 672000  | 3.1500          |
| 3.3121        | 5.75  | 680000  | 3.1312          |
| 3.3195        | 5.82  | 688000  | 3.1440          |
| 3.3195        | 5.88  | 696000  | 3.1191          |
| 3.3091        | 5.95  | 704000  | 3.1397          |
| 3.3091        | 6.02  | 712000  | 3.1485          |
| 3.3089        | 6.09  | 720000  | 3.1340          |
| 3.3089        | 6.15  | 728000  | 3.1385          |
| 3.3062        | 6.22  | 736000  | 3.1358          |
| 3.3062        | 6.29  | 744000  | 3.1296          |
| 3.3102        | 6.36  | 752000  | 3.1260          |
| 3.3102        | 6.42  | 760000  | 3.1428          |
| 3.3088        | 6.49  | 768000  | 3.1372          |
| 3.3088        | 6.56  | 776000  | 3.1404          |
| 3.3096        | 6.63  | 784000  | 3.1362          |
| 3.3096        | 6.69  | 792000  | 3.1408          |
| 3.3079        | 6.76  | 800000  | 3.1350          |
| 3.3079        | 6.83  | 808000  | 3.1461          |
| 3.3099        | 6.9   | 816000  | 3.1420          |
| 3.3099        | 6.96  | 824000  | 3.1213          |
| 3.3015        | 7.03  | 832000  | 3.1366          |
| 3.3015        | 7.1   | 840000  | 3.1401          |
| 3.3045        | 7.17  | 848000  | 3.1295          |
| 3.3045        | 7.24  | 856000  | 3.1323          |
| 3.3085        | 7.3   | 864000  | 3.1368          |
| 3.3085        | 7.37  | 872000  | 3.1275          |
| 3.3061        | 7.44  | 880000  | 3.1326          |
| 3.3061        | 7.51  | 888000  | 3.1377          |
| 3.309         | 7.57  | 896000  | 3.1407          |
| 3.309         | 7.64  | 904000  | 3.1324          |
| 3.3024        | 7.71  | 912000  | 3.1187          |
| 3.3024        | 7.78  | 920000  | 3.1514          |
| 3.2955        | 7.84  | 928000  | 3.1351          |
| 3.2955        | 7.91  | 936000  | 3.1308          |
| 3.3122        | 7.98  | 944000  | 3.1405          |
| 3.3122        | 8.05  | 952000  | 3.1291          |
| 3.304         | 8.11  | 960000  | 3.1244          |
| 3.304         | 8.18  | 968000  | 3.1409          |
| 3.3046        | 8.25  | 976000  | 3.1355          |
| 3.3046        | 8.32  | 984000  | 3.1416          |
| 3.3022        | 8.38  | 992000  | 3.1258          |
| 3.3022        | 8.45  | 1000000 | 3.1332          |
| 3.3004        | 8.52  | 1008000 | 3.1430          |
| 3.3004        | 8.59  | 1016000 | 3.1282          |
| 3.3045        | 8.66  | 1024000 | 3.1287          |
| 3.3045        | 8.72  | 1032000 | 3.1368          |
| 3.3047        | 8.79  | 1040000 | 3.1362          |
| 3.3047        | 8.86  | 1048000 | 3.1268          |
| 3.3044        | 8.93  | 1056000 | 3.1329          |
| 3.3044        | 8.99  | 1064000 | 3.1245          |
| 3.2961        | 9.06  | 1072000 | 3.1271          |
| 3.2961        | 9.13  | 1080000 | 3.1300          |
| 3.2999        | 9.2   | 1088000 | 3.1369          |
| 3.2999        | 9.26  | 1096000 | 3.1425          |
| 3.3012        | 9.33  | 1104000 | 3.1213          |
| 3.3012        | 9.4   | 1112000 | 3.1285          |
| 3.3008        | 9.47  | 1120000 | 3.1353          |
| 3.3008        | 9.53  | 1128000 | 3.1367          |
| 3.3028        | 9.6   | 1136000 | 3.1294          |
| 3.3028        | 9.67  | 1144000 | 3.1340          |
| 3.3043        | 9.74  | 1152000 | 3.1330          |
| 3.3043        | 9.8   | 1160000 | 3.1380          |
| 3.2976        | 9.87  | 1168000 | 3.1198          |
| 3.2976        | 9.94  | 1176000 | 3.1290          |
| 3.3048        | 10.01 | 1184000 | 3.1458          |
| 3.3048        | 10.08 | 1192000 | 3.1274          |
| 3.3038        | 10.14 | 1200000 | 3.1181          |
| 3.3038        | 10.21 | 1208000 | 3.1279          |
| 3.3066        | 10.28 | 1216000 | 3.1230          |
| 3.3066        | 10.35 | 1224000 | 3.1320          |
| 3.3019        | 10.41 | 1232000 | 3.1203          |
| 3.3019        | 10.48 | 1240000 | 3.1349          |
| 3.3037        | 10.55 | 1248000 | 3.1323          |
| 3.3037        | 10.62 | 1256000 | 3.1343          |
| 3.2868        | 10.68 | 1264000 | 3.1262          |
| 3.2868        | 10.75 | 1272000 | 3.1266          |
| 3.3033        | 10.82 | 1280000 | 3.1283          |
| 3.3033        | 10.89 | 1288000 | 3.1290          |
| 3.2984        | 10.95 | 1296000 | 3.1177          |
| 3.2984        | 11.02 | 1304000 | 3.1234          |
| 3.2982        | 11.09 | 1312000 | 3.1310          |
| 3.2982        | 11.16 | 1320000 | 3.1409          |
| 3.303         | 11.23 | 1328000 | 3.1330          |
| 3.303         | 11.29 | 1336000 | 3.1281          |
| 3.2976        | 11.36 | 1344000 | 3.1286          |
| 3.2976        | 11.43 | 1352000 | 3.1283          |
| 3.2923        | 11.5  | 1360000 | 3.1146          |
| 3.2923        | 11.56 | 1368000 | 3.1387          |
| 3.2988        | 11.63 | 1376000 | 3.1278          |
| 3.2988        | 11.7  | 1384000 | 3.1225          |
| 3.299         | 11.77 | 1392000 | 3.1341          |
| 3.299         | 11.83 | 1400000 | 3.1211          |
| 3.2993        | 11.9  | 1408000 | 3.1026          |
| 3.2993        | 11.97 | 1416000 | 3.1223          |
| 3.2942        | 12.04 | 1424000 | 3.1200          |
| 3.2942        | 12.1  | 1432000 | 3.1246          |
| 3.3062        | 12.17 | 1440000 | 3.1325          |
| 3.3062        | 12.24 | 1448000 | 3.1388          |
| 3.297         | 12.31 | 1456000 | 3.1371          |
| 3.297         | 12.37 | 1464000 | 3.1272          |
| 3.3033        | 12.44 | 1472000 | 3.1231          |
| 3.3033        | 12.51 | 1480000 | 3.1316          |
| 3.291         | 12.58 | 1488000 | 3.1393          |
| 3.291         | 12.65 | 1496000 | 3.1269          |
| 3.3054        | 12.71 | 1504000 | 3.1363          |
| 3.3054        | 12.78 | 1512000 | 3.1249          |
| 3.2908        | 12.85 | 1520000 | 3.1310          |
| 3.2908        | 12.92 | 1528000 | 3.1213          |
| 3.2987        | 12.98 | 1536000 | 3.1223          |
| 3.2987        | 13.05 | 1544000 | 3.1134          |
| 3.2965        | 13.12 | 1552000 | 3.1168          |
| 3.2965        | 13.19 | 1560000 | 3.1230          |
| 3.2931        | 13.25 | 1568000 | 3.1132          |
| 3.2931        | 13.32 | 1576000 | 3.1196          |
| 3.301         | 13.39 | 1584000 | 3.1287          |
| 3.301         | 13.46 | 1592000 | 3.1145          |
| 3.3004        | 13.52 | 1600000 | 3.1291          |
| 3.3004        | 13.59 | 1608000 | 3.1145          |
| 3.2992        | 13.66 | 1616000 | 3.1292          |
| 3.2992        | 13.73 | 1624000 | 3.1248          |
| 3.2974        | 13.79 | 1632000 | 3.1315          |
| 3.2974        | 13.86 | 1640000 | 3.1112          |
| 3.2993        | 13.93 | 1648000 | 3.1217          |
| 3.2993        | 14.0  | 1656000 | 3.1362          |
| 3.2934        | 14.07 | 1664000 | 3.1199          |
| 3.2934        | 14.13 | 1672000 | 3.1276          |
| 3.2964        | 14.2  | 1680000 | 3.1164          |
| 3.2964        | 14.27 | 1688000 | 3.1172          |
| 3.305         | 14.34 | 1696000 | 3.1320          |
| 3.305         | 14.4  | 1704000 | 3.1269          |
| 3.3022        | 14.47 | 1712000 | 3.1107          |
| 3.3022        | 14.54 | 1720000 | 3.1097          |
| 3.2969        | 14.61 | 1728000 | 3.1176          |
| 3.2969        | 14.67 | 1736000 | 3.1282          |
| 3.2976        | 14.74 | 1744000 | 3.1195          |
| 3.2976        | 14.81 | 1752000 | 3.1154          |
| 3.3004        | 14.88 | 1760000 | 3.1147          |
| 3.3004        | 14.94 | 1768000 | 3.1094          |
| 3.2908        | 15.01 | 1776000 | 3.1313          |
| 3.2908        | 15.08 | 1784000 | 3.1280          |
| 3.2896        | 15.15 | 1792000 | 3.1304          |
| 3.2896        | 15.21 | 1800000 | 3.1329          |
| 3.3061        | 15.28 | 1808000 | 3.1198          |
| 3.3061        | 15.35 | 1816000 | 3.1258          |
| 3.3056        | 15.42 | 1824000 | 3.1253          |
| 3.3056        | 15.49 | 1832000 | 3.1200          |
| 3.2921        | 15.55 | 1840000 | 3.1384          |
| 3.2921        | 15.62 | 1848000 | 3.1225          |
| 3.2895        | 15.69 | 1856000 | 3.1284          |
| 3.2895        | 15.76 | 1864000 | 3.1201          |
| 3.293         | 15.82 | 1872000 | 3.1256          |
| 3.293         | 15.89 | 1880000 | 3.1166          |
| 3.2963        | 15.96 | 1888000 | 3.1218          |
| 3.2963        | 16.03 | 1896000 | 3.1193          |
| 3.2908        | 16.09 | 1904000 | 3.1204          |
| 3.2908        | 16.16 | 1912000 | 3.1325          |
| 3.3039        | 16.23 | 1920000 | 3.1091          |
| 3.3039        | 16.3  | 1928000 | 3.1250          |
| 3.3011        | 16.36 | 1936000 | 3.1217          |
| 3.3011        | 16.43 | 1944000 | 3.1208          |
| 3.3003        | 16.5  | 1952000 | 3.1109          |
| 3.3003        | 16.57 | 1960000 | 3.1252          |
| 3.3012        | 16.63 | 1968000 | 3.1123          |
| 3.3012        | 16.7  | 1976000 | 3.1213          |
| 3.2885        | 16.77 | 1984000 | 3.1219          |
| 3.2885        | 16.84 | 1992000 | 3.1254          |
| 3.2982        | 16.91 | 2000000 | 3.1260          |
| 3.2982        | 16.97 | 2008000 | 3.1167          |
| 3.2962        | 17.04 | 2016000 | 3.1082          |
| 3.2962        | 17.11 | 2024000 | 3.1204          |
| 3.2889        | 17.18 | 2032000 | 3.1236          |
| 3.2889        | 17.24 | 2040000 | 3.1325          |
| 3.2892        | 17.31 | 2048000 | 3.1200          |
| 3.2892        | 17.38 | 2056000 | 3.1231          |
| 3.3028        | 17.45 | 2064000 | 3.1202          |
| 3.3028        | 17.51 | 2072000 | 3.1189          |
| 3.2889        | 17.58 | 2080000 | 3.1337          |
| 3.2889        | 17.65 | 2088000 | 3.1156          |
| 3.2985        | 17.72 | 2096000 | 3.1258          |
| 3.2985        | 17.78 | 2104000 | 3.1358          |
| 3.2949        | 17.85 | 2112000 | 3.1271          |
| 3.2949        | 17.92 | 2120000 | 3.1250          |
| 3.2987        | 17.99 | 2128000 | 3.1244          |
| 3.2987        | 18.05 | 2136000 | 3.1221          |
| 3.2884        | 18.12 | 2144000 | 3.1198          |
| 3.2884        | 18.19 | 2152000 | 3.1170          |
| 3.2918        | 18.26 | 2160000 | 3.1159          |
| 3.2918        | 18.33 | 2168000 | 3.1153          |
| 3.2995        | 18.39 | 2176000 | 3.1203          |
| 3.2995        | 18.46 | 2184000 | 3.1107          |
| 3.3003        | 18.53 | 2192000 | 3.1212          |
| 3.3003        | 18.6  | 2200000 | 3.1330          |
| 3.2921        | 18.66 | 2208000 | 3.1160          |
| 3.2921        | 18.73 | 2216000 | 3.1192          |
| 3.293         | 18.8  | 2224000 | 3.1164          |
| 3.293         | 18.87 | 2232000 | 3.1225          |
| 3.2969        | 18.93 | 2240000 | 3.1243          |
| 3.2969        | 19.0  | 2248000 | 3.1152          |
| 3.2891        | 19.07 | 2256000 | 3.1323          |
| 3.2891        | 19.14 | 2264000 | 3.1077          |
| 3.2903        | 19.2  | 2272000 | 3.1348          |
| 3.2903        | 19.27 | 2280000 | 3.1202          |
| 3.2986        | 19.34 | 2288000 | 3.1220          |
| 3.2986        | 19.41 | 2296000 | 3.1236          |
| 3.293         | 19.47 | 2304000 | 3.1224          |
| 3.293         | 19.54 | 2312000 | 3.1247          |
| 3.299         | 19.61 | 2320000 | 3.1235          |
| 3.299         | 19.68 | 2328000 | 3.1201          |
| 3.2898        | 19.75 | 2336000 | 3.1163          |
| 3.2898        | 19.81 | 2344000 | 3.1289          |
| 3.2956        | 19.88 | 2352000 | 3.1198          |
| 3.2956        | 19.95 | 2360000 | 3.1251          |
| 3.2926        | 20.02 | 2368000 | 3.1087          |
| 3.2926        | 20.08 | 2376000 | 3.1097          |
| 3.2958        | 20.15 | 2384000 | 3.1262          |
| 3.2958        | 20.22 | 2392000 | 3.1308          |
| 3.2862        | 20.29 | 2400000 | 3.1129          |


### Framework versions

- Transformers 4.35.0.dev0
- Pytorch 2.0.1+cu117
- Datasets 2.14.5
- Tokenizers 0.14.0
