transformers tensorboard safetensors mt5 text2text-generation generated_from_trainer base_model:google/mt5-small license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
license: apache-2.0
base_model: google/mt5-small
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: mt5-summarize-tr
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# mt5-summarize-tr

This model is a fine-tuned version of [google/mt5-small](https://huggingface.co/google/mt5-small) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 2.0312
- Rouge1: 0.3697
- Rouge2: 0.2183
- Rougel: 0.3248
- Rougelsum: 0.3253

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 2
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 16
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 90
- num_epochs: 10

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|
| 4.125         | 0.12  | 100  | 3.0694          | 0.2411 | 0.1271 | 0.2143 | 0.2131    |
| 3.5224        | 0.24  | 200  | 2.7433          | 0.3152 | 0.1669 | 0.2738 | 0.2743    |
| 3.2845        | 0.35  | 300  | 2.5655          | 0.3254 | 0.1738 | 0.2870 | 0.2863    |
| 3.1946        | 0.47  | 400  | 2.5012          | 0.3080 | 0.1659 | 0.2795 | 0.2798    |
| 3.2075        | 0.59  | 500  | 2.4685          | 0.3488 | 0.2036 | 0.3117 | 0.3120    |
| 3.0507        | 0.71  | 600  | 2.4249          | 0.3337 | 0.1947 | 0.2900 | 0.2898    |
| 3.0301        | 0.82  | 700  | 2.4027          | 0.3396 | 0.1949 | 0.2973 | 0.2987    |
| 3.032         | 0.94  | 800  | 2.3871          | 0.3165 | 0.1789 | 0.2777 | 0.2769    |
| 2.8716        | 1.06  | 900  | 2.3811          | 0.3446 | 0.2113 | 0.3104 | 0.3093    |
| 2.8563        | 1.18  | 1000 | 2.3448          | 0.3603 | 0.2248 | 0.3251 | 0.3247    |
| 2.8408        | 1.3   | 1100 | 2.3438          | 0.3387 | 0.1945 | 0.2960 | 0.2963    |
| 2.7755        | 1.41  | 1200 | 2.3275          | 0.3473 | 0.2038 | 0.2910 | 0.2920    |
| 2.8019        | 1.53  | 1300 | 2.2786          | 0.3438 | 0.2021 | 0.3029 | 0.3022    |
| 2.7457        | 1.65  | 1400 | 2.2380          | 0.3308 | 0.1887 | 0.2985 | 0.2985    |
| 2.7216        | 1.77  | 1500 | 2.2517          | 0.3515 | 0.2112 | 0.3079 | 0.3052    |
| 2.71          | 1.88  | 1600 | 2.2712          | 0.3397 | 0.1978 | 0.2955 | 0.2932    |
| 2.6489        | 2.0   | 1700 | 2.2678          | 0.3441 | 0.2059 | 0.3047 | 0.3049    |
| 2.6232        | 2.12  | 1800 | 2.2526          | 0.3242 | 0.1901 | 0.2874 | 0.2864    |
| 2.5419        | 2.24  | 1900 | 2.2526          | 0.2985 | 0.1636 | 0.2611 | 0.2610    |
| 2.6232        | 2.36  | 2000 | 2.2108          | 0.3454 | 0.2053 | 0.2995 | 0.2999    |
| 2.5973        | 2.47  | 2100 | 2.2166          | 0.3363 | 0.1986 | 0.2975 | 0.2968    |
| 2.5738        | 2.59  | 2200 | 2.1973          | 0.3263 | 0.1802 | 0.2803 | 0.2812    |
| 2.5279        | 2.71  | 2300 | 2.1978          | 0.3186 | 0.1778 | 0.2829 | 0.2857    |
| 2.5613        | 2.83  | 2400 | 2.1921          | 0.3539 | 0.2126 | 0.3125 | 0.3166    |
| 2.5856        | 2.94  | 2500 | 2.1830          | 0.3719 | 0.2293 | 0.3295 | 0.3307    |
| 2.5181        | 3.06  | 2600 | 2.1531          | 0.3581 | 0.2177 | 0.3245 | 0.3239    |
| 2.4885        | 3.18  | 2700 | 2.1702          | 0.3409 | 0.1980 | 0.3034 | 0.3051    |
| 2.4172        | 3.3   | 2800 | 2.1562          | 0.3387 | 0.1872 | 0.2927 | 0.2925    |
| 2.5102        | 3.41  | 2900 | 2.1398          | 0.3387 | 0.2004 | 0.2947 | 0.2944    |
| 2.5086        | 3.53  | 3000 | 2.1253          | 0.3453 | 0.1935 | 0.3031 | 0.3029    |
| 2.4235        | 3.65  | 3100 | 2.1530          | 0.3495 | 0.1990 | 0.2932 | 0.2936    |
| 2.364         | 3.77  | 3200 | 2.1215          | 0.3564 | 0.2057 | 0.3049 | 0.3072    |
| 2.4752        | 3.89  | 3300 | 2.1223          | 0.3250 | 0.1818 | 0.2764 | 0.2772    |
| 2.3531        | 4.0   | 3400 | 2.1100          | 0.3453 | 0.1945 | 0.2931 | 0.2943    |
| 2.3746        | 4.12  | 3500 | 2.1162          | 0.3494 | 0.1976 | 0.2991 | 0.2998    |
| 2.3519        | 4.24  | 3600 | 2.1069          | 0.3496 | 0.2118 | 0.3061 | 0.3052    |
| 2.3154        | 4.36  | 3700 | 2.1217          | 0.3544 | 0.2170 | 0.3257 | 0.3269    |
| 2.3163        | 4.47  | 3800 | 2.1092          | 0.3445 | 0.2007 | 0.3044 | 0.3045    |
| 2.3641        | 4.59  | 3900 | 2.1098          | 0.3484 | 0.2072 | 0.3127 | 0.3126    |
| 2.298         | 4.71  | 4000 | 2.1019          | 0.3571 | 0.2190 | 0.3211 | 0.3224    |
| 2.3414        | 4.83  | 4100 | 2.0955          | 0.3534 | 0.2002 | 0.2949 | 0.2957    |
| 2.3812        | 4.95  | 4200 | 2.1027          | 0.3531 | 0.2058 | 0.3009 | 0.3011    |
| 2.2614        | 5.06  | 4300 | 2.1107          | 0.3379 | 0.1902 | 0.2891 | 0.2886    |
| 2.2528        | 5.18  | 4400 | 2.0953          | 0.3789 | 0.2229 | 0.3263 | 0.3243    |
| 2.3287        | 5.3   | 4500 | 2.0977          | 0.3671 | 0.2120 | 0.3200 | 0.3180    |
| 2.2855        | 5.42  | 4600 | 2.1044          | 0.3567 | 0.2085 | 0.3030 | 0.3057    |
| 2.2969        | 5.53  | 4700 | 2.0925          | 0.3459 | 0.2103 | 0.3126 | 0.3119    |
| 2.2438        | 5.65  | 4800 | 2.0879          | 0.3577 | 0.2020 | 0.2999 | 0.2996    |
| 2.2088        | 5.77  | 4900 | 2.0862          | 0.3662 | 0.2212 | 0.3222 | 0.3259    |
| 2.2319        | 5.89  | 5000 | 2.0839          | 0.3530 | 0.2040 | 0.3108 | 0.3130    |
| 2.2879        | 6.01  | 5100 | 2.0761          | 0.3693 | 0.2236 | 0.3285 | 0.3288    |
| 2.2508        | 6.12  | 5200 | 2.0810          | 0.3924 | 0.2424 | 0.3440 | 0.3448    |
| 2.2634        | 6.24  | 5300 | 2.0760          | 0.3722 | 0.2170 | 0.3162 | 0.3165    |
| 2.1484        | 6.36  | 5400 | 2.0725          | 0.3624 | 0.2110 | 0.3097 | 0.3115    |
| 2.1863        | 6.48  | 5500 | 2.0663          | 0.3820 | 0.2332 | 0.3338 | 0.3348    |
| 2.1935        | 6.59  | 5600 | 2.0762          | 0.3906 | 0.2299 | 0.3421 | 0.3425    |
| 2.224         | 6.71  | 5700 | 2.0536          | 0.3788 | 0.2362 | 0.3337 | 0.3336    |
| 2.1938        | 6.83  | 5800 | 2.0663          | 0.3551 | 0.2017 | 0.3047 | 0.3044    |
| 2.2589        | 6.95  | 5900 | 2.0602          | 0.3782 | 0.2298 | 0.3365 | 0.3360    |
| 2.1185        | 7.07  | 6000 | 2.0632          | 0.3617 | 0.2170 | 0.3237 | 0.3239    |
| 2.1924        | 7.18  | 6100 | 2.0508          | 0.3771 | 0.2211 | 0.3284 | 0.3295    |
| 2.2012        | 7.3   | 6200 | 2.0602          | 0.3714 | 0.2184 | 0.3315 | 0.3315    |
| 2.1617        | 7.42  | 6300 | 2.0606          | 0.3311 | 0.1800 | 0.2883 | 0.2862    |
| 2.1557        | 7.54  | 6400 | 2.0444          | 0.3827 | 0.2318 | 0.3386 | 0.3383    |
| 2.1965        | 7.65  | 6500 | 2.0391          | 0.3777 | 0.2334 | 0.3401 | 0.3414    |
| 2.1599        | 7.77  | 6600 | 2.0393          | 0.3648 | 0.2121 | 0.3154 | 0.3152    |
| 2.1505        | 7.89  | 6700 | 2.0483          | 0.3905 | 0.2411 | 0.3485 | 0.3496    |
| 2.191         | 8.01  | 6800 | 2.0412          | 0.3815 | 0.2373 | 0.3327 | 0.3337    |
| 2.1513        | 8.12  | 6900 | 2.0461          | 0.3745 | 0.2252 | 0.3363 | 0.3374    |
| 2.1232        | 8.24  | 7000 | 2.0474          | 0.3672 | 0.2183 | 0.3264 | 0.3270    |
| 2.1249        | 8.36  | 7100 | 2.0422          | 0.3846 | 0.2270 | 0.3379 | 0.3388    |
| 2.1743        | 8.48  | 7200 | 2.0358          | 0.3707 | 0.2086 | 0.3192 | 0.3184    |
| 2.1684        | 8.6   | 7300 | 2.0391          | 0.3532 | 0.2061 | 0.3048 | 0.3035    |
| 2.1257        | 8.71  | 7400 | 2.0388          | 0.3773 | 0.2198 | 0.3218 | 0.3219    |
| 2.1186        | 8.83  | 7500 | 2.0356          | 0.3655 | 0.2135 | 0.3091 | 0.3077    |
| 2.054         | 8.95  | 7600 | 2.0308          | 0.3739 | 0.2276 | 0.3208 | 0.3208    |
| 2.1534        | 9.07  | 7700 | 2.0253          | 0.3649 | 0.2150 | 0.3154 | 0.3157    |
| 2.0906        | 9.18  | 7800 | 2.0327          | 0.3752 | 0.2233 | 0.3173 | 0.3174    |
| 2.0703        | 9.3   | 7900 | 2.0314          | 0.3610 | 0.2100 | 0.3153 | 0.3150    |
| 2.1088        | 9.42  | 8000 | 2.0289          | 0.3636 | 0.2106 | 0.3171 | 0.3174    |
| 2.0719        | 9.54  | 8100 | 2.0298          | 0.3682 | 0.2145 | 0.3225 | 0.3223    |
| 2.0873        | 9.66  | 8200 | 2.0294          | 0.3729 | 0.2185 | 0.3225 | 0.3228    |
| 2.0953        | 9.77  | 8300 | 2.0316          | 0.3729 | 0.2185 | 0.3225 | 0.3228    |
| 2.0825        | 9.89  | 8400 | 2.0312          | 0.3697 | 0.2183 | 0.3248 | 0.3253    |


### Framework versions

- Transformers 4.35.2
- Pytorch 2.1.0+cu121
- Datasets 2.16.0
- Tokenizers 0.15.0
