transformers tensorboard safetensors bart text2text-generation generated_from_trainer base_model:GanjinZero/biobart-base license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
license: apache-2.0
base_model: GanjinZero/biobart-base
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: fine-tuned-BioBART-50-epochs-1024-input-128-output
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# fine-tuned-BioBART-50-epochs-1024-input-128-output

This model is a fine-tuned version of [GanjinZero/biobart-base](https://huggingface.co/GanjinZero/biobart-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.9109
- Rouge1: 0.1191
- Rouge2: 0.0252
- Rougel: 0.105
- Rougelsum: 0.1059
- Gen Len: 16.2

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 50

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
| No log        | 1.0   | 151  | 8.7986          | 0.0    | 0.0    | 0.0    | 0.0       | 14.54   |
| No log        | 2.0   | 302  | 4.6009          | 0.007  | 0.0022 | 0.0066 | 0.0067    | 4.73    |
| No log        | 3.0   | 453  | 1.9851          | 0.1025 | 0.0246 | 0.091  | 0.0906    | 13.95   |
| 6.1578        | 4.0   | 604  | 1.7001          | 0.0763 | 0.0172 | 0.0666 | 0.0674    | 10.25   |
| 6.1578        | 5.0   | 755  | 1.6023          | 0.1303 | 0.0277 | 0.1167 | 0.1164    | 15.08   |
| 6.1578        | 6.0   | 906  | 1.5322          | 0.0795 | 0.0176 | 0.0732 | 0.0736    | 14.54   |
| 1.4113        | 7.0   | 1057 | 1.4998          | 0.0972 | 0.0241 | 0.0839 | 0.0838    | 13.47   |
| 1.4113        | 8.0   | 1208 | 1.4808          | 0.0992 | 0.0238 | 0.0894 | 0.0898    | 14.28   |
| 1.4113        | 9.0   | 1359 | 1.4964          | 0.1249 | 0.0214 | 0.111  | 0.1106    | 12.36   |
| 0.8834        | 10.0  | 1510 | 1.4858          | 0.1459 | 0.0363 | 0.1235 | 0.1237    | 15.63   |
| 0.8834        | 11.0  | 1661 | 1.4990          | 0.1578 | 0.0403 | 0.1379 | 0.139     | 15.92   |
| 0.8834        | 12.0  | 1812 | 1.5210          | 0.1327 | 0.0253 | 0.1212 | 0.1209    | 15.11   |
| 0.8834        | 13.0  | 1963 | 1.5381          | 0.1372 | 0.038  | 0.1255 | 0.1251    | 15.45   |
| 0.5229        | 14.0  | 2114 | 1.5559          | 0.1383 | 0.0348 | 0.1263 | 0.1263    | 16.49   |
| 0.5229        | 15.0  | 2265 | 1.5824          | 0.1509 | 0.0369 | 0.1336 | 0.1325    | 15.78   |
| 0.5229        | 16.0  | 2416 | 1.6369          | 0.128  | 0.0298 | 0.1176 | 0.1185    | 14.12   |
| 0.2708        | 17.0  | 2567 | 1.6393          | 0.1362 | 0.0429 | 0.1229 | 0.1229    | 15.77   |
| 0.2708        | 18.0  | 2718 | 1.6599          | 0.1521 | 0.0402 | 0.1329 | 0.1333    | 15.34   |
| 0.2708        | 19.0  | 2869 | 1.6705          | 0.1293 | 0.0265 | 0.1165 | 0.1166    | 16.51   |
| 0.1203        | 20.0  | 3020 | 1.6943          | 0.141  | 0.0289 | 0.1273 | 0.1275    | 15.69   |
| 0.1203        | 21.0  | 3171 | 1.6969          | 0.1253 | 0.0337 | 0.1081 | 0.1085    | 16.35   |
| 0.1203        | 22.0  | 3322 | 1.7431          | 0.1319 | 0.0272 | 0.1185 | 0.1185    | 15.63   |
| 0.1203        | 23.0  | 3473 | 1.7434          | 0.1357 | 0.0343 | 0.1253 | 0.125     | 16.39   |
| 0.0509        | 24.0  | 3624 | 1.7507          | 0.1375 | 0.0325 | 0.1233 | 0.1231    | 16.79   |
| 0.0509        | 25.0  | 3775 | 1.7776          | 0.1222 | 0.0328 | 0.1121 | 0.1121    | 16.18   |
| 0.0509        | 26.0  | 3926 | 1.7733          | 0.1265 | 0.0216 | 0.1166 | 0.117     | 16.25   |
| 0.0257        | 27.0  | 4077 | 1.8001          | 0.1238 | 0.0239 | 0.1116 | 0.1113    | 16.44   |
| 0.0257        | 28.0  | 4228 | 1.7955          | 0.1173 | 0.0221 | 0.103  | 0.1046    | 16.64   |
| 0.0257        | 29.0  | 4379 | 1.8143          | 0.1311 | 0.0273 | 0.1186 | 0.1183    | 16.78   |
| 0.0164        | 30.0  | 4530 | 1.8108          | 0.1331 | 0.0296 | 0.1219 | 0.1226    | 15.64   |
| 0.0164        | 31.0  | 4681 | 1.8184          | 0.1245 | 0.0339 | 0.1134 | 0.1143    | 16.55   |
| 0.0164        | 32.0  | 4832 | 1.8545          | 0.1101 | 0.0217 | 0.0982 | 0.0998    | 16.09   |
| 0.0164        | 33.0  | 4983 | 1.8550          | 0.1421 | 0.0322 | 0.1292 | 0.1296    | 16.07   |
| 0.0117        | 34.0  | 5134 | 1.8573          | 0.1309 | 0.0292 | 0.1192 | 0.1193    | 16.0    |
| 0.0117        | 35.0  | 5285 | 1.8453          | 0.1254 | 0.0238 | 0.1133 | 0.1139    | 16.55   |
| 0.0117        | 36.0  | 5436 | 1.8724          | 0.1167 | 0.0241 | 0.1024 | 0.1035    | 15.89   |
| 0.0089        | 37.0  | 5587 | 1.8761          | 0.1345 | 0.0275 | 0.1206 | 0.1208    | 15.87   |
| 0.0089        | 38.0  | 5738 | 1.8772          | 0.1338 | 0.0301 | 0.1216 | 0.1228    | 16.78   |
| 0.0089        | 39.0  | 5889 | 1.8654          | 0.134  | 0.0264 | 0.1193 | 0.1196    | 16.85   |
| 0.0071        | 40.0  | 6040 | 1.8812          | 0.129  | 0.0287 | 0.1181 | 0.1177    | 16.12   |
| 0.0071        | 41.0  | 6191 | 1.8838          | 0.1238 | 0.0274 | 0.1134 | 0.1134    | 16.29   |
| 0.0071        | 42.0  | 6342 | 1.8752          | 0.1334 | 0.0262 | 0.1209 | 0.1214    | 16.66   |
| 0.0071        | 43.0  | 6493 | 1.8993          | 0.1238 | 0.0254 | 0.1111 | 0.1113    | 16.31   |
| 0.0056        | 44.0  | 6644 | 1.8963          | 0.1279 | 0.0346 | 0.1133 | 0.1154    | 16.07   |
| 0.0056        | 45.0  | 6795 | 1.9079          | 0.1225 | 0.0261 | 0.108  | 0.1084    | 16.09   |
| 0.0056        | 46.0  | 6946 | 1.9132          | 0.129  | 0.025  | 0.1157 | 0.1154    | 16.26   |
| 0.0045        | 47.0  | 7097 | 1.9120          | 0.1419 | 0.0362 | 0.1275 | 0.1278    | 15.78   |
| 0.0045        | 48.0  | 7248 | 1.9069          | 0.1316 | 0.0253 | 0.1161 | 0.1165    | 16.38   |
| 0.0045        | 49.0  | 7399 | 1.9099          | 0.1206 | 0.0259 | 0.1074 | 0.1077    | 16.32   |
| 0.0041        | 50.0  | 7550 | 1.9109          | 0.1191 | 0.0252 | 0.105  | 0.1059    | 16.2    |


### Framework versions

- Transformers 4.36.2
- Pytorch 1.12.1+cu113
- Datasets 2.16.1
- Tokenizers 0.15.0
