transformers tensorboard safetensors bart text2text-generation generated_from_trainer base_model:GanjinZero/biobart-base license:apache-2.0 autotrain_compatible endpoints_compatible region:us
---
license: apache-2.0
base_model: GanjinZero/biobart-base
tags:
- generated_from_trainer
metrics:
- rouge
model-index:
- name: fine-tuned-BioBART-50-epochs-1024-input-224-output
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# fine-tuned-BioBART-50-epochs-1024-input-224-output

This model is a fine-tuned version of [GanjinZero/biobart-base](https://huggingface.co/GanjinZero/biobart-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.2443
- Rouge1: 0.1837
- Rouge2: 0.028
- Rougel: 0.1489
- Rougelsum: 0.1474
- Gen Len: 42.57

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 50

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1 | Rouge2 | Rougel | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:------:|:------:|:------:|:---------:|:-------:|
| No log        | 1.0   | 151  | 8.7928          | 0.0036 | 0.0    | 0.0035 | 0.0036    | 14.33   |
| No log        | 2.0   | 302  | 3.8946          | 0.0    | 0.0    | 0.0    | 0.0       | 4.27    |
| No log        | 3.0   | 453  | 1.2664          | 0.129  | 0.0304 | 0.1174 | 0.1169    | 15.04   |
| 5.8095        | 4.0   | 604  | 1.1007          | 0.1115 | 0.0252 | 0.0884 | 0.0882    | 24.25   |
| 5.8095        | 5.0   | 755  | 1.0410          | 0.1446 | 0.0292 | 0.1129 | 0.1126    | 50.47   |
| 5.8095        | 6.0   | 906  | 0.9980          | 0.1189 | 0.029  | 0.0855 | 0.0856    | 36.31   |
| 0.976         | 7.0   | 1057 | 0.9752          | 0.1261 | 0.0328 | 0.094  | 0.0944    | 33.23   |
| 0.976         | 8.0   | 1208 | 0.9652          | 0.153  | 0.0347 | 0.1144 | 0.114     | 37.16   |
| 0.976         | 9.0   | 1359 | 0.9640          | 0.1473 | 0.0266 | 0.1214 | 0.1209    | 30.78   |
| 0.6318        | 10.0  | 1510 | 0.9684          | 0.1687 | 0.0364 | 0.1366 | 0.1355    | 33.88   |
| 0.6318        | 11.0  | 1661 | 0.9682          | 0.1856 | 0.0405 | 0.1448 | 0.143     | 32.34   |
| 0.6318        | 12.0  | 1812 | 0.9893          | 0.1693 | 0.0288 | 0.1293 | 0.1283    | 45.33   |
| 0.6318        | 13.0  | 1963 | 1.0004          | 0.1821 | 0.0464 | 0.1479 | 0.147     | 29.72   |
| 0.3943        | 14.0  | 2114 | 1.0180          | 0.1659 | 0.0233 | 0.1306 | 0.1289    | 30.25   |
| 0.3943        | 15.0  | 2265 | 1.0289          | 0.188  | 0.0402 | 0.1525 | 0.1519    | 36.03   |
| 0.3943        | 16.0  | 2416 | 1.0434          | 0.1928 | 0.05   | 0.158  | 0.1575    | 35.88   |
| 0.2174        | 17.0  | 2567 | 1.0522          | 0.1824 | 0.0453 | 0.1508 | 0.1491    | 35.06   |
| 0.2174        | 18.0  | 2718 | 1.0749          | 0.1804 | 0.0447 | 0.1512 | 0.1498    | 35.17   |
| 0.2174        | 19.0  | 2869 | 1.0854          | 0.1774 | 0.0417 | 0.1483 | 0.1456    | 36.54   |
| 0.1076        | 20.0  | 3020 | 1.0986          | 0.1606 | 0.0389 | 0.1308 | 0.1309    | 37.39   |
| 0.1076        | 21.0  | 3171 | 1.1028          | 0.1727 | 0.0445 | 0.1421 | 0.1408    | 40.47   |
| 0.1076        | 22.0  | 3322 | 1.1114          | 0.1776 | 0.0401 | 0.1431 | 0.1402    | 52.41   |
| 0.1076        | 23.0  | 3473 | 1.1294          | 0.1876 | 0.0374 | 0.1552 | 0.153     | 36.39   |
| 0.0497        | 24.0  | 3624 | 1.1328          | 0.1862 | 0.0443 | 0.1551 | 0.1526    | 47.9    |
| 0.0497        | 25.0  | 3775 | 1.1398          | 0.198  | 0.0513 | 0.1599 | 0.1577    | 52.27   |
| 0.0497        | 26.0  | 3926 | 1.1626          | 0.1723 | 0.0306 | 0.1414 | 0.1397    | 38.1    |
| 0.025         | 27.0  | 4077 | 1.1681          | 0.1867 | 0.0412 | 0.1547 | 0.152     | 37.74   |
| 0.025         | 28.0  | 4228 | 1.1717          | 0.1454 | 0.0217 | 0.1103 | 0.1088    | 39.31   |
| 0.025         | 29.0  | 4379 | 1.1802          | 0.1668 | 0.023  | 0.13   | 0.1285    | 44.39   |
| 0.0155        | 30.0  | 4530 | 1.1900          | 0.1732 | 0.0335 | 0.1434 | 0.1419    | 41.42   |
| 0.0155        | 31.0  | 4681 | 1.1900          | 0.1828 | 0.0411 | 0.1542 | 0.1527    | 39.23   |
| 0.0155        | 32.0  | 4832 | 1.1996          | 0.1902 | 0.0346 | 0.1536 | 0.1523    | 44.7    |
| 0.0155        | 33.0  | 4983 | 1.1976          | 0.1878 | 0.0511 | 0.1517 | 0.1509    | 40.07   |
| 0.0103        | 34.0  | 5134 | 1.2044          | 0.1847 | 0.0402 | 0.1481 | 0.1478    | 47.21   |
| 0.0103        | 35.0  | 5285 | 1.2020          | 0.1915 | 0.0351 | 0.1504 | 0.1489    | 56.49   |
| 0.0103        | 36.0  | 5436 | 1.2098          | 0.1941 | 0.0385 | 0.1546 | 0.1535    | 48.35   |
| 0.008         | 37.0  | 5587 | 1.2223          | 0.1842 | 0.0414 | 0.1499 | 0.1478    | 41.0    |
| 0.008         | 38.0  | 5738 | 1.2183          | 0.1899 | 0.0349 | 0.1526 | 0.1516    | 41.29   |
| 0.008         | 39.0  | 5889 | 1.2294          | 0.1928 | 0.0384 | 0.1578 | 0.1568    | 39.19   |
| 0.0062        | 40.0  | 6040 | 1.2325          | 0.2012 | 0.046  | 0.1653 | 0.1637    | 42.46   |
| 0.0062        | 41.0  | 6191 | 1.2352          | 0.1862 | 0.0326 | 0.1519 | 0.1496    | 39.36   |
| 0.0062        | 42.0  | 6342 | 1.2343          | 0.1956 | 0.0369 | 0.1635 | 0.1618    | 41.77   |
| 0.0062        | 43.0  | 6493 | 1.2371          | 0.1838 | 0.0299 | 0.1529 | 0.1521    | 42.28   |
| 0.0049        | 44.0  | 6644 | 1.2351          | 0.191  | 0.0358 | 0.1574 | 0.1565    | 41.81   |
| 0.0049        | 45.0  | 6795 | 1.2403          | 0.1898 | 0.035  | 0.1522 | 0.1505    | 41.59   |
| 0.0049        | 46.0  | 6946 | 1.2394          | 0.2021 | 0.0465 | 0.1702 | 0.1686    | 43.89   |
| 0.004         | 47.0  | 7097 | 1.2412          | 0.1981 | 0.0416 | 0.164  | 0.1632    | 41.01   |
| 0.004         | 48.0  | 7248 | 1.2427          | 0.1765 | 0.0246 | 0.1447 | 0.1439    | 41.12   |
| 0.004         | 49.0  | 7399 | 1.2427          | 0.1806 | 0.0279 | 0.1473 | 0.1461    | 42.85   |
| 0.0036        | 50.0  | 7550 | 1.2443          | 0.1837 | 0.028  | 0.1489 | 0.1474    | 42.57   |


### Framework versions

- Transformers 4.36.2
- Pytorch 1.12.1+cu113
- Datasets 2.16.1
- Tokenizers 0.15.0
