

# mt0-definition-no-xl

This model is a version of [mt0-xl](https://huggingface.co/bigscience/mt0-xl) fine-tuned on Bokmålsordboka.

It achieves the following results on the evaluation set:
- Loss: 1.9882
- Rouge1: 31.4539
- Rouge2: 16.1017
- Rougel: 30.6959
- Rougelsum: 30.6888
- Gen Len: 8.9348

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 128
- total_eval_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20.0

### Training results

| Training Loss | Epoch | Step | Validation Loss | Rouge1  | Rouge2  | Rougel  | Rougelsum | Gen Len |
|:-------------:|:-----:|:----:|:---------------:|:-------:|:-------:|:-------:|:---------:|:-------:|
| 2.6123        | 1.0   | 497  | 1.8900          | 28.8529 | 13.6138 | 28.1471 | 28.1199   | 8.9495  |
| 2.0642        | 2.0   | 994  | 1.8383          | 30.5095 | 15.2505 | 29.8018 | 29.7658   | 8.6406  |
| 1.7611        | 3.0   | 1491 | 1.8447          | 30.9812 | 15.6932 | 30.2339 | 30.2413   | 8.7151  |
| 1.5284        | 4.0   | 1989 | 1.8619          | 30.9706 | 15.4516 | 30.1888 | 30.1911   | 9.3787  |
| 1.3422        | 5.0   | 2486 | 1.8895          | 30.9451 | 15.5242 | 30.1826 | 30.1837   | 9.2026  |
| 1.1862        | 6.0   | 2983 | 1.9224          | 31.3072 | 15.959  | 30.5538 | 30.5404   | 8.8816  |
| 1.0526        | 7.0   | 3480 | 1.9882          | 31.4465 | 16.095  | 30.6871 | 30.6739   | 8.7294  |
| 0.9384        | 8.0   | 3978 | 2.0583          | 31.1434 | 15.7298 | 30.287  | 30.2831   | 9.6134  |
| 0.8408        | 9.0   | 4475 | 2.1237          | 30.7808 | 15.4943 | 29.9606 | 29.9589   | 9.6527  |
| 0.7592        | 10.0  | 4972 | 2.1987          | 31.0097 | 15.5823 | 30.1202 | 30.1151   | 9.9255  |


### Framework versions

- Transformers 4.30.2
- Pytorch 1.13.1+rocm5.2
- Datasets 2.12.0
- Tokenizers 0.12.1

{"language": [false], "tags": ["text2text-generation", "definition-modeling"], "metrics": ["rouge"], "widget": [{"text": "Ha egen brygge og b\u00e5thus. Hva betyr b\u00e5thus?", "example_title": "Definition generation"}], "model-index": [{"name": "mt0-definition-no-xl", "results": []}]}
transformers pytorch mt5 text2text-generation definition-modeling no autotrain_compatible endpoints_compatible text-generation-inference region:us