
#  XLM-RoBERTa (base) Middle High German Charter Masked Language Model
This model is a fine-tuned version of xlm-roberta-base on Middle High German (gmh; ISO 639-2; c. 1050–1500) charters of the [monasterium.net](https://www.icar-us.eu/en/cooperation/online-portals/monasterium-net/) data set.

## Model description
Please refer this model together with to the [XLM-RoBERTa (base-sized model)](https://huggingface.co/xlm-roberta-base) card or the paper [Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al.](https://arxiv.org/abs/1911.02116) for additional information.

## Intended uses & limitations
This model can be used for sequence prediction tasks, i.e., fill-masks.

## Training and evaluation data
The model was fine-tuned using the Middle High German Monasterium charters.
It was trained on a Tesla V100-SXM2-16GB GPU.

## Training hyperparameters
The following hyperparameters were used during training:
- num_train_epochs: 15
- learning_rate: 2e-5
- weight-decay: 0,01
- train_batch_size: 16
- eval_batch_size: 16
- num_proc: 4
- block_size: 256


## Training results

| Epoch | Training Loss | Validation Loss |
|-------|---------------|------------------|
| 1     | 2.423800      | 2.025645         |
| 2     | 1.876500      | 1.700380         |
| 3     | 1.702100      | 1.565900         |
| 4     | 1.582400      | 1.461868         |
| 5     | 1.506000      | 1.393849         |
| 6     | 1.407300      | 1.359359         |
| 7     | 1.385400      | 1.317869         |
| 8     | 1.336700      | 1.285630         |
| 9     | 1.301300      | 1.246812         |
| 10    | 1.273500      | 1.219290         |
| 11    | 1.245600      | 1.198312         |
| 12    | 1.225800      | 1.198695         |
| 13    | 1.214100      | 1.194895         |
| 14    | 1.209500      | 1.177452         |
| 15    | 1.200300      | 1.177396         |

Perplexity: 3.25

## Updates
- 2023-03-30: Upload


## Citation
Please cite the following papers when using this model.

```
@misc{xlm-roberta-base-mhg-charter-mlm,
  title={xlm-roberta-base-mhg-charter-mlm},
  author={Atzenhofer-Baumgartner, Florian},
  year         = { 2023 },
  url          = { https://huggingface.co/atzenhofer/xlm-roberta-base-mhg-charter-mlm },
  publisher    = { Hugging Face }
}
```
{"language": ["gmh", "de"], "license": "gpl-3.0", "widget": [{"text": "Ich Ott von Zintzndorff vergich mit dem offenn prief vnd tun chunt alln den leutn, di in sehnt oder hornt lesn, daz ich mit wolbedachtm mut vnd mit guetem rat vnd willn czu der czeit, do ich ez wol getun mochtt, den erbern herrn vnd fursten apt Englschalchn cze Seydensteten vnd sein gnants Gotshavs daselbs gancz vnd gar ledig sage vnd lazze aller der ansproch, die ich ...  han auf seiner guter ains, des Schoephls lehn auf dem Graentleinsperg gnant in Groestner pharr gelegn, also, daz ich vnd alle mein erbn furbaz dhain ansprach dar vmb habn welln noch schulln in dhainn wegn, weder wenig noch vil. Vnd dar vmb czu eine steten vrchund gib ich dem vorgnantn Apt Englchalchn vnd seim wirdign Gotshaws cze Seydenstet den prief, versigelt mit meim des egnantn Ottn von Czintzndorff, vnd mit hern Dytrichs des Schenchn von Dobra anhangunden Insigeln, der das durch meinn willn cze gezeug der obgeschribn sach an den prief hat gehang. Das ist geschehn vnd der prief ist gebn nach Christs gepurd vber Drewtzehn hundert Jar, dar nach im Sibn vnd fumftzgisten Jar, am Eritag in den Phingstveyrtagn."}]}
transformers pytorch xlm-roberta fill-mask gmh de arxiv:1911.02116 license:gpl-3.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #xlm-roberta #fill-mask #gmh #de #arxiv-1911.02116 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

XLM-RoBERTa (base) Middle High German Charter Masked Language Model
===================================================================


This model is a fine-tuned version of xlm-roberta-base on Middle High German (gmh; ISO 639-2; c. 1050–1500) charters of the URL data set.


Model description
-----------------


Please refer this model together with to the XLM-RoBERTa (base-sized model) card or the paper Unsupervised Cross-lingual Representation Learning at Scale by Conneau et al. for additional information.


Intended uses & limitations
---------------------------


This model can be used for sequence prediction tasks, i.e., fill-masks.


Training and evaluation data
----------------------------


The model was fine-tuned using the Middle High German Monasterium charters.
It was trained on a Tesla V100-SXM2-16GB GPU.


Training hyperparameters
------------------------


The following hyperparameters were used during training:


* num\_train\_epochs: 15
* learning\_rate: 2e-5
* weight-decay: 0,01
* train\_batch\_size: 16
* eval\_batch\_size: 16
* num\_proc: 4
* block\_size: 256


Training results
----------------


Epoch: 1, Training Loss: 2.423800, Validation Loss: 2.025645
Epoch: 2, Training Loss: 1.876500, Validation Loss: 1.700380
Epoch: 3, Training Loss: 1.702100, Validation Loss: 1.565900
Epoch: 4, Training Loss: 1.582400, Validation Loss: 1.461868
Epoch: 5, Training Loss: 1.506000, Validation Loss: 1.393849
Epoch: 6, Training Loss: 1.407300, Validation Loss: 1.359359
Epoch: 7, Training Loss: 1.385400, Validation Loss: 1.317869
Epoch: 8, Training Loss: 1.336700, Validation Loss: 1.285630
Epoch: 9, Training Loss: 1.301300, Validation Loss: 1.246812
Epoch: 10, Training Loss: 1.273500, Validation Loss: 1.219290
Epoch: 11, Training Loss: 1.245600, Validation Loss: 1.198312
Epoch: 12, Training Loss: 1.225800, Validation Loss: 1.198695
Epoch: 13, Training Loss: 1.214100, Validation Loss: 1.194895
Epoch: 14, Training Loss: 1.209500, Validation Loss: 1.177452
Epoch: 15, Training Loss: 1.200300, Validation Loss: 1.177396


Perplexity: 3.25


Updates
-------


* 2023-03-30: Upload


Please cite the following papers when using this model.



TAGS
#transformers #pytorch #xlm-roberta #fill-mask #gmh #de #arxiv-1911.02116 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 

TAGS
#transformers #pytorch #xlm-roberta #fill-mask #gmh #de #arxiv-1911.02116 #license-gpl-3.0 #autotrain_compatible #endpoints_compatible #region-us 
