# German FinBERT For Sentiment Analysis (Pre-trained From Scratch Version, Fine-Tuned for Financial Sentiment Analysis)
<img src="https://github.com/mscherrmann/mscherrmann.github.io/blob/master/assets/img/publication_preview/germanBert.png?raw=true" alt="Alt text for the image" width="500" height="300"/>


German FinBERT is a BERT language model focusing on the financial domain within the German language. In my [paper](https://arxiv.org/pdf/2311.08793.pdf), I describe in more detail the steps taken to train the model and show that it outperforms its generic benchmarks for finance specific downstream tasks. 

This model is the [pre-trained from scratch version of German FinBERT](https://huggingface.co/scherrmann/GermanFinBert_SC), after fine-tuning on a translated version of the [financial news phrase bank](https://arxiv.org/abs/1307.5336) of Malo et al. (2013). The data is available [here](https://huggingface.co/datasets/scherrmann/financial_phrasebank_75agree_german).

## Overview  
**Author** Moritz Scherrmann
**Paper:** [here](https://arxiv.org/pdf/2311.08793.pdf)  
**Architecture:** BERT base 
**Language:** German  
**Specialization:** Financial sentiment
**Base model:** [German_FinBert_SC](https://huggingface.co/scherrmann/GermanFinBert_SC)


### Fine-tuning

I fine-tune the model using the 1cycle policy of [Smith and Topin (2019)](https://arxiv.org/abs/1708.07120). I use the Adam optimization method of [Kingma and Ba (2014)](https://arxiv.org/abs/1412.6980) with
standard parameters.I run a grid search on the evaluation set to find the best hyper-parameter setup. I test different
values for learning rate, batch size and number of epochs, following the suggestions of [Chalkidis et al. (2020)](https://aclanthology.org/2020.findings-emnlp.261/). I repeat the fine-tuning for each setup five times with different seeds, to avoid getting good results by chance.
After finding the best model w.r.t the evaluation set, I report the mean result across seeds for that model on the test set.

### Results

Translated [Financial news phrase bank](https://arxiv.org/abs/1307.5336) (Malo et al. (2013)), see [here](https://huggingface.co/datasets/scherrmann/financial_phrasebank_75agree_german) for the data:
- Accuracy: 95.95%
- Macro F1: 92.70%


## Authors  
Moritz Scherrmann: `scherrmann [at] lmu.de`


For additional details regarding the performance on fine-tune datasets and benchmark results, please refer to the full documentation provided in the study.

See also:  
- scherrmann/GermanFinBERT_SC
- scherrmann/GermanFinBERT_FP
- scherrmann/GermanFinBERT_FP_QuAD
{"language": ["de"], "license": "apache-2.0", "widget": [{"text": "STS Group AG erh\u00e4lt Gro\u00dfauftrag von f\u00fchrendem Nutzfahrzeughersteller in Nordamerika und plant Bau eines ersten US-Werks"}, {"text": "Zuk\u00fcnftig soll jedoch je Gesch\u00e4ftsjahr eine Mindestdividende in H\u00f6he von EUR 2,00 je dividendenberechtigter Aktie an die Aktion\u00e4rinnen und Aktion\u00e4re ausgesch\u00fcttet werden."}, {"text": "Comet passt Jahresprognose nach Q3 unter Erwartungen an"}]}
transformers pytorch bert text-classification de arxiv:2311.08793 arxiv:1307.5336 arxiv:1708.07120 arxiv:1412.6980 license:apache-2.0 autotrain_compatible endpoints_compatible region:us