ECBERT-base-mlm

This model is a pretrained version of answerdotai/ModernBERT-base on 25,581 texts (available here) using MLM but not yet fine-tuned on the monetary policy sentiment analysis task. The best model achieves the following results on an out-of-sample test set (Graimond/ECBERT-idioms-dataset):

• Accuracy: 40.00%

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

• Training data: Graimond/ECBERT-mlm-dataset
• Evaluation data: Graimond/ECBERT-idioms-dataset

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-5
• weight_decay=0.01
• per_device_train_batch_size=16
• seed: 42
• epochs: 20

Training results

Epoch	Training Loss	Validation Loss
1	1.905000	1.903329
2	1.689700	1.764568
3	1.600900	nan
4	1.476500	1.683352
5	1.381200	1.629597
6	1.367300	nan
7	1.230300	1.628195
8	1.142700	1.567721
9	1.131800	1.618517
10	1.139700	nan
11	1.086200	nan
12	1.072500	1.560426
13	0.984800	1.556072
14	0.958500	1.606674
15	0.955600	1.619744
16	0.920500	1.581421
17	0.882300	1.535872
18	0.877900	1.565936
19	0.803100	nan
20	0.815700	1.604986

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu121
• Datasets 3.2.0
• Tokenizers 0.21.0