---
license: apache-2.0
tags:
- token-classification
datasets:
- conll2003
- conllpp
language:
- en
metrics:
- f1: 92.85
- f1(valid): 96.71
- f1(CoNLLpp(2023)): 92.35
- f1(CoNLLpp(CrossWeigh)): 94.26
---


# Roberta-Base-CoNLL2003

This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on the conll2003 dataset.

## Model Usage

We made and used the original tokenizer with [BPE-Dropout](https://aclanthology.org/2020.acl-main.170/). 
So, you can't use AutoTokenizer but if subword normalization is not used, original RobertaTokenizer can be substituted.

Example and Tokenizer Repository: [github](https://github.com/4ldk/CoNLL2003_Choices)

```python
from transformers import RobertaTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = RobertaTokenizer.from_pretrained("4ldk/Roberta-Base-CoNLL2003")
model = AutoModelForTokenClassification.from_pretrained("4ldk/Roberta-Base-CoNLL2003")

nlp = pipeline("ner", model=model, tokenizer=tokenizer, grouped_entities=True)
example = "My name is Philipp and live in Germany"

nlp(example)

```


## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-5
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: AdamW with betas=(0.9,0.999), epsilon=1e-08, and weight decay=0.01
- lr_scheduler_type: linear with warmup rate = 0.1
- num_epochs: 20
- subword regularization p = 0.0 (= trained without subword regularization)

And we add the sentences following the input sentence in the original dataset. Therefore, it cannot be reproduced from the dataset published on huggingface.
For detail in [our github repository](https://github.com/4ldk/CoNLL2003_Choices/blob/develop/src/utils.py)).

### Training results

#### CoNNL2003

It achieves the following results on the evaluation set:
- Precision: 0.9707
- Recall: 0.9636
- F1: 0.9671

It achieves the following results on the test set:
- Precision: 0.9352 
- Recall: 0.9218
- F1: 0.9285

#### CoNNLpp(2023)

[Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023](
https://aclanthology.org/2023.acl-long.459.pdf)
([github](https://github.com/ShuhengL/acl2023_conllpp))

- Precision: 0.9244
- Recall: 0.9225
- F1: 0.9235

#### CoNLLpp(CrossWeigh)

[CrossWeigh: Training Named Entity Tagger from Imperfect Annotations](https://aclanthology.org/D19-1519/)
([github](https://github.com/ZihanWangKi/CrossWeigh))

- Precision: 0.9449
- Recall: 0.9403
- F1: 0.9426


### Framework versions

- Transformers 4.35.2
- Pytorch 2.0.1+cu117