
# DaNE+

This is a version of [DaNE](https://huggingface.co/datasets/dane), where the original NER labels have been updated to follow the ontonotes annotation scheme. The annotation process used the model trained on the Danish dataset [DANSK](https://huggingface.co/datasets/chcaa/DANSK) for the first round of annotation and then all the discrepancies were manually reviewed and corrected by Kenneth C. Enevoldsen. A discrepancy include notably also includes newly added entities such as `PRODUCT` and `WORK_OF_ART`. Thus in practice a great deal of entities were manually reviews. If there was an uncertainty the annotation was left as it was.

The additional annotations (e.g. part-of-speech tags) stems from the Danish Dependency Treebank, however, if you wish to use these I would recommend using the latest version as this version here will likely become outdated over time.


## Process of annotation

1) Install the requirements:
```
--extra-index-url pip install prodigy -f https://{DOWNLOAD KEY}@download.prodi.gy
prodigy>=1.11.0,<2.0.0
```

2) Create outline dataset
```bash
python annotate.py
```

3) Review and correction annotation using prodigy:
Add datasets to prodigy
```bash
prodigy db-in dane reference.jsonl
prodigy db-in dane_plus_mdl_pred predictions.jsonl
```

Run review using prodigy:
```bash
prodigy review daneplus dane_plus_mdl_pred,dane  --view-id ner_manual --l NORP,CARDINAL,PRODUCT,ORGANIZATION,PERSON,WORK_OF_ART,EVENT,LAW,QUANTITY,DATE,TIME,ORDINAL,LOCATION,GPE,MONEY,PERCENT,FACILITY
```

Export the dataset:
```bash
prodigy data-to-spacy daneplus --ner daneplus --lang da -es 0
```

4) Redo the original split:

```bash
python split.py
```

task_categories:token-classification task_ids:named-entity-recognition task_ids:part-of-speech annotations_creators:expert-generated annotations_creators:machine-generated language_creators:found multilinguality:monolingual size_categories:1K<n<10K source_datasets:dane source_datasets:extended|other-Danish-Universal-Dependencies-treebank source_datasets:DANSK language:da license:cc-by-sa-4.0 region:us
{"annotations_creators": ["expert-generated", "machine-generated"], "language_creators": ["found"], "language": ["da"], "license": ["cc-by-sa-4.0"], "multilinguality": ["monolingual"], "size_categories": ["1K<n<10K"], "source_datasets": ["dane", "extended|other-Danish-Universal-Dependencies-treebank", "DANSK"], "task_categories": ["token-classification"], "task_ids": ["named-entity-recognition", "part-of-speech"], "paperswithcode_id": "dane", "pretty_name": "DaNE+", "dataset_info": {"features": [{"name": "text", "dtype": "string"}, {"name": "ents", "list": [{"name": "end", "dtype": "int64"}, {"name": "label", "dtype": "string"}, {"name": "start", "dtype": "int64"}]}, {"name": "sents", "list": [{"name": "end", "dtype": "int64"}, {"name": "start", "dtype": "int64"}]}, {"name": "tokens", "list": [{"name": "dep", "dtype": "string"}, {"name": "end", "dtype": "int64"}, {"name": "head", "dtype": "int64"}, {"name": "id", "dtype": "int64"}, {"name": "lemma", "dtype": "string"}, {"name": "morph", "dtype": "string"}, {"name": "pos", "dtype": "string"}, {"name": "start", "dtype": "int64"}, {"name": "tag", "dtype": "string"}]}], "splits": [{"name": "train", "num_bytes": 7886693, "num_examples": 4383}, {"name": "dev", "num_bytes": 1016350, "num_examples": 564}, {"name": "test", "num_bytes": 991137, "num_examples": 565}], "download_size": 1627548, "dataset_size": 9894180}}
TAGS
#task_categories-token-classification #task_ids-named-entity-recognition #task_ids-part-of-speech #annotations_creators-expert-generated #annotations_creators-machine-generated #language_creators-found #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-dane #source_datasets-extended|other-Danish-Universal-Dependencies-treebank #source_datasets-DANSK #language-Danish #license-cc-by-sa-4.0 #region-us 


# DaNE+

This is a version of DaNE, where the original NER labels have been updated to follow the ontonotes annotation scheme. The annotation process used the model trained on the Danish dataset DANSK for the first round of annotation and then all the discrepancies were manually reviewed and corrected by Kenneth C. Enevoldsen. A discrepancy include notably also includes newly added entities such as 'PRODUCT' and 'WORK_OF_ART'. Thus in practice a great deal of entities were manually reviews. If there was an uncertainty the annotation was left as it was.

The additional annotations (e.g. part-of-speech tags) stems from the Danish Dependency Treebank, however, if you wish to use these I would recommend using the latest version as this version here will likely become outdated over time.


## Process of annotation

1) Install the requirements:


2) Create outline dataset


3) Review and correction annotation using prodigy:
Add datasets to prodigy


Run review using prodigy:


Export the dataset:


4) Redo the original split:



# DaNE+

This is a version of DaNE, where the original NER labels have been updated to follow the ontonotes annotation scheme. The annotation process used the model trained on the Danish dataset DANSK for the first round of annotation and then all the discrepancies were manually reviewed and corrected by Kenneth C. Enevoldsen. A discrepancy include notably also includes newly added entities such as 'PRODUCT' and 'WORK_OF_ART'. Thus in practice a great deal of entities were manually reviews. If there was an uncertainty the annotation was left as it was.

The additional annotations (e.g. part-of-speech tags) stems from the Danish Dependency Treebank, however, if you wish to use these I would recommend using the latest version as this version here will likely become outdated over time. ## Process of annotation

1) Install the requirements:


2) Create outline dataset


3) Review and correction annotation using prodigy:
Add datasets to prodigy


Run review using prodigy:


Export the dataset:


4) Redo the original split:
TAGS
#task_categories-token-classification #task_ids-named-entity-recognition #task_ids-part-of-speech #annotations_creators-expert-generated #annotations_creators-machine-generated #language_creators-found #multilinguality-monolingual #size_categories-1K<n<10K #source_datasets-dane #source_datasets-extended|other-Danish-Universal-Dependencies-treebank #source_datasets-DANSK #language-Danish #license-cc-by-sa-4.0 #region-us 
 # DaNE+

This is a version of DaNE, where the original NER labels have been updated to follow the ontonotes annotation scheme. The annotation process used the model trained on the Danish dataset DANSK for the first round of annotation and then all the discrepancies were manually reviewed and corrected by Kenneth C. Enevoldsen. A discrepancy include notably also includes newly added entities such as 'PRODUCT' and 'WORK_OF_ART'. Thus in practice a great deal of entities were manually reviews. If there was an uncertainty the annotation was left as it was.

The additional annotations (e.g. part-of-speech tags) stems from the Danish Dependency Treebank, however, if you wish to use these I would recommend using the latest version as this version here will likely become outdated over time. ## Process of annotation

1) Install the requirements:


2) Create outline dataset


3) Review and correction annotation using prodigy:
Add datasets to prodigy


Run review using prodigy:


Export the dataset:


4) Redo the original split: