BERT named entity recognition
220250247
BOND: BERT Assisted Open Domain Named Entity Recognition with Distant Supervision
We study the open domain named entity recognition (NER) problem under distant supervision. The distant supervision, though does not require large amounts of manual annotations, yields highly incomplete and noisy distant labels via external knowledge bases. To address this challenge, we propose a new computational framework BOND, which leverages the power of pre trained language models (e.g. BERT and RoBERTa) to improve the prediction performance of NER models. Specifically, we propose a two stage training algorithm: In the first stage, we adapt the pre trained language model to the NER tasks using the distant labels, which can significantly improve the recall and precision; In the second stage, we drop the distant labels, and propose a self training approach to further improve the model performance. Thorough experiments on 5 benchmark datasets demonstrate the superiority of BOND over existing distantly supervised NER methods. The code and distantly labeled data have been released in https:/github.com/cliang1453/BOND.
KDD
Chen Liang Yue Yu Haoming Jiang Siawpeng Er Ruijia Wang Tuo Zhao Chao Zhang
202734422
Portuguese Named Entity Recognition using BERT CRF
Recent advances in language representation using neural networks have made it viable to transfer the learned internal states of a trained model to downstream natural language processing tasks, such as named entity recognition (NER) and question answering. It has been shown that the leverage of pre trained language models improves the overall performance on many tasks and is highly beneficial when labeled data is scarce. In this work, we employ a pre trained BERT with Conditional Random Fields (CRF) architecture to the NER task on the Portuguese language, combining the transfer capabilities of BERT with the structured predictions of CRF. We explore feature based and fine tuning training strategies for the BERT model. Our fine tuning approach obtains new state of the art results on the HAREM I dataset, improving the F1 score by 3.2 points on the selective scenario (5 NE classes) and by 3.8 points on the total scenario (10 NE classes)
ArXiv
Fabio Souza Rodrigo Nogueira Roberto de Alencar Lotufo
218469094
Chinese clinical named entity recognition with variant neural structures based on BERT methods
Clinical Named Entity Recognition (CNER) is a critical task which aims to identify and classify clinical terms in electronic medical records. In recent years, deep neural networks have achieved significant success in CNER. However, these methods require high quality and large scale labeled clinical data, which is challenging and expensive to obtain, especially data on Chinese clinical records. To tackle the Chinese CNER task, we pre train BERT model on the unlabeled Chinese clinical records, which can leverage the unlabeled domain specific knowledge. Different layers such as Long Short Term Memory (LSTM) and Conditional Random Field (CRF) are used to extract the text features and decode the predicted tags respectively. In addition, we propose a new strategy to incorporate dictionary features into the model. Radical features of Chinese characters are used to improve the model performance as well. To the best of our knowledge, our ensemble model outperforms the state of the art models which achieves 89.56% strict F1 score on the CCKS 2018 dataset and 91.60% F1 score on CCKS 2017 dataset.
J. Biomed. Informatics
Xiangyang Li Huan Zhang Xiaohui Zhou
219721586
Using BERT and Augmentation in Named Entity Recognition for Cybersecurity Domain
The paper presents the results of applying the BERT representation model in the named entity recognition task for the cybersecurity domain in Russian. Several variants of the model were investigated. The best results were obtained using the BERT model, trained on the target collection of information security texts. We also explored a new form of data augmentation for the task of named entity recognition.
NLDB
Mikhail Tikhomirov Natalia V Loukachevitch Anastasiia Sirotina Boris V Dobrov
219179830
Exploring Cross sentence Contexts for Named Entity Recognition with BERT
Named entity recognition (NER) is frequently addressed as a sequence classification task with each input consisting of one sentence of text. It is nevertheless clear that useful information for NER is often found also elsewhere in text. Recent self attention models like BERT can both capture long distance relationships in input and represent inputs consisting of several sentences. This creates opportunities for adding cross sentence information in natural language processing tasks. This paper presents a systematic study exploring the use of cross sentence information for NER using BERT models in five languages. We find that adding context as additional sentences to BERT input systematically increases NER performance. Multiple sentences in input samples allows us to study the predictions of the sentences in different contexts. We propose a straightforward method, Contextual Majority Voting (CMV) to combine these different predictions and demonstrate this to further increase NER performance. Evaluation on established datasets, including the CoNLL'02 and CoNLL'03 NER benchmarks, demonstrates that our proposed approach can improve on the state of the art NER results on English, Dutch, and Finnish, achieves the best reported BERT based results on German, and is on par with other BERT based approaches in Spanish. We release all methods implemented in this work under open licenses.
COLING
Jouni Luoma Sampo Pyysalo
221191488
Chinese Named Entity Recognition Based on BERT with Whole Word Masking
Named Entity Recognition (NER) is a basic task of natural language processing and an indispensable part of machine translation, knowledge mapping and other fields. In this paper, a fusion model of Chinese named entity recognition using BERT, Bidirectional LSTM (BiLSTM) and Conditional Random Field (CRF) is proposed. In this model, Chinese BERT generates word vectors as a word embedding model. Word vectors through BiLSTM can learn the word label distribution. Finally, the model uses Conditional Random Fields to make syntactic restrictions at the sentence level to get annotation sequences. In addition, we can use Whole Word Masking (wwm) instead of the original random mask in BERT's pre training, which can effectively solve the problem that the word in Chinese NER is partly masked, so as to improve the performance of NER model. In this paper, BERT wwm (BERT wwm is the BERT that uses Whole Word Masking in pre training tasks) BERT, ELMo and Word2Vec are respectively used for comparative experiments to reflect the effect of bert wwm in this fusion model. The results show that using Chinese BERT wwm as the language representation model of NER model has better recognition ability.
ICCAI
Chao Liu Cui Zhu Wenjun Zhu
227231779
A Semi Supervised BERT Approach for Arabic Named Entity Recognition
Named entity recognition (NER) plays a significant role in many applications such as information extraction, information retrieval, question answering, and even machine translation. Most of the work on NER using deep learning was done for non Arabic languages like English and French, and only few studies focused on Arabic. This paper proposes a semi supervised learning approach to train a BERT based NER model using labeled and semi labeled datasets. We compared our approach against various baselines, and state of the art Arabic NER tools on three datasets: AQMAR, NEWS, and TWEETS. We report a significant improvement in F measure for the AQMAR and the NEWS datasets, which are written in Modern Standard Arabic (MSA) and competitive results for the TWEETS dataset, which contains tweets that are mostly in the Egyptian dialect and contain many mistakes or misspellings.
WANLP
Chadi Helwe Ghassan Dib Mohsen Shamas Shady Elbassuoni
222417478
Named Entity Recognition in Judicial Field Based on BERT BiLSTM CRF Model
The recognition of named entity in judicial documents is the key to realize automatic trial, and how to effectively distinguish the entities from text is the focus of this paper. However, in special fields, such as the judicial field, many experiments show that the artificial features selection based on domain knowledge have a great influence on the results of the neural network models. Therefore, how to obtain a better named entity recognition performance in judicial field without relying on artificial features is a problem to be solved. In this paper, we propose a neural network model based on BERT BiLSTM CRF. Firstly, we use the BERT pre trained language model to generate the word vectors according to the context of the words, enhance the semantic representation of words, then the word vector sequence is input into BiLSTM CRF for training. Experiments show that our method is effective, at the same time, it solves the problem that the traditional word vector representation maps the word into a single vector and cannot characterize the ambiguity of words.
2020 International Workshop on Electronic Communication and Artificial Intelligence (IWECAI)
Lu Gu Wenjing Zhang Yaoting Wang Bo Li Song Mao
210064296
LTP: A New Active Learning Strategy for Bert CRF Based Named Entity Recognition
In recent years, deep learning has achieved great success in many natural language processing tasks including named entity recognition. The shortcoming is that a large amount of manually annotated data is usually required. Previous studies have demonstrated that active learning could elaborately reduce the cost of data annotation, but there is still plenty of room for improvement. In real applications we found existing uncertainty based active learning strategies have two shortcomings. Firstly, these strategies prefer to choose long sequence explicitly or implicitly, which increase the annotation burden of annotators. Secondly, some strategies need to invade the model and modify to generate some additional information for sample selection, which will increase the workload of the developer and increase the training/prediction time of the model. In this paper, we first examine traditional active learning strategies in a specific case of BiLstm CRF that has widely used in named entity recognition on several typical datasets. Then we propose an uncertainty based active learning strategy called Lowest Token Probability (LTP) which combines the input and output of CRF to select informative instance. LTP is simple and powerful strategy that does not favor long sequences and does not need to invade the model. We test LTP on multiple datasets, and the experiments show that LTP performs slightly better than traditional strategies with obviously less annotation tokens on both sentence level accuracy and entity level F1 score. Related code have been release on this https URL
ArXiv
Mingyi Liu Zhiying Tu Zhongjie Wang Xiaofei Xu
218564040
Named Entity Recognition in Spanish Biomedical Literature: Short Review and Bert Model
Entity Recognition (NER) is the first step for knowledge acquisition when we deal with an unknown corpus of texts. Having received these entities, we have an opportunity to form parameters space and to solve problems of text mining as concept normalization, speech recognition, etc. The recent advances in NER are related to the technology of contextualized word embeddings, which transforms text to the form being effective for Deep Learning. In the paper, we show how NER model detects pharmacological substances, compounds, and proteins in the dataset obtained from the Spanish Clinical Case Corpus (SPACCC) To achieve this goal, we train from scratch the BERT language representation model and fine tune it for our problem. As it is expected, this model shows better results than the NER model trained over the standard word embeddings. We further conduct an error analysis showing the origins of models' errors and proposing strategies to further improve the model's quality.
2020 26th Conference of Open Innovations Association (FRUCT)
Liliya Akhtyamova