
# quinten-datalab/AliBERT-7GB: AliBERT: is a pre-trained language model for French biomedical text. 


# Introduction

AliBERT: is a pre-trained language model for French biomedical text. It is trained with masked language model like RoBERTa.

Here are the main contributions of our work:
  <ul>
  <li>
    A French biomedical language model, a language-specific and domain-specific PLM, which can be used to represent French biomedical text for different downstream tasks.
  </li>
  <li>
    A normalization of a Unigram sub-word tokenization of French biomedical textual input which improves our vocabulary and overall performance of the models trained.
  </li>
  <li>
      It is a foundation model that achieved state-of-the-art results on French biomedical text.
  </li>
  </ul>

The Paper can be found here: https://aclanthology.org/2023.bionlp-1.19/ 

# Data
The pre-training corpus was gathered from different sub-corpora. It is composed of 7GB French biomedical textual documents. The corpora were collected from different sources. Scientific articles are collected from ScienceDirect using an API provided on subscription and where French articles in biomedical domain were selected. The summaries of thesis manuscripts are collected from "Système universitaire de documentation (SuDoc)" which is a catalog of universities documentation system. Short texts and some complete sentences were collected from the public drug database which lists the characteristics of tens of thousands of drugs. Furthermore, a similar drug database known as "Résumé des Caractéristiques du Produit (RCP)" is also used to represent a description of medications that are intended to be utilized by biomedicine professionals.



# How to use alibert-quinten/Oncology-NER with HuggingFace

Load quinten-datalab/AliBERT-7GB fill-mask model and the tokenizer used to train AliBERT:

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification,pipeline

tokenizer = AutoTokenizer.from_pretrained("quinten-datalab/AliBERT-7GB")

model = AutoModelForTokenMaskedLM.from_pretrained("quinten-datalab/AliBERT-7GB")

fill_mask=pipeline("fill-mask",model=model,tokenizer=tokenizer)
nlp_AliBERT=fill_mask("La prise de greffe a été systématiquement réalisée au niveau de la face interne de la [MASK] afin de limiter la plaie cicatricielle.")

[{'score': 0.7724128365516663,
  'token': 6749,
  'token_str': 'cuisse',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la cuisse afin de limiter la plaie cicatricielle.'},
 {'score': 0.09472355246543884,
  'token': 4915,
  'token_str': 'jambe',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la jambe afin de limiter la plaie cicatricielle.'},
 {'score': 0.03340734913945198,
  'token': 2050,
  'token_str': 'main',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la main afin de limiter la plaie cicatricielle.'},
 {'score': 0.030924487859010696,
  'token': 844,
  'token_str': 'face',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la face afin de limiter la plaie cicatricielle.'},
 {'score': 0.012518334202468395,
  'token': 3448,
  'token_str': 'joue',
  'sequence': 'La prise de greffe a été systématiquement réalisée au niveau de la face interne de la joue afin de limiter la plaie cicatricielle.'}]
```

# Metrics and results
The model has been evaluted in the following downstream tasks

## Biomedical Named Entity Recognition (NER)
The model is evaluated on two (CAS and QUAERO) publically available Frech biomedical text.
#### CAS dataset

<style type="text/css">
.tg  {border-collapse:collapse;border-spacing:0;}
.tg .tg-baqh{text-align:center;vertical-align:top}
.tg .tg-0lax{text-align:center;vertical-align:top}
</style>
<table class="tg">
<thead>
  <tr>
    <th>Models</th>
    <th class="tg-0lax" colspan="3">CamemBERT</th>
    <th class="tg-0lax" colspan="3">AliBERT</th>
    <th class="tg-0lax" colspan="3">DrBERT</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>Entities</td><td>P<br></td><td>R</td><td>F1</td><td>P<br></td><td>R</td><td>F1</td><td>P<br></td><td>R</td><td>F1</td>
  </tr>
  <tr>
    <td>Substance</td><td>0.96</td><td>0.87</td><td>0.91</td><td>0.96</td><td>0.91</td><td>0.93</td><td>0.83</td><td>0.83</td><td>0.82</td>
  </tr>
  <tr>
    <td>Symptom</td> <td>0.89</td> <td>0.91</td> <td>0.90</td> <td>0.96</td> <td>0.98</td> <td>0.97</td> <td>0.93</td> <td>0.90</td> <td>0.91</td>
  </tr>
  <tr>
    <td>Anatomy</td> <td>0.94</td> <td>0.91</td> <td>0.88</td> <td>0.97</td> <td>0.97</td> <td>0.98</td> <td>0.92</td> <td>0.93</td> <td>0.93</td> 
  </tr>
  <tr>
    <td>Value</td> <td>0.88</td> <td>0.46</td> <td>0.60</td> <td>0.98</td> <td>0.99</td> <td>0.98</td> <td>0.91</td> <td>0.91</td> <td>0.91</td> 
  </tr>
  <tr>
   <td> Pathology</td> <td>0.79</td> <td>0.70</td> <td>0.74</td> <td>0.81</td> <td>0.39</td> <td>0.52</td> <td>0.85	<td>0.57</td> <td>0.68</td>
  </tr>
  <tr>
    <td>Macro Avg</td> <td>0.89  </td> <td>0.79</td> <td>0.81</td> <td> 0.94</td> <td>0.85</td> <td>0.88</td> <td> 0.92</td> <td> 0.87</td> <td>0.89</td>  
  </tr>
</tbody>
</table>
Table 1: NER performances on CAS dataset

#### QUAERO dataset

<table class="tg">
<thead>
  <tr>
    <th>Models</th>
    <th class="tg-0lax" colspan="3">CamemBERT</th>
    <th class="tg-0lax" colspan="3">AliBERT</th>
    <th class="tg-0lax" colspan="3">DrBERT</th>
  </tr>
</thead>
<tbody>
<tr>
<td>Entity </td> <td>  P </td> <td> R </td> <td> F1 </td> <td> P </td> <td> R </td> <td> F1 </td> <td>  P </td> <td> R </td> <td> F1 </td>
</tr>
<tr>
<td>Anatomy </td> <td>  0.649 </td> <td> 0.641 </td> <td> 0.645 </td> <td>  0.795 </td> <td> 0.811 </td> <td> 0.803 </td> <td> 0.736 </td> <td>   0.844 </td> <td> 0.824  </td>
</tr>
<tr>
<td>Chemical </td> <td>  0.844 </td> <td> 0.847 </td> <td> 0.846 </td> <td>  0.878 </td> <td> 0.893 </td> <td> 0.885 </td> <td>  0.505 </td> <td>   0.823 </td> <td> 0.777  </td>
</tr>
<tr>
<td>Device </td> <td>  0.000 </td> <td> 0.000 </td> <td> 0.000 </td> <td>  0.506 </td> <td> 0.356 </td> <td> 0.418 </td> <td>  0.939 </td> <td>   0.237 </td> <td> 0.419  </td>
</tr>
<tr>
<td>Disorder </td> <td>  0.772 </td> <td> 0.818 </td> <td> 0.794 </td> <td>  0.857 </td> <td> 0.843 </td> <td> 0.850 </td> <td>  0.883 </td> <td>   0.809 </td> <td> 0.845  </td>
</tr>
<tr>
<td>Procedure </td> <td>  0.880 </td> <td> 0.894 </td> <td> 0.887 </td> <td>  0.969 </td> <td> 0.967 </td> <td> 0.968 </td> <td>  0.944 </td> <td>  0.976 </td> <td> 0.960  </td>
</tr>
<tr>
<td>Macro Avg </td> <td>  0.655 </td> <td> 0.656 </td> <td> 0.655 </td> <td>  0.807 </td> <td> 0.783 </td> <td> 0.793 </td> <td>  0.818 </td> <td>  0.755 </td> <td> 0.782  </td>
</tr>
</tbody>
</table>
Table 2: NER performances on QUAERO dataset

##AliBERT: A Pre-trained Language Model for French Biomedical Text
{"language": ["fr"], "license": "mit", "library_name": "transformers", "tags": ["Biomedical", "Medical", "French-Biomedical"], "Mask token": [["MASK"]], "widget": [{"text": "A l\u2019admission, l\u2019examen clinique mettait en \u00e9vidence : - une hypotension art\u00e9rielle avec une pression [MASK] \u00e0 6 mmHg.", "example_title": "Example 1"}, {"text": "Le patient a \u00e9t\u00e9 diagnostiqu\u00e9 avec une [MASK] lobaire  aigu\u00eb et a \u00e9t\u00e9 trait\u00e9 avec des antibiotiques appropri\u00e9s", "example_title": "Example 2"}, {"text": "En mars 2001, le malade fut op\u00e9r\u00e9, mais vu le caract\u00e8re h\u00e9morragique de la tumeur, une simple biopsie surr\u00e9nalienne a \u00e9t\u00e9 r\u00e9alis\u00e9e ayant montr\u00e9 l\u2019aspect de [MASK] malin non Hodgkinien de haut grade de malignit\u00e9.", "example_title": "Example 3"}, {"text": "La cytologie urinaire n\u2019a mis en \u00e9vidence que des cellules [MASK] normales et l\u2019examen cyto-bact\u00e9riologique des urines \u00e9tait st\u00e9rile.", "example_title": "Example 4"}, {"text": "La prise de greffe a \u00e9t\u00e9 syst\u00e9matiquement r\u00e9alis\u00e9e au niveau de la face interne de la [MASK] afin de limiter la plaie cicatricielle.", "example_title": "Example 5"}]}
transformers pytorch roberta fill-mask Biomedical Medical French-Biomedical fr license:mit autotrain_compatible endpoints_compatible region:us