
# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This model card describes the AD-BERT model, which was initialized from [Bio+ClinicalBERT](https://arxiv.org/abs/1901.08746) trained on NWEDW AD-related progress notes.

## Model Details

## Pretraining Data
The `AD-BERT` model was trained on AD related progress notes from NMEDW (private).

## Model Pretraining 

### Note Preprocessing
We preprocessed each progress note derived from the EHRs as follows:
(1)	Deidentification: Clinical notes contain legally Protected Health Information (PHI), such as patient names, addresses, and phone numbers, which should not be released to the public and should not be used for most research applications. We use the package Philter56 to remove PHI for clinical notes.
(2)	Cleaning: We removed non-ASCII characters from the notes and replaced multiple contiguous white spaces with one blank space.  
(3)	Splitting: Each note is split into sections by the newline character (‘\n’). Each section is as an input to the model independently for section representation. And the embedding vector for a patient generated by the MaxPooling of all the section representations.

### Pretraining Procedures
The model was trained using code from [Hugging Face transformer repository](https://github.com/huggingface/transformers/tree/v1.0.0/examples/lm_finetuning) on a Tesla V100 X 32 GB GPU. Model parameters were initialized with Bio+Clinical BERT.

### Pretraining code 
```
INPUT=<customer defined name>
export TRAIN_FILE=<path to the training corpus>
export INITIAL_PATH=<path to the initialized model> (here is Bio+Clinical BERT)

CUDA_VISIBLE_DEVICES=1 python3 ./lm_finetuning/simple_lm_finetuning.py  \
    --train_corpus $TRAIN_FILE   \
    --bert_model $INITIAL_PATH    \
    --do_lower_case     \
    --output_dir ./AD_bert_simple_lm_${INPUT}/   \
    --on_memory   \
    --do_train     &
```

## How to use the model

Load the model via the transformers library:
```
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained(<path to model>)
model = BertModel.from_pretrained(<path to model>)
```

## More Information

Refer to the original paper, [AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease](https://arxiv.org/abs/2212.06042)  for additional details and performance on MCI-to-AD prediction tasks.

## Questions?

Email chengsheng.mao@northwestern.edu with any questions.




{}
transformers pytorch arxiv:1901.08746 arxiv:2212.06042 endpoints_compatible region:us
TAGS
#transformers #pytorch #arxiv-1901.08746 #arxiv-2212.06042 #endpoints_compatible #region-us 


# Model Card for Model ID



This model card describes the AD-BERT model, which was initialized from Bio+ClinicalBERT trained on NWEDW AD-related progress notes.

## Model Details

## Pretraining Data
The 'AD-BERT' model was trained on AD related progress notes from NMEDW (private).

## Model Pretraining 

### Note Preprocessing
We preprocessed each progress note derived from the EHRs as follows:
(1)	Deidentification: Clinical notes contain legally Protected Health Information (PHI), such as patient names, addresses, and phone numbers, which should not be released to the public and should not be used for most research applications. We use the package Philter56 to remove PHI for clinical notes.
(2)	Cleaning: We removed non-ASCII characters from the notes and replaced multiple contiguous white spaces with one blank space.  
(3)	Splitting: Each note is split into sections by the newline character (‘\n’). Each section is as an input to the model independently for section representation. And the embedding vector for a patient generated by the MaxPooling of all the section representations.

### Pretraining Procedures
The model was trained using code from Hugging Face transformer repository on a Tesla V100 X 32 GB GPU. Model parameters were initialized with Bio+Clinical BERT.

### Pretraining code 


## How to use the model

Load the model via the transformers library:


## More Information

Refer to the original paper, AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease  for additional details and performance on MCI-to-AD prediction tasks.

## Questions?

Email URL@URL with any questions.




# Model Card for Model ID



This model card describes the AD-BERT model, which was initialized from Bio+ClinicalBERT trained on NWEDW AD-related progress notes. ## Model Details ## Pretraining Data
The 'AD-BERT' model was trained on AD related progress notes from NMEDW (private). ## Model Pretraining ### Note Preprocessing
We preprocessed each progress note derived from the EHRs as follows:
(1)	Deidentification: Clinical notes contain legally Protected Health Information (PHI), such as patient names, addresses, and phone numbers, which should not be released to the public and should not be used for most research applications. We use the package Philter56 to remove PHI for clinical notes.
(2)	Cleaning: We removed non-ASCII characters from the notes and replaced multiple contiguous white spaces with one blank space.  
(3)	Splitting: Each note is split into sections by the newline character (‘\n’). Each section is as an input to the model independently for section representation. And the embedding vector for a patient generated by the MaxPooling of all the section representations. ### Pretraining Procedures
The model was trained using code from Hugging Face transformer repository on a Tesla V100 X 32 GB GPU. Model parameters were initialized with Bio+Clinical BERT. ### Pretraining code ## How to use the model

Load the model via the transformers library: ## More Information

Refer to the original paper, AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease  for additional details and performance on MCI-to-AD prediction tasks. ## Questions?

Email URL@URL with any questions.
TAGS
#transformers #pytorch #arxiv-1901.08746 #arxiv-2212.06042 #endpoints_compatible #region-us 
 # Model Card for Model ID



This model card describes the AD-BERT model, which was initialized from Bio+ClinicalBERT trained on NWEDW AD-related progress notes. ## Model Details ## Pretraining Data
The 'AD-BERT' model was trained on AD related progress notes from NMEDW (private). ## Model Pretraining ### Note Preprocessing
We preprocessed each progress note derived from the EHRs as follows:
(1)	Deidentification: Clinical notes contain legally Protected Health Information (PHI), such as patient names, addresses, and phone numbers, which should not be released to the public and should not be used for most research applications. We use the package Philter56 to remove PHI for clinical notes.
(2)	Cleaning: We removed non-ASCII characters from the notes and replaced multiple contiguous white spaces with one blank space.  
(3)	Splitting: Each note is split into sections by the newline character (‘\n’). Each section is as an input to the model independently for section representation. And the embedding vector for a patient generated by the MaxPooling of all the section representations. ### Pretraining Procedures
The model was trained using code from Hugging Face transformer repository on a Tesla V100 X 32 GB GPU. Model parameters were initialized with Bio+Clinical BERT. ### Pretraining code ## How to use the model

Load the model via the transformers library: ## More Information

Refer to the original paper, AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease  for additional details and performance on MCI-to-AD prediction tasks. ## Questions?

Email URL@URL with any questions.
TAGS
#transformers #pytorch #arxiv-1901.08746 #arxiv-2212.06042 #endpoints_compatible #region-us 
# Model Card for Model ID



This model card describes the AD-BERT model, which was initialized from Bio+ClinicalBERT trained on NWEDW AD-related progress notes.## Model Details## Pretraining Data
The 'AD-BERT' model was trained on AD related progress notes from NMEDW (private).## Model Pretraining### Note Preprocessing
We preprocessed each progress note derived from the EHRs as follows:
(1)	Deidentification: Clinical notes contain legally Protected Health Information (PHI), such as patient names, addresses, and phone numbers, which should not be released to the public and should not be used for most research applications. We use the package Philter56 to remove PHI for clinical notes.
(2)	Cleaning: We removed non-ASCII characters from the notes and replaced multiple contiguous white spaces with one blank space.  
(3)	Splitting: Each note is split into sections by the newline character (‘\n’). Each section is as an input to the model independently for section representation. And the embedding vector for a patient generated by the MaxPooling of all the section representations.### Pretraining Procedures
The model was trained using code from Hugging Face transformer repository on a Tesla V100 X 32 GB GPU. Model parameters were initialized with Bio+Clinical BERT.### Pretraining code## How to use the model

Load the model via the transformers library:## More Information

Refer to the original paper, AD-BERT: Using Pre-trained contextualized embeddings to Predict the Progression from Mild Cognitive Impairment to Alzheimer's Disease  for additional details and performance on MCI-to-AD prediction tasks.## Questions?

Email URL@URL with any questions.