
# CLMBR-T-Base

This is a 141 million parameter autoregressive foundation model pretrained on 2.57 million deidentified EHRs from Stanford Medicine. 

This is the model from [(Wornow et al. 2023)](https://arxiv.org/abs/2307.02028), and is based on the CLMBR architecture originally described in [(Steinberg et al. 2021)](https://www.sciencedirect.com/science/article/pii/S1532046420302653)

As input, this model expects a sequence of coded medical events that have been mapped to Standard Concepts within the [OMOP-CDM vocabulary](https://ohdsi.github.io/CommonDataModel/index.html). The model generates representations of patients which can then be used for downstream prediction tasks.

## Model Details

### Model Description

- **Developed by:** Shah lab @ Stanford University
- **Funded by:** Stanford Healthcare
- **Shared by:** Shah lab @ Stanford University
- **Model type:** CLMBR [(Steinberg et al. 2021)](https://www.sciencedirect.com/science/article/pii/S1532046420302653)
- **Language(s) (NLP):** Electronic health record codes
- **License:** CC-BY NC 4.0
- **Finetuned from model:** N/A -- trained from scratch

### Model Sources

- **Website:** [https://ehrshot.stanford.edu/](https://ehrshot.stanford.edu/)
- **Gitub:** [https://github.com/som-shahlab/ehrshot-benchmark/](https://github.com/som-shahlab/ehrshot-benchmark/)
- **Paper:** [EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models](https://arxiv.org/abs/2307.02028)

## Uses

This model is intended to generate representations for patients based on the structured data within their electronic health record. 
These representations can then be used for downstream tasks such as predicting diagnoses, detecting anomalies, or doing propensity score matching for causal inference.

### Direct Use

You will likely want to tune the model for your downstream use case. 

### Out-of-Scope Use

This model is for research purposes only. It is not for use in any real-world decision making that impacts patients, providers, or hospital operations.

## Bias, Risks, and Limitations

This model was trained on a corpus of 2.57 million patients from Stanford Medicine. 
The model will thus reflect the patterns of how care is delivered at Stanford Medicine, in addition to the racial and socioeconomic makeup of Stanford Medicine's patient base. 
This model may not generalize well to other hospitals and demographic mixes. 

While this is technically a generative model, we have not tested its generative abilities and thus do not anticipate it being used to generate synthetic EHR records. 
We aim to explore its generative abilities in future work.

## How to Get Started with the Model

Use the code below to get started with the model.

First, download the necessary libraries.
```bash
# Create Python 3.10 environment
conda create --name ehrshot_env python=3.10 -y
conda activate ehrshot_env

# Install requirements
pip install torch==2.1.1 femr==0.2.0 datasets==2.15.0 flash_attn==2.3.6 transformers==4.35.2
```

Second, run the following Python script to run inference on a single patient:
```python
import femr.models.transformer
import torch
import femr.models.tokenizer
import femr.models.processor
import datetime

model_name = "StanfordShahLab/clmbr-t-base"

# Load tokenizer / batch loader
tokenizer = femr.models.tokenizer.FEMRTokenizer.from_pretrained(model_name)
batch_processor = femr.models.processor.FEMRBatchProcessor(tokenizer)

# Load model
model = femr.models.transformer.FEMRModel.from_pretrained(model_name)

# Create an example patient to run inference on
example_patient = {
    'patient_id': 30,
    'events': [{
        'time': datetime.datetime(2011, 5, 8),
        'measurements': [
            {'code': 'SNOMED/1'},
        ],
    },
    {
        'time': datetime.datetime(2012, 6, 9),
        'measurements': [
            {'code': 'SNOMED/30'},
            {'code': 'SNOMED/103'}
        ],
    }]
}
batch = batch_processor.convert_patient(example_patient, tensor_type="pt")

# Run model
with torch.no_grad():
    patient_ids, times, reprs = model(batch)
    print(patient_ids)
    print(times)
    print(reprs)
```

## Training Details

Full training details are provided in our accompanying paper, [EHRSHOT (Wornow et al. 2023)](https://arxiv.org/abs/2307.02028).

### Training Data

The model is trained on 2.57 million patients from the [Stanford Medicine Research Data Repository (STARR)](https://academic.oup.com/jamiaopen/article/6/3/ooad054/7236015), which contains EHR data from both Stanford Health Care (primarily adult care)
and Lucile Packard Children’s Hospital (primarily pediatric care). 
The dataset contains only structured data (i.e. no clinical text or images) and covers demographics (e.g. age, sex, race), diagnoses, procedures, laboratory results, medication prescriptions, and other coded clinical observations. 
The data is formatted according to the [Observational Medical Outcomes Partnership Common Data Model (OMOP-CDM)](https://ohdsi.github.io/CommonDataModel/cdm53.html). 
All data that we work with is deidentified.

### Training Procedure

We train our model using an autoregressive next code prediction objective, i.e. predict the next code in a patient's timeline given their previous codes.

#### Preprocessing

We use the [FEMR](https://github.com/som-shahlab/femr/tree/main) Python library for data preprocessing.

#### Training Hyperparameters

* Learning rate: 0.00001
* Context window size: 496
* Internal dropout: 0
* Layers: 12
* Hidden dimension: 768

## Evaluation

We evaluate this model on [the EHRSHOT benchmark](https://ehrshot.stanford.edu). 

Information on this benchmark, tasks, and results are detailed in [Wornow et al. 2023](https://arxiv.org/pdf/2307.02028.pdf)

## Technical Specifications

This model uses the CLMBR architecture from [(Steinberg et al. 2021)](https://www.sciencedirect.com/science/article/pii/S1532046420302653). 
The objective is an autoregressive next token prediction task.
Please see [Wornow et al. 2023](https://arxiv.org/pdf/2307.02028.pdf) for more details on the specific model architecture.

## Citation

**BibTeX:**
```
@article{wornow2023ehrshot,
  title={EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models}, 
  author={Michael Wornow and Rahul Thapa and Ethan Steinberg and Jason Fries and Nigam Shah},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}
```
## Model Card Authors

Michael Wornow, Ethan Steinberg, Rahul Thapa, Jason Fries, Nigam H. Shah

## Model Card Contact

Michael Wornow (mwornow@stanford.edu)
{"license": "cc-by-nc-4.0", "library_name": "clmbr", "tags": ["healthcare", "femr", "medical"], "extra_gated_prompt": "You agree to all terms outlined in 'The EHRSHOT Credentialed Health Data License' (see https://shahlab.stanford.edu/ehrshot_license). Access requires a verified CITI training certificate using the same process outlined by PhysioNet (see https://physionet.org/about/citi-course/) Please provide proof via the verification URL, which takes the form https://www.citiprogram.org/verify/?XXXXXX. You agree to not use the model to conduct experiments that cause harm to human subjects.", "extra_gated_fields": {"Full Name": "text", "Email": "text", "Affiliation": "text", "CITI Certification Verification URL": "text", "I agree to all terms outlined in 'The EHRSHOT Credentialed Health Data License'": "checkbox", "I agree to use this model for non-commercial use ONLY": "checkbox"}}
clmbr safetensors healthcare femr medical arxiv:2307.02028 license:cc-by-nc-4.0 region:us