|
--- |
|
license: apache-2.0 |
|
pipeline_tag: image-text-to-text |
|
base_model: |
|
- epfl-llm/meditron-7b |
|
- microsoft/rad-dino |
|
base_model_relation: merge |
|
library_name: transformers |
|
tags: |
|
- RRG |
|
- Radiology Report Generation |
|
- Chest X-ray |
|
- Multimodal Large Language Models |
|
--- |
|
<br> |
|
|
|
# **Libra Model Card** |
|
|
|
**Version**: Libra-v1.0 |
|
|
|
## Overview |
|
|
|
**Libra** is a multimodal Large Language Model (LLM) specialized in **radiology report generation**, particularly **chest X-ray** interpretations. It can produce detailed _Findings_ sections with **temporal comparisons** (e.g., comparing a current chest X-ray with prior ones). Libra integrates the following key components: |
|
|
|
- **RAD-DINO**: A vision encoder pre-trained on medical imaging datasets for robust feature extraction from chest X-rays. |
|
- **Meditron-7B**: A 7B-parameter large language model (based on Llama-2) specialized in medical text generation. |
|
- **Temporal Alignment Connector (TAC)**: A custom adapter that fuses features across multiple time points to enable temporal comparisons. |
|
|
|
This model card provides an overview of Libra’s architecture, training methodology, limitations, and recommended usage guidelines. |
|
|
|
--- |
|
|
|
## Paper and Resources |
|
|
|
For more detailed information regarding Libra’s methodology, theoretical foundation, and performance benchmarks, please refer to the following resources: |
|
|
|
- **Project Website**: [Libra v1.0](https://x-izhang.github.io/Libra_v1.0/) |
|
- **Paper**: [arXiv:2411.19378](https://arxiv.org/abs/2411.19378) |
|
- **Code Repository**: [X-iZhang/Libra (GitHub)](https://github.com/X-iZhang/Libra) |
|
|
|
--- |
|
|
|
## Training Strategy |
|
|
|
Libra is trained in a **two-stage process**: |
|
|
|
1. **Temporal Feature Alignment** |
|
- Trains TAC to effectively fuse and align features from different time points (current and previous chest X-rays). |
|
- Focuses on capturing notable changes (e.g., appearance or progression of opacities, devices, and lines). |
|
|
|
2. **Fine-Tuning for Radiology Report Generation** |
|
- The language model part is fine-tuned on a large dataset of paired chest X-ray images and radiology reports. |
|
- Emphasizes the generation of the _Findings_ section, especially incorporating temporal descriptors. |
|
|
|
--- |
|
|
|
## Intended Use |
|
|
|
Libra is primarily designed to **assist** clinical practitioners, researchers, and medical students in generating chest X-ray reports. Key applications include: |
|
|
|
- **Clinical Decision Support**: Providing draft findings that can be refined by a radiologist. |
|
- **Educational Tool**: Demonstrating example interpretations and temporal changes for training radiology residents. |
|
- **Research**: Facilitating studies on automated report generation and temporal feature learning in medical imaging. |
|
|
|
> **Important**: Outputs should be reviewed by qualified radiologists or medical professionals before final clinical decisions are made. |
|
|
|
--- |
|
|
|
## Limitations and Recommendations |
|
|
|
1. **Data Bias**: The model’s performance may be less reliable for underrepresented demographics or rare pathologies. |
|
2. **Clinical Oversight**: Always involve a medical professional to verify the results—Libra is not a substitute for professional judgment. |
|
3. **Temporal Inaccuracies**: Despite TAC’s focus on temporal alignment, subtle or uncommon changes may go unrecognized. |
|
4. **Generalization**: Libra’s performance on chest X-ray types or conditions not seen during training may be limited. |
|
|
|
--- |
|
|
|
## Ethical Considerations |
|
|
|
- **Patient Privacy**: Ensure the data is fully de-identified and compliant with HIPAA/GDPR (or relevant privacy regulations). |
|
- **Responsible Use**: Deploy Libra’s outputs carefully; they are not guaranteed to be error-free. |
|
- **Accountability**: Users and organizations must assume responsibility for verifying clinical accuracy and safety. |
|
|
|
--- |
|
|
|
## How to Cite ✒️ |
|
|
|
If you use Libra in academic or research contexts, please cite: |
|
|
|
```bibtex |
|
@misc{zhang2024libraleveragingtemporalimages, |
|
title={Libra: Leveraging Temporal Images for Biomedical Radiology Analysis}, |
|
author={Xi Zhang and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho}, |
|
year={2024}, |
|
eprint={2411.19378}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV}, |
|
url={https://arxiv.org/abs/2411.19378}, |
|
} |
|
``` |
|
## Disclaimer: |
|
This tool is for research and educational purposes only. It is not FDA-approved or CE-marked for clinical use. Users should consult qualified healthcare professionals for any clinical decisions. |
