metadata

license: apache-2.0
pipeline_tag: image-text-to-text
base_model:
  - epfl-llm/meditron-7b
  - microsoft/rad-dino
base_model_relation: merge
library_name: transformers
tags:
  - RRG
  - Radiology Report Generation
  - Chest X-ray
  - Multimodal Large Language Models

Libra Model Card

Version: Libra-v1.0

Overview

Libra is a multimodal Large Language Model (LLM) specialized in radiology report generation, particularly chest X-ray interpretations. It can produce detailed Findings sections with temporal comparisons (e.g., comparing a current chest X-ray with prior ones). Libra integrates the following key components:

RAD-DINO: A vision encoder pre-trained on medical imaging datasets for robust feature extraction from chest X-rays.
Meditron-7B: A 7B-parameter large language model (based on Llama-2) specialized in medical text generation.
Temporal Alignment Connector (TAC): A custom adapter that fuses features across multiple time points to enable temporal comparisons.

This model card provides an overview of Libra’s architecture, training methodology, limitations, and recommended usage guidelines.

Paper and Resources

For more detailed information regarding Libra’s methodology, theoretical foundation, and performance benchmarks, please refer to the following resources:

Project Website: Libra v1.0
Paper: arXiv:2411.19378
Code Repository: X-iZhang/Libra (GitHub)

Training Strategy

Libra is trained in a two-stage process:

Temporal Feature Alignment
- Trains TAC to effectively fuse and align features from different time points (current and previous chest X-rays).
- Focuses on capturing notable changes (e.g., appearance or progression of opacities, devices, and lines).
Fine-Tuning for Radiology Report Generation
- The language model part is fine-tuned on a large dataset of paired chest X-ray images and radiology reports.
- Emphasizes the generation of the Findings section, especially incorporating temporal descriptors.

Intended Use

Libra is primarily designed to assist clinical practitioners, researchers, and medical students in generating chest X-ray reports. Key applications include:

Clinical Decision Support: Providing draft findings that can be refined by a radiologist.
Educational Tool: Demonstrating example interpretations and temporal changes for training radiology residents.
Research: Facilitating studies on automated report generation and temporal feature learning in medical imaging.

Important: Outputs should be reviewed by qualified radiologists or medical professionals before final clinical decisions are made.

Limitations and Recommendations

Data Bias: The model’s performance may be less reliable for underrepresented demographics or rare pathologies.
Clinical Oversight: Always involve a medical professional to verify the results—Libra is not a substitute for professional judgment.
Temporal Inaccuracies: Despite TAC’s focus on temporal alignment, subtle or uncommon changes may go unrecognized.
Generalization: Libra’s performance on chest X-ray types or conditions not seen during training may be limited.

Ethical Considerations

Patient Privacy: Ensure the data is fully de-identified and compliant with HIPAA/GDPR (or relevant privacy regulations).
Responsible Use: Deploy Libra’s outputs carefully; they are not guaranteed to be error-free.
Accountability: Users and organizations must assume responsibility for verifying clinical accuracy and safety.

How to Cite ✒️

If you use Libra in academic or research contexts, please cite:

@misc{zhang2024libraleveragingtemporalimages,
      title={Libra: Leveraging Temporal Images for Biomedical Radiology Analysis}, 
      author={Xi Zhang and Zaiqiao Meng and Jake Lever and Edmond S. L. Ho},
      year={2024},
      eprint={2411.19378},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.19378}, 
}

Disclaimer:

This tool is for research and educational purposes only. It is not FDA-approved or CE-marked for clinical use. Users should consult qualified healthcare professionals for any clinical decisions.