Hugging Face's logo

eltorio
/
IDEFICS3_ROCO

Image-Text-to-Text
PEFT
Safetensors
English
Model card Files Community
IDEFICS3_ROCO / README.md
æLtorio
add preload dataset in Dockerfile
2b6a5b3 unverified
preview code
|
raw
history blame
4.47 kB
metadata
license: apache-2.0
datasets:
  - eltorio/ROCO-radiology
language:
  - en
  - fr
base_model:
  - HuggingFaceM4/Idefics3-8B-Llama3
pipeline_tag: image-to-text

IDEFICS3_ROCO

StageLicenseContributors WelcomeOpen In Colab

A Fine-tuned Radiology-focused Model based on Hugging Face's Idefics3 Model

This repository contains a fine-tuned version of the Hugging Face Idefics3-8B-Llama3 model, built on top of the Meta Llama 3.1 8B architecture. Our model, IDEFICS3_ROCO, has been fine-tuned on the Radiology Objects in Context (ROCO) dataset, a large-scale medical and multimodal imaging collection.

Model Information

  • Base Model: Idefics3-8B-Llama3
  • Fine-tuning Dataset: Radiology Objects in Context (ROCO)
  • License: Apache-2.0
  • Current Status: Fine-tuning process is currently halted at checkpoint 2350 (out of 12,267) (in branch bug-restart) due to limitations with Colab Free T4 GPU unit. Contributions to complete the fine-tuning process are welcome!

Training Progress Status

  • Current checkpoint: 2350/12267 (~19% completed) (in branch bug-restart)
  • Estimated remaining GPU time: ~57 hours
  • Hardware requirements: T4 GPU with >16GB VRAM
  • Last update: november, 8th 2024

Fine-tuning Code

The fine-tuning code is available as a Jupyter Notebook in the ROCO-radiology dataset repository on Hugging Face:

The Junyper Notebook Open In Colab contains the code to fine-tune the Idefics3-8B-Llama3 model on the ROCO dataset. The fine-tuning process is currently halted at checkpoint 640 (out of 24,000) due to limitations with Colab Free T4 GPU unit. Contributions to complete the fine-tuning process are welcome!

Contributions Welcome

If you have the resources to complete the fine-tuning process, we would appreciate your contribution. Please fork this repository, finish the fine-tuning process, and submit a pull request with your updates.

Citation

If you use this model in your work, please cite the original Idefics3 model and our fine-tuned model:

Contribution Guide

  1. Technical Requirements

    • Access to powerful GPU (T4, V100, A100 or equivalent)
    • Python environment with PyTorch
    • Disk space: ~100GB

  2. Getting Started

    • Fork the repository
    • Resume from checkpoint 2350/12267 (in branch bug-restart)
    • Follow instructions in ROCO-idefics3.ipynb Open In Colab

  3. Contact

Docker Image

A AI training docker image is available for this model. The image and includes all necessary dependencies to run the fine-tuning process. The image is available on Docker Hub: 

docker run --user=42420:42420 -it sctg/roco-idefics3:latest /start.sh hf_TOKEN

The Dockerfile is available in the IDEFICS_ROCO repository.

Acknowledgments

This work was made possible by the Hugging Face Transformers library and the ROCO-radiology dataset.