MedClip - Pretraining CLIP on medical data
MedClipAdvancements in computer vision and deep learning techniques carry the potential to make significant contributions to healthcare. The current state-of-the-art models for automated diagnosis and outcome prediction using medical imaging tend not to consider additional information such as medical reports.A multimodal model like CLIP pre-trained in medical data could allow new medical applications that combine text and image.ModelPre-trainedViTandSciBERTmodels can be found on the model hub.DatasetTheMIMIC-CXRdataset can be used for this task. For privacy reasons, the dataset in question has restricted access. Anyone who wants to participate in this project must obtain the necessary credentials to access the dataset.In my experience, getting access to MIMIC-CXR is not particularly complicated, it’s necessary to accept the terms of the license and take a short course on medical data management. It normally takes ~2 weeks to get such credentials.Available training scriptsA training script for this will be provided soon. (seePR)ChallengesCarrying out a proper evaluation of the model may be difficult
2021-06-25T12:19:44Z
I would like to join this project!
2021-06-25T12:27:20Z
@shpotesThat’s an interesting project to work on. I’ve worked on Transformers with MIMIC-CXR database earlier, and I would like to experiment how CLIP fares out.Regarding the database, I believe that one of the team member in a group can get access to the database for working out with it.Lets connect and form a group, if possible, to carry out this project forward.
2021-06-28T13:01:41Z
Hi! I’m very interested in applying NLP to medicine and would love to join this project!
2021-06-28T13:54:55Z
Hi, I am Sweta from India. I am working on deep learning for medical image analysis for my msc thesis, and am generally interested in applications of AI in medicine/healthcare. With this project, I will be able to work on a new dimension, i.e., NLP in healthcare. Hence, I am ver interested in joining this project and working with everyone to hone my NLP skills.My time zone is IST(GMT + 5:30).
2021-06-28T16:24:03Z
That’s a great idea - let’s officially define this project thenPutting everybody in the official sheethere. More people can still join!  Leave a comment here or in the sheet if you want to change something.
2021-06-28T16:35:45Z
I would like to join this project as well!
2021-06-28T16:39:04Z
Awesome! Added you to the team
2021-06-28T16:54:08Z
Hi, may I join as well?
2021-06-28T17:04:25Z
Sure! Just added you the team
2021-06-28T23:36:47Z
vasudev-sharma:I believe that one of the team member in a group can get access to the database for working out with it.According to thedataset license, It is not possible to share the dataset with anyone else (I assume that also includes any participant in the project).
2021-06-29T06:04:17Z
Hey, have been also doing a lot of deep learning in healthcare space, would love to join this!
2021-06-29T14:05:28Z
Hi,I am very interested in this topic as well. I work as Sr. Clinical Data Scientist for Stanford.Keep me posted in how I can contribute
2021-06-29T15:11:47Z
It would be important to see if the data can be used and if so how! Also maybe it might make sense to fine-tune the official CLIP weights on the medical data instead of pretraining from scratch ?@valhalla
2021-06-29T17:37:20Z
I also think that it might make more sense to finetune the officialCLIPweights! Applying for the data might take a few days though so if this project is going to be selected, we might want to take that into account!
2021-06-29T20:36:30Z
patrickvonplaten:It would be important to see if the data can be used and if so how!I think this initiative can qualify as “lawful use in scientific research”, so I don’t think there is any problem. In any case, I can communicate directly with the license owners and ask them about it.Also maybe it might make sense to fine-tune the official CLIP weights on the medical data instead of pretraining from scratch?Considering the amount of data available, fine-tuning will probably work better. The main reason why I proposed to train it from scratch instead of finetuning is the vocabulary. Since the medical vocabulary stands out for its large number of unusual terms in more standard domains with which standard tokenizers tend to have problems (See for instanceBeltagy et al., (2019)).I suppose that techniques such as recycling (de Vries & Nissim, 2021) or Adapters (Houlsby et al., 2019;Pfeiffer et al., 2020), could solve this problem.
2021-06-30T07:38:48Z
hi! I would be very interested in joining this project! I am a ML Engineer at Ferrum Health - a healthcare startup in San Francisco, working on both NLP and computer vision. I am familiar with DICOM and I am currently working with another of the MIMIC datasets (MIMIC-III).
2021-06-30T08:18:29Z
As Patrick said, please see if the data can be made available before the sprint!And regarding fine-tuning and medical vocabulary I think in this case we could maybe use a text encoder which is trained on medical data and then pair it with CLIP’s vision model instead of starting from scratch.
2021-06-30T08:18:49Z
Added you the team@kaushalyaand@edugp
2021-06-30T10:50:55Z
Hey, I find this project really interesting. Recently, I’ve been looking to delve deeper into Clinical NLP. Is there a vacancy?
No date available
No reply text available