

# Model Card: ClipMD

## Model Details
ClipMD is a medical image-text matching model based on OpenAI's CLIP model with a sliding window text encoder.

### Model Description

The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked sliding window elf-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The model was fine-tuned on the [ROCO dataset](https://github.com/razorx89/roco-dataset).

## Use with Transformers
```
from PIL import Image

from transformers import AutoProcessor,AutoModel

model = AutoModel.from_pretrained("Idan0405/ClipMD",trust_remote_code=True)
processor = AutoProcessor.from_pretrained("Idan0405/ClipMD")

image = Image.open("your image path")

inputs = processor(text=["chest x-ray", "head MRI"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
logits_per_image = outputs[0] # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
```
# See also
* [ClipMD repository on github.](https://github.cs.huji.ac.il/tomhope-lab/ClipMD)
* [ClipMD paper on arxiv](https://arxiv.org/abs/2303.13340)
{"language": ["en"], "tags": ["medical"], "model_type": "clip", "inference": false, "pipeline_tag": "zero-shot-image-classification"}
transformers pytorch clip feature-extraction medical zero-shot-image-classification custom_code en arxiv:2303.13340 doi:10.57967/hf/0898 region:us
TAGS
#transformers #pytorch #clip #feature-extraction #medical #zero-shot-image-classification #custom_code #en #arxiv-2303.13340 #doi-10.57967/hf/0898 #region-us 



# Model Card: ClipMD

## Model Details
ClipMD is a medical image-text matching model based on OpenAI's CLIP model with a sliding window text encoder.

### Model Description

The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked sliding window elf-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The model was fine-tuned on the ROCO dataset.

## Use with Transformers

# See also
* ClipMD repository on github.
* ClipMD paper on arxiv
# Model Card: ClipMD ## Model Details
ClipMD is a medical image-text matching model based on OpenAI's CLIP model with a sliding window text encoder. ### Model Description

The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked sliding window elf-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The model was fine-tuned on the ROCO dataset. ## Use with Transformers # See also
* ClipMD repository on github.
* ClipMD paper on arxiv
TAGS
#transformers #pytorch #clip #feature-extraction #medical #zero-shot-image-classification #custom_code #en #arxiv-2303.13340 #doi-10.57967/hf/0898 #region-us 
 # Model Card: ClipMD ## Model Details
ClipMD is a medical image-text matching model based on OpenAI's CLIP model with a sliding window text encoder. ### Model Description

The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked sliding window elf-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The model was fine-tuned on the ROCO dataset. ## Use with Transformers # See also
* ClipMD repository on github.
* ClipMD paper on arxiv
TAGS
#transformers #pytorch #clip #feature-extraction #medical #zero-shot-image-classification #custom_code #en #arxiv-2303.13340 #doi-10.57967/hf/0898 #region-us 
# Model Card: ClipMD## Model Details
ClipMD is a medical image-text matching model based on OpenAI's CLIP model with a sliding window text encoder.### Model Description

The model uses a ViT-B/32 Transformer architecture as an image encoder and uses a masked sliding window elf-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss.

The model was fine-tuned on the ROCO dataset.## Use with Transformers# See also
* ClipMD repository on github.
* ClipMD paper on arxiv