File size: 3,701 Bytes
2472fa2 dd33d56 2472fa2 dd33d56 2472fa2 dd33d56 4225cab dd33d56 4225cab dd33d56 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
---
tags:
- clip
- e-commerce
- fashion
- multimodal retrieval
- siglip
library_name: open_clip
pipeline_tag: zero-shot-image-classification
license: apache-2.0
datasets:
- Marqo/atlas
- Marqo/deepfashion-inshop
- Marqo/deepfashion-multimodal
- Marqo/fashion200k
- Marqo/iMaterialist
- Marqo/KAGL
- Marqo/polyvore
language:
- en
metrics:
- precision
- recall
- MRR
---
# Marqo FashionSigLIP Model Card
Marqo-FashionSigLIP leverages Generalised Contrastive Learning ([GCL](https://www.marqo.ai/blog/generalized-contrastive-learning-for-multi-modal-retrieval-and-ranking)) which allows the model to be trained on not just text descriptions but also categories, style, colors, materials, keywords and fine-details to provide highly relevant search results on fashion products.
The model was fine-tuned from ViT-B-16-SigLIP (webli).
**Github Page**: [Marqo-FashionCLIP](https://github.com/marqo-ai/marqo-FashionCLIP)
## Usage
The model can be seamlessly used with [OpenCLIP](https://github.com/mlfoundations/open_clip) by
```python
import open_clip
model, _, _ = open_clip.create_model_and_transforms('hf-hub:Marqo/marqo-fashionSigLIP')
_, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('ViT-B-16-SigLIP', 'webli')
tokenizer = open_clip.get_tokenizer('hf-hub:Marqo/marqo-fashionSigLIP')
```
## Benchmark Results
Average evaluation results on 6 public multimodal fashion datasets ([Atlas](https://huggingface.co/datasets/Marqo/atlas), [DeepFashion (In-shop)](https://huggingface.co/datasets/Marqo/deepfashion-inshop), [DeepFashion (Multimodal)](https://huggingface.co/datasets/Marqo/deepfashion-multimodal), [Fashion200k](https://huggingface.co/datasets/Marqo/fashion200k), [KAGL](https://huggingface.co/datasets/Marqo/KAGL), and [Polyvore](https://huggingface.co/datasets/Marqo/polyvore)) are reported below:
**Text-To-Image (Averaged across 6 datasets)**
| Model | AvgRecall | Recall@1 | Recall@10 | MRR |
|----------------------------|-------------|------------|-------------|-----------|
| FashionCLIP2.0 | 0.163 | 0.077 | 0.249 | 0.165 |
| Marqo-FashionSigLIP | **0.231** | **0.121** | **0.340** | **0.239** |
| OpenFashionCLIP | 0.132 | 0.060 | 0.204 | 0.135 |
| ViT-B-16-laion2b_s34b_b88k | 0.174 | 0.088 | 0.261 | 0.180 |
| ViT-B-16-SigLIP-webli | 0.212 | 0.111 | 0.314 | 0.214 |
**Category-To-Product (Averaged across 5 datasets)**
| Model | AvgP | P@1 | P@10 | MRR |
|----------------------------|-----------|-----------|-----------|-----------|
| FashionCLIP2.0 | 0.684 | 0.681 | 0.686 | 0.741 |
| Marqo-FashionSigLIP | **0.737** | **0.758** | **0.716** | **0.812** |
| OpenFashionCLIP | 0.646 | 0.653 | 0.639 | 0.720 |
| ViT-B-16-laion2b_s34b_b88k | 0.662 | 0.673 | 0.652 | 0.743 |
| ViT-B-16-SigLIP-webli | 0.688 | 0.690 | 0.685 | 0.751 |
**Sub-Category-To-Product (Averaged across 4 datasets)**
| Model | AvgP | P@1 | P@10 | MRR |
|----------------------------|-----------|-----------|-----------|-----------|
| FashionCLIP2.0 | 0.657 | 0.676 | 0.638 | 0.733 |
| Marqo-FashionSigLIP | **0.725** | **0.767** | **0.683** | **0.811** |
| OpenFashionCLIP | 0.598 | 0.619 | 0.578 | 0.689 |
| ViT-B-16-laion2b_s34b_b88k | 0.638 | 0.651 | 0.624 | 0.712 |
| ViT-B-16-SigLIP-webli | 0.643 | 0.643 | 0.643 | 0.726 | |