# ZH-CLIP: A Chinese CLIP Model
[![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Spaces-blue)](https://huggingface.co/spaces/thu-ml/ZH-CLIPZH-CLIP)

## Models
You can download **ZH-CLIP** model from [🤗 thu-ml/zh-clip-vit-roberta-large-patch14](https://huggingface.co/thu-ml/zh-clip-vit-roberta-large-patch14). The model structure is shown below:
* Vision encoder network structure is the same as [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14), and initialize with [laion/CLIP-ViT-L-14-laion2B-s32B-b82K](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K).
* Text encoder network struceure is the same as [hfl/chinese-roberta-wwm-ext-large](https://huggingface.co/hfl/chinese-roberta-wwm-ext-large) and initialized.
## Results

#### COCO-CN Retrieval (Official Test Set):
<table>
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="4">Text-to-Image</th>
      <th colspan="4">Image-to-Text</th>
    </tr>
    <tr>
      <th>R@1</th>
      <th>R@5</th>
      <th>R@10</th>
      <th>Mean</th>
      <th>R@1</th>
      <th>R@5</th>
      <th>R@10</th>
      <th>Mean</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Clip-Chinese</td>
      <td>22.60</td>
      <td>50.04</td>
      <td>65.24</td>
      <td>45.96</td>
      <td>22.8</td>
      <td>49.8</td>
      <td>64.1</td>
      <td>45.57</td>
    </tr>
    <tr>
      <td>mclip</td>
      <td>56.51</td>
      <td>83.57</td>
      <td>90.79</td>
      <td>76.95</td>
      <td>59.9</td>
      <td>87.3</td>
      <td>94.1</td>
      <td>80.43</td>
    </tr>
    <tr>
      <td>Taiyi-CLIP</td>
      <td>52.52</td>
      <td>81.10</td>
      <td>89.93</td>
      <td>74.52</td>
      <td>45.80</td>
      <td>75.80</td>
      <td>88.10</td>
      <td>69.90</td>
    </tr>
    <tr>
      <td>CN-CLIP</td>
      <td>64.10</td>
      <td>88.79</td>
      <td>94.40</td>
      <td>82.43</td>
      <td>61.00</td>
      <td>84.40</td>
      <td>93.10</td>
      <td>79.5</td>
    </tr>
    <tr>
      <td>altclip-xlmr-l</td>
      <td>62.87</td>
      <td>87.18</td>
      <td>94.01</td>
      <td>81.35</td>
      <td>63.3</td>
      <td>88.3</td>
      <td>95.3</td>
      <td>82.3</td>
    </tr>
    <tr>
      <td>ZH-CLIP</td>
      <td><strong>68.00</strong></td>
      <td><strong>89.46</strong></td>
      <td><strong>95.44</strong></td>
      <td><strong>84.30</strong></td>
      <td><strong>68.50</strong></td>
      <td><strong>90.10</strong></td>
      <td><strong>96.50</strong></td>
      <td><strong>85.03</strong></td>
    </tr>
  </tbody>
</table>

#### Flickr30K-CN Retrieval (Official Test Set):
<table>
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="4">Text-to-Image</th>
      <th colspan="4">Image-to-Text</th>
    </tr>
    <tr>
      <th>R@1</th>
      <th>R@5</th>
      <th>R@10</th>
      <th>Mean</th>
      <th>R@1</th>
      <th>R@5</th>
      <th>R@10</th>
      <th>Mean</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Clip-Chinese</td>
      <td>17.76</td>
      <td>40.34</td>
      <td>51.88</td>
      <td>36.66</td>
      <td>30.4</td>
      <td>55.30</td>
      <td>67.10</td>
      <td>50.93</td>
    </tr>
    <tr>
      <td>mclip</td>
      <td>62.3</td>
      <td>86.42</td>
      <td>92.58</td>
      <td>80.43</td>
      <td>84.4</td>
      <td>97.3</td>
      <td>98.9</td>
      <td>93.53</td>
    </tr>
    <tr>
      <td>Taiyi-CLIP</td>
      <td>53.5</td>
      <td>80.5</td>
      <td>87.24</td>
      <td>73.75</td>
      <td>65.4</td>
      <td>90.6</td>
      <td>95.7</td>
      <td>83.9</td>
    </tr>
    <tr>
      <td>CN-CLIP</td>
      <td>67.98</td>
      <td>89.54</td>
      <td>94.46</td>
      <td>83.99</td>
      <td>81.2</td>
      <td>96.6</td>
      <td>98.2</td>
      <td>92.0</td>
    </tr>
    <tr>
      <td>altclip-xlmr-l</td>
      <td>69.16</td>
      <td>89.94</td>
      <td><strong>94.5</strong></td>
      <td>84.53</td>
      <td>85.1</td>
      <td><strong>97.7</strong></td>
      <td><strong>99.2</strong></td>
      <td>94.0</td>
    </tr>
    <tr>
      <td>ZH-CLIP</td>
      <td><strong>69.64</strong></td>
      <td><strong>90.14</strong></td>
      <td>94.3</td>
      <td><strong>84.69</strong></td>
      <td><strong>86.6</strong></td>
      <td>97.6</td>
      <td>98.8</td>
      <td><strong>94.33</strong></td>
    </tr>
  </tbody>
</table>


#### Muge Text-to-Image Retrieval (Official Validation Set):
<table>
  <thead>
    <tr>
        <th rowspan="2">Model</th>
        <th colspan="4">Text-to-Image</th>
    </tr>
    <tr>
        <th>R@1</th>
        <th>R@5</th>
        <th>R@10</th>
        <th>Mean</th>
    </tr>
  </thead>
  <tbody>
    <tr>
        <td>Clip-Chinese</td>
        <td>15.06</td>
        <td>34.96</td>
        <td>46.21</td>
        <td>32.08</td>
    </tr>
    <tr>
        <td>mclip</td>
        <td>22.34</td>
        <td>41.15</td>
        <td>50.26</td>
        <td>37.92</td>
    </tr>
    <tr>
        <td>Taiyi-CLIP</td>
        <td>42.09</td>
        <td>67.75</td>
        <td>77.21</td>
        <td>62.35</td>
    </tr>
    <tr>
        <td>cn-clip</td>
        <td>56.25</td>
        <td><strong>79.87</strong></td>
        <td>86.50</td>
        <td>74.21</td>
    </tr>
    <tr>
        <td>altclip-xlmr-l</td>
        <td>29.69</td>
        <td>49.92</td>
        <td>58.87</td>
        <td>46.16</td>
    </tr>
    <tr>
        <td>ZH-CLIP</td>
        <td><strong>56.75</strong></td>
        <td>79.75</td>
        <td><strong>86.66</strong></td>
        <td><strong>74.38</strong></td>
    </tr>
  </tbody>
</table>

#### Zero-shot Image Classification:
<table>
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="11">Zero-shot Classification (ACC1)</th>
    </tr>
    <tr>
      <th>CIFAR10</th>
      <th>CIFAR100</th>
      <th>DTD</th>
      <th>EuroSAT</th>
      <th>FER</th>
      <th>FGVC</th>
      <th>KITTI</th>
      <th>MNIST</th>
      <th>PC</th>
      <th>VOC</th>
      <th>ImageNet</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Clip-Chinese</td>
      <td>86.85</td>
      <td>44.21</td>
      <td>18.40</td>
      <td>34.86</td>
      <td>14.21</td>
      <td>3.87</td>
      <td>32.63</td>
      <td>14.37</td>
      <td>52.49</td>
      <td>67.73</td>
      <td>22.22</td>
    </tr>
    <tr>
      <td>mclip</td>
      <td>92.88</td>
      <td>65.54</td>
      <td>29.57</td>
      <td>46.76</td>
      <td>41.18</td>
      <td>7.20</td>
      <td>23.21</td>
      <td>52.80</td>
      <td>51.64</td>
      <td>77.56</td>
      <td>42.99</td>
    </tr>
    <tr>
      <td>Taiyi-CLIP</td>
      <td>95.62</td>
      <td>73.30</td>
      <td>40.69</td>
      <td><strong>61.62</strong></td>
      <td>36.22</td>
      <td>13.98</td>
      <td><strong>41.21</strong></td>
      <td><strong>73.91</strong></td>
      <td>50.02</td>
      <td>75.28</td>
      <td>49.82</td>
    </tr>
    <tr>
      <td>CN-CLIP</td>
      <td>94.75</td>
      <td>75.04</td>
      <td>44.73</td>
      <td>52.34</td>
      <td>48.57</td>
      <td>20.55</td>
      <td>20.11</td>
      <td>61.99</td>
      <td><strong>62.59</strong></td>
      <td><strong>79.12</strong></td>
      <td>53.40</td>
    </tr>
    <tr>
      <td>Altclip-xlmr-l</td>
      <td>95.49</td>
      <td>77.29</td>
      <td>42.07</td>
      <td>56.96</td>
      <td><strong>51.52</strong></td>
      <td><strong>26.85</strong></td>
      <td>24.89</td>
      <td>65.68</td>
      <td>50.02</td>
      <td>77.99</td>
      <td><strong>59.21</strong></td>
    </tr>
    <tr>
      <td>ZH-CLIP</td>
      <td><strong>97.08</strong></td>
      <td><strong>80.73</strong></td>
      <td><strong>47.66</strong></td>
      <td>51.58</td>
      <td>48.48</td>
      <td>20.73</td>
      <td>20.11</td>
      <td>61.94</td>
      <td>62.31</td>
      <td>78.07</td>
      <td>56.87</td>
    </tr>
  </tbody>
</table>

## Getting Started
### Dependency
* python >= 3.9
* pip install -r requirements.txt
### Inference
You can clone code from https://github.com/thu-ml/zh-clip
```python
from PIL import Image
import requests
from models.zhclip import ZhCLIPProcessor, ZhCLIPModel  # Code in https://github.com/thu-ml/zh-clip

version = 'thu-ml/zh-clip-vit-roberta-large-patch14'
model = ZhCLIPModel.from_pretrained(version)
processor = ZhCLIPProcessor.from_pretrained(version)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=["一只猫", "一只狗"], images=image, return_tensors="pt", padding=True)

outputs = model(**inputs)
image_features = outputs.image_features
text_features = outputs.text_features
text_probs = (image_features @ text_features.T).softmax(dim=-1)
```
### Other Chinese CLIP Models
In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future.
| # | model | alias |
| :----: | :---------- | :---------- |
| 0 | [ZH-CLIP](https://github.com/thu-ml/zh-clip) | zhclip |
| 1	| [AltCLIP](https://github.com/FlagAI-Open/FlagAI/tree/master/examples/AltCLIP) | altclip |
| 2	| [Chinese-CLIP](https://github.com/OFA-Sys/Chinese-CLIP)	| cnclip |
| 3	| [TaiyiCLIP](https://github.com/IDEA-CCNL/Fengshenbang-LM)	| taiyiclip |
| 4	| [Multilingual-CLIP](https://github.com/FreddeFrallan/Multilingual-CLIP)	| mclip |
| 5	| [CLIP-Chinese](https://github.com/yangjianxin1/CLIP-Chinese)	| clip-chinese |

Usage in [inference.py](https://github.com/thu-ml/zh-clip/blob/main/inference.py)

{"license": "apache-2.0"}
transformers pytorch zhclip license:apache-2.0 endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #zhclip #license-apache-2.0 #endpoints_compatible #has_space #region-us 

ZH-CLIP: A Chinese CLIP Model
=============================


![Hugging Face Spaces](URL


Models
------


You can download ZH-CLIP model from thu-ml/zh-clip-vit-roberta-large-patch14. The model structure is shown below:


* Vision encoder network structure is the same as openai/clip-vit-large-patch14, and initialize with laion/CLIP-ViT-L-14-laion2B-s32B-b82K.
* Text encoder network struceure is the same as hfl/chinese-roberta-wwm-ext-large and initialized.


Results
-------


#### COCO-CN Retrieval (Official Test Set):



#### Flickr30K-CN Retrieval (Official Test Set):



#### Muge Text-to-Image Retrieval (Official Validation Set):



#### Zero-shot Image Classification:



Getting Started
---------------


### Dependency


* python >= 3.9
* pip install -r URL


### Inference


You can clone code from URL


### Other Chinese CLIP Models


In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future.



Usage in URL



#### COCO-CN Retrieval (Official Test Set): #### Flickr30K-CN Retrieval (Official Test Set): #### Muge Text-to-Image Retrieval (Official Validation Set): #### Zero-shot Image Classification:



Getting Started
--------------- ### Dependency


* python >= 3.9
* pip install -r URL ### Inference


You can clone code from URL ### Other Chinese CLIP Models


In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future.



Usage in URL
TAGS
#transformers #pytorch #zhclip #license-apache-2.0 #endpoints_compatible #has_space #region-us 
 #### COCO-CN Retrieval (Official Test Set): #### Flickr30K-CN Retrieval (Official Test Set): #### Muge Text-to-Image Retrieval (Official Validation Set): #### Zero-shot Image Classification:



Getting Started
--------------- ### Dependency


* python >= 3.9
* pip install -r URL ### Inference


You can clone code from URL ### Other Chinese CLIP Models


In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future.



Usage in URL
TAGS
#transformers #pytorch #zhclip #license-apache-2.0 #endpoints_compatible #has_space #region-us 
#### COCO-CN Retrieval (Official Test Set):#### Flickr30K-CN Retrieval (Official Test Set):#### Muge Text-to-Image Retrieval (Official Validation Set):#### Zero-shot Image Classification:



Getting Started
---------------### Dependency


* python >= 3.9
* pip install -r URL### Inference


You can clone code from URL### Other Chinese CLIP Models


In addition, to compare the effectiveness of different methods, the inference methods of other Chinese CLIP models have been integrated. For the convenience of use, the inference code has also been made public, and please contact us if there is any infringement. The code only implements models at the same level as clip-vit-large-patch14, but it may be adapted for the use of more different versions of models in the future.



Usage in URL