Converting CLIP to CoreML
Hi, I’d like to convert CLIP model to CoreML but got some errors. Has anyone managed to do this? If someone can help me go in the right direction it would be great.I thought of first converting the vision model. Here’s my code so far:import coremltools as ct
import torch
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained(model_version)
visual_model = model.vision_model
visual_model.eval()


# Trace the model with random data.
example_input_image = torch.rand(1, 3, 224, 224)

traced_model = torch.jit.trace(visual_model, example_input_image)
out = traced_model(example_input_image)There’s a couple of issues:RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module’s inputs. Consider using a constant container instead (e.g. forlist, use atupleinstead. fordict, use aNamedTupleinstead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.Warning:site-packages/transformers/models/clip/modeling_clip.py:222: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):Warning:/site-packages/transformers/models/clip/modeling_clip.py:262: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):
2023-02-08T21:04:32Z
I’ll tag@pcuenqhere
2023-02-08T23:57:47Z
Thanks@radames! And hello,@alkibijadRegarding the first issue, you might avoid getting adictin the output by passingreturn_dict = Falseto the invocation. I’m not sure this will be enough, as I think the default for the model is to return atuple, seethe documentation here.The other warnings are frequent when tracing or compiling models. They are usually fine, as long as you use the model (in production) with inputs of the same shape as the one you used for tracing. The warning means that if you use a different shape, then the code path might be different and the traced code may no longer be correct.I hope these are enough to get you started, please let us know how it goes!
2023-02-15T13:20:17Z
Thanks for the initial guidance@pcuenq&@radames! I definitely needed to read the docs a bit more, so I went through bothCLIP’s docsand theCoreML docs.I’ve managed to:trace the model. The two warnings are still present, but shouldn’t have any impact.Convert the model to CoreMLRun inference on the CoreML modelThe standard and traced model produce exactly the same output, as they should. But the convertedCoreML model produces slightly differentoutput.The conversion to CoreML yields no errors and warnings:Converting PyTorch Frontend ==> MIL Ops: 100%|█████████▉| 696/697 [00:00<00:00, 4630.25 ops/s]
Running MIL Common passes: 100%|██████████| 40/40 [00:00<00:00, 96.36 passes/s]
Running MIL Clean up passes: 100%|██████████| 11/11 [00:00<00:00, 56.34 passes/s]
Translating MIL ==> NeuralNetwork Ops: 100%|██████████| 838/838 [00:54<00:00, 15.37 ops/s]When I pass the exact same image through the models, the CoreML has different results. Here’s how my code looks (simplified a bit):# Load CLIP
processor = CLIPProcessor.from_pretrained(model_version)
model_pt = CLIPModel.from_pretrained(model_version)
model_pt.eval()

# Trace
# wrapped_model -> wrapped CLIPModel so that forward() function returns get_image_features()
example_input = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(model_pt, example_input)

# Convert traced model to CoreML
model_coreml = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input.shape)]
)

# Inference through all 3 models. Convert to numpy for easier comparison
image = ... # Load real image from path
processed_image = processor(text= None, images=[image], return_tensors="pt", padding=True)

res_pt = model_pt.get_image_features(processed_image).numpy()
res_traced = model_traced(processed_image).numpy()
res_coreml = model_coreml.predict({'input_image': processed_image.numpy()})['output_name']


# Compare outputs
print(np.array_equal(res_pt, res_traced)) # True -> standard and traced model produce the same results
print(np.array_equal(res_pt, res_coreml)) # False -> different output

# How close are the outputs?
print(np.allclose(res_pt, res_coreml, atol=1e-5)) # False
print(np.allclose(res_pt, res_coreml, atol=1e-2)) # False
print(np.allclose(res_pt, res_coreml, atol=1e-1)) # True -> Results differ ~0.1Does anyone know what could cause this difference in output? The weights are stored as float32, so there’s no 32 → 16 bit conversion.
2023-02-16T16:58:21Z
Hi@alkibijad! Glad to know that you are making progressRegarding your observation about the outputs, you are right: it is normal that the results are not numerically identical to the ones from PyTorch. Even if the weights are stored in 32-bit format, it doesn’t mean that the model has been converted using 32-bit precision, or runs with it.The default conversion procedure is not guaranteed to happen in float-32, as describedin this documentation. Furthermore, execution precision depends on the hardware yourunyour model on. Another factor that also adds some confusion is the conversionformat(the legacy “Neural Network” format, vs the newer “ML Program”). If you convert to Neural Network, then inference precision will be 32-bit when the model (or portions of it) run on CPU, but 16-bit on GPU and Neural Engine. Using ML Program, you can also run 32-bit precision on GPU (but not on Neural Engine).You can force conversion to happen in 32-bit mode. In ML-Program mode, you can even convert most of your operations using 16-bit precision but preserve some of them in 32-bit mode.To verify the correctness of the conversion, you can first run the result on CPU, selecting the appropriate compute units, and measure the error with respect to the outputs from the original model, but without expecting numerical equivalence.This articlesuggests a signal-to-noise metric to measure the difference. Then you can run predictions incorporating GPU and/or NE, and measure the quality again. It’s usually ok to use conversion defaults and run in 16-bit mode on all eligible devices, but depending on your project (and the model) you might need to override some settings or resort to more exotic things such as per-op precision specification using typed tensors.Let us know how it goes
2023-02-28T20:41:42Z
Thank you@pcuenqit’s nice of you to detail the answer that much! Much useful information in it.The difference in the output has no impact in my use case, so I can say that the CLIP model can be successfully converted to CoreML
2023-04-07T23:17:13Z
@alkibijadThanks for posting your findings while you went through the conversion process!Would you be able to post a snippet of the finished code that you were able to use to do the conversion to coreml? I’m attempting to do the same and have ran into a few issues.For example, if I try converting the model using the code below, I get the following error:# Load CLIP
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")
model_pt = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
model_pt.eval()

# Trace
example_input = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(model_pt, example_input)

# Convert traced model to CoreML
model_coreml = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input.shape)]
)Error:ValueError: You have to specify pixel_values
2023-04-08T03:09:46Z
Made some progress. If I run the below code I can get through the first step.Converting PyTorch Frontend ==> MIL Ops: 100%|█████████▉| 1379/1380 [00:01<00:00, 1155.22 ops/s]# Load CLIP
model_pt = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
visual_model = model_pt.vision_model
visual_model.eval()

example_input_image = torch.rand(1, 3, 224, 224)
model_traced = torch.jit.trace(visual_model, example_input_image, strict=False)

# Convert the traced model to CoreML
coreml_model = ct.convert(
    model_traced,
    inputs=[ct.TensorType(name="input_image", shape=example_input_image.shape)]
)Unfortunately, I get the following error:RuntimeError: PyTorch convert function for op ‘dictconstruct’ not implemented.
2023-04-20T06:05:11Z
I have this same error. Did you figure out how to fix it?
2023-04-28T12:38:17Z
Sorry for late reply. I had issues with converting the vision model only. The hack/solution was to trace the entire CLIP model with only image as input.Also, if you don’t need the output of full CLIP model (which is a dictionary/tuple I think), you can wrap CLIP into your own model, whoseforward passcallsclip_model.get_image_features(x). That way you’re sure to not have any dictionaries that could cause conversion issues.
2023-06-06T19:14:50Z
HeyDid you fix the issue?Cause i faced the same problemThank you
2023-07-12T08:15:38Z
Thank you so much
2023-07-18T09:30:53Z
I found a solution. I entered the source code file of CLIPVisionTransformer and located the “forward” function. I forcefully set the “return_dict” variable to False.
2023-12-12T15:08:48Z
I know this thread is a bit old at this point, but in case you are like me and trying to do this today and struggling to run these code snippets without error, this solution code snippet isso close. A super easy way to accomplish the# wrapped_model -> wrapped CLIPModel so that forward() function returns get_image_features()line in this code snippet is to hack theforwardmethod so that it always calls theget_image_featuresmethod instead, like so:model_pt.forward = lambda *args, **kwargs: model_pt.get_image_features(*args, **kwargs)If you add that, this entire code snippet posted in the solution should work properly.In addition, if you also want to see how the text encoder could be converted to CoreML, I found this repo to be super helpful:GitHub - mazzzystar/Queryable: Run OpenAI's CLIP model on iOS to search photos.