text
Explaining the 👑 of zero-shot open-vocabulary object detection: OWLv2 🦉
Explaining the 👑 of zero-shot open-vocabulary object detection: OWLv2 🦉
new_line
text
OWLv2 is scaled version of a model called OWL-ViT, so let's take a look at that first. 📝
OWLv2 is scaled version of a model called OWL-ViT, so let's take a look at that first. 📝
new_line
text
OWLViT is an open vocabulary object detector, meaning, it can detect objects it didn't explicitly see during the training. 👀
OWLViT is an open vocabulary object detector, meaning, it can detect objects it didn't explicitly see during the training. 👀
new_line
text
What's cool is that it can take both image and text queries! This is thanks to how the image and text features aren't fused together.
What's cool is that it can take both image and text queries! This is thanks to how the image and text features aren't fused together.
new_line
new_line
text
Taking a look at the architecture, the authors firstly do contrastive pre-training of a vision and a text encoder (just like CLIP). They take that model, remove the final pooling layer and attach a lightweight classification and box detection head and fine-tune.
Taking a look at the architecture, the authors firstly do contrastive pre-training of a vision and a text encoder (just like CLIP). They take that model, remove the final pooling layer and attach a lightweight classification and box detection head and fine-tune.
new_line
text
During fine-tuning for object detection, they calculate the loss over bipartite matches. Simply put, loss is calculated over the predicted objects against ground truth objects and the goal is to find a perfect match of these two sets where each object is matched to one object in ground truth. 
During fine-tuning for object detection, they calculate the loss over bipartite matches. Simply put, loss is calculated over the predicted objects against ground truth objects and the goal is to find a perfect match of these two sets where each object is matched to one object in ground truth. 
new_line
new_line
text
OWL-ViT is very scalable. You can easily scale most language models or vision-language models because they require no supervision, but this isn't the case for object detection: you still need weak supervision. Moreover, only scaling the encoders creates a bottleneck after a while.
OWL-ViT is very scalable. You can easily scale most language models or vision-language models because they require no supervision, but this isn't the case for object detection: you still need weak supervision. Moreover, only scaling the encoders creates a bottleneck after a while.
new_line
new_line
text
The authors wanted to scale OWL-ViT with more data, so they used OWL-ViT for labelling to train a better detector, "self-train" a new detector on the labels, and fine-tune the model on human-annotated data. 
The authors wanted to scale OWL-ViT with more data, so they used OWL-ViT for labelling to train a better detector, "self-train" a new detector on the labels, and fine-tune the model on human-annotated data. 
new_line
new_line
text
Thanks to this, OWLv2 scaled very well and topped leaderboards on open vocabulary object detection 👑 
Thanks to this, OWLv2 scaled very well and topped leaderboards on open vocabulary object detection 👑 
new_line
text
If you'd like to try it out, I will leave couple of links with apps, notebooks and more in the comments! 🤗  
If you'd like to try it out, I will leave couple of links with apps, notebooks and more in the comments! 🤗  
Explaining the 👑 of zero-shot open-vocabulary object detection: OWLv2 🦉
OWLv2 is scaled version of a model called OWL-ViT, so let's take a look at that first. 📝
OWLViT is an open vocabulary object detector, meaning, it can detect objects it didn't explicitly see during the training. 👀
What's cool is that it can take both image and text queries! This is thanks to how the image and text features aren't fused together.

Taking a look at the architecture, the authors firstly do contrastive pre-training of a vision and a text encoder (just like CLIP). They take that model, remove the final pooling layer and attach a lightweight classification and box detection head and fine-tune.
During fine-tuning for object detection, they calculate the loss over bipartite matches. Simply put, loss is calculated over the predicted objects against ground truth objects and the goal is to find a perfect match of these two sets where each object is matched to one object in ground truth. 

OWL-ViT is very scalable. You can easily scale most language models or vision-language models because they require no supervision, but this isn't the case for object detection: you still need weak supervision. Moreover, only scaling the encoders creates a bottleneck after a while.

The authors wanted to scale OWL-ViT with more data, so they used OWL-ViT for labelling to train a better detector, "self-train" a new detector on the labels, and fine-tune the model on human-annotated data. 

Thanks to this, OWLv2 scaled very well and topped leaderboards on open vocabulary object detection 👑 
If you'd like to try it out, I will leave couple of links with apps, notebooks and more in the comments! 🤗  
❤️
s3nh NeuralNovel joytafty osanseviero not-lain samusenps Skier8402 Erland mlabonne Dlbk taufiqdp Jofthomas clem rreed aidystark johko ayhaidar ceofast
👍
Dlbk clem rreed
https://cdn-avatars.huggingface.co/v1/production/uploads/1648113222875-6141a88b3a0ec78603c9e784.png
Merve Noyan
merve
user