 Hey, thank you for joining. This will be just a quick tour through a couple of Hugging Face Spaces applications I created to help with vision language model research. Each week, there's several models that are coming out open that are a fusion of, you know, a visioner and a language model decoder in some slight variances in implementation. And it was just a lot of work to kind of pull this information together. I was trying to summarize it in a paper, but this was, these apps were easier for me. And I still might create a paper, but for now, this is meeting its purpose for me, and hopefully it's helpful for you as well. So this, you know, initial sort of display here, here are some of the key vision language models that are in use. Some of them have been around a little bit older. And this article, as you can tell, it's a couple of months old, which is like 10 years in human years. But some of the things that are missing. MoonDream is here, which is cool. It's a very small vision language model, but I do not see Halegema from Google, and I don't see Microsoft's PHY3 vision model. So, it kind of gets back to the rapidity and how quickly things are changing in the vision language model space. This blog post is excellent from Hugging Face, though, in terms of describing the high-level architecture and it comes down to image encoders, the multimodal projector interface in between, and the text decoder that's implemented by the language model. So that's an important concept as I hop into the app. So in Hugging Face Spaces, there's two apps that are created. One is a Model Explorer and the other one is the Hugging Face Extractor. So the Model Explorer, what it does, it leverages a couple of bugging face capabilities. One is Gradio, which is the user interface that you see here. And the other is the Transformers library, which is, you know, how you can sort of standard way of interacting and interfacing with a model. So that's what I use here to, you know, be able to quickly sort of introspect and see the high level of details of a division language model. et cetera, that this view was the simplest. And also it was very easy to kind of go from this view that's output from, you know, the Transformers library into something more, more usable and professional. But anyway, this was a very quick head start. So here, this model that we're looking at is 53 Vision. And if you think back to what we saw before in terms of the standard layers for a vision language model, you have the vision encoder and then you have the projection layer or the interface, and then you have an LLM that it talks to. So in terms of Microsoft's model it uses a clip architecture which is slightly different in terms of the next one we'll look at from Pellegema's model and let's just leave it at that for now okay and so this is a different model here you go let's just go ahead and you can see it at work so i'm using the examples to drive this and so you know very quickly it's able to introspect the model based on the model definition. And here this is the Google's vision model. It's using Siglet for the vision decoder. It's using like a single linear layer in terms of at the projection or interface layer. And then at the language model layer, it's using Gemma. So often what I do when I'm trying to compare it, I'll put two models side by side on the screen and sort of compare interactively. Just to show the code a little bit, and this is where some of the value came for me in terms of rapidly iterating and kind of understanding what's required to interact with the various vision models. And I mean, if you're trying to thinking of being being plug and play and sort of moving sort of Lego pieces around, this is a very quick way to understand how you can best interact with the model. One of the key discoveries for me in terms of going through and digging in is that, you know, the secret sauce of, you know, what helps you interact with a model is in the config.json based on the standard Transformers architecture. I don't want to belabor the point too much, but those of you that have worked with plugging face models will kind of understand the value of this. You know, most models, particularly non-vision models, you'll be able to load with a standard class. But there were a few that I found that used a specialized class. And in those cases, you know, if the architecture, and this is kind of what was pulled from the config.json, if the architecture was of a specific type, like, you know, if it's a lava versus polygema versus edifex, this is a vision model from Huggingface. That anyway, it, for those that it did seem to have an issue with using the standard class, I was able to just quickly sort of switch it out to use something, um, use a more specialized class to load the model and to be able to view the architecture. The other quick lesson learned from, you know, dealing with the PHY vision model and also just the PHY language models was that by default they're using flash attention and so just by putting this logic here it's able to leverage flash attention appropriately when required so that's a quick sort of walkthrough of the model explorer i think is as you kind of go through you'll kind of understand sort of through, you'll kind of understand sort of what I was trying to do with this in, you know, easily extend it to your specific use case. You know, things change so quickly that for me that one of the essential things that helped me was that, you know, you always get errors and this is kind of like an error panel and this is one of my favorite ones that i wanted to get working but it was throwing an error and it has to do with the specialized class that it declares not being found in the transformers library so for me that's kind of my next sort of thing to troubleshoot but that's one of the other things. In terms of approach, errors are great. You learn lots from them. And in this case, I was able to catch the error and throw it here. The other thing that in terms of Hugging Face Spaces is that it provides an easy mechanism to be able to look at the logs. If it's not your space, you may not be able to see it. So if you clone it locally under your space, you should be able to get this working quite quickly as long as you're having a Hugging Face Pro account. So next, let's hop into the extractor. For me, as I said, some of the key learning in terms of understanding for the Model Explorer, how I can best determine what class do I need to use to be able to leverage the specific vision model or even large language model and so this gave a mechanism for me to like easily search right now it's focusing on space so all right let's take a look at a okay so this is taking a look at a model and now that i know that config.json has a lot of secret sauce you can search okay so in terms of the config.json it calls out right away that the specialized architecture or class to use with you know aligema is this aligema for conditional generation and Pala Gemma is this Pala Gemma for conditional generation. And, you know, anyway, so that's kind of the value of this. Also, one of the things that I use LLMs frequently for is to help do analysis. So even if I'm using like the UI chat GPT version, I can copy this and paste it into chat GPT and say, Hey, introspect this architecture. You know, how can I use it to do X, Y, Z, or, you know, if I copy the extract from two, I can say, Hey, compare these. So there's certain sort of shortcuts I took in terms of focusing on files of a certain size, because like for me, I was primarily interested in small files like config files so that I could use it for these types, this type of analysis easily. So if you want to customize it for your purpose, you can clone the repo and adjust as appropriate. You can clone the repo and adjust as appropriate. And so the other thing that was really powerful, like particularly with Gradio, does some cool things in terms of, you know, ability to create a very quick UI or multimodal use cases. As you can see, some of the icons may be a little bit small. You can use a webcam. You can, I guess, audio is enabled for this one, but it's very easy to do it. And I've found that, you know, finding working prototypes or examples that you like, that you like and then you know doing analysis on the files it is much quicker and easier than reading through and scrolling through a bunch of documentation so this is an example of a great demo that was created by Hugging Face and the Google Palogema team and it's got some really powerful stuff in terms of vision analysis, but certain aspects of it were a little bit more challenging, but code from scratch. And so I will have a clear so we just do this. I'll switch to something else. I'll tell it ask it to detect a bee. So what it's going to do, it's going to take this image, and then in terms of this, what it's doing here is it's identifying and it's going to use a bounding box to indicate the location. So it's still thinking, but yeah. So it's still thinking, but yeah. So here, what it's done, it's actually using the Halley-Gemmell model and another model to sort of overlay this bounding line. And in terms of, you know, being able to segment cats, it's kind of, it's similar. being able to segment cats it's kind of it's similar they take a different method of sort of highlighting and segmenting the cats but in terms of getting quickly to the point of understanding how that's done and the code and the models to do it, you know, that's, that gets to, you know, the key sort of scenario. Like, so what I did here, you know, the basis for some of the examples here is I took this and I cut and pasted it into the app so that it's no fun if you don't see me sort of find it. I mean, it runs so quickly. It's not even using a GPU to do this. But it's going out and it's pulling back the content. And what I have here, this is the app.py. So this is the main sort of logic. It's the Gradio app that's used here. But what you're able to do is you're able to take these examples, these prototypes that others have created, understand the approach that they've used, and then get to understanding the, some of the underlying secondary models used. Because in this case for being able to do the bounding boxes being able to do the sort of the overlay that's used for segment it's all it's here so i'm not going to go into the details but it's more just wanted to kind of go into the use case that you know there is such brilliant and insightful work that were went into building this meaningful demo and you know you can take these lessons learned and you know leverage it to build you know your excellent use cases so huge shout out to a hugging face and the google palette gemma team for sharing this the other quick thing i'll show here is that on the model explorer page here, I've sort of documented some of the things that were so useful for me on my learning journey. And one is the link to the Hugging Face overview of the LM. So that's what I showed right here and then the other is a blog post on ala Gemma that kind of goes in based on use cases you know how they they went about their approach and with the various use cases are meant to address and help address. So my last shout out is to, you know, the Google Model Explorer. This is something that I ran across at the time that I was trying to do the analysis. I couldn't get it to work for the vision language models just based on how they do the model fusion of the, you know, the vision encoder and the language model decoder. But it did work really well for, you know, large language models directly. So anyway, definitely worth following and paying attention to, you know, the evolution of this tool. I think that they, they will do it very well. It's a completely different use case that I had in mind when, um, I created my tools like the model Explorer and the Hugging Face Extractor for discovery analysis. So with that said, I'd like to thank you for your time and enjoy the rest of your day.
Unlocking the Mystery of Open Source VLMs:  Accelerate Your Prototyping with Model Explorer
In this exciting video, I dive deep into the world of Vision Language Models (VLMs) and unveil two innovative applications designed to supercharge your initial analysis and prototyping process. 🚀

🔍 Application Highlights:

Model Explorer: Watch as I showcase the Model Explorer, built from scratch using the powerful Transformers library. This tool allows you to quickly introspect and reverse engineer VLM architectures, offering insights into the implementation choices at each layer—from image encoders to interface/projection and text decoders. 🌐

HF Extractor: Discover the magic of the HF Extractor, a simple but effective tool for extracting the "secret sauce" from top-tier Hugging Face spaces. With this tool, you can enhance your own spaces by leveraging the innovative techniques and configurations used by eminent creators. 🧪

🔧 Why Watch?

- Gain an in-depth understanding of key VLM architectures.
- Learn how to accelerate your VLM prototyping and analysis.
- Get inspired by the innovative approaches and insights shared.
- Don't forget to give a shoutout to the amazing creators who inspire you! 🎉
- Join me on this journey of discovery and innovation. Whether you're a seasoned ML engineer or just starting out, these building blocks will revolutionize the way you approach VLMs.

👉 Watch now and take your Vision Language Model projects to the next level!

Chapters
00:00 Introduction
01:40 High-level Architecture
02:17 Model Explorer Walkthrough
05:00 Model Explorer - quick deep dive into the code
08:37 HF Extractor - Walkthrough
10:49 HF Extractor - using it to understand the building blocks of the PaliGemma demo space
15:27 Google Model Explorer