|
--- |
|
pipeline_tag: image-text-to-text |
|
tags: |
|
- florence2 |
|
- smollm |
|
- custom_code |
|
license: apache-2.0 |
|
--- |
|
## FloSmolV |
|
|
|
A vision model for **Image-text to Text** generation produced by combining [HuggingFaceTB/SmolLM-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM-360M-Instruct) and [microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base). |
|
|
|
The **Florence2-base** models generate texts(captions) from input images significantly faster. This text content can be input for a large language model to |
|
answer questions. **SmolLM-360M** is an excellent model by HuggingFace team to generate rapid text output for input queries. These models are combined together to produce a |
|
Visual Question Answering model which can produce answers from Images. |
|
|
|
## Usage |
|
|
|
Make sure to install the necessary dependencies. |
|
|
|
```bash |
|
pip install -qU transformers accelerate einops bitsandbytes flash_attn timm |
|
``` |
|
```python |
|
# load a free image from pixabay |
|
from PIL import Image |
|
import requests |
|
url = "https://cdn.pixabay.com/photo/2023/11/01/11/15/cable-car-8357178_640.jpg" |
|
img = Image.open(requests.get(url, stream=True).raw) |
|
|
|
# download model |
|
from transformers import AutoModelForCausalLM |
|
model = AutoModelForCausalLM.from_pretrained("dmedhi/flosmolv", trust_remote_code=True).cuda() |
|
model(img, "what is the object in the image?") |
|
``` |
|
|
|
You can find more about the model and configuration script here: https://huggingface.co/dmedhi/flosmolv/tree/main |
