How to count input tokens in vision model?
I used apply_chat_template for text model:tokens = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
tokens_in = len(tokens)But code fails when message have content list:{'role': 'system', 'content': [{'type': 'text', 'text': 'You are assistant'}]}, {'role': 'user', 'content': [{'type': 'text', 'text': '1'}]}]I need to count input tokens for images and text.
2024-06-02T06:39:53Z
Hi! What model you’re using for the chat template?Not all vision LLMs currently support chat template. The one you have as an example should work for Idefics models. In case you want to add a chat template for other VLMs, you can write a jinja template yourself and do as below, wherechat_templateis your own jinja template.tokens = tokenizer.apply_chat_template(messages, chat_template=chat_template, add_generation_prompt=True)For examplehereis the template used for Idefics, but you should make sure to use the chat format that your model was trained on for better performance.Also see thedocson writing you own template
2024-06-02T13:49:11Z
I accidentally use wrong tokenizer, i usehttps://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers/tree/mainTemplate is working, but now i have another issue. For 280x420 and small promt I get 96000 tokens (of course it is more then context, but generation is working). I need to make pricing according amount of tokens but I don’t understand how image processing, I saw that OpenAI have pricing for tiles.