text
Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
new_line
text
Paper:
Paper:
new_line
link
https://arxiv.org/abs/2404.01911
https://arxiv.org/abs/2404.01911
new_line
text
Model weights and training logs:
Model weights and training logs:
new_line
resource
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b
model
sashakunitsyn/vlrm-blip2-opt-2.7b
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b
new_line
new_line
text
We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
new_line
text
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
new_line
text
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
Paper:
https://arxiv.org/abs/2404.01911
Model weights and training logs:
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b

We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
❤️
samusenps nikgr victor clem osanseviero merve Kukedlc AlkaouDembele xziayro
➕
samusenps merve