text
🔥 New multimodal leaderboard on the hub: ConTextual!
🔥 New multimodal leaderboard on the hub: ConTextual!
new_line
new_line
text
Many situations require models to parse images containing text: maps, web pages, real world pictures, memes, ... 🖼️
Many situations require models to parse images containing text: maps, web pages, real world pictures, memes, ... 🖼️
new_line
text
So how do you evaluate performance on this task?
So how do you evaluate performance on this task?
new_line
new_line
text
The ConTextual team introduced a brand new dataset of instructions and images, to test LMM (large multimodal models) reasoning capabilities, and an associated leaderboard (with a private test set).
The ConTextual team introduced a brand new dataset of instructions and images, to test LMM (large multimodal models) reasoning capabilities, and an associated leaderboard (with a private test set).
new_line
new_line
text
This is super exciting imo because it has the potential to be a good benchmark both for multimodal models and for assistants' vision capabilities, thanks to the instructions in the dataset.
This is super exciting imo because it has the potential to be a good benchmark both for multimodal models and for assistants' vision capabilities, thanks to the instructions in the dataset.
new_line
new_line
text
Congrats to 
Congrats to 
mention
@rohan598
rohan598
text
, 
, 
mention
@hbXNov
hbXNov
text
, 
, 
mention
@kaiweichang
kaiweichang
text
 and 
 and 
mention
@violetpeng
violetpeng
text
 !!
 !!
new_line
new_line
text
Learn more in the blog: 
Learn more in the blog: 
link
https://huggingface.co/blog/leaderboard-contextual
https://huggingface.co/blog/leaderboard-contextual
new_line
text
Leaderboard: 
Leaderboard: 
resource
https://huggingface.co/spaces/ucla-contextual/contextual_leaderboard
space
ucla-contextual/contextual_leaderboard
https://huggingface.co/spaces/ucla-contextual/contextual_leaderboard
new_line
🔥 New multimodal leaderboard on the hub: ConTextual!

Many situations require models to parse images containing text: maps, web pages, real world pictures, memes, ... 🖼️
So how do you evaluate performance on this task?

The ConTextual team introduced a brand new dataset of instructions and images, to test LMM (large multimodal models) reasoning capabilities, and an associated leaderboard (with a private test set).

This is super exciting imo because it has the potential to be a good benchmark both for multimodal models and for assistants' vision capabilities, thanks to the instructions in the dataset.

Congrats to @rohan598, @hbXNov, @kaiweichang and @violetpeng !!

Learn more in the blog: https://huggingface.co/blog/leaderboard-contextual
Leaderboard: https://huggingface.co/spaces/ucla-contextual/contextual_leaderboard

👍
vladbogo BrigitteTousi Kukedlc osanseviero victor VictorSanh samusenps JasonZJ mvaloatto rohan598 puneetjindalisb whitew1994