text
Can you beat an AI at Raven puzzles?
Can you beat an AI at Raven puzzles?
new_line
new_line
resource
https://huggingface.co/spaces/HuggingFaceM4/ai_raven
space
HuggingFaceM4/ai_raven
https://huggingface.co/spaces/HuggingFaceM4/ai_raven
new_line
new_line
text
The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (
resource
https://huggingface.co/papers/2403.04732
paper
2403.04732
https://huggingface.co/papers/2403.04732
How Far Are We from Intelligent Visual Deductive Reasoning? (2403.04732)
text
).
).
new_line
new_line
text
But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!
new_line
new_line
text
Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.
new_line
new_line
text
Such puzzles can be procedurally generated at scale. 
Such puzzles can be procedurally generated at scale. 
resource
https://huggingface.co/datasets/HuggingFaceM4/RAVEN
dataset
HuggingFaceM4/RAVEN
https://huggingface.co/datasets/HuggingFaceM4/RAVEN
text
 is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
 is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 
new_line
new_line
text
We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.
new_line
new_line
text
Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
Can you beat an AI at Raven puzzles?

https://huggingface.co/spaces/HuggingFaceM4/ai_raven

The most powerful vision+language AI systems like Gemini or GPT4V struggle with this problem when used out-of-the-box (https://huggingface.co/papers/2403.04732).

But when properly trained, a small ~8B model can be very accurate at these IQ tests, solely based on visual inputs!

Raven's Progressive Matrices are visual intelligence tests invented in the 1930s designed to measure abstract reasoning and problem-solving ability. The test consists of a series of matrices or patterns with one part missing. The task for the test-taker is to identify the missing piece from a set of options.

Such puzzles can be procedurally generated at scale. https://huggingface.co/datasets/HuggingFaceM4/RAVEN is one example. The complexity of the puzzles is then controlled by the complexity of the generation procedure. 

We fine-tuned an early checkpoint of our upcoming vision-and-language model idefics2 on that dataset. The resulting checkpoint yields ~91% accuracy! No chain of thoughts, no pre-processing of the image, no additional inputs or metadata, just the RAVEN problem fed to the model as a standalone image (and a short instruction to the model “Which figure should complete the logical sequence?”), with the training objective being the standard cross-entropy.

Just another evidence that in a lot of cases, for a given well-scoped problem, you will be better off paying to collect & annotate data, and fine-tune a model on that data (i.e. build your own AI) than wastefully trying to solve that problem with a gigantic general-purpose model you call through a paid API!
❤️
samusenps osanseviero clefourrier
🔥
yizheapple
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/YurLUFRKxyg3R753mCF_V.jpeg
Kelvin Bruce 
Kelvin19
user