text
Good folks from 
Good folks from 
mention
@Microsoft
Microsoft
text
 have released an exciting breakthrough in GUI automation!
 have released an exciting breakthrough in GUI automation!
new_line
new_line
text
OmniParser – a game-changing approach for pure vision-based GUI agents that works across multiple platforms and applications.
OmniParser – a game-changing approach for pure vision-based GUI agents that works across multiple platforms and applications.
new_line
new_line
text
Key technical innovations:
Key technical innovations:
new_line
text
- Custom-trained interactable icon detection model using 67k screenshots from popular websites
- Custom-trained interactable icon detection model using 67k screenshots from popular websites
new_line
text
- Specialized BLIP-v2 model fine-tuned on 7k icon-description pairs for extracting functional semantics
- Specialized BLIP-v2 model fine-tuned on 7k icon-description pairs for extracting functional semantics
new_line
text
- Novel combination of icon detection, OCR, and semantic understanding to create structured UI representations
- Novel combination of icon detection, OCR, and semantic understanding to create structured UI representations
new_line
new_line
text
The results are impressive:
The results are impressive:
new_line
text
- Outperforms GPT-4V baseline by significant margins on the ScreenSpot benchmark
- Outperforms GPT-4V baseline by significant margins on the ScreenSpot benchmark
new_line
text
- Achieves 73% accuracy on Mind2Web without requiring HTML data
- Achieves 73% accuracy on Mind2Web without requiring HTML data
new_line
text
- Demonstrates a 57.7% success rate on AITW mobile tasks
- Demonstrates a 57.7% success rate on AITW mobile tasks
new_line
new_line
text
What makes OmniParser special is its ability to work across platforms (mobile, desktop, web) using only screenshot data – no HTML or view hierarchy needed. This opens up exciting possibilities for building truly universal GUI automation tools.
What makes OmniParser special is its ability to work across platforms (mobile, desktop, web) using only screenshot data – no HTML or view hierarchy needed. This opens up exciting possibilities for building truly universal GUI automation tools.
new_line
new_line
text
The team has open-sourced both the interactable region detection dataset and icon description dataset to accelerate research in this space.
The team has open-sourced both the interactable region detection dataset and icon description dataset to accelerate research in this space.
new_line
new_line
text
Kudos to the Microsoft Research team for pushing the boundaries of what's possible with pure vision-based GUI understanding!
Kudos to the Microsoft Research team for pushing the boundaries of what's possible with pure vision-based GUI understanding!
new_line
new_line
text
What are your thoughts on vision-based GUI automation?
What are your thoughts on vision-based GUI automation?
Good folks from @Microsoft have released an exciting breakthrough in GUI automation!

OmniParser – a game-changing approach for pure vision-based GUI agents that works across multiple platforms and applications.

Key technical innovations:
- Custom-trained interactable icon detection model using 67k screenshots from popular websites
- Specialized BLIP-v2 model fine-tuned on 7k icon-description pairs for extracting functional semantics
- Novel combination of icon detection, OCR, and semantic understanding to create structured UI representations

The results are impressive:
- Outperforms GPT-4V baseline by significant margins on the ScreenSpot benchmark
- Achieves 73% accuracy on Mind2Web without requiring HTML data
- Demonstrates a 57.7% success rate on AITW mobile tasks

What makes OmniParser special is its ability to work across platforms (mobile, desktop, web) using only screenshot data – no HTML or view hierarchy needed. This opens up exciting possibilities for building truly universal GUI automation tools.

The team has open-sourced both the interactable region detection dataset and icon description dataset to accelerate research in this space.

Kudos to the Microsoft Research team for pushing the boundaries of what's possible with pure vision-based GUI understanding!

What are your thoughts on vision-based GUI automation?
🔥
emanuelbsilva John6666 a9i aleksey-6 AtAndDev JoshOohAhh iky1e juliets
🚀
JoshOohAhh