text
Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.
Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.
new_line
new_line
text
Key aspects of Lumos include:
Key aspects of Lumos include:
new_line
new_line
text
* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
new_line
text
* STR Components:
* STR Components:
new_line
text
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
new_line
text
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
new_line
text
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.
new_line
new_line
text
Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.
Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2402.08017
paper
2402.08017
https://huggingface.co/papers/2402.08017
Lumos : Empowering Multimodal LLMs with Scene Text Recognition (2402.08017)
new_line
new_line
text
Congrats to the authors for their work!
Congrats to the authors for their work!
Meta Reality Labs has developed Lumos, a system that merges Multimodal Large Language Models (MM-LLMs) with Scene Text Recognition (STR) to boost the efficiency of various tasks such as multimodal question-answering and text summarization.

Key aspects of Lumos include:

* Hybrid Computing: Utilizes a combination of on-device and cloud computing to process inputs, aiming to reduce latency.
* STR Components:
  * Region of Interest (ROI) Detection: Focuses on text-rich areas within images for optimized text extraction.
  * Text Detection and Recognition: Ensures high-quality text recognition within the ROI.
  * Reading Order Reconstruction: Arranges recognized text to mimic natural reading order, essential for context understanding.

Lumos demonstrates significant improvement with 80% accuracy in question-answering benchmarks and a low word error rate.

Paper: https://huggingface.co/papers/2402.08017

Congrats to the authors for their work!
👍
Dlbk ybelkada ajibawa-2023 musfiqdehan osanseviero clem Esquare jjoker999 Mrdaffy bunnycore QuantumResearch merve ioanacroi
🤯
alielfilali01 Dlbk jeffboudier merve
❤️
alielfilali01 Dlbk clem merve
🤝
deepufrk Dlbk merve
/avatars/5c718886ea6262d920e1b2bbf268f3dc.svg
Quantum Research
QuantumResearch
user
https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
Vlad Bogolin
vladbogo
user