text
New foundation model on document understanding and generation in  transformers 🤩
New foundation model on document understanding and generation in  transformers 🤩
new_line
text
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
new_line
text
Demo 👉 
Demo 👉 
resource
https://huggingface.co/spaces/merve/UDOP
space
merve/UDOP
https://huggingface.co/spaces/merve/UDOP
new_line
text
It is a model that combines vision, text and layout. 📝
It is a model that combines vision, text and layout. 📝
new_line
text
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
new_line
text
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
new_line
text
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
new_line
text
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
new_line
text
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
new_line
text
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
new_line
text
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
new_line
text
You can learn more about the model from below resources (h/t to 
You can learn more about the model from below resources (h/t to 
new_line
mention
@nielsr
nielsr
text
), thanks a lot for reading 🤗
), thanks a lot for reading 🤗
new_line
text
Docs: 
Docs: 
link
https://huggingface.co/docs/transformers/main/en/model_doc/udop
https://huggingface.co/docs/transformers/main/en/model_doc/udop
text
 📚 
 📚 
new_line
text
Checkpoints: 
Checkpoints: 
resource
https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
collection
microsoft/udop-65e625124aee97415b88b513
https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513
text
new_line
text
Demo notebooks: 
Demo notebooks: 
link
https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP
https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP
text
 📕
 📕
New foundation model on document understanding and generation in  transformers 🤩
UDOP by MSFT is a bleeding-edge model that is capable of many tasks, including question answering, document editing and more! 🤯
Demo 👉 https://huggingface.co/spaces/merve/UDOP
It is a model that combines vision, text and layout. 📝
This model is very interesting because the input representation truly captures the nature of the document modality: text, where the text is, and the layout of the document matters!
If you know T5, it resembles that: it's pre-trained on both self-supervised and supervised objectives over text, image and layout.
To switch between tasks, one simply needs to change the task specific prompt at the beginning, e.g. for QA, one prepends with Question answering. 
As for the architecture, it's like T5, except it has a single encoder that takes in text, image and layout, and two decoders (text-layout and vision decoders) combined into one.
The vision decoder is a masked autoencoder (thus the capabilities of document editing).
For me, the most interesting capability is document reconstruction, document editing and layout re-arrangement. This decoder isn't released though because it could be used maliciously to fake document editing.
Overall, the model performs very well on document understanding benchmark (DUE) and also information extraction (FUNSD, CORD) and classification (RVL-CDIP) for vision, text, layout modalities.
You can learn more about the model from below resources (h/t to 
@nielsr), thanks a lot for reading 🤗
Docs: https://huggingface.co/docs/transformers/main/en/model_doc/udop 📚 
Checkpoints: https://huggingface.co/collections/microsoft/udop-65e625124aee97415b88b513 
Demo notebooks: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/UDOP 📕
❤️
samusenps rajistics not-lain cstr diwank