How to detect the relation between each objects
Hi guys! I’m using Layoulmv3 to extract information from the invoice. There is one thing I’m confused about:When using Layoutlmv3 for the token classification task, I got the output like this:Screenshot 2023-12-25 1027521152×272 100 KB(The red boxes and green boxes indicate the objects with the same class). I have a question: Are there any models to detect the relationship between 1 pair (1 green and 1 red)? For example:These 2 boxes horizontally belong to 1 item.Thank you so much!
2023-12-25T20:55:17Z
Hi,There’s aLayoutLMForRelationExtractionmodel which can be used for that purpose, got a notebook on that here:Transformers-Tutorials/LayoutXLM/Fine_tune_LayoutXLM_on_XFUND_(relation_extraction).ipynb at master · NielsRogge/Transformers-Tutorials · GitHub.However, for relation extraction I’d recommend to instead train a generative model like Donut or Pix2Struct to simply generate key-value pairs. Refer to my demo notebooks here:Transformers-Tutorials/Donut at master · NielsRogge/Transformers-Tutorials · GitHub.
2023-12-26T02:40:54Z
Thank you,@nielsrFor the Relation extraction task, the number of relations is predefined and has a fixed value. In the above image, there are 2 items in total so I understand that I can get 2 “relations” but for another case, I can get 3 “relations” (3 items), and seems like with this approach, you must detect lots of relations. Do you have any recommendations for this case?