Visual Segmentation for Information Extraction from Heterogeneous Visually Rich Documents
Sarkhel, Ritesh
Nandi, Arnab
Physical and digital documents often contain visually rich information. With such information, there is no strict ordering or positioning in the document where the data values must appear. Along with textual cues, these documents often also rely on salient visual features to define distinct semantic boundaries and augment the information they disseminate. When performing information extraction (IE), traditional techniques fall short, as they use a text-only representation and do not consider the visual cues inherent to the layout of these documents. We propose VS2, a generalized approach for information extraction from heterogeneous visually rich documents. There are two major contributions of this work. First, we propose a robust segmentation algorithm that decomposes a visually rich document into a bag of visually isolated but semantically coherent areas, called logical blocks. Document type agnostic low-level visual and semantic features are used in this process. Our second contribution is a distantly supervised search-and-select method for identifying the named entities within these documents by utilizing the context boundaries defined by these logical blocks. Experimental results on three heterogeneous datasets suggest that the proposed approach significantly outperforms its text-only counterparts on all datasets. Comparing it against the state-of-the-art methods also reveal that VS2 performs comparably or better on all datasets.
 This is going to be the outline of my talk. First I'm going to be motivating the importance of the information extraction problem that we solve in this paper. Next I'll be going over some of the limitations of the existing techniques that is there in our current literature and some of our observations that we made during our research that we think gives us a better solution to this problem. Next I will describe our major contributions, two key contributions of this work and demonstrate our performance on a real-world dataset. So we are surrounded by documents that are visually rich in our everyday life and these documents come in various shapes, sizes and forms. Examples include documents such as posters, leaflets, banners, etc. Now although these documents can be varied in terms of their layout format or content, they do have a major commonality that binds them together and qualifies them as a visually rich document, which is their usage of visual cues to highlight the semantic entities that appear in them. Taking a look at these two examples, we see that visual features such as text color, font size, text orientation and positioning, white space symmetry, even negative space sometimes can play an important role in highlighting what a text means and how important it is in the context of the entire document. Now we see that these documents are sometimes very useful information of ad hoc content, which means that the information that these documents contain are not easily available in an indexed database, which is why when presented with a large-scale database of these heterogeneous documents, it would benefit us very much if we can perform some sort of bottom-metered information extraction on these documents, thereby reducing the cost of human effort. Take the example of Alice here, shown in this slide. Alice is an owner of a small event management company in Columbus, Ohio and she wants to perform some market research for her company by extracting two named entities, event organizer and event place from a collection of documents that she has collected for her business. Now in this collection, she has both the scans of physical posters of local events as well as digital event flyers saved as PDFs. Now as the layout and as well as format of these collection is heterogeneous, our objective is to help Alice reduce some of the human effort that she needs to put in to create the database by extracting these two information into her local database by automating some of the tasks. That is the problem that we try to address in this work. Now going over to some of the limitations of the existing techniques, one possible solution that one might think of is getting a transcription of the document and use the existing techniques that we have at our disposal to perform information extraction. However, in our research we saw that this solution is not as straightforward as it seemed to be. As the visual heterogeneity of the document increases, it gets more and more difficult by the transcription engine to capture the semantic role played by the visual features within its transcription and therefore perform well in the downstream extraction task. An example can be shown here where from a human perspective, a human point of view, we can see that the white space separators shown on the document in the left within the red bounding boxes are clearly demarcated by the white space separator and those are two distinct semantic entities that appear in the document. However, when we transcribe this document and take a look at the text that appears on the right within the blue bounding box, these kind of get messed up and we observe that as the visual features in the document play a more important role than the textual cues, these sort of errors creep up very often. We have used Disrect. Another possible solution towards this information extraction problem would be using template based masks as rules and save them in our local database and apply them on a case by case basis. An example of three such rules is shown on the document in the left where each of the bounding boxes represent the named entity that appears in the document and given a test document, any text that appears within the bounding box, let's say labeled as event title, we can say that this is the text that represents the named entity event title in the test document. However, as some of the recent works has also shown that this is a very expensive avenue to trade on. As we are going for exact matches for any test document that we might encounter, we need to be vigilant to update and be ready for any layout that a test document may contain which makes it prohibitively expensive to deploy and maintain in a cost-restraint environment. We approached this information extraction problem from a different angle and we come up with some observations first. We say that a visually rich document irrespective of its format, layout or content can be represented as a bag of isolated but coherent visual elements. Now although the visual heterogeneity of the document itself can be significantly high or low, when we concentrate on each of these visual areas itself, this visual heterogeneity gets significantly reduced. An example can be seen on the left where each of the red bounding boxes represent these coherent visual areas and we can see that the visual features get reduced by a lot when we compare it to the entire document level context. Now our observation number two is that a few of these visual areas also act as interest points in the document, which means that these areas have higher semantic significance than the other documents than the other areas that appear in the document. So based on these observations, we come up with the following hypothesis. We say that a visually rich document irrespective of its layout or format or the content or the information it contains can be represented as a bag of visually isolated coherent areas. And to find these segmented areas, we need to incorporate or consider not only the visual features of the document, but the semantic features as well, underline the word visual here. Now, our second hypothesis is once we have managed to represent a document using this bag of visual areas representation, we can apply a divide and conquer approach of the information extraction task we were given to perform on the document on each of these segmented areas and return the results we get to a high accuracy that we expect. So our proposed information extraction workflow from these documents look like this. First, we get a representation of the document by segmenting it into a bag of isolated areas and then we perform information extraction task by dividing and conquering the named entity recognition task, which is the task that we will demonstrate our performance on each of these segmented areas. Now, moving on to each of these algorithms, both our contributions. First, let's talk about the segmentation algorithm VS2 segment. So the input to this segmentation algorithm is the visual rich document itself and we expect the output of the bag of isolated visual areas that represents the document in a unified way. So the constraint here that we are dealt with is that the visual rich documents may come in various formats. For example, they can be in XML format, HTML format, PDF format or any other format, but we want to be as generalized as we can be. So we are not allowed to use any format specific high level operators to perform this segmentation, which is why in our segmentation algorithm, we kind of limit ourselves to easily generate a visual and semantic features that can be extracted by the rendered image of the visual rich document that is input to us. Now the segmentation algorithm itself is a recursive algorithm, which incrementally breaks down the area, the visual area that's given as input to it into smaller areas and it updates the layout tree that represents the document itself. Now if a visual area that's being input to the segmentation algorithm is broken down into smaller areas, then those four smaller areas will be represented by four nodes in the layout tree and added as a child node of the parent node itself. Now the layout tree is initialized with the root node R representing the entire visual area covered by the document itself. Now as the segmentation algorithm converges, we return all the leaf nodes that we have found of the layout tree back to the user and those visual areas represented by the leaf nodes represent the segmented areas that we set out to get from the segmentation algorithm itself. Now we use both visual features and semantic features from the document to implement this segmentation algorithm and the explicit visual features we have used for this purpose are white space based separators that run across from one edge of the document to the other, cutting across the document in clear visual, as clear visual limiters. And the implicit visual features that we use to cluster the visual, atomic visual elements which are words in our case are chosen empirically and shown as these five features that we have used in our work here. So to kind of recap, so the VST segment algorithm is a recursive algorithm that takes in the rendered image of the document as input and gives us this representation of bag of isolated areas shown on the left where each of the bounding boxes represent a segmented area that we obtain from the algorithm. Now to perform this segmentation algorithm, we need to incorporate both semantic and visual features extracted from the document. Now let's move on to the information extraction algorithm which is why we proposed this segmentation algorithm in the first place. Now let's moving on to VST to select our second contribution. This VST to select algorithm takes in input the bag of visual areas representation of the document and the name identity we want to extract and it outputs the text extracted from the document corresponding to the name identity that we wanted to search for. For every name identity that we want to extract, we search for a syntactic pattern that represents the name identity within each of the segmented visual areas and we rank all the matches in the multimodal encoding space and return the top case. So we kind of follow the paradigm of divide and conquer information extraction algorithm in this workflow and go from there. Now one of the biggest constraints of this name identity recognition algorithm is we want to be robust towards the ad hoc nature of the content of visual rich documents that we may find in the wild which is why we learn the syntactic patterns that represent every name identity and the patterns that we search for within each of the areas using a distance supervision approach. For each appearance of the name identity in the holdout corpus, we find the dependency parse tree, tag them with some publicly available natural language processing tools and then mine for the most frequent sub trees that represent the name identity. Now the pattern that we get from this most frequent sub trees represent the sub name identity that we want to search for when we are trying to perform the information extraction task. Now once we have managed to find, managed to learn the syntactic pattern that we want to find for a name identity and we have also managed to find out which visual areas contain those patterns we want to rank them in a multi-modal encoding space. Now we do this by computing, we come by computing a weighted L1 distance from the closest near interest point in the document from each of the matched patterns. Now the interest point as I told before represents an area in the document that has higher semantic significance than the other areas in the document and we find this optimal subset of areas from the segmented visual areas we obtained from the segmentation algorithm VS2 segment by solving a multi-objective optimization problem. Each objective optimization problem represents a design principle that's used by graphic designers to highlight the semantics or the importance of a visual area within the document. We selected three such objectives in our work and they were selected empirically. The interest points that we found for our document that is for the event poster from Alice's collection is shown on the left where each of the bounding boxes represent an interest point which is a visual area with higher semantic significance in the event poster itself. So to recap, VS2 select leverages the local context boundaries that we obtain from the segmentation algorithm and performs a divide and conquer approach to identify the name identities that we wanted to extract from the document in the first place. And to combat for the ad hoc nature of the visually rich documents that we can find in the wild, the syntactic patterns representing each name identity are learned using distance supervision. Now let's see how we did on a real-world dataset. Due to the interest of time, I will only cover one of the experiments that we performed in our paper. There are detailed analysis and more experiments that can be found by going back to the paper itself. Now let's focus on the first experiment, the first example that I introduced in one of the beginning slides. What happens, how did we do on the real-world event poster dataset where we wanted to extract five distinct name identities for Alice from each of the event documents. Now these event posters or flyers are heterogeneous in nature as in both their layouts are varied and the formats they are saved in are also different from each other. We have two such formats in the entire collection which may contain digital images saved as JPEG files as well as PDFs. Now the name identities we extracted from each of the documents are shown here where you can see that each of the name identities represent an important details that one can expect to extract from an event poster especially the information consumer like Alice. Now to measure the performance of our proposed method, we wanted to find the accuracy of the extracted text by performing a two-phase evaluation paradigm. We say that the extracted text T identifies as name identity N is accurate if the position of the text appearing in the document is accurate as well as the name identity type N it is classified as is accurate also. And to perform these comparisons we compare it against human and energetic ground truth which is shown on the left here and the color legends are also given. So that in the end to end performance we see that we do significantly better against the text-based baselines where all of these visual features and the segmentation step is not considered from, are not considered a priori. We also compare our system against a rule-based method which is commercially available termed as report minor and see that we outform them by a large margin also. By performing a statistical significance test we see that we significantly outperform the text-only baseline by a p-value less than 0.05. So the main takeaways of this work are the following. We show that prior segmentation can help information extraction from visually-rich documents. We have proposed two key algorithms for this purpose, both of which one is called VH2 segment and the other is VH2 select and we believe that these findings opens up a lot of future exciting possibilities where the semantic relationship between the segmented areas can be explored to perform more complex information extraction tasks such as QA summarization, etc. With that my presentation today ends and I will ask for more questions.
 This is going to be the outline of my talk.  First I'm going to be motivating the importance of the information extraction problem that  we solve in this paper.  Next I'll be going over some of the limitations of the existing techniques that is there in  our current literature and some of our observations that we made during our research that we think  gives us a better solution to this problem.  Next I will describe our major contributions, two key contributions of this work and demonstrate  our performance on a real-world dataset.  So we are surrounded by documents that are visually rich in our everyday life and these  documents come in various shapes, sizes and forms.  Examples include documents such as posters, leaflets, banners, etc.  Now although these documents can be varied in terms of their layout format or content,  they do have a major commonality that binds them together and qualifies them as a visually  rich document, which is their usage of visual cues to highlight the semantic entities that  appear in them.  Taking a look at these two examples, we see that visual features such as text color, font  size, text orientation and positioning, white space symmetry, even negative space sometimes  can play an important role in highlighting what a text means and how important it is  in the context of the entire document.  Now we see that these documents are sometimes very useful information of ad hoc content,  which means that the information that these documents contain are not easily available  in an indexed database, which is why when presented with a large-scale database of these  heterogeneous documents, it would benefit us very much if we can perform some sort of  bottom-metered information extraction on these documents, thereby reducing the cost of human  effort.  Take the example of Alice here, shown in this slide.  Alice is an owner of a small event management company in Columbus, Ohio and she wants to  perform some market research for her company by extracting two named entities, event organizer  and event place from a collection of documents that she has collected for her business.  Now in this collection, she has both the scans of physical posters of local events as well  as digital event flyers saved as PDFs.  Now as the layout and as well as format of these collection is heterogeneous, our objective  is to help Alice reduce some of the human effort that she needs to put in to create  the database by extracting these two information into her local database by automating some  of the tasks.  That is the problem that we try to address in this work.  Now going over to some of the limitations of the existing techniques, one possible solution  that one might think of is getting a transcription of the document and use the existing techniques  that we have at our disposal to perform information extraction.  However, in our research we saw that this solution is not as straightforward as it  seemed to be.  As the visual heterogeneity of the document increases, it gets more and more difficult  by the transcription engine to capture the semantic role played by the visual features  within its transcription and therefore perform well in the downstream extraction task.  An example can be shown here where from a human perspective, a human point of view,  we can see that the white space separators shown on the document in the left within  the red bounding boxes are clearly demarcated by the white space separator and those are  two distinct semantic entities that appear in the document.  However, when we transcribe this document and take a look at the text that appears on  the right within the blue bounding box, these kind of get messed up and we observe that  as the visual features in the document play a more important role than the textual cues,  these sort of errors creep up very often.  We have used Disrect.  Another possible solution towards this information extraction problem would be using template  based masks as rules and save them in our local database and apply them on a case by  case basis.  An example of three such rules is shown on the document in the left where each of the  bounding boxes represent the named entity that appears in the document and given a  test document, any text that appears within the bounding box, let's say labeled as event  title, we can say that this is the text that represents the named entity event title in  the test document.  However, as some of the recent works has also shown that this is a very expensive avenue  to trade on.  As we are going for exact matches for any test document that we might encounter, we  need to be vigilant to update and be ready for any layout that a test document may contain  which makes it prohibitively expensive to deploy and maintain in a cost-restraint environment.  We approached this information extraction problem from a different angle and we come  up with some observations first.  We say that a visually rich document irrespective of its format, layout or content can be represented  as a bag of isolated but coherent visual elements.  Now although the visual heterogeneity of the document itself can be significantly high  or low, when we concentrate on each of these visual areas itself, this visual heterogeneity  gets significantly reduced.  An example can be seen on the left where each of the red bounding boxes represent these  coherent visual areas and we can see that the visual features get reduced by a lot when  we compare it to the entire document level context.  Now our observation number two is that a few of these visual areas also act as interest  points in the document, which means that these areas have higher semantic significance than  the other documents than the other areas that appear in the document.  So based on these observations, we come up with the following hypothesis.  We say that a visually rich document irrespective of its layout or format or the content or  the information it contains can be represented as a bag of visually isolated coherent areas.  And to find these segmented areas, we need to incorporate or consider not only the visual  features of the document, but the semantic features as well, underline the word visual  here.  Now, our second hypothesis is once we have managed to represent a document using this  bag of visual areas representation, we can apply a divide and conquer approach of the  information extraction task we were given to perform on the document on each of these  segmented areas and return the results we get to a high accuracy that we expect.  So our proposed information extraction workflow from these documents look like this.  First, we get a representation of the document by segmenting it into a bag of isolated areas  and then we perform information extraction task by dividing and conquering the named  entity recognition task, which is the task that we will demonstrate our performance on  each of these segmented areas.  Now, moving on to each of these algorithms, both our contributions.  First, let's talk about the segmentation algorithm VS2 segment.  So the input to this segmentation algorithm is the visual rich document itself and we  expect the output of the bag of isolated visual areas that represents the document in a unified  way.  So the constraint here that we are dealt with is that the visual rich documents may come  in various formats.  For example, they can be in XML format, HTML format, PDF format or any other format, but  we want to be as generalized as we can be.  So we are not allowed to use any format specific high level operators to perform this segmentation,  which is why in our segmentation algorithm, we kind of limit ourselves to easily generate  a visual and semantic features that can be extracted by the rendered image of the visual  rich document that is input to us.  Now the segmentation algorithm itself is a recursive algorithm, which incrementally breaks  down the area, the visual area that's given as input to it into smaller areas and it updates  the layout tree that represents the document itself.  Now if a visual area that's being input to the segmentation algorithm is broken down  into smaller areas, then those four smaller areas will be represented by four nodes in  the layout tree and added as a child node of the parent node itself.  Now the layout tree is initialized with the root node R representing the entire visual  area covered by the document itself.  Now as the segmentation algorithm converges, we return all the leaf nodes that we have  found of the layout tree back to the user and those visual areas represented by the  leaf nodes represent the segmented areas that we set out to get from the segmentation algorithm  itself.  Now we use both visual features and semantic features from the document to implement this  segmentation algorithm and the explicit visual features we have used for this purpose are  white space based separators that run across from one edge of the document to the other,  cutting across the document in clear visual, as clear visual limiters.  And the implicit visual features that we use to cluster the visual, atomic visual elements  which are words in our case are chosen empirically and shown as these five features that we have  used in our work here.  So to kind of recap, so the VST segment algorithm is a recursive algorithm that takes in the  rendered image of the document as input and gives us this representation of bag of isolated  areas shown on the left where each of the bounding boxes represent a segmented area  that we obtain from the algorithm.  Now to perform this segmentation algorithm, we need to incorporate both semantic and visual  features extracted from the document.  Now let's move on to the information extraction algorithm which is why we proposed this segmentation  algorithm in the first place.  Now let's moving on to VST to select our second contribution.  This VST to select algorithm takes in input the bag of visual areas representation of  the document and the name identity we want to extract and it outputs the text extracted  from the document corresponding to the name identity that we wanted to search for.  For every name identity that we want to extract, we search for a syntactic pattern that represents  the name identity within each of the segmented visual areas and we rank all the matches in  the multimodal encoding space and return the top case.  So we kind of follow the paradigm of divide and conquer information extraction algorithm  in this workflow and go from there.  Now one of the biggest constraints of this name identity recognition algorithm is we  want to be robust towards the ad hoc nature of the content of visual rich documents that  we may find in the wild which is why we learn the syntactic patterns that represent every  name identity and the patterns that we search for within each of the areas using a distance  supervision approach.  For each appearance of the name identity in the holdout corpus, we find the dependency  parse tree, tag them with some publicly available natural language processing tools and then  mine for the most frequent sub trees that represent the name identity.  Now the pattern that we get from this most frequent sub trees represent the sub name  identity that we want to search for when we are trying to perform the information extraction  task.  Now once we have managed to find, managed to learn the syntactic pattern that we want  to find for a name identity and we have also managed to find out which visual areas contain  those patterns we want to rank them in a multi-modal encoding space.  Now we do this by computing, we come by computing a weighted L1 distance from the closest near  interest point in the document from each of the matched patterns.  Now the interest point as I told before represents an area in the document that has higher semantic  significance than the other areas in the document and we find this optimal subset of areas from  the segmented visual areas we obtained from the segmentation algorithm VS2 segment by  solving a multi-objective optimization problem.  Each objective optimization problem represents a design principle that's used by graphic  designers to highlight the semantics or the importance of a visual area within the document.  We selected three such objectives in our work and they were selected empirically.  The interest points that we found for our document that is for the event poster from  Alice's collection is shown on the left where each of the bounding boxes represent an interest  point which is a visual area with higher semantic significance in the event poster itself.  So to recap, VS2 select leverages the local context boundaries that we obtain from the  segmentation algorithm and performs a divide and conquer approach to identify the name  identities that we wanted to extract from the document in the first place.  And to combat for the ad hoc nature of the visually rich documents that we can find  in the wild, the syntactic patterns representing each name identity are learned using distance  supervision.  Now let's see how we did on a real-world dataset.  Due to the interest of time, I will only cover one of the experiments that we performed in  our paper.  There are detailed analysis and more experiments that can be found by going back to the paper  itself.  Now let's focus on the first experiment, the first example that I introduced in one of  the beginning slides.  What happens, how did we do on the real-world event poster dataset where we wanted to extract  five distinct name identities for Alice from each of the event documents.  Now these event posters or flyers are heterogeneous in nature as in both their layouts are varied  and the formats they are saved in are also different from each other.  We have two such formats in the entire collection which may contain digital images saved as  JPEG files as well as PDFs.  Now the name identities we extracted from each of the documents are shown here where  you can see that each of the name identities represent an important details that one can  expect to extract from an event poster especially the information consumer like Alice.  Now to measure the performance of our proposed method, we wanted to find the accuracy of  the extracted text by performing a two-phase evaluation paradigm.  We say that the extracted text T identifies as name identity N is accurate if the position  of the text appearing in the document is accurate as well as the name identity type  N it is classified as is accurate also.  And to perform these comparisons we compare it against human and energetic ground truth  which is shown on the left here and the color legends are also given.  So that in the end to end performance we see that we do significantly better against the  text-based baselines where all of these visual features and the segmentation step is not  considered from, are not considered a priori.  We also compare our system against a rule-based method which is commercially available termed  as report minor and see that we outform them by a large margin also.  By performing a statistical significance test we see that we significantly outperform the  text-only baseline by a p-value less than 0.05.  So the main takeaways of this work are the following.  We show that prior segmentation can help information extraction from visually-rich documents.  We have proposed two key algorithms for this purpose, both of which one is called VH2 segment  and the other is VH2 select and we believe that these findings opens up a lot of future  exciting possibilities where the semantic relationship between the segmented areas can  be explored to perform more complex information extraction tasks such as QA summarization,  etc.  With that my presentation today ends and I will ask for more questions.
Sarkhel_Nandl_0_131.png Sarkhel_Nandl_132_959.png Sarkhel_Nandl_960_1391.png Sarkhel_Nandl_1392_1907.png Sarkhel_Nandl_1908_2039.png Sarkhel_Nandl_2040_2447.png Sarkhel_Nandl_2448_4811.png Sarkhel_Nandl_4812_8735.png Sarkhel_Nandl_8736_10355.png Sarkhel_Nandl_10356_11771.png Sarkhel_Nandl_11772_12419.png Sarkhel_Nandl_12420_12527.png Sarkhel_Nandl_12528_12575.png Sarkhel_Nandl_12576_14051.png Sarkhel_Nandl_14052_15971.png Sarkhel_Nandl_15972_16703.png Sarkhel_Nandl_16704_17555.png Sarkhel_Nandl_17556_17639.png Sarkhel_Nandl_17640_19955.png Sarkhel_Nandl_19956_22019.png Sarkhel_Nandl_22020_22751.png Sarkhel_Nandl_22752_22883.png Sarkhel_Nandl_22884_24131.png Sarkhel_Nandl_24132_25607.png Sarkhel_Nandl_25608_26243.png Sarkhel_Nandl_26244_26579.png Sarkhel_Nandl_26580_27455.png Sarkhel_Nandl_27456_27515.png Sarkhel_Nandl_27516_27563.png Sarkhel_Nandl_27564_27693.png
This is going to be the outline of my talk. First I'm going to be motivating the importance of the information extraction problem that we solve in this paper. Next I'll be going over some of the limitations of the existing techniques that is there in our current literature and some of our observations that we made during our research that we think gives us a better solution to this problem. Next I will describe our major contributions, two key contributions of this work and demonstrate our performance on a real-world dataset. So we are surrounded by documents that are visually rich in our everyday life and these documents come in various shapes, sizes and forms. Examples include documents such as posters, leaflets, banners, etc. Now although these documents can be varied in terms of their layout format or content, they do have a major commonality that binds them together and qualifies them as a visually rich document, which is their usage of visual cues to highlight the semantic entities that appear in them. Taking a look at these two examples, we see that visual features such as text color, font size, text orientation and positioning, white space symmetry, even negative space sometimes can play an important role in highlighting what a text means and how important it is in the context of the entire document. Now we see that these documents are sometimes very useful information of ad hoc content, which means that the information that these documents contain are not easily available in an indexed database, which is why when presented with a large-scale database of these heterogeneous documents, it would benefit us very much if we can perform some sort of bottom-metered information extraction on these documents, thereby reducing the cost of human effort. Take the example of Alice here, shown in this slide. Alice is an owner of a small event management company in Columbus, Ohio and she wants to perform some market research for her company by extracting two named entities, event organizer and event place from a collection of documents that she has collected for her business. Now in this collection, she has both the scans of physical posters of local events as well as digital event flyers saved as PDFs. Now as the layout and as well as format of these collection is heterogeneous, our objective is to help Alice reduce some of the human effort that she needs to put in to create the database by extracting these two information into her local database by automating some of the tasks. That is the problem that we try to address in this work. Now going over to some of the limitations of the existing techniques, one possible solution that one might think of is getting a transcription of the document and use the existing techniques that we have at our disposal to perform information extraction. However, in our research we saw that this solution is not as straightforward as it seemed to be. As the visual heterogeneity of the document increases, it gets more and more difficult by the transcription engine to capture the semantic role played by the visual features within its transcription and therefore perform well in the downstream extraction task. An example can be shown here where from a human perspective, a human point of view, we can see that the white space separators shown on the document in the left within the red bounding boxes are clearly demarcated by the white space separator and those are two distinct semantic entities that appear in the document. However, when we transcribe this document and take a look at the text that appears on the right within the blue bounding box, these kind of get messed up and we observe that as the visual features in the document play a more important role than the textual cues, these sort of errors creep up very often. We have used Disrect. Another possible solution towards this information extraction problem would be using template based masks as rules and save them in our local database and apply them on a case by case basis. An example of three such rules is shown on the document in the left where each of the bounding boxes represent the named entity that appears in the document and given a test document, any text that appears within the bounding box, let's say labeled as event title, we can say that this is the text that represents the named entity event title in the test document. However, as some of the recent works has also shown that this is a very expensive avenue to trade on. As we are going for exact matches for any test document that we might encounter, we need to be vigilant to update and be ready for any layout that a test document may contain which makes it prohibitively expensive to deploy and maintain in a cost-restraint environment. We approached this information extraction problem from a different angle and we come up with some observations first. We say that a visually rich document irrespective of its format, layout or content can be represented as a bag of isolated but coherent visual elements. Now although the visual heterogeneity of the document itself can be significantly high or low, when we concentrate on each of these visual areas itself, this visual heterogeneity gets significantly reduced. An example can be seen on the left where each of the red bounding boxes represent these coherent visual areas and we can see that the visual features get reduced by a lot when we compare it to the entire document level context. Now our observation number two is that a few of these visual areas also act as interest points in the document, which means that these areas have higher semantic significance than the other documents than the other areas that appear in the document. So based on these observations, we come up with the following hypothesis. We say that a visually rich document irrespective of its layout or format or the content or the information it contains can be represented as a bag of visually isolated coherent areas. And to find these segmented areas, we need to incorporate or consider not only the visual features of the document, but the semantic features as well, underline the word visual here. Now, our second hypothesis is once we have managed to represent a document using this bag of visual areas representation, we can apply a divide and conquer approach of the information extraction task we were given to perform on the document on each of these segmented areas and return the results we get to a high accuracy that we expect. So our proposed information extraction workflow from these documents look like this. First, we get a representation of the document by segmenting it into a bag of isolated areas and then we perform information extraction task by dividing and conquering the named entity recognition task, which is the task that we will demonstrate our performance on each of these segmented areas. Now, moving on to each of these algorithms, both our contributions. First, let's talk about the segmentation algorithm VS2 segment. So the input to this segmentation algorithm is the visual rich document itself and we expect the output of the bag of isolated visual areas that represents the document in a unified way. So the constraint here that we are dealt with is that the visual rich documents may come in various formats. For example, they can be in XML format, HTML format, PDF format or any other format, but we want to be as generalized as we can be. So we are not allowed to use any format specific high level operators to perform this segmentation, which is why in our segmentation algorithm, we kind of limit ourselves to easily generate a visual and semantic features that can be extracted by the rendered image of the visual rich document that is input to us. Now the segmentation algorithm itself is a recursive algorithm, which incrementally breaks down the area, the visual area that's given as input to it into smaller areas and it updates the layout tree that represents the document itself. Now if a visual area that's being input to the segmentation algorithm is broken down into smaller areas, then those four smaller areas will be represented by four nodes in the layout tree and added as a child node of the parent node itself. Now the layout tree is initialized with the root node R representing the entire visual area covered by the document itself. Now as the segmentation algorithm converges, we return all the leaf nodes that we have found of the layout tree back to the user and those visual areas represented by the leaf nodes represent the segmented areas that we set out to get from the segmentation algorithm itself. Now we use both visual features and semantic features from the document to implement this segmentation algorithm and the explicit visual features we have used for this purpose are white space based separators that run across from one edge of the document to the other, cutting across the document in clear visual, as clear visual limiters. And the implicit visual features that we use to cluster the visual, atomic visual elements which are words in our case are chosen empirically and shown as these five features that we have used in our work here. So to kind of recap, so the VST segment algorithm is a recursive algorithm that takes in the rendered image of the document as input and gives us this representation of bag of isolated areas shown on the left where each of the bounding boxes represent a segmented area that we obtain from the algorithm. Now to perform this segmentation algorithm, we need to incorporate both semantic and visual features extracted from the document. Now let's move on to the information extraction algorithm which is why we proposed this segmentation algorithm in the first place. Now let's moving on to VST to select our second contribution. This VST to select algorithm takes in input the bag of visual areas representation of the document and the name identity we want to extract and it outputs the text extracted from the document corresponding to the name identity that we wanted to search for. For every name identity that we want to extract, we search for a syntactic pattern that represents the name identity within each of the segmented visual areas and we rank all the matches in the multimodal encoding space and return the top case. So we kind of follow the paradigm of divide and conquer information extraction algorithm in this workflow and go from there. Now one of the biggest constraints of this name identity recognition algorithm is we want to be robust towards the ad hoc nature of the content of visual rich documents that we may find in the wild which is why we learn the syntactic patterns that represent every name identity and the patterns that we search for within each of the areas using a distance supervision approach. For each appearance of the name identity in the holdout corpus, we find the dependency parse tree, tag them with some publicly available natural language processing tools and then mine for the most frequent sub trees that represent the name identity. Now the pattern that we get from this most frequent sub trees represent the sub name identity that we want to search for when we are trying to perform the information extraction task. Now once we have managed to find, managed to learn the syntactic pattern that we want to find for a name identity and we have also managed to find out which visual areas contain those patterns we want to rank them in a multi-modal encoding space. Now we do this by computing, we come by computing a weighted L1 distance from the closest near interest point in the document from each of the matched patterns. Now the interest point as I told before represents an area in the document that has higher semantic significance than the other areas in the document and we find this optimal subset of areas from the segmented visual areas we obtained from the segmentation algorithm VS2 segment by solving a multi-objective optimization problem. Each objective optimization problem represents a design principle that's used by graphic designers to highlight the semantics or the importance of a visual area within the document. We selected three such objectives in our work and they were selected empirically. The interest points that we found for our document that is for the event poster from Alice's collection is shown on the left where each of the bounding boxes represent an interest point which is a visual area with higher semantic significance in the event poster itself. So to recap, VS2 select leverages the local context boundaries that we obtain from the segmentation algorithm and performs a divide and conquer approach to identify the name identities that we wanted to extract from the document in the first place. And to combat for the ad hoc nature of the visually rich documents that we can find in the wild, the syntactic patterns representing each name identity are learned using distance supervision. Now let's see how we did on a real-world dataset. Due to the interest of time, I will only cover one of the experiments that we performed in our paper. There are detailed analysis and more experiments that can be found by going back to the paper itself. Now let's focus on the first experiment, the first example that I introduced in one of the beginning slides. What happens, how did we do on the real-world event poster dataset where we wanted to extract five distinct name identities for Alice from each of the event documents. Now these event posters or flyers are heterogeneous in nature as in both their layouts are varied and the formats they are saved in are also different from each other. We have two such formats in the entire collection which may contain digital images saved as JPEG files as well as PDFs. Now the name identities we extracted from each of the documents are shown here where you can see that each of the name identities represent an important details that one can expect to extract from an event poster especially the information consumer like Alice. Now to measure the performance of our proposed method, we wanted to find the accuracy of the extracted text by performing a two-phase evaluation paradigm. We say that the extracted text T identifies as name identity N is accurate if the position of the text appearing in the document is accurate as well as the name identity type N it is classified as is accurate also. And to perform these comparisons we compare it against human and energetic ground truth which is shown on the left here and the color legends are also given. So that in the end to end performance we see that we do significantly better against the text-based baselines where all of these visual features and the segmentation step is not considered from, are not considered a priori. We also compare our system against a rule-based method which is commercially available termed as report minor and see that we outform them by a large margin also. By performing a statistical significance test we see that we significantly outperform the text-only baseline by a p-value less than 0.05. So the main takeaways of this work are the following. We show that prior segmentation can help information extraction from visually-rich documents. We have proposed two key algorithms for this purpose, both of which one is called VH2 segment and the other is VH2 select and we believe that these findings opens up a lot of future exciting possibilities where the semantic relationship between the segmented areas can be explored to perform more complex information extraction tasks such as QA summarization, etc. With that my presentation today ends and I will ask for more questions.
contributions, two key contributions of this work layout format or content, they do have a major information that these documents contain are not as the layout and as well as format of these the document and use the existing techniques that separator and those are two distinct semantic entities that appear in the document. However, textual cues, these sort of errors creep up very three such rules is shown on the document in the the named entity that appears in the document and works has also shown that this is a very a visually rich document irrespective of its compare it to the entire document level context. isolated coherent areas. And to find these consider not only the visual features of the document, but the semantic features as well, approach of the information extraction task we these documents look like this. First, we get a representation of the document by segmenting it into a bag of isolated areas and then we perform to this segmentation algorithm is the visual rich that we are dealt with is that the visual rich visual features we have used for this purpose are obtained from the segmentation algorithm VS2 boundaries that we obtain from the segmentation visually rich documents that we can find in the we see that we do significantly better against the text-based baselines where all of these segmentation can help information extraction from visually-rich documents. We have proposed two key