 Hello everyone and thank you for joining. I'm Federica Ventruto, a data scientist at GraphAware. GraphAware is a globally recognized graph technology company and the creator of Hume, our connected data analytics platform. And we'll present today a session about a graph-based entity resolution approach in order to achieve data harmony. So we will start with a bit of context and applications, why and when you should apply entity resolution. Then we'll discuss the process from a technical point of view. And in particular, we will focus our attention on the incremental or real time process. Next, we'll discuss a few data modeling options that we have in order to represent the resolution results. And after that, we will see a demo. We conclude with a few takeaways and question and answers. So entity resolution is a crucial process in data management that involves establishing connections between the scattered and incomplete records to resolve the same real world entity or the same real world profile. So let's suppose that we are ingesting data about people coming from different data sources. Now, according to the source of our data, we might have that the same person is referred to by using different identifiers, different ID that are, for instance, the license ID, the insurance ID, and so on. There are many options. Also, the same attribute could be written in different ways. So here we have the name, including the middle name. We might have the name written as last name, comma, first name, or two different attributes for first name and last name. So having the process, having a process that allows us to resolve and disambiguate these entities, these records, is fundamental because it allows us to reach a unified view of your entities, of our entities, in order to reach a cohesive understanding. This allows us to go beyond the classic silo situation in which different data sources do not interact to each other, and instead reaching a system in which we can create a real knowledge and the different sources interact so we can merge the information that we have from them. For this reason, entity resolution helps in reducing the complexity of the data and perform more accurate analysis and predictions. In our approach, entity resolution is a two phases process. The first phase that we call candidate selection and ranking essentially involves some custom defined rules in order to compute, to identify potential matches in the graph. So at the end of this process, we will have our graph with some additional similarity relationship, each of them with the related similarity score. At the end of this step, there is the resolution phase in which we employ some community detection algorithms to identify communities of nodes that are closely connected. And each community is likely to represent the same real world entity. So these communities are the end, the final results. So let's see these two phases in detail. The first step, the first phase is the candidate selection and the ranking. And it is a single process in our approach in the sense that we perform it by using a single query but i'd like for a moment to keep them separated so that we can start explaining the the selection and then see the rank phase so candidate selection aims to identify potential matches among records based on custom rules when it comes to the rules, we can distinguish them according to the type of information that they leverage in the graph. And the first type of rules is, of course, the ones that depend on the node attributes. Here in this picture, there is an example that is the same social security number. This is a rule of type node attribute because it is using the social security number that is indeed an attribute of type node attribute because it is using the social security number that is indeed an attribute of our person nodes that we are supposing to resolve in this case. Now these rules are very easy to manage but usually they are not enough because the problem I mean the problem that we have with the resolution is that we usually do not have unique identifiers that we can use to distinguish among different entities. So for this reason, we need to leverage the context of the graph and the indirect relationship linked to the nodes. And this is why there are other type of rules that we can define, among them there are graph pattern rules we say that the two entities are similar according to gap pattern rules when these two entities are linked to some common nodes through relationship or paths of the same type in this case for instance we can say that the two person are are satisfied a rule of type same address if they are linked to the same address or the same location through the same type of relationship. In this case, as others, there are also composite rules that combine both for the attributes and the patterns. So similar name and same address is a rule of this type as the name is essentially a property of the person nodes and for the same address we are checking a pattern in the graph. Now the rank process is to score potential matches in the graph and in our proposal we have two levels of scoring. The first one is represented by the similarity score that is computed by using a similarity measure. It returns, usually, results that goes from zero to one. And here we have a few examples. So in the case of the same social security number, these two entities have a similarity score of one as we are checking the exact matching of the social securities. So the score is one. While for the similar name and same address, we are considering a similarity measure, the similarity of the names of this person. So in this case, this similarity score is obtained by computing the similarity of the two records names. With this approach, it is also possible to define optionally a threshold so that we define a similar only pair of nodes that have a similarity that is higher than the fixed threshold so that we have the possibility to filter out some pairs. The second level of scoring is represented by the importance of the rule. This is a property of the rule itself and allows us to score the rules according to their importance. What we will consider in the end is the final value that we got by multiplying the similarity score and the importance score of the rule so this value that we you can see here is the final value that we will consider later on during the resolution phase and it's what we will call similarity similarity value or similarity score now i'd like to show you a few examples of the queries that we can use during the candidate selection and the ranking phase. And as you can see, this step can be performed by just using the cipher queries. And this is the query that we are using and that we can use for same social security number rule. This process, the candidate selection, is a process that we execute in a streaming mode in a sense that for each input node, we identify all the entities that match the rule that we are considering. In this case, all the person nodes that have essentially the same social security number. Then we compute the score, as I said before, by multiplying the rule score, in this case, the same social security number as a rule score of 0.8 and the similarity score. And then we return for each input node that is represented by this node ID. So in this case, start node identity, we will have the list the collection essentially because we are collecting them the the list of entities that matched for each of them we have the ID and the similarity score so that later on we can create the similarity relationships here there is another example in this case we are considering a similar name and same address rule. Again, we executed the process in a streaming mode, but in this situation, we have two different pieces of the query that are dealing with the selection part and the ranking part. So for the selection, we are using a text index on the name of the person and these are in order to streamline the process. Then we check the condition that is, we filter essentially according to the same address condition. Then there is the ranking phase. And in this case, we are using the Sorensidist similarity string similarity function computed by considering the names of the two person. The reason why we are using this function is because it returns normalized values between zero and one. So instead of normalizing the results of the full text, we just use this other function. That is another option that we have. But this is important because it allows us to show the way in which we can customize our rules by using different ranking functions, similarity functions, or splitting the candidate selection and then the candidate ranking as we prefer. In this case, we are also defining the similarity threshold so that we will consider only entities that matched that have a similarity score that is the Soren said is a string similarity value higher than the fixed threshold. And then also in this case, we compute the final score by multiplying the importance of the rule, in this case 0.5. It's just a value that we assigned according to the quality of the dataset that we had, and the similarity that is obtained by using the Sorensen Lisa. Now at the end of this process, we will have our similarity added into the graph so that among possible matches, we create this similarity relationship with the similarity value. Then we perform the resolution phase that employs community detection algorithms to identify communities of nodes likely to represent the same entity in our approach we are using we can we can use two different algorithms from the neo4j graph data science library the first one is weekly connected component or wcc that offers a broad resolution value resolution level in the sense that this algorithm works by identifying the connected components in the graph that are groups of connected nodes. So every time two entities are linked by a relationship, they will be in the same connected component. This is an extremely valuable algorithm in the sense that if our data is lacking or dirty, it avoids us to miss some important information. But in other cases, we can get very big components. But on the other hand, we can use the other algorithm that is Louvain that offers a finer level of resolution. Louvain is a hierarchical algorithm that can work in a weighted mode in the sense that it can take into account the similarity values so that these weights that we are standard to the relationship. And it is able to distinguish between communities of nots that are strongly connected among them and the ones that are not. So in this example that we are considering with these four Michael Alcon, while WCC returns a single community, because essentially these nodes are linked among them, Louvain identifies two communities. And these communities are composed by nodes that share the same social security number. This is not the case, it's because the same social security number relationship has a niger value. So this shows us the importance of defining the proper rules and assigning to them the right thresholds and importance. Now here we have the queries that we are using during this resolution phase. So this is the first one that we are using for creating the in-memory projected graph. So we consider the personal nodes, the ones that we are solving, and the similarity relationship among them. We are also considering the scores of the relationship and in the case of multiple relationship, we are summing up the scores. Then we are on the WCC without considering these weights, this score and the Lubain in a weighted mode because as you can see, we are taking into account the total score that we computed before. Also in this case, we are enabling the writing of intermediate communities because Luven is a hierarchical algorithm. So it offers the possibility to return all the intermediate communities identified by the algorithm before getting the final, the final, the optimal configuration of the communities. And so this is another a further level of customization that we can select the further level of uh yeah resolution that we can assign or we can define according to our data and the data sets that we are working on. Now, what we've described so far is what we call batch resolution. That is a process that is useful in the case of large scale and initial resolution. But the problem is that this process can take a considerable amount of time while we need a resolution that is able to to keep our resolution results up to date and it's quick enough because usually data changes over time and in small increments so we need for this reason a process that is able to address all the changes up in the data almost in real time. And this is the process that we call incremental resolution that essentially works by identifying only the portion of the graphs that are affected by some changes according to, I mean, compared to the previous status of the graph while the previous resolution was run. And it updates the resolution only on this portion of the graph, only on this subgraph, while remaining the... while leaving the remaining nodes untouched. So this process starts by identifying the items that are affected by some changes, and this is possible thanks to the Neo4j change data capture feature. In our example, we can suppose that this red node changed and that it's the social security number of the node that was updated. Now, after detecting this event, what we do is to run the candidate selection and the ranking only on the connected component of this red node, that is the affected node. Why are we considering the entire connected component? Well, if the social security number of this node changed, the previously established similarity relationship involving this node, because of the social security number are not valid anymore and we need to recompute them so with these with these updates the entire connected component might be affected so in order to keep our resolution correct and that we need to rerun the similarity process on the entire connected component. Now after doing this it can happen that the social security number of the red node is now the same of another node in another connected component. So while running the resolution we need to consider to apply the community algorithms on both the connected component of the affected nodes and the nodes that are involved in the newly established similarity relationship. This process is effective in the sense that we can address and resolve in a proper way according by addressing any different type of changes that can happen to the database. For instance, the addition of items, not so relationship or the updating of the deletion of them and all these possible cases that can be addressed through this process. And it's efficient because we are only updating a sub graph, a subset of our graph and we can run in very fast. Now let's say that we run our resolution. Now it comes to the data modeling to the representation of the resolution results. There are a few options that are possible. The first one is storing the resolution results as attributes of the nodes. In this case, we use the of the resolution, but other options are possible. So the main thing is that we are using, we are storing these results as an attribute of our records. The other option is represented by the hyperlink approach. So in this case, we are creating an empty node that is linked to all the entities resolved through it. And the third approach is very similar to the second one. But in this case, these nodes will be a fused node in the sense that it contains some consolidated attributes. So which one, which option to prefer depends of course on the use cases and the requirements. For instance, if you are working in a street access control scenario, you should prefer, you can prefer the first option, the attribute option or the hyperlink because allows us, allows you to check the information, check the resolution of your node without disclosing information about other nodes that you cannot you you do not have privileges to see while if you need to um traverse essentially these uh these resolution results in your queries because for instance you want to merge data coming from different data sources and the jump from this Michael Alcon to other coming from other sources you should prefer the approaches that indeed that essentially created this node because the queries will be simpler now I'd like to show you a demo through YUM. So this is the visualization of YUM. And here we have five nodes that we will consider during this example. While about the rules, the similarity rules that we are defining, we are considering the two rules that we explained before. So same social security number and similar name and same address. And then there is another rule that considers the similarity of the names and the match of the date of birth. Now, by observing this note, we can notice that these two notes have essentially the same social security number. And this is the fourth line that we can see in the caption. So this number and again by these two nodes share the same that of these three nodes are for sure similar among them. Also, these two nodes have the same social security number again on this property. So they will be similar according to the first rule and these two others share the same addresses will be similar because of the address pattern. Now, let me run the batch resolution, supposing that we are running the first resolution process, the initial phase. This is a workflow that essentially implements our batch resolution, and we created by using orchestra that is our orchestrator that allows you to create workflows that can implement complex business logics in a local setting now let me run it while i explain how it works in the meantime so in the first branch we are just setting a few things such as the entities that we are resolving and creating some of this that we will be doing the process. Then in the second one, we fetch the entities that we want to resolve. In this case, all the person that we have in the database. And adding to them a temporary label that is to be seen in process label. The reason of this is because we wanted to track the status of the similarity computation because the similarity computation is essentially the process that takes the most of the time. So in case something goes wrong but also for checking the status we wanted to add this label. So here this bigger branch deals with the similarity computation. So we have three rules and essentially this branch starts with the three lines, one for each rule. For each rule we are fetching the entities that we want to process according to the label that we added and then we apply the rule. This is very similar to the one that we have seen before just a few automations. And this is the case of the same social security number. Now, at the end of the branch, we check. And every time an entity has been processed according to all the rules that are defined here, we remove the temporary label. So that in the following branch, in this one, we are just counting the number of nodes that still have the temporary label. So we know that the similarity process finished when the number of these nodes is zero and we essentially can start the resolution phases. Here we perform the resolution phase. We are projecting the graph. The query is very similar to the one that we have seen before. And then we run WCC and the Louvain. Now I just want to show you the number of one that we have seen before. And then we run WCC and Louvain. Now I just want to show you the number of nodes that we are considering in the projection. That is the entire size of essentially all the person that we have in our data set and we process according to the CIRR because it is this number that we have seen here, 400. Then we store the results. And in this case, we are using the attribute mode. So let me reload this announcement. And we can notice that now we have two new attributes that are the result of the WCC and the result of Louvain. Now, if I enable the grouping strategy according to WCC, we can notice that all these entities belong to the same community. While if I enable Louvain, we have two different communities. One is composed by people that are born in 1994. These two have also the same social security number, while in this other we have people that again have the same social security number. So these results seem promising. Now if we want to check the resolution, the similarity rules and the similarity relationship created, we need to change the visualization. Because something that I didn't mention before is that we usually suggest the usage of an helper database in which we can store the similarity relationship and also run the graph data science algorithms. This is because in this way, we can avoid polluting the main graph with all these details that are only relevant for the purpose of the resolution. Otherwise, if an analyst wants to see them, can just change the visualization. Okay, now let me copy these component ID. And if I change the visualization and I open one visualization linked to the helper database, we should now see our entities. open one visualization linked to the helper database, we should now see our entities. So because we run the batch workflow, so we have now the entities that are visible in this visualization. So if I expand the person nodes that have the component ID equals, sorry, the one that I copied before, we should see our entities. Okay, let me disable the grouping strategy and expand all the similarity relationship so we can understand the way in which they are connected. In particular, we can notice that the connection that goes from a community to the other, according to Louvain, is because of the same address. Now I just wanted to show you the incremental part. So let's suppose that an analyst noticed that this relationship is wrong. This person has not this address. I can drop it by using our graph editing tool. And let me submit the changes so that it will be persist to the database. Now, let me run the incremental workflow. This is the workflow that is meant to run continuously, but in this case, just for the demo, I'm starting it manually. So it is very similar to the batch workflow, but it starts by using a Neo4j change data capture components. You can see here and we are essentially monitoring the changes that involve any item used in our similarity rules. So this selector is defined according to our rules. So for instance, we have a sort of security number, the names, the address relationship. Now, when an event happens, what we are doing is to identify the connected component of the affected item, and we are processing all the entities belonging to the connected component. So here we are running the similarity only for five nodes. Then we also run the resolution when these, yeah, here in this bench, and I can show you again the sides of the projection that we are considering for which we are running the resolution. Only five nodes, the five affected nodes. The process finished, so I can go back to the main visualization and reload all these nodes. And we can notice that now, enabling again WCC, we have two different communities according to this algorithm, because essentially we broke the relationship that was connecting the two of them. Now, this concludes our demo. We have seen both batch resolution and the incremental one. In our case, I think that it took less than a minute, but in general, the incremental is a very fast process and also effective. I just want to conclude just mentioning that we have seen that gaps offer a natural way to address safety resolution. We can for instance mention the simplicity in defining the similarity relationship even in the case in which we are considering contextual information of our nodes in direct associations, all the links. It is very simple to write queries because of the graph structure. Also, our community detection algorithms are very fast because we are working, they are optimized to work on relationships, and this is the way in which we stored our similarities. We have discussed some of the useful features that Neo4j provides for managing the real scenarios. For instance, the change data capture that we use in the incremental or real-time process, but also the community detection algorithms that we use them and the variety of indexes that exist. And we have described the fully customizable and explainable entity resolution process that DUM offers in which the only responsibility that is left to the analyst is the definition of the rules according to the data. Otherwise, if you would prefer an out of the box solution, DUM also integrates with the sensing that is a living entity resolution library. But that's maybe for another talk. So I'd like to thank you all for attending this session. I hope that you found it interesting and useful. And let me check if there are any questions.
NODES 2024 - Beyond the Silos  Achieving Data Harmony With Graph Based ER
Entity Resolution (ER) serves to interlink fragmented and dispersed data, facilitating the identification of records representing identical real-world entities. This function is pivotal for intelligence analysis, enriching investigations by ensuring comprehensive and uniform data merging.
 This session will showcase a robust end-to-end approach for precise and effective data consolidation, adaptable to batch or incremental processing and predominantly reliant on graphs. Key themes will encompass customizable similarity rules, harnessing node attributes and relationship patterns, and strategic utilization of Neo4j indexes and GDS. Additionally, we’ll delve into diverse data modeling strategies, evaluating their advantages and drawbacks and how to accommodate dynamic data changes.

With Federica Ventruto


Get certified with GraphAcademy: https://dev.neo4j.com/learngraph
Neo4j AuraDB https://dev.neo4j.com/auradb
Knowledge Graph Builder https://dev.neo4j.com/KGBuilder
Neo4j GenAI https://dev.neo4j.com/graphrag