 Welcome to Graph Power Hour. Hi, my name is Paco Nathan, and I'm very glad to be the host here for Graph Power Hour from Sensing. Today for our second episode, we're thrilled to bring you Claire Sullivan. Dr. Sullivan is one of my favorite instructors and writers and engineers in the whole space of data science and AI. I love her tutorials and she has a lot of fantastic material here to share today. Just as some background, Claire and I have been working together. I've been learning from Claire about some of the intricacies of Graphrag, particularly in the context of using more structured kinds of input. It's working with entity resolution, but really looking in depth, like where are some of the gotchas? So I'm super thrilled. Let me click here. The title is Graphrag, The Good, The Bad, and The Ugly. And if you're just tuning in uh we have just launched graph power recently this is episode number two and we definitely want to encourage everyone to stay to the end because we have a special surprise um we're going to do a little bit of a contest, not a contest, it's a random drawing, I should say. Let me use the right verbiage here, but stay to the end and a lucky winner will be getting, well, we have some board games. We'll announce more about that later on. And so without further ado, let me introduce here.ire take it away all right thank you paco just wanted to thanks for the invite to uh come and talk today also thanks to senzing for having me as well um so today's subject graph rag the good the bad and the ugly and we'll see some good and we'll see some bad and we'll see some real ugly i'll tell you you. But before we get started, I always like to know who it is that's in the room and whatnot. So I can kind of talk about the interesting stuff to whomever's here. So why don't we kick off with the first poll here. So this is the audience participation portion. Let's see. So do we, there we go. Okay. So what is your current role? Got some answers coming in. It's looking like we've got a good diversity of people. I love that. We've got data scientists and ML or AI engineers, data engineers, product managers, business leaders, other business owners. This is great. Yeah. So give this another couple seconds here. Yeah, that's interesting. It is really, really spread across different roles here. And a lot of software engineers. That's good to see. Yeah. All right. So let's get, let's stop that poll. Let's go to the next poll now. And the next poll is what is your level of experience with large and GraphFrag. Oh, interesting. Okay. We've got a lot of diversity in this one, too. So, you know, hopefully there's going to be something for just about everybody in this talk i'm glad to see this um a lot of people in their learning journey with it that's wonderful love it um give this one another couple seconds we'll go to the next one there's four of these so we'll go to number three here now okay and then question number three how familiar are you with entity resolution and sensing? Some good diversity here too. Love it. All right, maybe another couple seconds on this one and we'll jump down to the final poll. The piece de resistance. Do you use entity resolution in conjunction with your knowledge graph practices is that something you're doing today this is the one i've been waiting to hear from i'll be perfectly honest you you know despite what you're hearing today sometimes i do sometimes i don't and you know i every time i don, I'm feeling pretty shameful to be perfectly honest, because I think what we're going to see today is there's really good reason to be doing it. So, all right, we can give that one a stop now. So a lot of people are in the evaluating and production and cases, but a bunch of people are not the evaluating in production and cases but a bunch of people are not using it yet so hopefully today um when we look at the good the bad and the ugly we will get to see when you might want to consider doing it now i will tell you that um i'm going to be looking a lot at my slides but i'm going to also be having um the&A window open, and I'm going to try to be good and remember every time I get to like a transition slide that has a black background to it, I'm going to try to consciously look over at the Q&A window, and then we'll have time for questions and answers at the very end as well. So let's dive in. All right, a little bit of how we're going to structure the talk today. Start a little bit about my graph journey, which I think is perhaps a little different than most people who come to graphs through proper means. I had a very unusual introduction to graphs. Then talk a little bit about why knowledge graphs are hard. Okay. And then, you know, when you see something is hard, you say, you know what, and I mean, this is true in all of data science is we try to, you know, we have fancy models, but a lot of times it's just the simple stuff that works best. And so if knowledge graphs are this hard, why should we bother if we just have generative AI and text? And then I'm going to show you some, this is the real ugly part, the creating knowledge graphs in the before times, show you some of my sad tales of woe when it comes to creating knowledge graphs. And then what they look like today and why that's such an improvement. And then ask a question that I know I get asked a lot, I'm sure Paco gets asked a lot, if you already have your data in a graph, why do you need entity resolution? So we're going to get into that a little bit too. But you're all here because graphs are having a moment. And let's be real, they're having a moment because of large language models, LLMs and generative AI. I just was kind of going through my stash of articles and whatnot and came up with what are some of the three like what are the three that really kind of shaped my view of graphs this year on or well I should say the past 12 months when I say year um the this and these two that are on the left here we'll we'll actually be coming back to later in the talk the retrieval augmented generation or RAG for AI generated content. This is a great survey article. Anything you could possibly want to know about the science behind RAG is in that article. The next article, the one below it here, this benchmarking article by Juan Cicada and the folks at Data World. I saw the results of this presented at a conference, Data Day Texas, in case that sounded too much like one word. Data Day Texas in Austin, Texas in January. Great conference. And Juan presented some results that just really motivated why you want to consider using graphs for your RAG. You don't have to. You can use a vector database such as Pinecone. You can have Postgres act as a vector database, but talk a bit about why graphs are great at this. Then of course Microsoft put out this really great research blog post and new tool that they have their own GraphRag thing going on. Yes, graphs are truly having a moment and I'm really excited to be part of that domain right now. But I didn't start as a graph person. I started just as, believe it or not, a nuclear engineer who became a machine learning engineer at GitHub. And so while I was at GitHub, it was a picture of my computer in a hotel room. And I just, you know, I had a bunch of stickers on it like everyone does. And I'll be honest, I created this sticker that you see on the right here. I will tell you, I got a little nervous when I was putting this up. I was like, oh, if I take a photo of that with my phone, is it going to come up like some QR code to some, you know, inappropriate website or something like that? It won't. It's not a QR code. But does anybody know what that is? If you do, drop it into a public chat because it's a pretty unusual thing, I think. It's really fun. You obviously know it has something to do with graphs. So I'll give a couple seconds here. Okay. So it turns out software and GitHub being the largest repository of software is a graph. And lots of things are graphs. But when you talk about software compilers, software compilers actually work on this thing called an abstract syntax tree. And this is a very small one that I've shown here, an abstract syntax tree. And it basically says, you know, this variable is connected to this function and goes to this other thing. And there's these if statements and all of that. But the bottom line is that you can represent software as a graph. Now this was circa 2018 when I was working on this. And the idea here was we had access to all code across all of GitHub. And could we look at the abstract syntax trees of all code and detect duplicates of code? So, and not just duplicates by saying, you know, I'm writing a function to calculate a cosine in Python. Paco's writing the same thing in Python. Did we have exactly the same code? We're actually looking at here whether the code is written in a completely different language. It's looking at the entire code base. It's looking at individual functions within the code. And can you use some sort of approach to detect this function written in JavaScript is the same as this other function over in somebody else's repo that was written in, I don't know, Rust, for example. And so looking at these abstract syntax trees, I was just I was looking at these one day and I was like, you know what? This is a graph. Let's look at this as an adjacency matrix. The adjacency matrix is the mathematical definition of the graph. What you're seeing in this not-quite-QR code thing is the adjacency matrix of a piece of software written in JavaScript. I thought that was really cool. I had to turn it into a sticker and give it to all of my friends. But then I said, you know, my background actually, even before nuclear engineering was in astrophysics, and I was like, that looks like stars. Wait a second. If that looks like stars, I can apply image analysis techniques to it. So i started doing convolutional neural nets to set to detect is this code identical to anything else within a database and it turns out using that approach we got something like 97 accuracy at detecting duplicate code written in other languages from each other so that was that was a really exciting project i loved working on that project. And if you think about it, it almost kind of is the mathematical basis of what became things like GitHub Copilot today. So I did not work on Copilot. I don't want to make it sound like I'm taking credit for that. But the point being that there is a lot that can be done when you look at your data as a graph. And this was just a really unusual, fun form of data to look at. Okay, so now I'm getting over to my first black screen thing. I'm checking the QA window. I don't see any questions come in yet. So graphs are obviously having a moment because of generative AI. That being said, do we really need graphs if we're just analyzing text? I mean, I can throw my data into chat GPT and it can come back with answers and it's not in a graph, it's just words, right? Well, so here's another project that I worked on, a little demonstration that I did. This is a particularly popular data set for graph analytics called the quora data set and what it is it's um you know 2700 scientific publications and data science and they are broken into seven classes i will tell you that those classes are very imbalanced so not a fun problem and then they look at um the citation relationships between one paper to another. This isn't quite a proper graph schema here. If it was, I would have paper and it would look like a loop citing back to paper. What I'm trying to convey here is that one paper is citing another. Now, I don't know what these papers are. They're given an integer ID and they're given this subject, which is one of the seven classes. And then they're given features. What these features are is somebody has encoded the vocabulary of the abstracts. So now what we have are word vectors. So this is a data set that is both a graph and it has word vectors. And so the question is, I just wanted to say, well, how well do we do at classifying based on word vectors versus graph vectors? So graph embeddings, you can set up a graph, you can create embeddings. I did that using an algorithm called fast RP for anybody who cares. But the idea here is that I wanted to know if all I did was change the model or, you know, change which embeddings we're using. I'm not changing the model. I want to be clear about that. I used the same support vector classifier model. I did not change any of the hyper parameters. I kept those all the same. All we did was use the same machine learning model and give it word vectors or give it graph vectors. One caveat here, there is no access to how the word vectors were created. So I don't know the vocabularies. I don't know anything about that. It's possible that they could be tuned to do better. I'm just going with what's in the base core of data set. So again, both models were given the same task, both approaches given the same task, multi-class classification with this massively imbalanced data set. So when I did that using the word vectors, here's what I got. Okay, so this is our confusion matrix here. And for those of you who are not familiar with this thing, the x-axis, the predicted label is what did the model say it was versus the true label on the y-axis, what actually was it? So we care about the numbers on the diagonal and the closer they are to one, the better. And we average those numbers up and we get the mean accuracy of the entire algorithm of about 73%. Okay, so that's not bad. Now, I will tell you that the class four here, this true label of four, that's the one that had the fewest samples in it. And by fewest, I mean, like, you know, it was in the tens of samples, and everything else was in several hundred samples. So, you know, you can see that it got 50, it got class four correct 54% of the time. And that's just barely better than flipping a coin. So not great. Now, this was just using the word vectors. Now we're going to do the same model and we're going to use the graph vectors. So when we do that, what we see is that our mean accuracy went up by 13%. So 85%. And that was without any hyperparameter tuning, I hyperparameter tuned, and I got like a little better 87%. And you can see, look at this class four here, where it went up from 54% to 76%. So that should be a really encouraging thing that, you know, when you actually look at how things are related to other things, you can do a lot better, right? Whether it's just this really basic machine learning problem of classification, or when we start getting into some harder problems. So great, why don't we all do that? All right, well, let's talk a little bit about knowledge graphs. You know, what I was showing you there before, that's just kind of like a bread and butter graph. Knowledge graphs though, these are the things where we start looking at how information is related to other information. And it's what gives us, you know, potentially some really useful information for getting our LLMs to not hallucinate so much. So that retrieval augmented generation or RAG is really key to making sure the LLMs don't go make up information. But that means we have to convert our information into a graph. And why is that hard? It is hard. The goal here is that we're going to start with stuff. That's the technical term stuff. And we're going to go into a structured thing, another technical term. That stuff could be structured or unstructured. So for example, you could have a stack of PDF documents. That's your stuff. Text, really not structured. Really hard, messy to deal with. And that's probably the most interesting stuff. Like if we already had our stuff in a table, you know, that's very helpful. But like a lot of the interesting stuff that gen ai is going after is not you know in that happy space of being structured then there's this thing that we're trying to get it into which is a graph structure but you know when we talk about graphs usually like we have we talk about knowing what the schema of the graph is so you remember i had those green circles and a paper pointing to another paper. If I'm just giving you a whole bunch of PDFs, we don't know what that schema is. And that can make things very challenging. So then, I'm not the only person who uses this term, gigo, garbage in, garbage out. This is especially true for graphs. And especially true when you're trying to understand the relationships between entities within your text. Verbs, as you're gonna see here in a minute, verbs are those relationships. And verbs are very hard to properly get and attribute to nouns within text. So it's really easy to have very noisy graphs. Garbage in. Okay. If you're putting garbage into your rag, you're going to get garbage out. So this is another problem. Language is messy, like I said. Natural language processing NLP approaches, which kind of are the basis of generative AI, and I'm going to show you when I get to the ugly portion, you know, some, what this looked like a few years ago. They're really complicated, and they're especially complicated if you're talking technical language, And technical language is the place where RAG really shines. Because, you know, chat GPT and the like have been trained across the entire internet. Most of the internet is not spoken at the level of a PhD, you know, chemist. Most of it is spoken at the level of a Kardashian. And so we have to somehow find a way to teach our language models, you know, that technical terminology. RAF, sorry, RAG is the way to do that. But like I said, technical language is really, really difficult for it to get. And, you know, honestly, yeah, language is messy, but data is messy too. Just even nice, basic, happy data living in some SQL database is messy. I say to student data scientists, look, 80% of your life, you know, once you get a job is going to be actually being a data janitor. And so this GIGO problem really does not help. So why? If it's this hard, why? Why should we bother? Well, let me get back to this paper here that Juan Cicada and crew put out. And what Juan did was he had this whole very systematic approach. He had started with a set of SQL databases and I think it was based in the insurance industry. This article here, I really strongly recommend this article. Using that SQL database, he had a very regimented approach to asking easy questions and hard questions of it. So asking a question in natural language to then go query the database. And then you can assess whether the resulting information coming back from that query was correct or not. And what Juan saw was a significant increase in the accuracy of the language model when it used a graph versus not. So, you know, I've kind of highlighted this section here. There were differing degrees of difficulty, which is why you see 66.9% to 71%, 25% to 37%. Go read the paper. If you do not come away after reading that paper saying, I must use knowledge graphs for my RAG, I don't think I can sell it anymore. So, now we're coming up to another black screen here. I'm checking with the Q&A section. I don't see any questions that have come back. Okay, graph structures can do better with text than text-based approaches as we saw, for example, with that Quora database thing. However, they can be pretty difficult to create. Now I'm going to grab a quick drink here. So I want to talk about the ugly. And you know, I'm not proud to admit it. I created this many moons ago. This is a very unpleasant look at natural language processing. And this is what natural language processing was like circa three years ago. And all I have here, this is just the entry paragraphs of Barack Obama's Wikipedia page. And I'm using a standard natural language processing package called SPACI. And all SPACI has done here is just kind of gone through and highlighted the entities of this particular text. So you can see it might be, the font might be a little small, but like the purple entities correspond to people. So the very first one, Barack Hussein Obama the second. You'll notice the very second one of that same color is Obama, as is the third one, Obama. There's lots of places where it says Obama. And having, this is a spoiler alert for what's coming later in the talk, Barack Hussein Obama and Obama are those the same people. Maybe they are, maybe they're not. Could be referring to Michelle Obama. A little complicated, but that entity resolution, as you're going to see here, is going to be pretty important once we get a little further down here. We've got dates that are kind of in mint green. We've got geopolitical environments and kind of like this gold color. So let's look at this, though, in terms of getting into the weeds a bit here. Language is a graph, just like abstract syntax trees like I showed with that GitHub example. Here I've just taken the first sentence, and this is Spacey's attempt to try and attribute which words go with which other words. It's doing part of speech tagging so I can see proper nouns. I can see determinants, adjectives, things like this. This is complicated. OK, to try and come up with what is the verb here and then what subject. The subject is Barack Hussein Obama II. But natural language processing does not necessarily get those as one thing. You know, that's three and a half different words there. And so somehow we have to lump those all together as one entity. Similar, we've got American and politician. So like, you know, we want to, what we want this graph to look like is Obama is politician. And maybe politician has the property of American. And then there's this word and, and just throws everything off. Okay. I always hate it when and shows up in the sentence um the bottom line here is that the process of doing nlp to create a graph is wrought with peril and if you go look at any of my previous content on it you know it's it's it's so sad because honestly i went through and i did this actually earlier today i just gave it uh chat GPT this section and hey create me the subject verb object triples here and it just was so glorious. So that's why I call this the ugly is this is the before times the before generative AI times. So the steps of NLP to create a knowledge graph in those before times. The first thing we had to do is text cleaning because like if you're going to get into text that is not you know nice and happy wikipedia text you're going to have emojis and you're going to have you know all kinds of different character encodings and it's awful and you've got you know web links and you got to do a lot of cleaning before you can do anything then you have to tokenize meaning you have to find all of the different words um you remember You remove the ones that we don't really care about. For instance, words like the, and. These are called stop words. They add nothing to the actual mathematical analysis of the statement. Then we do this thing called stemming and limitization. And what this is, is if you have a word that is, say the word was, was is the past tense of be. And so when we go for the lemmas and the stems, you could say swimming, ing, we take the ing off. So the stem is swim. Okay, so we want to get those verbs particularly in their most basic form. Then we do the named entity resolution. So that's the Barack, Hussein, Obama, the second is all one thing. And then from all of that, then we have to somehow extract the subject, verb, object trickles. The subjects and objects are going to be nodes within the graph and the verbs are going to be relationships. Okay, so that's the process that I used here. And like if you do this, this was the graph I created a million years ago on Barack Obama's Wikipedia page. We can zoom in on parts of this. And, you know, one of the things that I was asking in this workshop that I taught on this was using the graph, find where Barack Obama was born. Now, first off, you had to realize that for whatever reason, the NLP process gave the phonetic spelling of Obama. Don't ask me why, just go with it. So if you look at that upper left, you can see Obama, and then you see B, United States, B, African American, Bear, Honolulu. Okay. So it turns out that, excuse me, Bear is the lemma of born. So you had to know that in order to ask the question, because again, this was before LLM days. You had to really understand how to use language to query this graph. It was highly unpleasant. It's going to be kind of hard to see, but if you look at the two that are on the right, there is an orange node that is Hillary Clinton. And the relationship between Obama and Hillary in this graph was the verb elect. I believe in the context of the Wikipedia article, is that he was elected over Hillary Clinton, the primary. And then the one on the bottom there is Osama bin Laden, and the relationship there is order. Okay, well, that's not going to make any sense. We don't have enough context there to understand that what is actually going on here is that Obama ordered the assassination of bin Laden. So if you wanted to do some sort of question answering with this graph, it was incredibly, incredibly difficult in the before times. It was made worse by lots of things. Like when you do natural language processing, there's just duplicates everywhere. And, you know, so I just I'm just printing out some of them. You can see there's a gazillion Barack Hussein Obama the seconds district officials officially state. We've got John Sidney McCain, you know, so this is like there's just so much mess in this graph. Duplicate it. De duplicating was just a never ending process with this graph. Duplicating was just a never-ending process with this approach. Then there's the concept of named entity recognition, NER, and it's hard. Okay, so I had nodes in this graph that were, you know, Obama terms, Barack Obama, President Barack Obama, Barack Hussein Obama II. Then if you just wanted to go with Obama, you're going to come up with Michelle Obama in this graph as well. And you have to have ways that if you're giving this information to something to try and answer questions about it, you have to somehow figure out how to resolve all of these different entities. Very important. You could do something like really super silly basic, like what's in the bottom here. And what I'm doing here is I'm calculating what's called the Levenstein distance between the names. And what that is, is like how many characters off is one of these things. So if I said Barack Obama, the name Barack Obama has a distance of zero. If I said like President Barack Obama, you have to change 10 characters to get that to match up. It's a really crude way of doing it. I do not recommend it because this is just wrought with peril. But let's actually get out of this scary space. It's gotten better, I promise. So has the making of graph improved in the age of Lms oh my gosh yes it has just unequivocally um so i gave this same bit of text um neo4j graph database company they um have released this really fun tool which you can get at this web page here and basically you can just give it text you can give it a pdf whatever and it will create the graph for you which is super handy i wish that existed when i was trying to do this a few years ago okay and so what you have here this is the entire graph of the very same text and you'll notice it's much smaller like before you know this is what my graph looked like okay so lots and lots of nodes and relationships and a lot of them really bad. Now this is what the graph looks like. Not only that, but if you see these nodes that are kind of like a light pink and the text has an underscore, like you'll see Barack underscore Obama, that's chunks of text. That's not even actually. It's just Neo4j saying, aha, I got this bit of data in and it's a chunk and this is my chunk. You'll see though that there's Hillary Clinton and Osama bin Laden are both in this graph. So it actually went through, it did some sort of prioritization on what should be in there and came up with a much smaller graph. So let's see. They actually have a nice little question answering bot built in. And so I asked how was Barack Obama involved with Osama bin Laden and it gives a correct answer just based off of the graph. How was Barack Obama involved with Hillary Clinton? Correct answer just based on the graph. So the point here is that yes, this graph is smaller, but it is better because the data is higher quality. Okay, it's got relationships that are higher quality and text that's higher quality. So what goes into the graph matters. You get better answers. This is why you really need to think about entity resolution on a graph. But I'm getting ahead of myself. I decided to play with this a little bit more. And so going back to that survey article I was talking about, you remember this? I said this was the article to end all articles survey of all the mathematical approaches that could be used for RAG. Okay. So I took that entire article and I threw it into that knowledge graph creator that Neo4j put out. And this is what it made. This is a portion of the graph. This isn't the whole thing. But it was kind of fun. And so I wanted to ask some questions of that article because this article is like 30 some odd pages long. So let's get a summary of the article. And, you know, I said, what are the three most important takeaways from this paper? And it gave them to me and they were great. I loved it. Okay, so let's ask some more questions. What are the top five methods for RAG? Okay, because I really wanted to just cut to the chase. Tell me what should I use? Just give me the answer. And I'm sorry, but I don't have enough information on the top five methods for RAG. Okay. Not exactly what I was hoping for here, but, you know, it's kind of fun. Like I clicked that little details button at the bottom of that window and it told me, you know, where it was getting that information from and you could go see it in the graph. Really useful tool for experimentation. But let's get into this just a little bit more. Because unlike that Wikipedia page, this article is highly technical. And you remember I said technical terms were pretty problematic. Okay. So I'm just looking at some of the nodes that I created. And I'm like, hmm, okay. So in this top bit, there's a node called edit-sum, and then there's another node called edit-sum. Now, those are probably the same thing, but these are highly technical terms that the language model did not know what to do with. Now, that would be a problem if I was trying to ask the language model a question about edit sum and expect to get back a good answer. Then I've got this, you know, these retrievers. I've got dense retriever and sparse retriever, which are related but different, but then I have retriever and I have g retriever. These are probably the same thing. We need to have better data in our graphs. Okay, so better data in our graphs, looking at the QA bar. Somebody famously recently said, knowledge graphs and entity resolution, like peanut butter and chocolate. I love that quote. So we have to have to have to consider cleaning up our graphs for RAG. And a great way to do that, depending on your data, a great way of doing that is with Senzing. And I want to direct you to a couple articles. The first article here, Paco wrote about entity resolved knowledge graphs. And then I wrote a follow-up article about doing graph rag with knowledge graphs and entity resolved knowledge graphs in this case. So looking a little bit at it, the data set, we used the same data set, three sources, safe graph, which is a data about places and locations and businesses and things like this. We've got this Department of Labor wage and hour compliance action. What this is looking at is Department of Labor grievance or violations by businesses. And then the Small Business Administration was giving out these loans, PPP loans, to various businesses. And, you know, so this is a fun set of data that you can try to link together. Paco linked it. Paco ran this data through Senzing and, you know, kind of his results are on the top. I'm going to talk about these schemas here in a minute, but just wanted to show you what it looks like when you put this data into Sensing and what the kind of information you get out. For instance, this is just G2 Explorer, which is one of the tools that comes with Sensing and lets you look at your data. I just came up with this. Show me this particular company called Desert Spring, which is one of the tools that comes with sensing and lets you look at your data. So I just came up with this, show me this particular company called Desert Springs Landscaping. Okay, so this particular data set is scoped down to the Las Vegas area, just to keep it within the free tier of sensing. And okay, great. So I've got this result that comes back and you see it's got this entity ID. Now, if you look to the right, it says six relationships were derived from this data. So let's get into that a little. So using that entity ID, I can then back out some more information about Desert Springs. And you can see that they have an entry within that PPP loan database, as well as SafeGraph. Now, if you look at that that there's some variations there um for example let's look at the business address you can see that the zip codes are different because one of them is using the the full you know five numbers plus four numbers the other one is just using four numbers um you know it's it's uh one of them has LLC at the end, the other one does not. You could probably get away with linking these things via text, but it's not going to be, it's not going to make you real happy at scale. That's not going to be something that's going to work for more complicated cases. At the bottom, you see some possible relationships. So Desert Springs Pools and Spas, Inc. Now we can all, and you also see Desert Spring renovations there. You know, we can all go with the idea that, you know, these are probably the same thing. We've got Desert Springs Group. You know, it would make sense for all of those type of businesses to be related. Turns out if you click into this and look at Desert Springs pools and spas, yeah, it's probably all related. You can see that that address is the same. But again, if you're doing this at scale with say business names that have absolutely nothing to do with each other, you're not necessarily going to get all of the correct information i'm going to show you an example of that um but what i did was i took that data now um usually when you do entity resolution you have a whole bunch of entities that you collapse into a single resolved entity which is what paco did and it's shown in this top graph here so this is mandalay bay and we've got all kinds of different entities that are related to Mandalay Bay because they're owned by Mandalay Bay. So you can go through Paco's article. Strongly recommend you do that. I did it slightly differently. So this hub and spoke approach, I wanted to have, because I know I was going to throw this into a large language model, I had this hypothesis that if I had more relationships within that graph that I was more likely to be able to ask questions and get some more detailed answers out of it. Jury's still out whether that hypothesis bared out, but I'm going to show you some examples. So this is just a little bit of the graph as I assembled it in Neo4j. Let's do some queries now. Okay, so I had the data both in SQL, so just tabular data, and I also had it in Neo4j. And I decided I was going to be asking questions of that data. So here's a question, and I'm using Python and Langchain here just for interfacing with the data. So I said, find all references to union cabs in all of the tables. Okay. And the output said that they found one reference. I had three tables. I had one table for each of the safe graph, the department of labor and the PPP loan databases. So three tables in my database. And it said that it could only find one instance of union cabs okay well i'm going to tell you a spoiler alert that's not true okay now i uh what do i have here okay i how many violations or does union cabs have any violations meaning department of labor violations it says it appears that union cabs does not have any violations recorded in the daily base database wrong okay let's actually now look at the graph around union cabs and what i see here is oops there's actually other entities that are not named union cabs that are part of this database so like this is this is one of the great things about a tool like sensing is it's going off of lots of other data points other than just a name or an address to try and say, are these the same thing? So we have LVCabs. We have NLVCabs. We have ABC Union Cab Company. Okay, they all happen to be cab companies. None of them are in existence anymore. They've all shut down. But. I said, tell me about Union Cab Company. And you can see Union Cab Company, also known as ABC Union Cab Company, was located at, it operated in the taxi industry, got some information. And you can see that it also has other related entities such as Vegas Western Cab, ANLV Cab Company. Okay. So like automatically you can see, I just got a lot richer information by linking my entities here. Super cool. Okay. So then let's how many violations have entities union cab company is related to been involved in? Aha. Now we have our answer in blue at the bottom. Entities related to union cab company have been involved in a total of six violations. That is the correct answer. Okay. So by the fact that we were able to take this tabular data that appeared somewhat interlinked and somewhat not, and we were able to resolve the entities of it, now our LLM is getting correct answers, whereas it wasn't before. Even though we were doing RAG, we were doing RAG with SQL before. We could have done RAG before and tried to link the entities manually just based on name or something or address, but it's not a great approach because like I showed you, those addresses can vary. The names can vary um you know we we can think about like if we were talking about people um let's say you've got records in your database and let's say you've got um elizabeth smith and beth smith or you know sometimes jack is the nickname for john how are you going to get that you you can't have like this endless supply of you know dictionary here of all the possible permutations of every single name and then if you start talking about names and other languages and names in other languages they can change based on somebody's age and you know where they're at in life and you know this gets incredibly incredibly complicated really fast so if you are not doing something to somehow create that higher quality data in your graph you are going to get the wrong answer in your rag so let's wrap that up really quick um we should give serious consideration how to achieve the best RAG performance. Now, what I showed you very early on in that core data set was using vectors, meaning word vectors. We have graphs. We should probably be looking at hybrid solutions using both graphs and vectors. And you can make vectors out of graphs too so maybe we could bring that in here as well so don't think of your rag as being you know this either or thing think of this as a both and thing to get the right answer second it doesn't matter the problem graph or otherwise this you know hopefully at this point you know, hopefully at this point, you know, we're all on the same page here. If you have better data in, you get better answers out. And as I hope you've seen, entity resolution is key to fighting that GIGO problem within RAG. Last thing I want to say is just about the rapid pace of development in this area. There is code that I have written for generative AI a month ago that doesn't work today because the tools are just changing that fast. This has a lot of implications for graphs and for otherwise. But what I want to just kind of leave you with is that these tools are evolving all the time. They're getting better all the time. Stick with it even when you do the same thing I do and pound your head against your keyboard. Oh my gosh, they changed the package again. And now my dependencies are all broken. You know what? It's still giving you better answers. And so just kind of want to put a shout out, you know, to those people who are developing these tools out there, you know, particularly in the open source community, like to encourage people to get involved in the open source community, help make these tools better. But yeah, at this point, I have come to the end of my talk and it is question time. So Paco, why don't you come back in here? Let's do some questions. Wonderful, thank you very much, Claire. That was fantastic. And I have a quick question for you just on that last point, starting out, you know, the tool from Neo4j that automagically constructs a graph, it's more of like a low-code kind of a thing. But then you show going into using Langchain. Along with the idea that these libraries are evolving very rapidly, as you were saying, what's the level of difficulty of working with them? Like, are they really approachable right now? Or like, are there any caveats you could give about that? Because I think that might help a lot of people trying to get into this. For sure. Yeah. So that Neo4j tool that I showed, that was put out by kind of the research arm. It an experimental thing um you know it's very much in beta but it's completely no code so all you do is you go to neo4j and you can spin up a free aura database or a db they call it and um is there any code demo uh Yes. Okay, we're going to come back to that one. So go, yeah. I don't know how to bring that question back up onto the screen, so I'm going to need some help there. You got it. Okay, cool. So yeah, all you do is drag and drop your file in. It creates the graph, and then there's a button you can click to visualize the graph, and then there's the question bot right there, so you don't have to program and then there's the question bot right there so you don't have to program anything it's super slick um what I showed for that uh thing with SQL and um the graph I I created the code myself and I think that's actually going to be a segue to that question if you can put that question up on the screen so yeah we had a question from Greg. There we go. Yes. Okay. So Greg, great question. If you go to, there's the two links and I think the slides are going to be made available. The, I am going to quickly scroll back to that though, just to get those links on the screen for people. Okay. So these two links, the top one there, that's Paco's article, the bottom one there is my article. Both have code in them. Mine, I can't remember Paco, does yours have a GitHub repo? Mine has a GitHub repo with it. And you can run top to bottom, yeah, just starting with the raw data, running it through Senzing, and then there's Jupyter Notebooks notebooks and it goes all the way through the whole thing um so absolutely code is code exists uh another question just came in let's see let me go back through uh christine had asked about support for rdf and like we're working in graphs. So, and I'm going to speak from the Neo4j perspective just because that's one that I'm most familiar with, but like RDFs can be pulled into any number of other graph databases. Neo4j has this thing called NeoS semantics. It's a package that works with RDFs. So yes, absolutely doable there. If you go to my GitHub page, and I'll tell you that it's just GitHub.com slash CJ 2001. There's a whole series there called bite-sized Neo4j for data science. And if you go into there, there's some videos and some content about how to create graphs from RDFs in there. So that's another place you can go for that one. Fantastic. Let's see, Mitash had a question. Is it preferable to have the knowledge-based schema predefined or use a tool like Knowledge Graph Builder from Neo4j and just let it run wild? If, okay, so if you have, if you're able to define your schema in advance, great, I love that. It's really hard though if you're talking about raw text. You could define a schema and you could say, I expect my verbs to be these things. I expect my nouns, my subjects and objects to be these things and then throw away everything that's not those things. You could do that. You're missing data when you do that. The Knowledge Graph Builder is, you know, it's great at detecting automatically what that schema and building it for you as you go. And the nice thing about graph databases is that you don't have to have a schema. It will create one for you, but like, you know, it's not like SQL where you say I've got a table and it's got this schema to it. You know, in graphs, it's schemeless, schema free. So you can say like on the fly, okay, now I'm gonna bring in a whole new thing that doesn't look like anything else in here. And that's fine. That's great. One question I have for you, probably a lot of people who are more familiar with working with say data warehouses, data lakes, they're used to having some notion of schema predefined, um, in your experience working on graph projects, uh, how often are you handed schema ontology control vocabulary upfront? Like how often does somebody actually provide that for you when you're starting out? Rarely. Exactly. And I mean, it could. I've had people who've given me what they thought was a good schema, and a stack of PDFs. And then we realized that that's not a good schema. So I've had to go the other way of, yeah, you said you wanted the graph to look like this, but you're actually missing the data that you're looking for by giving that schema. So I am more in favor of bring data in and find reasons to filter it out than starting with a very prescribed schema. That's just me, though. So iterate, iterate as you start to find out. Oh, schema and ontology. Ooh, now we're gonna. Okay, so. This is a tricky question. This is a tricky question, I love it. Okay, so when we talk about schema, what we're doing is we're making the database people in the world, we're making their hearts go pitter patter, right? Because we can understand that there's this concept of a table and it's got columns and it's really easy for us to select this column where this thing from that table. Okay. That's when I think schema, even when we're going kind of into a NoSQL domain, you know, we still are providing terms that we're searching over. When we're talking ontology, now we're getting messy. You know, we're getting into language and language, like I said, is messy. So like understanding how things relate to other things in language is, is, is a little more complicated than that. But Paco, I really want to hear your, your thoughts on this one. Well, it gets very messy because, you know, people put so much work into trying to resolve, uh, standards for ontology and RDF and L and all that. And the more you dig in, the more you find it doesn't really resolve unless you're in a tight domain. And then Google came along and said, hey, we have our version of RDF, we're going to call it schema.org. So now we're just going to confuse the whole thing. I think there's a lot of wisdom to what you're saying, like get your data in there and see what's actually there before you can really start to build out a data model. I mean, maybe if you're working in like catching bad guys and you have the, like Friedrich Lindenburg had that model of like event link, sorry, risk link event, you can have some kind of a use case specific model, but otherwise good luck because uh you've also got taxonomies which are different from ontology and probably woven through them and you probably start there and you've also got different different kinds of controlled vocabularies that get layered in uh so yeah it's tough bringing that back to the RDFs, this Barack Obama graph I tried to do it on, I also did, I was trying to create a graph once with RDFs off of Wikidata, and I wanted to create this graph of brew pubs in like England, I think, or no, Ireland. And I mean, why not? I like beer. And just sitting there and trying to understand what all I needed, what relationships or Q values, there's Q values and there's P values in Wicked Data. One of them is the relationship and one of them is the note. I always get it backwards, which one is which. But like I had to know absolutely every single relationship, every single Q value that I needed to look for. I couldn't just flexibly look for it based off of just language. And there were so many of them. With Barack Obama, I think I wound up coming up with a list of like 80 different relationships. And I still probably didn't hit all of them right I just it's it's just that it's when you're when you're talking about a very prescribed schema where you have like these P and Q values you're really locked in to what it is that you can type versus when you're talking about language and I can just be like tell me everything I care about about pubs in Ireland you know that's you're gonna get a lot more information back and not be locked into a very specific type. Ooh, what do we have here? Oh yes, I love it. The joy of pubs. So yeah, it's a really great question. Thank you for that one. But yet, I'm going back to that point about the importance of getting a data model, getting some kind of ontology. When you are using GraphRag, having good definitions between the nodes, good descriptions, semantic descriptions on relations, what impact does that have if you have just sort of a haphazard ontology or have a good one? Yeah. And that's the hardest part too, right? The relationships are the hardest part. Named entity recognition, you know, that's a fairly mature field in NLP, but getting the relationships as they link those entities together really hard. And if like, you know, there was that example where I was talking about using my ugly knowledge graph to figure out where Barack Obama was born. And I had to know inside out what, you know, what relationship type I was looking for in order to do that query properly in Cypher versus, you know, if you have a language model and I'll need to sit down and actually use that graph and throw it into, you know, this same approach like I did in the blog post, having a language model that would understand, you know, if I said, where's Barack Obama from, you know, okay, well, what verbs, you know, in that graph, what relationships could apply here, you know, so bear the dilemma of born, be could apply, could apply in certain cases, like if it's a place, but if it was, you know, if it was Barack Obama be president, okay, that's not going to be, you know, so letting the language model understand and have a little bit of flexibility and play over that, that schema that's already in this ugly graph could potentially help. There's a question Alan has, which is very closely related here. Can you generate a fuzzy ontology that includes nearest neighbor terms? If so, would it be helpful as a as a spring point okay yes and i can i can add something yeah i because because we talked about this before i think you just need to put that one out there that was a great one well with the uh contextual the nearest turning the nearest neighbors into a summary. What was that called? There is an idea. We're working with it. A little bit of a preview for a couple of upcoming episodes, but one of them is the idea of how do you link this structured and unstructured data uh if you look at the graph you look at a node and understand a textual description for like here is bob smith bob smith lives at 101 maine and is a member of xyz organization if you can start to to basically give a contextual summary uh for a node somewhere in the graph then you can use that as part of your embedding model for how to draw connections between what is the prompt, what are the text chunks, what are the elements in your graph, but have them contextualized so that if you're talking about MLP, do you mean natural language programming or do you mean low neuro-linguistic programming? you can start to make those kind of distinctions by having better context. The other thing though, too, is if you start to end a preview for maybe if we can get David Hughes and Amy Hodler up here, a preview of the idea of if you take a lot of embeddings of your SVOs, the subject word object triples that you generated as you're showing, and just put those somewhere kind of fuzzy and have embeddings of them, then it's a numbers game. The real relationships that tend to happen between certain types of subjects and objects will start to pop out. So we'll show that a little bit later on, but there's some interesting work in the space of relation extraction. And yeah, you're calling it fuzzy. You might want to call it associative. You might want to call it probabilistic, but all along there. I like probabilistic. Let's see here. Okay, great. Okay. Shared how the graph ML was able to get superior performance compared to word vectors. Can you elaborate on that? What features did you use? Did you use the graph embeddings as part of your feature set? Excellent question. Thank you, Matesh. I I did not use anything other than this approach of fast RP. Basically, you can kind of think of it as this money Carlo approach where you've got a node, and you start at that node, and you're going to randomly then hop out to other nodes and see where you can get to. And you're going to do that a handful of times and see which nodes you hit and you mathematically use that to create the vector. There's nothing about the any node properties that go into that approach. It literally is just what's connected to what else. So there are other, that's what I used, was FastRP just because it's incredibly, incredibly simple. And I was just wanting to demonstrate the power of just looking at what's connected to what else. But there are more sophisticated embedding approaches to graphs where you do bring in embeddings. GraphSage being one of them, there's a handful out there based on neural networks. And so then you are embedding property information, but that's not what I used in that particular one. And if you want to know more about that one, again, if you go to my GitHub which is just my GitHub alias is CJ2001. I have a whole bunch of links to blog posts and stuff in there and you'll you'll actually find a big blog post on that particular approach if you want some more information along with the code. fantastic how precise do natural language queries have to be to query knowledge graphs effectively how are vague queries handled oh this is this is a great question it's a can of worms too the answer is it depends um it depends on a lot of things for example what how good is the prompt so when you're when you're typing something to many of these bots, they've been told you're a helpful assistant, you have information on XYZ, things like this. And sometimes they include what we call few shot learning. So for example, if you're asked this question, here's a good answer. So when you give the prompt a lot of information, you can get away with a lot more on your queries in terms of how vague they are. But if you are, if you have not told that prompt much, if it's a maybe a less sophisticated model, you have to be much more specific. Now, if you read my blog post, you'll see that I was actually pretty specific in what I asked it because I did not spend any time doing anything with the prompt. The prompt was very basic. I think it was basically you are a helpful assistant because I didn't want to worry about prompt engineering and prompt tuning and things like that. So you'll notice like in that union cabs example i i had to tell it union cabs and companies related to them in order for the um for the bot to create the correct cypher query and i will tell you that was that took me some time i don't want to you know gloss over it you might read the blog post and think oh she just thought to ask that question immediately no i actually i spent a lot of time futzing around with the bot just to see what you know what i needed to do for it to generate the right cipher that's probably a whole blog post in and of itself right there but um but yeah it does depend in the case of a narrow and well-defined domain with a high requirement for data reliability, do you have any recommendations for tools and pipelines to map a schema-less knowledge graph to an ontology? And how about converting natural language questions into knowledge graph queries reliably? Okay, let me take the second half of this question first. Conver uh natural language questions into knowledge graph queries reliably um there is a built-in uh function in lang chain which is why i picked that one that will uh do knowledge graph rag um and that's in the blog post so if you go to the blog post you'll see that code there um the first half of that query i do not have a recommendation for tools on that one do you paco yeah i'm gonna put something into uh the comments right now but um there is one company i know of that's been doing a lot of work in that area called whyhow so whyhow.ai and they have a tool a knowledge graph builder uh you can import your triples if you're doing construction outside, like we were just showing. But that's one of the areas that they're looking at is if you have a well-defined domain and you have some strong notions of ontology, how things should fit ideally, how can you try to align the data that you have? And yeah, there's some others definitely. And also this area of entity linking is something that's been a little bit, people haven't been paying attention to it for a while. If you look at the spacey pipeline support so far, but there's some new work. So I think that we'll see more coming out. Yeah, that was so hard and that was why it was so ugly. And then I was crying tears when i saw how easy it was to do in chat gpt this morning so the other thing that's been on the table for a long time too is like coreference it's a really important problem when you're doing prompts and it's kind of been ignored in nl for a while, except for maybe core NLP. Yeah, I think Tomaj Britannic wrote a really nice article about co-reference. If you go look him up on Medium, he had a, this is a couple of years ago now, but it was a good article. Yeah, and there's another thing here too about Ask News. So yeah, a little shout out. Ask News is a project that comes out of emergentmethods.ai, and here is, I'll just put a knee to the arrow for it. And they're looking at fine tuning models like Liner for NER, and also, what was it, Phi3 Mini, for looking across the world at different news sources and being able to understand bias in news, but being able to put together a nice pricey report based off of, as Claire was showing here, going out, grabbing the news sources, blending them together into a graph. Good, okay, yeah, Tom got it too. Okay, oh, Roberto. This sounds like a Roberto I know. Hi, Roberto. This sounds like a Roberto I know. Hi, Ro. I'm a musician. One of the hardest things to do is to find people that might like my music. Beyond text analysis, are there tools to help with graph analysis of music? Interesting. Yeah, that, that, hey. Clara, I'll let you take an idea um there's tons of data sets out there have i seen it grabbed i i i don't know that i've seen that data turn into a graph um but it could be a lot of fun especially like as artists like um co-produce with other artists or maybe you know different um the production houses and whatnot i i think that there there could definitely be a lot of fun there tell you what roberto challenge show it to us make it yeah yeah there okay for whatever reason my search engine is is not doing too well right now. There has been some interesting work published by Spotify on how they worked with graph neural networks and doing graph representation of preferences. In the hardest cases they had were with audio books, but it also applies to what they were seeing in user preferences. Short answer is they spent a lot of money to be able to get just a little bit better than they've been doing before. Absolutely, let's go. And I mean, there's other things you might look at, like what Indora has done. And I mean, if you think about Spotify and their recommendation engine, they're really good at, you know, recommending music to you. So you got to know that there's a graph in the background there somewhere, right? And thank you, Fernique had posted here. A friend of ours, Dennis has a great project. I've been wanting to post this online. Oh, yes. It's about Afrobeat. So that's actually a really good example there. Very cool. And while we're at it, let me put in a link for GraphGeeks. So Dennis is right here. Yeah, there is a group called GraphGeeks.org, and definitely give a shout out. It's an excellent community if you have questions about getting involved in Knowledge Graph projects or learning about these kinds of technologies, it's a great place to go and ask questions too. Yeah. And both Paco and I hang out there. They have a nice Discord channel and Discord server and just a lot of really super helpful people. Awesome. Alrighty. Is it time time i think we should do a drawing i think it's time to do a drawing i can't wait all right here we go okay big bucks and no whammy spinning random numbers are being generated left and right. 12! Our lucky winner is 12, which is a great number. I love 12. And what do we, oh, on the channel, let me get back to there. Okay, our lucky winner. Our congratulations to Christine. Christine Kieran, who has just won. And Drummle here, I should announce. From Kaylee and Suzanne also, can we announce which game we're going with? Which game has Christine just won? Actually, no, wait, I'm going to take that back. I was instructed about this. I remember my instruction. We're going to have to figure out, we'll be back with you. We have a set of games, but we also ran into some restrictions like where we can get what shipped. So we have a set of board games that are kind of on a data science theme, not all. And we're going to have to talk with Christine and find out like geographically where we can ship what. But again, congratulations to Christine. And thank you very much for participating in our draw. Also, thank you very much, everyone for uh wonderful questions here q a definitely thank you again paco and sensing for hosting me wonderful talk claire thank you so much thank you if you would like to get in touch with us and learn more about any resolution uh we have a qr code there that has the url and also there's a way to sign up for our newsletter. We'll be having more of these usually about every month, maybe even more later on. But we'll be having our event newsletter. I know that, Claire, you've got some talks coming up, it sounds like, at Day-to-Day Texas in January. I will be at Day-to-Day Texas, yes. Great conference. Everybody should come. Huge graph focus. Me too. I'm really looking forward to that one. That's one of my favorite conferences. And I'll just add to, if you want to get a hold of me, we'll be doing more of these. So we'd love to see you here. Please get in touch. And with that, thank you, Claire. Thank you all. Thank you. you, Claire. Thank you all. Thank you. We'll see you next month.
GraphPowerHour EP2 GraphRAG good bad ugly Paco Nathan Clair Sullivan
Thanks to the GenAI revolution, more people than ever are turning to graphs to solve problems like retrieval-augmented generation ( GraphRAG ) and minimize hallucinations common in GenAI models. Graph structures excel at representing complex, interconnected data—critical for improving the contextual understanding that generative AI requires. By leveraging graphs, AI systems can better discover relationships, link entities, and enrich data, making them more effective at handling real-world complexity. As AI evolves, graphs are emerging as key enablers for more sophisticated, context-aware insights.

However, applying graphs to back GenAI applications is not without challenges. Extracting nodes and relationships from unstructured text via natural language processing (NLP) is difficult, as language is rich in context, idiomatic expressions, technical jargon, and implicit meanings that don’t neatly fit into graph structures. While large language models (LLMs) reflect general language usage, they often fail to capture subtle nuances and technical details, leading to oversimplified or inaccurate graph representations. This disconnect between the richness of language and the structured nature of graphs can hinder a system’s ability to generate significant insights.

In this talk, we will explore the use of graphs "from the trenches," examining real-world challenges encountered when applying them in complex systems. While graphs offer immense theoretical potential, translating that into practical applications often reveals unforeseen difficulties. From handling incomplete, ambiguous, or unresolved data to scaling large, interconnected datasets, the journey is rarely straightforward. We'll also discuss the challenges of integrating graphs with AI models, where the structured nature of graphs can clash with the messiness of real-world information. These practical insights will highlight the gap between the idealized use of graphs and the realities faced in the field.