 Introducing Neo4j 5, ushering in a new era in graph technology that is transforming the way we use connected data. With this latest update, Neo4j is more powerful than ever. You want enhanced performance? Neo4j is unmatched in the market. Neo4j was already 1,000 times faster than relational databases, and with version 5, Neo4j widens the performance lead with new features like K-Hop query optimization, graph pattern matching, and granular indexes. You want your data to scale? With 5, Neo4j has unbounded scalability. Autonomous clustering lets you scale out horizontally with less manual overhead and better use of computing resources. And when your datasets lie in multiple databases, Fabric instantly helps you create a composite graph of insights hidden in disparate silos. You want agility? Neo4j 5 has unparalleled operational agility. Neo4j fits your IT strategy whether it's on-premise, cloud, hybrid, or multi-cloud. And with Neo4j Ops Manager and a continuous release support model, it's easier than ever to manage your database in the cloud. Neo4j Ops Manager and a continuous release support model, it's easier than ever to manage your database in the cloud. Neo4j 5 helps you achieve unbounded performance, scale, and agility. Find out more at neo4j.com so Stamina For For over a decade, Neo4j has been helping the world make sense of data. Today, Neo4j, the leader in graph database and analytics technology, is unlocking new possibilities in generative AI. Together, Neo4j and large language models allow you to use a public, pre-trained LLM with your own data, subject to your privacy controls. Reduce hallucination through a powerful combination of deterministic facts and probabilistic conclusions. And enhance explainability and transparency through explicit knowledge representation for getting transparent reasoning and explainable AI. transparent reasoning and explainable AI. Neo4j's scalable and flexible technology seamlessly integrates with generative AI frameworks like Langchain, Vertex AI, OpenAI, and beyond, democratizing access to the world's information while setting a new standard for AI accuracy, transparency, and explainability. Unlock new possibilities in generative AI today at neo4j.com. There are many types of databases today. The tried and tested relational database, the new and popular DocumentDB, and many others. And all of these have their strengths. But one place they all fall short is in handling complex connections between your data. Sure, any database may be able to return friend of a friend queries of three or four degrees, but what if you need 20 degrees and need it quickly? That's where Neo4j comes in. Unlike other databases, Neo4j Graph Database doesn't need to compute the relationships between your data at query time. The connections are already there, stored right in the database. Because of this, queries of deeply connected data are orders of magnitude faster. We get used to the limitations of the tools we work with. Neo4j blows those limits wide open, enabling queries never before imagined at speeds never thought possible. That's why Neo4j has become a key technology driving business critical applications for hundreds of Fortune 500 companies, government agencies, and NGOs. Learn more at neo4j.com. Learn more at Neo4j.com. Are you interested in graph data science but don't know where to begin? The Neo4j Sandbox is the best way to get started and learn graph data science basics with interesting preloaded datasets like fraud detection, contact tracing, and Twitter trolls. It's easy to jump right in, follow our guides and explore visualization with Bloom. There's no download required and you can get up and running in less than a minute. Try it for free today. Ketutak សូវបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបានបា� Graph databases have been the fastest growing database category for the past decade. The reason for this is simple. Graph databases are highly optimized to understand and work with the complex relationships between your data. Unlike other kinds of databases, a native graph database doesn't need to search for data connections at query time. Those relationships are already there, stored right in the database. Because of this, queries of large, complex, connected data are orders of magnitude faster. A graph database doesn't necessarily need to replace your current database. By adding graph capabilities to your current infrastructure, you can keep the technology you already have, but greatly improve it with the power to uncover rich data connections, reduce time to market, and run faster queries. Taking advantage of the connections that already exist in your data enables you to get ahead of the competition. Every business needs to leverage data relationships, and leverage them faster and more efficiently. Graph databases deliver those capabilities. The speed and efficiency advantage of graph databases has driven new, real-time applications in fraud detection, customer 360, machine learning, and knowledge graphs, just to name a few. Because of this, graph databases have become a key technology creating competitive advantage for hundreds of Fortune 500 companies, government agencies, and NGOs. We asked 100 senior tech executives, CIOs, CTOs, and chief data officers what they need to bridge data silos, boost AIML projects, and open up new revenue streams. A massive 88% of them said the same thing. Knowledge graphs. But why? Leaders know the value of data, keenly aware that it holds the answers to their most pressing business questions. They need insights to improve decision making and enhance business performance. Enter knowledge graphs. A knowledge graph is a map of everything an organization knows about insights to improve decision making and enhance business performance. Enter Knowledge Graphs. A Knowledge Graph is a map of everything an organization knows about a given topic. Knowledge Graphs help business leaders by functioning as a non-disruptive insight layer on top of their existing data landscape. Knowledge Graphs drive intelligence into data to significantly enhance its value without changing any of the existing data infrastructure. Knowledge Graphs enhance existing technologies by providing better data management, better predictions, without changing any of the existing data infrastructure. Knowledge Graphs enhance existing technologies by providing better data management, better predictions, and better innovation. Last but not least, Knowledge Graphs lend themselves to multiple use cases like fraud detection, digital twin, patient journey, bill of materials management, recommendations, and IT operations across many industries, including financial services, healthcare, life sciences, manufacturing, supply chain, and retail. The C-suite is waking up to the possibility of Knowledge Graphs, shouldn't you? Here we go. Hello, sorry. Hello, everybody. Good morning, good evening, good afternoon. And thank you for joining to this Neo4j going meta episode. It's issue number 26. And today we look at a benchmark from a couple of guys at DataWorld that was released end of last year, early this year, I think on the role and on the impact of knowledge graphs and how they make LLMs more accurate. I think that's what we talk about today. It's very interesting in general, just this topic. But before we dive in, I should maybe give you a little space. Jesus, hi, and thank you for joining. Yeah, you're totally right and we'll provide all the details and give all the context but you're right, yeah, really excited to speak of episode 26 which is great and yeah it's a bit of a special one. I mean instead of you know starting from from scratch and taking a problem we are yeah we will start from these paper, this kind uh benchmark that was published a few months ago and and we'll uh you know unpick it as we said in the title we'll we'll you know break it down you know how do you say like deconstruct it and understand what's in there and and try to put it in the perspective of property graphs and uh and the knee of the j and i hope it's going to be a we will, in the process, touch on some topics. I like doing that when we refer to previous episodes and bring things that we've already discussed. So there'll be a bit of that today as well. So, yeah, that's the plan. Any announcements that we have for the coming days, weeks? I know we sometimes use this sort of an introduction to remind people that uh you know about the other sessions that we run as well or the events that we're involved in yeah we can we can do that at the end but obviously what i what i what i want to do uh is is mention the the training series that's that's coming up starting next week and i can post a link in the chat right now as well so if um if you are interested in learning new things, getting up to speed with Neo4j, especially in the Gen AI space, we have a training on how to deploy high-performance RAC applications on the 21st of March, so in two weeks' time or so. But nevertheless, if this is not so interesting to you, then obviously we have an intro session. we have something on geospatial data and others so if you are interested in this check it out and register for for the trainings but i'm i'll talk about this at the end of of the episode so i wanted to um don't divert too much but yeah i'm looking i'm looking forward to today i think to to talk about i mean there's always uh benchmarks and and everybody publishes them i don't think um we have to go into into much details generally but i think maybe a little little heads up so you know, whatever you do there in terms of benchmarking, it always needs to be looked at with some sort of grain of salt. I think this one is, is, is maybe a little different than the usual. Um, I would say in a bunch of commas, kind of like official or something like a, uh, a benchmark done by somebody, um, obviously promoting their product. So it always comes with this caveat that you say, okay, somebody wanted to make a point and obviously you pick the queries that kind of like work well or the setup kind of like is in the favor of this. So everything is always a little bit biased, I would say. This is not really objective usually. biased i would say this is not not really um objective uh usually um so you know it it all depends a little bit on on general the background the setup all of that so if you keep that in mind then then whatever the benchmark you look at i think that's important to to have um as a as a you know always always keep that in mind but generally generally, I think today we don't want to moan or we don't complain about benchmarks. So I think we take this relatively neutral, I would say, and we try to see what they did in this benchmark and how they see the LLMs and knowledge graphs work together. And then we can see, we can give our perspective. And this is not necessarily wrong or right or something like this. It's more maybe just a little different perspective here. And I think that makes total sense and it's a good way of looking at things. Yeah. And sometimes, you know, a lot of it is I mean this is a message that we're totally aligned with I mean and probably let's kick off a bit of an introduction so yes we're talking about this benchmark that you know my good friend and Juan Sakeda who works at Data that World and a team of individuals at that company run a few months ago. I think it was October, November when they published it. And the results were fantastic because what they were exploring, and you can probably, I hope it's readable from the title, but we have the links, right? Alex will be sharing these. So these are incredible. And basically there are, I mean, Juan is a big proponent of this idea of uh knowledge first i mean we're and actually that that connects with something that jim jim weber our chief scientist uses a lot in some of the keynotes right he says he used this quote i don't know what it comes from but like we're drawn drowned in data but we're starving from knowledge and sometimes you know that that is the case, right? And this example was a great exercise, a great experiment. So what they looked at, basically, it's we'll go into the details, is these days with LLMs, and we've done a bit of that in previous episodes, we are seeing as one of the use cases, one of the users of it, is have them generate code for us. So we give them a description of the query that we want, and we get the LLM to produce the structured query, right? And that's applicable to all languages, all models. So what Juan and the team did is, okay, so we can try to have an LLM generate some SQL queries in this case, or we can create some knowledge representation of the content of the database, pass it as context to the LLM, and then see if that improves the quality of the queries that the LLM produces. And that's the idea. LLM produces. So and that's the idea. So again, the premise is, we believe that a knowledge, explicit knowledge representation should improve the quality. And exactly. I mean, that's, that's what the results produce. So that you see some, I've collected a couple of posts from LinkedIn that were quite announced some of the results, you see that this, the improvement was substantial substantial and we're going to understand why. And what was interesting is that that was really impactful. I mean, lots of people reposted it and some people or some other organizations repeated, replicated that with different technologies, which is, again, an interesting and interesting thing. The idea we're going to look at it from the point of view of graphs, of course. But Juan and the Data World team, as you can see, they call it the impact or the importance of knowledge graphs on the interaction between LLMs and SQL databases. But they look at graphs in a slightly different way than we do. And that's exactly the content of today. But I was saying that some of the teams replicated these exercises, and they came to similar conclusions, right? And the idea, we can see that it's pretty intuitive, right? So sometimes the knowledge is hidden in the data, and we have to distill it. We have to make it explicit. That's something that we've talked a lot over the 25 previous episodes, so you will see that some of these things resonate there. So that was the premise. So Juan and the team believed that that was the case and they proved it with a collection of... I mean, we're not going to replicate, of course, the benchmark, but I'm going to give some clues and show how we could set it up in an Neo4j environment and what would be the advantages of doing it in that way and give an example of how could we run one or two queries. And if anyone gets inspired and wants to replicate it or do it in its entirety, that would be a fantastic follow-up to these. So, yeah, that's – and I'm following with one, not directly with Juan, which we'll go back to in a minute, but to someone from DBT where they – one of the things that they do, DBT is a company that does – takes care about the data transformation, right, in data warehouse environments. And one of the things that they have been going to market with is this idea of a semantic layer, which is a very, very similar, I mean, conceptually is exactly the same concept, right? So they believe that layering semantic, a business oriented representation of the data that's storing your data warehouse will enable, you know, consistent and more efficient and richer use of that data, rather than having multiple queries replicating the same logic in different BI tools in different consumers, have a single layer where you centralize that logic, you have this kind of semantic layer, semantic representation, and then everyone reads from the semantic layer. And that's exactly what they did. They took Juan's example. We'll look at a little bit of what that looks like. But the idea is what you see on the screen. So layering this semantic knowledge has a number of benefits. But now in the context of LLMs producing code, number of benefits. But now in the context of LLMs producing code, this is super relevant and super valuable additional context to get better quality. And again, this is the idea what I was mentioning, right? So they saw this layer on top of your data, so that consumer applications will have a consistent and sort of canonical if you want understanding some of these topics we've used a lot, right? Is this idea of having an ontology? So Juan will call that an ontology. These people will call it a semantic layer. But again, the idea is pretty much the same. So let's go down to the details and let's unpick the experiment. So what you see on, and these is two screen captures from the paper that you've just shared, Alex. I believe you put the PDF, right? So these are the two systems, the two scenarios that they compare. So the first one, so data lives in a SQL database, in a relational database, and it's a pretty, I would say, highly structured schema, and it's all public. We'll show the GitHub repo where they published all the assets. For that database, we have the DDL. We have the description of all the tables with the columns, with the data types, with the foreign keys, with the primary keys, a formal description of what's in the database. But that's a database level description. It's a low-level description of the structure of the database but that's a database level description it's kind of a low level uh description of the structure of the database now if we pass uh that sql ddl this description of your database along with a question to an llm without no without any training and and it is a zero shot prompt so you just literally pass the question in the context which is the ddl you just literally pass the question in the context, which is the DDL, GPT-4 will produce some SQL. And we'll see, I mean, we will not see today, but if you check the PDF, you'll see that the results are pretty rubbish. It falls in a number of ways. And that's understandable. That's understandable. And we're going to see why. Because sometimes, often, the questions are described in terms of business terms that don't exist necessarily at the data level. So it's pretty obvious that, and here we're surfacing a problem that's not new. When we define a database, if you think of the process, we start interestingly with an entity relationship model where we identify the entities, we identify the relationships between them. We kind of build the graph schema, that's funny. But then at some point we translate that into a collection of tables, we normalize it, and we end up with a low level representation. That's great for querying, and I'm not gonna criticize the SQL model, I mean, the relational model, it's been there for 50 years and it's super versatile and powerful, but there's a problem is that we've lost a lot of the semantics of what our data actually means and what happens i mean if you're an expert if you if you know your data model then you will be able to write queries but if not you will find a gap and that's exactly what happens to the ddl sorry to the deal to the ll, the large language model, tries to generate questions in schema terms, in SQL terms, based on business or domain terms, because there's this gap. And that's exactly what we have to fill. And that's the role of the knowledge graph in this context. And that's what we see on the right-hand side. So what Juan and the team did is they said, okay, we have same database and we have to query it with SQL, but what we're gonna do, instead of providing the SQL DDL, a low level description data, level description of the data, what we're gonna do is we're gonna provide an ontology. An ontology, as our regular attendees know, an ontology is a domain description. It's a schema, It's a graph schema. It's a description of the entities, the labels in your graph, the relationships between them, how they're connected, the properties. So it's a business. It's a semantic description of your data. So it's exactly that gap that we're missing. I mean, it's kind of an explanation of what's in my database in business terms. So that's, of course, if we give that to an LLM, we're giving them a massive advantage. Now, the question, and we'll get to that, is what's the cost in building that? But it's definitely worth it, as we're going to see, because the results are significantly improved. Now, the problem is not the problem, but the reality of their approach is that they took a rather complicated architecture, as you can see there. So they look at graphs from the RDF, you know, semantic stack point of view, which again is something that we've discussed several times in this series, and well, they provide the ontology. They ask GPT-4, the LLM, to generate SPARQL instead of SQL. Why? Well, because they want a query that's described in terms of the concepts defined in the ontology. So it's a query described in terms of semantic business oriented concepts. Now, the problem is that sometimes at some point, you're gonna have to translate that into SQL because the data lives in SQL. And that's where they have this additional level here. Not only you have to describe to your domain, but you have to say, my domain maps to my data in this particular way. So basically what I define as a policy or what I define as a policy holder, actually maps to the subset of the records in this table. And that's where things become even more complicated because, you know, you have to do that, right? Your LLM is going to generate great sparkle, but then you have to translate that sparkle into something that the database understands. And that's the approach, which is great. As you can see here, for them, the knowledge graph is slightly different from what we see in Neo4j commonly, where we combine instance data. I mean, everything, we live in a graph world, which means that our description is part of the graph as well as the data. So we're going to see in our approach that we have a much more simplified architecture where we have a self-describing graph that contains the ontology within, right? But that's what they... Let me pause here and see if you have any questions. Does that make sense, the approach that they take? And I'm still going to go in more detail in each of the components. Okay, good. But I think it is interesting to see, but I think it's an approach that I take. I mean, I guess that's what I wanted to say in the beginning, is with different backgrounds come different approaches to things. I mean, we often see this. And you try to take what you know and adopt it sort of. And I guess that's, they come from an RDF background. Obviously this is probably why they chose this route. In a different background, you might take a different approach and I guess we show a little bit how you could do it with the property graph model. I guess we show a little bit how you could do it with the property graph model. But it makes sense. I mean, it's still already better than just standard approach. And I think it makes sense to add the semantic layer, to add this ontology towards it. It's just, you know, this double mapping is also then, I would assume, but maybe I'm wrong here, but I would assume that this double mapping could lead to errors that are not really, I mean, the left model will create errors because of the LLM being out of its depth a little bit and doesn't really know what to come back with. And the right model could come back with errors due to some mapping issues that are, I don't know how much this virtualized knowledge graph is hand manually curated or if they also had this done automatically by an AI-assisted tool, but that could then lead to issues obviously because you transfer SPARQL into a knowledge graph back into an sql query that i could see issues yeah yeah it's inevitably more complex i mean one could argue that depending on the context uh it might make sense if you uh if you don't want to move the data from where it is right so we don't want to to move the data out of my s database. There's no question I'm going to move that into a full graph environment. But the complexity is there, and we're in this virtualized approach where there's a number of the creation, the maintenance, and then the performance as you can imagine, it's having to translate a Sparkle query into SQL or in any form of virtualized federated type of logical layer, and there's multiple products in the market that do that, we know that the complexity in doing query translation in an efficient way, in a performant way, it's not an easy job. So yeah, totally agree. It's a rather complex, I would say, architecture. But I would say let's put that on the side and let's focus more on how do we and the value of making. Because ultimately what we're doing here is we're making the semantics of our data explicit, which is a topic that we keep talking about here. And we totally align with that and we're going to see that there's value in this so creating this ontology and mapping it to our data is is definitely uh uh an investment worth its worth worth the effort right and and and they have quantified it with the results uh of the of the benchmark and they show how how the the quality of the um i mean it's a great way to automate that because you know otherwise how would you quantify the the value of the... I mean, it's a great way to automate that because, you know, otherwise, how would you quantify the value of this semantic layer? Well, now we can ask the LLM a collection of questions and see how they improve or how they don't, right? So, it's a way of validating this idea of let's let's bring the knowledge on top of our data well i think it's great to see that even if you are using a more not judging here but a more maybe complicated architecture still improves your output by an incredibly big amount so no matter where you come from and what your what's your tech stack kind of is if you add a knowledge graph to your to your gen ai llm interaction it will boost boost your performance and that's that's a good you know it's an absolute good conclusion here we've seen that uh that it has i mean you can you can repurpose that i mean we've seen that all the time and again, right? How an explicit semantic description of your data can be used to validate the quality of your data. It can be used to dynamically generate data ingestion pipelines. It can be used to, you know, so many now to improve the quality of LLM. So materializing and making this knowledge explicit is definitely an investment that's going to pay off. So that's the bottom line and that's their position. So yeah, let's move on a little bit and let's explore each one of the elements. And again, these, I think you have the link to the GitHub repo and we'll go to it in a minute. Yep. But they published in GitHub all the assets. So there's the data, there's the mappings, there's the ontology, there's the questions, everything's there. So the first thing that we see is the ontology, and that's not new. I mean, we've shown many of those over the last 20, 25 episodes. And we see that an ontology, like I said before, is basically a description of the entities and the relationships in your domain. It's a graph model. I picked two fragments there where they described the notion of a policyholder. A policyholder is a class, it's a category, it's a way of saying there's a type of nodes in the graph that's called policy holders and they're identified by URI and they have a label, a human readable version of it, and then they are related to other things. They will have properties, but in this case what we're saying is that there's a relationship that's called hasPol, that connect is an object property, which is RDF terms for a relationship. And that connects a policy to a policyholder. So basically, what we're saying is there's a policyholder definition that connects a policy to a policyholder. So that's the way in RDF. And again, I will not go too much into that. But this is a formal description of the data at the business level. So interestingly, as we're going to see in a minute, the notion of policy holder, and if some of you are looking at the assets in the GitHub repository, you will see that there's no table called policy holder, which is, you know, again, an implementation decision. And that's the kind of thing that I was mentioning before. So these semantics of your data are lost. And only if you understand the DDL and you understand how the data is encoded, you will be able to surface what policyholders are and how they relate to policies. And we'll see that. But at this level, the ontology just captures the collection of entities and relationships and properties with them. Make sense? And it's described in OWL, and we work with this many times in this series, so not much more to add. Yep. The second part is the thing where, you know, it gets a bit more complicated, The important part is the thing with, you know, it gets a bit more complicated, which is where we map that definition in the ontology to the elements in the database, in the relational database. It's this mapping here. And for that, they use another W3C. I don't know if it's a standard. I don't know what the status of it. It's a recommendation or a proposed. But it's this language called R2RML. So relational to RDF, markup language, something like that. I can't, I'll show a link in a minute. But this is a way of basically describing, okay, the instances of this particular type. And in this case, we're defining a policy holder. So, and in particular, the ID, the're defining a policy holder. In particular, the ID, they identify as a policy holder. It's done in this way. Basically, if you can read in here, there is a SQL query. I don't want to click because it's going to navigate to the next page, but you see where my mouse is, right? Yeah. So we're saying that you have to, I mean, this is what you see on the left-hand side, is a subset of the table in the schema. Like I was saying, it's a highly normalized model. We have policies that are a type of agreement. There's a master table agreement, and then there's this joint table called agreement party role, which connects a policy, which is a type of agreement, with an individual, which can be an organization or a person. And they use this role model. So it's a pretty complex and indirect collection of tables, right? So you have to do a number of joins to extract what a policyholder is. And if you read this query, what you're saying is, look at that, And if you read this query, what you're saying is, look at that, we're joining this agreement party role, which is this table here, with policy, right, which actually is a synonym of agreement because they have matching primary keys on a particular column. So you describe the join here, and you filter the tables that contain on the role code PH. So there you go. That's only if you really understand the model, you know that ph is policyholder. So it's a rather complicated way of expressing, I mean, complicated, it's SQL, but basically what we're doing here is we're surfacing that and we're saying when you run, I mean, that's the way of extracting from this relational model the notion of a policy holder. So does it make sense? So that's the kind of thing. And I take just the fraction, but if I believe you put, well, we can have a look. Let me just delete this. And I think I have the GitHub repo. So if you go to the data folder, inside the data folder, I think, here we go, is the R2RML. And unfortunately, GitHub is not doing the proper highlighting of the syntax. But anyway, you see that there's a rich collection of definitions, each one of them backed by a SQL query. So basically you're saying, you know, run this query in order to distill or to extract the surface this particular business level entity so that's that's how it works and that's how the translation is going to happen when we describe a query in sparkle terms we're going to use each of these translation to uh to generate uh the relevant uh sql uh to to hit the database. Does that make sense? Yeah. So this is the second part. So we looked at the ontology, we looked now at the mapping, and then finally, well, yes. You need, like I say, you need an engine. Once you've defined that, you've defined the ontology and the mapping, you need an engine that is capable of, for the generated Spark translated into sql given the given the the the mappings right and that's what i mean i've found um uh this uh open source uh r2 rml engine i suppose juan and the team use their own i mean the the data data that world platform probably uh has their own proprietary implementation. But this is one example, basically. You can go look at it. And that's essentially it. So you pass it a mapping, and it will translate Sparkle to SQL. Right. So that's pretty much it. Does that make sense? So that's pretty much it. Does that make sense? And again, just to complete it, I mentioned before that all the platforms, all the technologies have implemented something similar. And DBT, as I mentioned before, they have this idea of a semantic layer, which kind of replaced, they don't have the notion of a knowledge graph, they don't have a sparkle, they don't use ontologies, but they have a formal way of defining in their semantic layer using yaml pretty much the same thing right so as you can see they the what we call the ontology here they define their entities right they define there's a notion of a policy right and then further down in the same description of the semantic layer they say there's a this party role code which is the column that we saw in the in the sql query they have a natural language text description of it so it must filter on this and if party role equals ph then a party identifier corresponds to a policy holder right so it's a another way of of describing it uh um and and trying to kind of provide this meaning layer of meaning semantic layer on top of on top of the data. So, again, I have the link there if you want to have a look at it. But the concept is pretty much the same, except that, like I say, they don't use ontologies or Sparkle. So, this is basically a replacement of this in yellow highlighted segment. Correct. And instead of having the ontology, what they have is a semantic layer. So they pass the semantic layer to the LLM, and the LLM will generate queries to the semantic layer that then they project onto the underlying relational databases. Okay, cool. Makes sense? Yeah. But again, the idea is the same. Let's make explicit the entities and the relationships and the domain and the meaning of our data so that we can ask questions in a much more meaningful way. So now, I've pasted again the same idea. This is the definition of their semantic layer that they, I mean, initially it's not conceived to be, I mean, you can use it for interactions for interactions with lms but they use it to to integrate with any consumers catalogs bi tools any kind of data applications right okay good so that here's the the neo4j based version and and and you know i i you can see that i i got just halfway because i didn't get to replace sql database and i'm going to do it right now. Because in this case, what we're going to do is, because I've got the question several times. What does that look in Neo4j terms? This is of course Neo4j. It's not a SQL database because our approach is going to be based on the idea of, okay, we're going to move the data into Neo4j. Instead of having a layer on top, is going to be based on the idea of okay we're going to move the data into into neo4j so instead of having a a a layer on top we're going to create semantically described described data right so it's already data that that has built in the the the interpretation uh the semantic interpretation and what that would look like is something like this i mean i don't use this kind of uh um yellow box here but, but that's what it's going to look. So the structure is the same. I mean, we're going to have the data in Neo4j, but because Neo4j is self-contained, self-describing data, we don't need to, I mean, we could, and we've done it again in multiple episodes, but you can ask the graph, and we're going to see it in a minute. We can ask the graph, hey, tell me what your shape is. Tell me which are the entities in the graph, which are the types of nodes, which are the properties, which are the relationships, how they connect to each other. So that's what these kind of back arrow returns. So I can generate, I can give you an ontology of the data in the graph. So once the data is here, you have it for free. You don't need to add any layer because it's already semantic data. So when we ask the question to the LLM without no previous, it's no SQL, of course, it's going to generate cipher. So I've just tried to reuse the diagram. But the idea is this. So what we're going to do is, hey, LLM, generate the cipher for this question and the context, I don't need to produce it, you can get it from the database itself. So query the database for a self-description of what the entities are. And with that, you're gonna be able to generate a accurate cipher. So that's the idea. So, and now is when I'm gonna jump into the code and we're gonna show how we will go about this. We're going to build a property graph and we use some of the elements that we've shown in the past. We use the ontology as a target model. We'll use the data input tool to populate that from the data that they shared in the GitHub repository. Then we're going to see how we can prompt an LLM with exactly the information as we described here. Make sense? Yeah, sounds good. Cool. Good. And then once we have it, the idea is pretty simple. We're going to test how to... We'll test the LLM to generate the... We can do it programmatically and we can do it directly on ChatGPT. So we'll try both. So let's get started. So I think I have most of the elements here. And as usual, I'll make them available after the session. So I've emptied the graph. I'm going to empty, I think, what was it? I have a, here we go, a way of removing all the schema elements because I don't want anything in my graph. So it's totally empty. That's my sandbox instance. And we have here the repository that data world have made available. So we have a folder with all the CSV files. It's small in terms of, I mean, we don't care too much about the size of the data. We care more about the correctness of the queries. care too much about the size of the data. We care more about the correctness of the queries. So we have the ontology, which is here, right, with all the definition of the business entities, all the semantic entities in our data set. And then we have the collection of questions and results. But we'll get to that in a minute. So first thing we want to do is, okay, I'm going to build a graph out of this, and I want to automate most of it. Like I said on the slide, that's something that we did in episode five. You might remember it was called something like ontology-driven knowledge graph construction, and we had a target model. That's the model that we want to build and and our source data can be relational can be whatever but we can automate the generation of a graph and the population of a graph based on on this uh on this target ontology that's exactly what we're going to do because we have uh an ontology available which is which is this one here and one of the ways in which we did it was using the data import tool because we we that we could take the definition here. Remember that here we define classes, their properties, their relationships, and use that to generate a data importer model that we can then bring here. If you don't remember that, I'm going to show it and I'm sure you're going to very quickly remember what I'm talking about. remember what I'm talking about. So this is basically the call. I'm going to paste and explain here. So here we go. So the method is still in experimental mode because, you know, this data input model is, I mean, was evolving at the time. So I didn't want to have it, you know, generate models that were not valid. So that's why I still keep it in experimental state. But the idea is that we pass an ontology as first parameter. And this ontology is just the ontology that we're looking at here. Remember when you do it, you don't take this URL because this is the HTML view of it. It is always the raw one, right? So you can either click on it and get this url or uh just uh right click and copy the link address so that's that's what we're doing right so in this case i'm saying go read that ontology and that's exactly what i'm pasting here right just make sure that this is the one it's serialized as turtle and for this example again i don't want to cover the whole benchmark. I'm going to pick just two or three classes. And I said, okay, I'm going to take policy. And I take these three because they're particularly interesting because they are really far from the implementation in the data as we saw, right? So there's no notion of policy holder. It's actually a subset of the records in the table called role, whatever, right? So I picked these three categories and i identify them with the uri in the ontology i pick policy sorry policy i pick policy holder and i pick agent right so these three are uh all three defined in this ontology and it will uh go and pick the relevant bits so if i run that, it's going to go read it, and it generates a couple of things. It will generate something that is actually a model. It's a JSON file that I can copy here and save and then import from the data import tool. It's a description. It's extracted agent policy and policy holder as I described. These are the properties that have been identified. These are the properties that have been identified these are the relationships that have been identified so all good so i can either copy these and put it on a file or i can use the other flavor of my of this method which is stream produces the output as you see it here but if you do export i believe is it export it is yeah it will save it uh in your local drive and uh that's what i've done so i call it di model.json so i uh you know it does it for me i mean instead of having to copy it and put it on a file so you you choose the your your favorite approach and then what we have is uh like this. I've saved it before, and this is the model. So if I open with, I don't know, for example, with ZLime, that's what it looks like. So it's a JSON file. I will show a little bit the details of what's in there, but basically it's a definition of all the classes and the schema that then I can import from my data importer tool. And that's exactly what I'm gonna do. So I go to say open model and I have it quite handily already here. So I'm gonna open it and here we go. It does a bit of a funny layout, but essentially it's extracted the two, three entities that I want to populate, right? So I have policies, policy holders, and agents. You will see that for all of them, it's systematic at a URI, a comment, and a label, which we can delete. I mean, it's something that's sort of standard in the RDF world, but probably in most of the cases we don't care about, so I can happily delete them from here same in the policy holder i'll delete the uri the comment and the label and in the policy i'll do the same uri uh comment and the label and what's lacking and i'll add it here is a something that I'm going to call a policy ID. But OK, so I've used my ontology to create sort of target model that I want to populate. The next thing is I want to populate it based on the data. And again, in the like I mentioned before, in the data folder, we have a collection of CSV files that I've checked out. I mean, it's public data, so you can do it yourself as well. And and we can import it here. It's very straightforward. Like I say, I have it here. I'm going to import all the CSV files, and my data importer will do a bit of an analysis. For each one of them, it will give me the headers and a bit of a sample data of what's in them. There's plenty of them. I'm going to use only a couple of them to populate this graph, but I'm showing how easy it can be to replicate and move the data into a graph representation. So for example, the policy will come from the policy. CSV, all looking good. And I will select the fields. So the policy effect, let me make it to make it a bit more readable. Because some of the fields have long names. So the effective date will be something like effective date. Easy. Expiration date will be expiration date. The policy number will be policy number. And the policy ID will be the policy identifier. So we're good. And this happens to be the primary key. So I mark it there. So done with the policy, I get the green tick. Then policy holder. Policy holder, as we saw, it's an interesting one because it comes from this file. Agreement, no, sorry, it's party role, isn't it? I believe. Party or party role. I mean, there there's synonyms in this case, so I can say that there's a party identifier and it's a primary key done with policyholder and agent is another party, so it comes from exactly the same file. And this gets hang on where is it I hope I'm making it right so is it a yeah or maybe it's the party same thing so it's party identifier and this is the primary key. So the last bit that we need to do is how do we populate the relationship? So sold by agent is, it comes from the agreement party role file. And yeah, it has the, this is the join table that I show in the slides briefly before. So this is, if you go here, the agreement party role connects the policy, which is agreement with the party role. So that's the the file that I'm using and it contains the two references. So it will have the policy, which is the agreement identifier, and it will have the agent, which is the party identifier. There we go. And the same. Now, it's important here because here's where we are kind of injecting the semantics that we saw in the definition before. So we have to filter because not all of these are agent definitions. Let me bring it up and it will be here. So if we go agreement party role. You see, remember the PH and the AG. So PH means policyholder. AG means agent. So, in other words, what this means is we have to select the fields, select the records that contain AG, and those will be agents, and select the records that contain PH, and these will be policyholders. That's something that, fortunately, we can do here by saying I want to filter, and I'm going to say that we're going to populate agents when the party role code is AG for agent and similarly when the policyholder same mapping I select the file agreement party role and we have the agreement identifier, party identifier, and same filter, except that in this case the code will be ph for policyholder. Anyway, so we have... Sorry, a little bit too quick, but you see how I did it. So what I'm doing here is I'm applying these semantics of saying, I have an understanding of how my data is organized, and I'm building it into my graph in order to now populate explicit relationships called has policyholder and sold by agent. We will not have to translate it on the fly like we were doing in the federated virtualized approach, but we will have these properties populated. So once I'm happy with this model, all I have to do is run the input. I skipped through that, but before this, I connected to my sandbox. All you have to do is pass the credentials, the URL to connect, user password, and you're in. So this is already connected to my empty sandbox here. Just a reminder that this is entirely empty. I've just used it to generate the model, and I can run the input, and it should take just a few seconds because it's a very, very tiny data set, so it's already run, and now we should have something here, hopefully. Let's take a few seconds. So if I show this, we see the two instances that we're seeing before. So we have two policies, policy one and policy two. And we see that they happen, it's a simple data set, the same policy holder, which is one, and they're both sold by agent two. So there is agreement one, policy one has policy holder one, agreement two, and two has policy holder one, agreement two. It's a simple data set. But yeah, you get the idea. So now we have a graph. So we built into the ingestion, the pipeline process, the semantics of understanding this particular fragment of the model but now i have a model that i can query right this already has my uh my graph is already at the business level it's not a physical broken down into tables representation it's something that's self-describing and what i was saying before and remember my diagram is that my model my data i can ask it tell me what's your shape tell me what's your ontology and i can do that very easily and and that's you probably know i can call it call a db schema uh for example no type properties right and this will give me a list of all the properties. So there's a policy and the policy has a policy effective date, which is type string, and so on and so forth. So we can ask our graph to describe itself and provide an ontology in a form. I mean, this is a kind of a more simplified, well, not simplified, because it's as rich as an ontology, but it's serialized as a collection of records, which we're going to pass, as we said, to the LLM. But that's that. And the same with the relationships, right? We can do, I think, there's a APOC, it's meta relationship type property, something like that. And again, we said there's a sold by agent that connects a policy to an agent and there's a has policy. So it's again, this is this is a query that I can run to my database. I don't need to create any separate artifact to to to overlay on top because my data is semantic already. So by passing a combination of these two, which is basically a description of my node types, the properties and relationship types and the entities that they connect, my LLM is gonna do a pretty good job at generating valid cipher. And that's what we're gonna test very quickly. And again, as usual, I'm gonna share the code. This is the usual introduction, right? So I've already installed Neo4j, OpenAI. I mean, don't need pandas for this one, or even RDA Neo4j. But what we do is we connect to, I mean, we set our connection. I'm going to have to input baby again. Yeah, let's do it because I thought it was going to keep it from the last session but looks like it's lost it oh no it's running it sorry i shouldn't have maybe i can still stop it stop stop i don't need it stop stop stop anyway well there you go so we have already um uh set up you know the the the credentials and the and the connectivity with the with with the sandbox that I've just created. And what I can do here is run these two queries that I've just shown you. The one that gets the properties of the nodes and the properties of the relationships. So these are two, I call them metadata queries for nodes and relationships. And if I concatenate them uh that will produce hopefully there you go a kind of a textual description of what's in my graph and it's more than enough for the llm to produce is this readable should i make it a bit bigger yeah i just wanted to say yeah i think this is yeah this is better now so there you go so it's it's telling me there's a label these are the properties that this label has this is better now so there you go so it's it's telling me there's a label these are the properties that this label has this is these are the label and so on relationships connecting from label to label that's it very very simple but ultimately it contains exactly the same information as the ontology and now i'm going to put that in a prompt and this is something that we've seen in the past is very very simple so that's the question that i'm going to go and run in a minute but my prompt looks something like this so i'm saying given a property graph with the structure described by the following list of northern relationships and i'm going to inject here my structure that i've just used so that's dynamically generated from my from my graph write a cipher query that answers this question. And I additionally say, don't explain the query, just the query and nothing more. And I pass the question. And this is where I'm going to pick one of them, one that applied to agent, for example. And I've used this one again, because this is the kind of query that an LLM would have never got right. Because if you scan the list of tables in your relational schema, there's no notion of an agent. An agent, like we saw, is a role that connects to an agreement through a party role, blah, blah, blah. So no way it would have got it. But in a graph, it's suddenly there. And hang on, where am I? Here we go. So this is the prompt. If I run it, it will generate basically it embeds what we've just produced is description of the graph model and it embeds the query. And if I run it, it will produce, this is what I call my open AI with any, at the same time, I think I had here, chat GPT open, because I can, of course, copy. No, not the cell I wanted to copy. Anyway, let's see what it does. And I could manually, if you don't want to do it programmatically, but I can go boom. Oh, no, I don't want that. I don't want to copy the cell. I actually want to copy the output. so how can I do that whoops if I do command c hopefully that will do it and I will clean that and pass a proper prompt here we go so that's that's the question. And it should hopefully do a good job. There we go. So it produces, I mean, it's a pretty simple one, but the complexity here is not in the richness of the query. It's in the fact that we're asking a question at the semantic level. So if I copy this query and I come to my database, boom, it would tell me that there's only agent two, I mean, simple data set, but again, and it's sold two policies. So I'll, again, take a pause here and maybe a similar thing, you know, like I say, we've run it here and it will produce the same, exactly the same query. I mean, this, oh, interesting. In this case, it's taking the ID, instead of doing it globally, it's doing it for, it expects the idea of an agent to return the results for a specific agent. But you know, the idea is that it's using the right, the right schema, which is again, self-describing in the graph. So, you know, long story, but what we've done here, and before I open it up for questions, in case someone from the audience, or you, Alex, have some questions. What we've done is, OK, we can if we have if we're using a platform that does not support semantics, which is like a relational database, I have to overlay it. And that sometimes leads to a complex architecture. Like we saw before, you have to create these artifacts, the ontology, the mapping, and then not only create them and maintain them, but when I run queries, I have to apply a translation that who knows what's going to be the efficiency of that query. Now, if I take my data and apply this semantic definition and create a semantically enabled model like the one we've created in Neo4j, we now have a much richer description of my data. I don't have this party role, blah, blah, blah. I have terms that I understand and that my business users will understand, and that I can pass in the way that we've seen as context to the LLM to generate queries. And that's entirely dynamic. I don't need to maintain that because the moment I add a new type of entity, these queries that extract the shape of the graph will be naturally returning the additional results, which makes the architecture substantially simpler. So that's the idea. Good, I'll take a pause here, sip of coffee and and that's that's pretty much it so that that's again we could take this gives an idea of how we could maybe take the whole workbench or benchmark and and and maybe replicate it and see the see the results that we get with Neo4j and the complexity of the architecture that we would need to build to do that. But I didn't, of course, in this one-hour session, want to run the complete benchmark, but just a simple example of how to go about doing it. Yeah, I think that was great. Great unpacking, as Tony says in chat. So I think that really helps. And I guess, yeah, like you said, we need to, in this format, we cannot really do the whole thing. But I think what you showed today is a good example of how it's done with the knowledge graph, with graph model um working uh alongside this um this data set so gives you it should give you a good hopefully it should give you a good insight of of what you can do here what what the benefits are and what you you know what what you get in terms of results uh from from this edition and i think we think we showed in the previous episodes why a knowledge graph together with an LLM makes a lot of sense and why this is really an improvement. This is just another way of, you know, agreeing, I would say, with what we found already in the past and just, you know, using a slightly different stack with the property graph model. And again, we took the same settings and we use exactly the assets that they published and in the context where data lives in a relational database. But any Neo4j user should know that the data is GenAI ready in a way. Because the last step the the code the notebook that I published there would work on any model I mean if you think of it let me let me bring it bring it back again and share my entire screen if I share this let me bring back the notebook I mean mean, all this notebook does is, after the installation, it connects to your database, and it will go and retrieve a description. I've let this because initially I was hard coding it, but I completed it in the end. All it's doing is dynamically retrieving from the graph a description of its schema. What's in that graph? What type of entities? What's the relationship? What's the ontology of your graph? You retrieve it dynamically, I don't need to do anything. You run this query and it will work on any graph. You have the movie database, we'll run it and we'll get the entities. Whatever you have, these will retrieve that schema, and you can pass that. This code will work. All you need to do is change this to be a business relevant question in your domain, in your model, and these will just work and will generate a high quality cipher for you without having to maintain all these elements. So that's one of the added values. I mean, the outcomes, I would say, the takeaways from this session, that a graph is kind of a self-describing data set. It doesn't only contain the data, but it can tell you what it's like, what's its shape, what's its structure. Yeah, perfect. That was good. There was one question from Fabrice on chat. As you pay for tokens, it could be expensive with a big ontology schema, like a big prompt. I mean, yes, it could be. But if we think of it, I mean, the answer is it can grow. But the ontology will never be data size, instance data size. So you can use portions of the ontology if that becomes an issue, right? Here, we're working in the context of a single data set. And ontology is never going to contain more than what? 100 hundred entities, you know Few hundred if you're looking at an enterprise-wide one I mean an ontology can I think they can become big when they're combined with with additional additional elements But it's I mean if it becomes a concern, I would say maybe pick a portion of it, right? No, maybe you don't need to pass the... Now, the question of how to filter, how to select might be a tricky one. I mean, ontologies are sometimes broken down and divided in domains or in subontologies. So that might be another way of doing it. But yeah, for a point, it would be an interesting exercise to to understand i mean i haven't looked at them at the metrics but but measure a little bit of how many tokens are being exchanged in this case which is a small one maybe in a larger one and and see whether that can become a an issue no no i mean that always needs some fine tuning but i think that's generally yeah you know um i mean the question really is here do you want to get the right answer or is is the answer not relevant and if if if that is not a business problem then i guess it doesn't matter but if it is a business issue and can create problems then I think the cost for the tokens is not going to be the high priority. Let's take one more question before we wrap up today from Arunan. I worry that this may produce unpredictable results in production. More of a comment about LLM output rather than the structured nature of the ontology. it's true i mean it depends i mean i think in the case i set the temperature to zero to kind of i mean not guarantee but at least have the more predictable query generation but yes i mean any anything where you where you introduce an llm you know by definition is generative and therefore creative and therefore sometimes to some extent unpredictable. But, and again, there's a bit of a question here on where would you want to put a solution like this one? I probably, my personal opinion is that I think this type of helpers when generating queries will be more co-pilot style rather than something that you ask a question in English. I don't know if you will, maybe there are scenarios where you want to give business users or non-technical users access to a dataset. I don't know if it will be kind of entirely open-ended and just let them talk and hope that the LLM produces something meaningful, but yes, that's inevitable. And you will have to introduce guards and controls. And that's the nature of LLMs, right? So you can introduce some guards from the Knowledge Graph side. But that's going to be a factor, of course. MARTIN SPLITTINGER. Yeah, but I think the more you add there, the more more guards the more structure the more context the more the more that a knowledge graph can help avoid that and make sure that you don't get um you know it results you don't want to want to see so but yeah it needs it needs fine tuning it needs it needs uh it's not something you can just copy and paste and run with it. It needs, it needs a little bit of work. But once I think as this, this whole exercise today showed, I think it is, it is worth it in the end because the output is just so much better already with this showcase. And I think it'll just improve over, over time. Cool. Before we wrap up i i as i promised initially i just wanted to give you a quick um little um demo or rather promo of what's what's gonna happen in the future so again the um So again, the workshops that will be running in the next few weeks is available for registration. So if you want to go there, check it out. Also, O'Reilly is running a knowledge graphs and large language models boot camp. So perfect fit, I think. If that is something you are interested in, they offer also a 30-day trial for free. If you want to join these, they are happening actually next week and the week after. So check it out for that and you can subscribe for 30 days for free. And then next week, next stream will be together with Johannes where we'll talk about entity resolution and deduplication with Neo4j and GenAI. So you see obviously the topic GenAI as a whole is sticking around for a while. So we offer a bit more on that front. But that should also be an interesting session. Also, again, touching obviously knowledge graphs and how that all fits together with this entity resolution and entity duplication in the next episode. Yeah, and I think with that, we are at the end of today's episode. Again, thank you very much for your comments and your questions. Thank you for watching. Thank you very much for your comments and your questions. Thank you for watching. If there's anything else that comes up after this episode is now over, please use the YouTube comments, give it a like, and follow our channel to get notified for the next ones. And, yeah, thank you very much for presenting and for preparing and for sifting through that benchmark. I think it was not really as mentioned, but I think very much appreciated that you looked at all this data, lots of CSV files, lots of things. And just to point out that this takes a lot of work. And thank you very much for doing that, for making that presentable in a very concise one hour session today so thank you perfect you know and as always all those materials will be soon on the github repo so we'll we'll make sure that happens right yeah cool thank you very much uh see you around and uh yeah until next time uh bye bye everybody see you all bye
Going Meta - Ep 26: Unpicking the data.world Benchmark on the Role of KGs in LLM QA
Episode 26 of Going Meta – a series on graphs, semantics and knowledge

Jesús Barrasa: https://twitter.com/BarrasaDV
Repository: https://github.com/jbarrasa/goingmeta
Knowledge Graph Book:  https://bit.ly/3LaqE6b
Previous Episodes: https://neo4j.com/video/going-meta-a-series-on-graphs-semantics-and-knowledge/

A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databases: https://arxiv.org/abs/2311.07509
Benchmark Data GitHub: https://github.com/datadotworld/cwd-benchmark-data
r2rml: https://www.w3.org/TR/r2rml/
r2rml github: https://github.com/chrdebru/r2rml
dbt semantic layer: https://github.com/dbt-labs/semantic-layer-llm-benchmarking
Neo4j Sandbox: https://dev.neo4j.com/try
Going meta - Ep 5: Ontology-driven Knowledge Graph construction: https://youtube.com/live/05Wkg1p34ek
Data Importer: https://data-importer.neo4j.io

#neo4j #graphdatabase #semantic #knowledgegraph #benchmark #llm