 all right good morning good evening good afternoon everybody and yeah welcome back to going meta season two episode one it's been a while we took a few months break over summer and then um a little bit before that to to refresh to to recalibrate and now we're back back with going meta hi jesus welcome as well thank you you. It's so good to be back. It's been a while, but it's great that we're here. Yes, absolutely. No, it's been a constant thing in the back of our minds, I think. Constant. Thank you all for you all for chatting to us regularly, saying when you're coming back. A couple of messages on YouTube and on other mediums so it was great to see that um that there is um you know there was demand and there was a felt feel like people missed us so that's good we've missed it a bit haven't we i mean it's true that it's been summer but it's a it feels you know like there was something missing in our in our beginning of the month exactly also also for for us too right so we had uh we had the the conversation uh last week um we said yeah i mean this was uh like a very constant thing uh also for us to check in and to talk about things and uh to share what we think and what we saw but what recently happened uh with you all so um it was uh it was a constant that uh that we were you know i think we're happy to to to restart now um at the end of uh of season one we we kind of wrapped up a little bit we summarized a couple of things so we we had uh one bigger summarization episode i I would say, which we did a recap episode. I think that was episode number 20. And then in the last episode, 27, we kind of like paved in the way and said, okay, we need to look into more outside of knowledge graphs, outside of just ontologies. AI became the topic, I would say, for the last six, seven episodes that we had in season one. And we kind of like couldn't really move away from the topic. It was just clear that we had to have that. We had to talk about it. Everybody was talking about it. And it just made so much sense. And that's a great segue for us to talk about these developments and these interesting things that's happening around in the space and technology around AI. And we had the perfect, I mean, we had a show, we called it Going Meta, and we had the perfect vehicle, I would say, to talk about these things and to combine things and to take all of these different different approaches and to mix it together so it was great that it worked kind of like perfectly even when we when we when i think when we started with the whole going meta series that we didn't think about that it would go this way but it kind of like became a more or less natural path that led into it yeah yeah yeah and i think with kind of a, how would I say, I mean, build some of the, I mean, we've talked from the very beginning about knowledge graphs, about ontologies, about semantics, and these topics have somehow been, you know, like many others, been absorbed into the whole frenzy about Gen.ai and brought up very, very, very interesting experiments and interesting analysis that we've done. And we're going to pick it up exactly what we left it. And also listening to what we get on the feedback. So this is the topic that we're going to be talking about today, something that people have asked about. We've touched on the knowledge graph generation, but mostly on structured data. And using unstructured data is something that kept coming up in the feedback. So I'm really happy that we can address it. And also that there's this great project that we're going to be spending time on, which is the LLM Knowledge Graph Builder, which is going to be the center of today's session. So yeah, really timely. And I think it's a hot topic and I hope people are going to find it interesting because we're throwing these additional element of ontologies into it. Yeah, absolutely. it this additional element of ontologies into it yeah absolutely yeah and that's that's that's great how we we still keep combining the the the spirit i would say because some people were worried and said oh not yet another ai show no it's not going to be yet another ai show we we obviously we include ai to the things we are you know caring about most and knowledge graphs and anthologies that's taxonomies all of these these topics we talked about uh in the in the previous season and we will continue looking but we we add to it uh a uh an ai part i would say absolutely and and and and those of you who pay attention and who have followed us, you will find some elements that we have reused. And that's something that I love doing, we love doing, and we've done it in the past, like taking a topic and rethinking about in the context of Gen.AI, LLMs, and combining things. Because of the of the topics that we covered are sort of building blocks that when combined deliver more interesting results. And that's also a bit of what we're going to be doing today. So really looking forward to the session. Yeah, me too. Before we dive in, I just wanted to for anybody that hasn't watched a previous episode so don't worry if you are new to the to the show welcome even it's it's it's not a prerequisite kind of like to have watched the previous episodes definitely not you can always watch them they're all available in in a in a repository and on our um on our youtube playlist so you can you can watch them there so but don't worry about it. You don't have to have watched the previous episodes. Again, we are conversational, but we also have a demo part. So Jesus has prepared a little bit of a demo as usual. So it will be more of a hands-on thing, a little bit of a show and tell, I would say. And we go into some hands-on part. Everything will be shared in a GitHub repository afterwards. So you can always check that out afterwards if it's a new code we developed and if it's something new for you to try out. And obviously, because this is live and we have the possibility to interact with you who are watching live right now. If you have comments, if you have questions, if you have anything with us during the event, then obviously let us know, type it in chat. I'll see what we can find some time to answer these questions. Usually we find some time for it either at the end or in a space where it makes sense to break for a bit of a Q&A. And if you have questions afterwards, the best place to go to is probably the YouTube video link and just use the comment function there and add some comments there. And obviously, I look for your questions, your comments there, and see if we can answer them either in a future episode. That's always good, like Jesus said, that we can take your input, take your request, take your something you have come across and you think, hey, this would be a good topic for the going meta episodes in the future. That's always great to have that. Or if you didn't understand something quite or something didn't quite make sense for you, then let us know. Cool. Excellent. Yeah. cool excellent yeah then um i think we we can uh we can dive in uh in today's episode fantastic thank you very much so i assume people can see my screen and as we used to do we'll keep doing i'll i'll um open the session with with a few slides just to set the context, to explain a little bit of what we're going to do. And then we'll jump into the hands-on part where we see things in action. And today, what we want to do is create a knowledge graph. And we want to create it out of unstructured data. That's something that many people have tried before. And many people have suggested that think that ontologies are a prerequisite, an important component in driving or kind of guiding what this knowledge graph is supposed to look like and we're gonna see that that's true and that we have tools to make it really... I would say you know and this is a bit of an announcement also for you Alex that's uh that's true and that we have uh tools to make it really i would say you know and this is a bit of an announcement also for you alex that i thought that we're going to split these into in two sessions probably these in the next one because it's such a rich topic and and i wanted intentionally to spend you know today's session in like a like i said before in the llm uh uh knowledge graph builder because you know let me move to the next slide you know the idea of of today is is pretty simple so we have data out there that's unstructured and by unstructured we mean you know text data video data audio data that uh that's not in the form of you know typically tables csvs json something that we can easily parse and write scripts to ingest into a knowledge graph, but rather something that requires some sort of heuristic or some sort of intelligent approach like what LLMs are doing these days. And this, you probably will identify some of these drawings here as part of the ones that we typically use on how we consume these unstructured data. We apply some LLM processing, whether that's through some API or through some application, and we use the output in a structured form to populate the knowledge graph. What I've done is I've injected there this element, which is the ontology. That's an optional element, but important one, as we're going to see today. And that's the experiment that we're going to run. The idea, obviously, is that the ontology, we've talked about this many times, but an ontology is kind of an agreed sort of schema, vocabulary, domain definition of a particular area of interest. So if we, let's say, we're talking about news, we might have people, location, events, and a collection of categories that we describe in an ontology. We describe also how they relate to each other, and we use specific terms to refer to them. And if we make that public in a standard format, like we've seen in many of our previous episodes, what we can do is we programmatically can use that to drive the construction of a knowledge graph, and we can use that to drive the construction of the knowledge graph, and we can use that to expose our knowledge graph to consumers, and that enables interoperability among other things. And we know that ontologies can take us further. But in this case, it's kind of a consensuated, agreed vocabulary, agreed set of categories that kind of describe what our knowledge graph is going to look like. So that's exactly what we're going to do and uh very timely uh we we have this great project probably the hottest project this day in in neo4j labs which is the uh neo4j llm um uh knowledge graph builder right so this is the the the labs page and uh and linked to it it's uh like all the labs page and linked to it it's like all the labs projects is an open sourced one so it's linked to the to the github repository so you're more than welcome to to go check it out contribute explore look at it at the internals but most importantly because you know the idea is that this offers a code free tool to do the whole end-to-end process. Let's take unstructured data in a number of different forms, and it will take care of it and build the knowledge graph for us. So you have the links there. It's very well documented, so I totally recommend you go have a look at that. But I've also done my research, and I know that you, Alex, have had a session a a few months ago i think i think was just before the summer in may or june with the niels and and morgan where where they took you through i mean they both were heavily involved in the in the creation of this of this project and and it's a very interesting session where they take you through you know what it is how it's built and how to use it, I totally recommend, and you have the link there to the recording of that session and the post that Morgan wrote on it. I don't know if you want to add anything on that, Alex, but I think that's definitely something that people can go and check out. Yeah, absolutely. So it was a great, great, great episode. So it's a great tool in all in all. So it really is, like you said, a no code, almost no code tool that helps you get any kind of data, unstructured data into a knowledge graph. And it's working super easy and super quickly. And you can use the predefined app. So you don't have to set up anything, just an Aura instance, and you can use an Aura-free instance for that, and you get going, or you can go further and you can, as the code is completely available, you can deploy it yourself and decide you wanna tweak it some more, but out of the box, it works really well with free tools, and you can just put in anything you think of and see where it takes you. And it is a really nice experiment to try to see what difference does it make to add this knowledge graph and to add this feature and then have a chatbot-like interactivity with your data within a day, I would say. Yeah, absolutely. And I think it was so convenient for today's session because I want to focus, obviously, that's the subject of today's session, on how we can inject this kind of ontology as part of the process. Because we're going to see that by default, you can do it without, and we'll see the consequences of that and the quality of it and what happens when we start. I mean, I guess we're going to see and I don't want to anticipate the outcome and you will come to your own conclusions, but we'll see that the whole process of extracting unstructured data, I mean, something structured like a knowledge graph out of unstructured data, it's not an easy process. It's simplified significantly with these sort of tools, but there's a lot of guardrails that we're going to have to put in the way in order to produce something that we can use. But like I said, let's not anticipate, let's show it rather than tell it. So yeah, check this content out because I think it's really good quality and really useful. So today we're going to be working with data solos that I've been meaning to use for a while. You see the URL down there. But if you've been to London, you might have seen that all over the city, there are these blue plaques, these kind of buildings that indicate that some significant person has lived there, has a connection to that building. And there's a database online that you can explore, that's the URL. And I think it's quite interesting. It's, if you do a search, you can filter according to different criteria, you will find them where the location is, and you will find a detailed explanation on when was it created, who is it about. And for example, you're seeing there the place where Jimi Hendrix used to live in London. So it's a pretty cool data set. But I thought it's one of the many unstructured data sets that you find out there. And because that web content is something that the LLM Knowledge Graph Builder can deal with, I thought that's a good one to use. So that's what I'm going to show today. That's the dates that I'm going to try to build a knowledge graph out of. And I thought I'd mention that there's an interesting aspect to this tool, is that it produces a quite particular and rich knowledge graph, a graph that's formed of two parts. And again, that's a reference to some things we've done in previous sessions. But the data, unstructured data, typically will live in documents. And these documents might have internal structure. If they do, that's great. And you probably remember this session where we looked at some legislation, a document that has parts and sections and definitions. So we could kind of map in this document structure graph the structure of the document and then deal with each of the chunks and get insights from it. In the worst case, if it's just plain text, we might just chunk it. I guess you're familiar with this idea of chunking. It's a way of producing fragments of a document in order to be able to then embed and enable vector-based search. That's something that we've been doing again and again. But this is something that's captured, and I would say it's kind of the lower level graph. It's a graph of the structure of the document. In worst case, like I say, it's a sequence of chunks associated with a representation of a document, which in our case will be the web page. But on top of it, and more interesting, is the domain graph, where we extract the entities that are mentioned in these documents. So for example, in that page that I showed a minute ago, we will have a note representing the web page about the plaque of the building where Jimi Hendrix used to live and the different chunks that the Knowledge Graph Builder creates out of the page. But then on top of it, and these on top of is more kind of my way of thinking, because in the end they're just nodes connected to all the nodes, we will see entities like, hey, there's a person, there's a node representing a person, that's Jimi Hendrix, there's a node representing an address, it used to live there, there might be other things that are mentioned there. And these are entities that are extracted from this content, and that's, I would say, the more interesting part. And the two graphs are linked together. And like I say there, they will not be exhaustive, and it will depend on how you approach the process. But more importantly, they're complementary. And they're complementary for a number of reasons. But you mentioned before, Alex, that one of the things that we'll typically do to test our knowledge graph is to run some kind of chatbot conversation with it and that's built into the knowledge graph builder so one way of approaching this interaction is by doing vector search on the on the kind of lower level graph the document structure graph finding matching elements and then navigating up to the domain graph and finding related entities so this is the idea of graph rag. We've talked about it several times and we've shown how it improves and augments plain rag that would be just based on the vector search. So that's the kind of end product that you will see. And I thought I'd mention it because it's a very interesting aspect of what the Knowledge Graph Builder does. So that's it as an introduction. So time to try it out. So any questions on that? I'm sure you're familiar with the subject, Alex. You've talked about it in other episodes. So while I move to the Builder, so that's what it looks like. So from the page that I referenced before, you have a link to the... You mentioned it before, we could download it. You could run it locally with the Docker Compose. It's very easy and straightforward. But in my case, I'm going to run on the hosted one. So I took the easy the easy route so that's that's what you see if you go to that url and all you have to do which i did up front is connect it to my uh to my uh instance right so that would be your aura instance in my case i'm using a sandbox but you have to uh to have a neo4j instance that's reachable from from these these application and that's really everything you need to do in case you want to just try it out without any previous work. You just create a Neo4j instance, or you have some instance that's available online, reachable online, and it can be a sandbox. If you have an Aura-free instance you are using for something else, you can spin up a quick free sandbox if you have no Aura-free instance you take the Aura-free instance if you have something else available as a play playground you can you could just connect it and and go go from there exactly so this is yeah this is like an empty blank sandbox that I just created. So this is the browser interface. So you see that it's an empty one. And when you create it, whether that's in the sandbox or in Aura, you will get the credentials in the form of user password. You can even download the credentials in the file. The builder is really well built. And now I'm connected. So if I click here, I would disconnect. But in general, you type user and password, or you just drop the credentials file, and it will connect you to your instance. And then it's really, really seamless and straightforward. So I'm connected. So my NodeGraph builder is connected to my instance. I see I have the chatbot here ready. Let's leave that on the side for now. And I'm going to go to our data sources. So this is the page that I was talking about before. I mean, this is the page about Jimi Hendrix, but this is the blue plaques, plaques, sorry, page. And what I've done, sorry, I think I went with music and city of Westminster. So if you do a search, you find... I mean, these have done it manually. Of course, you can script that and be more exhaustive. But for our experiment, I thought I'll just explore a bunch of those. And I picked some famous composers that happened to be connected to the city of London. Chilpan, Berlioz, a bunch of those. So I have the URLs of the pages, because that's all you need, right? You go to the builder, and there's actually support in the, not in the hosted one, but in the one that you run locally. You can also use YouTube videos, but I think we've disabled them on the cloud one. But what we're going to use here today is web sources. So all I have to do is paste the URL. So in this case, it's the blue flag page for Handel, who used to also live in London. So I submit that. It will scan the page, and it will show us ready to import. This is a bit of a manual process. I hope that it doesn't take too long but I'm going to... my plan is to paste three or four pages there. Of course there's the demo effect that might kick in but hopefully there you go so we have one document i'm going to add another one for chopin who um who else we have uh mozart who was also i've saved the the urls up front but yeah essentially what i'm doing is adding them here and all the one and just to i didn't click submit there you go to add a bit of uh more modern day stuff i thought i put also jimi hendrix and john lennon who happen to be to have their own plaques in London. Anyway, so this is the manual part of it. So you have to one by one submit the URLs. But once you're done, it should be all be very straightforward. So one more. Jimi Hendrix. And... for Jimi Hendrix and um there we go that's it so for five of them should be sufficient but you see how and and by the way this is uh um you can use it through the app like we're doing here in an entirely code code free but since it's open source, you can programmatically do exactly the same and get in a sort of a headless approach the same result, which is more aligned with what we've seen in the past. Good, so we have the list of resources here, and let's do one thing. There's a button here, a bit hidden, which is Graph Enhancement, where you can do something quite interesting, which is central to our topic today, which is define the schema. So by default, you will find nothing. So it's an empty schema, which means, you know, go ahead, LLM, and extract whatever entities you find. be uh uh but no guardrails or no any any further input of what to expect uh what would make sense from a you know human perspective correct so what happens if we if we do that let's let's take jimi hendrix the page about him of course we have to select the LLM. I'll go with GPT-4.0, which is the default selection as well. But if I pick one, what will happen? And I say generate graph. What we're going to see is that there's some processing going on, and the result will be populated on my Neo4j instance in the cloud. And we're going to see that the result, you know, quite expectedly, it's a bit unwieldy because there's a lot of information in that page. If I don't put any gut rails, you will identify whatever is in that page and try to make it explicit in the form of a knowledge graph. That could be all right for an initial exploration, but it's certainly probably not what you want to do if you want to create a graph that's manageable. I guess that's what you can expect. As you mentioned earlier, the text will be chunked. Individual text lines or paragraphs will be isolated and linked together. And then each of those gets almost like a keyword or something or highlight okay and then you identify the things and that's what makes the graph it becomes yeah yeah exactly not very clearly defined yeah the problem is that it will you know it will be we should start seeing results here. So yes, you see the notion of a document with the chunks. The chunks are connected with the sequential relationship, like next chunk. And this is what defines what we said, the structure of the document, which I think it's probably a bit less interesting here. But we see that there's a number of entities that have been identified. So we've events we've found persons we've found instruments places songs that all these make sense the thing is because we didn't put any any schema definition any sort of shared vocabulary it will come up with the names that things are are are right and now the problem is you know when we reload the next document, that might grow without control. And that will define a graph that's hard to use. But not only that, this is, I think, something that I wanted to share after experimenting a little bit. And it's the fact that web pages these days, we should see here that the Graph Builder is done. So that's completed. We see here that it's done. The web pages contain a lot of noise in a way. So remember that we were, I think we've loaded the Jimi Hendrix one. And what I'm interested in is the key elements here, the text part. But there's lots of information in it. And that puzzled me when I was looking at it. For example, I said, well, there's a number of places, let's look at that. And we find that there is a bit of a number of unexpected things. For example, we have a mention of Stonehenge here. Wow, that's interesting. I mean, I know he played at the Isle of Wight festival, but was not aware of Stonehenge, right? And there's Dover Castle. And I said, what is that? And the thing is, you can expand it and see what it is related to. And in this case, it's kind of an orphan node. So it's an entity that has been extracted and you can look at the these are the chunks. So you can look at the these are the chunks so you can look at that's very very interesting because that gives you additional information on where in the page that comes from and you immediately see that hey that's not the content that i was after that looks like the header is like looks like the footers and all these things and if you look at the page guess what it is so if you click on the visit oops why is going on why it doesn't show it's interesting there you go so there you go stonehenge dover castle that's not i mean that that's part of the menu and you know fair enough that that's so all this to say that you know be careful when you load web content directly from the pages because you know these days and this is not the worst i mean i've tried i've experimented with the catalogs of the Tate Modern or the National Gallery and the way they're built, there's so much additional information that you will be surprised to find things that you were not expecting. So probably some pre-processing, maybe an idea would be to extract the relevant context, the content, the text, and put it in the form maybe of files, right, in an S3 bucket or something like that and consume that text rather than directly the web page. Do some noise before. Exactly. So this is just a kind of an observation and I suppose that's the way, if I remember correctly from the session that you run with Niels and Morgan, I think they use the Langchain loaders. I suppose these are standard loaders that would go in and suck all the content from the web page, all the HTML you don't know upfront. That's part of what we're going to be doing today. We're going to put some guardrails in there. We're going to say, when you import information, there might be places, there might be other things, but I want to take control of the kind of things that you import, because otherwise, it will be hard to manage. And we see, for example, there's a band member relationship. So we have a Gatorade that's a band member relationship so we have a a gatorist that's a band member of so you know uh kind of questionable in a way so it highlights the complexity of extracting something uh like a knowledge graph out of unstructured data so it takes this normal approach which is you didn't give me any any hints on what we're after so i will take you know every subject every predicate every object and i will build the graph out of it. So somehow that's the approach that it takes. And it's not wrong, but I want to kind of guide that a little bit with my ontology. Does that make sense? That makes total sense. Yeah. And I think if you have in a world where you only have one document, maybe this approach actually works as is. But like you said, if you add two, three, four documents, it'll become quickly a little bit messy. And already now we just loaded one document that's relatively, I would say, not very complex in essence, but already you have, and now the relationship type spent away, but we had, or away but uh we had all the labels we had like 20 i think different different relationships already um created and and node labels uh we have here what do you have 10 or so maybe um so it imagine if you upload 10 or so files and this grows almost indefinitely then you have quickly you have a huge mess and you don't really can follow anything because it just becomes very very unstructured that's right that's right no and that that's exactly it so so one one thing that i did is okay so i'm gonna go to this graph enhancement and I'm going to try different things because, you know, it's interesting that they offer already a number of interesting options. So I can say, okay, I want to get the schema for my text fragment and I can take, I copied the text of the description about, I think, Handel. So I copy the text there, right? So I can ask the builder to analyze that and extract some potentially relevant entities and relationships. So it's kind of learning in a way an ontology. I mean, it talks about a person, a location, musical works, events, and there's relationships. And I can say, I'm interested in these, in those. I don't want these. So I can introduce a level of control. Now, I can even enter them myself, or I can go and take a standard vocabulary. And one thing that I thought would be an interesting one to do is go to a very popular one. I think it's one of the most... I don't know if people would call it an ontology, but it's a controlled vocabulary that's shared in the form of an ontology with a collection of classes, relationships, properties, etc. And it defines a number of schemas, entities like events, persons. So it's kind of... I thought it would be a good idea. And we've used it in the past, Alex. You will probably remember in previous episodes. So it defines entities and terms for some of the things that I want to extract from the pages that I'm exploring. For example, I know we're using... We have addresses, right? Where's the plaque in London? So there's the notion of an address. If I do a search here, there's a postal address, which I might want to use. So I might want to, instead of randomly giving it a name, I'm going to use postal address according to the schema.org vocabulary. So I can come and say, okay, I want to use this term and I'm going to pass it to the to the schema.org vocabulary. So I can come and say, okay, I want to use this term and I'm going to pass it to the builder. So I want it to extract postal addresses, right? That would be one category. Another one is we've talked about these are musicians and there will be probably compositions, right? Creations, right? Creation, yeah. Music composition. There's the notion of music composition. So, I mean, I could go on like that, but there's a bunch of these already in schema.org, which I'm keen to reuse. For example, there will be music compositions, and I realize I'm not passing it as a formal ontology. And we'll see the problem with that, because at the moment in its current version, it supports the definition of labels or categories, as we call it. Labels is the way we define categories in Neo4j and relationships. But for example, you cannot specify the properties inside these nodes or inside these labels, or you cannot say how these relationships connect things. For example, in this case, I would expect people to appear. So there will be persons, and I suppose there will be the notion of person. It's schema.org person. So I can copy that too and I will add it to the to the builder so I know that a person is connected to a music composition through the relationship author I believe it is so we can we have music composition or we can have even the composer here that's uh the person organization who wrote uh composition maybe composer is a subcategory of of authors so that's that's a person connected to a music composition perfect so we can we can use that so we can say composer is the relationship that i want to to extract as well so I'll put it on the relationship but I'm not being as you know you can see that I'm not being explicit about composer it's meant to be between person and music composition so it's a relationship that will be there and well I could go on right but I could create my own and I will do. For example, if you notice, an interesting thing is that at the top of the page, there's a description of which government entity. In this case, it's English heritage, but might be London or whatever. So there's an organization that erects the plaque and I was interested in that so I wanted to maybe capture maybe the government organization or some form of organization I think there was an organization I believe there's even a government organization somewhere so there you go. So there's a government organization. Well, anyway, like I say, if there's an existing one in schema.org, I'll pick it up and pass it to the Knowledge Graph Builder. And the connection, there's obviously not a relationship for this government organization erects that plague so I'm going to create my own so it doesn't need to be all coming from I'll take care of that later on specifying that this is not coming from schema.org but created by me what else so we have music composition we have persons and um uh well there you go that that's that's just an example but i'm i'm feeding the the somehow a part of the ontology to my uh to my knowledge graph builder and we're going to see what the effect of that is going to be so let's make sure that i I'm running the same as I'm running my test. So we have a posted address, we have a music composition, a person and a government organization, and then we have the composer. I think I used author in my test. Let's see how this goes with composer. And erected plague at, which is just to capture that line so okay so let's use that so i just need to close and i'm going to run uh the generate graph for the remaining uh pages right so if i run that that will um looks like it's running two in parallel so we should be seeing uh the the results popping up in the graph but this time it should be a bit more controlled. It should be a bit more less, I mean, I wouldn't call it random, but a bit more using the specified relationships and categories that I've defined. So I'm going to close some of these because I have way too many. And let's give it a bit of time so it's chunking. So now what we're going to see is that yes, it's going to use the terms that I've passed, but then I have to double check that these terms are being used in the intended way. So for example, like I mentioned before, that a composer or author connects a person to the music composition and not the other way around or not in any other way. That's not what I'm expecting. Let's see. It's taking a bit of time. Let's give it time to process. Do we have any questions so far, Alex? Just to give it. Yeah, we can take the time and answer some questions. So first, a couple of comments from YouTube. First, last, they say awesome, that works. And they built something, similarly, they tried to build something where they feed unstructured data into a graph, but they weren't aware of this tool. So maybe that helps you define your work a little bit closer and helps you. You don't have to hack it together. So that's great. Hopefully it helps you on that journey and maybe you together um so that's great hope hopefully it helps helps you on that journey and maybe you can let us know what you uh what you do with it and how how you proceed and there's a question from cameron does it make sense to introduce an ontology of a web page as organ is organizing principle and then derive further patterns, ontologies from the unstructured data itself? Sure. I mean, I think we've kind of, you know, sort of didn't anticipate that question, but it's a little bit what we did, maybe with one single, it was a one shot exercise. So we took just the content for, I think it was for Handel and you put it there and it will analyze and try to give you the kind of entities so you kind of learn it from an example and then you can decide which of the categories and the relationships you want to keep and the ones that you want to do. So I would say you can learn it from your data, yes, or you can either because of course we have to think how people would use it in the real world, right? So let's think of an organization, right? So you have an enterprise ontology, you have an agreed vocabulary, you have a catalog where you have standard canonical terms for certain things, so you want to use those terms in your knowledge graph. So it's very common the case where that exists and you might need to extend it for certain elements, but in some cases of of course, you will just literally learn it from your data. And what I wanted to convey here in today's session is that it will be an iterative exercise. And I'm a bit surprised that this is taking that long. I mean, this is the danger of doing it and run the other two. Should we? Yeah. I don't know if we can. But sometimes, yeah, I mean, it's. Cancel these and maybe process the other two. So we have a document processing. Yeah, I should have probably gone with the local install. It will be a safer bet for for a demo but it's um but we'll see that's um you know it's worked of course during the test as it always does but yeah we have some more questions so um so maybe we can we can see hopefully this this uh this goes otherwise we'll uh we'll see what we can do but yeah sometimes this is a bit um yeah unreliable uh with a demo unfortunately um the the sorry now i'm dropped my yeah here from himans, they ask, how to validate the relationship generated by the LLM for knowledge graph creation? You are reading my mind. I mean, that's exactly what I'm going to show the moment we get some of these loaded. Let me start again. So if I delete the files, maybe, I don't know if that's going to help, but delete the entities. And let's put it again. Let's take a single one. For example, the first one that I had for handle. Because that's exactly what comes next. Let me paste it here, hit run, and then talk about it. So submit that. Yeah. I don't know if it's a question of limited resources. And again, I'm sure you will have an entirely different experience if you run it locally, and that's probably what I should have tried. But anyways, yeah, sorry. Generate this graph. I hope we've preserved the schema. That's good, yeah. But yes, I mean, that's exactly what comes next. So what I'm going to show is that, as has happened in the test, some of these, I mean, the relationships that I've passed as input are used, but sometimes not in the right way. And when we see cases, oh, that's great. That seems to be progressing a bit better. So we see cases where I don't know where I are. Let's now put maybe another one. I don't want to be too over optimistic, but it looks like we could. So sometimes you will see a, what did we call it, like an author, a person authoring another person and I say, hey, hang on, that's not right. I said that a person would be the author or the composer of a music composition. So these sort of things you will detect, sort of violation of the expected use. There you go. So is it completed? So this is completed. Fantastic. So let's kick off the other one. So I'm going to start this one, but we can start looking at the generate the graph. So let's look at what came out of it. there you go so we have the the relationships that we define so we have the composer the erected plaque at etc etc and we have persons and we have posted addresses music composition and government organizations so all looks you know decent uh so far but let's let's see what what that looks like so for exampleoser and I would expect there to be... Let's change a little bit the presentation so we have Handle there and another node representing... Yeah, let's make that a bit bigger as well. Yeah, maybe you can consume so we have a sort of a duplicate there and we have a number of music compositions that looks pretty good so that's actually much better than i than i thought so let's try to find another one for example what about postal address a composer erected plan at there we go that should be hopefully some some of those might not be correct so there you go so we have a that's that's something uh that i was not expecting so uh i was not expecting mozart to erect a plaque uh and address so that's uh uh and and we find some mistakes of all the nature because what about government organizations? So we have London County Council that might hopefully have erected some plaque. There you go. And these are the chunks, right? I understand that when we explore it here in the browser, these are kind of flattened, but think of these as the chunks that if I expand it, they will normally be connected to the document where they come from. So that would be... This is the next chunk. There you go. It might not be very readable. This is the document and if we expand it, this will probably be connected to... I don't want to expand it because it's going to have an explosion, but these are the kind of people that's connected to the entities that are mentioned in this chunk. And the relationships. I'm surprised it's a lot cleaner than I was hoping and so what I saw basically but you know that's great anyway. So we have a... here we go, that's one example. So there you go. So this is Mozart's first symphony, which is described there, and he's the composer of that, of course. Now, Maria Anna Mozart is definitely not the composer of Wolfram Mozart. So that's one case of... And Leopold probably is his father, so he's probably not the composer of one. So sometimes the relationships are not used in the intended way, so the right relationships are used, but not following the domain and range kind of expected. And we have a tool for that, you might remember, right? So we have a way and there's several. These days there's... I don't know if you've had a live session on that but there's the pydantic framework where you can basically define constraints on what the expectation on the graph is what what should it look like and and we've done it with shackle here so that's what i think we talked about it a little bit in a previous episode but we didn't uh and that was not a specific go-to episode about this yeah exactly. Yeah, exactly. So what I've done is I've created a very, very simple shackle shape. You might remember how we use that. So there's in new semantics that there's a, I'm sorry, it was shackle. We can import a collection of shapes and all you have to do is specify you know the serialization that you're going to use this is turtle and I just need to paste it here if I use the inline and I've created these this small fragment that I'm going to show there's a couple of episodes on these if you want to dive deeper into this but essentially what I'm saying is I'm defining a shape here, very simple one saying there's nodes of type person, and there's an author relationship. These author relationships are supposed to, I mean, we didn't call it author. We call it composer, didn't we? You called it composer, yes. A previous example, let's make sure that we use it right. So here we call it Composer. Right. Let's fix that. So this relationship is called Composer. And this is standard syntax. This is Shackle language, which is based on RDF. We've talked about it before, and it's a way of defining the shape of your graph, defining constraints. So there's a property called Composer that's meant to connect the person to a music composition, right? And this is the type of the target sort of node. I'm going to do the same. So basically, this should identify violations of this type, right? So this is wrong. A person cannot be the composer of another person. And the same with the government organization so there's erected plaque at which is connecting that to a postal address if it's something else that should be wrong so i've just defined two basic constraints so that sort of answers the question that has been raised so how do i control that effectively the llm behind this tool is following my guidance and using the ontology in the right way well there's no way no way because the LLMs to some extent, or at least not in its current version, but what I can do is define constraints on my graph that will detect any case in which that's not used as I was expecting it to. So now remember how I can run the validation just by saying, okay, go ahead and shackle validate, and it will produce a summary of all the cases where this is not respected, right? For example, we see that there is a node 115, which is of type person, where the composer, which is meant to be of type Music Composition, is not. Right? So we can find the details about it. So we can find about that node. So there will be a node that's connected through a, we said, what was it? Composer to something else. And let's return this subgraph. But I want to focus specifically on this one, so where id of n equals is the 1, 1, 5, right? That's one of the problematic ones so we have a case there well effectively that's what we saw before right this is wrong so this is not the composer of these other persons so I can use that and I'm doing it manually here but these of course can be automated so I could say you know whenever you find something like that, you can delete the relationship. And the same with the other rule that we defined. We can uncover these sort of situations and control the graph that we're generating. I hope that answers that question. Now the second part and the one that I want to use to conclude is, hey, we created that graph following an ontology that I passed as input in a kind of a bit flaky way because it's not really the ontology. I passed the labels and the relationships, but these are standard terms that are defined in the schema, the ontology vocabulary, so I should be able to present them and export them afterwards and the answer is of course we can and this is something that we've also done in previous sessions and all I have to do and this is going to be done in one go but I'm sure you will I'll share the code afterwards as always but what I'm what I'm doing here basically is I'm saying, okay, I'm using one namespace, which is schema.org. That's the vocabulary, the ontology that I've used. And you remember that I created some relationships of my own. And I've put that, it doesn't exist, but imagine I've defined that in my own ontology. So I've called that JB Extensions, which is my own schema, my own ontology, because it doesn't need to perfectly align with schema.org i can add my own extensions so that's a placeholder for that yeah but it's very important to to think about it i mean if if you do this in in your uh company or for your project you you might be relying to schema.org or some other ontology for for general purpose terms but you will for sure have your own definition of some topics, some concepts, some things, and it makes sense to add that in addition to make it more relevant to your set of documents, to your data in essence. Absolutely. And all I have to do afterwards is map the internal name, which for some reason in the Graph Builder lower cases the rest of it, but I just follow the standard naming in schema.org. The one change that I have to do is we didn't use author, we used composer, I think. Composer, so that's the term in schema.org. And I define these mappings. And this one, erected-plag at, is my own terminology. So I define that. This is persisted in Neo. And now I can export it. Something it didn't like. Why was that expected? Oh, I didn't put a semicolon. Silly me. There you go. Looks like that was the oh yeah there was oh this double dash that that's not valid a bit silly so I suppose it broke there so let's run this second part perfect yeah I am good So we now have the mappings defined. And what I can do now is, okay, I'm going to take a subgraph, for example, and let's fix this, right? So we're going to, for example, when I find these type of violations uh i can say when um i mean i'm again doing it manually here in general whenever you find uh something here and an object where oh um it's not of type uh what was it supposed to be? It was a music composition. How did we call it? Music composition, yeah. Music composition where actually we're not that. If we find something connected to a node and we knew that that was problematic, what we're going to do is we're going to remove we're going to delete this relationship right so i'm not going to delete the the target because maybe uh leopold mozart is you know it's something that we want to we want to keep uh no actually it's in this direction if we want to be more precise. But this should... What did I write wrong? So I'll double where. So, and again, no return, just delete. Yes. Okay, so it's deleted one relationship. So now it should look better. So if we go back to the subgraph around this node. There it was. Yeah. Why is there no? Because there might have been a, okay. So we fixed one, but not the, so that's, maybe let's pick another one so for example that um well i mean it doesn't really make a massive difference i mean we could take um one one um six one 8, for example, 1, 21. There you go, Queen Charlotte. That's another case where not. But anyway, there's some errors. But anyway, what I wanted to show just to conclude, because I'm conscious of time, is that now we can export that. And since we've defined the mappings, we can say, okay. And remember, there's a built-in http endpoint in the in the browser so all we have to do is rdf um neo4j is the name of the database uh and cypher there's a cypher endpoint what i can write my um uh for example let's see what query uh let me find something interesting here so we have uh this um Mozart who's the composer this is node 111 composer right so we can take one simple query and say I'm going to pass the cipher oops And the query is the one that I was showing before, but for Node, we said 1, 1, 1, right? So let me... 1, 1, 1, yeah. This is the reason, yeah. So it would be something like match a Node N connected through, I hope that that doesn't fail me composer of something where id of n is 111 and we return subgraph as g which we have to name like this. I hope I've made no mistake, but I have, of course, match, subgraph and composer or ID. Let me see if this is a valid query because that's common. Yep. Okay, that's a very interesting error. Oh, of course, it has an embedding. That's not. Oh, that would be the, I mean, it's trying to serialize. I mean, I could drop the embedding, but it's going to take me a bit more time. And I think that's, I mean, we can just for the sake of the example, I mean, it looks like there's an option actually in the importer where we can decide whether we want to, where is it, post-processing jobs, materialize, enable hybrid search. I think there's an option. And it's a shame we don't have Morgan here to tell us. But I think you can enable, calculate embeddings of, or maybe not through the application, but you can optionally define whether you want the embeddings to be calculated for the extracted entities and not only for the chunks. But just to see it working, we could maybe be a bit of a hack, but we could say for this one, for 1, 1, 1, we could delete. I mean, don't do this at home, we could say add delete so it's uh delete or is it remove remove n dot embedding and we don't need to define the subgraph here there you go oh Oh, no changes. Oh, because there's not. Okay, so we have that. What did I do? You just returned. You removed the remove part of it. Hang on. So match n where idN and return. You just returned it. Yeah. Hang on. Let's go back. Remove command.embedding. And maybe I'm probably spending more time that I should on this. But that didn't change anything. You mistyped embedding. Oh, there you go. Thank you, embedding. So that should remove a property. Yeah, and we're now going to do the same with the other one, with the first symphony, which is 114. So we're going to do the same. There you go. And hopefully, if we now try to re-serialize that as we did before, that's going to produce some nice RDF out of it and entirely defined according to the schema of the orgontology. So it's exporting a person connected through another node of type music composition, right? And with a label that's the name and the comment, etc., etc. So all these, you know, I hope it didn't, it was not too confusing with the tiny bits at the end, but what we've done is we kind of somehow injected the ontology as target model in the knowledge graph generation part. And then we verified that the resulting graph was correct using our constraints. And then we showed that we can export it according to schema.org, producing some perfectly valid RDF. So yeah, I think that was the whole. I mean, I hope that the that was the whole i mean i i i i hope that the the the concept was was clear when understood i'll share the the different bits that i've used uh in the session but uh happy to take despite being over time any questions if there are any yeah we can we can look at a couple questions so so to um as as we have time a little bit um one question comes from uh from time trader and i think i don't think we can answer this but i just wanted to read it out is so support for rules domain range data type etc coming in future releases um i don't know if you're aware of of something i i i don't know i mean i i am mean, that's something that I should, and maybe I thought for a minute, but it was too late to bring either Niels or Morgan as guests to the session. We have a follow up episode next month. Maybe we ask if they can join us then. And then we can answer some of the questions. I know they have a pipeline. I know they're working on things, constantly updating stuff. So it might well be that this is on their agenda, on things they want to add. I don't know if and when. And I would expect to be able to do that to some extent in the programmatic approach. I mean, maybe not through the app, because the app is limited in the sense that you can only enter labels and relationship names. But maybe through code, you can introduce additional constraints. But, you know, we'll definitely answer that with Morgan. Yeah. And something from, I don't know, more of a comment than a question here. First, last is the guy I mentioned earlier that they built their own tool that does something similar. For them, validation required basically collecting feedback or reinforcement from users and or manual validation. validation required basically collecting feedback or reinforcement from users and or manual validation so i think that's a that's a big a big topic which we we showed today and we showed in previous episodes i think as well as like you constantly have to validate if if this still makes sense if this is is the right schema if this is something that that needs updating or that something is you cannot just um and I think this was one of the last episodes where we had this model agent and this query agent kind of like playing back and forth together, LLM playing the roles of humans here. But I think you have to have some kind of validation to look into this. Otherwise, that's right. Yeah. And the idea was to show that I think someone commented somewhere that it's no magic. Of course, there's no magic to some extent, because it's amazing that you can pass some text to an LLM and it will be able to detect, hey, this is an entity, this is a connection between two entities. So it will do a good part of the job for you but but in order to get something usable and something that you can really build applications on you're going to have to first iterate you're probably going to have to start small you know with a a small set of categories relationships a small schema small ontology and then iterate over that and and along with that you will have to put guardrails to make sure that whatever comes out of these llm process is what you expect so so yes i mean it's it's uh it's a crafty you know process but it's uh it's definitely magic to some extent i mean we can do things that we could not even imagine you know not not that long ago and think of the flexibility that that that this that this brings right so yeah yeah um and maybe maybe one more uh question here from don uh he asks have there been any previous sessions on automating the mapping of existing ontologies or data wall schemas to create graphs i think we did we did a couple of those i don't't remember exactly which episode numbers. Do you remember anything in particular? 25, 27, I think we did twice. I mean, we looked at structured data, we looked at the schema of some Kaggle datasets and we mapped it. I mean, actually we didn't map it. What we did is we kind of learned of um uh a schema out of it but the mapping would be exactly the same so we took one approach you know one shot llm based and another one a sort of agentic approach where we kind of create a a proposed uh schema then there was another agent being the critic of it and iterating a couple of times yeah that's what that's what i meant meant before yeah exactly so yeah check check a couple of times. Yeah, that's what I meant before. Yeah, exactly. So yeah, check this episode out. I think that's interesting in the previous sessions. And then to complete, maybe do one more question, because I think that's also an interesting one from, and I'm sorry, I probably mispronounced your username here, Pankothandaraman. And they ask, can we use ontologies to store session data of a chatbot text data in a session in Neo4j? I mean, you don't really need, well, you could use an ontology, of course, to define, but, you know, we have examples of that. I mean, persisting as a graph, all the interactions of, you know, conversations with the chatbot and the kind of the chunks, the fragments of the graph that are being used to answer them. So I'm sure you've run sessions with Dan Bukowski and company that have built a lot of examples around that. So yes, we can definitely share. Yep. Check out these episodes. So yeah. Cool. I know we're heavily over time, so just let me browse through these super quickly. But I wanted to mention it before we close. It's the Road to Nodes. Workshops are coming up on the... Oops. This is not what I wanted to do. I lost Jesus this is not what I wanted to do. Now I lost Jesus. This is what I wanted to do. Sorry, Jesus. I wanted to briefly mention the Road to Nodes session. So on the 19th of September, we start with Neo4j Fundamentals. They will be in two time zones. One is EMEA Asia friendly, and the other one is North America friendly or America in general friendly. So check these out. App building on Aura graph visualization with Neo4j Bloom and also mastering GraphRack. Also this one happening at two time zones throughout September and October up to nodes, which brings you to the next point. Obviously it's happening on the 7th of november you can register now um i'm looking forward to uh lots of great sessions we will announce the agenda in a in a few days i think i have seen so many interesting sessions so it's it's really going to be a beautiful event again looking forward to to that in in november and then where do you want to go next i saw a question earlier on how do you get in touch with Neo4j, with the community, check out the community forum, check out the Neo4j Discord server, go to Graph Academy for interesting sessions. There is one on knowledge graphs coming in the future. So stay tuned, but also there is something on how to work with LLM, how to build chatbots is already available and aura free obviously um if you want to play with the llm knowledge graph builder app uh and and use user database use you can use aura free and uh go from there and yeah with that um thank you for jesus again for welcome back to to season two it's been fun it's been great i'm looking forward to next episode we can definitely continue this and hopefully maybe uh niels or morgan have time to join us um in in a month uh that we can uh we can discuss more and uh and see what else we can do enhanced the enhancing knowledge graph builder with ontologies i think that's um that's interesting and yeah thank you for your time thank you everybody for watching and uh
Going Meta - S02 Ep01: Using Ontologies to Guide Knowledge Graph Creation from Unstructured Data
Season 02 Episode 01 of Going Meta – a Series on Semantics, Knowledge Graphs and All Things AI
Topic: Using Ontologies to Guide Knowledge Graph Creation from Unstructured Data

Jesús Barrasa: https://twitter.com/BarrasaDV
Repository: https://github.com/jbarrasa/goingmeta
Knowledge Graph Book: https://neo4j.com/knowledge-graphs-practitioners-guide/
Previous Episodes: https://neo4j.com/video/going-meta-a-series-on-graphs-semantics-and-knowledge/

LLM Knowledge Graph Builder: https://neo4j.com/labs/genai-ecosystem/llm-graph-builder/
Knowledge Graph Builder App Video: https://youtube.com/live/NbyxWAC2TLc
Blog: https://neo4j.com/developer-blog/graphrag-llm-knowledge-graph-builder/

0:00 Introduction and Welcome 
1:33 Recap of Season 1
2:24 Discussion on AI Integration 
4:03 Today's Topic: Knowledge Graph Builder with Ontologies
7:53 Demo Overview and Setup 
18:00 Differences in Graph Structures
25:03 Schema and Ontology in Graph Building 
29:20 Challenges with Unstructured Data 
51:03 Validation and Constraints with Shacl 
1:10:00 Q&A and Closing Remarks 


#neo4j #graphdatabase #ai #knowledgegraph #genai #graphrag #llm #ontology