 Hello, everybody. Good morning. Good evening. Good afternoon. And thank you for joining this Neo4j Live. Today, we talk about a personal knowledge vault or creating of a personal knowledge vault with Neo4j GraphRack and some chat GPT prompting. So I'm looking forward to this episode today. With me today, I have Mike Murley. Hi, Mike. How are you doing today? Greetings. I'm great. How are you? I'm good, I'm good. It's finally summer here in Germany, so I'm looking forward to it. So it's a good day. Cool, yeah. How are things over where you are? Oh great, we finally also got into summer, although I'm still wearing my toque. And yeah, I was just out in the mountains last week. It was a little snowy and cold going in, but coming back out, it was actually, we finally got up into the teens in Celsius. Yeah, yeah. It's up in the mountains, it's still cooler. So that's for sure. But when you do some exercising, I guess you don't want it to be too hot anyway. So it's true. Cool. No. So today we talk about the personal knowledge world before we do that. Maybe Mike, do you want to say a few words about uh who you are uh and and what you do when you don't create knowledge vaults sure um yeah so i i i'm the currently i'm the director of machine learning and artificial intelligence for a company based in calgary canada called r curve and a big part of that role is helping clients kind of go through and understand use cases for advanced technologies and for AI. A lot of the work that we do of course is confidential. So one of the reasons I undertook this project was to create a working example that allows our, myself and our team to explore how we would go about applying some of the latest and very rapidly moving ai technologies including of course knowledge graphs to the domain of helping organizations make better use of their knowledge and their data and in particular we do a lot of very applied work, modeling physical systems, energy, environmental, and mining, to name just a few. Architectural engineering construction is another big one. And those are areas where you have a lot of complexity, a lot of different systems, a lot of contributing systems into making a model of the world that these companies are operating in. And you need to have a lot of flexibility with respect to the data. And one of the challenges that we'll talk about today, of course, is I think there's, it's this impedance mismatch between the lack of computers to understand and be able to model data the way people think of it. So we were forced to kind of put data into a certain form of the machines to be able to model data the way people think of it. So we were forced to kind of put data into a certain form for the machines to be able to work with it. But I think there's a real innovation when you look at graph and when you look at that combined with large language models that unlocks a whole powerful new paradigm for the way we work with data. Yeah, that's a very, very good problem we all have. I think we have faced before and we will face probably for the foreseeable future until it all hopefully can be outsourced to some kind of AI assistance that would just take the documents you provide to them and say, hey, here you go, take this. It's in my format, in my way of organizing things. But you do you and you know what you do with it and then put it in a, in a form that is slightly different every time. Sometimes it's first name last. It just starts with an address, right? I mean, sometimes it's first name, last name, sometimes it's first and last name in one thing, sometimes it's last name and then the first name, you know, it's, it's in, in just very simple examples, it can become complicated. And, you know, God forbid talking about more complicated things like approval In just very simple examples, it can become complicated. And, you know, God forbid talking about more complicated things like approval processes for, you know, you mentioned a couple of things like construction roads or pharma companies process in getting a, you know, Midsom approved, stuff like that. It's a complicated process yeah it was it was interesting i was at a conference recently building sustainability conference in tempe arizona um on an ai panel for that and that was i think one of the biggest things that folks there were talking about was when you're on the construction site things are moving around you know you don't have time necessarily to be putting you know whatever to the things you're trying to observe into these little boxes. Yeah. And I think now to one of the things I'll try and show illustrate today, and the example that we're looking at is this idea of a personal knowledge vault. And again, it's a proxy for other places where we would apply this technology and sort of the business side in that it allows you to much more flexibly collect data. So I'm hoping to illustrate a bit of that. And then, of course, the knowledge graph really has a nice, it's a very human way of modeling the data. It's kind of the way I like to think of it. It provides a lot of flexibility and fluidity. And you can have, I don't have it quite working yet in the knowledge vault, but as the data is coming in or coming in, you can have then downstream agents't have it quite working yet in the knowledge vault, but as the data is coming in or coming in, you can have then downstream agents that are going in and post-processing and extracting entities and doing named entity recognition and whatnot to construct a much richer context around the data. And it's this very fluid schema, schema when you need it, but it doesn't get in the way. That's one of the things I really like about Graph. And it allows you to kind of evolve your understanding of the data as you go. Yeah, yeah. Yeah, that's always a good point to bring up every now and then. Is the schema less possibility with Neo44j you can obviously do it um as as you like it we had a neo4j fundamentals uh course last week with martin and we the question came up as well what what is what do i need to do here and then is is is uh if i have if i don't have a schema is this going to be completely chaotic but yeah it can be, yeah. I mean, if you don't think about it and just dump everything in there, but it gives you also the freedom to do and expand beyond the initial scope of a project and in the initial ideation you had, maybe you realize, oh, this could also be a good addition to this. And then you just add it in and the schema works with you and you can, you can expand it in a way that, uh, that it makes sense. And it, uh, it connects with the rest of your data. That's already there. You don't have to completely start again. Totally. Um, so I've got a couple of slides I could jump into and just to set the stage. If that's. Yeah, that, that sounds good. Um, as as as always i already see a couple questions in uh in in chat so i'd like to encourage that always if you have questions if you have comments then then don't be don't be shy uh let us know and i can read them out um um any um any time so i i see see a couple of questions already. So we do the intro first, and then I'll look up what came in. Just quickly before this scrolls away, but I'll keep my eye open, obviously, and scroll back up. Martin posted on X if this video is going to be available as an archive later. So the answer to that is yes. as an archive later. But so the answer to that is yes. If you check out on YouTube, youtube.com slash at Neo4j, you'll see all the videos that have been made in this series previously and you can watch them at any time. So that's an easy question to answer. One point, Mike, I'd like to ask you to, if you could click the little button that says studio.restream is sharing your screen, if you could click on hide there so it goes away otherwise it's a little bit on the oh this on the middle yeah exactly got it thank you cool and that um floor is yours okay well yeah if somebody has a question you know throw it up there and i can't see the questions at the moment but um okay perfect yeah so as i mentioned this this is kind of a two this this project that i've initiated um is operating at two levels so one of the things with this is um as i mentioned a lot of the work that we do at our curve is confidential we have had a lot of people a lot of executives a lot of companies asking us a lot of questions about large language models, how they could be applied, what the risks are, how they would manage the risks and a myriad of things like that. It is challenging talking about some of these things because you do want to have, it's nice to have an applied example that you can use to show folks because you do wanna have, it's nice to have an applied example that you can use to show folks exactly the impact that this new technology has and how graph kind of factors into that. So I set this project up. This is something I've been interested in for a long time and have been working on for a long time. Prior to R-curve, I was working at an environmental company where we were using, starting to use knowledge graphs to work with complex environmental data and reporting on that. And then that led to my starting a company called Minome Technologies, where we actually developed a system that was using Neo4j to continuously integrate data together for many sources. And then that was acquired by R curve a few years ago. And since then, we've just been expanding the number of organizations that we work with in this domain. Something that came up, so I have kind of on the personal side of this, I have a relative who's experiencing functional memory challenges. And it struck me that when, you know, the large language model thing showed up, if it were possible for me to hook, to create a system, using some of the technology that we developed with Neo4j for kind of constructing the graph side of things, hooked up to an LLM, you could help a person who, you know, they can't use file folders and the complicated ways we use to store data anymore, but they can still write and they can still express and their memories and whatnot. So they're writing their memoirs now. So a couple of weeks ago, I went and set this system up for them to help them start to capture a bunch of the memories. And then as I continue to evolve, the functionality will help them start to generate some of the writing work by allowing them to dynamically retrieve the knowledge without having to know where it's stored. So a bit of context on that piece. Yeah, so back when all this kind of knowledge stuff started the idea was, and if you go back and you read through all this, I won't go through this in detail. I've got a post on this up on Medium that explores this in some detail in terms of how we've ended up where we are with the way knowledge systems work. Prior to the systems that we use now, there was this, these folks, you see on the screen here, imagined the potential of computing, revolutionizing and unleashing, augmenting the human intellect was the term that I think Douglas Engelbart used for it. And the original vision for this was knowledge collaboration without barriers. So being able to move knowledge and knowledge and data and applications, there wasn't a differentiator between those things. So if you look back at the Xerox PARC, the work that was done with Smalltalk and whatnot, it was not really a difference between the applications that you worked with and the data that you had. And you could dynamically share that around across without any boundaries, or at least that was the intent. But what ended up happening was, is over the time, when we started off with this notion, the computing devices and the computing that was available in the 1980s, and you see this machine behind me, that was my first computer, was very, very limited. So while the WYSIWYG paradigm, the desktop metaphor that emerged, you know, that Jobs saw at Xerox and brought across to the Apple and then was eventually migrated over to Windows that we all now work with. So the apps, the way apps work, the way we store files with file folders and whatnot, all that stuff came across in that same era where computing was extremely limited. And what that meant was we were forced to deal with a series of conventions. You had to have an application, the application had to store the data in a certain way, in tabular data forms specifically, or generally speaking, you had to store the data in a certain way in tabular data forms specifically, or generally speaking, you had to store your files in a certain way. And what that has meant, that combined with the aspect of commercialization and the organizations that make these systems want your data to stay with them, is you have this tyranny of apps. And I think there's some stats on this where the average user has somewhere between 80 and 100 apps on their mobile devices, let alone the stuff that goes on the enterprise where there's some stats on this where the average user has somewhere between 80 and 100 apps on their on their mobile devices let alone the stuff that goes on the enterprise where there's hundreds of applications that a person has to interact with on a daily basis it makes it really tough for a normal person to know where did i store this bit of information let alone somebody who may be experiencing functional memory challenges and it's even worse when you talk about this domain of files. There's a whole history behind the file thing in terms of where it came from. And it's actually fascinating to study. And again, I've written about that in some detail. So you can look that up on a Medium post. But very quickly, in order to store a file, you have to know which application you're working with. Is it Excel? Is it Word? Is it the other things that could make files, music programs, composition, whatnot? You have to know which system you're storing the thing in. So if you're in an organization, I work with all of these. And then we have clients who use other systems. And then you've got all the things like Documentum and whatnot. All of them have this file folder structure. There's always some kind of naming scheme that somebody comes up with that's supposed to be the naming scheme to solve all the naming scheme problems, which never works. And then you have to know what name to give the file. So you're putting an awful lot of cognitive load on a user who then has to navigate all this complexity to put something in a little bucket somewhere. And then they have to remember all the things that they did to get that back, which was at Dropbox, was at Google Drive, SharePoint. And I don't know if most folks here probably use the search tools these things all provide. They're all terrible. And you can't search across them. So you can't search across them you know so you can't search across dropbox you can't like there's there's no unification of any of these things it's very very complicated um so the crux of it is working with knowledge on our current computing devices requires users to learn complex metaphors taxonomies and numerous disconnected apps, each that it has its own method for storing and retrieving data. So the idea here is, okay, we've got this new powerful world of LLMs. They do have this little problem though, in that they're not databases. They're a very useful thing in terms of being able to, and they were kind of the missing link, if you will, between the a very useful thing in terms of being able to, and they were kind of the missing link, if you will, between, uh, this is the type of thing that we were trying to accomplish with the Meno paradigm, which was, uh, kind of along the lines that I've been scribing, enabling people just to ask for something and get it back, describe the thing they're looking for and get it back. The problem was, is it's, you're still having to try and translate the person's intent, uh intent into some sort of machine understandable query. That's the piece that LLM start to help out with is that you can have the LLM act as the interface, like a human capable, human level interface, if you will. They can capture the data in a very human way. It can store the data for the person. It can help contextualize the data, but you still need the database and behind the scenes to actually act as the anchor for where these things live. And again, I think it's interesting that Microsoft published that article on GraphRag last week because it's long been my sense that graph is the way to do this sort of thing because it gives you that, again, human-like capability, linked relationships-based method for describing the way you store your data. And then you anchor that context for the LLM into that graph, and it's quite a powerful thing. So the use case I undertook to help, again, this is a multi-tiered project. One is to help provide a method that we can use to talk to clients about this, but it's also to help a person that I know out with this exact challenge of, I can't use file folders anymore, but I can just talk to this thing, and it will capture this information and knowledge for me. And then I can get it back very easily just by asking for it. And the way that I've done this is I decided I wanted to use chat GTP and a GTP as the interaction interpreter. And I'll talk a little bit about why I chose that instead of building a standard user interface. I'm using Langchain and Langraph through a Python fast API to handle the API layer. That's hosted up in DigitalOcean. And of course, I'm using Neo4j as the database, the vector embeddings and all the power that comes along with having a full database as opposed to just a vector database. And there is there is a difference there. And I'm using kind of a simplified form of graph right now. I do hope to get more sophisticated with it as I get into this more. The initial intent with this thing was to really just focus on a few key use cases. So most of those are on the capturing side and then on the basic retrieval side, I haven't done a lot with context yet. That's gonna be the next stage that I go through as I build this thing out more. The idea though is I wanted to be able to capture notes from sources such as books, handwritten notes, or voice transcription. So these are all things that in the GPT-4-0 model, hopefully I'll show you a couple of things. It's pretty amazing. The thing can read my handwriting, and I can't even read my handwriting. It's like, wow, okay. Pretty incredible. and I can't even read my handwriting. It's like, wow, okay. Pretty incredible. So then the other thing I do, I use this thing myself too for all the knowledge and all the stuff that I read. I put everything that I read into it. And somebody I think earlier asked or early on before the session had asked about are you hooking this thing up to things like OneNote? And the answer to that is yes, I will be. I haven't done it yet. I do have it hooked up to SharePoint. So that's actually where my user in question was storing all of the things that they've written over the last 10 or 15 years or so. So I actually do have a thing that goes out and grab the documents from that and process and decompose them into the graph structure, graph document structure. I wanted to also use some of the land graph stuff so i do have a little land graph agent in there where you can direct these agents to go out and research a topic on your behalf and i'll show you a bit of that where the agent can go into the personal your private knowledge or personal knowledge vault pick up stuff from there it also searches out to an external public source. In this case, I've got it looking at Tavoli because that was an easy one to set up. But you can certainly add others to that. And then retrieve word documents and other sorts of things out of the vault. That's enough of that. Let's get some demos going here. Any questions so far? A couple of comments. So you mentioned already the tools like Obsidian or OneNote and stuff like that. So I think they're trying to find a workflow that resonates, trying to find how to connect these things. I guess you can share a little bit what you learned, what your experiences was with connecting different tools. I think you have it a little bit easier if you stay with one ecosystem. I mean, like you mentioned this in the beginning, Microsoft Fabric was mentioned by Creative Sense. It's like makes it, tries, I guess, Microsoft's version of trying to connect at least their initial tools, which are not really connected at all, to make it a little bit easier for users to know, okay, I have something on Teams, I have something on email, on Outlook, I guess, or something on SharePoint, on OneDrive. How do I find all of this in these mixed things? But usually people are not strictly in one bubble, I would say, then they have maybe an iPhone and a different system already. Then they have maybe Slack and work instead of Teams and maybe a Google Mail instead of Microsoft. It's going to be, it makes it a bit more complicated. But I think it's good that the tools are getting there. I think it's great that the vendors themselves realize that this is a problem. I think it's great that the vendors themselves realize that this is a problem. I think it's not going to be solved so very quickly. No, I don't think so. Again, it's this is kind of and again, I get into a larger philosophical discussion about this in one of the medium posts. And it's something I do like talking about. And I actually did. Yeah, I mean, Alan Kay is a really interesting person to look up on that subject. He has a lot of very interesting views on how the commercialization piece of this does start to act as a barrier to the actual, to leveraging the full power of the technology. Because it does create other motivations that are actually counter to the user's benefit. that are actually counter to the user's benefit. And we do see that, I think, in some of the, the way some of the systems are, have been developed in terms of, you know, the desire to pull things back into clouds or into spaces where it's difficult to get the data out of. I think banks are a great example of that, particularly here in Canada. If you've ever tried to pull your data out of a bank to do a budget with it, it's actually quite difficult. So yeah, let's, let's give this thing a go. So let's see if this works now. So whenever I need to zoom in a little bit, or do I need to zoom in more? Okay. No, I just, I think it's over. Yeah, this is, yeah. That should be fine. Yeah. Okay. So you can see what's going on here. So whenever I read something now, what I do is I've now got an engine in here where, and you can see it. Okay. So I'll describe what's going on. So on the right-hand side, you got chat GDP. Inside of chat GDP, I've got a thing, a GDP that I've constructed called Mike's Personal Knowledge Vault. And it's a private GTP, so you can't mess with it unless you happen to have the link. But what this is doing is there's a series of intents that have been mapped in, and I'll show you that too here in a sec, where I basically, and I've got an api and an api hosted in digital ocean you see on the left hand side um and the api itself is constructed with it's got a number of rest it's a rest l endpoint written in python um and i'll show you some of the code with that too and i'm eventually going to make this public once i get it so it's not embarrassing to publish because right now right now it's kind of hacky um but it's got a couple of different bits to it so one is the thing that's grabbing the data and kind of communicates with chat gdp and it uses you know it's got a secured endpoint and stuff like that to do this um and then in behind the scenes there's also uh i'm using python celery asynchronous framework to set up the processing side of it so whenever you add something into this thing, so you see this document you had here, let's just grab that. Oops, come here you. Yeah. And we'll go over to our friendly graph database here. Comment, I just read it out because it's great from TWB. Never let great be the enemy of good enough mike good call it works you know for most part i do get you'll notice there'll be some errors and stuff that show up here but that's fine um so and probably what's happened here nature.com yeah okay What's happened here? Nature.com. Yeah, okay. Yeah. So the way this thing works is when you drop stuff into it, what it'll do is it'll go out and start to try and pull the document back. And then it'll start to decompose that down into this graph document structure. Sometimes like it looks like in this case, I hit a robot blocker. So this article, so let's try it with this one and we'll see the processing kick off. So what ends up happening is when I submit that, it grabs, there's an agent that goes out and grabs the initial article bit, pulls it in. When that's contributed to the graph database, you'll see the salary worker kicks off. And what this does is this starts to go through and it's the thing that's doing to the graph database, you'll see the Celery worker kicks off. And what this does is this starts to go through and it's the thing that's doing the Langchain decomposition of that thing down of the incoming HTML down into a graph document structure. In this case, I'm using a structure that is basically, let's pull it up so you can see it. Again, I'll get this one. So you're actually seeing the stream of output there from the linux host um i'll do this one because maybe it can't i think venture beat is typically not blocked so it comes in a bit better yeah there we go um so you can then see that the you know it comes in with the page and then what ends up happening is the page will then have these these little chunks that are generated or sorry chunks that are generated out of it and then there's questions associated with each of little chunks that are generated out of it, and then there's questions associated with each of the chunks that are generated in a summary. So that's kind of the general pattern that I'm using essentially to pull apart the documents so that you end up with a bigger document like this, you'd end up with something that has quite a number of different uh elements or pages to it and then each page has its you know the various questions and i'm doing this again this is part of the graph right pattern where you can um and i've got a thing in the code actually where one of the things you want to be able to do with this it's a little different than vector databases is if somebody asks a question you want to have multiple retrieval strategies so one retrieval strategy that's actually pretty straightforward is you know somebody asks a question, you want to have multiple retrieval strategies. So one retrieval strategy that's actually pretty straightforward is, you know, somebody types a question in. And one of the things that I'm going to do, and I'll show you the chat thing in a sec here, but is we basically go in. So here's the endpoint chat with sources. One of the things that's really important for the when we're working with clients on the side of this is you want to give back the sources when somebody asks a question. Same thing with the user that I'm working with. They need to have the sources come back so you know where the data are coming from. The retrieval strategy in this case, and again, it's fairly simple still, but what I'm doing is when the question comes in, you want to determine whether the question is specific or general. So we fire this thing over to the LLM to get an assessment of that. And this is this is something, you know, this is really cool stuff. Even if it's simple, it's still pretty cool. So then what I do is I just set that based on that, I can set the index to target either the child level. So the small chunks and I'm using the langchain semantic decomposition utility which um goes through and it actually is pretty nice at being able to break the document up based on semantically relevant uh components so it does a great job of kind of finding sentences and stuff so where this becomes also important is if you know when we're doing this for companies that are doing the environmental, taking large environmental documents and making them conversational. So if somebody, you know, environmental impact regulation or environmental impact assessment documents are a great example of this, where if you've got a big project that's going in, you know, like a mining project where there's a bunch of stakeholders that are going to be impacted by this, the stakeholders are typically non-technical. The people who write these reports are very, very technical. So you have this impedance mismatch between the level of the documentation that's being provided and the, you know, the nature of the stuff that people are able to understand. LLMs and again, graph databases are really good for this because if somebody asks a question, I live in the, you know, that's a pretty general question. So they level objects to get sort of a more general response back from, you know, the intent that the person's giving. If the person happens to be really into fishing and they happen to fish bull trout in a certain river, they ask a question very specific okay um i'm talking about the south saskatchewan river how's that how's the uh what's the potential for light contamination to affect the you know the the bull trout and then the thing will be able to target the child chunks the small chunks to actually get very specific information back about that thing the person has asked, which is pretty neat and extremely useful. So, and then the other thing that this does, it's quite impressive, is you can do stuff like this. So usually I do this on my phone. So I've got the app, you know, one of the advantages of having the if you see that. But I could make one of the bigger. Do you want to show it again? Sure. What are the advantages having a lot of reflections? There you go. Yeah. Of doing is this is a GDP is the as the interface is you can actually have the GDP resident on all the person's mobile devices. So, for example, if I want to capture something out of a book that I'm reading, or again, this is for more targeted of my user, because they're typically a little bit more analog. So I want to capture a page out of the Richard Dawkins book, The Blind Watchmaker. So stick this in here. Need to do. And I can go read the text on this page and store it as a note called my file folders of work and let's submit that to this does if life is, we will see all sorts of things happen. I found the GTP4 old model is really good at being able to read stuff. And it can do quite a good job of reading things out of page-based text like this. And out of things like handwriting and whatnot. Confirm. No, we got an error. Okay. Bummer. Let's see if it gets it this time. Sometimes it'll try a couple times and get it. Again, this is one I've been working on recently. So what it's trying to do is it's trying to upload the image to my server. And that's one area where the OpenAI API sometimes is a little messy. We can, while we have, maybe we answer a couple of questions. Sure. maybe we answer a couple of questions. Sure. There was one question from Shang who asks, how does the document get ingested into the graph? And if this is the Minumi processor or how, how would you explain the process of getting the documents in the graph? Yeah. How do you, yeah. So that basically the way that works and you'll see Oh, how do you? Yeah. So that basically the way that works and you'll see it go here once I get this. So what will happen here is this should go. I see what's happening, right? OK, let's try this. What is happening is the the I'm using the openAI front end, like the ChatGDP side of it, this bit. There's a couple of different ways. So if I'm coming in this side, I'm actually using the, as you can see here, the ChatGDP itself basically has read the text in that page. Then I can actually take that text and store it into the, I store that as a node in the database here. Okay. Then there's a post processor that gets kicked off, that goes through and actually takes the text out of this full text here, which you probably can't see all that well on the screen, but you can see all the stuff, all the text that kind of comes out of whatever the input is. Okay, yeah, yeah, yeah. And then what ends up happening is if I go to the code here and bring that up, we basically then get into this processing function down here. Let's see, where's the tasks processing, here we go. I'll make this bigger so you can see it. And this process is exactly the same regarding of the actual type of document coming in. So if we've got say a file coming in from tasks, here we go. If we've got a file coming in from a SharePoint library, if you've got a file coming in from this one here. So this is the SharePoint document processing task. It's exactly the same pattern, regardless of what's coming in. In this case, it's coming in from SharePoint. So I basically go in, I grab the document from the SharePoint side of it, the file name and then I pass this off I do basically commit here to get the to get the base node in and what I do is I generate a uid name source of it text all that kind of stuff and then it throws it over to a processing engine okay and basically that iterates through and generates the pages um and then the child documents by using the uh you'll you'll see it kind of in here somewhere where the um uh there's a the actual processing function which basically there's the hypothetical question processor which generates the questions uh and then stores those summary generator yeah this is another part of the chain uh and then stores those summary generator. Yeah, this is another part of the chain and then the actual. So, yeah, and for for chunking, what what are you using again? So I'm using the Langchain semantic chunker. Yeah. Yeah. So that thing basically allows me to go through and take the documents and breaks them down into semantically relevant chunks. So originally I was just doing naive chunking, which basically was I was saying, OK, just break these things up into 540 token bits. But I found that was kind of messy. Yeah. So or not very accurate, accurate i guess would be a better way of describing it um the semantic chunker does a better job and it does actually make a difference in terms of like if you're breaking things in the middle of a sentence um that sometimes then means you're you're answering part of the equation doesn't work as well as you might like it to cool so yeah uh where's my, I've got too many browsers running. Okay, there we go. Let's try this again. There we go. So now that that worked. So that now you can actually see what's going on is let's get this, let's get this guy here. Grab that node. So again, getting back to that question, there's your document process coming in off this note. Then what ends up having this, that triggers the Celery worker, which then you can see it processing the chunks. So it breaks the document down into chunks, processes the chunk, breaks the document down into chunks, and that keeps going in the background. So that's a background thread process. The nice thing about that then is, you know, those are long, if you're dealing with a document that's hundreds of pages, that can be a very long running operation. So I didn't want those operations to be blocking. So that's why I put that on the async background thread. It uses So that's why I put that on async background thread. It uses actually, yeah. So that then what ends up happening is you can continue on doing whatever the user can keep doing their thing. And then in behind the scenes, you get this really nice little graph pattern kind of coming out the other side of it with the page text here and then the chunks. And let's see if we look at the chunk here. You'll notice that that chunk is really nicely bounded around the sentence structure. Yeah, that's cool. And that OCR capability, because there was another question from chat from Veronique, that is built-in default capabilities from ChatGPT. This is literally just OpenAI's 4.0 model. Check this out. Add this as a note called fieldObserve. Okay, let's do this even better. Okay, read, if I get this to work, this is really cool. I did this this morning, I'm like, oh man, read this note, correct any spelling mistakes and add the notice thing called field observations. and add the notice thing called field observations. So this is revolutionary. So somebody now, you can actually set up things now. Can you show the note again in a bigger, so that it is- Yeah, I'll pull it up here. So people can see. Let's do this thing. Cause this blew my mind when I got this working, I was like, what? Yeah, so here's my really lovely handwriting dr mike is in office like it's in the office you look at this and it's like yeah there's no way this is gonna work um but the wild thing about this so imagine this like all these years we've been forcing people to oh you need to have a form and you got to put this thing in this place in this form and yet the form form has to be done in a certain way. And you have to write the, all this stuff goes out the window because what can you do? You just have the person go with their notebook. They write a bunch of stuff down, um, in whatever the, you know, whatever order they see fit. Cause these things like the, the order of these don't matter. As long as you've got some basic structure here, the LLM is now smart enough, confirm this, which I mean, look at this. There's the site name, the date, observations, the spelling's all corrected, and it even breaks it out. Like I can go in and post-process this now with Neo4j and go, okay, give me an observations node. Give me a landowner node. What does the landowner care about? This is revolutionary stuff. And I can go list. So get the node, display the text, and explain why the landowner is concerned what that should do is that should call back into my library it should get that note so you So you see it coming through here. If life is good, we'll get that note back. Wow. You can imagine now this for field people, for people who are working in the field and they're having to deal with all these horrible, you know, the paradigms that we've imposed on them over all these years. You don't have to do that anymore. So I go list. So, I mean, I got so many examples of this. It's pretty wild. Forms data capture thing. Oh, yeah. So then if you get into stuff like this, so let's ask it a question. So we'll jump a little bit here sideways. So as you know, I basically inside of this thing, I've got all the articles that I read and stuff. So there's, if I bounce this back out, almost 200,000, I'm almost at my aura limit. Yeah, well for the free one, but I mean, it's scalable to whatever, so it doesn't matter. But yeah, so I've almost got 200,000 nodes in there of basically documents. I think it's about 2000 documents that I've got stored in this thing at this juncture of stuff that I've read. Most of it's on AI, but there's other subjects in there as well. So if I fire this interaction off, what we should see is it should, in a second here, it's a little slow this morning. Oh, it's probably because I'm streaming. Yeah. So it knows it's a question. So it should chat sources. Here we go. So this should explain why we, you know, we want to use graph databases for this stuff. And of course, this is now only taking from the documents you have stored in your. Yeah, here come the sources now. Oh, okay. I see. Yeah. Yep. So then we can should be able to go. Okay. Using source three, let's go provide a topic to research further on this subject graph and LLM. Source three is Langley, blah, blah, blah, that one. Actually, let's try two. Two might be a little better, but it doesn't really matter. So this is where this also gets really interesting is, you know, now I've got this base of knowledge out there that's got a bunch of stuff in it. So I can then start having these conversations and asking a question. So when I'm writing articles now, this is how I'm actually doing it. So now what it's done is it's actually using this source. So in behind the scenes, it's actually got all the texts from that article, kind of captured in context. So then you can get this down. So research focus, right. And then what I'm going to try and do here. Okay. So now generate research on knowledge graphs and private data sets with LLIMES. Now, if this intent works, what this should do is, so I've got inside of this thing, the only other thing I did um was uh good um what i did is i set up i used a land graph and this is really mind-blowing this this as well so land graph for folks i don't know how familiar folks are with it there's a couple of these things out there the next the next phase of this evolution of this this domain is this idea of autonomous ai agents and back when we were doing the mean ohm thing we had built that entire system the data link as it was called was built on the premise of uh what they call multi-agent system design and this is the idea behind that is you have a whole bunch of agents that are acting in some kind of a pattern uh oh um oh bad request for your own huh oh it's on the tab release side. Okay. So that's an error outside of my control. Good. Anyway, it'll still complete, but that's all right. Yeah, the agents basically then can go out. You set up a bunch of agents to accomplish an objective, and they go do their thing. And then they communicate back. And then you use the graph basically as an anchor for the, for managing the agent behavior and the agent interactions. So there's a couple of them out there now, but the problem with the agents in the past was the, you had to use deterministic decision logic to try and control how they would respond to situations like the one that just came up on the left there. So if something goes wrong, if you have some bad data or if the data needs to have more subtle handling, all these sorts of things were pretty complicated to do. LandGraph, Autogen is another one. I think Lama Index has an agent paradigm now. If you inject AI into these things, you now have a really nice, if you will, probabilistic method for handling fairly complicated situations so that the agents can actually make their own decisions about stuff. And it's pretty powerful. So this is again, where the, you know, robotic process automation and, and these, all these sorts of workflows you see in the enterprise will become revolutionized by this. So a good example is health and safety, like when we were doing health and safety stuff for capturing hazard IDs. So there's this whole pattern around hazard identification, near misses and incidents. And the handling of these types of things, you know, go up exponentially based on the level of urgency associated with it. So a hazard is, you can identify hazards like this is a potential problem, needs to be reported, but it can be mitigated and your miss is something where something has happened and it almost caused an injury or a problem but it didn't you know luckily nothing or fortunately nothing happened and then it you know a full incident is obviously something bad happened um for example when we're out wandering around in the woods last week one of the things you want to know about is, hey, are there any bears in the area? So if you're out there and you report, you know, you've got your phone here and you say, oh, there's some bear scat. Take a picture of it. You drop it into the thing. There's a bear in the area. We're not in any immediate danger, but other people in the area should be aware of this. You fire that into one of these systems. You know, it can identify it can probably identify the type of bear it is based on looking at the type of scat or the track or whatever it is. It can then, the agent could then go out and make a decision about, okay, which people are in the region, which one, who do I need to notify? Yeah. And off it goes, right? In this case, what I'm doing is something a little simpler where I've got a curation agent that basically can go or that I got too many windows open as usual. Yeah, here we go. So what the thing will do is when you fire it off, it'll it'll go out, these agents will go and I've got a search agent, curation agent, a writer agent, a critique agent and design agent. And then these are put in these are nodes in a workflow and then they're joined together with edges so it's a graph structure so one of the reasons i picked land graph for this is because i thought oh great i can actually then take this and um actually project the the thing that the the workflow is doing the graph back into neo4j um so that research that kind of came out the other side of this, and it's pulling from two sources right now, as I mentioned, it's pulling from inside. So pulling in from the private knowledge vault, or personal knowledge vault, and it's also searching to an external source, Tavoli, or you can basically any source could be a corporate source, could be whatever. And then its objective is to generate a cohesive output based on curating the information that it finds by having a writer agent that basically is prompted to take the sources that the curator gives it. So it picks five of the things that come out of the search. The search grabs 10 from each source. that come out of the search. The search grabs 10 from each source. The curator basically then renders that down, basically picking the top five items that are most in its opinion, relevant to the topic at hand. The writer then writes the article up. In this case, I'm coaching it to be a newspaper writer, but you could also say scientific journal, whatever. Or you could have a take on some other persona and you have an editor that then is challenging the curator, like the writer, hey, have you actually done a good job at this? And then it publishes it out into a form that then is in the style of a Wikipedia like article. So this is based on, there was a publication that actually described this process. So the neat thing about this, as I'd mentioned, is it comes out as a graph structure. So here's the graph here and here's the actual output here, which will pull up into the into the action and pull it back another way, too. But so the first step is it does a search on those things that I mentioned. Now, the really cool thing with this is you can actually see, and I haven't got the surface in the interactions yet, but we know exactly where this stuff came from. So the thing that comes out the other side, this article that it produces, comes from these external sources that were pulled in and from these pages of these internal sources. And you can trace that back to the exact document that it came from. So when you're talking about an enterprise and you're talking about, you know, important critical knowledge that people are making decisions on, you have to have traceability. And this is something that really the graph makes possible because I can track exactly what the agents are doing, the steps that they took. I can tell, you know, the different actions that were, that happened and whatnot. And I know exactly which sources and which chunks even that this, this output was derived from. And then you can create feedback loops with this, where you can then push this back into the system to improve the responses and everything else. So it, you know, the potential here, again, is to provide tremendous traceability and grounding of the of the responses that the LLM is giving. So questions about any of that, I guess. Yeah, that's really cool. We had a bunch of questions. So I can read a couple of these and then we can discuss some more. I think one question that maybe particularly to this, what you showed here last is probably interesting, came in from Jumbo and he asks, how do you delete or update a document from your graph? So if you have something in your database already and how do you update that? Is that currently not a thing? It's on the roadmap. Yes, you're absolutely right. So one of the, a couple of things I really wanna do is detecting duplicates and versioning, that're absolutely right. So one of the couple of things I really want to do is, is detecting duplicates, and versioning that sort of thing. So if something comes in twice, or even entity recognition. So again, this is where this, this is again, where the graph is quite useful. Because all of the utilities for doing that sort of thing are built into Neo4j. Similarity detection, you know, the embedding side of it, all that stuff into Neo4j, similarity detection, the embedding side of it, all that stuff makes it possible for you to basically go, hey, does this thing already exist? Is it similar to the other things? You can then make decisions about, do I actually bring it in again and link it as being similar or do I just say, no, it's already there and inform the user? There's a bunch of different patterns that could come into play. My thought is that even with the user that I'm kind of working with, they have a lot of duplicates, different versions of things. So ideally what would happen is you keep all the versions and then you'd use a method to kind of triage the things like, oh, you've got five versions of this. This looks like the newest one based on these criteria. What do you want to do with the other versions? You know, this sort of, again, getting it into the interaction model so that that information is actually showing up or these actions can then be taken in a conversational fashion inside of here rather than doing it in a standard UX. Yeah. Cool. A couple of, there were a couple other questions on entity resolution and duplicates. So I think that's, the more data you have, the more of a problem this is going to be. But we can, yeah, if this is a future development, then maybe we can talk about this later. There was a question on the choice of LLMs. So I know you have probably worked with a couple. So I don't know if you wanna give a recommendation. I mean, what are you using? What are good LLMs for, you know, text to cipher is something that comes up. Stuff like that. I mean, is there, is there a, do you have some recommendations or you have some kind of, you know, yeah, experience from the past from where you, you decided to deploy this with GPT, um, you know, and anything you can share here? Yeah. I mean, this is one of the great things that's happened. So, you know, I do a lot of presentations on this subject. And, you know, there's some comparisons that are sometimes made with this technology to other sorts of technology paradigm shifts that have happened. One of the great things in the case of this one compared to other situations, I guess the one that's often used is the Manhattan Project. The nice thing about this particular case, though, is that while there are certainly risks and challenges, and we won't get into that in this conversation, but the great thing is that a lot of the stuff, the work has been open sourced and is available in that way. So there are a lot of models out there and they're proliferating rapidly. In the context of the work I do at R-Curve, what we do is we're agnostic in terms of specific platforms. We will use the best available technology for the thing the client wants to accomplish. In this case, I chose GDP 4.0 and OpenAI's kind of method. Microsoft does have quite a good story on this front as well. I think for most enterprises who are, if they happen to live in that world, that's a really great option. AWS has got some stuff they're working on. Of course, Google's got some great technology. We have done a number, or Databricks is a huge one. they've got a nice engine for the hosting LLMs and stuff. So there are a number out there. And again, I think it comes down to picking the one that works best for the context that you're operating in. And, and then kind of keeping track of it. Like that's one of the hard things right now is it's such an arms race in terms of the capabilities of these things. With respect to the text cipher thing. I think the work that Tomas Britannic is doing, if you look up his stuff, he's got some really great, I think the text decipher model he's been fine tuning looks really promising and it's really evolved quite a bit. So that would be the one I'd check out there. And of course, then I think Neo4j has got a really nice story around the Olami side of the equation, which is also quite a nice model because it's pretty lightweight. So another version I was thinking of doing of this thing was actually using a local LLMm which i think you can do with olama um because it's a bit lighter than you know then you're not having to use the full chat gdp thing or gdp 4o thing and token you know paying for tokens and all that kind of stuff um yeah so um maybe maybe one more question, because we are approaching the end of the hour already. Time flies. So you have some local data and you have something from the internet. How do you avoid kind of, or is it maybe if you wanted to avoid, maybe in your case, you don't mind so much, but how would you say walling off or fencing off your personal knowledge vault that it actually becomes a vault so that an external system cannot access that. And how do you fence that off? How can basically, in this case, it looks like GPT could access your local data as well as look in the internet. And how is that from your point of view? Is that a problem? Is that something you, how do you circumvent that private data that should not be uploaded to the internet becomes in the internet? Yeah, it's kind of part of what I'm trying to test out with this, actually. That's quite a good question because it is a big concern. So certainly inside of this thing, one of the things I did do, and you'll notice it here, is I'm paying for the, you know, I'm using the paid-for version of opening eyes in Gizmo and also paying for their API. Now, according to their documentation, when you do that, they are not supposed to be using the information to train their models. So I'm relying on that guarantee that that is indeed the case. The other thing I'm doing, of course, as well as the, you know, all of the data itself reside inside of the graph. So this is isolated and you can kind of see as I've been going through these interactions, this thing will ask me for permission to communicate with the link.benome.com. And it's passing parameters back and forth. Now, of course, when the data comes back into this side of the fence and to JackGDP itself, it is inside of this environment. So I am trusting the fact that these guys are not harvesting this stuff that I'm sending in here. With respect to the internet side of it, this thing is a closed system. I mean, you can go to the URL, but it's using ssl and the usual sorts of a lot of stuff so unless you have an authentication uh password blah blah to get into it you won't be able to hopefully get into my my knowledge vault um assuming i've done that correctly which yeah um yeah so that's kind of the way it's set up um Yeah, so that's kind of the way it's set up. I guess you could. I mean, you're using a public LLM, but you could as well decide, I want to keep it completely private, locked up, and I'm not going to let it talk to an outside LLM, but I have my own. Absolutely. And then the other thing, too, is, again, the nice thing with the Neo4j side of it, of course, is there is security. You can apply security down to the node and relationship level inside of this environment, and it can be wired into the various auth providers. So, you know, one of the things I can do with this, I haven't done it yet, but it is also on my roadmap, is to, when I've got the user user node here is to then basically and you say i've got another user in here pete which you may remember pete from other neo4j lectures in the past um yeah the idea here is that these are all these nodes are all connected back to me um and there's and the other thing that's on here is there's a user action node. So I do have contributor. So I have basic roles that I've set up inside of this thing. And then there's also this user action node that then tracks the actions that I think it's a bit messy there. But the various actions that I've taken, you know, Mike has added stuff. And these are the sorts of actions they've taken on the content as the system is running. So you have traceability on that. Eventually what'll happen is, you know, once we get into sort of the multi-tenant aspect of this, I can have a public and a private side. So, okay, this information I want to make public, this information I want to keep to myself, I want to share it with this person, and so on. And again, graph enables you to do that. It's a really nice way of being able to make sections of the graph available by having just a simple link through a node that's public. Anything that's kind of hooked up to that node then becomes available to open other things, maybe connect to certain people, this kind of an idea. Super. That was a great demo. Thank you to you, Mike. I cannot just repeat. Bernd, this is a great job. Jaffa has a wonderful demo. So yeah, absolutely. It was really cool, really, really interesting. Thank you very much, Mike. The GitHub repo is kind of available. I put that on in the chat and in the video description. Yeah, so that bit basically, so I have published the processing bit of it here. Where did it go? Repositories and processor, this one. So this is, it's a bit older now, but this basically gives you the backend pattern. I don't think the semantic chunkers in this one, but it gives you the salary structure and it gives you the decomp structure for, you know, basically the stuff that the taking an input and then breaking it down into a simple null graph document kind of structure is what this does. I am planning to open source the rest of the code at some point here, but again, I want to get it. It's kind of like performing, when I perform with music and stuff, I want to practice it enough so that it's not wrong sounding, but this information that's here is essentially, music and stuff, I want to practice it enough so that it's not wrong sounding. Yeah. But this information that's here is essentially this is still relevant in terms of the way the structure like I haven't changed the structure very much since I first wrote this bit a little while ago. So this is all still basically the same. Yeah. And Peter writes, he's officially immortalized in Mike Morley's world. Yeah. Yeah, that's true. Cool. If people would like to follow you, where's the best place to do that, Mike? I publish a bunch of stuff on LinkedIn. So our curve is, we're pretty active in terms of the company really has a great ethos around, you know, putting useful knowledge out there. So a number of my team members, we've got a really great series coming up on from my colleague, Farhad is actually going to be publishing a web series on a bunch of LLMs and AI related subjects. So all of that LinkedIn is probably the best place to look. I do have a series also running on the medium side of things, and those tend to be very long, but, you know, I try and do a lot of research and on the subjects that I post up there and try and put quite a bit of thought into those things. So, you know, I'm not a big person in terms of wanting to publish snippets i like kind of getting in depth with stuff so i encourage you to check that out yeah if you have questions um yeah you can i think linkedin's probably the best to try and get me in terms of answering stuff and i try and answer as many as i can cool i'll i'll share the link to your personal LinkedIn also in the video description. So if you would like to follow Mike, then that's best to do that. For now, I think we are at the end of today's episode. I have one more thing I'd like to mention on the 12th. So this Friday, we have a special Euro Championship soccer football championship special exploring World Cup history with Neo4j. So my colleague Pierre looks at the FIFA data set from almost from the beginning of the World Cups, I think. And we have put that in NeoDash to visualize the data. And also we use NeoConverse to do a virtual chat interaction with a chat assistant basically on that data set. So that's something a little bit similar to this, what we showed today, more public data, obviously, but similar concepts. So if you're interested in that, let us know and obviously follow the channel to tune in on Friday. Notes 2024 registration page is open. So I'll encourage you all to register for that. That's going to be a fun and a very interesting event later in November. And yeah, if you want to know more about Neo4j, know more about the community, more about the interesting stuff to use and to work with graphs, then look at the Graph Academy for interesting courses, free online courses. Check out Aura Free, that same tool that Mike also uses for his vault to play around with a graph database with no installation needed, or just hang out in the community forum or the Discord server and talk to people who are as much a fan of graphs as we are. Yeah. And with that, thank you all for watching. Thank you very much, Mike, again, for the great demo, the cool showcase, and the amazing demo, really. I can only tell this has been really, really cool stuff. Thanks. Looks like I got a couple bugs to fix, but that just means I'm on the right track, I guess. Yeah. Sure. Yeah, with that, thank you all for watching. Thank you for participating in a lively discussion today. That was fun as well. And I hope to see you soon at another live stream or in the future somewhere else. Take care, everybody, and see you soon. Thank you very much for hosting me and thanks to everybody.
Neo4j Live: Personal Knowledge Vault with Neo4j GraphRAG
Explore how to build a Personal Knowledge Vault using Neo4j's GraphRAG pattern. Discover how to transform website URLs into structured graph documents, enabling advanced Retrieval Augmented Generation (RAG). This session will demonstrate the integration of Neo4j, FASTapi, Celery, and RabbitMQ to create an efficient, scalable knowledge management system.

Guest: Mike Morley

Github Repo: https://github.com/clockworknowledge/menome_processor
Mike on Medium: https://clockworknowledge.medium.com/
Mike Morley on LinkedIn: https://www.linkedin.com/in/mikemorley/

0:00 - Introduction and Greetings
1:00 - Overview of Personal Knowledge Vault
1:53 - Mike Murly Introduction and Role
3:33 - Challenges in Managing Complex Data
5:56 - Schema Flexibility in Neo4j
7:29 - Demo Introduction and Community Interaction
13:04 - Combining LLMs and Graph Databases
17:23 - Capturing and Processing Documents
23:42 - Using Celery for Background Processing
30:42 - Demonstration of AI-Powered Data Entry
57:13 - Future Development and Enhancements
59:02 - Community Questions and Closing Remarks
1:03:17 - Additional Resources and Upcoming Events

#neo4j #graphdatabase #graphrag #rag #genai #llm #knowledgegraph #knowledge