 Hi there, I'm Sebastian. I am a postdoc at the University of Heidelberg and I will talk today about our ecosystem for manipulating and then using knowledge. It is not strictly constrained to biomedical knowledge, but that's kind of our environment, so that's what we kind of tune into. If you want to follow along, we have this demo dataset here, which I created recently, which you can also try to start up on your own. If you want to chat with the model, you need an OpenAI API key at the moment. You can start the repository if you don't, but you will not be able to use the model. If you don't, I did the same here just now. So I omitted the OpenAI key step because I already have it in my environment. And then basically what you do is clone and then docker compose up you need docker for that obviously and this is a data set that comes from one of the neo4j data set repositories this one here it's called pole and i will be talking a bit about the general And I will be talking a bit about the general background of the project and then go into a bit of demonstration of what it does. So generally, we'd like to democratize knowledge representation, and this is a bit abstract. So we could also say we want to increase the accessibility of knowledge and the reproducibility of the pipelines that build the knowledge and the reusability and the flexibility of these knowledge pipelines. And bottom line, this comes down to increasing the community, like people telling each other what they have done and then communicating also through some more permanent solutions like repositories and i will be touching on two main points one is the knowledge engineering in biomed specifically but i think in other cases as well we have the problem that we need to engineer our knowledge to make it usable and the second one is then the more recent example of LLM technology and making the same what we do for the knowledge also available for the communication with the LLM. So if we look at biomedical data in general, I think it's fair to say that we do have a lot of data, but we still don't know absolutely everything about the problems we try to solve. I think it's fair to say we are quite far from knowing everything. As we see in this little animation in the background, it's pretty hard to comprehend with one human brain what is going on in individual cells, for example. And this brings me to what I call knowledge engineering, which is also what humans do. We extract knowledge, for example, from the papers we read. We represent the knowledge in our heads. Most of us don't need to think a lot about how we do this. And then we use the knowledge to generate ideas, draw conclusions and so on. If we want to teach machines how to do this we need to deal with the constant updates that we have and the arbitrary formats that our data come in. And so the first framework that we developed is called BioCypher and takes care of the knowledge representation aspect. It is a modular framework to make it flexible and reusable. And there are three types of modularity. The one is on the input side, where we have different adapters that deal with different input data sources. So depending on the format, depending on what exactly they contain, and very technical things like what are the column names or something like this. The second modularity component is ontology. So we want to instill some domain knowledge into our models. So in the biomedical case, we could, for example, use the gene ontology, disease ontology, cell ontology, and so on. We bring them together using a very constrained configuration in YAML format, which is nice to then also communicate to language models. And the third modular component then is the output side, where you could use a label property graph, such as Neo4j, which is kind of one of our standard use cases, but also simple representations like a Pandas data frame. And this then ultimately should help you to maintain and analyze your data but then also to archive it in a very reproducible fashion and share it alongside applications that you may do on the data but the knowledge isn't everything we also need some insights we need to generate some value from the knowledge and this is what brings me to the second framework which kind of started out as just a front-end application of some sort of langchain implementation that we've heard in several other talks today already. This was originally just a demonstration of what we envision, and it's basically a feedback loop between a researcher and a large language model. We engineer the prompts and then we bring some domain specific things into it. For example, we make the model aware and tune the model for the output of particular bioinformatics methods that we would like to interpret. We can also connect databases for injecting prior knowledge, but also as a long-term memory storage of the model. And then we can bring in also more complicated chaining applications in the vein of long chain. And since we noticed that not only we had these ideas, but that many people that we knew and collaborated with also have, for example example web applications that they would like to fuel using the modern capability of lms we decided to do another framework which now we call biochatter which is all of this functionality the base functionality incorporated into a python library and then we can distribute these functionalities among the different front ends that we use. So how do we make these two modalities communicate? Particularly since the knowledge graphs in question more look like this hairball here and the language model in their abstracted form look like this. And this is what brings me to this configuration that I mentioned before. So we are using ontologies as installation of domain expertise into the machine learning space, because human experts learn lots of things from growing up and then learning how to interact with the world and then also learning medicine or biology which makes them much different than specialist language models and what we can do in BioCypher is we can project the resources that we load in our pipelines onto specific ontologies. And we can even merge them. So you could, for example, have a very generic base ontology such as BioLink, but then if you're interested in sequences, you can hybridize the sequence ontology subtree to it there, and then you can profit from both implementations of ontologies. And this then brings us to how we connect these things. We have the knowledge graph and this YAML configuration that I mentioned. And it's just an example where we would say we have a gene and a disease and a gene to disease association. And with this, we can also generate the representation, how it would look in the knowledge graph in terms of writing the cipher query. And this we give to the Bioshatter framework to then create this conversational loop to enable the LLM to ask the knowledge graph if it deems to do so. But we can also think about other applications like an orchestration setup where we have a primary AI that leads the conversation, but then also determines that it may need to do a task, which is, for example, parameterizing an algorithm, doing a linear regression or something more complicated, or calling a vector database for some semantically similar text embeddings that we can also use to do in context learning later. And all of this we try to collect in this biochatter framework, which should be like an aggregation of these functionalities that we would like to use. So with this, I need to thank my collaborators and sponsors. And I need to say that everything that I've mentioned is free and open source. And you can find us on GitHub in the individual repositories. There are also some other repositories of demonstrations or implementations, documentation and a demo of the BioCypher. The biochatter slash chat GSE app is also to be found on chat.biochatter.org. And now I'm going to go through this demo repository and see what it actually does for us. So this is the demo repository that you can run yourself derived from this Neo4j example. Here is a sandbox of the Neo4j example that I created on the Neo4j sandbox server with this kind of guide, which we will use to go along. Here is the database that was started by the Docker container, and this is the app that was started by the docker container and this is the app that was started by the docker container. The same app that you can see on chat.bioscipher.org but the online version is kind of limited and you can only look at a demo. Oops. So this is the same repository, the demo repository, and usually when I create one of these pipelines, I go through a process that I have also recorded here in this Markdown file. So, the first thing is to explore the dataset. Which classes are in there? How large is it? How heterogeneous is it? And then kind of try to find a way around it. What I did in this case is I used the Neo4j dump, and I exported it as a CSV, and you can see that this is not super readable, but it contains all the information. We have persons, locations, phones, emails, and then at the very bottom, we also have some relationships, phones, emails, and then at the very bottom, we also have some relationships. And all the adapter that I mentioned needs to do is then find its way around this CSV and present it to the BioCypher pipeline in somewhat standardized format. So kind of like a data model, in a sense. And we put this together using a Docker Compose. And this is also available in our template repository. So if you want to build a thing like this, I would recommend to start from the template repository, which already contains these steps. This step I added. But in general, we built the BioCypher representation of the data. Then we use these two containers to import and then also deploy in a Neo4j instance. We use two steps because the two-step procedure enables to mount read-only databases, which is nice if you want to share something on the internet but don't want people to manipulate your database. If you only want a local modifiable database, you could get away with one step here. But this is the most generic interface so we do this and then i also added this chat gse container image here that can access the graph that is started on the on the docker network the orchestration of the build process is in this case a small orchestration file. For using BioCypher, you just import it and then you need one or multiple adapters. And the great thing about these adapters is that they're reusable. So we have this component board on GitHub where we can look for input adapters, output adapters, ontology adapters, and entire pipelines that use these adapters. So this is the IGVF project and we also use this board to generate a meta graph with the BioCipher pipeline so that you can have a knowledge graph about the knowledge graphs that we build. You can find this on the GitHub. And once you have the BioCipher instance with the configuration and the adapter, you simply need to get the nodes and edges from the adapter. And then in this case, we write it because we want to use the Neo4j Admin import thing. And we write the import call and the schema info which is an extended version of this configuration that I've shown in the demo in the presentation and we also have some useful information afterwards so we can also run this to kind of see what running BioCypher looks like. And here we see that we start up BioCypher, we do some logging, we load the ontology, where we do the mapping, and then the adapter generates nodes and edges into an import call file. And then we also get the types of entities, depending on the ontology that we loaded, that are in the graph. So in this case we have objects and persons, and some of these persons are officers. We have crimes and phone calls, and we have certain facts like someone is involved, an object is involved in a crime, it is investigated by an officer, some person knows another person the crime occurred at a location and so on so this is fairly straightforward but it requires some configuration this is what what is not shown here because it's kind of the default setup bioscipher looks for this you can also change it but bioscipher looks for this. You can also change it, but BioCypher looks for this. We have basic settings. So this is, for example, where we tell it that we use this ontology. So in this markdown file of the creation process, we are now at this step. We find a suitable ontology that maps our classes effectively and has the correct scope for the question. And I don't know my way around crimes very well. So I just did a Google search and I identified this ontology that is kind of basic but useful for this case. And then what we also can configure is what exactly are the nodes and edges that we have in the knowledge graph. So we start with the person. The person is not in the ontology, so we can also attach it to the ontology by saying it is a child of the actor in the ontology. Then we need to state what we expect as an input label for a person. And here I just put the exact label that we would find in the csv so we don't need to do any processing of that field we can just pass it on and then we also can filter some properties so for the person we want name surname nhs number and a phone and an email what bioscipher then does after you run the pipeline is that it also generates a schema info DML with extended information that is useful for communicating with the LLM. For example, we explicitly encode whether something is a relationship or not, which is useful because sometimes we represent relationships as nodes. So we reify these relationships and then you could get confused about whether something is a relationship or just an object. And also whether it's present, so we don't send anything to the language model which we know isn't present in the knowledge graph because that will just return an empty result. And that's how we deal with kind of the connectivity, which we'll see in a bit. And then lastly, what we need, of course, is that the adapter goes through the CSV file and adapts it to the BioCipher format. And for this, we do a simple configuration, which is kind of duplicated effort at the moment and we're working on making this more more streamlined. But it is very similar to defining a data model in the sense that we say we have different node types, persons, offices, locations and so on. And then for all of these node types, maybe we also have particular fields properties that we want to use like name, surname and so on. And we do the same for the edges. None of the edges that we actually use has a property, so it stops there. And then we also have the adapter with the main functionality of getting the data. So knowing what it looks like and then doing the ETL stuff, but then also doing some maybe abstraction processes. abstraction processes. And the most important function, of course, is getNodes and getEdges, where we pass the knowledge then to BioCipher in the format that it expects, which is for nodes, an ID, a type, and a property dictionary, and for the edges, ID, type, and properties, but also a start and end node. And we can do some more complex stuff. For example, we could abstract away the telephones in the original poll dataset that Neo4j uses. We have a telephone number, which is owned by a person, and the telephone receives calls or makes calls. I didn't want to have that in this database, so I made a simple function to just get the phone number projected onto the person that owns it and remove the phone note so that the call, either someone made a call or received a call, goes from the person directly to the phone call and not through the telephone number. And basically that's it for the knowledge graph creation process. And then we go to the usage in Biochatter. And for this, as an example, we have this ChatGSE front end here, which has some other functions if you're interested in those check them out for example we do prompt engineering to have robust prompt sets and in context learning where you can use a vector database to do semantic search of a couple of interesting documents that you think are relevant to your question. But what I wanted to talk about today is the Knowledge Graph connectivity. So we have here the Knowledge Graph connection interface. Usually it looks for the local host, but as we are running in a Docker container, we need to set this to the container name that we're using, which is deploy, the deploy step in the Docker pipeline. And since the graph was built by BioCypher and we have this line here in the creation of the knowledge graph, it already has the schema info as a node in the graph. as a node in the graph. Otherwise you would be able to give this schema info YAML file to the front end here via upload, but we don't need to do that because it's already loaded from the graph. It's this node here, which has basically all of this YAML file in a property. So then we can go to the question-answer scheme. If you have loaded a schema configuration or info file, you are presented with the content. So this is again the YAML file, but just as a usability, user-friendliness measure in the website. And then you can start asking your questions. For example, if we say here in the kind of model guide that they have, let's look at the types of primes and the number of times each occurred. So instead of writing this cipher query, you could just say what are the top primes maybe that works already no it doesn't All right. So this gives you a cipher query, which says match C crime, return C type as count order descending it's only one if we rerun this query don't run the language model again but we just update the query that is sent to the knowledge graph that is connected so we can see a couple more of them and it seems like violence and sexual offenses are the most and then public order and then criminal damage in ar, which is the same that we should receive with this query. Oh, maybe the sandbox has disconnected. That's unfortunate. But let's go on. So what is interesting about this connectivity, I think, is that the language models appear to be understanding fairly limited questions by, for example, if I enter who primed the most, which barely contains any information about what the relationships and nodes are called in the knowledge graph, it still manages to create something that works. And if we return the entire person node, then we can see a bit more information. So, of course, this is all randomly generated data, but Philip Williamson was the one that was associated to the most crimes. And it also works for fairly complicated problems. So, I saw this one, the vulnerable persons investigation, where you have, where you try to match for a person that has many friends that were associated with crimes, but that wasn't associated with any crimes themselves. So if you manage to ask the language model in the right fashion then this also works so let's see who has most...involved in crimes that was not involved in crimes themselves. And we see that it replicates the general pattern of the query. You don't get a result yet. Ah, because we have isRelatedTo, which is the family relationships. If we use the designation in the graph directly, let's see if we can get the language model to return the right thing. And then we kind of see something that looks like it's accurate. So at least it returns the result. So we have a person that knows another person that was party to a crime, where the first person wasn't party to the crime, and then we just count these friends and order descending by number of friends. Let's see the first five people that would be vulnerable to these, and it's freeman with seven friends bonnie gilbert with six actually robertson with three and so on of course we need to validate whether all of this is actually true but at least we get the query and we can modify it and this is basically all that i wanted to show it's very much work in progress and it's all open source and very collaborative. We are currently writing the paper about the BioChatter repository. So if you want to get involved, you can do so at your own pace, either by joining our Zulip chat, writing me an email or directly on GitHub. And now just have a look whether there are any questions there is something about the ontology yes so as i said i i'm not hugely invested in crime investigations so what I did was googling something like crime ontology this here probably and then I found a couple of papers and decided on one of them because I was able to access the web page where they had the actual ontology. And this is basically about it. So it took me a bit of reading to figure out whether there is actually a usable ontology. And actually, BioCypher helps you in looking at the ontologies. usable ontology and actually bioscipho helps you in in looking at the ontologies if you uncomment this here then we we can see maybe i comment this so that we don't need to run everything again So this particular function here, if you have a configuration already, is a nice overview. But even without having a configuration, if you just load the BioCypher with this ontology as the ontology to use, then you can manipulate this NetworkX object that we have as an internal representation. And then you see what is in the ontology. This includes now some of the things that I did in the schema configuration as well. So for example, making a person a child of an actor. This is why the person is already in this representation here but this is the way that i usually work with these things all right i think we're at the end of our time and i don't see any other questions so that leaves me to just thank you for your attention. Remember to get in touch if you're at all interested in these types of things. And thanks to Neo4j for hosting me. And enjoy the rest of North 23.
NODES 2023 - BioCypher/BioChatter: An Ecosystem for Connecting Knowledge Graphs and LLMs
A synergistic relationship exists between knowledge graphs and Large Language Models (LLMs) in biomedical research. In this session, Sebastian will present open-source frameworks for knowledge graph creation and LLM interaction to increase accessibility for the research community. You'll learn about the BioCypher library, which enables the creation of knowledge graphs for various biomedical applications through its Python framework. The BioChatter library will also be discussed, showcasing its role in facilitating interactions between human researchers, LLMs, knowledge graphs, and vector databases. Attending this session will give you insights into advanced research workflows, fostering accessibility and reproducibility in your work.