 So good afternoon everyone. It's a pleasure to talk with you today and thank you very much for your time to listen to a very important topic for myself and that is how generative AI knowledge graphs are going to transform clinical trial programming. programming. So just to let everyone know that I'm not incentivized in any way with the work that we've been doing with Virizion and that the work is covered with my employer under an NDA. And also that the views and opinions in this presentation are solely those of my own and don't necessarily represent those of my employer. Here's an overview of what I'll be presenting to you today. I'll be giving you my thoughts and opinions on all of these aspects around the use of this technology in clinical stat programming. And we'll go through three major sections, provide an introduction for you, and then see how the challenges of our industry bridge into the topic of today, which is developing our future systems while we're, so to speak, driving the car. So I think it's important to consider on a very fundamental level the reason that we're here today, and that is around the topic of communication. If you think about what we do for clinical step programming, it really is about attempting to communicate the outcomes of our clinical trials. outcomes of our clinical trials. So that is quantitative measures of our medicines and therapies to communicate the safety and efficacy measurements that we've observed and collected from patients and to communicate those to third parties like regulatory authorities and other stakeholders. And we do that for the evaluation of the health value of our therapies and medicines. And ultimately, what I'm seeking in this presentation is to change for the better how we do this. So we or I have some ideas on how this can be done more effectively. And I hope in sharing these with you, it might also not only give you more to work with, but you might have some ideas to share back. And ultimately, in some kind of exchange and working together, we might be able to improve and change for the better how we're communicating our clinical data to regulatory authorities for the benefit of patients. So please keep that in mind throughout the rest of what I'm sharing with you. It's nice to make a presentation a little bit more personal to identify really with the people doing the work. Work is only done by people. And so if we can gain some sympathy with the challenges that they face, then we might be able to more effectively help them. So I'm introducing a fictional character here, Surya. She's a statistical programming analyst and her team, they write SAS code to get their work done. So they're all SAS experts and they manage large quantities of code each and every day. So yeah, a so-called mountain of SAS code is something that they need to manage, maintain, modify, create, and so forth on a daily basis. So, they do this to deliver or to create the tables, listings, and figures, which collectively support everything that's contained in the clinical study report. So this is the goal of Surya and her team. Now, the challenges that Surya faced in 2024 seem to be quite similar to the challenges that she faced in 1994. And I'm making this statement based on the fact that the technology has not changed much over the last 20 or 30 years. Data storage hasn't evolved much further than tables held in SAS 7 BDAT files. And then these files are stored on a file system in some kind of structured directory hierarchy. And on a daily basis, this is time-consuming and labor-intensive to work with already at the very outset. Now, what Syria needs to do with her data, the transformation from the collected data that comes from sites through to table listings and figures and onto the regulatory authorities, this all takes place using a statistical programming language and a proprietary one at that, most commonly still today, the SAS programming language. So that's labor intensive and time intensive and requires specialized training. The way that data storage is managed and the way that the data are transformed have not evolved either in the decades since these processes began. So this is usually performed by one person. They'll be contributing to a study, and they'll undertake these processes alone. And on top of that, a second programmer will undertake what's called double programming, that is, they will reprogram from scratch any new transformations which need to be done. And then the two results are compared. And if they're the same, then the code is considered to be correct and the program is considered to do what it's expected to do. Of course, it could be that they've both made precisely the same mistake and it may not be so unreasonable to consider this outcome. This is is labor intensive, and it's not a process which allows change and innovation. So going further on the point C, the solo programmer, it brings about a mindset of siloing everything. So working on everything alone, keeping all of the data carefully disconnected from other systems and other people. And I think to some degree, this is reasonable in the sense that working with Working with patient data does require a lot of care, given that it represents personal measurements of people's personal, most personal and intimate information, that is their health. And so it brings about, though, a mindset from the very fundament of the process of keeping systems disconnected and not allowing an open way of exchanging, collaborating and developing more effective ways to perform the work that we perform. that we perform. So these are the challenges which hinder Surya and her team from innovating, working more effectively, and working more quickly, ultimately to bring therapies to patients more effectively, in a more trustful way, and more quickly. effectively in a more trustful way and more quickly. So since entering the pharma industry what surprised me was that the tech world and the software development industry have some great tools and products in place to develop their code very effectively. And they've been developing these tools for decades now. And the most obvious thing to me was version control. I did not see really any awareness around committing code at certain states and forking code to develop features and merging back after code review these features back into the production branch. So the fact that programmers are working solo and not having any awareness of version control was really just such an obvious win to me that could be gained here and today. And the same also applies to data. Although this is a more recent development, the ability to version databases is something which I think would also very much level up how Surya and her team can work effectively knowing which version of the data that their code is running on and what the providence of that particular snapshot of the data is would be tremendously helpful for ensuring quality and ensuring quality and truthfulness about specific code and data combinations. Agile development methods. The ability for teams to collaborate, so build off each other's ideas, critically assess other people's ideas and review the code of our colleagues, provide suggestions and iteration. So to continually redo what has been done to improve it. This is really the foundation of modern software development and something which I think is sorely lacking. And I'm a huge fan of the agile development methodologies, stand-ups and sprint cycles, code reviews, all of these things, I think offer a lot of benefits to simply doing better what is already being performed by Surya and her team. is already being performed by Surya and her team. And the last point here is to turn back the tide of siloing of all aspects of our processes and to move to an, well, I call it an open-ish source model. model, certain aspects of the work that we do needs to remain hidden from many stakeholders like the compound structures that we work with. But the tools that we use, so the programming languages and the processes that we use these, we don't make money from those specific things. So it doesn't make sense to hide them, but rather to democratize them and to collectively improve the tools and processes that we work with to get our tables, listings, and figures and other documents to the regulatory authorities. I make just a mention here that blockchain has been considered to democratize the data, patient data that we work with. So this would in a trustless way, allow regulators to gain value to patient data and at the same time allow patients to control or have more granular control in who might be able to use their data and for what purpose they might be able to use their data. Okay, so even though there are some steep challenges to Surya's work, and despite the existence for some time now of many tools and processes used by other industries, How are we going to get from where we stand today to where we would like to be in the future where we can be much faster in supporting the requirements of patients and deliver what they need in a more trustful and higher quality way? way. We need to consider what's holding us back and the incumbents of our industry. So these are legacy technology companies and infrastructure. These are a tremendous hurdle. Hundreds of millions of dollars have been invested to develop and maintain the infrastructure that we have today. And the companies and people who have done this have a vested interest to keep doing it. And so there's considerable resistance by these parties to moving things forward. by these parties to moving things forward. And we need to take this on board if we're going to attempt to make changes to what we have today. So the mindset that exists is what I consider a closed source mindset. And this is largely the way that people think. And people, oftentimes, again, they're not given to change. And with the great benefits from the profits of the industry, a lot of managers are quite comfortable doing things the way they've been doing them. And would rather avoid risk and just keep doing things the way that they've been doing them in the past. So mindset is contributing a lot there. In that regard, generation change is limiting us. So our rate of change at the moment seems to match the rate of our species producing new generations. It's really only the younger talent now that come with new ideas that are really motivated to bring change. And so this is also something we can't wait another 20 years to bring about the changes which are ready today. And so let's be mindful of that. The code bases that we have in place, so referring to all of the SAS code that Siri and her team have written for decades now, these are astonishingly complex. So we have many macros and nested macros, which obfuscate or hide much of the code that is actually executed for a specific task. And trying to untangle these interdependencies is really an enormous task, even for those who are deeply familiar with the study, let alone anyone who might have joined a serious team recently, or might be going back and looking over their old code from five years ago. So this just creates a huge inertial resistance to seeing how we can change what we're doing today. So lastly is motivation. And I think I've always mentioned to my children that if they're not, if they tell me they're not capable of doing something, I usually challenge them by saying that they're not motivated enough to change something. And Tony Robbins, I think the quote from him, that change happens when the pain of staying the same is greater than the pain of change, I think is something important to keep in mind here. How motivated are we? Well, I think if you're watching this, I think you are motivated to make that change. So let's dive in and see where we can go. I also have a quote here from one of my favorite stoic philosophers, and that is simply to accept what we have and be willing to have and be willing to commit to what's needed to change. And so no matter how motivated you are, we need people who can commit the time and the money and the people to make the changes that are needed, because it will take considerable resources to make the changes that stand in front of us. So we need a way to move forward, which is achievable and which is going to bring value to us. So I'm proposing that if we can get a handle on our data and code using a semantic data model to create knowledge graphs, this will be a huge step forward for the industry. Note that I've highlighted overlay because, as I'll show you in a moment, this is the approach that's been taken most recently. in a moment. This is the approach that's been taken most recently. But just to mention that data and code really in the end, in its essence, is ETL or extraction, transformation, and loading. We could be doing this much better. At the moment, we're using SAS. We're writing SAS programs to do the extraction, transforming, and, so to speak, loading between capture of the data from sites through to the tables, listings, and figures, or TLFs. But we could be doing that much better. Now, previous efforts or efforts until today to build the ontology that will be needed for a knowledge graph have been top down. So standards around the data models and the transformations that should be made, These are all written by people for people and handed down to programmers to then implement. And the way that knowledge graphs have been built have been based on these human texts. And so don't necessarily apply or don't translate precisely to the SAS code, which has been written. So that's why I refer to this as a top down effort to develop a knowledge graph ontology, and then see how that can be applied to the code. So here's an example of a knowledge graph, and I won't dwell on the details here, but suffice to say that each of the circles there represents some kind of some data point, discrete data point in a clinical trial data. So here we're working with vital signs and the lines that join the circles represent relationships between the different data points. So we bridge across here the different data models of ADAM and SDTM in the different colors there and so we have all of these relationships linking the data together. And what I want to highlight here is that on the right side there, we have the concept of vital signs. And that maps all the way over to the measurements which are taken from the patients. And so a number of transformations take place, which are all interconnected. And if you make one change in one part of the graph, then it's because of the built-in relationships, this is directly represented on the other side of the graph. So one change propagates thanks to the linking of the data to all other connected data points. And this is only a very small example. And as a complete clinical study would be about 500 times this size. So you can imagine that even one single change to one of these data points can affect the rest of the study. And without having any linking in place, what will typically happen when Surya is running her programs is that the program will, there'll be an error and she'll have to untangle the entire execution run of her SAS code to try to find out what change, where has caused an error that she's observing. And with knowledge graphs, we have this information almost, well, essentially instantaneously. So this is the reason why we want to, why I propose that knowledge graphs will be our bridge into the future of clinical stat programming. And one example that's available today, which I think is the best to date, is built by the OpenStudy Builder team. It's an open source initiative from developers or some key contributors here from Nova Nordisk and besides also one or two from Bayer, the company where I'm working at. One thing to note here with this slide is that there's no free lunch to developing knowledge graphs top-down. There's an enormous amount of work required to build the ontology, which then allows a proper understanding and configuration of the semantic model for the clinical data. Okay. So we're talking about teams of specialized programmers who are familiar with graph theory, who would be required to continue developing and contributing this across the industry. So there will be much work required to bring this forward. So if building an ontology and semantic data model is highly specialized and deeply labor intensive, and we would like to do this. We would like to accelerate this so that we can bring the benefits of knowledge graphs to start programming more quickly. There might be another way that we can do this. And the proposition here is to build a semantic data model bottom up. And by bottom up, I mean from the code. And it won't only be faster but there's a lot more that will be gained from it as well. So I have this sketch of how I understand the process of clinical stat programming to be related across the various levels of information. So at the top here, we have what's called the business process. So these are the transformations of a variable from the incoming raw data from the site all the way through to the clinical study report and finally onto what the regulatory authorities will receive. Here I have variable X, so this could be any variable in a clinical trial, and I represent the number of transformations that it will go through towards its goal. So that is the data level. In the middle here, I've placed the SAS code. So this is the SAS code that is written, which effectively is the business execution, transforming the data in order to be in the correct format for regulatory authorities to be able to review effectively. Now, in order to generate this SAS code, Surya and her team are taking a mixture of, I guess, pseudocode and English text. So talking about SDTM metadata standards and Atom metadata standards, we have also define XML, which is part of the development process for both SDTM and Adam, and there'll be other things as well. These, these, this information is derived from purely English text, so documentation, statistical analysis plan definitions, and other study specifics, which which have been created by the study statistician and other stakeholders in the study. So these documents are used by Surya and her team to create the SAS code. When I talk about bottom up creation of knowledge graphs, I'm referring to using the SAS code to generate the knowledge graph. At the moment, systems like Open Study Builder use the top-level standards, the SAP definitions and so forth, to create an ontology and then from the ontology, build a knowledge graph. But if we have a system which can read SAS code, we can generate the knowledge graphs directly based on what logic is held in the SAS code because the logic in the SAS code perfectly represents the intentions of the documentation here. So let me explain a little bit further how this might work and also go further into the benefits that I think this approach offers. So I've just described both of those points. So I've just described both of those points. And yeah, okay. So then the bottom-up knowledge graph will underpin the semantic data model for existing data and code. So I think one really important aspect here is that by building knowledge graphs based on the SAS code, there's no loss of information that can occur because the business code, sorry, the SAS code is the business execution that has taken place or will take place in a study in delivering the tables, listings, and figures. So yeah, there's no room for error there and no loss or possibility of loss of information when translating between text and code. The other thing is that once you have such a system in place that can read the SAS code, you can apply this to the current work that you're doing, but you can just as easily apply it to all previous SAS code that's been written. And so that's why I refer to the bottom-up approaches, being able to change the wheels while the car is driving because there's no need to spend time building ontologies. You can take a machine that can read SAS code to generate the knowledge graphs, and there's no extra work required. There's no extra work required. And I think another really important aspect of taking this approach, as LLMs have now come into the picture in the last year and a half, a lot of developers have considered how an LLM might perform to read SAS code and see what kind of insights can be gained from that and also other applications. code in a deterministic manner is critical in being able to extract all aspects of the code logic and not have any room for error. So just a note on what it means to have it deterministically read. Just a moment. deterministically read. Just a moment. Yeah, deterministically passed code. So determinism versus probabilism. These differ in the in by way of a deterministic system has, if you provide the same input and the same transformation, then you will always get the same output. And there's no randomness involved in that outcome. A probabilistic system takes, if you keep the transformation the same and you keep the input the same, you may not necessarily get the same output because this incorporates some amount of randomness. And this is what makes LLMs valuable is being able to distill outcomes from somewhat noisy or data that incorporates some or systems which incorporate randomness. They can still produce insights. But these insights come with a probability of likelihood, which is less than 100%. And, okay, sometimes 90% or 95% probability of an outcome is enough for what we're doing, or better than what we've been doing. But when it comes to reading SAS code that describes the creation of tables, listings, and figures, we need 100% surety. And that only comes with a deterministic system. So machine-read, machine-passed code is also fully automatable this is something that essentially doesn't require any human input once once you've built a system it only has to be checked for consistency and and so there is of course some there, but we're talking about much, much, much less compared to having to build knowledge graphs from the top down. So again, I want to highlight the fact that this provides an opportunity for us to, we can keep doing what we're doing, but we can do it much, much more effectively if we allow a machine to create knowledge grass from our existing SAS code. Because it's deterministic, it's 100% explainable and there are no hallucinations. universal logic from the code, then we also have a clear understanding of the logic and this can be used for other processes. So this single source of truth, so to speak, can feed documentation creation. So definitions and descriptions for humans to read. So here you start to see that rather than spending time creating standards and documentation, which then we hand down to the SAS programmers, we can now start to have the existing SAS programs and new SAS programs generate draft documentation from this source of business execution, the SAS code. With universal code logic, we are halfway to translating to any other language, well, any other open source language, certainly. So R on Python becomes quite easily achievable to have the programs that are created be available in R on Python. And this could go a long way to help us step or break the stranglehold, so to speak, that SAS code has around our industry. I wouldn't propose this as a motivation, but it's still something that can be appealing to some teams and companies. I think I made it clear when going through the knowledge graph that we gain end-to-end traceability once we have a complete dependency map. So in this regards, how one data point or transformation or line of code affects any other data point or transformation in a study, we know this immediately without any searching or hidden lines of code in macros or other hard-to-reach places. or other hard-to-reach places. So we were thinking, perhaps, many of us earlier last year, that LLMs might make stat programmers redundant or less valuable, that they might replace humans in some regard. Is this the future that I see? It's not. I don't see this future. And for the reasons that are described in the previous couple of slides, because they're probabilistic and that determinism is really at the heart of the drug development process, and especially if we would like to understand outcomes in patients, how that might relate back to the way the drug works, something that the industry has been trying and continues to try harder to do, to not just take empirical results, but to actually understand why their drugs are working the way that they are and then to use that information to better inform the drug development process. Probabilistic systems, essentially one way. So LLMs won't allow us to take the information. They won't produce any improvement in feeding back information from patients back into the development process. So that's one reason where I see their exclusion. And I think the way that they can help us do repetitive tasks in many aspects of drug development, however, will be valuable. So if they're going to accelerate human efforts, of drug development, however, will be valuable. So if they're gonna accelerate human efforts, not so far seeing anything revolutionary here, but if from my experience, here's already where we will see some value from this technology. Most obvious one being human text generation. And the most valuable one for us in this discussion is the CSR. So being able to instantaneously produce a draft text that can then be reviewed quickly by, well, not reviewed quickly, but generated quickly and available for review. So in generating drafts quickly, we can already save potentially weeks of time. So that really is a value add from LLMs. Same applies to documentation of code. To be able to produce drafts of this quickly and on the fly is also saving Surya time and her team from having to document or create documents of the code that they've written by hand. And the same also goes when they're writing their SAS code to have a virtual assistant provide some recommendations on the fly. That's also something that is accelerating and improving the code of quality that's being written. And being able to navigate more effectively the tabular and relational database stores that we have. Typically, and relational database stores that we have. Typically, medically trained people have Excel open and might be clumsily or working to avoid being clumsy, spending the time and effort to carefully filter those tables or search for those tables is really a time-consuming process. And being able to simply speak or type exactly what you're looking for through a large language model interface will accelerate those processes. Lastly is the creation of synthetic data. So where our patient data is tightly access controlled to ensure compliance with legal requirements of a country or area, it means that we can't really work easily with clinical data. And that limits what data scientists can do easily with clinical data. So typically, it's very difficult to work with more than one study at a time. So it's hard to gain broader insights from clinical data. It's not to say it's impossible, but there's really a lot of additional effort required to do that. And the result is that not much data science takes place on clinical data today that, for example, takes place on other non-human data like bus schedules or price data from other industries. So if we can create synthetic versions of those data which which are not in any way identifiable with with the original data set so truly completely new new data sets not not merely de-identified data sets then then this will really help data scientists it of course what's critical here is that the data retains the interdependencies between variables and features and intercorrelations so that analytical outcomes from machine learning models that we would find in the synthetic data mirror or would be the same as what we would find in the original data set. This is a delicate balancing act. And here, LLMs have shown potential and promise to take this part of or this aspect of our domain further. So, I've seen some preliminary results there that are very promising for this technology. To summarize what I've presented for you today, the challenges which Surya's team have faced to communicate clinical trial outcomes are really now limiting our efforts to bring medicines to patients. And so we need to do better. We need to change the way we're doing stat programming today in order to meet the business opportunities and the health needs of patients. So I think we now have the motivation, the tools have been there and are becoming available as we speak and I think it's more important to understand that the software development industry and companies like Verizion are available and open to partnering with us to help us get where we need to go. So that, yeah, I'm a deep believer in the use of knowledge graphs to help us gain these advances. So we need something that's practical and achievable but at the same time we need that effort to to deliver results so in this regard an automated knowledge graph generation from code this will allow us to upgrade our processes on the fly without any tools required. And lastly, LLMs are finding many use cases outside those requiring explainability. So, but I don't so far see them offering the kind of innovations that we need to really make big steps forward. Lastly, just to thank Thomas, CEO of Farizian, for all of the exchange that we've had around this topic and the work that his team are doing together with Bayer to develop some knowledge graph technology for clinical step programming. And of course, to Emil Hittfeldt for using the slide theme that he's developed. And onto Q&A. And in case you're interested to contact me after this, I'm quite happy to take questions. Thank you.
How Generative AI and Knowledge Graphs Will Transform Clinical Trial Programming
The advent of modern clinical trials to evaluate drug safety and effectiveness ushered in the age of clinical data programming. To date the pharmaceutical industry relies mainly on proprietary statistical programming languages to curate, transform and apply statistical tests to create the documents regulatory authorities require to assess a drugs safety and effectiveness for patients. We will yet depend on this process for years to come. However, the number of drugs and the clinical data they generate is increasing and the time to bring them to market to meet patient needs is decreasing.

New approaches are thus required to communicate clinical data insights to regulatory authorities with increased speed and confidence. This presentation describes the main challenges we see today and considers where the opportunities exist to overcome these. We focus on the application of machine learning technologies including semantic data modelling or 'knowledge graphs'. Additionally, with the recent availability of LLM technology, popularly known as chatGPT, we also consider if and how this will support the domains evolution to meet the demands of current and future clinical trial data processing.

Speaker: Karl Brand, Bayer
Karl is originally from Ontario, Canada. He graduated as a geneticist from Melbourne University in Australia and obtained his PhD in circadian genomics from the Erasmus Medical Center (EMC) in Rotterdam, The Netherlands. A subsequent post-doctoral fellowship within the Dept. of Bioinformatics, EMC supported by the Dutch Center for Translational Molecular Medicine identified predictive biomarkers in patients with heart disease through the commercial partnering of both clinical and basic research. At Bayer since 2016 and now in his role as a study biomarker lead, he uses his experience with open source and emerging digital technologies and frameworks to drive innovation around clinical trial data towards faster, deeper, more trustful insights on behalf of patients mana

Resources
- Learn more about us: https://www.verisian.com/
- Join the Verisian Community: https://www.verisian.com/community
- Follow us on LinkedIn:  https://www.linkedin.com/company/verisian/