 You're listening to Gradient Dissent, a show about making machine learning work in the real world. And I'm your host, Lucas B. Wald. Today, I'm talking to Max and Sergey. Max Jaderberg is the chief AI officer at Isomorphic Labs, and Sergey Yeknen is the CTO. Isomorphic Labs is a drug discovery company spun out of DeepMind, and so we talk about the drug discovery process itself and how it's been impacted by recent advances in deep learning. I hope you enjoy this conversation. Well, why don't we start by, could you guys introduce yourselves and your company? Yeah, sounds good. My name is Sergey. I'm the CTO here at Isomorphic Labs. I have been with the company for two and a bit years. Actually, me and Max started like maybe a week or two apart. Max first, me second. But, you know, we feel like we've been there since the very beginning. And, you know, I've had a long history in tech, working in a number of different areas, building tech products for different companies and various sectors and financial technology and risk management and geospatial software in sort of e-commerce and then also spent quite a bit of time working in healthcare for the past 10 years plus really focusing on research and the cancer genomics landscape and then also took some of that research into the industry working at a company called Sophia Genetics where I was CTO and then you know now for the last two years plus in the drug discovery space. for the last two years plus in the drug discovery space. Awesome. And yeah, I'm Max Jaderberg, the Chief AI Officer at ISO. Yeah, started with Sergey. I think there was four or five people in the company when we started. Been building out all of the machine learning models, the research, and how we apply those models to drug discovery here. I was at DeepMind beforehand for seven, eight years, working on a lot of core deep learning, generative modeling, reinforcement learning, these big challenge domains that DeepMind loves, with a background in, before that, in computer vision and deep learning, really building out some of these first networks for computer vision with deep neural networks and had a company in that space as well. And so I guess actually neither of you are approaching this directly from a bio background then. Yeah, I mean, I would say we're learners somewhere along the journey. I've been, you know, in the sort of healthcare slash bio space for about 10 years now, but I always feel like this is such a deep field, and there's also so many different disciplines as well. You always feel like you kind of know next to nothing, but you get comfortable with being uncomfortable about things. Yeah, there's a lot of leaning into asking the stupid questions, probably repeatedly, a few months apart, and yeah, just trying to really take on board this new science, which is fascinating. And biology in particular is so deep that I don't think you can ever really feel like you've finished that. But we're quite lucky in the sense that we have some amazing colleagues and fantastic world expert chemists and biologists around us that really help us along the way. Well, so that's great. So you can empathize with me and probably most of the audience here and maybe kind of take us along for the journey. Maybe the place to start is what drug discovery is at a high level and how ML fits into that process today. Yeah, sounds good. Do you want to take it away? Yeah. I mean, you know, very high level. What you're trying to do with drug discovery is effectively modulate the pathway of a disease. So there's a disease and a whole patient population out there that we want to help by designing drugs and these drugs go into the body and modulate on some functional level the process of this disease in the body. So when we think about this at ISO, we're really thinking about designing small molecules, the sort of things that you can take as a pill. And these molecules will be absorbed into the blood, into the cells even, and attach themselves to these proteins, which are the functional building blocks of people. And by attaching themselves to that, they'll either disrupt or modulate the functional behavior of those proteins and so change the disease state of the person and so you know drug design is all about okay what exactly are those molecules that are a going to attach themselves to these specific proteins that are involved in disease or a whole suite of proteins and also be you know good drugs in the sense that you can actually take them as a pill. They'll get into the bloodstream. They'll get to the right part of the body, and they won't cause any toxic side effects, for example. And so what's the typical process of doing drug discovery? What did it look like 20 years ago? Maybe what does it look like today? Sorry, I think it's a great question. I think one thing that, you know, Max picked up on there, that's really important is something that our CSO Miles often tells us that this is a drug design process, less so than a drug discovery process in the sense that we don't find these drugs. We want to be able to design them specifically to solve a problem. And this is actually a bit emblem you know deep understanding of chemistry by human experts a lot of intuition you know some understanding of the disease but it's a very much trial and error process in the sense that you might form a hypothesis based in your experience that well this type of molecule might fit really well into the type of pocket in a protein where I wanted to go. And then you would go and you would make those molecules, you know, you'd synthesize them in a lab and then you would actually test, you know, does it go there? Does it bind? Does it have some functional impact on, you know, the disease of interest for you? And so when you measure it, oftentimes you would discover, no, actually it doesn't. And so, you know, drug design is actually a very frustrating discipline in that there's a high degree of failure. Even currently, when we think about how this process is performed, it takes 13 plus years on average for one drug to reach the market. It costs $3 billion plus per drug on average. And the process is really rife with kind of toil and failure. And one of our hopes is, of course, to be able to build a technology that will be able to reason over the space in a much more rational manner, such that it becomes less about this, you know, trial and error and much more about rational design. And this is one of the things that excites us the most about what we're doing. And so I guess what my impression of ML applied to drug discovery is it kind of had some sort of early hype cycle, maybe five or 10 years ago, where people got excited and then a little bit of, you know, kind of disappointment. And it sort of seems to be re-emerging with a lot of enthusiasm. Do you think my outsider perspective is accurate? And I guess, do you think that there's some technology that's driving that feeling? Yeah, I think there's something to what you're saying. to what you're saying. There was a first wave of companies, I think, that were really taking this approach of using machine learning to aid product discovery. And I think there's sort of two fundamental strains of how you can think about machine learning in this space. One, and it goes towards this notion of are you building local models or global models that generalize? And I think in the earlier days of machine learning for drug discovery, a lot of focus was on building local models. And what I mean by local model, it's where you take a small amount of data on the order of thousands of data points that are very related to the exact problem that you want to solve today or tomorrow. And you train a small model like a small NLP, even random forest SVM, against that data. And then you apply that model around, you know, just in that local region, around the sort of data that you already know about. And so this can be really helpful if, for example, you know the specific part of molecular space that you're designing around. You've got some initial wet lab data, you know, real experimental data about what works, what doesn't. And you train a little model and you help use that to interpolate and extrapolate a little bit around this region of molecular space. And sorry, before I let you go further, what does that data look like? I'm trying to picture, like, well, what are the, you know, what are the rows and columns here? Yeah, you know, think about the problem of does this small molecule bind to a particular protein? Now, in the world of local data, those rows and columns would literally just be particular protein. Now, in the world of local data, those rows and columns would literally just be the chemical formula of that molecule and the activity of that molecule. In the most critical, that could be a one or a zero. And how is that data collected? Like, they put the molecule on a test tube and shake it or something? Yeah, yeah. There's like a few forms of shaking and the whole protocols for how you shake and for how long and you know it depends on the the protein that you're measuring against each each protein is might have a different set of subtleties in terms of how you set it up or how you might measure activity of this molecule against that protein um but ultimately you know we think about this for example in a in a test tube test, an in vitro test, so to speak. Got it. Okay, so that's the past. Sorry, there's like millions of potentially different proteins, and there's many, many more different molecules. if you were building this model for a program, you would literally have the one protein that you're interested in, and then some small number of potentially related molecules. And so the coverage of the space was quite small in that name, right? Yeah. Got it. So in that world, when you train this model on this small amount of data, it's very specific to just this protein that you're targeting, very specific to even the bit of molecular space that you have data or molecules around, which means it's limited in its utility. It's limited to that particular drug design program, and it might be useful, and in many cases it is, but at the end of the day, you can't really walk away with that model and reuse it on the next drug design program because it's too specific. So you would have to do that process again on the next program. And then I think what we've seen since maybe that first wave in 2015, 2016 are actually fundamentally new models. And I think really emblematic of this is AlphaFold and AlphaFold2. These are sort of models which I would call global models in the sense that they train on as wide a variety of data as possible in molecular space. And as a result and of course as a result of amazing deep learning and research, these models after training they actually generalize outside of the distribution that they've been trained on to completely novel spaces in terms of protein sequences, in terms of the molecules that they're able to predict accurately. So it's that type of model, these global models, that we're very much focused on building at ISO because we believe that actually training on the most data gives us the best support for any drug design program that might come our way. Of course, we can specialize these models further if there is specific data, but really it gives us the best starting point and we're able to use this platform again and again on different drug design programs. And so just to make sure I'm following what you're saying, the input here is like a molecule and the output is some behavior. Is that what you're saying? Yeah. So an example model would be the input being a protein and a molecule, a small molecule that could be a drug. And the output would be, okay, do these things interact? I see. And how do you feed a molecule or a protein into a model? Do you actually treat it like a string of chemical structure? Do you put in the name of the protein? How does that work? Yeah, that's the cool thing about these general models is you can actually just specify these inputs as strings, strings of amino acids, or what's called SMILE strings for molecules, for example. There's a lot of knowledge about the actual structure of these objects. So although it's just a string, in the neural network or the way that you process that string, you can start embedding all of the known structure about these objects, the bonds that are there, and even how these might look in 3D to be inputted into the neural network and to be featurized by the neural network. And I mean, how do you feed that in? Do you actually somehow preserve the graph structure or is this literally just like a longer and longer string sort of specifying all the aspects of how the molecule looks? It really depends on the application on the model. Like a lot of this is very empirical. So, you know, sometimes raw string works. Sometimes you want to embed the graph structure. Sometimes it's point clouds in 3D. Yeah, it really depends on the application here. And, you know, a lot of the work here, like, you know, the rest of deep learning is very empirically driven. I guess, like, one thing that is worth also mentioning, Lucas, is that drug design isn't really one machine learning problem. It's actually a whole wide variety of different problems that call for specific representations of these systems. And when we think about even the moment when AlphaFold2, for example, was announced, there's a lot of hype around like, well, maybe AlphaF 2 like has solved drug design and we're done basically and we're not done in fact it's opened up the door for really amazing breakthroughs and it's really fundamentally solved the problem of predicting protein structure but we feel there's probably on the order of ten or so you know alpha-fold like challenges that need to be solved in order to actually be able to resolve this, this really complex set of questions, scientific questions around how do we design a drug that is going to solve, you know, solve the ultimate problem and have a whole bunch of other desirable properties as, as it's administered. But I think what I heard you saying earlier is that, you know, one of the big changes versus in 2015 is that, you know, kind of solving one task in the space informs other tasks. So you have some sense of like, I'm sort of imagining the foundation models that we have in language. Is that a fair analogy? Yeah, I think it's a fair analogy. Yeah, I think it's a fair analogy. And this is the sort of thing that we're starting to observe and really work out as a community is what are those foundational modeling problems in chemistry and biology that really translates into lots and lots of downstream performance and lots and lots of different tasks. It's a really exciting time for this field because I think there are a few hints of this bubbling up with things like sequence modeling on protein, sequence modeling on DNA, on base pairs, on structure modeling with AlphaFold and everything that can be done with that model. If you just take AlphaFold 2, that's a single supervised learning task of predicting the 3D coordinates of proteins and protein interactions. But if you look at the depth of publications that use this model, that hack on this model, that extend this model further, you can see there's so much dark knowledge in this model that allows it to be used for many, many very useful downstream tasks that aren't directly structured prediction, but structured prediction clearly is a precursor to thinking about these other downstream problems. But doesn't it kind of require a consistent representation? I would imagine if the core model had, you know, say the molecular structure encoded in the way you feed in data, it'd probably be hard to feed in data as like a point cloud or something, or are these models sort of multimodal in the same way that like GPT-4 is? A lot of these models operate in sequence space. So whether they're protein sequence models or alpha fold itself, like the input is a sequence. But as we know from, you know, vision space, from NLP, there's lots of opportunity to even use a model that's been trained in a single modality and extend it into other modalities as well, even without keeping these trunks frozen. So, you know, I think there can be a lot of creativity around how we use these models and how we leverage them outside of the particular training domain. I think one thing worth mentioning as well here is that, you know, in biology, there's room for several foundation models, I feel, because there's actually multiple scales and resolutions at which you can look at these different phenomena. And so, you know, at sort of the highest resolution level, you can look at sort of subatomic quantum level effects, and then you can step up to sort of atomic molecular level interactions. But then one level up is modeling what is going on inside cells or how to, you know, how to model the behavior or the function or the fate of a particular cell. But then, of course, these get organized into tissues and these get organized into organs. And then there's whole humans and even how humans interact with their environment. And so I think we have to be quite specific about which level of resolution we're talking about and we can build models that reason over that space. And there's also, we feel, a lot of opportunity to actually then think about how we can integrate, how we can reason over multiple levels of resolution. But, you know, this is very much an open research question. I guess I feel like, you know, the NLP space, which is something I'm more familiar with, you know, people used to talk a lot about, you know, kind of higher order patterns, you know, like, you know, semantics and things, and even the data you would feed in would typically be already chunked into words and sentences. And then there's been sort of a general trend to be, you know, less opinionated in the data you feed in and, you know and maybe operate on a granular level, like just treating information as a set of tokens with less priors. Is there a similar thing happening as the data size increases in the realm of biology? Yeah, we're seeing a very similar trend where more and more of the input representation gets to the lower and lower granular level. I think two examples would be actually AlphaFold itself and moving to this AlphaFold latest model that we've been developing with Google DeepMind, we really think about tokenization of the particular elements and treating things in that sort of homogenous way allows more scalable architectures to be used. And for these neural networks to actually extract that high-level structure themselves. Same things with this recent model like Evo, which is doing sequence modeling on the base pair resolution. Again, putting minimal structure in, but larger amounts of data can hopefully allow a model to internally extract these high-level representations, which could then be quite generic and be used for many downstream tasks. So I think it's a similar trend here. Are there any differences in the way that you model this kind of data? Are you using transformers? Is it functionally just like the LLMs that come out, or are there any kind of key differences that you have to do? Yeah. The big difference, Lucas, is that the scale of data is many orders of magnitude lower than in something like NLP or computer vision. So although we want to move in this direction, there's still not enough data, in my opinion, to have completely flat sequence models and extract, I would say, enough about the world of proteins from just sequence data. You can get far, and there's many protein sequence models that have done very well. But I also think there's a lot of opportunity to actually inject priors, whether these are physical priors, biological priors, chemical priors, inject priors, whether these are physical priors, biological priors, chemical priors, structural priors, into the way we represent the data, the architectures that process that, and also on the loss side, the types of loss functions we use, or theory losses or other modeling tasks that are associated with this. So I think that's where the difference occurs today. But just because of the quantity of data, you don't quite have enough to just say, hey, we'll just stick a super deep flat transformer and let it run for a month or two. At the same time, I want to say that I think in the space, kind of the context length, the input size can be very, very large, actually, depending on, obviously, like how much of the system you want to be able to describe. But if we think about a problem where we want to be able to predict something about, you know, a clinical outcome or something like a disease prognosis for a person, you know, one might envision, you know, wanting to put in the whole genome, which could be, you know, hundreds of gigabytes of data, essentially just a genome, but you may want to be able to also add, you know, other supplementary information around some of the imaging they may have done or different interventions, you know, all of the basically like clinical history. And so when we think about how to model that, we need to be able to, you able to model quite large input sizes. And even though we've seen quite a lot of progress, for example, with the latest version of Gemini on really large context links, I think there's a lot to be still done there in terms of being able to support these kind of large scale inputs. And to your point, though, of course, we use Transformers, an absolute workhorse of our neural network stack today. And then the question comes into how do we transform these inputs to really be best processed by these generic and scalable neural network modules? Just in the same way as for VIT, you have the question of, okay, how do we transform images in a way that's congruent with transformers? And okay, you've got this patch-based mechanism. It's fairly straightforward for images. What's that equivalent for proteins, for structure, for genomes? These are all things waiting to be, we're along that path of discovering that, but I think it's really early days. Interesting. So you don't feel like it's resolved. I mean, I guess I'm imagining with strings, you could do a really similar tokenization strategy, but I guess with a graph structure or something, or point cloud, it's kind of less obvious. Yeah. Lots of these things could be natively thought of as graph structures. There's a lot of people working on graph neural networks for this sort of space. But even think about, okay, how would we then transform a graph into something that could be ingested by a transformer? And does any of this actually make sense when we really think about the biology or think about this thing as a molecule rather than just a generic graph? And there's a lot of subtleties. There are things that work, but I just think as a field, we're still quite early in this and there'll be lots of space to innovate. Even when you think about genomes, right, you could think ostensibly, OK, that's a linear sequence. You can follow like a very simple tokenization method. And indeed, actually, many of the current methods do that. But that may sort of ignore the fact that there's massive amounts of structure that are not linear in the genome. fact that there's massive amounts of structure that are not linear in the genome. There is secondary tertiary structure in actually how those molecules are packed inside a cell, what could be actually the units that divide interpretable units of information in the genome. Oftentimes, even different pieces can swing together for a particular event. And so I think when we think about data representation, it's important to also inform our tokenization strategy by actually some understanding of the underlying biology and sort of, you know, how these molecules actually behave in the real world. Although it's kind of funny as I listen to you, you sound like, you know, NLP researchers 10 years ago. I kind of, you know, I. I feel like the tokenization strategies, from my perspective, get simpler and simpler and more and more painfully losing information and priors that seem really important. I wonder how this will play out. I'd love for us to be in that regime. I'm curious. I remember the Nature paper that came out with ImageNet a long time regime. I'm curious. You know, I remember like the nature paper that came out with ImageNet a long time ago. And I think one of the things that really was just so amazing about it, was taking, you know, CNNs trained on ImageNet and then applying them to detecting, you know, melanoma. And it kind of makes me wonder, like, is there ever a world where you would take, you know, models trained in entirely different, you know, domains where there's tons of data? Would they have any bearing on the domain that you're in? I feel like the, you know, the multitask applications just seem to be expanding beyond, you know, kind of anyone's wildest dreams. applications just seem to be expanding beyond anyone's wildest dreams. Could you imagine a world where GPT or Gemini becomes relevant for what you do? Yeah, I mean, it's really interesting. There are these papers, I'm thinking about an Igor Mordech paper, for example, where you take a pre-trained transformer large language model, and then you can just project arbitrary new problems into token space and just bootstrap off the internal reasoning that happens in a large language model for a completely different modality. I don't think we've ever tried that, you know, going from language into natural language into protein language. But I'm really curious to see if this sort of thing could emerge. Or similarly, if you start training a large language model on DNA, does any of that functionality, for next token prediction, does any of that functionality translate into something when you're thinking about, I don't know, protein structure? Yeah, I mean, I don't know, protein structure. Yeah, I mean, I think, you know, it's possible and we can get some transfer there. But, you know, in my mind, there's also, for example, a difference between solving a problem enough where you can write an interesting paper about it. And I think there's potentially lots of space for that. Or like solving a problem where it's solved in the real world and you know this there's a big difference for me there and you know especially at ISO our biggest focus is to solve problems in the real world and my experience tells me that there isn't necessarily like an easy lift and shift like that you need to spend a lot of time actually with the data and with the domain to still be able to get that real thorough solution. I'll give you as an example, my experience at Sophia Genetics previously where we were doing segmentation of tumors on the basis of MRI and CAT scans. And of course, we tried a whole bunch of fairly sort of established vanilla methods from vision. But, you know, what we learned actually is tumors are not like a lot of the other objects that you might normally train your model on. They're not, you know, they're not cats and dogs and sort of other images on the internet. And so, you know, the morphology of a tumor, the space it takes, it's, you know, it can have a lot of different variability. And it can be, for example, very globular, but it can also be very diffuse as it develops in the organ. And so, you know, we found that we really need to build quite specific models in order to be able to reach that performance where I'd be willing to start using that model to actually, you know, guide clinical decision making, for example, right? And so this is where I'm kind of going with that difference between what's enough to write a paper about versus what's enough to put into a medical device, for example. And some of this really interestingly comes down to that use in, you know, training these models for actual use in novel science. And it's that novel science bit which is quite interesting because with machine learning, what is machine learning? Well, you take your training distribution and you try and fit a statistical model to it. But when you're applying this model to novel science, by definition, you're trying to use this maybe in the very, very tails of that training distribution, maybe even completely outside of that. And that's quite unusual often when we think about the application of machine learning compared to, for example, self-driving cars. And so there's a lot of subtleties in how do you transfer or how do you set up your training distribution, how do you set up your model training? How do you even apply that model downstream to actually do novel science? Do you think that there is overlap with what you do in research in material science? I feel like we've had a lot of folks come on here talking about kind of doing chemistry. Like, is that relevant to you? Are there data sets that you can draw on from that? Or is biology just kind of a completely different world? I think there's a lot of interesting similarities, especially to my previous point on training these networks to then extrapolate and discover new things. Somewhat annoyingly today, it feels like that inorganic chemistry is kind of a different world to the organic chemistry that we're working with. The types of problems, the types of data are different. Saying that, fundamentally, physics is the same for these systems, right? So at some level, there has to be some notion of overlap or something we could transfer. But I haven't seen much of that intersection between those two fields like material science and, for example, organic chemistry. I guess switching gears a little bit, how do you think about open source in your world? I feel like there's sort of this big debate around open source with these language models I feel like there's this big debate around open source with these language models and these multimodal models. And I think there is starting to be a little bit of open source models coming out. How do you think about that at Isomorphic? Yeah, I guess the open source community, I think, is a great benefit, and we've all benefited from that, and I think many also participate in it. So I think it's a great working ecosystem. I think it's important to basically balance the ability to contribute to that ecosystem as well with the ability to develop specific, unique capabilities that actually sort of sustain a business. And so I think, you know, that's what we're doing here at Isomorphic Clamps and so on. Totally. What about data? I mean, I would think that, you know, here the data sets are probably much more proprietary than folks would be used to. Where do you think that goes? than folks would be used to. Where do you think that goes? Well, I think there's actually quite a lot to be said for public data sets. As we've seen, you know, tools like AlphaFold have been able to do amazing by working on, you know, being trained on the protein data bank. And, you know, I hope that statement doesn't trivialize the protein data bank because it's actually an amazing resource, the result of like, you know, thousands of years collectively of research. So, you know, I think there's still a lot to be said. And certainly, you know, I would imagine any organization that goes into this space, you know, seriously would want to look at these public commercial data sets at the first instance. But there's, of course, also a lot of unique data sets. And, you know, one question to be asked is,, how useful might that data be for training the type of models that, you know, me and Max have been talking about. If you're thinking about building a general model that works across all of chemical space, you know, you can see potentially limited usefulness in data points that are very near to each other in that space, actually. And so when we think about data, you know, we think very carefully about data set diversity, being able to cover as much as possible. Of course, some of these spaces, you know, we think about the space of, for example, all of drug-like molecules is something like 10 to the power of 60 molecules large. So sort of any notion of coverage of that space. I think, you know, we'll be a long time synthesizing data for that, but it's still quite important when we think about data set design. And so when one thinks about data, we basically need to go both to data sets that already exist, but also be ready to generate new data sets that will help us actually, you know, go into and get great performance on areas that have never been tried by humans before. You know, it's interesting, I guess, I don't want to take this in a weird direction, but I sort of have to flag, I mean, 10 to the 60 possible drugs actually seems quite small, right? i mean that that actually is much smaller than i was expecting i mean i feel like you could compress that you know until like 30 characters or less but i think maybe now i'm wondering do i actually even understand what the space of possible drugs is here i think it's pretty huge um I'm not sure I can think about rationalizing the 10 to the 60 space when you think about what each one of those objects is. It's this unique 3D arrangement of atoms, and it's not just about the 3D arrangement, but that arrangement in complex with the environment in the cell, so the protein, and that arrangement, you know, these are not just identical atoms, but these are different charges, you know, that precise, both local and global arrangement changes that whole electron density around that 3D structure. Right. So totally. So, but how do you get that down to 10 to the 60? I mean, I'm just picturing, So totally. But how do you get that down to 10 to the 60? I mean, I'm just picturing, you know, if I could string together, you know, 60 atoms of choice, like that would really that humans have developed of drug-likeness that basically constrains the set of molecules. And you can create infinite numbers of molecules, but there are certain constraints around size, for example. So one of the things that needs to happen for your drug to work, especially if it's a small molecule, is it needs to be able to go into the cell. And so the cell is only going to be able to take in molecules of a certain size. And then there's other issues like these molecules need to be soluble. These molecules need to be able to go through membranes. They need to have other behaviors. And so basically you can write a few heuristics that constrain that space to 10 to the power of 60. But, you know, in fact, you may need to search a much larger space in the end. But this is just something that helps us, you know, think a little bit around some boundaries around what the usable sort of useful molecule space might look like. Okay, well, Tim can decide if they want to leave that in. But that was actually kind of enlightening for me. Thank you. Do you, at Isomorphic, have an ambition to take the drugs all the way to commercial viability, or do you kind of stop short of that? How does it work? Yeah, it's a fascinating space, actually, and one that's been really interesting to learn more about as we have worked in this space for a couple of years. But I think there are different stages of that drug design pipeline. And a particular asset has an amount of value that is essentially corresponding to the amount of risk that is inherently left in that asset. So when you think about, you know, the stages, it's sort of like target discovery. And so when you've identified a really cool target, there's already some value in that. And then you go into hit ID, sort of lead optimization stages. At some point, you're going to go through preclinical studies, clinical trials, you know, one, two phases, one, two, three. At each point in time, you will have addressed and managed certain amount of risk. And so the value of your program goes up. And so there isn't actually one single answer as to where to stop. It really depends on your own appetite to take on the risk and on your own ability to actually do something with that. And so when we think about where should one stop, you know, we have created, of course, Isomorphic Labs to bring a unique angle of how machine learning can help transform drug design. And so that's one, you know, lens that we apply. form drug design. And so that's one lens that we apply. And then the other one is basically how much of sort of that process are we willing to go through in order to maximize the value of each molecule. And so as we rationalize our portfolio, we're basically working very actively to say, well, we think this needs to go over here. And that's a sweet spot for that particular asset. And for this one over here, we should be able to partner with somebody who we think will be much better able to actually execute, for example, the clinical trial stage of that. And so that's a great idea to work with them on that molecule. Interesting. Would you do your own clinical trials? Wouldn't that be quite a departure from your core competency? Or is there like ways that ML could play a role there? I see lots of opportunities for ML to play a role in that. And, you know, there are ones that we're thinking about as well. To me, you know, the most obvious is, you know, when you have a, there's kind of two problems. One is you want to identify a disease and then you want to find the right molecule for patients with that disease. But there's also sort of the other side of that problem, which is when you've designed a molecule, you want to find the best patients for that. And so part of that is solved within the clinical trial sort of phase of drug development. But even after that molecule is on the market, we have things like companion diagnostics. How can I find patients with the right, for example, genomic signature to be best fit to benefit from this medication? And so there's plenty of opportunities for us to apply machine learning and thinking about, for example, how does the genetics of someone influence, you know, the type of disease that they have, or how can we better classify diseases based on molecular signatures. And so, you know, we see all of these as great opportunities for us to be able to optimize things like clinical trials. If you think about the numbers, Lucas, basically 90% of molecules that enter clinical trials actually fail. So, you know, these are not great numbers. And these are, you know, some of the ones that are behind what we're seeing as these like really high costs, really long timeframes for bringing these molecules to market. And so they seem like really amazing opportunities to actually be able to go afterwards. And so I think there's, you know, there's a lot to be said for working in that area. And I guess, dumb question from the outside, but why do so many molecules fail? I mean, you described like a pretty simple sounding task, like, okay, like, does it like, you know, bind to a protein or not? And like, that's the plan. Presumably, you'd like a pretty simple sounding task. Like, okay, like, does it like, you know, bind to a protein or not? And like, that's the plan. Presumably, you'd know if it binds to the protein before you ever, you know, put it in any living organism, I would think. I mean, is it because the drugs turn out to have like unanticipated side effects or what happens along the way? Yeah, I mean, there's the side effects, but just back to this point, okay, of course, you test that this drug binds to the protein, but maybe that's just in isolation in a test tube. Now, when it's in a cell, does it actually get into the cell and bind to the protein? Okay, even then, does that cause the functional effect that you hypothesized this would cause, that this binding event would cause on this whole signaling pathway, for example, that this protein might be part of? You might have just had this biology wrong. So even your hypothesis of the signaling pathway might be wrong. You might be targeting completely the wrong protein. So when you start when you start you can measure all this efficacy in cells even and when you go into looking at the effect on the disease you don't see any effect there. And that's before you even begin to look at all the the toxicity as you were talking about just how this this molecule breaks down, what effect that can have on the body. But also, you know, sometimes these molecules, more often than not, they don't just hit one protein target, they'll actually end up hitting a whole range of targets. And so what's the effect of that hitting of completely unintended targets? That can have some very serious, you know, consequences. So yeah, there's quite a lot that can go some very serious you know consequences so yeah there's quite there's quite a lot that can go wrong in this pipeline and you know as we talk about this you can see that there's lots and lots of opportunities to understand and model more and more about this microscopic world about disease biology how all of these things connect together. And as we understand more about the different pieces of this puzzle, that means that when we go to design a molecule and we select the target that we're designing against, we can do that in a way which we have higher and higher confidence in our hypotheses of this will actually produce the desired effect on the disease that we're trying to address. How do you decide what diseases to target? Like, is there like an experimental phase where you just kind of contemplate like anything and sort of, you know, see what looks promising? Or do you kind of pick like one disease where you feel like sure that there's a big market and there's sort of like a known kind of pathway and you just sort of like look to, you know, find something new to do there? Yeah, that's a great question, Lucas. In fact, you know, just like deciding how far to take your molecules, this is another quite complex problem. And, you know, I feel like we're actually very lucky to be able to build, you know, these general models at ISO that allow us to target many different diseases. This is one of the things that excites me the most about what we're building here in terms of technology. And so when we think about that space, actually to begin with, anything is game. And then you need to form some hypotheses indeed around what market opportunities exist there. You need to be able to think about the technology that we have, where does it actually perform best? And so we need to have a set of opinions about what is going to be more or less tractable. Every drug design program is a long mission that takes many years, it takes many millions of dollars. And so the commitment to go and do one is actually a substantial decision. And so you end up building this kind of model that builds in all of these factors around what is the disease burden, what is the patient need there, what is the market opportunity, what is the market opportunity what is the technology fit all of these different factors and then that allows you to go into particular disease areas and then you look deeper and you know you you need to find potentially unique important targets and you have to just basically decide where where are you going to enter you know another another aspect of this is, you know, who else is doing what in that space as well? You know, how far have people already advanced their understanding about a particular disease? And so if you go in there, are you going to be building, for example, a first-in-class medicine? Nobody has a medicine for this particular indication. Or are you going to be building a best-in-class medicine? Somebody has already a drug on the market, and are you going to be building a best-in-class medicine? Somebody has already a drug on the market and are you going to be able to build something better because the economics and all of it is quite different in these two cases. Is there a medicine out in the market today that I might use that was discovered using these machine learning techniques? I don't believe we have anything on the market at the moment that can be sort of straightforwardly linked to having been discovered using these types of techniques at the very least. I mean, I think people have been using, as Max has said, you know, various sort of types of machine learning for probably the past two decades. That said, there's a number of molecules and clinical trials now that would have been a number of molecules in clinical trials now that would have been developed sort of with at least what I would call, you know, the last generation of machine learning methods. I mean, do these new advances in sort of the early part of the funnel mean that there's now lots more candidate medicines that seem likely to work that people would want to try like i would think that these techniques would work much better at kind of the top of the funnel if you even think of it like a funnel i'm imagining a funnel that sort of goes from like you know kind of cheap candidates that seem to work in a test tube all the way to like gone through every um every trial is there sort of like a glut now at the top of the funnel of like now we got really good at that and we need to figure out like which ones to send through like are there going to be like kind of changes downstream coming from all this effort i focus on the top you know i i think i think this is part of the opportunity at least when we think about hit identification or is you know it can look a bit like a funnel, right? You start with more stuff and over time more things drop out because you realize the negative aspects against that particular design or that particular molecule. But of course, if you have techniques that allow you to discover or design more and more initial hits, and then as you go through those stages of further and further experimentation or further design even though some things might drop out you still have tons in your funnel that you can take forward and then by having a wider array of molecules that you're taking forwards by the end of your design process when you're thinking about okay what do we actually lead with and even thinking about, you would hope to have much higher quality molecules there and a lot more to choose from for then subsequent clinical studies. I see. So you think that your rate will go up, like more than 10% will get through the funnel because there's more options and you can actually model what's going to happen through the life of the drug? Yeah. I mean, I think we're going to be much better at being able to tell what's going to happen. And in fact, we already are. And so I very much expect that rate to go up. There's many different inputs into that. But I think it's worth saying, Lucas, like nothing is really easy in this space. It's not really like a low hanging fruit kind of space where, you know, there hadn't been like big technological breakthroughs and then sort of technology comes in and kind of swoops in and just solves it all. because it's really cutting edge science, you know, at the edge of sort of what is known to humanity now. And there's been a massive industry of, you know, the world's smartest, most impassioned people thinking about the space. And so, you know, breakthroughs, I think, are going to come by developing these fundamental capabilities. They're going to advance our knowledge. And then we're going to apply them in very smart ways to be able to solve the hardest problems so i would say it's less of a question of like picking up a whole wide of wide range of like simple problems and just going to town on them it's actually unlocking things that nobody's been able to do before you know we have this disease we have this target we have no idea like where to bind to the target and how to actually make it work. And then by being able to model that space rationally and, you know, rather quickly in silico, we should be able to make inroads into some of these like really fundamentally difficult diseases. Well, why do you say that there's not a lot of low-hanging fruit? I would imagine being able to model this task in Silico versus actually trying it in a lab would just be a massive increase in efficiency. Is it because the modeling isn't reliable? No, it's not that. I mean, I think it is a massive increase in efficiency, but it's a very complex, multifaceted problem, right? And so I think while we're going to be able to massively improve the speed, massively able to improve the accuracy with which we're making predictions, and so I expect things to contract, but actually the whole problem of designing a drug from identifying a target to sort of having something that's been proven to be efficacious and, you know, non-toxic and have all the other properties, I think is going to be a series of very, very hard challenges. And so, you know, I think we need to set our expectations accordingly as well in that sense. How far along are you in that journey? Like, can you talk about like how many steps have happened with your furthest along drugs? Yeah, well, look, the way to think about Isomorphic Labs is we're only two and a little bit of years in existence. you know two and a little bit of years in existence and so you know of course our first port of focus has been to develop our technology platform and so this is something that we have been doing since the very start of course building on top of alpha fold that has been sort of a key enabling technology in the space that gives one a foothold into this you know structure based and silico drug design space and so we have developed a number of these methods and then over the course of the past year we have started doing actual drug design programs using the technology that we have developed and so you know a year in drug discovery is a fairly short period of time and so we have a number of these programs that are both our own set of our own internal targets, as well as the ones that we have partnered for with, you know, the two partnership announcements that we have made with Eli Lilly and Novartis. And so I would say it's quite early days for that portfolio, but we have been making, you know, pretty steady progress through that. And already today, we're seeing how all of that modeling work that we've been doing over the last two years is actually changing the way that the chemists are approaching the day-to-day drug design. And that's super exciting and really speaks to this longer-term inflection point in how chemists approach drug design long term. What is the role that chemists play? I would think in language modeling today, I don't think there's a lot of linguists involved at this point. Yeah, how does that work? It's interesting you say that there's not a lot of linguists involved at this point, but then, you know, the fact of the matter is every single machine learning researcher who works on NLP is an expert at language, right? Just by being a human. It's the same thing with computer vision, that every single person is an expert actually at computer vision or human vision. And so it's very easy to take these internal priors and directly translate them into machine learning models and your workflows and even on the product side how you then take those models and change the way you do you know you write a blog or you write a script with a large language model we all have these intuitions internally. It's very different in chemistry space because me and Sergey, for example, we're not native chemists. So we wouldn't have that native intuition about how chemistry can relate or can be maximally exploited by some of these models. So I think there's two ways that chemists come into this picture. One is on helping us develop this platform, helping us build these models and really attacking this fundamental science in the right way with machine learning and deep learning. And the other place that chemists come in, of course, is on the actual drug design. Maybe there'll be a world, I'm sure there will be, where we can press play and out pops your drug. But before, at that point from a model, we're of course going to have human drug designers, chemists in the loop, working with these models, being very creative together in that process to design molecules together. Okay, we always end with two questions and I want to give some space for them because I'm really interested in your answers. One thing that I would love to hear about is kind of on that journey from, you know, more research paper oriented work to trying to make this like really, you know, productionized and really working for your use case. What have been the been the unexpected challenges? Well, I would say, from my perspective, and sort of like building on the theme that Max has just covered, is the data understanding is a really difficult problem, essentially, right? We cannot just eyeball, you know, a chemical reaction, or we cannot eyeball something that happens inside a cell at a molecular level and be like, oh, yeah, that makes sense to me. Like, that's a good data point. You know, we should keep that one. So I think it's, you know, it's a big perennial challenge in the space to really be able to have great experimental design, to be able to generate really high quality data sets, to be able to actually use and training these methods. And so, you know, basically making progress in the space is, you know, linked a lot to our ability to design these data sets and understanding. And I think it's kind of a challenging scientific pursuit, essentially, to be able to really make heads or tails of a lot of these phenomena. Yeah. And then on the other side of the sort of model development spectrum, I think how these models should be applied to actual drug design, to actual science, I think is always work in progress and ever evolving as these models and the characteristics change. Again, it comes back to this point of, we train these models on a generic distribution of data, and then we try and stretch and apply them to the very tails and the very frontiers of what this model could know. Taking those raw capabilities, a raw capability like AlphaFold, and then applying it to a very particular drug design challenge, a target that no one knows anything about, the very frontier of medicinal chemistry, poses lots of challenges. tier of medicinal chemistry poses lots of challenges. There's of course going to be lots and lots of dark knowledge in this model that we need to work out how to extract for this very particular use case. And I've seen again and again in this context, even in previous contexts at DeepMind that there needs to be a lot of creative work in how you extract that juice, how you extract that dark knowledge for these really particularly challenging applications. And that's really a magic spot as well. Really exciting, but very difficult as well. Can you say anything about how you've done that or how you've approached that? A lot of this would, I'd say, fall under the hacking sort of moniker. And I use that in the best sort of way. I think we see again and again, it doesn't matter what domain actually we're deep learning, that you put these models in front of people who are really passionate about applying them and really extracting juice out of them. And people will find amazing ways to mold and hack and just completely mash up or pipeline these systems together in a way that they were never trained for, no one ever thought about before. But actually you do this and you can find some way to interpret that statistical inference in a completely new way. And it really gives value to people. And we see the same thing day to day you know for chemists for example so you're talking about the chemists you're working with or the machine learning engineers primarily doing this they'll be side by side doing this together um so yeah the chemists might not be doing the native hacking but the machine learning engineer will do that and they'll do it together. I mean, actually, I do want to add that to me, this has been probably, you asked about the hardest things. One of the things that has been the most enjoyable is at being able to actually integrate these disciplines together. You know, we often hear about some horror stories where, you know, company X for, you know, AI for drug discovery goes off, raises money to do this and build technology. And the technology doesn't quite work yet. You know, you need to do drug discovery. And so you go off and you end up having this disconnect between essentially your drug discovery teams and your teams building the tech. And one of the things that we have built into kind of the DNA of the company is we're in this together to solve these problems together. And so this is, you know, reflected into how we've structured projects. You know, we have chemists deeply involved in all of our machine learning projects as subject matter experts, as folks that are, you know, team members. We sit together as teams sort of interspersed with each other. And, you know, similarly, when we think about doing the drug discovery programs, our machine learning engineers and researchers are deeply embedded in those programs to make sure that we're getting the best of the tech. And so to me, this has been like a really integral part of actually how to crack this for real. It's funny. I really try not to inject weights and biases into these conversations at all, but I kind of can't help myself because I always wonder if people are using our visualization stuff for bio examples. Like, you know, we have a molecule viewer that always, it demos really well because it just sort of seems like science. I think it probably demos better to like the non-bio customers because everyone fantasizes about being a scientist that can look at structure and see that. But is that actually like a useful way to look at data to literally like see the molecule or how do you actually look at the data? I mean, I guess it's so much harder than with texture images. What's the process? Yeah. And it'll depend on who wants to view the data. If you're a medicinal chemist, you will happily sit there and look at a page of hundreds of molecules. Literally like a picture of them or like the scientific like notation of them from my chemistry class the 2d picture graph notation of them roll through that and you know these amazing people you know i think about like someone used to sit next to eric like just looks the molecule be like because real off characteristics about that just intuitively um but then, of course, like, as we scale up the number of molecules that we could be, you know, assessing or think about generating, then this could go into thousands or hundreds of thousands. And you could think about, you know, very unique visualizations or landscapes of these things, as we're able to measure more and more properties, or predict more and more properties with our models, we can then think about how to surface those properties, that whole multidimensional characterization of each individual molecule and then project that in a space that a chemist can browse through and really start to intuit about. So there's lots of opportunities. Actually, this raises a great point as well, Lucas, even about the wider, like about the wider field, which is product, like how do we actually, we can develop really amazing tech, we can build really great models, but we want those models to solve real world problems. And, you know, product plays a really crucial role in that. And I feel like that is a muscle in sort of the wider tech space that we've been able to grow really well like how to obsess over what that problem is like who are the users what do we need to build to really move the needle and so we've like integrated that as well you know we have a product team that are working really closely with the machine learning side to really understand these capabilities and both learn and give feedback into that research roadmap. But then on the other side, also working very closely with chemists and biologists to understand those workflows. And so we don't just have a collection of models. You're not just running a bunch of collabs as you use this. You have sort of an interface, you have a fully formed product that you can log on to, and you can run your drug design project inside. And that puts all of those models into the context of what you're doing as a chemist, and, you know, helps with your workflow, and you need to be able to visualize all of this. And so, you And so I think it's a key actually aspect of how we think the translation of some of this technology into real world impact really, really works. And I feel like that's an important piece even for the wider industry as we have a lot of cool technology that is looking for that real product insight and product application. Totally. Well, if you have any suggestions of visualizations or queries you'd like to push into us, we would be overjoyed to get that form of feedback. We'll start featuring. Please. I would love to hear them. We could take it out of this conversation, but I'm dying to hear it. And my last question, to take things in a totally different direction, if you don't mind, but this has often been interesting is like, when you think about, you know, machine learning in general could be inside your field or outside of your field. Is there any kind of research topic you think is under explored or like a research result that really excited you that you think didn't get, you know, the attention it served? Or maybe another way of thinking about it is if you didn't have your current job, is there something interesting that you would want to look into? Well, on my side, maybe is, you know, I'm really passionate about healthcare. So I think my topics are, you know, quite often about healthcare so i think my topics are are are you know quite often about healthcare but i you know i see i see sort of neuroscience as like one of the biggest mysteries that is like left in our overall understanding of actually how biology works we you know we still have quite basic ideas about how brains work and how we form like these larger you know larger sort of concepts around our world models and our memories and so on and so to me you know if I was to think about a domain where you know we haven't really had that yet like that aha moment yet that would be the domain and I think there's lots to be said for machine learning in that space, but also for, you know, other forms of modeling, simulation, and so on. But, you know, I would look for that as a key, you know, key space to really make impact in. On my side, I think at the very core, you know, I love deep learning. I've sort of been at the core, always developing new neural network modules and ways that these pieces could be put together or how you add new functionality into neural networks. you know um ways of conditioning ways of modulating attention mechanisms and and playing with these things and putting everything together in new ways i think it's great fun i said because we get to you know look at all of this develop new stuff and leverage all of it for completely new data types so mashing up all these things in in ways that aren't defined in the original research. I think if it wasn't in ISO space, you know, I think back to some of the later research I was doing at DeepMind on open-ended learning, really thinking about, okay, how do we use then these fundamental building blocks of deep learning and create these scalable learning systems that can just learn even without human-labeled data or just bootstrapping perhaps off the initial bit from human-labeled data against environments, whether that's simulations or the real world, to learn more and more and more about the world, ultimately so that you can come along and give this agent any task down the line. So I love that space as well. Awesome. Yeah, me too. Any research paper you want to point us to there that we could put in the show notes? I'll plug the X-Land work, which was some of the work from us from DeepMind, which was baby steps along that way of, you know, how do you just create these environment universes and train agents and populations of agents to get into space? So at the end of the day, you start getting these agents which can exhibit, you know, start to exhibit general capabilities to zero shot to any new task. Awesome. Well, that topic seems more and more relevant these days. Yeah. How to supply that in the language space. Or maybe in bio, you know, what would happen. Yeah. I think there's lots of analogs to how we can do drug design here to be able to. Very cool. Well, thank you so much for your time. It was a lot of fun. It's been a pleasure. Yeah. Great to see you. Yeah. Good to see you. And yeah and great to work with you guys. I really mean it on future requests. I also think healthcare is the coolest ML applications and I personally would just love to give you guys better tools. It seems like the one where it's the hardest to actually look at your data and I do feel like actually looking at your data is like this the key to success broadly yeah i mean as we ramp up the usage of the of weights and biases i'm sure there'll be lots of ideas coming i really don't mind either like really specific tiny stuff like i feel like that's the stuff we never get but if anything is just like irritating you just don't be shy like you know tell you tell your team just send it to me yeah yeah yeah i mean it's a it's a key piece of our sort of technology infrastructure and i think you know people have loved working with it but there's always more things there's always more scaling and so i'm sure there'll be no shortage of requests we'll definitely take care of your word lucas. Okay, great. Really appreciate it. Have a great day. Cheers, man. Thanks so much for listening to this episode of Grading Descent. Please stay tuned for future episodes.
Accelerating drug discovery with AI: Insights from Isomorphic Labs
In this episode of Gradient Dissent, Isomorphic Labs Chief AI Officer Max Jaderberg, and Chief Technology Officer Sergei Yakneen join our host Lukas Biewald to discuss the advancements in biotech and drug discovery being unlocked with machine learning.

🎙 Listen on Apple Podcasts: http://wandb.me/apple-podcasts
🎙 Listen on Spotify: http://wandb.me/spotify 

With backgrounds in advanced AI research at DeepMind, Max and Sergei offer their unique insights into the challenges and successes of applying AI in a complex field like biotechnology. They share their journey at Isomorphic Labs, a company dedicated to revolutionizing drug discovery with AI. In this episode, they discuss the transformative impact of deep learning on the drug development process and Isomorphic Labs' strategy to innovate from molecular design to clinical trials.

You’ll come away with valuable insights into the challenges of applying AI in biotech, the role of AI in streamlining the drug discovery pipeline, and peer into the  future of AI-driven solutions in healthcare.

✅ Subscribe to Weights & Biases →  https://bit.ly/45BCkYz

⏳Timestamps:
00:00 Episode introduction and guest overview
05:42 Max's transition from DeepMind to Isomorphic Labs
12:37 Sergei's tech background and move to healthcare
18:54 Early challenges at Isomorphic Labs
25:58 Integrating AI into drug discovery
32:16 Impact of machine learning on drug design
39:07 Introducing AI to drug discovery teams
47:29 AI's role in predicting drug effects
54:55 Future prospects of AI in biotech
01:01:00 AI ethics in healthcare
01:05:00 Broader AI applications in healthcare
01:09:00 Reflections on AI's impact on medicine & wrap-up

🎙 Get our podcasts on these platforms:
Apple Podcasts: http://wandb.me/apple-podcasts
Spotify: http://wandb.me/spotify
Google: http://wandb.me/gd_google
YouTube: http://wandb.me/youtube

Connect with Sergei Yakneen & Max Jaderberg:
https://www.linkedin.com/in/maxjaderberg/ 
https://www.linkedin.com/in/yakneensergei/ 
https://twitter.com/SergeiIakhnin 
https://twitter.com/maxjaderberg 

Follow Weights & Biases:
https://twitter.com/weights_biases 
https://www.linkedin.com/company/wandb