DeepMind's AlphaFold 2 Explained! AI Breakthrough in Protein Folding! What we know (& what we don't)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google deepmind deep mind alphago alphazero alphafold protein dna rna folding casp casp14 alphafold 2 blog hassabis biology translation amino acid transformer convolution residual spatial graph refine gradient descent van der waals torsion angles google ai google brain nobel prize msa multiple sequence alignment covariation evolution contact prediction distogram
#deepmind #biology #ai

This is Biology's AlexNet moment! DeepMind solves a 50-year old problem in Protein Folding Prediction. AlphaFold 2 improves over DeepMind's 2018 AlphaFold system with a new architecture and massively outperforms all competition. In this Video, we take a look at how AlphaFold 1 works and what we can gather about AlphaFold 2 from the little information that's out there.

OUTLINE:
0:00 - Intro & Overview
3:10 - Proteins & Protein Folding
14:20 - AlphaFold 1 Overview
18:20 - Optimizing a differentiable geometric model at inference
25:40 - Learning the Spatial Graph Distance Matrix
31:20 - Multiple Sequence Alignment of Evolutionarily Similar Sequences
39:40 - Distance Matrix Output Results
43:45 - Guessing AlphaFold 2 (it's Transformers)
53:30 - Conclusion & Comments

AlphaFold 2 Blog: https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology
AlphaFold 1 Blog: https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery
AlphaFold 1 Paper: https://www.nature.com/articles/s41586-019-1923-7
MSA Reference: https://arxiv.org/abs/1211.1281
CASP14 Challenge: https://predictioncenter.org/casp14/index.cgi
CASP14 Result Bar Chart: https://www.predictioncenter.org/casp14/zscores_final.cgi

Paper Title: High Accuracy Protein Structure Prediction Using Deep Learning

Abstract:
Proteins are essential to life, supporting practically all its functions. They are large complex molecules, made up of chains of amino acids, and what a protein does largely depends on its unique 3D structure. Figuring out what shapes proteins fold into is known as the “protein folding problem”, and has stood as a grand challenge in biology for the past 50 years. In a major scientific advance, the latest version of our AI system AlphaFold has been recognised as a solution to this grand challenge by the organisers of the biennial Critical Assessment of protein Structure Prediction (CASP). This breakthrough demonstrates the impact AI can have on scientific discovery and its potential to dramatically accelerate progress in some of the most fundamental fields that explain and shape our world.

Authors: John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Kathryn Tunyasuvunakool, Olaf Ronneberger, Russ Bates, Augustin Žídek, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Anna Potapenko, Andrew J Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Martin Steinegger, Michalina Pacholska, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, Demis Hassabis.

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 It will change everything. DeepMind solves 50 year old grand challenge. The game has changed. DeepMind's latest AI breakthrough achieves historic new milestone, helps solve how diseases invade cells, improve protein folding prediction, AI breakthrough it also wipes your butt automatically. It is the newest DeepMind big publication. Actually, it's not a publication yet. But so what happened and I'm sure you've heard this is that every year there is this competition of protein folding prediction. So proteins are the structures that fold in a given way. And we'll go into that in a bit. But basically every year there is this competition and the results of this year's competition came out and they looked something like this. Namely, every entry here you see is a team participating in that competition of protein folding prediction. And there is one team which is DeepMind's system Alpha Fold 2, which completely dominates all the others. To the point where the problem is now considered to be solved. Now solved in this case simply means that you're past a certain number in this test set. And if you're past that certain number, your predictions are useful enough so that other scientists can basically take them and base work on them. So that's what it means for this protein folding problem to be solved. Now we don't have much information on Alpha Fold 2 yet, other than it's really good. And like a blog post and a bunch of advertisement videos by DeepMind, they are writing a paper on it. But today I want to go into this blog post and maybe parse out what we can gather from that blog post. And I also want to go actually through the Alpha Fold 1 paper. So as you can see, the performance here increased drastically with Alpha Fold 2. But you know, guesses are high that the system is going to be somewhat similar to Alpha Fold 1, of which we do have a paper. So today we'll go into Alpha Fold 1. We'll go into some speculations of Alpha Fold 2. I can already give you my speculation. It's transformers. It's attention that all of a sudden made this big jump together with probably a few other improvements to the Alpha Fold 1 system. Basically, transformers continuing to dominate the entire field. So where do we start? It's probably best. By the way, if this is not a great meme template, I don't know what is. Just saying. Just saying. Yeah. So let's actually start with the problem itself. I realize if you're here, you're probably a machine learning person, might not know too much about protein folding. So these things here are computer representations of proteins. They don't really look that way, but sort of similar. A protein essentially is a chain of amino acids. So an amino acid, where do we have this? Right here. Amino acids are these what they're called basic building blocks of life. Since the proteins are what make the cell do things. So protein are sort of the workers in the cell. They are used as signaling molecules, receptors. They are parts of your muscles. Actually, the parts that move are proteins. So they are all the work doers. Whatever something needs to work in a cell to do mechanical or work, proteins are involved. And amino acids are the building blocks of proteins. So each amino acid has a given certain common structure. And there are 21 of them. So all the proteins in the world are simply made out of chains of these 21 amino acids. And these chains, they are formed. And so there's always this sort of body that can link up to other bodies of amino acids. It's very similar. If you maybe know how DNA is structured, it's a very similar concept. Except in DNA, there are four different bases. Here there are 21 amino acids. And each amino acid is a little bit different. In each amino acid has like a tail that hangs off. So the tail can be, you know, look like this, or it can look like this, like with a side chain. Or there is there one where it's like maybe a cyclic one. I'm not sure. Maybe you can look out here or it can have sort of no tail at all. I think that's the case for glycine. So the important part is depending on this tail, the properties, the chemical properties of the amino acids are different. And then what happens next is really interesting. Once this amino acid chain is built in this. So this is the central dogma of modern biology is that you have DNA. And DNA is translated to RNA. Sorry. And then it's translated to. So it's read off, copied to RNA, which is sort of a DNA clone. And then the RNA is translated into the amino acid chain. And there is always three, three pieces of DNA mapped to one amino acid. This is very much like a compiler. Notably, the interesting part is that these steps right here, this compilation steps are done by proteins. So there are proteins that do these things. So nature in a very real sense is its own compiler. So this here you can see as like the binary. And this here is like the source code. But what happens once you build this chain of amino acid and you set it out into the cell, because of these different properties of these side chains, they're also called residues. These chain begins to fold. So this is if you know a bit of chemistry, you might know that these are these are sort of atoms that are linked with covalent bonds in this case. And it can be that part of this chain is rather like electrically negatively charged. And here part of this chain might be like electrically positively charged in a given place over a given other place. And it also depends on the surrounding medium, of course. And that means that in this case, for example, these two things will attract. And so if you release this amino acid chain, what you're going to get is sort of a bend where now the chain sort of bends. And these two, this chain right here, this tail goes like here, this tail goes like here. I'm sorry, if there is no if there is no if there is no I don't even know what to call it, pyrene rings or something like this. If there isn't an amino acid with that, I apologize. But the point is that these two things attract and sort of form this shape. And this shape is very important. We know that proteins and proteins consist of it can be hundreds, thousands, tens of thousands of these amino acids in a chain. The proteins function is, interestingly, largely determined by its structure, by its 3D structure, not necessarily by the actual amino acid. So technically you can substitute amino acids for each other. So this amino acid here can be could be substituted for another amino acid that maybe isn't the same, but is has the same properties of its side chain, such that if the structure is still the same, the protein would perform the same function. So that that is is very special property of proteins, namely their 3D structure largely determines their function. So, for example, in this step here, when you read off the RNA to the DNA, as you know, the RNA is sorry, the DNA is like this double strand of connected base pairs. And in order to replicate the DNA or to read it off, there is a there more or let's call it. There is also this step of DNA replication, right, where you copy the DNA in mitosis. In order to do that, you need to split off the two strands. You need to split it up because you want to get like a protein needs to get here to actually read it off. For that, there is a protein, a specific protein that will insert right here to split up the DNA, which is called a helicase. And that really is very important how that protein is shaped. So the shape needs to be actually such that it kind of removes these bonds from each other. So the shape is very, very important for a protein. And conceivably, you could build a helicase from many, many different amino acid sequences as long as it has the same shape. Now, I think something like something like fundamental like a helicase is probably conserved in the evolutionary tree. But I hope you get the point. The shape is super duper important. Now, the shape isn't just arbitrary. There are some amino acid chain is called the primary structure. And then the first thing that happens is that two very distinct kind of sub shapes appear. So often repeating shapes, these things, I think, are called alpha helicase or helix. This is a helix. And this here is I don't know what's in English. It's probably called a strand or something like this. These are like long sheets like I think they're called beta strands. And these things form. These are often repeated sequences. And then the third tertiary structure is when the whole thing starts to kind of fold on itself and so on and give itself the the final structure. So this is part, I guess, of the RNA polymerase, which is the molecule that reads DNA and outputs RNA. And there are many, many, many proteins. Now, since the shape is so important, it is vital that we know of it. And technically, technically, this is what why this problem is 50 years old, I guess. They say it's a 50 year old problem. I think that's due to the fact that 50 years ago, a Nobel laureate said the following. Since a protein is fully determined by its amino acid chain and since the amino acid chain determines the structure that it's going to do because of these kind of chemical properties, it should be possible to read in the amino acid sequence or read in the DNA sequence. We know what amino acid sequence results and output the shape of a protein. However, this is an extremely complicated problem. It turned out to be because they're very subtle interactions. They're not always the same. It depends, right? Like somewhere out here, there could be some amino acid with like some weird chain that, you know, everything folds on itself all the time. So at some point, these get in contact and the changes kind of the local properties here. So this is a very, very difficult problem to solve. And people have sort of tried to do this and now apparently deep mind the first system that does this to such a satisfaction that it's beneficial. All right. Now I lost my train of thought. Yeah. So the shape prediction, what happened so far is what you have to do is you'd have to sort of do this, determine this experimentally. So you'd have to take these proteins and crystallize them and then like shoot X-rays at them and then infer the structure. You can do that from crystallized proteins because I think it's due to crystals or like very regular accumulations of proteins. So if you look at a snowflake, that is if we knew nothing about the water molecule that it's like H2O, if we knew nothing of that, we could just look at a snowflake and determine this structure, this specific angles here from the snowflake. We would just look at the snowflakes and if someone tells us, look, that's all the same material, that's all water, we could infer what the water molecule looks like just by analyzing snowflakes because they're crystals. And the pretty much the same here is you build, you make crystals out of these materials, you shoot X-rays at them. And then you sort of reason over the patterns that come out. This is very, very difficult, very expensive. And so to solve this problem computationally is super important. Now we'll get to this graphic in a minute. This is sort of the only thing we know about AlphaFold2 is this graphic right now, because they have not yet released. The paper or any descriptions of the model, as I said. But what we'll do is we'll go into AlphaFold1. So this is AlphaFold1. And AlphaFold1 was participating in the same competition two years ago and was already dominant there, but not yet dominant to the point of having, quote unquote, solved the problem just better than other systems. So this is the basic structure of AlphaFold1. So what do you have right here? Let's give ourselves an overview. So the overview is the following. There are two different stages to this algorithm. Stage one is over here and stage two is over here. Maybe it's easiest to start with stage two. So the output of stage one is this thing right here, a distance and torsion distribution prediction. So this matrix here that's kind of tilted on its side, I believe there are more down here. Right. OK. So what you do right here is you take an amino acid sequence and you line it up right here. You line it up. This is the amino acid sequence. It's a bit harder if there's like a split, but let's just say a protein is... Actually, there can't be a split. Sorry, that's in the amino acids. I'm dumb. So a protein is a single chain of these amino acids. There can be multiple sort of parts to a bigger protein conglomerate. But there is this chain. You line it up here and here. So now we're building sort of a pairwise matrix between the sequence and itself. And this pairwise matrix is going to be a distance matrix. So what we are going to do is we're going to input some features about this sequence of amino acids. That's what we get as an input. And we're going to predict for any pair. So we have the sequence and we're going to predict for any pair how far are they apart? So of course, here the answer is always kind of zero. They're zero apart. But you might say, you know, these two are five apart and these two here are seven apart. But these two here are only one apart. So it's reasonable, you know, that the final structure, these two are close together. We don't worry about close together right now. We just worry about for each two, we'll predict how far they are apart. OK, so this is you can view this as, you know, a machine learning problem, right? You have an input sequence and you simply want to predict the distance matrix. So here you can see that. In fact, you can see the top and bottom. One is the predicted and one is the real. I don't even remember which one's which. You can see that this system does a pretty good job at that. There are minute differences. You really go look like down here, you can see a bit of a difference over here. There is a bit of a difference. But in general, this system does a pretty good job. So this is the output of stage one is this matrix. It's a bunch of other it's like also the torsion angles and so on. But the main thing is you predict the distances between those two. That's what you take as a input to stage two. So what stage two does is stage two builds a model of this molecule. And the model is sort of a differentiable geometrical model. So they say they. Where is it? I don't get these nature papers like they're split into two parts, but then they are they largely say the same things. I am absolutely confused by them. So we're going to jump around a fair bit. They say we parameterize protein structures by the backbone torsion angles of all residues and build a differentiable model of protein geometry to compute the coordinates for all residues. And thus the inter residue distances. So what they do is essentially they build a computer model of these amino acids. And these are parameterized by the torsion angles. Now, the torsion angle is simply the angle between any two of them. So this would be like a torsion angle of 180 degrees. And then if it folds like this, it would be torsion angle of 90 degrees and so on. And you need two torsion angles because you're in 3D. But essentially the torsion angles determine the structure of the protein. So it's one way of parameterizing it. So they build a differentiable model, a differentiable model of protein geometry. Now, the important thing is they don't do any learning with this differentiable model. The purpose of this differentiable model is such that what you can do now, if you have a differentiable model, you can run gradient descent. So imagine they pretty much lay it out right here. So they have the x, x is the output of your differentiable geometry, right, of your torsion angles. Let's just call it this Greek letter phi, psi, whatever. If x is the output and now x goes into your loss function. So x goes into your loss function and the loss function simply compares x. To the predicted x. So the loss function will take in x and it will compare it to the x that you predicted from this thing here. So we start off with a flat chain, maybe. Actually, I think we start off with some initialization because they also predict the torsion angles directly. Right here, they predict the torsion angles directly. And that's what we initialize from. But let's just say we initialize from the flat chain. And then because this is differentiable, we do so your L is x minus x prime. And what we do is we derive the loss with respect to the angle, to the torsion angle. So we can do this since this is differentiable. So now we know how do we need to change the angle, which is this thing right here, in order to make the loss smaller. And maybe it says, actually, you need to turn it down, right, make the angle smaller. And we do that. Okay, cool. Now it's only 90 degrees. And then we do it again and again and again. And you can see that by changing all the angles such that this loss is smaller, we end up through steps, step, step, step. We in our computer model, we sort of replicate this process that happens in nature, where what we feed in is how far any two amino acids should be apart. And by running gradient descent, just gradient descent on the torsion angles, we figure out what do the angles need to be in order to make this happen. So first, we predict all the distances, and then we figure out how do we need to set the angles such that these distances are fulfilled. These are not true distances. These are predicted distances, right? So everything depends on how well we can predict these distances. But once we have them, we can sort of replicate in our computers the process as it happens in nature, except in nature, the whole folding is dependent on these all these chemical interactions and so on. And now we do none of this. We simply look see how do we need to fold in order to make these distances in our computer model like these like the distance between this and this and this and this. Any two distances may agree with the distances that we have predicted right here. And you can see that over time, this as you run gradient descent, this goes up. This this TM score was up the root mean square distance goes down between and then you of course can compare it if you have a test set with stuff that people have already figured out, you can analyze these metrics and see that indeed, you do get the correct folding. It's also pretty interesting that so here in blue and red, I believe you have. Yeah, exactly. So the the helix in blue and the strands in red. So in this case, you from if you have this folded structure or partially folded structure, you can already see that these sort of substructures emerge like this is a helix, right? As you can see, and then you sort of made this maybe a strand and so on. There are ways to heuristically classify that. And you can see that if you look at the database, right, you can see that this here is a strand. These are helices, and this is a strand and these are helix. This is a strand and so on. And you can see that the model here is what the model thinks at the beginning. It doesn't get many things correct, though it does some, but then over time, it sort of refines its guesses until at the end, it's pretty much equal to what the database to what the true sample is. And here is simply the distribution of, I guess, confidence about these things and the torsion angles right here. So it, as you can see, this two step process is the key here to do that. Now, Alpha Fold 2 conceivably probably changes this a little bit. But again, we're not sure. The step one right here is a deep learning system. So step two is simply a gradient descent procedure that you run at inference time, right? This at training, you can you can just do step one. So step one is is the machine learning bit. So the goal is to output this distance, this distance tensor right here. And there are more things than distances, as we said, there are torsion angles and so on. But ultimately, you want to output this distance matrix. And how do they do it? You can already see it's a deep neural network. So you want to build a input data point, let's say, of L by L, which is sequence length by sequence length. So you want to collect some features, you don't know the distances yet, right? But you can collect some features that are either pairwise features between these two things, right? So here, maybe this is, I don't know, leucine, and this is what's a different amino acid glycine. And in here, you want to put features, maybe it can be features for that position, right? Maybe leucine here is at the 100th position in the in this particular protein, and this is at the 90th position. So we want to put in some features of that that you can derive from a data set. You can put in correlation statistics in general between these two amino acids. You can even put in just single features. So you have these tiled L by one features, which is just features for the sequence itself, not pairwise features. But what you do is you simply replicate them along along any given dimension right here. You always put the same features. This is very common in conv nets. And you can even do a scalar feature. So there are some scalar features. And what you would do is you would simply fill an entire plane with that scalar feature, all the same number. It's just easier to do it like this because it fits into the convolutional architecture. Well, so you want to provide all kinds of features and the features they provide are, you know, plentiful. And a lot of them do introduce some domain tools, domain expertise and so on. But once they have that, they simply take that sort of image with many, many channels and they predict this image if you want. So it's just an image to image translation problem. And they do this via a convolutional neural network. As you can see, there are 220 residual convolutional blocks. Now, I assume that most of the viewers of this video are familiar what convolutional neural networks are. If not, I'm deeply sorry, but we'll not go into that. But you can see they sort of they tile this tensor right here and they tile it differently from from from instance to instance. So they tile it in the training procedure. They always tile it differently. That's a form of data augmentation. But ultimately, you slide over this image with this 64 by 64 ConvNet and you produce the image on the right. Here you can see an inherent weakness of these approaches, namely that this thing can only ever look at 64 amino acids at a time. So now that can that can be the same if you're on the diagonal of this. Let's say let's say this is not 64 by 64, but three by three. If you're on the diagonal, you would only consider three amino acids and their interactions with each other. Right. Any to any interactions with each other. If you're off the diagonal, what you would consider is maybe these three amino acids and these three amino acids. And you would only consider you consider features for maybe for those three, but interactions only in between like the these not interactions actually within the same amino acids. So you're the thing that you can look at any point in time is going to be very limited. Right. And these so these distances that you get out here, they necessarily cannot directly depend on, let's say, this amino acid right here. You always have this limited view of your protein that sort of local. Now, people argue that that's actually enough if you look at maybe the green connections right here in order to establish them. What's most important is the vicinity of these of this amino acid and the immediate vicinity of this amino acid. And, of course, the interaction between those two vicinities. But it is quite conceivable that this green thing down here being so close will actually sort of push the two apart and sort of do this interaction, which, in my understanding, would not be covered by a system like this. And that's where alpha fold two, I believe, is is one point where it makes the big gains that it does. Now, the features that go in here, as I said, they are they're quite plentiful. One of the more interesting features is this MSA, these multiple sequence alignment. And I believe they're they're up right here. Yeah, sequences. So here they introduce them in recent years. The accuracy of structure predictions has improved through the use of evolutionary covariation data that are found in sets of related sequences. Sequences that are similar to the target sequence are found by searching large data sets of protein sequences derived from DNA sequencing and aligned to the target sequence to generate a multiple sequence alignment. Correlated changes in the positions of two amino acid residues across the sequences of MSA can be used to infer which residues might be in contact. So what what this I've searched out one of the papers right here, and this is from a paper called improved contact prediction proteins using pseudo likelihoods to infer POTS models. The entire basis here is that here is your chain of amino acid that you're considering. And this is you. This is the human. And they actually have one like a very similar graphic in their blog post. But we'll draw this ourselves. I'll just kind of sort of copy it. And what you do is you go and look into your database. Right. This this is the amino acid sequence. And each amino acid can actually be abbreviated by a single letter since they're 21. And luckily, the holy alphabet creators have given us what 26. So that fits. So each of these can be done by like S Y C M D and so on. Can be then you go look into your database and your database is of sort of all of life. And you go look for similar sequences and there are tools that you can very quickly see through databases and get out similar sequences to yours. And those are sequences that are overlapping in amino acid sequence. Right. So you could find in the fish. This is an alpha. This is not a fish in the fish. There is a similar sequence right here in the iron. Like this is OK in the whatever this is. This might be a horsey. No, this is not a horse. Let's make an alligator out of this. So in the alligator, raw does the alligator have? There might be a sequence and so you get the point. My drawing skills are to be criticized in another video. So you search for all of these similar sequences just by amino acid sequence and from the correlations, you can derive something. For example, I've already told you that sometimes you can substitute an amino acid and the sort of function of the protein isn't really affected. And this may be what you can see right here. So in the human, this is maybe a D, but or sorry, maybe this here. It's a C, but in the in the let's call this an M in the fish, it's a C2. But, you know, in the alligator, it's a P and in the cockroach, it's K and so on. You can see that maybe if the alignment is good, right, this is sort of from the same protein or from a protein that does maybe the same thing in these life forms, because life is continuous. Often these things are preserved or slightly modified. So here there are variations that happen in life, right? Mutations, variations. And so we can safely maybe assume that, you know, a K, whether there's a K or a P or a C in this particular point, it doesn't really matter. The shape doesn't seem to be too affected. So that's step one. And now so this might be this protein, this amino acid right here, you see, whether it's this chain or whether it's this chain, maybe doesn't really matter for the function of the protein. However, if you look at two proteins that are in contact, what needs to happen? So if my protein here has this chain and the other protein has sort of is in contact, that means there is like a chemical interaction between the two. So now if a mutation happens, if a mutation happens and the protein is still functioning the same way, but the mutation happened, let's say it's now this right here, that must mean the shape is still the same sort of. And that must mean that probably if one of them changed, the other one probably changed sort of analogously at the same time because structure is preserved, function is preserved. So structure is preserved. And since structure is determined by chemical interactions, one of the parts changed. That means probably the other part has changed as well. So maybe now this is sort of this chain right here. So what you would expect to see in the statistics is that if one changes, the other one changes accordingly. So there can be variations, right? There can be mutations. But if the mutation happens in one of them, a corresponding mutation should happen in the other one as well. Otherwise, the protein would be non-functional and the organism would sort of die. Not always, but you know, this is kind of a statistics game. And this is what you see here. Like the fish has an S like the human and an H right here. But the alligator has an F and a W right here. And then in the cockroach, you see the S and the H again, and so on. And here down here, you see the F and the W again. And this is an indication that these the correlation here is an indication that these two things might be in contact with each other. Now, there have been systems, for example, in this paper right here, that directly go from these statistics to contact predictions and so on. Alpha Fold simply takes in this stuff as features. So this right here, all of this, there can be, I think they derive 488 features from this. So this goes down here. I think they say it again. As I said, this is confused. Like here, article stops, references, article starts again. Thanks. And they like say almost the same things. It's just a little bit more detailed, but it's not longer. So here they derive 484 features from these multiple sequence alignment for each residue pair. Right. So in our big tensor right here, right here, each dot, each thing right here already now has 400. So each one of these already has 484 features and then some more. Right. This is already this is from the MSA, but then more features. So they incorporate lots of features right here. Where are we at? Here. They incorporate lots of features. In addition, we provide the network with features that explicitly represent gaps and deletions. They also represent scalar features and so on. So here you can see they have scalar features, sequence length features, amino acid type, profiles, HH blitz profiles. These are all sort of these comp bio tools, these genetic tools and so on. You also have sequence length features. These are these 484 features and so on. So these are all akin. There are some positional. One of these acts as positional encodings and so on. So lots of features, input, convolutional network, output, the distance matrix. And that's that. Right. So there you have the inputs, the distance matrix from the distance matrix. You can run gradient descent to get the protein structure at inference time. And they make some pretty cool points. Not only do they compare the distance matrices, but they here is the not only the single prediction for the distance, but they, of course, output a probability distribution. They bin all of these distances. They output a probability distribution. And you can see that the black line in these histograms. So this is this is for a particular thing. This is for this this red line, this red row right here. It's the extraction. So it's for one of the amino acid, the distribution of probabilities of distance bins. With each of the other ones. So this is number 29. And we look at the distance between number 29 and one, two, three, and so on. The black line represent the represents, I think, eight angstroms, which is generally considered the barrier for being in contact or not being in contact. And here it's colored in blue if not in contact and in green if in contact. And the red bar represents the true distance. And you can see this is pretty accurate. So whenever the network predicts blue, usually the red line is on the right of the black line. And if the network predicts no, sorry, these green and blue is the ground truth. So whenever it's blue, the network's distribution is usually shifted towards the right. And whenever it's green, the network's distribution is shifted towards the left. There are some failure cases, as you can see right here, the network predicts a higher distance than the than the the truth. Right. You can also see what's pretty interesting is that the most accurate predictions sort of the highest confidence, the smallest variation in distribution are around here, which is exactly around. So 29 would be in the middle right here. And that's where you find the most accurate predictions, of course, since local local distances are much more easier. And then as you go farther away, you get less sure. And this is a cool thing. So here you can see model prediction versus true distance fits fairly well. But you can also see that here they plot the standard deviation of their prediction. And you can see that the the means are very close, but the higher the sort of standard deviation, the less sure the model is. So there seems to be a there seems to be like a built in confidence metric. Right. So you can see the distance error it makes here are bigger and also its standard deviation is bigger at the same time, which means that you can sort of look at the standard deviation of this distribution right here. And that is an estimate for how sure how confident the model is in its prediction. And apparently that's something that in Alpha Fold 2, the the model relies upon very, very crucially. So here you these are just on the bottom, you see one of these residual blocks here, more distance matrices. They do a lot of analysis in this article, which is pretty cool. So you can go into it fairly far. They also have look at what the network pays attention to. And it makes a lot of sense like it pays attention to kind of these these helices and then these interactions between the helices and the parts where it's close in close contact with and so on. But now we want to go into Alpha Fold 2. Alpha Fold 2. Now the what we have isn't much we have this graphic right here, which is also in the article. It's probably better we go to the blog post to the blog post is like a fluff piece saying we they are going to publish a paper. But of course, they don't have it yet because we've just gotten the results. Yeah, they have they have these these cool these videos were like, ah, so good. As I said, I like there's so many Twitter threads with. I'm not usually up for the hype, but this is the best thing and so on and everyone's everyone's hyping and I thought, is it really up to me to be the grumpy one here. But then I couldn't find anything to be grumpy about. So this is what we what we get. Let's see. It's it's deep mind. I expect them to not fully maybe release the code. Maybe they will. But in Alpha Fold 1, they've released like half the code, which is already pretty cool. So there are open source implementations based on that. So again, nothing to be grumpy about. All right. So what can we what can we say? They say a folded, folded protein can be thought of as a spatial graph. And then this is kind of a new word they introduced. But ultimately, it's simply this distance matrix that we've seen before is a representation of that spatial graph. Right. It's simply a graph of nodes and the edges say whether or not they're in contact or respectively how far they are apart, where the residues are nodes and edges connect the residues in close proximity. This graph is important for understanding the physical interactions within proteins as well as their evolutionary history. For the latest version of Alpha Fold used at CAS 14, that's this challenge. We created an attention based neural network system trained end to end that attempts to interpret the structure of this graph while reasoning over the implicit graph that it's building. I look this it's sound like this. This is fluff. Maybe. I don't know. But this here attention based. OK. So I'm going to guess for sure that they've replaced this convent with and with a transformer style with an attention attention layer or multiple attention layers. They say it uses evolutionary evolutionarily related sequences, multiple sequence alignment and the representation of amino acid residue pairs to refine this graph. This is this is what we've already seen. So use these other sequences plus like a lot of stats that you can gather from the data sets on amino acid pairs in order to develop this this graph. And the graph is distance, the distance matrix or other things we'll see in just a second. They say by iterating this process, the system develops strong predictions of the underlying physical structure of the protein and is able to determine highly accurate structures in a matter of days. Additionally, Alpha Fold can predict which parts of each predicted protein structure are reliable using an internal confidence measure. Again, this is something that we've already sort of seen in Alpha Fold 1 that there is sort of an internal confidence measure. And the part here is they say by iterating this process, which could mean that it's no longer just this two stage approach, but it could be an actually fully cycling approach that sort of goes back to the neural network to refine the structure that it's building with the gradient descent procedure. It's entirely possible. So this is the graphic of Alpha Fold 2. You can see at the very beginning, you have protein sequence. And at first you have this embed and outer embed and outer sum, which I'm going to guess this is just kind of features for pairs or individual amino acids. This this is correlation statistics from your data set. It can be chemical properties, whatever. It's just a bunch of features that you can attach to each of these amino acids in the sequence. The other path here is this genetic search and embed. So this is what we've already seen with the MSA. I told you they have the same graphic. So there's human, there's fishy, there's rabbit. And you simply search for sequences in your database. It could even be from other humans that are similar. And from that from those, you can also derive features. So here is where I'm a bit confused. You can see they build up this again, this square matrix right here. I mean, this it already screamed attention before. Right. So I'm going to guess they no longer limit themselves to the maybe maybe to the 64 by 64. Maybe they do something bigger. Maybe they use local attention. Who knows? I'm going to guess they use attention to and these this here is simply given by an attention layer of some sort to go into the next to just this is basically I would guess this is a big transformer right here. The interesting part is that it appears to interact much like much like the original transformer, maybe encoder decoder here. They pass information around. So this top thing isn't amino acid sequence to amino acid sequence like to itself, but it appears to be a matrix that you build up between the amino acid sequence and these sequences you built. So I would guess that they are no longer, let's say happy with simply inputting the features of these algorithms that go over these other sequences. But now they also want to sort of put these features through through steps of transformations. So again, I would guess this is an attention layer. And how can we interpret this matrix? As you can see, this matrix relates individual amino acids in the sequence to other species. So I would guess that this square here represents something like how important is this particular location in the chain, which is a purple thing in the human. How important is that in the in the in the chicken or how related is that to the chicken at that particular position or as a whole? I don't know. Probably DeepMind doesn't know. Like they probably just ship these features in here, right? And then they just ship it through transformers. They pass information around. I don't know whether it's just in this direction and then in this direction or whether there's like an arrow right here conceivably. But in any case, it seems like they've replaced what was a conv net. So no longer friends with ConvNet. New best friend is transformer. And then at the end, you see what they get out is these pairwise distances again. Now, it's also not really clear because I would expect maybe an arrow going like this. If they again use these pairwise distances to predict the structure. I don't know. OK. Or if that's just a side output. I would guess they still actually use the pairwise distances and the confidence score. Again, you can it might be something very similar that we saw again being the sort of standard deviation on the predicted distances. But they could also refine that. And then the last thing is I don't know if this iterative process is simply referring to there being multiple layers of this attention and passing around. So the passing around will simply be like you stack the representations on top of each other. I don't know if this is the iterative procedure or if there is actually like the structure module actually sort of builds the structure and then goes back. And then you consult the neural network again and then you build some more of the structure and so on. I can't tell right now. It's quite conceivable that they they do like that the search here is not only gradient descent, but is actually informed by the neural network. So you sort of go back and refine, though I don't know. There doesn't seem to be any features in the neural networks that would represent that would represent whatever you could read from a partially built 3D model. So, you know, the boring guess is that the part two is very is a lot of the same. But there could also be substantial improvements in that part. All right. I hope this was this was sort of a good overview. So, as I said, the paper isn't out yet. If you want to cite this, I guess you can you can refer to the blog post and here they say until we've published a paper on this work, please cite high accuracy instruction prediction using deep learning by these people. I just want to highlight shout out to to Anna, who was educated right here. She was an intern. So in a way, I'm actually saying that this is my discovery and I take full responsibility for it. You're welcome. World shout out to Anna. Very nice job. Good work. Good work to all of these people. Yeah, I hope that was enough. If I got something horribly wrong, please tell me in the comments and share the video out if you liked it. Other than that, have fun. Bye bye.
 It will change everything. DeepMind solves 50 year old grand challenge. The game has changed.
 DeepMind's latest AI breakthrough achieves historic new milestone, helps solve how diseases invade cells,
 improve protein folding prediction, AI breakthrough it also wipes your butt automatically.
 It is the newest DeepMind big publication. Actually, it's not a publication yet.
 But so what happened and I'm sure you've heard this is that every year there is this competition of protein folding prediction.
 So proteins are the structures that fold in a given way. And we'll go into that in a bit.
 But basically every year there is this competition and the results of this year's competition came out and they looked something like this.
 Namely, every entry here you see is a team participating in that competition of protein folding prediction.
 And there is one team which is DeepMind's system Alpha Fold 2, which completely dominates all the others.
 To the point where the problem is now considered to be solved.
 Now solved in this case simply means that you're past a certain number in this test set.
 And if you're past that certain number, your predictions are useful enough so that other scientists can basically take them and base work on them.
 So that's what it means for this protein folding problem to be solved.
 Now we don't have much information on Alpha Fold 2 yet, other than it's really good.
 And like a blog post and a bunch of advertisement videos by DeepMind, they are writing a paper on it.
 But today I want to go into this blog post and maybe parse out what we can gather from that blog post.
 And I also want to go actually through the Alpha Fold 1 paper.
 So as you can see, the performance here increased drastically with Alpha Fold 2.
 But you know, guesses are high that the system is going to be somewhat similar to Alpha Fold 1, of which we do have a paper.
 So today we'll go into Alpha Fold 1. We'll go into some speculations of Alpha Fold 2.
 I can already give you my speculation. It's transformers.
 It's attention that all of a sudden made this big jump together with probably a few other improvements to the Alpha Fold 1 system.
 Basically, transformers continuing to dominate the entire field.
 So where do we start? It's probably best.
 By the way, if this is not a great meme template, I don't know what is. Just saying. Just saying.
 Yeah. So let's actually start with the problem itself.
 I realize if you're here, you're probably a machine learning person, might not know too much about protein folding.
 So these things here are computer representations of proteins.
 They don't really look that way, but sort of similar.
 A protein essentially is a chain of amino acids.
 So an amino acid, where do we have this? Right here.
 Amino acids are these what they're called basic building blocks of life.
 Since the proteins are what make the cell do things.
 So protein are sort of the workers in the cell.
 They are used as signaling molecules, receptors.
 They are parts of your muscles.
 Actually, the parts that move are proteins.
 So they are all the work doers.
 Whatever something needs to work in a cell to do mechanical or work, proteins are involved.
 And amino acids are the building blocks of proteins.
 So each amino acid has a given certain common structure.
 And there are 21 of them.
 So all the proteins in the world are simply made out of chains of these 21 amino acids.
 And these chains, they are formed.
 And so there's always this sort of body that can link up to other bodies of amino acids.
 It's very similar. If you maybe know how DNA is structured, it's a very similar concept.
 Except in DNA, there are four different bases.
 Here there are 21 amino acids.
 And each amino acid is a little bit different.
 In each amino acid has like a tail that hangs off.
 So the tail can be, you know, look like this, or it can look like this, like with a side chain.
 Or there is there one where it's like maybe a cyclic one. I'm not sure.
 Maybe you can look out here or it can have sort of no tail at all.
 I think that's the case for glycine.
 So the important part is depending on this tail, the properties, the chemical properties of the amino acids are different.
 And then what happens next is really interesting.
 Once this amino acid chain is built in this.
 So this is the central dogma of modern biology is that you have DNA.
 And DNA is translated to RNA.
 Sorry.
 And then it's translated to.
 So it's read off, copied to RNA, which is sort of a DNA clone.
 And then the RNA is translated into the amino acid chain.
 And there is always three, three pieces of DNA mapped to one amino acid.
 This is very much like a compiler.
 Notably, the interesting part is that these steps right here, this compilation steps are done by proteins.
 So there are proteins that do these things.
 So nature in a very real sense is its own compiler.
 So this here you can see as like the binary.
 And this here is like the source code.
 But what happens once you build this chain of amino acid and you set it out into the cell, because of these different properties of these side chains, they're also called residues.
 These chain begins to fold.
 So this is if you know a bit of chemistry, you might know that these are these are sort of atoms that are linked with covalent bonds in this case.
 And it can be that part of this chain is rather like electrically negatively charged.
 And here part of this chain might be like electrically positively charged in a given place over a given other place.
 And it also depends on the surrounding medium, of course.
 And that means that in this case, for example, these two things will attract.
 And so if you release this amino acid chain, what you're going to get is sort of a bend where now the chain sort of bends.
 And these two, this chain right here, this tail goes like here, this tail goes like here.
 I'm sorry, if there is no if there is no if there is no I don't even know what to call it, pyrene rings or something like this.
 If there isn't an amino acid with that, I apologize.
 But the point is that these two things attract and sort of form this shape.
 And this shape is very important.
 We know that proteins and proteins consist of it can be hundreds, thousands, tens of thousands of these amino acids in a chain.
 The proteins function is, interestingly, largely determined by its structure, by its 3D structure, not necessarily by the actual amino acid.
 So technically you can substitute amino acids for each other.
 So this amino acid here can be could be substituted for another amino acid that maybe isn't the same,
 but is has the same properties of its side chain, such that if the structure is still the same, the protein would perform the same function.
 So that that is is very special property of proteins, namely their 3D structure largely determines their function.
 So, for example, in this step here, when you read off the RNA to the DNA, as you know, the RNA is sorry, the DNA is like this double strand of connected base pairs.
 And in order to replicate the DNA or to read it off, there is a there more or let's call it.
 There is also this step of DNA replication, right, where you copy the DNA in mitosis.
 In order to do that, you need to split off the two strands.
 You need to split it up because you want to get like a protein needs to get here to actually read it off.
 For that, there is a protein, a specific protein that will insert right here to split up the DNA, which is called a helicase.
 And that really is very important how that protein is shaped.
 So the shape needs to be actually such that it kind of removes these bonds from each other.
 So the shape is very, very important for a protein.
 And conceivably, you could build a helicase from many, many different amino acid sequences as long as it has the same shape.
 Now, I think something like something like fundamental like a helicase is probably conserved in the evolutionary tree.
 But I hope you get the point.
 The shape is super duper important.
 Now, the shape isn't just arbitrary.
 There are some amino acid chain is called the primary structure.
 And then the first thing that happens is that two very distinct kind of sub shapes appear.
 So often repeating shapes, these things, I think, are called alpha helicase or helix.
 This is a helix.
 And this here is I don't know what's in English.
 It's probably called a strand or something like this.
 These are like long sheets like I think they're called beta strands.
 And these things form.
 These are often repeated sequences.
 And then the third tertiary structure is when the whole thing starts to kind of fold on itself and so on and give itself the the final structure.
 So this is part, I guess, of the RNA polymerase, which is the molecule that reads DNA and outputs RNA.
 And there are many, many, many proteins.
 Now, since the shape is so important, it is vital that we know of it.
 And technically, technically, this is what why this problem is 50 years old, I guess.
 They say it's a 50 year old problem.
 I think that's due to the fact that 50 years ago, a Nobel laureate said the following.
 Since a protein is fully determined by its amino acid chain and since the amino acid chain determines the structure that it's going to do because of these kind of chemical properties,
 it should be possible to read in the amino acid sequence or read in the DNA sequence.
 We know what amino acid sequence results and output the shape of a protein.
 However, this is an extremely complicated problem.
 It turned out to be because they're very subtle interactions.
 They're not always the same. It depends, right?
 Like somewhere out here, there could be some amino acid with like some weird chain that, you know, everything folds on itself all the time.
 So at some point, these get in contact and the changes kind of the local properties here.
 So this is a very, very difficult problem to solve.
 And people have sort of tried to do this and now apparently deep mind the first system that does this to such a satisfaction that it's beneficial.
 All right. Now I lost my train of thought.
 Yeah. So the shape prediction, what happened so far is what you have to do is you'd have to sort of do this, determine this experimentally.
 So you'd have to take these proteins and crystallize them and then like shoot X-rays at them and then infer the structure.
 You can do that from crystallized proteins because I think it's due to crystals or like very regular accumulations of proteins.
 So if you look at a snowflake, that is if we knew nothing about the water molecule that it's like H2O,
 if we knew nothing of that, we could just look at a snowflake and determine this structure, this specific angles here from the snowflake.
 We would just look at the snowflakes and if someone tells us, look, that's all the same material, that's all water,
 we could infer what the water molecule looks like just by analyzing snowflakes because they're crystals.
 And the pretty much the same here is you build, you make crystals out of these materials, you shoot X-rays at them.
 And then you sort of reason over the patterns that come out.
 This is very, very difficult, very expensive.
 And so to solve this problem computationally is super important.
 Now we'll get to this graphic in a minute.
 This is sort of the only thing we know about AlphaFold2 is this graphic right now, because they have not yet released.
 The paper or any descriptions of the model, as I said.
 But what we'll do is we'll go into AlphaFold1.
 So this is AlphaFold1.
 And AlphaFold1 was participating in the same competition two years ago and was already dominant there,
 but not yet dominant to the point of having, quote unquote, solved the problem just better than other systems.
 So this is the basic structure of AlphaFold1.
 So what do you have right here?
 Let's give ourselves an overview.
 So the overview is the following.
 There are two different stages to this algorithm.
 Stage one is over here and stage two is over here.
 Maybe it's easiest to start with stage two.
 So the output of stage one is this thing right here, a distance and torsion distribution prediction.
 So this matrix here that's kind of tilted on its side, I believe there are more down here.
 Right. OK.
 So what you do right here is you take an amino acid sequence and you line it up right here.
 You line it up. This is the amino acid sequence.
 It's a bit harder if there's like a split, but let's just say a protein is...
 Actually, there can't be a split. Sorry, that's in the amino acids. I'm dumb.
 So a protein is a single chain of these amino acids.
 There can be multiple sort of parts to a bigger protein conglomerate.
 But there is this chain. You line it up here and here.
 So now we're building sort of a pairwise matrix between the sequence and itself.
 And this pairwise matrix is going to be a distance matrix.
 So what we are going to do is we're going to input some features about this sequence of amino acids.
 That's what we get as an input.
 And we're going to predict for any pair.
 So we have the sequence and we're going to predict for any pair how far are they apart?
 So of course, here the answer is always kind of zero.
 They're zero apart. But you might say, you know, these two are five apart and these two here are seven apart.
 But these two here are only one apart.
 So it's reasonable, you know, that the final structure, these two are close together.
 We don't worry about close together right now.
 We just worry about for each two, we'll predict how far they are apart.
 OK, so this is you can view this as, you know, a machine learning problem, right?
 You have an input sequence and you simply want to predict the distance matrix.
 So here you can see that. In fact, you can see the top and bottom.
 One is the predicted and one is the real.
 I don't even remember which one's which.
 You can see that this system does a pretty good job at that.
 There are minute differences.
 You really go look like down here, you can see a bit of a difference over here.
 There is a bit of a difference.
 But in general, this system does a pretty good job.
 So this is the output of stage one is this matrix.
 It's a bunch of other it's like also the torsion angles and so on.
 But the main thing is you predict the distances between those two.
 That's what you take as a input to stage two.
 So what stage two does is stage two builds a model of this molecule.
 And the model is sort of a differentiable geometrical model.
 So they say they. Where is it?
 I don't get these nature papers like they're split into two parts,
 but then they are they largely say the same things.
 I am absolutely confused by them.
 So we're going to jump around a fair bit.
 They say we parameterize protein structures by the backbone torsion angles of all residues
 and build a differentiable model of protein geometry to compute the coordinates for all residues.
 And thus the inter residue distances.
 So what they do is essentially they build a computer model of these amino acids.
 And these are parameterized by the torsion angles.
 Now, the torsion angle is simply the angle between any two of them.
 So this would be like a torsion angle of 180 degrees.
 And then if it folds like this, it would be torsion angle of 90 degrees and so on.
 And you need two torsion angles because you're in 3D.
 But essentially the torsion angles determine the structure of the protein.
 So it's one way of parameterizing it.
 So they build a differentiable model, a differentiable model of protein geometry.
 Now, the important thing is they don't do any learning with this differentiable model.
 The purpose of this differentiable model is such that what you can do now,
 if you have a differentiable model, you can run gradient descent.
 So imagine they pretty much lay it out right here.
 So they have the x, x is the output of your differentiable geometry, right, of your torsion angles.
 Let's just call it this Greek letter phi, psi, whatever.
 If x is the output and now x goes into your loss function.
 So x goes into your loss function and the loss function simply compares x.
 To the predicted x.
 So the loss function will take in x and it will compare it to the x that you predicted from this thing here.
 So we start off with a flat chain, maybe.
 Actually, I think we start off with some initialization because they also predict the torsion angles directly.
 Right here, they predict the torsion angles directly.
 And that's what we initialize from. But let's just say we initialize from the flat chain.
 And then because this is differentiable, we do so your L is x minus x prime.
 And what we do is we derive the loss with respect to the angle, to the torsion angle.
 So we can do this since this is differentiable.
 So now we know how do we need to change the angle, which is this thing right here, in order to make the loss smaller.
 And maybe it says, actually, you need to turn it down, right, make the angle smaller.
 And we do that. Okay, cool. Now it's only 90 degrees.
 And then we do it again and again and again.
 And you can see that by changing all the angles such that this loss is smaller, we end up through steps, step, step, step.
 We in our computer model, we sort of replicate this process that happens in nature, where what we feed in is how far any two amino acids should be apart.
 And by running gradient descent, just gradient descent on the torsion angles, we figure out what do the angles need to be in order to make this happen.
 So first, we predict all the distances, and then we figure out how do we need to set the angles such that these distances are fulfilled.
 These are not true distances. These are predicted distances, right?
 So everything depends on how well we can predict these distances.
 But once we have them, we can sort of replicate in our computers the process as it happens in nature, except in nature, the whole folding is dependent on these all these chemical interactions and so on.
 And now we do none of this.
 We simply look see how do we need to fold in order to make these distances in our computer model like these like the distance between this and this and this and this.
 Any two distances may agree with the distances that we have predicted right here.
 And you can see that over time, this as you run gradient descent, this goes up.
 This this TM score was up the root mean square distance goes down between and then you of course can compare it if you have a test set with stuff that people have already figured out, you can analyze these metrics and see that indeed, you do get the correct folding.
 It's also pretty interesting that so here in blue and red, I believe you have.
 Yeah, exactly. So the the helix in blue and the strands in red.
 So in this case, you from if you have this folded structure or partially folded structure, you can already see that these sort of substructures emerge like this is a helix, right?
 As you can see, and then you sort of made this maybe a strand and so on. There are ways to heuristically classify that.
 And you can see that if you look at the database, right, you can see that this here is a strand.
 These are helices, and this is a strand and these are helix.
 This is a strand and so on. And you can see that the model here is what the model thinks at the beginning.
 It doesn't get many things correct, though it does some, but then over time, it sort of refines its guesses until at the end, it's pretty much equal to what the database to what the true sample is.
 And here is simply the distribution of, I guess, confidence about these things and the torsion angles right here.
 So it, as you can see, this two step process is the key here to do that.
 Now, Alpha Fold 2 conceivably probably changes this a little bit.
 But again, we're not sure. The step one right here is a deep learning system.
 So step two is simply a gradient descent procedure that you run at inference time, right?
 This at training, you can you can just do step one.
 So step one is is the machine learning bit.
 So the goal is to output this distance, this distance tensor right here.
 And there are more things than distances, as we said, there are torsion angles and so on.
 But ultimately, you want to output this distance matrix.
 And how do they do it? You can already see it's a deep neural network.
 So you want to build a input data point, let's say, of L by L, which is sequence length by sequence length.
 So you want to collect some features, you don't know the distances yet, right?
 But you can collect some features that are either pairwise features between these two things, right?
 So here, maybe this is, I don't know, leucine, and this is what's a different amino acid glycine.
 And in here, you want to put features, maybe it can be features for that position, right?
 Maybe leucine here is at the 100th position in the in this particular protein, and this is at the 90th position.
 So we want to put in some features of that that you can derive from a data set.
 You can put in correlation statistics in general between these two amino acids.
 You can even put in just single features.
 So you have these tiled L by one features, which is just features for the sequence itself, not pairwise features.
 But what you do is you simply replicate them along along any given dimension right here.
 You always put the same features. This is very common in conv nets.
 And you can even do a scalar feature. So there are some scalar features.
 And what you would do is you would simply fill an entire plane with that scalar feature, all the same number.
 It's just easier to do it like this because it fits into the convolutional architecture.
 Well, so you want to provide all kinds of features and the features they provide are, you know, plentiful.
 And a lot of them do introduce some domain tools, domain expertise and so on.
 But once they have that, they simply take that sort of image with many, many channels and they predict this image if you want.
 So it's just an image to image translation problem. And they do this via a convolutional neural network.
 As you can see, there are 220 residual convolutional blocks.
 Now, I assume that most of the viewers of this video are familiar what convolutional neural networks are.
 If not, I'm deeply sorry, but we'll not go into that.
 But you can see they sort of they tile this tensor right here and they tile it differently from from from instance to instance.
 So they tile it in the training procedure. They always tile it differently.
 That's a form of data augmentation. But ultimately, you slide over this image with this 64 by 64 ConvNet and you produce the image on the right.
 Here you can see an inherent weakness of these approaches, namely that this thing can only ever look at 64 amino acids at a time.
 So now that can that can be the same if you're on the diagonal of this.
 Let's say let's say this is not 64 by 64, but three by three.
 If you're on the diagonal, you would only consider three amino acids and their interactions with each other.
 Right. Any to any interactions with each other.
 If you're off the diagonal, what you would consider is maybe these three amino acids and these three amino acids.
 And you would only consider you consider features for maybe for those three, but interactions only in between like the these not interactions
 actually within the same amino acids.
 So you're the thing that you can look at any point in time is going to be very limited.
 Right. And these so these distances that you get out here, they necessarily cannot directly depend on, let's say, this amino acid right here.
 You always have this limited view of your protein that sort of local.
 Now, people argue that that's actually enough if you look at maybe the green connections right here in order to establish them.
 What's most important is the vicinity of these of this amino acid and the immediate vicinity of this amino acid.
 And, of course, the interaction between those two vicinities.
 But it is quite conceivable that this green thing down here being so close will actually sort of push the two apart
 and sort of do this interaction, which, in my understanding, would not be covered by a system like this.
 And that's where alpha fold two, I believe, is is one point where it makes the big gains that it does.
 Now, the features that go in here, as I said, they are they're quite plentiful.
 One of the more interesting features is this MSA, these multiple sequence alignment.
 And I believe they're they're up right here.
 Yeah, sequences. So here they introduce them in recent years.
 The accuracy of structure predictions has improved through the use of evolutionary covariation data that are found in sets of related sequences.
 Sequences that are similar to the target sequence are found by searching large data sets of protein sequences derived from DNA sequencing
 and aligned to the target sequence to generate a multiple sequence alignment.
 Correlated changes in the positions of two amino acid residues across the sequences of MSA can be used to infer which residues might be in contact.
 So what what this I've searched out one of the papers right here, and this is from a paper called improved contact prediction proteins using pseudo likelihoods to infer POTS models.
 The entire basis here is that here is your chain of amino acid that you're considering.
 And this is you. This is the human.
 And they actually have one like a very similar graphic in their blog post. But we'll draw this ourselves.
 I'll just kind of sort of copy it. And what you do is you go and look into your database.
 Right. This this is the amino acid sequence.
 And each amino acid can actually be abbreviated by a single letter since they're 21.
 And luckily, the holy alphabet creators have given us what 26.
 So that fits. So each of these can be done by like S Y C M D and so on.
 Can be then you go look into your database and your database is of sort of all of life.
 And you go look for similar sequences and there are tools that you can very quickly see through databases and get out similar sequences to yours.
 And those are sequences that are overlapping in amino acid sequence.
 Right. So you could find in the fish.
 This is an alpha. This is not a fish in the fish.
 There is a similar sequence right here in the iron. Like this is OK in the whatever this is.
 This might be a horsey. No, this is not a horse. Let's make an alligator out of this.
 So in the alligator, raw does the alligator have?
 There might be a sequence and so you get the point. My drawing skills are to be criticized in another video.
 So you search for all of these similar sequences just by amino acid sequence and from the correlations, you can derive something.
 For example, I've already told you that sometimes you can substitute an amino acid and the sort of function of the protein isn't really affected.
 And this may be what you can see right here. So in the human, this is maybe a D, but or sorry, maybe this here.
 It's a C, but in the in the let's call this an M in the fish, it's a C2.
 But, you know, in the alligator, it's a P and in the cockroach, it's K and so on.
 You can see that maybe if the alignment is good, right, this is sort of from the same protein or from a protein that does maybe the same thing in these life forms, because life is continuous.
 Often these things are preserved or slightly modified. So here there are variations that happen in life, right?
 Mutations, variations. And so we can safely maybe assume that, you know, a K, whether there's a K or a P or a C in this particular point, it doesn't really matter.
 The shape doesn't seem to be too affected. So that's step one.
 And now so this might be this protein, this amino acid right here, you see, whether it's this chain or whether it's this chain, maybe doesn't really matter for the function of the protein.
 However, if you look at two proteins that are in contact, what needs to happen?
 So if my protein here has this chain and the other protein has sort of is in contact, that means there is like a chemical interaction between the two.
 So now if a mutation happens, if a mutation happens and the protein is still functioning the same way, but the mutation happened, let's say it's now this right here,
 that must mean the shape is still the same sort of. And that must mean that probably if one of them changed, the other one probably changed sort of analogously at the same time because structure is preserved, function is preserved.
 So structure is preserved. And since structure is determined by chemical interactions, one of the parts changed.
 That means probably the other part has changed as well. So maybe now this is sort of this chain right here.
 So what you would expect to see in the statistics is that if one changes, the other one changes accordingly.
 So there can be variations, right? There can be mutations.
 But if the mutation happens in one of them, a corresponding mutation should happen in the other one as well.
 Otherwise, the protein would be non-functional and the organism would sort of die.
 Not always, but you know, this is kind of a statistics game.
 And this is what you see here. Like the fish has an S like the human and an H right here.
 But the alligator has an F and a W right here. And then in the cockroach, you see the S and the H again, and so on.
 And here down here, you see the F and the W again.
 And this is an indication that these the correlation here is an indication that these two things might be in contact with each other.
 Now, there have been systems, for example, in this paper right here, that directly go from these statistics to contact predictions and so on.
 Alpha Fold simply takes in this stuff as features. So this right here, all of this, there can be, I think they derive 488 features from this.
 So this goes down here. I think they say it again.
 As I said, this is confused. Like here, article stops, references, article starts again. Thanks.
 And they like say almost the same things. It's just a little bit more detailed, but it's not longer.
 So here they derive 484 features from these multiple sequence alignment for each residue pair.
 Right. So in our big tensor right here, right here, each dot, each thing right here already now has 400.
 So each one of these already has 484 features and then some more.
 Right. This is already this is from the MSA, but then more features.
 So they incorporate lots of features right here.
 Where are we at? Here. They incorporate lots of features.
 In addition, we provide the network with features that explicitly represent gaps and deletions.
 They also represent scalar features and so on.
 So here you can see they have scalar features, sequence length features, amino acid type, profiles, HH blitz profiles.
 These are all sort of these comp bio tools, these genetic tools and so on.
 You also have sequence length features. These are these 484 features and so on.
 So these are all akin. There are some positional. One of these acts as positional encodings and so on.
 So lots of features, input, convolutional network, output, the distance matrix.
 And that's that. Right. So there you have the inputs, the distance matrix from the distance matrix.
 You can run gradient descent to get the protein structure at inference time.
 And they make some pretty cool points. Not only do they compare the distance matrices, but they here is the not only the single prediction for the distance,
 but they, of course, output a probability distribution. They bin all of these distances. They output a probability distribution.
 And you can see that the black line in these histograms. So this is this is for a particular thing.
 This is for this this red line, this red row right here.
 It's the extraction. So it's for one of the amino acid, the distribution of probabilities of distance bins.
 With each of the other ones. So this is number 29. And we look at the distance between number 29 and one, two, three, and so on.
 The black line represent the represents, I think, eight angstroms, which is generally considered the barrier for being in contact or not being in contact.
 And here it's colored in blue if not in contact and in green if in contact. And the red bar represents the true distance.
 And you can see this is pretty accurate. So whenever the network predicts blue, usually the red line is on the right of the black line.
 And if the network predicts no, sorry, these green and blue is the ground truth.
 So whenever it's blue, the network's distribution is usually shifted towards the right.
 And whenever it's green, the network's distribution is shifted towards the left.
 There are some failure cases, as you can see right here, the network predicts a higher distance than the than the the truth.
 Right. You can also see what's pretty interesting is that the most accurate predictions sort of the highest confidence, the smallest variation in distribution are around here, which is exactly around.
 So 29 would be in the middle right here. And that's where you find the most accurate predictions, of course, since local local distances are much more easier.
 And then as you go farther away, you get less sure. And this is a cool thing. So here you can see model prediction versus true distance fits fairly well.
 But you can also see that here they plot the standard deviation of their prediction.
 And you can see that the the means are very close, but the higher the sort of standard deviation, the less sure the model is.
 So there seems to be a there seems to be like a built in confidence metric. Right.
 So you can see the distance error it makes here are bigger and also its standard deviation is bigger at the same time, which means that you can sort of look at the standard deviation of this distribution right here.
 And that is an estimate for how sure how confident the model is in its prediction. And apparently that's something that in Alpha Fold 2, the the model relies upon very, very crucially.
 So here you these are just on the bottom, you see one of these residual blocks here, more distance matrices.
 They do a lot of analysis in this article, which is pretty cool. So you can go into it fairly far.
 They also have look at what the network pays attention to.
 And it makes a lot of sense like it pays attention to kind of these these helices and then these interactions between the helices and the parts where it's close in close contact with and so on.
 But now we want to go into Alpha Fold 2. Alpha Fold 2.
 Now the what we have isn't much we have this graphic right here, which is also in the article.
 It's probably better we go to the blog post to the blog post is like a fluff piece saying we they are going to publish a paper.
 But of course, they don't have it yet because we've just gotten the results.
 Yeah, they have they have these these cool these videos were like, ah, so good.
 As I said, I like there's so many Twitter threads with. I'm not usually up for the hype, but this is the best thing and so on and everyone's everyone's hyping and I thought, is it really up to me to be the grumpy one here.
 But then I couldn't find anything to be grumpy about. So this is what we what we get.
 Let's see. It's it's deep mind.
 I expect them to not fully maybe release the code. Maybe they will.
 But in Alpha Fold 1, they've released like half the code, which is already pretty cool.
 So there are open source implementations based on that. So again, nothing to be grumpy about.
 All right. So what can we what can we say?
 They say a folded, folded protein can be thought of as a spatial graph.
 And then this is kind of a new word they introduced.
 But ultimately, it's simply this distance matrix that we've seen before is a representation of that spatial graph.
 Right. It's simply a graph of nodes and the edges say whether or not they're in contact or respectively how far they are apart,
 where the residues are nodes and edges connect the residues in close proximity.
 This graph is important for understanding the physical interactions within proteins as well as their evolutionary history.
 For the latest version of Alpha Fold used at CAS 14, that's this challenge.
 We created an attention based neural network system trained end to end that attempts to interpret the structure of this graph while reasoning over the implicit graph that it's building.
 I look this it's sound like this.
 This is fluff. Maybe. I don't know.
 But this here attention based. OK.
 So I'm going to guess for sure that they've replaced this convent with and with a transformer style with an attention attention layer or multiple attention layers.
 They say it uses evolutionary evolutionarily related sequences,
 multiple sequence alignment and the representation of amino acid residue pairs to refine this graph.
 This is this is what we've already seen.
 So use these other sequences plus like a lot of stats that you can gather from the data sets on amino acid pairs in order to develop this this graph.
 And the graph is distance, the distance matrix or other things we'll see in just a second.
 They say by iterating this process, the system develops strong predictions of the underlying physical structure of the protein and is able to determine highly accurate structures in a matter of days.
 Additionally, Alpha Fold can predict which parts of each predicted protein structure are reliable using an internal confidence measure.
 Again, this is something that we've already sort of seen in Alpha Fold 1 that there is sort of an internal confidence measure.
 And the part here is they say by iterating this process, which could mean that it's no longer just this two stage approach,
 but it could be an actually fully cycling approach that sort of goes back to the neural network to refine the structure that it's building with the gradient descent procedure.
 It's entirely possible. So this is the graphic of Alpha Fold 2.
 You can see at the very beginning, you have protein sequence.
 And at first you have this embed and outer embed and outer sum, which I'm going to guess this is just kind of features for pairs or individual amino acids.
 This this is correlation statistics from your data set.
 It can be chemical properties, whatever.
 It's just a bunch of features that you can attach to each of these amino acids in the sequence.
 The other path here is this genetic search and embed.
 So this is what we've already seen with the MSA.
 I told you they have the same graphic.
 So there's human, there's fishy, there's rabbit.
 And you simply search for sequences in your database.
 It could even be from other humans that are similar.
 And from that from those, you can also derive features.
 So here is where I'm a bit confused.
 You can see they build up this again, this square matrix right here.
 I mean, this it already screamed attention before.
 Right. So I'm going to guess they no longer limit themselves to the maybe maybe to the 64 by 64.
 Maybe they do something bigger.
 Maybe they use local attention. Who knows?
 I'm going to guess they use attention to and these this here is simply given by an attention layer of some sort to go into the next to just this is basically I would guess this is a big transformer right here.
 The interesting part is that it appears to interact much like much like the original transformer, maybe encoder decoder here.
 They pass information around.
 So this top thing isn't amino acid sequence to amino acid sequence like to itself, but it appears to be a matrix that you build up between the amino acid sequence and these sequences you built.
 So I would guess that they are no longer, let's say happy with simply inputting the features of these algorithms that go over these other sequences.
 But now they also want to sort of put these features through through steps of transformations.
 So again, I would guess this is an attention layer.
 And how can we interpret this matrix?
 As you can see, this matrix relates individual amino acids in the sequence to other species.
 So I would guess that this square here represents something like how important is this particular location in the chain, which is a purple thing in the human.
 How important is that in the in the in the chicken or how related is that to the chicken at that particular position or as a whole?
 I don't know. Probably DeepMind doesn't know.
 Like they probably just ship these features in here, right?
 And then they just ship it through transformers.
 They pass information around.
 I don't know whether it's just in this direction and then in this direction or whether there's like an arrow right here conceivably.
 But in any case, it seems like they've replaced what was a conv net.
 So no longer friends with ConvNet.
 New best friend is transformer.
 And then at the end, you see what they get out is these pairwise distances again.
 Now, it's also not really clear because I would expect maybe an arrow going like this.
 If they again use these pairwise distances to predict the structure.
 I don't know.
 OK.
 Or if that's just a side output.
 I would guess they still actually use the pairwise distances and the confidence score.
 Again, you can it might be something very similar that we saw again being the sort of standard deviation on the predicted distances.
 But they could also refine that.
 And then the last thing is I don't know if this iterative process is simply referring to there being multiple layers of this attention and passing around.
 So the passing around will simply be like you stack the representations on top of each other.
 I don't know if this is the iterative procedure or if there is actually like the structure module actually sort of builds the structure and then goes back.
 And then you consult the neural network again and then you build some more of the structure and so on.
 I can't tell right now. It's quite conceivable that they they do like that the search here is not only gradient descent, but is actually informed by the neural network.
 So you sort of go back and refine, though I don't know.
 There doesn't seem to be any features in the neural networks that would represent that would represent whatever you could read from a partially built 3D model.
 So, you know, the boring guess is that the part two is very is a lot of the same.
 But there could also be substantial improvements in that part.
 All right. I hope this was this was sort of a good overview.
 So, as I said, the paper isn't out yet.
 If you want to cite this, I guess you can you can refer to the blog post and here they say until we've published a paper on this work, please cite high accuracy instruction prediction using deep learning by these people.
 I just want to highlight shout out to to Anna, who was educated right here.
 She was an intern. So in a way, I'm actually saying that this is my discovery and I take full responsibility for it.
 You're welcome. World shout out to Anna.
 Very nice job. Good work. Good work to all of these people.
 Yeah, I hope that was enough.
 If I got something horribly wrong, please tell me in the comments and share the video out if you liked it.
 Other than that, have fun. Bye bye.