JEPA - A Path Towards Autonomous Machine Intelligence (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper jepa h-jepa yann lecun lecun agi artificial general intelligence openreview
#jepa #ai #machinelearning 

Yann LeCun's position paper on a path towards machine intelligence combines Self-Supervised Learning, Energy-Based Models, and hierarchical predictive embedding models to arrive at a system that can teach itself to learn useful abstractions at multiple levels and use that as a world model to plan ahead in time.

OUTLINE:
0:00 - Introduction
2:00 - Main Contributions
5:45 - Mode 1 and Mode 2 actors
15:40 - Self-Supervised Learning and Energy-Based Models
20:15 - Introducing latent variables
25:00 - The problem of collapse
29:50 - Contrastive vs regularized methods
36:00 - The JEPA architecture
47:00 - Hierarchical JEPA (H-JEPA)
53:00 - Broader relevance
56:00 - Summary & Comments

Paper: https://openreview.net/forum?id=BZ5a1r-kVsf

Abstract: How could machines learn as efficiently as humans and animals?  How could machines learn to reason and plan?  How could machines learn representations of percepts and action plans at multiple levels of abstraction, enabling them to reason, predict, and plan at multiple time horizons?  This position paper proposes an architecture and training paradigms with which to construct autonomous intelligent agents. It combines concepts such as configurable predictive world model, behavior driven through intrinsic motivation, and hierarchical joint embedding architectures trained with self-supervised learning.

Author: Yann LeCun

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello there, today we're looking at a path towards autonomous machine intelligence by Jan LeCun, also called the JEPA paper. Actually, I think only I call it the JEPA paper. But JEPA is a new architecture that Jan LeCun proposes as a part of this paper and we're gonna go into it as he himself describes it as the corner piece of this method. So you will learn what one of the Godfathers and Touring Award winners thinks of how we should reach machine intelligence or at least one proposal of it. The abstract reads how could machines learn as efficiently as humans and animals? How could machines learn to reason and plan? How could machines learn representations of percepts and action plans at multiple levels of abstraction enabling them to reason, predict and plan at multiple time horizons? These things are largely all open problems in current deep learning. Efficient learning especially. Deep learning is notoriously data-hungry. Reasoning and planning is something that a lot of these things can't do at least according to some people. And certainly reasoning, predicting, planning at multiple time horizons. These kind of things including abstraction. All of these things are still sort of out of the realm of current deep learning. So here is Jan LeCun's position paper as he calls it of how to reach these things. So he also says the text is written with as little jargon as possible and using as little mathematical prior knowledge as possible so as to appeal to readers with a wide variety of backgrounds. Now I don't want to actually go through the whole paper because the whole paper is what 69 pages long or so but I'll present to you sort of the core piece which is the JEPA architecture and just a little bit around that so you know what's going on. And I think it's pretty cool. Here he states the main contributions of the paper are the following. First an overall cognitive architecture in which all modules are differentiable and many of them are trainable. This is going to be one of the more wishy-washy hand wavy pieces of the paper. We'll quickly look at it. Then JEPA and hierarchical JEPA, a non generative architecture for predictive world models that learn a hierarchy of representations. So there should immediately you should see that you have a non generative architecture but for predictive world models which is going to be interesting. How can you be non generative yet still predict stuff? We're going to see that in fact the predictions happen in the latent space kind of like mu zero if you will. Third a non-contrastive self supervised learning paradigm that produces representations that are simultaneously informative and predictable. And the key thing here is going to be this non-contrastive part. Lacan makes a big deal out of pitching essentially pitting contrastive and non-contrastive methods and arguing why non-contrastive methods should be preferred above contrastive methods mostly due to the curse of dimensionality. Lastly a way to use H-JEPA at the basis of predictive world models for hierarchical planning under uncertainty. So the H here is going to be for the hierarchical extension or the hierarchical arrangement of the JEPA architecture. He says impatient readers may prefer to jump directly to the aforementioned sections will do exactly that. So there is a bit about world models and why it's important and here is kind of the entire proposed architecture. Now as I said this is a little bit hand wavy so there is essentially a world model which is you know pretty important and that's going to be the centerpiece right here that predicts the state of the world forward in time. So this is the actual world and the world model is trying to predict that. It's going to interact with this actor module right here. Obviously the actor is going to be what actually does the action however the actor could also act inside of the world model in sort of a simulated reality and plan forward what would happen if I were to do something or it could interact with the world model to find the best action to do and that's exactly what we're going to see. The short-term memory here is going to be used to train that world model and also to train that critic so essentially the things that happen in the world are going to be stored into the short-term memory and then the critic can be updated from that but will not look into that very well very much. Perception module right here is a module that takes the whatever the world gives and makes it available as a representation or as a perception. This is going to be the let's say the entry point to the systems that we have and this is very much the closest that we have to something that's actually working which is obviously our current deep learning systems they're very good at perception. So there is one thing I've left out which is this configurator right here. The configurator is sort of the master module that configures all the other modules depending on what situation they're in and so on and this is is definitely like there's a lot of hand-waving right here is like yeah yeah we can just have like a top-down configurator that configures stuff and I don't want to I don't want to go too much into it because there's not too much to go into but also it's not the core of the paper. We're going to go what we're going to go into the world model here specifically. So first of all he describes a two different ways of let's say acting in the world and here we are for the first time introduced to kind of like the notation of this paper which is very much in diagrams. So this is what he calls a mode one perception action episode. This goes very much with like Kahneman I believe it was Kahneman like mode one and mode two reasoning or thinking. So mode one is sort of reactive you simply go from perception of the world to action without much thought. It's kind of subconscious and this is encapsulated here. So we start with the world we get like some sort of so sort of observation we put this through the encoder right here that's going to give us a latent representation. This encoder is that that perception perception module that we saw before. Now different things happen but only actually one path is critical namely this goes to the actor right here this is the actor and the actor sends back an action to the world. As you can see this is a straightforward signal routing to the actor and back oh it even says actor right here. It says even this reactive process does not make use of the world model nor the cost. So there is a cost module that we saw which tells sort of how much something is whether it's good or bad this can be intrinsic motivation this can be external reward anything like this we can compute it however in this very basic loop the actor has been trained already to just act on a percept. At inference time the actor doesn't need to look at the cost anymore in order to act. This is what we're very used to from current like model free reinforcement learning algorithms they simply train the actor using the reward but then once it's inference time they simply let the actor act and rely on that training. This is a mode one perception action episode. In contrast to that we are introduced to the mode two perception action episode. This is a little bit more involved you can see here that we are rolling out the world model forward in order to do something and what do we do again we have an input here we go through the encoder this is probably a wrong color as it's the same we go through the encoder however now we are going to roll out the world model across different time steps and how are we going to roll out the world model we're going to use the actor right here so the actor is going to take that state that gets from the encoder and propose an action this is the same actor as before it's just sort of a trained thing that's proposing some action okay good enough we can use that into the world model together with the latent prediction you realize right here the predictor here this thing it takes whatever comes out of the encoder right here that means it takes a latent state of the world and it predicts the next latent state of the world that's why he calls this non generative these these world models and and these encoders they all go to latent space and then they predict stuff in latent space so in fact it doesn't predict the world it predicts the latent state of the world which enables it to focus on what's truly important for the task obviously modulo how well you can train this thing to actually do that and how you can prevent it from collapse we'll get to all of that however you'll notice that now we can give the actor the representation it proposes an action we can actually use the world model to predict the next state from that next state we can ask the actor for an action the actor gives us an action and we can predict the next state now what does that give us in fact that gives us quite a bit let's let's assume let's just assume that episodes are always the same length and forget about this forget about this forget about this episodes are always the same length this length right here and you won't get any reward or anything or any intrinsic reward until the very end like until the very end there's kind of like a reward or a cost or something like this well we can compute it which is fine we could already do that before it's informative but we didn't do anything with it however once we have that whole loop done if all of these things are differentiable what we can do is we can say well this action sequence right here right now would give us like a reward of five okay can we make that bigger well since everything's differentiable I can certainly use back propagation and gradient descent to ask how would this action need to change in order to make this thing go higher right maybe I need to switch to a different action now it's six well can I also change that action to make it go higher oh well I can now it's seven and so on so I can modify I can optimize all of these actions at inference time using gradient descent right this is if this is not familiar to you it's kind of the same as if you construct an adversarial example to an image classifier that's also gradient descent at inference time so here gradient descent isn't used to train any of these modules we assume that training is done gradient descent is used in order to improve this initial action sequence to a more optimal set of actions and we do that you know we improve these actions here we're using gradient descent through all these modules until we have completely optimized the action sequence and which means that this very first action is probably a very good action like hopefully a better action than was first proposed by the naive actor and then we can take that action and feed it to the world as an action so this is mode to perception action episode this is kind of the model thinking about the future and figuring out through forward-looking what do I need to do what do I need to change to improve the outcome how can I how can I make stuff better and that necessarily uses this world model right and obviously this is just more general if you include all of these costs which you can have after every step you can include some kind of discount factors and yada yada yada yeah so inference time optimization isn't new but it is sort of how the car sees a way one way of how to make these things plan forward so the text says through an optimization or search procedure the actor infers a sequence of actions that minimizes the total energy so these things are called energy and note that it doesn't necessarily need to be optimization it could also be search it could be evolutionary search it could be tree search anything that actually tries to improve the action sequence at inference time an instance of classical model predictive control this is an instance of classical model predictive control with receding horizon planning all right and this here is how we would train such a thing so not such a thing sorry let's assume that we have the two modes we have this naive actor and we use the naive actor to propose sequences for the longer like for for this thing right we propose that first sequence using the new fact naive actor in mode one mode two language there is such a thing as if you do something often and you do it consciously at some point it becomes subconscious right like muscle memory or something like this well how could this work this is how this could work in this framework so you'd have essentially these actions right here are the ones that we have come up through this whole planning process through this whole optimization process well what you can do is you can simply ask the actor or take that output from the initial actor and then you can try to make these things as close as possible right you have all the things right here everything's differentiable so you can train the actor to essentially match those better actions because you know the actor would propose one action however this other action you found to be superior using your world model now obviously that requires you to have a good world model but if you have that then you can improve this low-level actor and at some point that initial action sequence that it proposes will already be close to optimal it's kind of an approximation that you distill into this actor so this is first introduction to the system right here we're going to look a little bit more into how these systems should actually work and here starts a discussion of two things the first one is self supervised learning and the second one is energy-based models the first one is sort of a training paradigm of how to train models using unsupervised data the second one is I want to say a way of thinking about these models it's a formulation of a system and we'll get to it and they are connected so self supervised learning Lacan sees this in the following terms I have a piece of data which is this whole block right here and I try to predict I try to like mask out the piece which is this right hand side right here like I pretend I don't know it and then I use the thing I do know and I try to predict the thing I don't know it's not exactly that however in fact what I want to do is I don't want to predict the thing I don't know I want to create this thing called an energy function an energy function tells me how well these two things fit together and this is going to become clearer in just a second but the way it's formulated right here is that to capture the dependencies between the observed parts of the input and possibly unobserved parts of the input so this is supposed to well it's gonna as I said it's gonna get clearer in just one second but what you want to do is you want to train a system that sees the data space in this format right here which is going to be so-called energy landscape so if you have imagine this is a video sequence right here so there is a bunch of frames and a bunch of frames and frames frames frames frames frames right here so if you have this energy landscape right here you're trying to relate first like the start of a video sequence to the end of a video sequence you can imagine this in a very high dimensional space essentially where all the frames here are concatenated to to a big vector and all the frames here as well and the energy function or the system that you train should assign a very low energy to all of the video sequences that are let's say realistic or in other words here is the X whenever X is this video sequence then and Y is this video sequence then the energy function should assign a low energy to that if the two could actually follow each one another so if Y could follow X if Y would be a logical continuation of X in video space the energy function should assign a low value to that this formulation is very cool because it means if we don't need to predict Y from X directly because there could be multiple video sequences right following that same beginning and that means if we were to just predict Y then we would probably train the system I mean we can still do it but we can probably we will probably train the system to say no there is one correct continuation however if we train the energy function the energy function could assign a low value to any possible continuation as long as it assigns a high value everywhere else we're good so we're trying to produce systems that behave like this now I for I used to think energy function and training loss are the same thing but I know that young Lacan is very adamant about the thing that an energy function is sometime something that you minimize at inference time while the training loss is something that you minimize at training time sometimes they are very similar and overlapping for example a lot of times the the energy function and the training loss are the same formula and by training the system you actually immediately cause it to minimize that energy at inference time simply by forward passing in the model however we can do more with energy functions which we're going to see right now now we introduce latent variable energy based models this is the same formulation as before we have an X and a Y and we have an energy function that tells us how well those two are compatible with each other which is going to be this thing right here however as we've seen there could be many Y that are possible for a given X right so just by seeing X we can't tell you know which of the wise is compatible and that's why we introduce a latent variable Z so this Z right here is going to capture all the information about Y that isn't directly in X for example if we have a video of some some car right the car ah no obviously we have the tracks and they split right here and they go right here and there's a bunch of people and there is a person so the trolley car problem if we have the trolley car problem and it goes down this is the video sequence is up to here right and we don't know how the lever is this is hidden from us there are two possible continuations one here one here the we can't tell just from X X is here and Y is the continuation so the variable Z we introduce it to capture that information in this case the variable Z is either left or right it's binary variable and in order if we have an X and we have a Y in order to compute that energy that tells us how well the two are compatible we need to minimize over Z so what we need to do is if we have a particular Y let's say we actually have the Y where the card goes here right so it goes on the lower track we ask how well do these two video sequences follow from one another well the answer is they follow very well from one another because certainly the card going here is one possible continuation and that means that we had to search over all the possible futures which means we had to minimize over Z so we considered Z going up or Z being down and we determined the Z being down leads to the lower energy and that is in fact a very low energy now what happens if we actually input a video sequence that isn't that isn't let's say we input a video sequence instead of this so the cart is here it goes here and then the next video sequence is of I don't know like a Teletubby so there's a Teletubby it's a sequence like it's an episode from Teletubbies so these two things don't follow from one another and again we do the same thing we minimize over Z but no matter whether we think the lever is up or down as the minecart approaches it never it's never a good continuation that there is that followed the next frames are an episode of Teletubbies so that's how you think about latent variable energy based models is that there's a hidden variable the hidden variable captures everything that is sort of not captured in X about Y and we minimize over that latent variable to get the actual energy which means we're looking for the the value of the latent variable that is most that makes X and Y most compatible and yeah so this is also going to be quite powerful which means that if we already know that X and Y are compatible with one another then minimizing over Z if we have a good energy function minimizing over Z could actually tell us something about the latent structure of the world so we could infer Z or if we have this model trained then if we have an X we could actually sample some Z values in order to maybe produce different future or different possibilities of Y this gives us a lot of freedom to handle uncertainty in the world or simply unobserved structure in the world now there is a problem with these types of architecture and that is going to be collapse if you've noticed that we simply introduced this variable Z right here and we said well it contains everything that's not contained in X but there is actually no restriction for that the if we train this model just with let's say gradient descent and some loss and will make all of these variables unrestricted then very quickly the like the model will become basically useless because let's say our loss function is how well we can predict from X and Z how well we can predict Y right that's the general form now we minimize over we minimize over the values of Z which means that if we simply set Z equals to Y we can always perfectly predict Y and that means X just becomes completely useless and the prediction function just becomes the identity function this is known as collapse and we don't want it what we want to do is restrict Z for example so that like here it can only take two particular values while X and Y are sequences of video frames so that that doesn't happen or we can do it with some architectures so let's look at different configurations right here of these energy-based models in any case D here is the D is the energy or the compatibility function what if we have a deterministic encoder that gives us the latent representation of X and then we use a predictor module in order to predict Y so we'll just predict Y directly and then compare it with the true Y and then we have a loss in between them this cannot collapse because well we need to predict the actual Y now let's introduce one of these latent variables and we're in exactly the situation that I just described again we compute the representation for X but we'll introduce this Z that can vary over a certain domain which gives us a very a domain that we can control for the output of this predictor right here if we now try to predict Y from Z and X we can as I said just set Z to Y and we'd always be good so this can collapse what about this thing right here the auto encoder this seems oh this is just the same as the first architecture this is the same as the first architecture except just Y goes in so instead of X and Y we just have Y goes through an encoder gets a latent representation goes through a decoder that gives you back the gives you back an estimation of oneself and as you know an auto encoder if you don't restrict it somehow in the middle here then it can just become the identity function again and be useless and the last one is this joint embedding architecture now this is looks or sounds an awful lot like the thing that the paper is describing and as you can see it can in fact collapse so we're going to have an encoder for X and an encoder for Y these could be the same but don't have to be they're going to give us two latent representations but or then we use an energy function to compute how well these two latent representations fit together maybe with the help of a latent variable now if the encoders right here simply always output the a constant vector and this one does too and the constant vector is in fact the same constant vector then we're always good right we always output the same vector and this cost function up here we always say yeah they're completely equal this is completely cool they match together super well so this can definitely collapse and we need to do something against it this is a the main discussion here that leads us into contrastive versus restrictive or regularized architectures and this is going to lead us to the gear architecture now it's going to be JEPA but we're building it up slowly so how do we design the loss to prevent collapse now remember where we are we started with self super with we started with recognizing that self supervised learning is probably a good thing because we can do it without labels right we can handle multiple domains with this all we need to do is we need to pretend to not know some part of the input and use the other part to predict something about that unknown part we then said okay we want to formulate this as an energy based model where we'll obtain a model that assigns a low energy to all the compatible pairs of inputs and a high energy to all the incompatible pairs of inputs and that means at inference time we can do a lot of things for example minimize that energy in order to find pairs that go really well together or if we have a pair we can we can look at the energy and judge how well that fits for example you could interpret something like clip as an simple energy based model that simply computes at inference time that energy and if you view these VQGAN plus clip optimization procedures that were really cool before dully was or mini dully was open-sourced then this is exactly minimizing an energy at inference time so just so you can imagine something below it we then introduced latent variables into the mix saying well for a given beginning of a video for example there's going to be multiple continuations and this could be captured in a latent variable this could also be for a given left side of the picture there can be multiple right hand sides and so on this can be captured in latent variables and to compute the energy we need to minimize we then discovered that this is probably prone to a thing called collapse among other things other like other aspects of this architecture are also prone to collapse and now we need to do something against it there are two ways of doing something against it there is contrastive training or regularization now contrastive training you might be aware of that so on the left hand side you have the situation of like a half trained system so this half trained system already has some training examples that have a relatively low energy but there are still some that have a high energy so training means that at the end we want to end up with a model that assigns a low energy to certainly all the training examples and some space around it so we want the energy at the low energy region to extend to these training examples and maybe cut out a bit from that middle right here push the energy up a little bit to say well actually these samples in that space are not compatible with one another so contrastive methods are very very classic methods I don't actually know if clip is trained as a contrastive method but many many sort of of these image image or self supervised image training procedures are certainly contrastive what they'll do is they'll have an image they are going to make two variations of that image maybe by random cropping and data augmentation and so on then they'll take another image like a third image from the database and get they're going to make also a variation of that and then they use the embedding models to embed all of those already so embed embed embed this into latent space so this here would be your standard ResNet encoder or something like this this is usually used in image pre training right and no no no so this will give you a data point somewhere in high dimensional space and then what you do is you try to pull the two that are from the same image together and you push the ones that are from different images apart this is contrastive training and it relies on you coming up with these negative samples so what you want to do is you want to create these contrastive samples that you just kind of jiggle the data points around a bit that you have in with using either augmentations or just some sort of distortions and so on now what we've done right here is we've chosen random negatives but we could also actually mine hard negatives that are very close to the training data however this quickly runs into problems as you know there's the curse of dimensionality if you will have a data point and you want to wiggle it into different directions those directions increase exponentially as you go up in dimensions so this whole approach of finding training examples or finding negative examples around a training example to do the contrastive training is getting less and less tenable in the higher you go with the dimensions and therefore Yandaka advertises for something different which calls regularized methods now regularized methods have other means of restricting that space that is low a low energy region so there is no there are no constructed data points outside here that you know make the energy high here and low here but there is a natural tendency of the system like obviously you enforce you enforce the system you encourage the system to keep the region where the energy is low very small and this is done through regularization and we'll see how this is done in this joint embedding predictive architecture so this is the basic module we've already seen it this was the thing before that was no almost almost so this is almost the same as before but again we have our X and our Y two points that we want to check if they're compatible with one another will embed both of them using deterministic encoders this gives us latent representations of X and Y so X could be the last state of the world Y could be the next state of the world so we map these to the latent representations then we'll use this predictor right here to predict the latent representation of Y from the latent representation of X okay this is the an important part here that differentiates us from before before we try to predict Y directly now we try to predict the latent representation of Y from X we're going to make use of a latent variable right here I guess this is optional but it's built into this model right here so this controls which Y or which latent representation we're getting so Z can vary over this domain right here which then leads the S of Y this thing here to vary over this squiggly domain right here so this probably means that Z could vary over a relatively simple domain but through the power of neural networks this is going to be transformed into some complicated manifold like as I said does the current car turn left or right gives rise to an entirely different series of video frames and this is then going into the energy function whether or not the representation of Y is compatible with the predicted representation of Y now since we are actually trying to predict the representation this energy function right here is probably very simple like something like a cosine distance or an L2 distance or something like this that actually makes the representations equal energies can be much more complicated but yeah so here it repeats the main advantage of JEPA is that it performs predictions in representation space eschewing the need to predict every detail of Y and enabling an elimination of irrelevant details by the encoders obviously that's also a thing that's going to be subject to collapse so he says you know these encoders they could just throw away everything that's not relevant about X and Y because we never need to predict Y directly from something in here right we don't do that so we can just forget about stuff that is not important now how why aren't we forgetting about all the stuff and here is where this regularization comes in so how to train a model like this well the first of all we obviously train it by minimizing this predictive error right here this is the basis right we actually want to predict the latent representation of Y from this thing or sorry from the latent representation of X right we want to predict this thing we actually need to compute the loss between these two things that's exactly this D function right here this is the core right this is unchanged from before however we have a couple of regularizers here to prevent collapse first of all we regularize Z this thing right here what do we do we minimize the information content of Z and that means as before we said well if we let Z just be anything that we want and given that we minimize over Z at inference time this Z can just become equal to Y and make D be zero all the time so this is not good so we need to minimize we need to regularize Z before I said Z could just capture the state of the lever left or right right then you know that there is so much more information in the latent representation of the future video frames that Z cannot possibly even if we minimize over this binary variable cannot possibly capture all of that so restricting the domain of Z is certainly a way to regularize it we can also I guess classically regularize it with some L2 regularization we could quantize it we could apply sparsity regularization anything like this that limits the Z this latent variable that we minimize over is needed right here to prevent collapse the other things that are needed are the things that you see right here so these are regularizers on the information content of the latent representation so what we want to do is we maximize the information content that the latent representation of the encoded signal of the encoder perception has about that about that variable itself well I guess it doesn't need to be actually about that variable it simply needs it simply means we need to maximize the information content of that variable how are we going to achieve that there are also various ways of maximizing the information content essentially it just means that if that variable always has the same value it doesn't have much information inside of it so what we can do for example we can use a mini batch approach and have many X right here X X 1 X 2 X 3 X 4 right and these if these are all independent we encode all of them we get a mini batch of latent representations and we can do something like we say well all of these need to be different right and they're for example their covariance matrices must be identity or something like this so there are various ways and a lot of Yandere also points to some papers for example Vic reg and Barlow twins that have already or can be framed in ways like this but this is a general framework minimize the information content of the latent variable and maximize the information content of the encoded signals which makes sure that there isn't a collapse this directly counteracts that down here I believe yeah exactly we have Vic reg as a system so direct implementations of this you can see right here the L2 loss between the representations the regularization here I don't exactly know how that's regularized doesn't say here but then the maximizing of the information content here is or here of this thing is done via via regularizing the covariance matrix right here so yeah at the last thing that he says here is that we could also bias Jepa to learn useful representations saying it would be useful to have a way to bias the system towards representations that contain information relevant to a class of tasks this can be done by adding prediction heads that take the latent representation as an input and are trained to predict variables that are easily derived from the data and known to be relevant to the task so now we're essentially going into the domain of I don't know natural language pre training with with something like t5 or t0 where you just kind of throw tasks at the system and hope and jointly train all the tasks and hope that you know it learns latent representations that are kind of useful for language tasks Lacan says you could also in addition to doing all of this you could also attach some kind of a prediction head right here and then have another loss from a supervised signal or maybe a imitation learning in reinforcement learning or something like this all of this is entirely possible because without it without having these heads right you now have a system that just sort of does an information trade-off right it just kind of trades off these these different regularizers right here and tries to get like as much information transmitted through this path here about the latent representation of why like it tries to it tries to counteract all of these regularizers it tries to minimize the information right here because then it can do a better job it tries to maximize the information content here as much as it can you counteracted via regularization so you're just kind of playing this information game with the variables right here and it is up I would say to the designers of the system to set the parameters on all of these different loss terms correctly such that the latent representations are useful and I also think a big big big part here is on the data itself like the entirety of usefulness without prediction heads of the system is just down to the data right if you have data if you want to learn something about let's say different chess positions like you want to pre train a chess computer with this thing right you better input data that has different chess positions that differentiate themselves in the relevant aspects of chess positions and it's probably not a good idea that you always have the same chess position but you vary the sort of the shades of gray in the chessboard right so this thing will sort of learn what is predictable from the data that it gets so you better make sure that that data the variation in that data captures what you need to get out of it right so what can we do with this we can arrange it in a hierarchical fashion so this is going to lead us to hierarchical JEPA which is going to be the final the super sane form right here of the model in fact if you think about this going back to the very beginning where we asked ourselves how could we use a fully differentiable system to plan ahead in time well if you consider this to be you know your state of the world for example or frames in a video or something like this you could arrange this system like we did are doing here to predict over multiple time steps right yeah as as we do right here so the lower level predicts over short time frames while the higher level you can see over here that this latent representation is in fact obtained from the latent representation of the lower level by a second encoder and then makes predictions over a longer period of time so the hierarchical arrangement of these things is entirely possible and we can use that to do hierarchical planning so this goes back to the very beginning we at the beginning we saw how can we do mode to planning if we have such a world model right and now we're going to do this in a hierarchical fashion so what do we do again say this is the state of the world and we know at some point we have a desired outcome like a cost function or a reward or something like this well if we have trained such a multi-layer predictive model in latent space what we can do is we can do what we did at the beginning at this higher level right here so we're just gonna do this thing up here first which means that we're going to ask this high level actor and we'll get to what high level actions are but assume there are high level actions for example let's say I need to get to the airport right the high level actions are simply you know I'm gonna go out of the house I'm gonna get in the car I'm gonna drive to the airport and I'm gonna park the car there those are high level actions and low level actions would be the actual you know movements you do so we can ask this high level actor to give us high level actions we can roll out the world model with it until we are here we can use back propagation or search or some other optimization technique in order to refine these actions as well as we can right and then we have here targets for these low level actions now before these things on the lower level were themselves kind of rewards that we get from from the world but this is now up here and the rewards on the lower level are simply how well we match those targets that are given by the higher level so this this action this high level action right here could be get in the car right so now get in the car becomes the target and we can use our lower level planning algorithm in order to determine the best actions again using proposals back propagation optimization and so on to get in the car in fact we can do it for all of these to match all these higher level actions which gives us entire action sequence that would optimally fulfill the plan to to match these higher level actions and you know if we're super duper engaged we could also optimize all of the different levels together until we have the optimal sequence of lower level and higher level actions in order to reach this goal right here at that point we can be relatively sure that this first action right here will serve us just well and we can actually send that to the world get the next state and do it all over again we can even use the short-term memory or something like this in order to start at a better place for next time already although the short-term memory here is used to store states in order to train the train the loss modules and the critics this is if you are actually in an uncertain environment you could even introduce these latent variables right here which you can infer so if you want to reach a certain goal right here you can infer the latent variables also through some sort of optimization procedure or you can sample the latent variables in order to give you different continuations of your world model up to you and there are various possibilities that open up with these with probabilistic world models but I don't want to go too much into this I think I hope you get the concept by now of how to think about these things again this we are again in the space where we have the models trained and we need to do inference time inference time decision of what action to take right training this thing is a different game training this thing is done via this method oh sorry this general method by regularizing by minimizing the prediction error in the latent space okay I think that was it for the paper the rest is about the rest of the architecture designing and training the actor data streams designing the configurator yeah this it gets a bit hand-wavy at that point I mainly wanted to bring the mainly wanted to bring the the JEPA architecture to you and you hope you understand that yeah so there's a bit of broader relevance of the proposed approach could this architecture be the basis of basis of a model of on animal intelligence now it's the answer is maybe but I found this paragraph here pretty pretty astounding the presence of a cost module that drives the behavior of the agent by searching for optimal actions suggest that autonomous intelligent agents of the type proposed here will inevitably possess the equivalent of emotions but that's escalated quickly in an analogous way to animal and humans machine in emotions will be the product of an intrinsic cost or the anticipation of outcomes from a trainable critic cool could this be a path towards machine common sense to which he says I speculate the common sense may emerge from learning world models that capture the self-consistency and mutual dependencies of observations in the world allowing an agent to fill in missing information and detect violations of its world model I mean this isn't entirely possible it's certainly like a sense of common sense like one aspect of common sense he makes another other few points saying scaling is not enough mainly criticizing kind of like you know can we just scale up GPT-3 in order to get intelligence and to which he says probably not reward is not enough which is sort of a criticism of this thing of can we just train reinforcement learning like to to to you know can we just train reinforcement learning more and more to reach it and not only is it some horribly sample inefficient but also if it lacks a kind of a world model he also says it's not enough yeah horribly extremely sample inefficient so one aspect of the paper is how do we learn more efficiently do we need symbols for reasoning this is an interesting question and he says maybe as far as I understand it he says probably at very high abstraction levels these sort of latent variables or states of the world might become so discontinuous that it's essentially symbolic at that point at which point one could also use kind of like tree search or so instead of a back prop gradient descent yeah like heuristic search methods including Monte Carlo tree search or other gradient free methods since things are so discontinuous so that is it a remain question a remaining question is whether the type of reasoning proposed here can encompass all forms of reasoning that humans and animals are capable of that certainly is the case so this was the paper again the core con the core suggestion right here is this model or these types of models where you have an energy based model the energy is kind of like a cost function that you attempt to minimize at inference time you can use this for planning in an actor by at inference time sort of deciding what actions would maximize that reward or minimize that energy or maximize the whatever using your world models in latent space right you can do this hierarchically by starting with the higher layers and the higher of determining high level actions which are essentially targets for the lower levels to match at any stage you'll do inference inference time optimization of the action sequence all of this can be trained using this arrangement right here where you do train your predictor and your encoders such that you can very well predict the latent representation of a part of the input this is self supervised learning from another part of the input however in order for this model to not collapse you need to regularize the latent variable and you need to regularize the information content of the latent representations that come out of the encoder lastly yeah I think I think that was it I hope you also got the idea behind the difference between contrastive and regularized methods contrastive methods sort of try to generate data that is goes well together and generate data that doesn't especially generate these these negatives here however due to the curse of dimensionality that gets less and less feasible as you go to higher dimensions in your latent representations on the other hand regularized methods don't suffer this problem as much and as we saw a regularizer can be put on any height of dimensional variables that was the wrong graphic but JEPA is exactly such a regularized method and does not rely on contrastive training you can still do it obviously but it doesn't it can be trained without because it prevents collapse through regularization yeah I hope also it became clear kind of what an energy function is and how to use latent variables inside of energy functions and this here no this here still a bit of a mystery how this all should work together but as I said it's more of a position paper and a vision and I think the JEPA is the core piece of this paper so I hope you enjoyed this leave a link to the paper let me know what you think in the comments and yeah I'll see you around bye bye
 Hello there, today we're looking at a path towards autonomous machine
 intelligence by Jan LeCun, also called the JEPA paper. Actually, I think only I
 call it the JEPA paper. But JEPA is a new architecture that Jan LeCun
 proposes as a part of this paper and we're gonna go into it as he himself
 describes it as the corner piece of this method. So you will learn what one of the
 Godfathers and Touring Award winners thinks of how we should reach machine
 intelligence or at least one proposal of it. The abstract reads how could machines
 learn as efficiently as humans and animals? How could machines learn to
 reason and plan? How could machines learn representations of percepts and action
 plans at multiple levels of abstraction enabling them to reason, predict and plan
 at multiple time horizons? These things are largely all open problems in current
 deep learning. Efficient learning especially. Deep learning is notoriously
 data-hungry. Reasoning and planning is something that a lot of these things
 can't do at least according to some people. And certainly reasoning,
 predicting, planning at multiple time horizons. These kind of things including
 abstraction. All of these things are still sort of out of the realm of current
 deep learning. So here is Jan LeCun's position paper as he calls it of how to
 reach these things. So he also says the text is written with as little jargon as
 possible and using as little mathematical prior knowledge as possible
 so as to appeal to readers with a wide variety of backgrounds. Now I don't want
 to actually go through the whole paper because the whole paper is what 69 pages
 long or so but I'll present to you sort of the core piece which is the JEPA
 architecture and just a little bit around that so you know what's going on.
 And I think it's pretty cool. Here he states the main contributions of the
 paper are the following. First an overall cognitive architecture in which all
 modules are differentiable and many of them are trainable. This is going to be
 one of the more wishy-washy hand wavy pieces of the paper. We'll quickly look at
 it. Then JEPA and hierarchical JEPA, a non generative architecture for
 predictive world models that learn a hierarchy of representations. So there
 should immediately you should see that you have a non generative architecture
 but for predictive world models which is going to be interesting. How can you be
 non generative yet still predict stuff? We're going to see that in fact the
 predictions happen in the latent space kind of like mu zero if you will. Third
 a non-contrastive self supervised learning paradigm that produces
 representations that are simultaneously informative and predictable. And the key
 thing here is going to be this non-contrastive part. Lacan makes a big
 deal out of pitching essentially pitting contrastive and non-contrastive
 methods and arguing why non-contrastive methods should be preferred above
 contrastive methods mostly due to the curse of dimensionality. Lastly a way to
 use H-JEPA at the basis of predictive world models for hierarchical planning
 under uncertainty. So the H here is going to be for the hierarchical extension or
 the hierarchical arrangement of the JEPA architecture. He says
 impatient readers may prefer to jump directly to the aforementioned sections
 will do exactly that. So there is a bit about world models and why it's
 important and here is kind of the entire proposed architecture. Now as I said this
 is a little bit hand wavy so there is essentially a world model which is you
 know pretty important and that's going to be the centerpiece right here that
 predicts the state of the world forward in time. So this is the actual world and
 the world model is trying to predict that. It's going to interact with this
 actor module right here. Obviously the actor is going to be what actually does
 the action however the actor could also act inside of the world model in sort of
 a simulated reality and plan forward what would happen if I were to do
 something or it could interact with the world model to find the best action to
 do and that's exactly what we're going to see. The short-term memory here is
 going to be used to train that world model and also to train that critic so
 essentially the things that happen in the world are going to be stored into
 the short-term memory and then the critic can be updated from that but
 will not look into that very well very much. Perception module right here is a
 module that takes the whatever the world gives and makes it available as a
 representation or as a perception. This is going to be the let's say the entry
 point to the systems that we have and this is very much the closest that we
 have to something that's actually working which is obviously our current
 deep learning systems they're very good at perception. So there is one thing I've
 left out which is this configurator right here. The configurator is sort of
 the master module that configures all the other modules depending on what
 situation they're in and so on and this is is definitely like there's a lot of
 hand-waving right here is like yeah yeah we can just have like a top-down
 configurator that configures stuff and I don't want to I don't want to go too
 much into it because there's not too much to go into but also it's not the
 core of the paper. We're going to go what we're going to go into the world model
 here specifically. So first of all he describes a two different ways of let's
 say acting in the world and here we are for the first time introduced to kind of
 like the notation of this paper which is very much in diagrams. So this is what he
 calls a mode one perception action episode. This goes very much with like
 Kahneman I believe it was Kahneman like mode one and mode two reasoning or
 thinking. So mode one is sort of reactive you simply go from perception of the
 world to action without much thought. It's kind of subconscious and this is
 encapsulated here. So we start with the world we get like some sort of so sort of
 observation we put this through the encoder right here that's going to give
 us a latent representation. This encoder is that that perception perception
 module that we saw before. Now different things happen but only actually one path
 is critical namely this goes to the actor right here this is the actor and
 the actor sends back an action to the world. As you can see this is a
 straightforward signal routing to the actor and back oh it even says actor
 right here. It says even this reactive process does not make use of the world
 model nor the cost. So there is a cost module that we saw which tells sort of
 how much something is whether it's good or bad this can be intrinsic motivation
 this can be external reward anything like this we can compute it however in
 this very basic loop the actor has been trained already to just act on a
 percept. At inference time the actor doesn't need to look at the cost anymore
 in order to act. This is what we're very used to from current like model free
 reinforcement learning algorithms they simply train the actor using the reward
 but then once it's inference time they simply let the actor act and rely on
 that training. This is a mode one perception action episode. In
 contrast to that we are introduced to the mode two perception action episode.
 This is a little bit more involved you can see here that we are rolling out the
 world model forward in order to do something and what do we do again we
 have an input here we go through the encoder this is probably a wrong color
 as it's the same we go through the encoder however now we are going to roll
 out the world model across different time steps and how are we going to roll
 out the world model we're going to use the actor right here so the actor is
 going to take that state that gets from the encoder and propose an action this
 is the same actor as before it's just sort of a trained thing that's proposing
 some action okay good enough we can use that into the world model together with
 the latent prediction you realize right here the predictor here this thing it
 takes whatever comes out of the encoder right here that means it takes a latent
 state of the world and it predicts the next latent state of the world that's
 why he calls this non generative these these world models and and these
 encoders they all go to latent space and then they predict stuff in latent space
 so in fact it doesn't predict the world it predicts the latent state of the
 world which enables it to focus on what's truly important for the task
 obviously modulo how well you can train this thing to actually do that and how
 you can prevent it from collapse we'll get to all of that however you'll notice
 that now we can give the actor the representation it proposes an action we
 can actually use the world model to predict the next state from that next
 state we can ask the actor for an action the actor gives us an action and we can
 predict the next state now what does that give us in fact that gives us quite
 a bit let's let's assume let's just assume that episodes are always the same
 length and forget about this forget about this forget about this episodes
 are always the same length this length right here and you won't get any reward
 or anything or any intrinsic reward until the very end like until the very
 end there's kind of like a reward or a cost or something like this well we can
 compute it which is fine we could already do that before it's informative
 but we didn't do anything with it however once we have that whole loop
 done if all of these things are differentiable what we can do is we can
 say well this action sequence right here right now would give us like a reward of
 five okay can we make that bigger well since everything's differentiable I can
 certainly use back propagation and gradient descent to ask how would this
 action need to change in order to make this thing go higher right maybe I need
 to switch to a different action now it's six well can I also change that action
 to make it go higher oh well I can now it's seven and so on so I can modify I
 can optimize all of these actions at inference time using gradient descent
 right this is if this is not familiar to you it's kind of the same as if you
 construct an adversarial example to an image classifier that's also gradient
 descent at inference time so here gradient descent isn't used to train
 any of these modules we assume that training is done gradient descent is
 used in order to improve this initial action sequence to a more optimal set
 of actions and we do that you know we improve these actions here we're using
 gradient descent through all these modules until we have completely
 optimized the action sequence and which means that this very first action is
 probably a very good action like hopefully a better action than was first
 proposed by the naive actor and then we can take that action and feed it to the
 world as an action so this is mode to perception action episode this is kind
 of the model thinking about the future and figuring out through forward-looking
 what do I need to do what do I need to change to improve the outcome how can I
 how can I make stuff better and that necessarily uses this world model right
 and obviously this is just more general if you include all of these costs which
 you can have after every step you can include some kind of discount factors
 and yada yada yada yeah so inference time optimization isn't new but it is
 sort of how the car sees a way one way of how to make these things plan forward
 so the text says through an optimization or search procedure the actor infers a
 sequence of actions that minimizes the total energy so these things are called
 energy and note that it doesn't necessarily need to be optimization it
 could also be search it could be evolutionary search it could be tree
 search anything that actually tries to improve the action sequence at inference
 time an instance of classical model predictive control this is an instance of
 classical model predictive control with receding horizon planning all right and
 this here is how we would train such a thing so not such a thing sorry let's
 assume that we have the two modes we have this naive actor and we use the
 naive actor to propose sequences for the longer like for for this thing right we
 propose that first sequence using the new fact naive actor in mode one mode
 two language there is such a thing as if you do something often and you do it
 consciously at some point it becomes subconscious right like muscle memory or
 something like this well how could this work this is how this could work in this
 framework so you'd have essentially these actions right here are the ones
 that we have come up through this whole planning process through this whole
 optimization process well what you can do is you can simply ask the actor or
 take that output from the initial actor and then you can try to make these
 things as close as possible right you have all the things right here
 everything's differentiable so you can train the actor to essentially match
 those better actions because you know the actor would propose one action
 however this other action you found to be superior using your world model now
 obviously that requires you to have a good world model but if you have that
 then you can improve this low-level actor and at some point that initial
 action sequence that it proposes will already be close to optimal it's kind of
 an approximation that you distill into this actor so this is first introduction
 to the system right here we're going to look a little bit more into how these
 systems should actually work and here starts a discussion of two things the
 first one is self supervised learning and the second one is energy-based
 models the first one is sort of a training paradigm of how to train
 models using unsupervised data the second one is I want to say a way of
 thinking about these models it's a formulation of a system and we'll get to
 it and they are connected so self supervised learning Lacan sees this in
 the following terms I have a piece of data which is this whole block right
 here and I try to predict I try to like mask out the piece which is this right
 hand side right here like I pretend I don't know it and then I use the thing I
 do know and I try to predict the thing I don't know it's not exactly that
 however in fact what I want to do is I don't want to predict the thing I don't
 know I want to create this thing called an energy function an energy function
 tells me how well these two things fit together and this is going to become
 clearer in just a second but the way it's formulated right here is that to
 capture the dependencies between the observed parts of the input and
 possibly unobserved parts of the input so this is supposed to well it's gonna
 as I said it's gonna get clearer in just one second but what you want to do is
 you want to train a system that sees the data space in this format right here
 which is going to be so-called energy landscape so if you have imagine this is
 a video sequence right here so there is a bunch of frames and a bunch of frames
 and frames frames frames frames frames right here so if you have this energy
 landscape right here you're trying to relate first like the start of a video
 sequence to the end of a video sequence you can imagine this in a very high
 dimensional space essentially where all the frames here are concatenated to to a
 big vector and all the frames here as well and the energy function or the
 system that you train should assign a very low energy to all of the video
 sequences that are let's say realistic or in other words here is the X
 whenever X is this video sequence then and Y is this video sequence then the
 energy function should assign a low energy to that if the two could
 actually follow each one another so if Y could follow X if Y would be a logical
 continuation of X in video space the energy function should assign a low
 value to that this formulation is very cool because it means if we don't need
 to predict Y from X directly because there could be multiple video sequences
 right following that same beginning and that means if we were to just predict Y
 then we would probably train the system I mean we can still do it but we can
 probably we will probably train the system to say no there is one correct
 continuation however if we train the energy function the energy function
 could assign a low value to any possible continuation as long as it assigns a
 high value everywhere else we're good so we're trying to produce systems that
 behave like this now I for I used to think energy function and training loss
 are the same thing but I know that young Lacan is very adamant about the thing
 that an energy function is sometime something that you minimize at inference
 time while the training loss is something that you minimize at training
 time sometimes they are very similar and overlapping for example a lot of times
 the the energy function and the training loss are the same formula and by
 training the system you actually immediately cause it to minimize that
 energy at inference time simply by forward passing in the model however we
 can do more with energy functions which we're going to see right now now we
 introduce latent variable energy based models this is the same formulation as
 before we have an X and a Y and we have an energy function that tells us how
 well those two are compatible with each other which is going to be this thing
 right here however as we've seen there could be many Y that are possible for a
 given X right so just by seeing X we can't tell you know which of the wise is
 compatible and that's why we introduce a latent variable Z so this Z right here
 is going to capture all the information about Y that isn't directly in X for
 example if we have a video of some some car right the car ah no obviously we
 have the tracks and they split right here and they go right here and there's
 a bunch of people and there is a person so the trolley car problem if we have
 the trolley car problem and it goes down this is the video sequence is up to here
 right and we don't know how the lever is this is hidden from us there are two
 possible continuations one here one here the we can't tell just from X X is here
 and Y is the continuation so the variable Z we introduce it to capture
 that information in this case the variable Z is either left or right it's
 binary variable and in order if we have an X and we have a Y in order to compute
 that energy that tells us how well the two are compatible we need to minimize
 over Z so what we need to do is if we have a particular Y let's say we
 actually have the Y where the card goes here right so it goes on the lower track
 we ask how well do these two video sequences follow from one another well
 the answer is they follow very well from one another because certainly the card
 going here is one possible continuation and that means that we had to search
 over all the possible futures which means we had to minimize over Z so we
 considered Z going up or Z being down and we determined the Z being down leads
 to the lower energy and that is in fact a very low energy now what happens if we
 actually input a video sequence that isn't that isn't let's say we input a
 video sequence instead of this so the cart is here it goes here and then the
 next video sequence is of I don't know like a Teletubby so there's a Teletubby
 it's a sequence like it's an episode from Teletubbies so these two things
 don't follow from one another and again we do the same thing we minimize over Z
 but no matter whether we think the lever is up or down as the minecart approaches
 it never it's never a good continuation that there is that followed the next
 frames are an episode of Teletubbies so that's how you think about latent
 variable energy based models is that there's a hidden variable the hidden
 variable captures everything that is sort of not captured in X about Y and we
 minimize over that latent variable to get the actual energy which means we're
 looking for the the value of the latent variable that is most that makes X and Y
 most compatible and yeah so this is also going to be quite powerful which means
 that if we already know that X and Y are compatible with one another then
 minimizing over Z if we have a good energy function minimizing over Z could
 actually tell us something about the latent structure of the world so we
 could infer Z or if we have this model trained then if we have an X we could
 actually sample some Z values in order to maybe produce different future or
 different possibilities of Y this gives us a lot of freedom to handle
 uncertainty in the world or simply unobserved structure in the world now
 there is a problem with these types of architecture and that is going to be
 collapse if you've noticed that we simply introduced this variable Z right
 here and we said well it contains everything that's not contained in X but
 there is actually no restriction for that the if we train this model just
 with let's say gradient descent and some loss and will make all of these
 variables unrestricted then very quickly the like the model will become
 basically useless because let's say our loss function is how well we can predict
 from X and Z how well we can predict Y right that's the general form now we
 minimize over we minimize over the values of Z which means that if we simply
 set Z equals to Y we can always perfectly predict Y and that means X
 just becomes completely useless and the prediction function just becomes the
 identity function this is known as collapse and we don't want it what we
 want to do is restrict Z for example so that like here it can only take two
 particular values while X and Y are sequences of video frames so that that
 doesn't happen or we can do it with some architectures so let's look at different
 configurations right here of these energy-based models in any case D here
 is the D is the energy or the compatibility function what if we have a
 deterministic encoder that gives us the latent representation of X and then we
 use a predictor module in order to predict Y so we'll just predict Y
 directly and then compare it with the true Y and then we have a loss in
 between them this cannot collapse because well we need to predict the
 actual Y now let's introduce one of these latent variables and we're in
 exactly the situation that I just described again we compute the
 representation for X but we'll introduce this Z that can vary over a certain
 domain which gives us a very a domain that we can control for the output of
 this predictor right here if we now try to predict Y from Z and X we can as I
 said just set Z to Y and we'd always be good so this can collapse what about
 this thing right here the auto encoder this seems oh this is just the same as
 the first architecture this is the same as the first architecture except just Y
 goes in so instead of X and Y we just have Y goes through an encoder gets a
 latent representation goes through a decoder that gives you back the gives
 you back an estimation of oneself and as you know an auto encoder if you don't
 restrict it somehow in the middle here then it can just become the identity
 function again and be useless and the last one is this joint embedding
 architecture now this is looks or sounds an awful lot like the thing that the
 paper is describing and as you can see it can in fact collapse so we're going
 to have an encoder for X and an encoder for Y these could be the same but don't
 have to be they're going to give us two latent representations but or then we
 use an energy function to compute how well these two latent representations
 fit together maybe with the help of a latent variable now if the encoders
 right here simply always output the a constant vector and this one does too
 and the constant vector is in fact the same constant vector then we're always
 good right we always output the same vector and this cost function up here
 we always say yeah they're completely equal this is completely cool they match
 together super well so this can definitely collapse and we need to do
 something against it this is a the main discussion here that leads us into
 contrastive versus restrictive or regularized architectures and this is
 going to lead us to the gear architecture now it's going to be JEPA
 but we're building it up slowly so how do we design the loss to prevent collapse
 now remember where we are we started with self super with we started with
 recognizing that self supervised learning is probably a good thing
 because we can do it without labels right we can handle multiple domains with
 this all we need to do is we need to pretend to not know some part of the
 input and use the other part to predict something about that unknown part we
 then said okay we want to formulate this as an energy based model where we'll
 obtain a model that assigns a low energy to all the compatible pairs of inputs
 and a high energy to all the incompatible pairs of inputs and that
 means at inference time we can do a lot of things for example minimize that
 energy in order to find pairs that go really well together or if we have a
 pair we can we can look at the energy and judge how well that fits for example
 you could interpret something like clip as an simple energy based model that
 simply computes at inference time that energy and if you view these VQGAN plus
 clip optimization procedures that were really cool before dully was or mini
 dully was open-sourced then this is exactly minimizing an energy at
 inference time so just so you can imagine something below it we then
 introduced latent variables into the mix saying well for a given beginning of a
 video for example there's going to be multiple continuations and this could be
 captured in a latent variable this could also be for a given left side of the
 picture there can be multiple right hand sides and so on this can be captured in
 latent variables and to compute the energy we need to minimize we then
 discovered that this is probably prone to a thing called collapse among other
 things other like other aspects of this architecture are also prone to
 collapse and now we need to do something against it there are two ways of doing
 something against it there is contrastive training or regularization now
 contrastive training you might be aware of that so on the left hand side you
 have the situation of like a half trained system so this half trained
 system already has some training examples that have a relatively low
 energy but there are still some that have a high energy so training means
 that at the end we want to end up with a model that assigns a low energy to
 certainly all the training examples and some space around it so we want the
 energy at the low energy region to extend to these training examples and
 maybe cut out a bit from that middle right here push the energy up a little
 bit to say well actually these samples in that space are not compatible with
 one another so contrastive methods are very very classic methods I don't
 actually know if clip is trained as a contrastive method but many many sort of
 of these image image or self supervised image training procedures are certainly
 contrastive what they'll do is they'll have an image they are going to make two
 variations of that image maybe by random cropping and data augmentation and so on
 then they'll take another image like a third image from the database and get
 they're going to make also a variation of that and then they use the embedding
 models to embed all of those already so embed embed embed this into latent space
 so this here would be your standard ResNet encoder or something like this
 this is usually used in image pre training right and no no no so this will
 give you a data point somewhere in high dimensional space and then what you do
 is you try to pull the two that are from the same image together and you push the
 ones that are from different images apart this is contrastive training and
 it relies on you coming up with these negative samples so what you want to do
 is you want to create these contrastive samples that you just kind of jiggle the
 data points around a bit that you have in with using either augmentations or
 just some sort of distortions and so on now what we've done right here is we've
 chosen random negatives but we could also actually mine hard negatives that
 are very close to the training data however this quickly runs into problems
 as you know there's the curse of dimensionality if you will have a data
 point and you want to wiggle it into different directions those directions
 increase exponentially as you go up in dimensions so this whole approach of
 finding training examples or finding negative examples around a training
 example to do the contrastive training is getting less and less tenable in the
 higher you go with the dimensions and therefore Yandaka advertises for
 something different which calls regularized methods now regularized
 methods have other means of restricting that space that is low a low energy
 region so there is no there are no constructed data points outside here
 that you know make the energy high here and low here but there is a natural
 tendency of the system like obviously you enforce you enforce the system you
 encourage the system to keep the region where the energy is low very small and
 this is done through regularization and we'll see how this is done in this joint
 embedding predictive architecture so this is the basic module we've already
 seen it this was the thing before that was no almost almost so this is almost
 the same as before but again we have our X and our Y two points that we want to
 check if they're compatible with one another will embed both of them using
 deterministic encoders this gives us latent representations of X and Y so X
 could be the last state of the world Y could be the next state of the world so
 we map these to the latent representations then we'll use this
 predictor right here to predict the latent representation of Y from the
 latent representation of X okay this is the an important part here that
 differentiates us from before before we try to predict Y directly now we try to
 predict the latent representation of Y from X we're going to make use of a
 latent variable right here I guess this is optional but it's built into this
 model right here so this controls which Y or which latent representation we're
 getting so Z can vary over this domain right here which then leads the S of Y
 this thing here to vary over this squiggly domain right here so this
 probably means that Z could vary over a relatively simple domain but through the
 power of neural networks this is going to be transformed into some complicated
 manifold like as I said does the current car turn left or right gives rise to an
 entirely different series of video frames and this is then going into the
 energy function whether or not the representation of Y is compatible with
 the predicted representation of Y now since we are actually trying to predict
 the representation this energy function right here is probably very simple like
 something like a cosine distance or an L2 distance or something like this that
 actually makes the representations equal energies can be much more
 complicated but yeah so here it repeats the main advantage of JEPA is that it
 performs predictions in representation space eschewing the need to predict
 every detail of Y and enabling an elimination of irrelevant details by the
 encoders obviously that's also a thing that's going to be subject to collapse
 so he says you know these encoders they could just throw away everything that's
 not relevant about X and Y because we never need to predict Y directly from
 something in here right we don't do that so we can just forget about stuff that
 is not important now how why aren't we forgetting about all the stuff and here
 is where this regularization comes in so how to train a model like this well the
 first of all we obviously train it by minimizing this predictive error right
 here this is the basis right we actually want to predict the latent representation
 of Y from this thing or sorry from the latent representation of X right we want
 to predict this thing we actually need to compute the loss between these two
 things that's exactly this D function right here this is the core right this
 is unchanged from before however we have a couple of regularizers here to prevent
 collapse first of all we regularize Z this thing right here what do we do we
 minimize the information content of Z and that means as before we said well if
 we let Z just be anything that we want and given that we minimize over Z at
 inference time this Z can just become equal to Y and make D be zero all the
 time so this is not good so we need to minimize we need to regularize Z before
 I said Z could just capture the state of the lever left or right right then you
 know that there is so much more information in the latent representation
 of the future video frames that Z cannot possibly even if we minimize over this
 binary variable cannot possibly capture all of that so restricting the domain of
 Z is certainly a way to regularize it we can also I guess classically regularize
 it with some L2 regularization we could quantize it we could apply sparsity
 regularization anything like this that limits the Z this latent variable that
 we minimize over is needed right here to prevent collapse the other things that
 are needed are the things that you see right here so these are regularizers on
 the information content of the latent representation so what we want to do is
 we maximize the information content that the latent representation of the
 encoded signal of the encoder perception has about that about that variable
 itself well I guess it doesn't need to be actually about that variable it
 simply needs it simply means we need to maximize the information content of that
 variable how are we going to achieve that there are also various ways of
 maximizing the information content essentially it just means that if that
 variable always has the same value it doesn't have much information inside of
 it so what we can do for example we can use a mini batch approach and have many
 X right here X X 1 X 2 X 3 X 4 right and these if these are all independent we
 encode all of them we get a mini batch of latent representations and we can do
 something like we say well all of these need to be different right and they're
 for example their covariance matrices must be identity or something like this
 so there are various ways and a lot of Yandere also points to some papers for
 example Vic reg and Barlow twins that have already or can be framed in ways
 like this but this is a general framework minimize the information
 content of the latent variable and maximize the information content of the
 encoded signals which makes sure that there isn't a collapse this directly
 counteracts that down here I believe yeah exactly we have Vic reg as a
 system so direct implementations of this you can see right here the L2 loss
 between the representations the regularization here I don't exactly know
 how that's regularized doesn't say here but then the maximizing of the
 information content here is or here of this thing is done via via regularizing
 the covariance matrix right here so yeah at the last thing that he says here is
 that we could also bias Jepa to learn useful representations saying it would
 be useful to have a way to bias the system towards representations that
 contain information relevant to a class of tasks this can be done by adding
 prediction heads that take the latent representation as an input and are
 trained to predict variables that are easily derived from the data and known
 to be relevant to the task so now we're essentially going into the domain of I
 don't know natural language pre training with with something like t5 or t0 where
 you just kind of throw tasks at the system and hope and jointly train all
 the tasks and hope that you know it learns latent representations that are
 kind of useful for language tasks Lacan says you could also in addition to doing
 all of this you could also attach some kind of a prediction head right here and
 then have another loss from a supervised signal or maybe a imitation
 learning in reinforcement learning or something like this all of this is
 entirely possible because without it without having these heads right you
 now have a system that just sort of does an information trade-off right it just
 kind of trades off these these different regularizers right here and tries to get
 like as much information transmitted through this path here about the latent
 representation of why like it tries to it tries to counteract all of these
 regularizers it tries to minimize the information right here because then it
 can do a better job it tries to maximize the information content here as much as
 it can you counteracted via regularization so you're just kind of
 playing this information game with the variables right here and it is up I
 would say to the designers of the system to set the parameters on all of these
 different loss terms correctly such that the latent representations are useful
 and I also think a big big big part here is on the data itself like the entirety
 of usefulness without prediction heads of the system is just down to the data
 right if you have data if you want to learn something about let's say
 different chess positions like you want to pre train a chess computer with this
 thing right you better input data that has different chess positions that
 differentiate themselves in the relevant aspects of chess positions and it's
 probably not a good idea that you always have the same chess position but you
 vary the sort of the shades of gray in the chessboard right so this thing will
 sort of learn what is predictable from the data that it gets so you better make
 sure that that data the variation in that data captures what you need to get
 out of it right so what can we do with this we can arrange it in a hierarchical
 fashion so this is going to lead us to hierarchical JEPA which is going to be
 the final the super sane form right here of the model in fact if you think about
 this going back to the very beginning where we asked ourselves how could we
 use a fully differentiable system to plan ahead in time well if you consider
 this to be you know your state of the world for example or frames in a video
 or something like this you could arrange this system like we did are doing here
 to predict over multiple time steps right yeah as as we do right here so the
 lower level predicts over short time frames while the higher level you can
 see over here that this latent representation is in fact obtained from
 the latent representation of the lower level by a second encoder and then makes
 predictions over a longer period of time so the hierarchical arrangement of these
 things is entirely possible and we can use that to do hierarchical planning so
 this goes back to the very beginning we at the beginning we saw how can we do
 mode to planning if we have such a world model right and now we're going to do
 this in a hierarchical fashion so what do we do again say this is the state of
 the world and we know at some point we have a desired outcome like a cost
 function or a reward or something like this well if we have trained such a
 multi-layer predictive model in latent space what we can do is we can do what
 we did at the beginning at this higher level right here so we're just gonna do
 this thing up here first which means that we're going to ask this high level
 actor and we'll get to what high level actions are but assume there are high
 level actions for example let's say I need to get to the airport right the
 high level actions are simply you know I'm gonna go out of the house I'm gonna
 get in the car I'm gonna drive to the airport and I'm gonna park the car there
 those are high level actions and low level actions would be the actual you
 know movements you do so we can ask this high level actor to give us high level
 actions we can roll out the world model with it until we are here we can use
 back propagation or search or some other optimization technique in order to
 refine these actions as well as we can right and then we have here targets for
 these low level actions now before these things on the lower level were
 themselves kind of rewards that we get from from the world but this is now up
 here and the rewards on the lower level are simply how well we match those
 targets that are given by the higher level so this this action this high
 level action right here could be get in the car right so now get in the car
 becomes the target and we can use our lower level planning algorithm in order
 to determine the best actions again using proposals back propagation
 optimization and so on to get in the car in fact we can do it for all of these to
 match all these higher level actions which gives us entire action sequence
 that would optimally fulfill the plan to to match these higher level actions and
 you know if we're super duper engaged we could also optimize all of the different
 levels together until we have the optimal sequence of lower level and
 higher level actions in order to reach this goal right here at that point we
 can be relatively sure that this first action right here will serve us just
 well and we can actually send that to the world get the next state and do it
 all over again we can even use the short-term memory or something like this
 in order to start at a better place for next time already although the short-term
 memory here is used to store states in order to train the train the loss modules
 and the critics this is if you are actually in an uncertain environment you
 could even introduce these latent variables right here which you can infer
 so if you want to reach a certain goal right here you can infer the latent
 variables also through some sort of optimization procedure or you can sample
 the latent variables in order to give you different continuations of your
 world model up to you and there are various possibilities that open up with
 these with probabilistic world models but I don't want to go too much into
 this I think I hope you get the concept by now of how to think about these things
 again this we are again in the space where we have the models trained and we
 need to do inference time inference time decision of what action to take right
 training this thing is a different game training this thing is done via this
 method oh sorry this general method by regularizing by minimizing the
 prediction error in the latent space okay I think that was it for the paper
 the rest is about the rest of the architecture designing and training the
 actor data streams designing the configurator yeah this it gets a bit
 hand-wavy at that point I mainly wanted to bring the mainly wanted to bring the
 the JEPA architecture to you and you hope you understand that yeah so there's
 a bit of broader relevance of the proposed approach could this architecture
 be the basis of basis of a model of on animal intelligence now it's the answer
 is maybe but I found this paragraph here pretty pretty astounding the presence of
 a cost module that drives the behavior of the agent by searching for optimal
 actions suggest that autonomous intelligent agents of the type proposed
 here will inevitably possess the equivalent of emotions but that's
 escalated quickly in an analogous way to animal and humans machine in emotions
 will be the product of an intrinsic cost or the anticipation of outcomes from a
 trainable critic cool could this be a path towards machine common sense to
 which he says I speculate the common sense may emerge from learning world
 models that capture the self-consistency and mutual dependencies of observations
 in the world allowing an agent to fill in missing information and detect
 violations of its world model I mean this isn't entirely possible it's
 certainly like a sense of common sense like one aspect of common sense he makes
 another other few points saying scaling is not enough mainly criticizing kind
 of like you know can we just scale up GPT-3 in order to get intelligence and
 to which he says probably not reward is not enough which is sort of a criticism
 of this thing of can we just train reinforcement learning like to to to you
 know can we just train reinforcement learning more and more to reach it and
 not only is it some horribly sample inefficient but also if it lacks a kind
 of a world model he also says it's not enough yeah horribly extremely sample
 inefficient so one aspect of the paper is how do we learn more efficiently do
 we need symbols for reasoning this is an interesting question and he says maybe
 as far as I understand it he says probably at very high abstraction
 levels these sort of latent variables or states of the world might become so
 discontinuous that it's essentially symbolic at that point at which point
 one could also use kind of like tree search or so instead of a back prop
 gradient descent yeah like heuristic search methods including Monte Carlo
 tree search or other gradient free methods since things are so
 discontinuous so that is it a remain question a remaining question is whether
 the type of reasoning proposed here can encompass all forms of reasoning that
 humans and animals are capable of that certainly is the case so this was the
 paper again the core con the core suggestion right here is this model or
 these types of models where you have an energy based model the energy is kind of
 like a cost function that you attempt to minimize at inference time you can use
 this for planning in an actor by at inference time sort of deciding what
 actions would maximize that reward or minimize that energy or maximize the
 whatever using your world models in latent space right you can do this
 hierarchically by starting with the higher layers and the higher of
 determining high level actions which are essentially targets for the lower levels
 to match at any stage you'll do inference inference time optimization of
 the action sequence all of this can be trained using this arrangement right
 here where you do train your predictor and your encoders such that you can very
 well predict the latent representation of a part of the input this is self
 supervised learning from another part of the input however in order for this
 model to not collapse you need to regularize the latent variable and you
 need to regularize the information content of the latent representations
 that come out of the encoder
 lastly yeah I think I think that was it I hope you also got the idea behind the
 difference between contrastive and regularized methods contrastive methods
 sort of try to generate data that is goes well together and generate data that
 doesn't especially generate these these negatives here however due to the curse
 of dimensionality that gets less and less feasible as you go to higher
 dimensions in your latent representations on the other hand
 regularized methods don't suffer this problem as much and as we saw a
 regularizer can be put on any height of dimensional variables that was the wrong
 graphic but JEPA is exactly such a regularized method and does not rely on
 contrastive training you can still do it obviously but it doesn't it can be
 trained without because it prevents collapse through regularization yeah I
 hope also it became clear kind of what an energy function is and how to use
 latent variables inside of energy functions and this here no this here
 still a bit of a mystery how this all should work together but as I said it's
 more of a position paper and a vision and I think the JEPA is the core piece
 of this paper so I hope you enjoyed this leave a link to the paper let me know
 what you think in the comments and yeah I'll see you around bye bye