Neural Ordinary Differential Equations
https://arxiv.org/abs/1806.07366

Abstract:
We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

Authors:
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud
 Hello and welcome. Today we're going to look at Neural Ordinary Differential Equations by Rick Chen, Julia Rubinova, Jesse Bettencourt and David Dovenoe. This has been quite an interesting kind of paper to see because it's a bit special. We're going to go over parts of it, not the full paper, just kind of the important parts because the paper is quite packed and we'd rather explain it in parts and kind of get the gist of it. So basically what they do is they say we introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black box differential equation solver. These continuous depth models have constant memory cost, adapt their evaluation strategy to each input, can explicitly trade numerical precision for speed. It sounds awesome, honestly. It sounds really cool and it sounds really new. Let's jump in. What they say is let's look at kind of classic neural networks, especially residual neural networks. What residual neural networks do is in each hidden layer they kind of have a representation. This is kind of their hidden representation at layer t. What they do is they then add something. If you don't know a recurrent neural network is where you have, let's say this is your hidden state ht, and in a classic neural network you would just have a weight matrix here, blah blah blah blah. You do a matrix multiplication to get ht plus 1. So to get the next kind of the next hidden state you do a matrix multiplication by a big weight matrix here w. In a residual neural network what you do is you have this weight matrix w, you multiply it to get delta ht plus 1 and you take ht and you add the two. You add ht and delta ht plus 1 to arrive at ht plus 1. That's a residual network. It basically doesn't learn the transformation to the next layer but it learns how is the next representation different from this representation. What do I need to add to this representation to get to the next representation? It's reasoned that for deep networks since each layer only does a little bit of transformation we should basically bias it towards keeping the representation the same and just kind of changing it a little bit. So this is the inherent bias, the identity transform. So that's a residual network. This here is characterized by f of kind of theta and ht. So this is kind of the this is what we called delta h. It's now called f. So this f would be the kind of neural network layer and theta would be the parameters of it. So the weight matrix in our case. They say okay what if you do many of those? So they say basically what this is is kind of a time process. It's kind of you have a state and the next state and the next state and you always learn how to go to the next state to the next state and so on. What if you go very deep and what if you look at this as a time process and kind of make these steps very small? Make these super small and basically what if you have many many infinitely many layers? I say well okay this then becomes a dynamic process. Basically an ordinary differential equation where I say okay my time is now continuous and I look at it as a linearization as a local linearization basically and I say okay I basically specify how to get from this time to the next instance of time. The next instant the next infinitesimally small instance of time by specifying this f and in the continuous case this is to say that the derivative of the hidden state is now parameterized by a neural network. So if you know what a differential equation is it has like a start state and then what you do is you specify how at each point in time so that's t at each point in time how does the gradient look so maybe the gradient looks like this and then what an ODE solver will do is the ODE solver will say okay the gradients we're gonna do an infinite small step in this direction and then it goes back to f. What's the gradient at this infinitely small step next in time and then f would say well the gradient is like this and then the ODE solver will go like okay I need to be a little bit flatter so I go here so what's the gradient at this time okay maybe it's up this I need to go up here so the ODE solver will kind of construct a curve and at each point it needs to look that whatever f says is the gradient is actually the gradient right if this is the gradient this is the gradient this is the gradient so that's that's kind of how an ODE works and that's they say okay you can actually look at residual networks here as being a discrete time analog to such an ODE so what we want to do is actually we want to specify we want to actually and this is the the crazy part right or the cool part is we want to do this for neural networks basically we simply specify an ODE and the start state here the start state is let's say if you want to build an MNIST classifier it's our it's our image right the start state is our MNIST image and we're simply training a neural network such that the ODE that the equation if you solve it the curve at the end will arrive at the correct class I mean that's that's I'm skipping a few parts here about dimensionalities and so on right because you need to keep in the same dimension but in essence they say here we start out with our input and we train the neural network to give us the correct gradients the correct derivatives of this curve at each point in time such that when you solve the ODE at the end point you are going to be at the correct label so that's this is the input to your task basically and this is the output right but instead of having a neural network go from input to output you have a neural network that parameterizes how you go from each step in time to the next one what's what's the gradient at each point in time that's that's the kind of gist of it and that's that's kind of really cool it's a really new approach alright so they give various advantages of this and so here is this demonstrated again right you are here this is your input and you want to go to the output and then the loss of the loss that you specify it can depend on kind of either on the output as in like an image classifier or it can depend on intermediate states this is it's kept general right so the way they go about it is they say well okay but so the neural network now specifies how to get from one step to the next right here and the neural network has parameters right so we we need to train this network such that the correct output is given to some input right we actually need to train it so we need to we need to some how way to train these parameters theta and they say okay we do gradient descent on theta like in a classic neural network but now we need it's not it's not so easy right it's not one pass through this function it's like infinitely many passes through this function until you arrive here and then if you basically need to somehow get a gradient with respect to these parameters here so they say this again the loss of the this is the loss of the end state right is the loss of the start state plus the the integral over time of this is derivative which is basically this curve and the curve is given by an ODE solver where we input all these things so we need gradients with respect to that how do we do that and they give away here of saying okay we could either kind of back propagate through the ODE solver but that would you know depend on the ODE solver and so on but there's another method there's called what's called the we need the what's called the adjoint so this is reverse mode differentiation of an ODE solution adjoint sensitivity method solves an augmented ODE backwards in time so basically what you need to do is you forward propagate you come here right and then what you can do is you can solve the second ODE so you can generate a second curve here this one and don't worry about these little jumps here you can solve the second curve and the second curve together with the first and second curve you can then compute the gradients you need right so the second curve is is basically simply something like the application of the chain rule to the continuous domain and you need to you need to adjust these jumps here only when your loss depends on intermediate states this is this is kind of the offset caused by including or not including the loss so let's dive a bit further into this adjoint state what's the red curve the red curve is called a and what's a a is a curve and this is the differential equation for it again we specify the curve a by specifying its start state and its derivative and from its start state and its derivative at each time the ODE solver is able to construct the curve entirely so a t it says here is del L to del ZT this means how does the loss depend on this ZT on the hidden state right how does the loss depend on the hidden state at time T so it doesn't even have to be any of these points here how does the loss depend on this hidden state here and in order to find that out you would need to go you would need to develop the the curve until here right and then calculate the loss and then back propagate through here but you can do this by calculating this adjoint thing so as you can see here is a demonstration it's an example right so the start state here is simply given by the loss how does the loss of this state how does the loss depend on this state well simply by plugging it into the into the loss equation right so your losses might be a cross entropy loss or something how does the loss do that depend on this state here well we go we go from this state that we already know and we know how in reverse time so backwards in time this sensitivity of the loss develops so we go and we develop this curve until here and we say aha this point influences this loss in this much basically right so so and if the loss explicitly depends on this point then we have to we have to calculate in this offset since this point here only depends on this time up till here and then it changes so there is there's a discontinuation but don't worry about that too much basically what we can do is we can calculate the curve in a forward pass curve and the loss in the forward pass then we can do a second pass backward again by an ODE solve to say how does the how does the loss depend on each one of the states here of the hidden states right so that's the second point but that's not all because we're ultimately not interested in the how the loss depends on the state where the we're interested in how the loss depends on these parameters that tell us how to get from one hidden state to the next but luckily we can then simply evaluate this integral that depends as you can see here on a and on Z we can evaluate this and get the gradients for the the parameters right so I also have to say the parameters are static so the parameters are given over the entire duration of this they're they're the same and it's simply what changes is time alright so this is how you can get this is how you can get gradients with respect to parameters and the cool thing is now you can train these you can actually train this neural network here that tells you how to go from one state to the next such that if you input the digit 2 as an image well you can output to I mean not exactly but that's that's the point right you can by by going through this motion by going through this od solve so that's I mean that's immensely cool they actually define how to do this here in one forward one kind of backward pass you can solve everything at the same time it's it's pretty cool and they evaluate their their net and they compare it with a different bunch of other nets and they interestingly show that so basically with an od solver you can never kind of tell how many evaluations it's going to do because it's going to get increasing like it's increasingly accurate over time so you let it run and maybe it's going to first generate a curve that's like something like this right and then it needs to say crap okay I need to go back and refine and then it maybe goes the curve like this and so on so it gets continually closer over time and for that it needs to kind of query it's like a query it needs to query this this F so you need to give it the function as an invaluable function and it goes and just okay I need to I need to know it here okay I got it from here okay I need to know it here okay I got it from oh no I didn't get it okay I need also need to know it here all right and so you can never know how much they will evaluate but you basically have a parameter to trade off accuracy and how much they evaluate that's what they show here so the the less error they want in their forward pass the more forward passes they have to do that's this curve the more forward passes they do the more time they have to invest right that's this curve but interestingly the more forward passes the time required for forward passes or the evaluations required for passes increases also the evaluation required for backward passes but not by much so that the backward passes require about half the amount of evaluations that's forward passes which is encouraging since the the backward passes don't go kind of overboard like if you had to back propagate through the operations of the ODE solver itself and they also show as your training epoch continues that the ODE solver requests more and more evaluations for so for the same epoch basically or the same samples within different epochs which means as it gets more accurate kind of needs to know more and more and more about the the samples basically about the test the training samples which is all basically showing that this kind of works yeah so they they kind of to get into normalizing flows which I don't want to get into here much because we haven't done a video on that yet we'll do one but they basically show that it's it's quite easy to do normalizing flows in a continuous fashion and the topic normalizing flows it's in itself pretty cool what they do at the end is they say okay what we can now do is we can actually take sequential data so now we've just talked about let's input one data point get out let's say a label or something which we can actually do sequential data and let's for example have an RNN encoder for our sequential data so here here these are data points right these are measurements like a blood pressure of a of a person and what we can do is we can do a variational autoencoder we've talked about this we can have an RNN encoder parameterize a distribution and then as a decoder have this ODE neural network and basically what that allows us to do is that allows us to deal with time steps that are not regularly sampled and so we can extrapolate from the data point at time yeah times not regular samplings like or with RNNs you basically forced to have always the same time step difference otherwise you have a very tough time but with this since these are continuous flows you're basically you can basically unroll them and evaluate them at whatever time you want so they have pretty cool experiments here where they kind of try to learn these kind of spiraling behaviors and you see on top the RNN decoder will get all jaggy and so on where as the so so basically as the the neural ordinary differential equation will generate quite let's say smooth things and also it can extrapolate as you can see here it can it can go the red the red thing is the extrapolation only there's only data where the green dots are so that's pretty cool you can see the RNN sometimes isn't able to kind of continue the flow as you can see in here it extrapolates wrongly so the this kind of I mean it's toy it's a toy example but these kind of dynamics are pretty cool and they also show here when they learn the spirals and vary one dimension of the latent code that is given by the encoder then the flow goes from clockwise it goes from to to counter clockwise as you see here I've turned this in I've drawn this in wrong but so it's pretty pretty cool what these these things learn and since it's only small data right now small models but I'm pretty sure this is going to develop further and be a cool just a cool way cool alley of research cool idea and looking forward to what they come up next alright so that was it for today a bit shorter but I hope this was somewhat clear enough all right have a great day
 Hello and welcome. Today we're going to look at Neural Ordinary Differential
 Equations by Rick Chen, Julia Rubinova, Jesse Bettencourt and David Dovenoe.
 This has been quite an interesting kind of paper to see because it's a bit special.
 We're going to go over parts of it, not the full paper, just kind of the
 important parts because the paper is quite packed and we'd rather
 explain it in parts and kind of get the gist of it. So basically what they do is
 they say we introduce a new family of deep neural network models. Instead of
 specifying a discrete sequence of hidden layers we parameterize the
 derivative of the hidden state using a neural network. The output of the network
 is computed using a black box differential equation solver. These
 continuous depth models have constant memory cost, adapt their evaluation
 strategy to each input, can explicitly trade numerical precision for speed.
 It sounds awesome, honestly. It sounds really cool and it sounds really new.
 Let's jump in. What they say is let's look at kind of classic neural
 networks, especially residual neural networks. What residual neural
 networks do is in each hidden layer they kind of have a representation.
 This is kind of their hidden representation at layer t. What they do
 is they then add something. If you don't know a recurrent neural network
 is where you have, let's say this is your hidden state ht, and in a classic neural
 network you would just have a weight matrix here, blah blah blah blah. You do a
 matrix multiplication to get ht plus 1. So to get the next kind of the next
 hidden state you do a matrix multiplication by a big weight matrix
 here w. In a residual neural network what you do is you have this
 weight matrix w, you multiply it to get delta ht plus 1 and you take ht and you
 add the two. You add ht and delta ht plus 1 to arrive at ht plus 1.
 That's a residual network. It basically doesn't learn the transformation to the
 next layer but it learns how is the next representation different from
 this representation. What do I need to add to this representation to get to
 the next representation? It's reasoned that for deep networks
 since each layer only does a little bit of transformation we should basically
 bias it towards keeping the representation the same and just kind of
 changing it a little bit. So this is the inherent bias, the identity
 transform. So that's a residual
 network. This here is characterized by f of kind of theta and ht. So this
 is kind of the this is what we called delta h. It's now called f. So this
 f would be the kind of neural network layer and theta would be the
 parameters of it. So the weight matrix in our case. They say okay what if you do
 many of those? So they say basically what this is is kind of a time
 process. It's kind of you have a state and the next state and the next state
 and you always learn how to go to the next state to the next state and so on.
 What if you go very deep and what if you look at this as a time process and
 kind of make these steps very small? Make these super small and basically
 what if you have many many infinitely many layers? I say well okay this
 then becomes a dynamic process. Basically an ordinary differential
 equation where I say okay my time is now continuous and I look at it as a
 linearization as a local linearization basically and I say okay I basically
 specify how to get from this time to the next instance of time. The next
 instant the next infinitesimally small instance of time by specifying this f
 and in the continuous case this is to say that the derivative of the hidden
 state is now parameterized by a neural network. So if you know what a
 differential equation is it has like a start
 state and then what you do is you specify how at each point in time
 so that's t at each point in time how does the gradient look so maybe the
 gradient looks like this and then what an ODE solver will do is the ODE solver
 will say okay the gradients we're gonna do an infinite small step in this
 direction and then it goes back to f. What's the gradient at this
 infinitely small step next in time and then f would say well the gradient is
 like this and then the ODE solver will go like okay I need to be a little bit
 flatter so I go here so what's the gradient at this time okay maybe it's up
 this I need to go up here so the ODE solver will kind of construct a curve
 and at each point it needs to look that whatever f says is the gradient is
 actually the gradient right if this is the gradient this is the gradient this
 is the gradient so that's that's kind of how an ODE works and that's they say
 okay you can actually look at residual networks here as being a discrete time
 analog to such an ODE so what we want to do is actually we want to specify we
 want to actually and this is the the crazy part right or the cool part is we
 want to do this for neural networks basically we simply specify an ODE and
 the start state here the start state is let's say if you want to build an MNIST
 classifier it's our it's our image right the start state is our MNIST image and
 we're simply training a neural network such that the ODE that the equation if
 you solve it the curve at the end will arrive at the correct class I mean
 that's that's I'm skipping a few parts here about dimensionalities and so on
 right because you need to keep in the same dimension but in essence they say
 here we start out with our input and we train the neural network to give us the
 correct gradients the correct derivatives of this curve at each point
 in time such that when you solve the ODE at the end point you are going to be at
 the correct label so that's this is the input to your task basically and
 this is the output right but instead of having a neural network go from input
 to output you have a neural network that parameterizes how you go from each step
 in time to the next one what's what's the gradient at each point in time
 that's that's the kind of gist of it and that's that's kind of really cool it's
 a really new approach alright so they give various advantages of this and so
 here is this demonstrated again right you are here this is your input and you
 want to go to the output and then the loss of the loss that you specify it can
 depend on kind of either on the output as in like an image classifier or it can
 depend on intermediate states this is it's kept general right so the way they
 go about it is they say well okay but so the neural network now specifies how to
 get from one step to the next right here and the neural network has parameters
 right so we we need to train this network such that the correct output is
 given to some input right we actually need to train it so we need to we need
 to some how way to train these parameters theta and they say okay we do
 gradient descent on theta like in a classic neural network but now we need
 it's not it's not so easy right it's not one pass through this function it's like
 infinitely many passes through this function until you arrive here and then
 if you basically need to somehow get a gradient with respect to these
 parameters here so they say this again the loss of the this is the loss of the
 end state right is the loss of the start state plus the the integral over time of
 this is derivative which is basically this curve and the curve is given by an
 ODE solver where we input all these things so we need gradients with respect
 to that how do we do that and they give away here of saying okay we could either
 kind of back propagate through the ODE solver but that would you know depend on
 the ODE solver and so on but there's another method there's called what's called the we
 need the what's called the adjoint so this is reverse mode differentiation of
 an ODE solution adjoint sensitivity method solves an augmented ODE
 backwards in time so basically what you need to do is you forward propagate you
 come here right and then what you can do is you can solve the second ODE so you
 can generate a second curve here this one and don't worry about these little
 jumps here you can solve the second curve and the second curve together with
 the first and second curve you can then compute the gradients you need right so
 the second curve is is basically simply something like the application of the
 chain rule to the continuous domain and you need to you need to adjust these
 jumps here only when your loss depends on intermediate states this is this is
 kind of the offset caused by including or not including the loss so let's dive
 a bit further into this adjoint state what's the red curve the red curve is
 called a and what's a a is a curve and this is the differential equation for it
 again we specify the curve a by specifying its start state and its
 derivative and from its start state and its derivative at each time the ODE
 solver is able to construct the curve entirely so a t it says here is del L to
 del ZT this means how does the loss depend on this ZT on the hidden state
 right how does the loss depend on the hidden state at time T so it doesn't
 even have to be any of these points here how does the loss depend on this hidden
 state here and in order to find that out you would need to go you would need to
 develop the the curve until here right and then calculate the loss and then
 back propagate through here but you can do this by calculating this adjoint
 thing so as you can see here is a demonstration it's an example right so
 the start state here is simply given by the loss how does the loss of this state
 how does the loss depend on this state well simply by plugging it into the into
 the loss equation right so your losses might be a cross entropy loss or
 something how does the loss do that depend on this state here well we go we
 go from this state that we already know and we know how in reverse time so
 backwards in time this sensitivity of the loss develops so we go and we
 develop this curve until here and we say aha this point influences this loss in
 this much basically right so so and if the loss explicitly depends on this
 point then we have to we have to calculate in this offset since this
 point here only depends on this time up till here and then it changes so there
 is there's a discontinuation but don't worry about that too much basically what
 we can do is we can calculate the curve in a forward pass curve and the loss in
 the forward pass then we can do a second pass backward again by an ODE solve to
 say how does the how does the loss depend on each one of the states here
 of the hidden states right so that's the second point but that's not all because
 we're ultimately not interested in the how the loss depends on the state where
 the we're interested in how the loss depends on these parameters that tell us
 how to get from one hidden state to the next but luckily we can then simply
 evaluate this integral that depends as you can see here on a and on Z we can
 evaluate this and get the gradients for the the parameters right so I also have
 to say the parameters are static so the parameters are given over the entire
 duration of this they're they're the same and it's simply what changes is
 time alright so this is how you can get this is how you can get gradients with
 respect to parameters and the cool thing is now you can train these you can
 actually train this neural network here that tells you how to go from one state
 to the next such that if you input the digit 2 as an image well you can output
 to I mean not exactly but that's that's the point right you can by by going
 through this motion by going through this od solve so that's I mean that's
 immensely cool they actually define how to do this here in one forward one kind
 of backward pass you can solve everything at the same time it's it's
 pretty cool and they evaluate their their net and they compare it with a
 different bunch of other nets and they interestingly show that so basically
 with an od solver you can never kind of tell how many evaluations it's going to
 do because it's going to get increasing like it's increasingly accurate over
 time so you let it run and maybe it's going to first generate a curve that's
 like something like this right and then it needs to say crap okay I need to go
 back and refine and then it maybe goes the curve like this and so on so it gets
 continually closer over time and for that it needs to kind of query it's like
 a query it needs to query this this F so you need to give it the function as an
 invaluable function and it goes and just okay I need to I need to know it here
 okay I got it from here okay I need to know it here okay I got it from oh no I
 didn't get it okay I need also need to know it here all right and so you can
 never know how much they will evaluate but you basically have a parameter to
 trade off accuracy and how much they evaluate that's what they show here so
 the the less error they want in their forward pass the more forward passes
 they have to do that's this curve the more forward passes they do the more
 time they have to invest right that's this curve but interestingly the more
 forward passes the time required for forward passes or the evaluations
 required for passes increases also the evaluation required for backward passes
 but not by much so that the backward passes require about half the amount of
 evaluations that's forward passes which is encouraging since the the backward
 passes don't go kind of overboard like if you had to back propagate through
 the operations of the ODE solver itself and they also show as your training epoch
 continues that the ODE solver requests more and more evaluations for so for the
 same epoch basically or the same samples within different epochs which
 means as it gets more accurate kind of needs to know more and more and more
 about the the samples basically about the test the training samples which is
 all basically showing that this kind of works yeah so they they kind of to get
 into normalizing flows which I don't want to get into here much because we
 haven't done a video on that yet we'll do one but they basically show that it's
 it's quite easy to do normalizing flows in a continuous fashion and the topic
 normalizing flows it's in itself pretty cool what they do at the end is they say
 okay what we can now do is we can actually take sequential data so now
 we've just talked about let's input one data point get out let's say a label or
 something which we can actually do sequential data and let's for example
 have an RNN encoder for our sequential data so here here these are data points
 right these are measurements like a blood pressure of a of a person and what
 we can do is we can do a variational autoencoder we've talked about this we
 can have an RNN encoder parameterize a distribution and then as a decoder have
 this ODE neural network and basically what that allows us to do is that allows
 us to deal with time steps that are not regularly sampled and so we can
 extrapolate from the data point at time yeah times not regular samplings like
 or with RNNs you basically forced to have always the same time step
 difference otherwise you have a very tough time but with this since these are
 continuous flows you're basically you can basically unroll them and evaluate
 them at whatever time you want so they have pretty cool experiments here where
 they kind of try to learn these kind of spiraling behaviors and you see on top
 the RNN decoder will get all jaggy and so on where as the so so basically as the
 the neural ordinary differential equation will generate quite let's say
 smooth things and also it can extrapolate as you can see here it can it
 can go the red the red thing is the extrapolation only there's only data
 where the green dots are so that's pretty cool you can see the RNN
 sometimes isn't able to kind of continue the flow as you can see in here it
 extrapolates wrongly so the this kind of I mean it's toy it's a toy example but
 these kind of dynamics are pretty cool and they also show here when they learn
 the spirals and vary one dimension of the latent code that is given by the
 encoder then the flow goes from clockwise it goes from to to counter
 clockwise as you see here I've turned this in I've drawn this in wrong but so
 it's pretty pretty cool what these these things learn and since it's only small
 data right now small models but I'm pretty sure this is going to develop
 further and be a cool just a cool way cool alley of research cool idea and
 looking forward to what they come up next alright so that was it for today a
 bit shorter but I hope this was somewhat clear enough all right have a great day