GP-ILQG: Data-driven Robust Optimal Control for
Uncertain Nonlinear Dynamical Systems

arXiv:1705.05344v1 [cs.RO] 15 May 2017

Gilwoo Lee

Siddhartha S. Srinivasa

Matthew T. Mason ∗

Abstract
As we aim to control complex systems, use of a simulator in model-based reinforcement learning is becoming more common. However, it has been challenging
to overcome the Reality Gap, which comes from nonlinear model bias and susceptibility to disturbance. To address these problems, we propose a novel algorithm that
combines data-driven system identification approach (Gaussian Process) with a
Differential-Dynamic-Programming-based robust optimal control method (Iterative
Linear Quadratic Control). Our algorithm uses the simulator’s model as the mean
function for a Gaussian Process and learns only the difference between the simulator’s prediction and actual observations, making it a natural hybrid of simulation
and real-world observation. We show that our approach quickly corrects incorrect
models, comes up with robust optimal controllers, and transfers its acquired model
knowledge to new tasks efficiently.

1

Introduction

As we aim to control more complex robotic systems autonomously, simulators are being more
frequently used for training in model-based reinforcement learning [7]. A simulator allows us to
explore various policies without damaging the robot, and is also capable of generating a large amount
of synthetic data with little cost and time.
However, we often observe that simulator-based policies perform poorly in real world, due to model
discrepancy between the simulation and the real world. This discrepancy arises from two fundamental
challenges: (1) system identification to match the simulation model with the real world requires
the exploration of a large state space at the risk of damaging the robot, and (2) even with good
system identification, there is still discrepancy due to the limitations of a simulator’s ability to render
real-world physics.
Stochastic optimal control algorithms attempt to partially address this issue by artificially injecting
noise into the simulation during training [5, 24], or by explicitly modeling multiplicative noise [22,
20].
If the task domain is predefined, exploration can be limited to task-specific trajectories, and system
identification can be coupled with trajectory optimization [21]. Some recent works have suggested
policy training with multiple models, which results in a policy robust to model variance [11, 3].
While these methods have shown some successful results, they are still limited by the expressiveness
of the simulator’s model. If the true model is outside of the simulator’s model space, little can be
guaranteed.
Thus, although these algorithms produce more robust policies, they fail to address the fundamental
issue: there is unknown but structured model discrepancy between the simulation and the real world.
In this paper, we propose a novel algorithm that addresses both model bias and multiplicative noise.
Our work is based on the following key insight:
∗

gilwool, siddh, mason@cs.cmu.edu. All authors are affiliated with Robotics Institute, Carnegie
Mellon University.

[Simulation]
Robust-ILQG
Policy π

Augmented model

[Real World]
Rollout

Model Learning
Data

Figure 1: GP-ILQG overview
Explicitly correcting model bias and incorporating the correction as well as our
uncertainty of the correction in optimal control enables lifelong learning of the
system and robust control under uncertainty.
Our algorithm iterates over simulation-based optimal control, real-world data collection, and model
learning, as illustrated in Figure 1. Starting from a potentially incorrect model given by the simulator,
we obtain a control policy, with which we collect data in the real world. This data feeds into model
learning, during which we correct model bias and estimate our uncertainty of the correction. Both the
correction and its uncertainty are incorporated into computing a robust optimal control policy, which
then gets used to collect more data.
Our approach improves any simulator beyond the scope of its model space to match real-world
observations and produces an optimal control policy robust to model uncertainty and multiplicative
noise. The improved simulator uses previous real-world observations to infer the true model when it
explores previously visited space, but when it encounters a new region, it relies on the simulator’s
original model. Due to this hybrid nature, our algorithm shows faster convergence to the optimal
policy than a pure data-driven approach [13] or a pure simulation-based approach. Moreover, as it
permanently improves the simulator, it shows even faster convergence in new tasks in similar task
domain.

2

Related Work

Most model-based reinforcement learning has both model learning (system identification) and policy
optimization components [7]. The data for a model comes either from real world or simulation, and
is combined to construct a model via nonlinear function approximators such as Locally Weighted
Regression [2], Gaussian Processes [17], or Neural Networks [12]. Once the model is built, a
typical policy gradient method computes the derivatives of the cost function with respect to control
parameters [4, 8].
If an analytic model is given, e.g., via equations of motion or as a simulator2 , one can use classical
optimal control techniques such as Differential Dynamic Programming (DDP) [6], which compute a
reference trajectory as well as linear feedback control law. For robustness, Iterative Linear Quadratic
Gaussian Control (ILQG) [22] or H-∞ Control [20] can be used to incorporate multiplicative noise.
Variants of [22] have been used to generate guiding policies for data-driven RL methods [8, 28].
Recently, there have been some attempts to combine DDP or ILQG with data-driven models by
replacing analytical models with locally linear models [10, 25] or nonlinear models [13, 15, 26, 14]
learned by Gaussian Processes or Neural Networks.
2

We consider black-box simulators as analytical models, as derivatives can be taken by finite differencing.

2

The goal of our algorithm is closely aligned with those of [27] and [1]. [27] has proposed a framework
in which the robot maintains two controllers, one for the simulator and another for the real world, and
aims to narrow the difference. [1] assumes a deterministic real world, constructs an optimal policy
based on the simulator’s deterministic model, and evaluates its performance in the real world, while
successively augmenting the simulator’s model with time-dependent corrections based on real-world
observations. Our method considers a stochastic system and the correction is not time-dependent.

3

Approach

We work with stochastic nonlinear dynamical systems, whose evolution is described by the stochastic
differential equation
dx = f (x, u) dt + F(x, u) dω
(1)
where the applied control u ∈ Rm transitions the state x ∈ Rn of the system according to a linear
sum of the system dynamics f ∈ Rn and state-dependent amplification F ∈ Rn×p of Brownian noise
increment dω ∈ Rp .
We assume that we have full knowledge of F and partial knowledge of f , e.g., from a simulator. We
represent f as the sum of simulated component fsim and unknown residual ,
f (x, u) = fsim (x, u) + (x, u).
Then, in a discrete setting with fixed step size δt, (1) can be rewritten as the following:
√
∆x = (fsim +)δt + F ξF δt
(2)
√
where ξF ∼ N (0, Ip×p ) and δt appears because the noise covariance increases linearly with time.
We omit x and u for clarity.
Nonzero model bias results in nonzero difference between the model’s prediction fsim δt and the
actual δx. From (2), the difference is equivalent to
√
∆x − fsim δt =  δt + F ξF δt
In expectation,
√
E[ δt] = E[∆x − fsim δt − F ξF δt] = E[∆x − fsim δt]
as the mean of ξF is 0.
In order to correct the nonzero model bias, we approximate  δt from data. From rollouts of
trajectories, we collect a set of {x, u, ∆x − fsim δt} tuples. Assuming that the the covariance of  δt
grows linearly with δt, we use a Gaussian Process (GP) [17], which estimates  δt as a Gaussian
distribution:

(x, u)δt ∼ N ˆ(x, u)δt, Σ(x, u)δt
whose derivation we provide in Section 3.1. Let Σ(x, u) be decomposed into G(x, u) G(x, u)> , with
G ∈ Rn×n . Now (2) can be rewritten as the following:
√
√
∆x = (fsim + ˆ) δt + F ξF δt + G ξG δt
(3)
where ξG ∼ N (0, In×n ) .
The original ILQG handles stochastic systems without uncertainty, but the above formulation makes
it straightforward to extend ILQG to incorporate uncertainty, which we refer to as Robust-ILQG (see
Section 3.2).
Our approach is summarized in Algorithm 1. Initially, with zero data and no policy, we start by using
ILQG3 to obtain a locally optimal policy for the suboptimal model provided by fsim . Then, with
rollouts of the computed policy, we compute the error between f and fsim , with which we train a GP
that estimates . We run Robust-ILQG again, but with the updated dynamics. This process is repeated
until the policy converges.
3

Robust-ILQG is equivalent to ILQG in the absence of .

3

Algorithm 1 GP-ILQG
Require: π, fsim , F, D
if D 6= ∅ then
[ˆ
, G] ← GP(D)
end if
if π == N IL then
π ← Robust-ILQG(fsim + ˆ, F, G)
end if
while ∆π > γ do
Collect {(x, u, δx − f δt)} with rollouts of π.
D ← D ∪ {(x, u, δx − f δt)}
[ˆ
, G] ← GP(D)
π ← Robust-ILQG(fsim + ˆ, F, G).
end while

150

100

Incorrect Model + GP
Incorrect Model
Target
Observation

50

0
−10

−5

0

5

10

15

20

Figure 2: Use of Gaussian Process to correct an incorrect model. Near previous observations,
the model’s prediction is corrected to match the target. When the test input is far from the prior
observations, predictions resort to the incorrect model. Shaded area indicates 95% confidence interval.

One major advantage of our approach is that it is straightforward to use the improved model for
a new task. Given observations learned from a previous task, Algorithm 1 starts with an estimate
of . Whenever the algorithm explores a previously visited state-control space, the learner corrects
any simulator error and provides smaller uncertainty covariance; in a new region, the algorithm
relies on the original model fsim with larger uncertainty covariance than the explored region. This
results in a more accurate model, and as Robust-ILQG takes into account uncertainty, it also results
in a more robust policy than those that rely only on the simulator or on a locally learned dynamics.
When using only a partially-correct simulated model, the policy generated is always limited by the
simulator’s accuracy; when using a locally learned dynamics, the robot knows very little outside
previously explored regions. Our algorithm, which combines both, our algorithm quickly outperforms
the simulator-only approaches and requires less data than the latter.
3.1

Gaussian Process with the Simulator as a Mean Function

Gaussian Process Regression [17] is a nonlinear regression technique that has been successfully used
in many model-based reinforcement learning approaches [4, 13]. A Gaussian Process is a stochastic
process in which any finite subset is jointly Gaussian. Given a set of inputs and corresponding
outputs {xi , yi }ni=1 the distribution of {yi }ni=1 is defined by the covariance given by a kernel function
k(xi , xj ).
4

We use a variant of Gaussian Processes that uses a nonzero mean function. With f : X → Y as the
mean function, the prediction for test input x becomes the following:
>
E[y] = f (x) + kxX
K −1 (Y − f (X))
>
var(y) = kxx − kxX
K −1 kxX

where kxX is the covariance between test input x and training set X, K is the covariance matrix of
among the elements of X. In this formulation, the GP provides a posterior distribution given f (x)
and observations.
Using the simulator as the mean function for a Gaussian Process allows us to combine the simulator
and real-world observations smoothly. Near previous observations, the simulator’s prediction is
corrected to match the target. When the test input is far from the previous observations, the predictions
resort to the simulator. See Figure 2 for an illustration.
e = [x> u> ]> to be the input and δx to be the
As we have both x and u as the input, we define x
output, and use fsim as the mean function. Then, the GP serves to correct the error, δx − fsim δt. We
use the ARD (Automatic Relevance Determination) squared exponential kernel function:
ej ) = σf2 exp(−(e
ej )> Λ−1 (e
ej ))
k(e
xi , x
xi − x
xi − x
where σf2 is the signal variance and Λ controls the characteristic length of each input dimension. These
hyperparameters are optimized to maximize log-likelihood of data using conventional optimization
methods. For multiple dimensions of Y , we train one GP per dimension and treat each dimension to
be independent.
3.2

Robust-ILQG

In order to address both uncertainty and stochasticity in computing an optimal policy, we propose a
new algorithm, which we refer to as Robust-Iterative Linear Quadratic Gaussian Control (RobustILQG). Robust-ILQG is an extension to the original ILQG [22], a variant of Differential Dynamic
Programming (DDP) for stochastic systems. It makes second-order approximation of the value
function to the second order and first-order approximation of the dynamics. The main difference
between ILQG and Robust-ILQG is that the latter takes into account uncertainty in dynamics, as we
will see in the following derivation.
We use a discretized version of a continuous system, with a fixed step size δt. From the equation
√
∆x = f (x, u)δt + F(x, u) δtξF
(4)
√
√
= (fsim + ˆ)δt + F ξF δt + G ξG δt
The state x0 at the next timestep can be written as

√
√
x0 = x + (fsim + ˆ)δt + F ξF δt + G ξG δt

Given a trajectory {x0 , u0 , · · · xT }, the total cost is given as


T
−1
X
J(x0 ) = E lT (xT ) +
l(xi , ui )

(5)

(6)

i=0

where lT is the final cost and l is the running cost. Our objective is to find a deterministic policy
π : X → U that minimizes this total cost.
We consider the local variation of value function with respect to change in x and u. The Bellman
equation gives us the following:


∗
0 0
0∗
V (x + δx) = l(x + δx, u + δu ) + E V (x + δx )
(7)
where V is the value function at a particular timestep and V 0 is the value function at the next timestep.
δx0∗ refers to the variation of the next state given an optimal control change δu∗ .
Analogous to ILQR which does not have stochastic components, we can analytically derive the
local optimal improvement δx∗ by taking first order approximation of dynamics and second order
approximation of the value function.
5

The local deviation of the next state from its nominal (deterministic) state can be written as the
following:
δx0 = Aδx + Bδu + CξF + DξG

(8)

where A, B linearizes the deterministic component of dynamics at the current time step, defined as
A = I + (fsimx + ˆx )δt
B = (fsimu + ˆu )δt
and C, D captures the deviation from nominal (deterministic) state caused by stochastic terms
√
C = (F + Fx ⊗δx + Fu ⊗δu) δt
√
D = (G + Gx ⊗δx + Gu ⊗δu) δt.
where ⊗ denotes tensor-vector multiplication as defined in Section 6 such that Fx ⊗δx, Gx ⊗δx are
matrices of size n × p and Fu ⊗δu, Gu ⊗δu are of size n × m.
Assuming that the variation of value at the next timestep can be written in a quadratic form,
1
V 0 (x0 + δx0 ) = s0 + s0> δx0 + δx0> S0 δx0
2

(9)

the local variation of cost-to-go at the current time step can be written as 4
1 >
1 >
>
0 0
0
V (x + δx) = l(x, u) + l>
x δx + δx lxx δx + lu δu + δu luu δu + E[V (x + δx )]
2
2
Note that, when taking expectation for s> δx0 , the stochasticity and uncertainty terms disappear as the
means of ξF and ξG are zero, while for the second-order terms such as δx0> (·)δx0 , their covariances
must be taken into account.5 .
Expanding above equation with the definitions of partial derivatives,
V (x + δx) = l + s0 + s0> (Aδx + Bδu) + lx δx + lu δu


1
> 0
>
>
+
(Aδx + Bδu) S (Aδx + Bδu) + δx lxx δx + δu luu δu
2

1
+ tr(CC> S0 ) + tr(DD> S0 ) δt
2

(10)

where the trace terms can be re-written as the following:
tr(CC> S0 ) = tr

X

(i)
(i)
(i) > 0
(F(i) + F(i)
+ F(i)
x + Fu )(F
x + Fu ) S



i

=

X

> 0 (i)
(F(i) + Fx(i) + F(i)
+ Fx(i) + F(i)
u ) S (F
u )

i

where the superscripts (i) denote the ith column of the corresponding matrix and subscripts denote
their partial derivatives.
4
5

We assume that lux is zero.
e.g., for a randomvariable ξ ∼ N (0, Σ), E[ξ > Sξ] = tr(ΣS)

6

Then, combining the like terms together, (10) becomes
1 X (i)> 0 (i) 1 X (j)> 0 (j)
= l + s0 + δt
F
S F + δt
G
S G
2
2
i
j
|
{z
}
q


X
X
(i)> 0 (i)
(j)> 0 (j)
δx
+ s0> A + l>
+
δt
F
S
F
+δt
G
S
G
x
x
x
i

j

|

{z

}

q>



X
X
1
(j)> 0 (j)
+ δx> A> S0 A + lxx + δt
Fx(i)> S0 F(i)
+δt
G
S
G
δx
x
x
x
2
i
j
|
{z
}
Q



+ s0> B + l>
u + δt

X

(i)
F(i)> S0 Fu
+δt

i

X

G(j)> S0 G(j)
u


δu

j

{z

|

}

g>



X
X
(j)> 0 (j)
0 (i)
+ δx> A> S0 B + δt
F(i)>
S
F
+δt
G
S
G
δu
x
u
x
u
i

|

j

{z

}

G>

X
X
1
(j)> 0 (j)
S Gu ) δu
Gu
S0 F(i)
F(i)>
+ δu> (B> S0 B + luu + δt
u +δt
u
2
j
i
|
{z
}
H

Minimizing this with respect to δu gives the optimal δu∗ :
δu∗ = −H−1 g − H−1 Gδx,
Plugging this δu∗ back into (10), we get a quadratic form of V (x + δx):
1
V (x + δx) = s + δx> s + δx> Sδx
2
with
S = Q − G> H−1 G
s = q − G> H−> g
1
s = q − g> H−> g
2
Note that, in the absence of uncertainty terms, this is equivalent to iLQG as introduced in [22], and
further, in the absence of both uncertainty and stochsaticity, this is equivelent to iLQR [9].
To make a local improvement of the nominal trajectory, we perform a backward pass to update the
(T )
(T )
value function and the optimal δu∗ for each timestep, starting with ST = lxx , sT = lx , sT = lT .
During a forward pass, the nominal trajectory is updated by applying δu = −αH−1 g − H−1 Gδx
to the deterministic part of the system. We use backtracking line-search to find α that minimizes the
total cost.
For a long horizon task, Model Predictive Contol can be used with Robust-ILQG, which is to run
Robust-ILQG for a short horizon repeatedly. We use this approach in our second experiment in
Section 4.2.

4

Experiments and Analysis

We consider two simulated tasks: cart-pole swing-up and quadrotor control. For each task, one set
of model parameters was used as the “simulator”, and another set of model parameters was used
7

as the “real-world.” Multiplicative noise was added to dynamics and Gaussian noise was added to
observation. We compare GP-ILQG’s performance with three optimal control algorithms: (1) ILQG
using the “real-world” model, (2) ILQG using the incorrect “simulator” model, (3) Probabilistic DDP
(PDDP) [13], which is a variant of ILQG that relies only on data. For both GP-ILQG and PDDP, the
same set of data and same GP implementation were used at each iteration.
As data gets large, computing Gaussian Process becomes computationally expensive, and thus
it is common to use a subset of the data for computing the mean and covariance [17]. In our
experiments, we keep all observations and uniformly sub-sample 300 data points at each iteration6 .
GP hyperparameters are optimized with the Gaussian Process for Machine Learning Toolbox [18].
If the learner’s prediction error for a validation set is higher than that of its previous learner, we
re-sample and re-train.
Incorrect Model

(b) Task 2 Cost

(a) Task 1 Cost

PDDP

(c) Task 1 U-Trajectory

Torque (N)

1

1
Normalized Cost

GP-ILQG

Correct Model

0.5

0.5

0

−20

0

0
0

10

20

Training Iterations

0

10

20

Training Iterations

0

50

100

Timesteps (0.04s/step)

Figure 3: Cartpole swing-up.

4.1

Cart-Pole Swing-Up

In the cart-pole problem, the state is defined as [x, ẋ, θ, θ̇], where x is the position of the cart along
the x−axis and θ is the angle of the pole from the vertical downright position. Control input is the
x-directional force (N).
The model parameters for the “simulator” and “real-world” model are given in Section 6. The
real-world model has a 30% longer pole.
We run two tasks in this experiment. In the first task, the initial state is [0, π/4, 0, 0]. Figure 3(a)
is the normalized cost for the first task. While both GP-ILQG and PDDP converges to the optimal
performance, GP-ILQG is converges much quickly, within the first 2 iterations.
The difference between GP-ILQG and PDDP is more noticeable in the second task (Figure 3(b)),
which starts from a different initial state [0, −π/4, 0, 0]. Both GP-ILQG and PDDP use the learner
used in the previous task, but the initial cost for PDDP is significantly higher than GP-ILQG. We
believe that this is because both algorithms explore an unexplored region in the first few iterations.
While GP-ILQG relies on the simulator’s inaccurate model in the unexplored region, PDDP has no
information to make meaningful advancement until enough data is collected.
What is more noticeable is the improved performance of GP-ILQG over the simulator-based ILQG.
The latter’s suboptimal policy results in significantly higher cost in general. Figure 3(c) shows the
control sequences generated by the final policies of the four algorithms. GP-ILQG’s control sequence
is almost identical to the optimal sequence, and PDDP closely follows the optimal sequence as well.
However, the simulator-based ILQG’s control sequence is quite different due to its incorrect model.
6

For better results, more advanced techniques such as Sparse Pseudo-input GP [19] or Sparse Spectral
Gaussian Process [16] can be used.

8

Incorrect Model
(b) Task 2 Cost

(a) Task 1 Cost

GP-ILQG

(c) Task 1 Z-Trajectory

1

1

5
Height (m)

Normalized Cost

Correct Model

0.5

0.5

0

0
0

5

4
3.5

0

10

4.5

5

10

Iterations

Iterations

0

100

200

Timesteps (0.02s/step)

Figure 4: Quadrotor control

4.2

Quadrotor

We use the quadrotor model introduced in [23]. The model has 12-dimensional state, x =
[p, v, r, w]> , where p (m) and v (m/s) refers to the quadrotor’s position and velocity in 3D space, r
is orientation (rotation about axis r by angle krk), and w (rad/s) is angular velocity. It has 4 control
inputs, u = [u1 , u2 , u3 , u4 ]> , which represent the force (N) exerted by the four rotors. The dynamics
is given as the following:
ṗ = v
v̇ = −ge3 + (

X

ui ) exp([r]e3 − kv v)/m

1
1
1
ṙ = w + [r]w + (1 − krk/ tan( krk)[r]2 /krk2
2
2
2
ẇ = J −1 (ρ(u2 − u4 )e1 + ρ(u3 − u1 )e2
+ km (u1 − u2 + u3 − u4 )e3 − [w]Jw)
where ei are the standard basis vectors, g = 9.8m/s2 is gravity, kv is the drag coefficient of rotors,
m (kg) is the mass, J (kg m2 ) is the moment of inertia matrix, and ρ (m) is the distance between the
center of mass and the center of rotors, and km is a constant relating the force of rotors to its torque.
[·] refers to the skew-symmetric cross product. The model parameters used are included in Section 6.
The real-world model is 40% heavier than the simulator’s model.
We evaluate the performance of two tasks. The quadrotor starts at a position near (0m, 0m, 5m)
with zero velocity. In the first task, the goal is to move the quadrotor forward to (4m, 0m, 5m) in 4
seconds. The second task is to drive the quadrotor to (2m, 1m, 7m) in 4 seconds. The cost function
was set to track a straight line from the initial position to the goal, with higher cost for maintaining
the height.
In this experiment, we were not able to run PDDP to convergence with the same data used in GPILQG. We believe that this arises from the same problem we saw in the second task of cart-pole:
PDDP has insufficient data to infer the unexplored state-space. We note that the original PDDP
algorithm requires random trajectories as its initial data set instead of random variations of a single
nominal trajectory. While our experiment does not indicate that PDDP is incapable of this task7 , it
highlights our algorithm’s data efficiency. Even with the small set of task-specific data, GP-ILQG
converges in the first two iterations in the initial task (Figure 4(a) and converges immediately to the
optimal policy in the second task (Figure 4(b)). Figure 4(c) compares the trajectories generated by the
three algorithms. It shows that while our algorithm closely tracks the desired height, the simulator’s
suboptimal controller fails to recover from the vertical drop due to its incorrect mass model.
7

A similar experiment with quadrotor control was shown to be successful in [15].

9

5

Conclusion

In this paper, we proposed a novel algorithm that combines real-world data with a simulator’s model to
improve real-world performance of simulation-based optimal control. Our approach uses a Gaussian
Process to correct a simulator’s nonlinear model bias beyond the scope of its model space while
incorporating the uncertainty of our estimate in computing a robust optimal control policy. Through
simulated experiments, we have shown that our approach converges to the optimal performance
within a few iterations and is capable of generalizing the learned dynamics for new tasks.
Although our algorithm is capable of correcting significant model errors, it is limited by the quality
of the initial policy based on the simulator’s incorrect model. For example, the simulator’s model
can be sufficiently different from the true model such that the initial policy results in catastrophic
damage to the robot. Our algorithm is incapable of measuring this initial uncertainty, although it can
be improved by providing an initial set of expert-generated trajectories. We leave this as a future
research direction.

6

Appendix

6.1

Stochasticity and uncertainty terms

We define the partial derivatives for stochasticity and uncertainty terms as the following:


(p)
Fx ⊗δx , F(1)
x δx, ..., Fx δx


(n)
Gx ⊗δx , G(1)
x δx, ..., Gx δx

(11)
(12)

(j)
where F(i)
x , Gx refer to the partial derivatives of i, j-th columns of F and G with respect to x.

6.2

Models used in experiments

We have used the following model parameters for the cart-pole and quadrotor experiments. “Simulator”
refers to the incorrect model in the simulator. “Real World” is the model used during rollouts and
policy evaluation; its model parameters are not revealed to our algorithm.
6.2.1

Cartpole

Cart Mass
Pole Mass
Pole Length
6.2.2

Simulator

Real World

1 kg
1 kg
1m

1 kg
1 kg
1.3 m

Quadrotor

kv is a constant relating the velocity to an opposite force, caused by rotor drag and induced inflow. m
(kg) is the mass, J (kg m2 ) is the moment of inertia matrix, ρ (m) is the distance between the center
of mass and the center of the rotors.

kv
km
m
J
ρ

Simulator

Real World

0.15
0.025
0.5
0.05 I
0.17

0.15
0.025
0.7
0.05 I
0.17

References
[1] P. Abbeel, M. Quigley, and A. Y. Ng. Using inaccurate models in reinforcement learning. In
W. W. Cohen and A. Moore, editors, Proceedings of the 23th International Conference on
10

Machine Learning (ICML-06), pages 1–8, 2006. URL http://www.machinelearning.org/
proceedings/icml2006/001_Using_Inaccurate_Mod.pdf.
[2] C. G. Atkeson, A. W. Moorey, and S. Schaalz. Locally weighted learning. Artif Intell Rev, 11
(1-5):11–73, 1997.
[3] A. Boeing and T. Bräunl. Leveraging multiple simulators for crossing the reality gap. In
Control Automation Robotics & Vision (ICARCV), 2012 12th International Conference on,
pages 1113–1119. IEEE, 2012.
[4] M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to policy
search. In Proceedings of the 28th International Conference on machine learning (ICML-11),
pages 465–472, 2011.
[5] D. Huh and E. Todorov. Real-time motor control using recurrent neural networks. In Adaptive
Dynamic Programming and Reinforcement Learning, 2009. ADPRL’09. IEEE Symposium on,
pages 42–49. IEEE, 2009.
[6] D. H. Jacobson and D. Q. Mayne. Differential dynamic programming. 1970.
[7] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The
International Journal of Robotics Research, 32(11):1238–1274, 2013.
[8] S. Levine and V. Koltun. Guided policy search. In ICML (3), pages 1–9, 2013.
[9] W. Li and E. Todorov. Iterative linear quadratic regulator design for nonlinear biological
movement systems.
[10] D. Mitrovic, S. Klanke, and S. Vijayakumar. Adaptive optimal feedback control with learned
internal dynamics models. In From Motor Learning to Interaction Learning in Robots, pages
65–84. Springer, 2010.
[11] I. Mordatch, N. Mishra, C. Eppner, and P. Abbeel. Combining model-based policy search with
online model learning for control of physical humanoids. In Robotics and Automation (ICRA),
2016 IEEE International Conference on, pages 242–248. IEEE, 2016.
[12] K. S. Narendra and K. Parthasarathy. Identification and control of dynamical systems using
neural networks. IEEE Transactions on neural networks, 1(1):4–27, 1990.
[13] Y. Pan and E. Theodorou. Probabilistic differential dynamic programming. In Advances in
Neural Information Processing Systems, pages 1907–1915, 2014.
[14] Y. Pan and E. A. Theodorou. Data-driven differential dynamic programming using gaussian
processes. In American Control Conference (ACC), 2015, pages 4467–4472. IEEE, 2015.
[15] Y. Pan, X. Yan, E. Theodorou, and B. Boots. Scalable reinforcement learning via trajectory
optimization and approximate gaussian process regression. NIPS Workshop on Advances in
Approximate Bayesian Inference, 2015.
[16] J. Quiñonero-Candela, C. E. Rasmussen, A. R. Figueiras-Vidal, et al. Sparse spectrum gaussian
process regression. Journal of Machine Learning Research, 11(Jun):1865–1881, 2010.
[17] C. E. Rasmussen. Gaussian processes for machine learning. 2006.
[18] C. E. Rasmussen and H. Nickisch. Gaussian processes for machine learning (gpml) toolbox.
Journal of Machine Learning Research, 11(Nov):3011–3015, 2010.
[19] E. Snelson and Z. Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances in
neural information processing systems, 18:1257, 2006.
[20] A. A. Stoorvogel. The h-infinity control problem: A state space approach. 1993.
[21] J. Tan, Z. Xie, B. Boots, and C. K. Liu. Simulation-based design of dynamic controllers for
humanoid balancing. In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International
Conference on, pages 2729–2736. IEEE, 2016.
[22] E. Todorov and W. Li. A generalized iterative lqg method for locally-optimal feedback control of
constrained nonlinear stochastic systems. In American Control Conference, 2005. Proceedings
of the 2005, pages 300–306. IEEE, 2005.
[23] J. van den Berg. Extended lqr: Locally-optimal feedback control for systems with non-linear
dynamics and non-quadratic cost. In Robotics Research, pages 39–56. Springer, 2016.
11

[24] J. M. Wang, D. J. Fleet, and A. Hertzmann. Optimizing walking controllers for uncertain inputs
and environments. In ACM Transactions on Graphics (TOG), volume 29, page 73. ACM, 2010.
[25] A. Yamaguchi and C. G. Atkeson. Differential dynamic programming with temporally decomposed dynamics. In Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International
Conference on, pages 696–703. IEEE, 2015.
[26] A. Yamaguchi and C. G. Atkeson. Neural networks and differential dynamic programming for
reinforcement learning problems. In Robotics and Automation (ICRA), 2016 IEEE International
Conference on, pages 5434–5441. IEEE, 2016.
[27] J. C. Zagal, J. Ruiz-del Solar, and P. Vallejos. Back to reality: Crossing the reality gap in
evolutionary robotics. In IAV 2004 the 5th IFAC Symposium on Intelligent Autonomous Vehicles,
Lisbon, Portugal, 2004.
[28] T. Zhang, G. Kahn, S. Levine, and P. Abbeel. Learning deep control policies for autonomous
aerial vehicles with mpc-guided policy search. In Robotics and Automation (ICRA), 2016 IEEE
International Conference on, pages 528–535. IEEE, 2016.

12

