\section{Introduction}
Solutions of general relativistic Liouville's equation ($grl$) in
a prescribed space-time
have been considered by some investigators.
Most authors have sought its solutions as functions of the
constants of motion, generated by Killing vectors of the space-time in
question.   See for example Ehlers (1971), Ray and
Zimmerman (1977),
Mansouri and Rakei (1988), Ellis, Matraverse and Treciokas (1983), Maharaj and
Maartens (1985, 1987), Maharaj (1989), and Dehghani and Rezania (1996).

In applications to self gravitating stars and stellar systems, however, one
should combine Einstein's
field equations and $grl$.   The resulting nonlinear equations can
be solved in certain approximations.
Two such methods are available;
 the {\it post-Newtonian (pn) approximation} and the {\it weak-field
} one.
In this paper we adopt the first approach to study a self gravitating
system imbeded in an otherwise flat space-time.
In sect. 2, we derive the $grl$ in the $pn$ approximation.
In sect. 3 we seek static solutions of
the post-Newtonian Liouville's
equation ($pnl$).     We find two integrals of $pnl$ that are the
$pn$
generalizations of the energy and angular momentum integrals of the
classical Liouville's equation.
Post-Newtonian polytropes, as simultaneous solutions of $pnl$ and Einsteins
equation,
are discussed and calculated in sect.
 4.     Section 5 is devoted to concluding remarks.

The main objective of this paper, however, is to set the stage for the second
paper in this series (Sobouti and Rezania, 1998).     There, we study a class
of
non static oscillatory solutions of $pnl$, different from the conventional $p$
and $g$ modes of the system.      They are associated with oscillations in
the space-time metric, without disturbing the classical equilibrium of the
system.    In this respect they might be akin to the so called {\it
gravitational wave} modes that some authors have advocated to exist in
relativistic systems.     See, for example,
Andersson, Kokkotas and Schutz (1995), Baumgarte and Schmidt (1993),
Detweiler and Lindblom (1983, 1985),
Kojima (1988),
Kokkotas and Schutz (1986, 1992), Leaver (1985),
Leins, Nollert and Soffel (1993),
Lindblom, Mendell and Ipser (1997), and
Nollert and Schmidt (1992).
\section{Liouville's equation in post-Newtonian
approximation}
The one particle  distribution function of a gas of collisionless particles
with identical masse $m$, in the restricted seven dimensional phase space
\stepcounter{sub}
\begin{equation}
P(m):\;\;g_{\mu \nu} U^\mu U^\nu = -1
\end{equation}
satisfies $grl$:
\stepcounter{sub}
\begin{equation}
{\cal L}_Uf = (U^\mu \frac{\partial}{\partial x^\mu} - \Gamma_{\mu\nu}^i U^\mu
U^\nu \frac{\partial}{\partial U^i}) f(x^\mu,U^i) = 0,
\end{equation}
where $(x^\mu,U^i)$ is the set of configuration and velocity coordinates in
 $P(m)$, $f(x^\mu,U^i)$ is a distribution function,
${\cal L}_U$
is Liouville's operator in the $(x^\mu,U^i)$ coordinates, and $\Gamma^i
_{\mu\nu}$ are Christoffel's symbols.
Greek indices run from 0 to 3 and Latin indices from
1 to 3.   We use the convention
$c = 1$ except in numerical calculations of section 4.
  The four-velocity of the particle and
its classical velocity are related as
\stepcounter{sub}
\begin{equation}
U^\mu = U^0 v^\mu;\;\;\;\;  v^\mu = (1, v^i = dx^i/dt),
\end{equation}
where $U^0(x^\mu,v^i)$ is to be determined from Eq. (1).
In the $pn$ approximation, we need an expansion of
${\cal L}_U$ up to order $\bar{v}^4$, where $\bar{v}$ is a typical
Newtonian speed.   To achieve this goal we transform
$(x^\mu,U^i)$ to $(x^\mu,v^i)$.
Liouville's operator transforms as
\stepcounter{sub}
\begin{equation}
{\cal L}_U = U^0 v^\mu (\frac{\partial}{\partial x^\mu} + \frac{\partial v^j}
{\partial x^\mu} \frac{\partial}{\partial v^j}) - \Gamma^i _{\mu\nu}U^{0^2}
v^\mu v^\nu \frac{\partial v^j}{\partial U^i} \frac{\partial}{\partial v^j},
\end{equation}
where  $\partial v^j/ \partial x^\mu$ and $\partial v^j/ \partial U^i$ are
determined
from the inverse of the transformation matrix (see appendix A).
Thus,
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\frac{\partial v^j}{\partial x^\mu} = - \frac{U^0}{2Q} v^j
\frac{\partial g_{\alpha \beta}}{\partial x^\mu} v^\alpha v^\beta,\\
\stepcounter{subeqn}
&&\frac{\partial v^j}{\partial U^i} =
\;\; \frac{1}{Q} v^j (g_{0i} + g_{ik} v^k);\hspace{2cm}\;\;\mbox{for $i
\neq j$},\nonumber\\
&&\\
&&\hspace{.81cm}=-\frac{1}{Q} (U^{0^{-2}} + \sum_{k \neq i} v^k (g_{0 k}
+ g_{kl} v^l));\;\;  \mbox{for $i = j$},\nonumber
\end{eqnarray}
where
\stepcounter{subeqn}
\begin{equation}
Q = U^0 (g_{0 0} + g_{0 l} v^l).
\end{equation}
Using Eqs.(5) in Eq.(4), one finds
\stepcounter{sub}
\stepcounter{subeqn}
\begin{equation}
{\cal L}_U f =U^{0} {\cal L}_v f=0,
\end{equation}
or
\stepcounter{subeqn}
\begin{equation}
{\cal L}_v f(x^\mu,v^i) = 0,
\end{equation}
where
\stepcounter{subeqn}
\begin{eqnarray}
&&{\cal L}_v= v^\mu (\frac{\partial}{\partial x^\mu} - \frac{U^0}{2Q} v^j
\frac{\partial g_{\alpha \beta}}{\partial x^\mu} v^\alpha
v^\beta\frac{\partial}{\partial v^j})
- \Gamma^i _{\mu\nu}v^\mu v^\nu\{\sum_{j\neq i} \frac{1}{Q} v^j (g_{0i}
+ g_{ik} v^k)  \frac{\partial}{\partial v^j}  \nonumber \\
&&\hspace{4cm} - \frac{1}{Q} (U^{0^{-2}} + \sum_{k \neq i} v^k (g_{0 k}
+ g_{kl} v^l)) \frac{\partial}{\partial v^i}   \},
\end{eqnarray}
We note that the post-Newtonian hydrodynamic equations are obtained from
integrations
of Eq. (6a) over the ${\bf v}$-space rather than Eq. (6b) (see appendix B).
We expand
${\cal L}_v$ up to the order $\bar{v}^4$.  For this purpose, we need
expansions of Einstein's field equations, the metric tensor, and the affine
connections up to various orders.
Einstein's field equation with harmonic coordinates
condition, $g^{\mu\nu}\Gamma^{\lambda}_{\mu\nu}=0$, yields (see
Weinberg 1972):
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\nabla^2\; ^2g_{00} = - 8\pi G\; ^0T^{00},\\
&&\nabla^2\; ^4g_{00} = \frac{\partial^2\; ^2g_{00}}{\partial t^2} +
\;^2g_{ij} \frac{\partial^2\; ^2g_{00}}{\partial x^i \partial x^j} -
(\frac{\partial\; ^2g_{00}}{\partial x^i})(\frac{\partial\; ^2g_{00}}
{\partial x^i})\nonumber\\
\stepcounter{subeqn}
&&\hspace{2cm}- 8\pi G (\;^2T^{00} - 2\; ^2g_{00}\; ^0T^{00} +\;
^2T^{ii}),\\
\stepcounter{subeqn}
&&\nabla^2\; ^3g_{i 0} = 16 \pi G\; ^1T^{i0},\\
\stepcounter{subeqn}
&&\nabla^2\; ^2g_{ij} = - 8 \pi G \delta_{ij}\; ^0T^{00}.
\end{eqnarray}
The symbols $^ng_{\mu\nu}$ and $^nT^{\mu\nu}$ denote the $n$th order terms in
${\bar v}$ in
the metric and in the energy-momentum tensors, respectively.    Solutions of
these equations are
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&^2g_{00} = - 2 \phi,\\
\stepcounter{subeqn}
&&^2g_{ij} = - 2 \delta_{ij} \phi, \\
\stepcounter{subeqn}
&&^3g_{i 0} = \eta_i,               \\
\stepcounter{subeqn}
&&^4g_{00} = - 2 \phi^2 - 2 \psi,
\end{eqnarray}
where
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\phi({\bf x},t) = -G \int \frac{^0T^{00}({\bf x}',t)}{\vert{\bf x} -
{\bf x}' \vert} d^3 x', \\
\stepcounter{subeqn}
&&\eta^i ({\bf x},t) = -4 G \int \frac{^1T^{i 0} ({\bf x}',t)}{\vert {\bf x} -
{\bf x}' \vert} d^3 x',\\
\stepcounter{subeqn}
&&\psi({\bf x},t) = - \int \frac{d^3 x'}{\vert {\bf x} - {\bf x}' \vert}
(\frac{1}{4\pi} \frac{\partial^2 \phi({\bf x'},t)}{\partial t^2}
+ G\; ^2T^{00}({\bf x}',t)\nonumber\\
\stepcounter{subeqn}
&&\hspace{6cm}+G\;^2T^{ii}({\bf x'}, t)),
\end{eqnarray}
where bold characters denote the three vectors.
Substituting Eqs. (8) and (9) in (6c) gives
\stepcounter{sub}
\begin{eqnarray}
{\cal L}_v &= &{\cal L}^{cl}+{\cal L}^{pn} \nonumber\\
                  & =&\frac{\partial}{\partial t} + v^i
\frac{\partial}{\partial x^i} - \frac{\partial \phi}{\partial x^i}
\frac{\partial}{\partial v^i} \nonumber\\
                &&- [(4\phi + {\bf v}^2) \frac{\partial \phi}{\partial x^i} -
\frac{\partial \phi}{\partial x^j} v^i v^j -  v^i \frac{\partial
\phi}{\partial t} + \frac{\partial \psi}{\partial x^i} \nonumber\\
                  &&+ (\frac{\partial \eta_i}{\partial x^j} -  \frac{\partial
\eta_j}{\partial x^i}) v^j+\frac{\partial\eta_i}{\partial t}]
\frac{\partial}{\partial v^i}
\end{eqnarray}
where ${\cal L}^{cl}$ and ${\cal L}^{pn}$ are the
classical Liouville
operator and its post-Newtonian correction,
respectively.      Equation (6b) for the
distribution function $f(x^\mu,v^i)$ becomes
\stepcounter{sub}
\begin{equation}
({\cal L}^{cl}+{\cal L}^{pn}) f(x^\mu,v^i) = 0.
\end{equation}
The three scalar and vector potentials $\phi,\psi$ and
$\eta\hspace{-.2cm}\eta\hspace{-.2cm}\eta$
can now be given in terms of the distribution function.
    The energy-momentum tensor in terms of $f(x^\mu, U^i)$ is
\stepcounter{sub}
\begin{equation}
T^{\mu\nu}(x^\lambda)=\int \frac{U^\mu U^\nu}{U^0} f(x^\lambda, U^i)
\sqrt{-g}d^3U,
\end{equation}
where $g=det(g_{\mu\nu})$.     For various orders of $T^{\mu\nu}$
 one finds
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&^0T^{00}(x^\lambda) = \int  f(x^\lambda,v^i) d^3 v, \\
\stepcounter{subeqn}
&&^2T^{00}(x^\lambda) = \int (\frac{1}{2}v^2 + \phi(x^\lambda))
f(x^\lambda,v^i) d^3 v,\\
\stepcounter{subeqn}
&&^2T^{ij}(x^\lambda)= \int v^i v^j f(x^\lambda,v^i) d^3 v, \\
\stepcounter{subeqn}
&&^1T^{0i}(x^\lambda) = \int v^i f(x^\lambda,v^i) d^3 v.
\end{eqnarray}
Substituting Eqs. (13) in (9) gives
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\phi({\bf x},t) =-G \int \frac{f({\bf x'},t,{\bf v')}}{\vert {\bf
x} - {\bf x'} \vert} d \Gamma',\\
\stepcounter{subeqn}
&&\eta\hspace{-.2cm}\eta\hspace{-.2cm}\eta ({\bf x},t) = -4G \int \frac{{\bf
v'} f({\bf x'},t, {\bf v')}}{\vert {\bf x} - {\bf x'} \vert} d\Gamma'\\
&&\psi({\bf x},t) = \frac{G}{4 \pi} \int \frac{\partial^2
f({\bf x''},t,{\bf v''})/\partial t^2 }{
\vert {\bf x} - {\bf x'} \vert \vert  {\bf
x'} - {\bf x''} \vert} d^3x'd\Gamma'' \nonumber\\
&&\hspace{2cm}- \frac{3}{2}G \int \frac{{\bf v'}^2 f({\bf x'},t, {\bf
v'})}{\vert {\bf x} - {\bf x'} \vert} d \Gamma'\nonumber\\
\stepcounter{subeqn}
&&\hspace{2cm}+ G^2 \int \frac{f({\bf x'},t,{\bf v'}) f({\bf
x''},t,{\bf v''})}{\vert {\bf x} - {\bf x'}
\vert \vert {\bf x'} - {\bf x''} \vert} d \Gamma' d
\Gamma'',
\end{eqnarray}
where $d\Gamma=d^3xd^3v$.      Equations (11) and (14) complete the
$pn$ order of Liouville's equation for self gravitating systems imbeded in
a flat space-time.
\section{Integrals of post-Newtonian Liouville's equation}
In an equilibrium state $f({\bf x}, {\bf v})$ is time-independent.
Macroscopic velocities along with the vector potential
$\eta\hspace{-.2cm}\eta\hspace{-.2cm}\eta$ vanish.       Equations (10)
and (11) reduce to
\stepcounter{sub}
\begin{eqnarray}
&&({\cal L}^{cl}+{\cal L}^{pn})f({\bf x},{\bf v})=[(v^i\frac{\partial}
{\partial x^i}-
\frac{\partial\phi}{\partial x^i}\frac{\partial}{\partial v^i})\nonumber\\
&&\hspace{3cm}-(\frac{\partial\phi}{\partial x^i}(4\phi+v^2)-
\frac{\partial\phi}{\partial x^j}v^iv^j+\frac{\partial\psi}{\partial x^i})
\frac{\partial}{\partial v^i}]f=0,
\end{eqnarray}
One easily verifies that the following, a generalization of the classical
energy integral, is a solution of Eq. (15)
\stepcounter{sub}
\begin{eqnarray}
&&E=\frac{1}{2}v^2+\phi+2\phi^2+\psi + const.
\end{eqnarray}
Furthermore, if $\phi({\bf x})$ and $\psi({\bf x})$ are spherically symmetric,
which actually is the
case for an isolated system in an asymptotically flat space-time,
the following generalization of angular momentum are also integrals of
Eq. (15)
\stepcounter{sub}
\begin{eqnarray}
&& l_i=\varepsilon_{ijk}x^jv^kexp(-\phi),
\end{eqnarray}
where $\varepsilon_{ijk}$ is the Levi-Cevita symbol.      Static distribution
functions maybe constructed as functions of $E$ and even functions of
$l_i$.       The reason for restriction
to even functions of $l^{pn}_i$ is to insure the vanishing of $\eta^i$,
the condition for validity of Eq. (15).
\section{Polytropes in post-Newtonian approximation}
As in classical polytropes we
consider the distribution
function for a polytrope of index $n$ as
\stepcounter{sub}
\begin{eqnarray}
&&F_n(E)=\frac{\alpha_n}{4\pi\sqrt{2}}(-E)^{n-3/2};\;\;
\mbox{for}\;\; E< 0, \nonumber\\
&&\hspace{1.3cm}=0\hspace{3.00cm}\mbox{for}\;\; E> 0,
\end{eqnarray}
where $\alpha_n$ is a constant.
By Eqs. (13) the corresponding orders of the energy-momentum tensor are
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&^0T^{00}_n=\alpha_n\beta_n(-U)^n,\\
\stepcounter{subeqn}
&&^2T^{00}_n=\alpha_n\beta_n\phi (-U)^n +\alpha_n\gamma_n
(-U)^{n+1},\\ \stepcounter{subeqn}
&&^2T^{ii}_n\;\;=\delta_{ij}\;^2T^{ij}=2\alpha_n\gamma_n (-U)^{n+1},\\
\stepcounter{subeqn}
&&^1T^{0i}_n=0,
\end{eqnarray}
where
\stepcounter{sub}
\begin{eqnarray}
&&\beta_n=\int^1_0(1-\eta)^{n-3/2}\eta^{1/2}d\eta=\Gamma(3/2)\Gamma(n-1
/2)/\Gamma(n+1),\\
\stepcounter{sub}
&&\gamma_n=\int^1_0(1-\eta)^{n-3/2}\eta^{3/2}d\eta=\Gamma(5/2)\Gamma(n-1/
2)/\Gamma(n+2),
\end{eqnarray}
and $U=\phi+2\phi^2+\psi$ is the gravitational potential in $pn$ order.
It will be chosen zero at the
surface of the stellar configuration.     With this choice, the escape velocity
$v_e=\sqrt{-2U}$ will mean escape to the boundary of the system rather
than to infinity.   Einstein's equations, Eqs. (7), (8) and (9), lead to
\stepcounter{sub}
\begin{eqnarray}
&&\nabla^2\phi=4\pi G\; ^0T^{00}=
4\pi G\alpha_n\beta_n(-U)^n,\\
\stepcounter{sub}
&&\nabla^2\psi=4\pi G (^2T^{00}+^2T^{ii})
=4\pi G\alpha_n\beta_n\phi (-U)^n\nonumber\\
&&\hspace{1cm}+12\pi G\alpha_n\gamma_n(-U)^{n+1}.
\end{eqnarray}
Expanding $(-U)^n$ as
\stepcounter{sub}
\begin{equation}
(-U)^n=(-\phi)^n[1+n(2\phi+\frac{\psi}{\phi})+\cdots],
\end{equation}
ans Substituting it in Eqs. (22) and (23) gives
\stepcounter{sub}
\begin{eqnarray}
&&\nabla^2\phi=4\pi
G\alpha_n\beta_n[(-\phi)^n-2n(-\phi)^{n+1}-n(-\phi)^{n-1}\psi+\cdots],\\
\stepcounter{sub}
&&\nabla^2\psi=4\pi
G\alpha_n\beta_n
\{(3\frac{\gamma_n}{\beta_n}-1) (-\phi)^{n+1} \nonumber\\
&&\hspace{3cm}- [3(n+1)\frac{\gamma_n}{\beta_n}-n] [2(-\phi)^{n+2}+
(-\phi)^n\psi]+\cdots\},
\end{eqnarray}
These equations can be solved numerically by an iterative scheme.    We
introduce three
dimesionless quantities
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&-\phi\equiv \lambda \theta,\\
\stepcounter{subeqn}
&&-\psi\equiv \lambda^2 \Theta,\\
\stepcounter{subeqn}
&&\hspace{.35cm}r\equiv a\zeta,
\end{eqnarray}
where, in terms of $\rho_c$, the central density,
$\lambda=(\rho_c/\alpha_n\beta_n)^{1/n}$ and
$a^{-2}=4\pi G\rho_c/\lambda$.      Equations
(25) and (26) in various iteration orders reduce to
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\nabla_{\zeta}^2\theta_o + \theta_o^n=0,\\
\stepcounter{subeqn}
&&\nabla_{\zeta}^2\Theta_o +(3\frac{\gamma_n}{\beta_n}-1) \theta_o^{n+1}=0,\\
\stepcounter{subeqn}
&&\nabla_{\zeta}^2\theta_1+\theta_1^n = qn(2\theta_o^{n+1} - \theta_o^{n-1}
\Theta_o),\\
&&\nabla_{\zeta}^2\Theta_1+(3\frac{\gamma_n}{\beta_n}-1) \theta_1^{n+1}=
\nonumber\\
\stepcounter{subeqn}
&&\hspace{2cm}q [3(n+1)\frac{\gamma_n}{\beta_n}-n] (2\theta_o^{n+2}-
\theta_o^n\Theta_o),
\end{eqnarray}
where
$\nabla_{\zeta}^2= \frac{1}{\zeta^2}
\frac{d}{d\zeta}(\zeta^2\frac{d}{d\zeta})$.
The subscripts 0 and 1 refer to orders of the iteration.   The
dimensionless parameter $q$ is defined as
\stepcounter{sub}
\begin{equation}
q=\frac{4\pi G \rho_c a^2}{c^2}=\frac{R_s}{R} \frac{1}{2\zeta_1\mid\theta_o'
(\zeta_1)\mid},
\end{equation}
where $R_s$ is the Schwarzschild radius, $R=a\zeta_1$ is the radius of system,
$\zeta_1$ is defined by $\theta_o(\zeta_1)=0$ and $c$ is the light speed.
The order of magnitude of $q$ varies from $10^{-5}$  for white dwarfs to
$10^{-1}$ for neutron stars.       For future reference, let us also note that
\stepcounter{sub}
\begin{equation}
-U=\lambda[\theta_1 + q (\Theta_1 - 2\theta_1^2)].
\end{equation}
We use a forth-order Runge-Kutta method to find
numerical solutions of the four coupled nonlinear differential Eqs. (28).
At the center we adopt
\stepcounter{sub}
\begin{equation}
\theta_a(0)=1;\;\;\;\theta'_a(0)=\frac{d\theta_a}{d\zeta}\mid_0=0;\;\;
\;\;\; a=0,1.
\end{equation}
In tables 1 and 2, we summarize the numerical results for the Newtonian and
post-Newtonian polytropes for different polytropic indices and $q$ values.
The $pn$ corrections tend to reduce the radius of the polytrope.
The
higher the polytropic index the smaller this radius.      The same is true,
of course, for higher values of $q$.\\
\section{Concluding remarks}
As we discussed in section 1, some authors is debated to exist a new modes of
oscillations in relativistic stellar systems.      They believed that these
modes generated by the perturbations in space time metric and no have analogue
in Newtonian systems.       They used  the general relativistic hydrodynamics
to distinguish them.
Although this way is routine but one needs to assume some thermodynamic
concepts
that may be fault in relativistic regime.    Hence, to reject these conceptual
problems, we choosed general relativistic Liouville's equation that is the
purely dynamical theory.       The combination Liouville and Einstein equations
enable one to study the behavior of relativistic systems without ambiguity.
Therefore, in this paper, we used the $pn$ approximation to obtain the
Einstein-Liouville equation for a relativistic self gravitating stellar system.
    We found two integrals,
generalization of the classical energy and angular momentum, that are
satisfying $pnl$ in equilibrium state.      These solutions enable one to
construct an equilibrium model for the system in $pn$ approximation.
Polytropic models, the most familiar stellar models, are constructed in $pn$
approximation.     In tables 1 and 2, we compared these models with its
Newtonian correspondence.     The $pn$ corrections tend to reduce the radius of
polytrope.      The higher the polytropic index the smaller this radius.
We introduced a parameter $q$, Eq. (29), to enter the effect of central density
of system in calculations.      Increasing values of $q$ reduce the
radius of system.       In the second paper (Sobouti and Rezania 1998), we
study time-dependent oscillations of a relativistic system in $pn$
approximation.

\newpage
\setcounter{sub}{0}
\setcounter{subeqn}{0}
\renewcommand{\theequation}{A.\thesub\thesubeqn}
\noindent {\large{\bf Appendix A: Derivation of Eqs. (5)
}}\\ Consider a general coordinate transformation $(X, U)$ to
$(Y, V)$.   The corresponding partial derivatives transform as
\[\left( \begin{array}{c}
\partial/\partial X\\ \partial/\partial U
          \end{array} \right)\;= M
\left( \begin{array}{c}
\partial/\partial Y\\ \partial/\partial V
          \end{array} \right)\;,\]

\stepcounter{sub}
\begin{equation}
\hspace{5.9cm}=
\left( \begin{array}{cc}
 \partial Y/\partial X &\partial V/\partial
X\\
\partial Y/\partial U &\partial V/\partial U
            \end{array}\right)
\left( \begin{array}{c}
\partial/\partial Y\\\partial/\partial V
          \end{array} \right) ,
\end{equation}
where $M$ is the $7\times 7$ Jacobian matrix of transformation.   Setting
$X=Y=x^{\mu}$, $V=v^i$ and $U=U^i$ for our problem, one finds
\stepcounter{sub}
\stepcounter{subeqn}
\begin{equation}
M=\left( \begin{array}{cc}
\partial x^{\mu}/\partial x^{\nu}&\partial v^i/\partial
x^{\nu}\\
\partial x^{\mu}/\partial U^j&\partial v^i/\partial U^j
\end{array}\right),
\end{equation}
and its inverse
\stepcounter{subeqn}
\begin{equation}
M^{-1}=\left( \begin{array}{cc}
\partial x^{\mu}/\partial x^{\nu}&\partial U^i/\partial
x^{\nu}\\
\partial x^{\mu}/\partial v^j&\partial U^i/\partial v^j
\end{array}\right).
\end{equation}
One easily finds
\stepcounter{sub}
\stepcounter{subeqn}
\begin{eqnarray}
&&\partial x^{\mu}/\partial x^{\nu}=\delta_{\mu\nu};\;\;\;\;\;\;\;
\partial x^{\mu}/\partial v^j=0,\\
\stepcounter{subeqn}
&&\partial U^i/\partial x^{\nu}=v^i\partial U^0/\partial
x^{\nu}
=\frac{{U^0}^3v^i}{2}\frac{\partial g_{\alpha\beta}}{\partial
x^{\nu}}v^{\alpha}v^{\beta},\\
\stepcounter{subeqn}
&&\partial U^i/\partial v^j=U^0\delta_{ij}+v^i \partial
U^0/\partial v^j
=U^0\delta_{ij}+{U^0}^3v^ig_{j\beta}v^{\beta}.
\end{eqnarray}
Inserting the latter in $M^{-1}$ and inverting the result one arrives at $M$
from which Eqs. (5) can be read out.\\
\newpage
\setcounter{sub}{0}
\renewcommand{\theequation}{B.\thesub\thesubeqn}
\noindent {\large{\bf Appendix B: Post-Newtonian hydrodynamics}}\\
Mathematical manipulations in the development of this work has been tasking.
To ensure that no error has crept in the course of calculations we have tried
to infer the post-Newtonian hydrodynamical equations from the post-Newtonian
Liouville equation derived earlier.        From Eq. (6a) one has
\stepcounter{sub}
\begin{eqnarray}
{\cal L}_U^{pn}f&&=U^0({\cal L}^{cl}+{\cal L}^{pn})f\nonumber\\
&&=[(1-\phi+\frac{1}{2}{\bf v}^2){\cal L}^{cl}+{\cal L}^{pn}]f,
\end{eqnarray}
where ${\cal L}^{cl}$ and ${\cal L}^{pn}$ are given by Eq. (10).
    We integrate ${\cal L}_U^{pn}f$ over the ${\bf v}$-space:
\stepcounter{sub}
\begin{equation}
\int {\cal L}_U^{pn}fd^3v=\int [(1-\phi+\frac{1}{2}{\bf v}^2) {\cal
L}^{cl}+{\cal L}^{pn}]fd^3v.
\end{equation}
Using Eqs. (12) and (13), one finds the continuity equation
\stepcounter{sub}
\begin{eqnarray}
&&\frac{\partial}{\partial
t}(\;^0T^{00}+\;^2T^{00})+\frac{\partial}{\partial x^j}(\; ^1T^{0j}
+\; ^3T^{0j})-\;^0T^{00}\frac{\partial \phi}{\partial t} =0,\nonumber\\
\end{eqnarray}
which is the $pn$ expansion of the continuity equation
\stepcounter{sub}
\begin{equation}
T^{0\nu}_{\;\;\;;\nu}=0,
\end{equation}
Next,
we multiply ${\cal L}_U^{pn}f$ by $v^i$ and integrate over the ${\bf
v}$-space:
\stepcounter{sub}
\begin{equation}
\int v^i {\cal L}_U^{pn}fd^3v=\int v^i[(1-\phi+\frac{1}{2}{\bf v}^2) {\cal
L}^{cl}+{\cal L}^{pn}]fd^3v.
\end{equation}
After some calculations one finds
\stepcounter{sub}
\begin{eqnarray}
&&\frac{\partial}{\partial
t}(\;^1T^{0i}+\;^3T^{0i})+\frac{\partial}{\partial x^j}(\; ^2T^{ij}
+\; ^4T^{ij})\nonumber\\
&&\;\;\;+\; ^0T^{00}[\frac{\partial}{\partial x^i}(\phi+2\phi^2+\psi)+
\frac{\partial
\eta_i}{\partial t}]+\; ^2T^{00}\frac{\partial
\phi}{\partial x^i}\nonumber\\
&&\;\;\;+\; ^1T^{0j}(\frac{\partial \eta_i}{\partial x^j}-\frac{\partial
\eta_j}{\partial x^i}-4\delta_{ij}\frac{\partial \phi}{\partial t})+ \;
^2T^{jk}
(\delta_{jk}\frac{\partial \phi}{\partial x^i}-4\delta_{ik}\frac{\partial \phi}
{\partial x^j})=0.
\end{eqnarray}
The latter is the correct $pn$ expansion of
\stepcounter{sub}
\begin{equation}
T^{i\nu}_{\;\;\;;\nu}=0;\;\;\;\;i=1,2,3.
\end{equation}
See Weinberg 1972, QED.
  \vspace{2cm}\\

{\Large{\bf References:}}\\
\begin{itemize}
\item[ ] Andersson N., Kokkotas K. D., Schutz B. F., 1995, M. N. R. A. S.,
{\bf 274}, 1039
\item[ ] Baumgarte, T. W., Schmidt, B. G., 1993, Class. Quantum. Grav.,
{\bf 10}, 2067
\item[ ] Dehghani M. H., Rezania V., 1996, Astron. Astrophys., {\bf 305}, 379
\item[ ] Detweiler S. L., Lindblom L., 1985, Ap. J., {\bf 292}, 12
\item[ ] Ehlers J., 1977, in: Sachs R. K. (ed.) Proceedings of the international
summer school of
\indent Physics ``Enrico Fermi'', Course 47
\item[ ] Ellis G. R., Matraverse D. R., Treciokas R., 1983, Ann. Phys. {\bf
150}, 455
\item[ ] Ellis G. R., Matraverse D. R., Treciokas R., 1983, Ann. Phys. {\bf
150}, 487
\item[ ] Ellis G. R., Matraverse D. R., Treciokas R., 1983, Gen. Rel. Grav.
{\bf 15}, 931
\item[ ] Kojima, Y., 1988, Prog. Theor. Phys.,
{\bf 79}, 665
\item[ ] Kokkotas K. D., Schutz B. F., 1986, Gen. Rel. Grav., {\bf 18},
913
\item[ ] Kokkotas K. D., Schutz B. F., 1992, M. N. R. A. S., {\bf
255}, 119
\item[ ] Leaver, E. W., 1985, Proc. R. Soc. London A,
{\bf 402}, 285
\item[ ] Leins, M., Nollert, H. P., Soffel, M. H., 1993, Phys. Rev. D,
{\bf 48}, 3467
\item[ ] Lindblom L., Detweiler S. L., 1983, Ap. J. Suppl., {\bf 53}, 73
\item[ ] Lindblom, L., Mendell, G., Ipser, J. R., 1997, Phys. Rev. D,
{\bf 56}, 2118
\item[ ] Maartens R., Maharaj S. D., 1985, J. Math. Phys. {\bf 26}, 2869
\item[ ] Maharaj S. D., Maartens R., 1987, Gen. Rel. Grav. {\bf 19}, 499
\item[ ] Maharaj S. D., 1989, Nouvo Cimento 163B, No. 4, 413
\item[ ] Mansouri A., Rakei A., 1988, Class. Quantum. Grav. {\bf 5}, 321
\item[ ] Nollert, H. -P., Schmidt, B. G., 1992, Phys. Rev. D,
{\bf 45}, 2617
\item[ ] Ray J. R., Zimmerman J. C., 1977, Nouvo Cimento 42B, No. 2, 183
\item[ ] Sobouti Y., Rezania V., 1998, Astron. Astrophys., (Submitted to)
\item[ ] Weinberg S., 1972, Gravitation and Cosmology. John Wiley \& Sons, New
york
\end{itemize}
\newpage
\noindent Table 1.  A comparison of the Newtonian and post-Newtonian polytropes
at certain selected radii for $n$=1, 2 and 3 and
different values $q$.\vspace{1cm}\\
Table 2.   Same as Table 1. $n$=4 and 4.5.
\newpage
\begin{center}
Table 1.\vspace{.2cm}\\
\begin{tabular}{|c|c|c|c|c|c|}\hline
n &Polytropic & Newtonian   &\multicolumn{3}{c|}{$pn$ polytrope,
$\theta+q(\Theta-2\theta^2)$}\\\cline{4-6}
  &radius, $\zeta$ &polytrope, $\theta$& $q=10^{-5}$ & $q=10^{-3}$ &
$q=10^{-1}$ \\\hline   & 0.0000000  & 1.00000 & 1.00000 & 1.00000 & 1.00000\\
  & 1.0000000  & 0.84145 & 0.84145 & 0.84143 & 0.83936\\
  & 2.0000000  & 0.45458 & 0.45458 & 0.45433 & 0.42949\\
1 & 2.9233000  & 0.07408 & 0.07407 & 0.07334 &  0.00000    \\
  & 3.1388500  & 0.00087 & 0.00086 & 0.00000 &         \\
  & 3.1415500  & 0.00001 & 0.00000 &         &      \\
  & 3.1415930  & 0.00000 &         &         &      \\\hline
  & 0.0000000  & 1.00000 & 1.00000 & 1.00000 & 1.00000\\
  & 1.0000000  & 0.84868 & 0.84868 & 0.84863 & 0.84394\\
  & 2.0000000  & 0.52989 & 0.52988 & 0.52945 & 0.48609\\
  & 3.0000000  & 0.24188 & 0.24187 & 0.24031 & 0.13289\\
2 & 3.4737000  & 0.13904 & 0.13902 & 0.13770 &  0.00000     \\
  & 4.3394800  & 0.00171 & 0.00169 & 0.00000 &      \\
  & 4.3527000  & 0.00002 & 0.00000 &         &       \\
  & 4.3529000  & 0.00000 &         &         &       \\\hline
  & 0.0000000  & 1.00000 & 1.00000 & 1.00000 & 1.00000\\
  & 1.0000000  & 0.85480 & 0.85480 & 0.85473 & 0.84773\\
  & 2.0000000  & 0.58284 & 0.58283 & 0.58230 & 0.52894\\
  & 3.0000000  & 0.35939 & 0.35938 & 0.35824 & 0.24016\\
  & 4.0000000  & 0.20927 & 0.20925 & 0.20764 & 0.03226\\
3 & 4.1939500  & 0.18690 & 0.18688 & 0.18520 &  0.00000     \\
  & 6.8435000  & 0.00228 & 0.00225 & 0.00000 &              \\
  & 6.8963000  & 0.00002 & 0.00000 &         &               \\
  & 6.8967000  & 0.00000 &         &         &              \\\hline
\end{tabular}
\end{center}
\newpage
\begin{center}
Table 2.\vspace{.2cm}\\
\begin{tabular}{|c|c|c|c|c|c|}\hline
n &Polytropic & Newtonian   &\multicolumn{3}{c|}{$pn$ polytrope,
$\theta+q(\Theta-2\theta^2)$}\\\cline{4-6}
  &radius, $\zeta$ &polytrope, $\theta$& $q=10^{-5}$ & $q=10^{-3}$ &
$q=10^{-1}$ \\\hline   & 0.0000000  & 1.00000 & 1.00000 & 1.00000 & 1.00000\\
  & 2.0000000  & 0.62294 & 0.62293 & 0.62235 & 0.56326\\
  & 4.0000000  & 0.31804 & 0.31802 & 0.31645 & 0.14194\\
  & 5.1541000  & 0.22574 & 0.22572 & 0.22383 &  0.00000    \\
  & 8.0000000  & 0.10450 & 0.10448 & 0.10221 &         \\
4 &12.0000000  & 0.02972 & 0.02970 & 0.02716 &         \\
  &14.0000000  & 0.00833 & 0.00830 & 0.00570 &         \\
  &14.6468000  & 0.00265 & 0.00262 & 0.00000 &         \\
  &14.9680000  & 0.00003 & 0.00000 &         &      \\
  &14.9713400  & 0.00000 &         &         &      \\\hline
  & 0.0000000  & 1.00000 & 1.00000 & 1.00000 & 1.00000\\
  & 2.0000000  & 0.63965 & 0.63964 & 0.63905 & 0.57857\\
  & 4.0000000  & 0.36053 & 0.36651 & 0.35897 & 0.18628\\
  & 5.7468600  & 0.24334 & 0.24332 & 0.24135 & 0.00000\\
4.5 & 8.0000000& 0.16173 & 0.16171 & 0.15946 &     \\
  &12.0000000  & 0.09015 & 0.09013 & 0.08766 &      \\
  &16.0000000  & 0.05402 & 0.05399 & 0.05141 &       \\
  &20.0000000  & 0.03230 & 0.03227 & 0.02962 & \\
  &24.0000000  & 0.01782 & 0.01779 & 0.01510 & \\
  &28.0000000  & 0.00747 & 0.00744 & 0.00472 & \\
  &30.2689200  & 0.00282 & 0.00279 & 0.00000 &\\
  &31.7792300  & 0.00004 & 0.00000 &         &    \\
  &31.7878400  & 0.00000 &         &         &  \\\hline
\end{tabular}
\end{center}
\end{document}


\section{Introduction}
\label{sec:introduction}

    Reinforcement Learning (RL) is a powerful algorithmic paradigm used to solve sequential decision-making problems and has resulted in great success in various types of environments, e.g., mastering the game of GO \citep{silver2017mastering}, playing computer games \citep{mnih2013playing} and controlling power systems \citep{qiu2021multi,wang2021multi}. The majority of these successes depends on an implicit assumption that \textit{the testing environment is identical to the training environment}. However, this assumption is too strong for most realistic problems, for example, controlling a robot. There are several situations where mismatches might appear between training and testing environments in robotics: (1) \textit{Parameter Perturbations} indicates that a large number of environmental parameters, e.g. temperature, friction factor could fluctuate after deployment and thus deviate from the training environment \citep{kober2013reinforcement}; (2) \textit{System Identification} estimates a transition function from finite experience. This estimation is biased compared with the real world model \citep{schaal1996learning}; (3) \textit{Sim-to-Real} learns a policy in a simulated environment and performs on real robots for reasons of safety and efficiency \citep{openai2019learning}. Henceforth, it is essential to develop Robust RL algorithms that can adapt to unseen testing environments for practical applications.

    Robust Markov Decision Processes (Robust MDPs) \citep{iyengar2005robust, wiesemannrobust} is a common framework for analyzing the robustness of RL algorithms. Compared with regular MDPs with a single transition model $\mathcal{P}(s'|s,a)$, Robust MDPs consider an uncertainty set of transition models $\mathbb{P}=\{\mathcal{P}\}$ to better describe the perturbation of the transitions. This formulation is sufficiently general to cover various scenarios for robot learning problem.

    Robust RL aims to learn a robust policy under the worst-case scenario for all transition models $\mathcal{P}\in\mathbb{P}$. If the transition model $\mathcal{P}$ is viewed as an adversarial agent and the uncertainty set $\mathbb{P}$ is its action space, one can reformulate Robust RL as a zero-sum game \citep{ho2018fast}. In general, solving such a problem is NP-hard \citep{wiesemannrobust}. \citet{derman2021twice}, however, adopted the Legendre-Fenchel transform to avoid excessive mini-max computations by converting the minimization over the transition model to a regularization on the value function. Furthermore, it enjoys the additional advantage that it provides more possibilities to design novel regularizers for different types of transition uncertainties. Its extension to  continuous state space problems (required, e.g., for controlling robots) is unclear, which directly motivates the work of this paper.

    In this paper, we (1) extend the robustness-regularization duality method to continuous control tasks; (2) propose the \textbf{U}ncertainty \textbf{S}et \textbf{R}egularizer (USR) on existing RL frameworks for learning robust policies; (3) learn an adversarial uncertainty set through the value function when the actual uncertainty set is unknown in some scenarios; (4) evaluate USR on the Real-world Reinforcement Learning (RWRL) benchmark, showing improvements for a robust performance in perturbed testing environments with unknown uncertainty sets.

\section{Preliminaries}
\label{sec:preliminaries}

\subsection{Robust MDPs}
\label{sec:robust_mdp}
    The mathematical framework of Robust MDPs \citep{iyengar2005robust, wiesemannrobust} extends regular MDPs in order to deal with  uncertainty about the transition function.
    A Robust MDP can be formulated as a 6-tuple $\langle \mathcal{S},\mathcal{A}, \mathbb{P}, r, \mu_0,  \gamma \rangle$, where $\mathcal{S}, \mathcal{A}$ stand for the state and action space respectively, and $r(s,a): \mathcal{S} \times \mathcal{A} \to \mathbb{R}$\ stands for the reward function. Let $\Delta_{|S|}$ and $\Delta_{|A|}$ be the probability measure on $S$ and $A$ respectively. The initial state is sampled from an initial distribution $\mu_0 \in \Delta_{\mathcal{S}}$, and the future rewards are discounted by the discount factor $\gamma \in [0,1] $.
    The most important concept in robust MDPs is the uncertainty set $\mathbb{P}=\left\{ \mathcal{P}(s'|s,a): \mathcal{S} \times \mathcal{A} \to \Delta_{\mathcal{S}} \right\}$ that controls the variation of transition $\mathcal{P}$, compared with the stationary transition $\mathcal{P}$ in regular MDPs. Let $\Pi=\left\{ \pi(a|s): \mathcal{S} \to \Delta_{\mathcal{A}} \right\}$ be the policy space; the objective of Robust RL can then be formulated as a mini-max problem,

    \begin{equation}
    \label{eq:robust_objective}
        J^* = \max_{\pi \in \Pi} \min_{\mathcal{P} \in \mathbb{P}} \mathbb{E}_{ \pi, \mathcal{P} } \left[ \sum_{t=0}^{+\infty} \gamma^t r(s_t, a_t) \right].
    \end{equation}

\subsection{Robust Bellman equation}
\label{sec:robust_bellman}
    While \citet{wiesemannrobust} has proved NP-hardness of this mini-max problem with an arbitrary uncertainty set, most recent studies \citep{iyengar2005robust,  ho2018fast, derman2021twice, nilim2004robustness, roy2017reinforcement, mankowitz2019robust, dermansoftrobust,  wang2021online, grand-clement2021scalable} approximate it by assuming a rectangular structure on the uncertainty set, i.e., $\mathbb{P} = \mathop{\times}_{ (s,a) \in \mathcal{S} \times \mathcal{A} }^{} \mathbb{P}_{sa}, {\rm where }\ \mathbb{P}_{sa}=\{ \mathcal{P}_{sa}(s'): \Delta_{\mathcal{S}} \}$ denotes the local uncertainty of the transition at $(s, a)$. In other words, the variation of transition is independent at every $(s, a)$ pair. Under the assumption of a rectangular uncertainty set, the robust action value function $Q^{\pi}(s, a)$ under policy $\pi$ must satisfy the following robust version of the Bellman equation \citep{bellman1966dynamic} such that

    \begin{equation}
    \label{eq:bellman_equation}
    \begin{split}
          Q^{\pi}(s, a)  &= r(s, a) +  \min_{\mathcal{P}_{sa} \in \mathbb{P}_{sa} }\gamma \sum_{s', a'}\mathcal{P}_{sa}(s') \pi(a'|s') Q^{\pi}(s', a') \\
             &=  r(s, a) +  \min_{\mathcal{P}_{sa} \in \mathbb{P}_{sa} }\gamma \sum_{s'}\mathcal{P}_{sa}(s') V^{\pi}(s').
    \end{split}
    \end{equation}

    \citet{nilim2004robustness} have shown that a robust Bellman operator admits a unique fixed point of Equation \ref{eq:bellman_equation}, the robust action value function $Q^{\pi}(s, a)$.

\subsection{Robustness-regularization duality}
\label{sec:duality}
    Solving the minimization problem in the RHS of Equation \ref{eq:bellman_equation} can be further simplified by the Legendre-Fenchel transform \citep{rockafellar1970convex}. For a function $f: X \to \mathbb{R}$, its convex conjugate is $f^{*}\left(x^{*}\right):=\sup \left\{\left\langle x^{*},x\right\rangle -f(x)~\colon ~x\in X\right\}$. Define $\delta_{\mathbb{P}_{sa}}(\mathcal{P}_{sa}) =0$ if $\mathcal{P}_{sa} \in {\mathbb{P}_{sa}} $ and $+\infty$ otherwise, Equation \ref{eq:bellman_equation} can be transformed to its convex conjugate (refer \citet{derman2021twice} for detailed derivation),

    \begin{equation}
    \label{eq:bellman_equation_dual}
    \begin{split}
         Q^{\pi}(s, a) &=  r(s, a) +  \min_{\mathcal{P}_{sa} }\gamma \sum_{s'}\mathcal{P}_{sa}(s') V^{\pi}(s') + \delta_{\mathbb{P}_{sa}}(\mathcal{P}_{sa}) \\
         &=  r(s, a) - \delta^*_{\mathbb{P}_{sa}}(-V^{\pi}(\cdot)).
    \end{split}
    \end{equation}

    The transformation implies that the robustness condition on transition can be equivalently expressed as a regularization constraint on the value function, referred to as the robustness-regularization duality. The duality can extensively reduce the cost of solving the minimization problem over infinite transition choices and thus is widely studied in the robust reinforcement learning research community \citep{husain2021regularized, eysenbach2021maximum, brekelmans2022your}.

    As a special case, \citet{derman2021twice} considered a $L_2$ norm uncertainty set on transitions, i.e., $\mathbb{P}_{sa} = \{  \bar{\mathcal{P}}_{sa} + \alpha \tilde{\mathcal{P}}_{sa} : \| \tilde{\mathcal{P}}_{sa} \|_2 \le 1 \}$, where $\bar{\mathcal{P}}_{sa}$ is usually called the nominal transition model. It could represent  prior knowledge on the transition model or a numerical value of the training environment. The uncertainty set denotes that the transition model is allowed to fluctuate around the nominal model with some degree $\alpha$. Therefore, the corresponding Bellman equation in Equation \ref{eq:bellman_equation_dual} becomes $Q^{\pi}(s, a) =  r(s, a) + \gamma   \sum_{s'}\bar{\mathcal{P}}_{sa}(s') V^{\pi}(s') - \alpha\|V^{\pi}(\cdot)\|_2$. Similarly, the $L_1$ norm has also been used as uncertainty set on transitions \citep{wang2021online}, i.e., $\mathbb{P}_{sa} = \{  \bar{\mathcal{P}}_{sa} + \alpha \tilde{\mathcal{P}}_{sa} : \| \tilde{\mathcal{P}}_{sa} \|_1 \le 1 \}$, and the Bellman equation becomes the form such that $Q^{\pi}(s, a) =  r(s, a) + \gamma  \sum_{s'}\bar{\mathcal{P}}_{sa}(s') V^{\pi}(s') - \alpha \max_{s'}|V^{\pi}(s')|$. This robustness-regularization duality works well with finite state spaces, but is hard to directly extend to infinite state spaces since both $\|V^{\pi}(\cdot)\|_2$ and $\max_{s'}|V^{\pi}(s')|$ regularizers need calculations on the infinite dimensional vector $V^{\pi}(\cdot)$.

\section{Uncertainty Set Regularized Robust Reinforcement Learning}
\label{sec:method}
    Having introduced the robustness-regularization duality and the problems regarding its extension to continuous state space problems in Section \ref{sec:duality}, here, we will first present a novel extension to continuous state space with the uncertainty set defined on the parameter space of the transition function. We will then utilize this extension to derive a robust value iteration that can be directly plugged into existing RL algorithms. Furthermore, to deal with the unknown uncertainty set, we propose an adversarial uncertainty set and visualize it in a simple \textit{moving to target} task.

\subsection{Uncertainty Set Regularized Robust Value Iteration (USR-RVI) }
\label{sec:rvi}
   For environments with continuous state spaces, the transition model $\mathcal{P}(s'|s,a)$ is usually represented as a parametric function $\mathcal{P}(s'|s,a;w)$, where $w$ denotes the parameters of the transition function. Instead of defining the uncertainty set on the distribution space, we directly impose a perturbation on $w$ within a set $\Omega_w$. Consequently, the robust objective function (Equation \ref{eq:robust_objective}) becomes $J^* = \max_{\pi \in \Pi} \min_{w \in \Omega_w} \mathbb{E}_{ \pi, \mathcal{P}(s'|s,a;w) } \left[ \sum_{t=0}^{+\infty} \gamma^t r(s_t, a_t) \right]$. We further assume the parameter $w$ fluctuates around a nominal parameter $\bar{w}$, such that $w=\bar{w} + \tilde{w}$, with $\bar{w}$ being a fixed parameter and $\tilde{w} \in \Omega_{\tilde{w}} =\{ w-\bar{w} | w \in \Omega_w\}$ being the perturbation part. Inspired by Equation \ref{eq:bellman_equation_dual}, we can derive a robust value iteration algorithm on the parametric space for continuous control problems as shown in Proposition \ref{prop:rvi}.

   \begin{proposition}
        Suppose the uncertainty set of $w$ is $\Omega_w$ (i.e., the uncertainty set of $\tilde{w}=w-\bar{w}$ is $\Omega_{\tilde{w}}$),
        the robust value iteration on parametric space can be represented as follows:
        \begin{equation}
        \label{eq:rvi}
             Q^{\pi}(s, a) =  r(s, a) + \gamma \int_{s'} \mathcal{P}(s'|s,a; \bar{w}) V^{\pi}(s') ds' - \gamma \int_{s'} \delta^*_{\Omega_{\tilde{w}}} \left[-\nabla_{w}\mathcal{P}(s'|s,a; \bar{w}) V^{\pi}(s') \right] ds',
        \end{equation}
    where $\delta_{\Omega_w}(w)$ is the indicator function that equals $0$ if $w \in \Omega_w$ and $+\infty$ otherwise,  and $\delta^*_{\Omega_w}(w')$ is the convex dual function of $\delta_{\Omega_w}(w)$.
    \label{prop:rvi}
    \end{proposition}

    The complete derivation is given in Appendix \ref{sec:robust_value_iteration}. Intuitively, Proposition \ref{prop:rvi} shows that ensuring robustness on parameter $w$ can be transformed into an additional regularizer on value iteration that relates to the product of the state value function $V^{\pi}(s')$ and the derivative of the transition model $\nabla_{w}\mathcal{P}(s'|s,a;\bar{w})$.
    Taking the $L_2$ uncertainty set (also used in \citet{derman2021twice}) as a special case, i.e., $\Omega_w = \{\bar{w} + \alpha \tilde{w} : \| \tilde{w} \|_2 \le 1 \}$, where $\bar{w}$ stands for the parameter of the nominal transition model $\mathcal{P}(s'|s,a';\bar{w})$, the robust value iteration in Proposition \ref{prop:rvi} becomes

    \begin{equation}
        Q^{\pi}(s, a) =  r(s, a) + \gamma   \int_{s'}\mathcal{P}(s'|s,a;\bar{w}) V^{\pi}(s')ds' - \alpha \int_{s'} \|\nabla_w\mathcal{P}(s'|s,a;\bar{w})V^{\pi}(s')\|_2 ds'.
    \label{eq:l2_usr}
    \end{equation}

    Following the idea of Q-Learning \citep{suttonreinforcement}, a Bellman operator can be naturally derived from Equation \ref{eq:l2_usr} to learn a robust policy. The two integrals in Equation \ref{eq:l2_usr} can be calculated exactly or approximated by sampling methods. We will discuss these in details in the following section.

\subsection{Uncertainty Set Regularized Robust Reinforcement Learning (USR-RRL)}
\label{sec:usr_rrl}
    USR-RVI proposed in the last section naturally applies to model-based RL, as model-based RL learns a point estimate of the transition model $P(s'|s, a; \bar{w})$ by maximum likelihood approaches \citep{kegl2021modelbased}. Since calculating the derivatives in USR-RVI is computationally involved, we choose to construct a local transition model with mean and covariance as parameters only (cf. comments in Section \ref{sec:limitations}).
    Specifically, suppose one can access a triple (state $s$, action $a$ and next state $x$) from the experience replay buffer, a local transition model $P(s'|s,a;\bar{w})$ is constructed as a Gaussian distribution with mean $x$ and covariance $\Sigma$ (with $\Sigma$ being a hyperparameter), i.e., the nominal parameter $\bar{w}$ consists of $(x, \Sigma)$. With this local transition model, we now have the full knowledge of $\mathcal{P}(s'|s,a;\bar{w})$ and   $\nabla_w\mathcal{P}(s'|s,a;\bar{w})$, which allows us to calculate the RHS of Equation \ref{eq:l2_usr}.
    To further approximate the integral calculation in Equation \ref{eq:l2_usr}, we sample $M$ points $\{s_1, s_2, ..., s_M\}$ from the local transition model $\mathcal{P}(s'|s,a;\bar{w})$ and use them to approximate the target action value function by $Q^{\pi}(s, a) \approx r(s, a) + \gamma \sum_{i=1}^M  \left[   V^{\pi}(s_i) -  \alpha \| \nabla_w\mathcal{P}(s_i|s,a;\bar{w})V^{\pi}(s_i)\|_2 / \mathcal{P}(s_i|s,a;\bar{w}) \right]$, where $\bar{w}=(x, \Sigma)$. With this approximated value iteration, the Bellman operator is guaranteed to converge to the robust value of current policy $\pi$. Policy improvement can be further applied to learn a more robust policy.

\subsection{Adversarial uncertainty set}
\label{sec:adv}

    \begin{wrapfigure}{r}{0.55\textwidth}
      \begin{center}
        \includegraphics[width=0.55\textwidth]{figures/adversarial_uncertainty_set.pdf}
      \end{center}
      \caption{Generation of adversarial uncertainty set.}
    \label{fig:adv_uc}
    \end{wrapfigure}

    The proposed method in Section \ref{sec:usr_rrl} relies on the prior knowledge of the uncertainty set of the parameter space. The $L_p$ norm uncertainty set is most widely used in the Robust RL and robust optimal control literature. However, such a fixed uncertainty set may not sufficiently adapt for various perturbation types. The $L_p$ norm uncertainty set with its larger region can result in an over-conservative policy, while the one with a smaller region may lead to a risky policy. In this section, we learn an adversarial uncertainty set through the agent's policy (and value function).

    The basic idea of the adversarial uncertainty set is to provide a broader uncertainty range to parameters that are more sensitive to the value function, which is naturally measured by the derivative. The agent learning on such an adversarial uncertainty set is easier to adapt to the various perturbation types of parameters. We generate the adversarial uncertainty set in 5 steps as illustrated in Figure \ref{fig:adv_uc}, (1) \textit{sample} next state $s'$ according to the distribution $\mathcal{P}(\cdot|s,a;\bar{w})$, given the current state $s$ and action $a$; (2) \textit{forward} pass by calculating the state value $V(s')$ at next state $s'$; (3) \textit{backward} pass by using the reparameterization trick \citep{kingma2014autoencoding} to compute the derivative $g(\bar{w}) = \nabla_{w}V(s';\bar{w})$; (4) \textit{normalize} the derivative by $d(\bar{w})= g(\bar{w})/[\sum_i^W g(\bar{w})_i]^2]^{0.5}$; (5) \textit{generate} the adversarial uncertainty set $\Omega_w = \{  \bar{w} + \alpha \tilde{w} : \| \tilde{w} / d(\bar{w})\|_2 \le 1 \}$.

\subsection{Visualizing Adversarial Uncertainty Set}
\label{sec:vis_adv}
    To further investigate the characteristics of the adversarial uncertainty set, we visualize it in a simple \textit{moving to target} task: controlling a particle to move towards a target point $e$ (Figure \ref{fig:uc_example}.a). The two-dimensional state $s$ informs the position of the agent, and the two-dimensional action $a=(a_1, a_2)$ ($\|a\|_2=1$ is normalized) controls the force in two directions. The environmental parameter $w=(w_1, w_2)$ represents the contact friction in two directions respectively. The transition function is expressed as $s' \sim \mathcal{N}(s+(a_1w_1, a_2w_2), \Sigma)$, and the reward is defined by the progress of the distance to the target point minus a time cost: $r(s,a,s')=d(s,e)-d(s',e)-2$. The nominal value $\bar{w}=(1,1)$ (Figure \ref{fig:uc_example}.b) indicates the equal friction factor in two directions for the training environment. It is easy to conjecture that the optimal action is pointing towards the target point, and the optimal value function $V^*(s)=-d(s,e)$.

    We visualize the $L_2$, $L_1$ and the adversarial uncertainty set of the contact friction $w$ in Figure \ref{fig:uc_example}.(b,c,d) respectively, at a specific state $s=(4,3)$ and the optimal action $a^*=(-0.8, -0.6)$. We can see that compared with the $L_2$ uncertainty set, the adversarial uncertainty set extends the perturbation range of the horizontal dimensional parameter since it is more sensitive to the final return. As a result, the agent learning to resist such an uncertainty set is expected to generally perform well on unseen perturbation types, which will be verified in experiments in the next section.

    \begin{figure*}[ht!]
    \centering
        \begin{subfigure}[b]{1.0\linewidth}
    	    \centering        	    \includegraphics[width=\textwidth]{figures/uc_example.pdf}
        \end{subfigure}
    \caption{An example of uncertainty set.}
    \label{fig:uc_example}
    \end{figure*}

\section{Experiments}
\label{sec:experiments}
    In this section, we demonstrate the experimental results on the Real-world Reinforcement Learning (RWRL) benchmark \citep{dulac-arnold2019challenges}, to validate the effectiveness of the proposed regularizing USR for resisting the perturbations in the environment.

\subsection{Experimental Setups}
\label{sec:exp_setup}
    \paragraph{Task Description:} RWRL, whose back-end is the Mujoco environment \citep{todorov2012mujoco}), is a continuous control benchmark consisting of real-world challenges for RL algorithms (e.g., learning from limited samples, dealing with delays in system actuators, high dimensional state and action space). Using this benchmark, we will evaluate the proposed algorithm regarding the robustness of the learned policy in physical environments with perturbations over parameters of the state equations (dynamics). In more detail, we first train a policy through the interaction with the nominal environments (i.e., the environments with no perturbations), and then test the policy in the environments with perturbations within a range over relevant physical parameters.

    In this paper, we conduct experiments on six tasks: \textit{cartpole\_balance}, \textit{cartpole\_swingup}, \textit{walker\_stand}, \textit{walker\_walk}, \textit{quadruped\_walk}, \textit{quadruped\_run}, with growing complexity in state and action space. More details about the specifications of tasks are shown in Appendix \ref{sec:rwrl}. The perturbed variables and their value ranges can be found in the Table \ref{tab:rwrl_env} in Appendix.

    \paragraph{Evaluation metric:} A severe problem for Robust RL research is the lack of a standard metric to evaluate policy robustness. To resolve this obstacle, we define a new robustness evaluation metric which we call \textit{Robust-AUC} to calculate the area under the curve of the return with respect to the perturbed physical variables, in analogy to the definition of regular AUC \citep{huang2005using}. More specifically, a policy $\pi$ after training is evaluated in an environment with perturbed variable $P$ whose values $v \in [v_{min}, v_{max}]$ for $N$ episodes to obtain an $\alpha$\%-quantile return $r$. Then, these two sets of data are employed to draw a parameter-return curve $C(P, R)$ to describe the relationship between returns $r$ and perturbed values $v$. We define the relative area under this curve as Robust-AUC such that $rv(\pi, P)=\frac{Area(C(v, r))}{v_{max}-v_{min}}, v \in [v_{min}, v_{max}]$. Compared to the vanilla AUC, Robust-AUC describes the correlation between returns and the perturbed physical variables, which can sensitively reflect the response of a learning procedure (to yield a policy) to the unseen perturbations, i.e., the robustness. In practical implementations, we set $N=100$ and $\alpha=10$ and uniformly collect $20$ samples of $v$ to estimate the area of the curve.

    \paragraph{Baselines and Implementation of Proposed Methods:} We first compare USR with a standard version of Soft Actor Critic (SAC) \citep{haarnoja2018soft}, which stands for the category of algorithms without regularizers (\textit{None Reg}). Another category of baselines is to directly impose $L_p$ regularization on the parameters of the value function (\textit{L1 Reg}, \textit{L2 Reg}) \citep{liu2019regularization}, which is a common way to improve the generalization of function approximation but without consideration of the environmental perturbations; For a fixed uncertainty set as introduced in Section \ref{sec:usr_rrl}, we implement two types of uncertainty sets on transitions, \textit{L2 USR} and \textit{L1 USR}, which can be viewed as an extension of \citet{derman2021twice} and the work of \citet{wang2021online} work to continuous control task respectively; finally, we also evaluate the adversarial uncertainty set (Section \ref{sec:adv}), denoted as \textit{Adv USR}.

    \paragraph{Experimental Setups:} For a fair comparison, we utilize the same model specifications and hyperparameters for all algorithms in model-free RL and model-based RL respectively, except for the regularizer method and its coefficient. We select the best coefficient for each setting through validation. The detailed model specifications and hyperparameters are shown in Appendix \ref{sec:hyperparameters}. Each algorithm in the baseline is trained with 5 random seeds. During the test phase, for each environmental variable, we uniformly sample 20 perturbed values $v$ in the range of $[v_{min}, v_{max}]$. For each value $v$, we run 20 episodes (5 seeds, in total 100 episodes) and select the $10\%$-quantile as the return $r$ to calculate Robust-AUC defined previously.

\subsection{Main Results}
\label{sec:main_results}
    We show the parameter-return curve for all algorithms below in Figure \ref{fig:mfrl} and their Robust-AUC in Table \ref{tab:mfrl}. Due to page limits, the results of other tasks are presented in Appendix (Figure \ref{fig:mfrl_extra} and Table \ref{tab:mfrl_extra}). In addition to calculating Robust-AUC under different perturbations, we also rank all algorithms and report the average rank as an overall robustness performance of each task. Notably, \textit{L1 Reg} and \textit{L2 Reg} do not improve on the robustness, and even impair the performance in comparison with the non-regularized agent on simple domains (\textit{cartpole} and \textit{walker}). In contrast, we observe that both \textit{L2 USR} and \textit{L1 USR} can outperform the default version under some certain perturbations (e.g. \textit{L1 USR} in \textit{cartpole\_swingup} for pole\_length, \textit{L2 USR} in \textit{walker\_stand} for thigh\_length); they are, however, not effective for all scenarios. We argue that the underlying reason could be that the fixed shape of the uncertainty set cannot adapt to all perturbed cases. This is supported by the fact that  \textit{Adv USR} achieves the best average rank among all perturbed scenarios, showing the best zero-shot generalization performance in continuous control tasks.

    \begin{figure*}[ht!]
        \centering
        \begin{subfigure}[b]{1.0\linewidth}
        	\centering
    	    \includegraphics[width=\textwidth]{figures/cartpole_swingup.pdf}
    	    \caption{cartpole\_swingup}
    	\label{fig:cartpole_swingup}
        \end{subfigure}
        ~
        \begin{subfigure}[b]{1.0\linewidth}
    	    \centering        	    \includegraphics[width=\textwidth]{figures/walker_stand.pdf}
    	    \caption{walker\_stand}
    	\label{fig:walker_stand}
        \end{subfigure}
        ~
        \begin{subfigure}[b]{1.0\linewidth}
    	    \centering        	    \includegraphics[width=\textwidth]{figures/quadruped_walk.pdf}
    	    \caption{quadruped\_walk}
    	\label{fig:quadruped_walk}
        \end{subfigure}
    \caption{The parameter-return curve of model-free RL algorithms. All graphs are plotted with the $10\%$-quantile and 5\%-15\%-quantile shading. The vertical dashed line denotes the nominal value that all algorithms train on. Robust-AUC is illustrated after each label in the graph.}
    \label{fig:mfrl}
    \end{figure*}

    \begin{table}[ht]
    	\caption{Robust-AUC of Model-free RL algorithms.}
    	\begin{center}
    	 \scalebox{0.7}{
    		\begin{tabular}{llcccccc}
    			\toprule
    			\multirow{2}{*}{\textbf{Task Name}} &
    			\multirow{2}{*}{\textbf{Variables}} &
    			\multicolumn{6}{c}{\textbf{Algorithms}} \\
    			&& \textit{None Reg} & \textit{L1 Reg} & \textit{L2 Reg} & \textit{L1 USR} & \textit{L2 USR} & \textit{Adv USR} \\
		\midrule	\multirow{5}{*}{cartpole\_swingup}
		& pole\_length    & 393.41 & 319.73 & 368.16 & \textbf{444.93} & 424.48 & 430.91 \\
		& pole\_mass      & 155.25 & 96.85 & 131.35 & 175.28 & 159.61  & \textbf{193.13}   \\
	    & joint\_damping  & 137.20 & 140.16 & 165.01 & 164.21 & 169.88 & \textbf{170.39}  \\
	    & slider\_damping & 783.76 & 775.73 & 797.59 & 793.55 & 781.02 & \textbf{819.32}  \\
	    & average rank    & 4.5    & 5.75   & 3.75   & 2.5    & 3.25   & \textbf{1.25}  \\
		\midrule	\multirow{5}{*}{walker\_stand}
		& thigh\_length  & 487.02 & 461.95 & 497.38 & 488.71 & \textbf{511.16} & 505.88  \\
		& torso\_length  & 614.06 & 586.16 & 586.20 & 598.02 & 610.93 & \textbf{623.56}  \\
	    & joint\_damping  & \textbf{607.24} & 387.89 & 443.82 & 389.77 & 527.87 & 514.77  \\
	    & contact\_friction & 946.74 & \textbf{947.24} & 941.92 & 943.11 & 940.73 & 945.69  \\
	    & average rank & 2.50 & 4.75 & 4.25 & 4.25 & 3.00 & \textbf{2.25}    \\
	    \midrule	\multirow{5}{*}{quadruped\_walk}
		& shin\_length & 492.55 & 406.77 & 503.13 & 540.39 & 564.60 & \textbf{571.85}  \\
		& torso\_density & 471.45 & 600.86 & 526.22 & 442.05 & 472.80 & \textbf{602.09} \\
	    & joint\_damping & 675.95 & 711.54 & \textbf{794.56} & 762.50 & 658.17 & 785.11  \\
	    & contact\_friction & 683.80 & 906.92 & 770.44 & 777.40 & 767.80 & \textbf{969.73} \\
	    & average rank & 5.25 & 3.50 & 3.00 & 3.75 & 4.25 & \textbf{1.25}   \\
    	\bottomrule
    		\end{tabular}
    		}
    	\end{center}
    \label{tab:mfrl}
    \end{table}

\section{Related Work}
\label{sec:related_work}

    Robust Reinforcement Learning (Robust RL) has recently become a popular topic \citep{iyengar2005robust, dulac-arnold2019challenges, kirk2022survey, moos2022robust}, due to its effectiveness in tackling perturbations. Besides the transition perturbation in this paper, there are other branches relating to action, state and reward. We will briefly discuss them in the following paragraphs. Additionally, we will discuss the relation of Robust RL, sim-to-real and Bayesian RL approaches, which are also important topics in robot learning.

   \paragraph{Action Perturbation:} Early works in Robust RL concentrated on action space perturbations. \citet{pinto2017robust} firstly proposed an adversarial agent perturbing the action of the principle agent, training both alternately in a mini-max style. \citet{tessler2019action} later performed action perturbations with probability $\alpha$ to simulate abrupt interruptions in the real world. Afterwards, \citet{kamalaruban2020robust} analyzed this mini-max problem from a game-theoretic perspective and claimed that an adversary with mixed strategy converges to a mixed Nash Equilibrium. Similarly, \citet{vinitsky2020robust} involved multiple adversarial agents to augment the robustness, which can also be explained in the view of a mixed strategy.

   \paragraph{State Perturbation:} State perturbation can lead to the change of state from $s$ to $s_p$, and thus might worsen agent's policy $\pi(a|s)$ \citep{pattanaik2017robust}. \citet{zhang2021robust, oikarinen2021robust} both assume an $L_p$-norm uncertainty set on the state space (inspired by the idea of adversarial attacks widely used in computer vision \citep{szegedy2014intriguing}) and propose an auxiliary loss to encourage learning to resist such attacks. It is worth noting that state perturbation is a special case of transition perturbation, and that state perturbations with fixed transitions can be absorbed into transition perturbations, i.e., $\mathcal{P}(s'_p|s,a;\bar{w})=\mathcal{P}(s'|s,a;w)$, which can be covered by the framework proposed in this paper.

   \paragraph{Reward Perturbation:} The robustness-regularization duality has been widely studied, especially when considering reward perturbations \citep{husain2021regularized, eysenbach2021maximum, brekelmans2022your}. One reason is that the policy regularizer is closely related to a perturbation on the reward function without the need for a rectangular uncertainty assumption. However, it restricts the scope of these works as reward perturbation, since it can be shown to be a particular case of transition perturbation by augmenting the reward value in the state \citep{eysenbach2021maximum}. Besides, the majority of works focus on the analysis of regularization to robustness, which can only analyze the effect of existing regularizers instead of deriving novel regularizers for robustness as in the work we present here.

    \paragraph{Sim-to-Real:} Sim-to-Real is a key research topic in robot learning. Compared to the Robust RL problem, it aims to learn a robust policy from simulations for generalization in real-world environments. Domain randomization is a common approach to ease this mismatch in sim-to-real problems \citep{peng2018sim, openai2019learning}. However, \citet{mankowitz2019robust} has demonstrated that it actually optimizes the average case of the environment rather than the worst-case scenario (as seen in our research), which fails to perform robustly during testing. More recent active domain randomization methods \citep{mehta2020active} resolve this flaw by automatically selecting difficult environments during the training process. The idea of learning an adversarial uncertainty set considered in this paper can be seen as a strategy to actively search more valuable environments for training.

    \paragraph{Bayesian RL:} One commonality between Bayesian RL and Robust RL is that they both store uncertainties over the environmental parameter (posterior distribution $q(w)$ in Bayesian RL and uncertainty set $\Omega_w$ in Robust RL). Uncertainties learned in Bayesian RL can benefit Robust RL in two ways: (1) Robust RL can define an uncertainty set $\Omega_w = \{w: q(w) > \alpha \}$ to learn a robust policy that can tolerate model errors, which is attractive for offline RL and model-based RL; (2) A soft robust objective with respect to the distribution $q(w)$ can ease the conservative behaviors caused by the worst-case scenario \citep{dermansoftrobust}.

\section{Conclusion}
\label{sec:conclusion}
    In this paper, we adopt the robustness-regularization duality method to design new regularizers in continuous control problems to improve the robustness and generalization of RL algorithms. Furthermore, to deal with unknown uncertainty sets, we design an adversarial uncertainty set according to the learned action state value function and incorporate it into a new regularizer. While further validation on real world systems is necessary, the proposed method already showed great promise regarding generalization and robustness under environmental perturbations unseen during training, which makes it a valuable addition to RL for robot learning.

\section{Limitations}
\label{sec:limitations}
    The main limitation of the proposed method is possibly the efficiency of the uncertainty set regularizer. Depending on the derivative of the model parameters, it is expensive to compute for environments with a large number of parameters. To address this problem, a workable method is to detect important variables automatically and only apply regularization on these dimensions. Besides, while USR performs well on complex simulated environments in this paper, it should involve more tests on real-world robots in the future.

\clearpage

\acknowledgments{This research receives funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 953348 (ELO-X). The authors thank Jasper Hoffman, Baohe Zhang for the inspiring discussions.}