\section{Introduction}\label{sec_intro}
We often need to estimate the \emph{maximum expected value} of a set of random variables (RVs), when only noisy estimates for each of the variables are given.\footnote{Without loss of generality, we assume that we want to \emph{maximize} rather than \emph{minimize}.}
For instance, this problem arises in
optimization in stochastic decision processes and in algorithmic evaluation.

Formally, we consider a finite set $V = \{ V_1, \ldots, V_M \}$ of $M \geq 2$ independent RVs with finite means $\mu_1, \ldots, \mu_M$ and variances $\sigma^2_1, \ldots, \sigma^2_M$. We want to find the value of $\mu_*(V)$, defined by
\begin{equation}\label{maxE}
\mu_*(V) \equiv \max_i \mu_i \equiv \max_i E \{ V_i \} \enspace.
\end{equation}
We assume the distribution of each $V_i$ is unknown, but a set of noisy samples $X$ is given. The question is how best to use the samples to construct an estimate $\hat{\mu}_*(X) \approx \mu_*(V)$. We write $\mu_*$ and $\hat{\mu}_*$ when $V$ and $X$ are clear from the context. It is easy to construct consistent estimators, but we are also interested in the quality for small sample sizes. The mean squared error (MSE) is the most common metric for the quality of an estimator, but sometimes (the sign of) the bias is more important. Unfortunately, as we discuss in Section \ref{sec_bias}, no unbiased estimators for $\mu_*$ exist.

A common estimator is the \emph{maximum estimator} (ME), which constructs estimates $\hat{\mu}_i \approx \mu_i$ and then uses $\hat{\mu}_* \equiv \max_i \hat{\mu}_i$. When $X_i \subset X$ contains direct samples for $V_i$, and $\hat{\mu}_i$ is the average of $X_i$, the ME is simply the maximum sample average.
The ME on average overestimates $\mu_*$. This bias has been rediscovered several times, for instance in economics \cite{Steen:2004} and decision making \cite{Smith:2006}. It can lead to overestimation of the performance of algorithms \cite{Varma:2006,Cawley:2010}, and poor policies in reinforcement learning algorithms \cite{vanHasselt:2011,vanHasselt:2011phd}.
It is related to \emph{over-fitting} in model selection, \emph{selection bias} in sample selection \cite{Heckman:1979} and feature selection \cite{Ambroise:2002}, and the \emph{winner's curse} in auctions \cite{CapenClappCampbell:1971}.

The most common alternative to avoid this
bias is \emph{cross validation} (CV) \cite{Larson:1931,Mosteller:68}. If CV is used to construct each $\hat{\mu}_i$, and thereafter to estimate $\mu_*$ (as described in Section \ref{sec_CV}), this is called nested CV or ``double cross'' \cite{Stone:74}. Unfortunately, (nested)
CV can lead to a large variance. Perhaps surprisingly,
we show the absolute bias of CV can be larger than
the bias of the ME that we are trying to prevent.
However, the bias of CV is provably negative, which can
be an advantage.

In this paper, we give general distribution-independent bounds for the bias
and variance of the ME and CV. We present a new variant of
CV and show that it is very dependent
on $V$ which CV estimator is most accurate in terms
of MSE. Therefore, it is non-trivial to construct accurate
CV estimators without some knowledge about the
distributions of $V$. We discuss why standard
10-folds CV is often not a bad choice for model selection,
but show that in other settings other estimators may be
much more accurate.

We now discuss two motivating examples to highlight the practical importance of this general topic.

\paragraph{Learning Algorithms}
Many learning algorithms explicitly maximize noisy values to update their internal parameters. For instance, in reinforcement learning \cite{Sutton:98book} the goal is to find strategies that maximize a reward signal in a (sequential) decision task. Inaccurate biased estimators for $\mu_*$ can have adverse effects on the speed of learning and the strategies that are learned
\cite{vanHasselt:2011}.

\paragraph{Evaluation of Algorithms}
Most machine-learning algorithms have tunable parameters. \emph{Internal parameters}, such as the Lagrangian multipliers of a support vector machine (SVM) \cite{Vapnik:95book}, are optimized by the algorithm. \emph{Hyper-parameters}, such as the choice of kernel function in a SVM, are often tuned manually or chosen with domain knowledge.
Other relevant choices by the experimenter---such as which algorithms to consider and the representation of the problem---can be summarized as \emph{meta-parameters}.

Typically, we evaluate a set $C$ of configurations, where each $c_i \in C$ denotes a specific algorithm with specific hyper- and meta-parameters. Often, each evaluation is noisy, due to (pseudo-)randomness in the algorithms or inherent randomness in the problem. The performance of $c_i$ is then a random variable $V_i$, and we want to construct an estimate $\hat{\mu}_*$ for the best performance $\mu_*$ñ.\footnote{Sometimes we are more interested in the configuration that optimizes the performance than in the resulting performance, but often the performance itself is at least as important. In part, this depends on whether the focus of the research is on the algorithms or on the problem.}

For instance, \citet{Varma:2006} note that the ME results in
overly-optimistic prediction errors and propose to use nested CV. They evaluate an SVM for various hyper-parameters on an artificial problem, with an actual error of 50\%. The estimate by the ME is 41.7\% and nested CV results in $54.2\%$. Varma and Simon argue that the latter exceeds 50\% because nested CV removes a sample from each training set. However, in fact the difference between 50\% and $54.2\%$ is a demonstration of a completely different general bias that we discuss in Section \ref{sec_CV}.
This bias has received very little attention, although---as we will show---it is not in general smaller than the bias caused by using the ME.

\paragraph{Overview}
In the rest of this section, we discuss related work and (notational) preliminaries.
In Section \ref{sec_bias}, we discuss the bias of estimators in general. In Section \ref{sec_estimators}, we discuss the properties of the ME and of CV, including bounds on their bias and variance. In Section \ref{sec_examples} we discuss concrete settings with empirical
illustrations. Section \ref{sec_disc} contains a discussion and Section \ref{sec_conc}
concludes the paper.

\subsection{Related Work}
The \emph{bootstrap} \cite{Efron:1993} is a resampling method that can be used to estimate the bias of an estimator, in order to reduce this bias. Based on this, Tibshirani and Tibshirani \cite{Tibshirani:2009} propose an estimator for $\mu_*$ for model selection in classification. Inevitably---see Section \ref{sec_nonexistence}---the resulting estimate is still biased, and it is typically more variable that the original estimate. Also specifically considering model selection for classifiers, Bernau et al. \cite{Bernau:2011} propose a smoothed version of nested CV. The resulting estimator performs similar to normal (nested) CV, which in turn is shown to typically be more accurate than the approach by Tibshirani and Tibshirani. In this paper we focus on CV and the ME, which are by far the most widely used.

The problem of estimating $\mu_*$ is related to the \emph{multi-armed bandit} framework \cite{robbins1952some,Berry_bandit:85}, where the goal is to find the \emph{identity} of the action with the maximum expected value rather than its \emph{value}. The focus in the literature on multi-armed bandits is often on how best to collect samples. In contrast, in this paper we assume that a set of samples is given. A discussion on how best to collect samples to minimize the bias or MSE is outside the scope of this paper, although we do note that minimizing online regret \cite{Lai:1985,Auer:2002} does not necessarily correspond to minimizing the online MSE of the estimator.

\subsection{Preliminaries}\label{sec_prelim}

The measurable domain of $V_i$ is $\mathcal{X}_i$, and $f_i : \mathcal{X}_i \to \mathbb{R}$ denotes its probability density function (PDF), such that $\mu_i = \intg{x\;f_i(x)}{x}{\mathcal{X}_i}{}$. For conciseness, we assume $\mathcal{X}_i = \mathbb{R}$.
We assume these PDFs $f_i$ are unknown and therefore $\mu_* = \max_i \mu_i$ can not be found analytically.

We write $\hat{\mu}_i(X)$ for an estimator for $\mu_i$ based on a sample set $X$. Similarly, $\hat{\mu}_*(X)$ is an estimator for $\mu_*$. We write $\hat{\mu}_i$ and $\hat{\mu}_*$ when $X$ is clear from the context. If $X_i \subset X$ is a set of unbiased samples for $V_i$, $\hat{\mu}_i$ might be the sample average.
In that case, $\hat{\mu}_i$ is unbiased for $\mu_i$. In general, $\hat{\mu}_i$ can be biased for $\mu_i$. As discussed in the next section, no general unbiased estimator for $\mu_*$ exists, even if all $\hat{\mu}_i$ are unbiased.

The following definitions will be useful below, when stating necessary and sufficient conditions for a strictly positive or a strictly negative bias. The set of \emph{optimal} indices for RVs $V$ is defined as
\begin{equation}\label{chEB_eq_optimal_estimator}
\mathcal{O}(V) \equiv \left\{ i \,\left|\, \mu_i = \mu_* \right. \right\} \enspace.
\end{equation}

The set of \emph{maximal} indices for samples $X$ is defined as
\begin{equation}\label{max_indices}
\mathcal{M}(X) \equiv \left\{ i \,\left|\, \hat{\mu}_i(X) = \max_j \hat{\mu}_j(X) \right. \right\} \enspace.
\end{equation}
An estimator is called optimal or maximal whenever its index is optimal or maximal, respectively. Note that optimal estimators are not necessarily maximal and maximal estimators are not necessarily optimal.

\section{The Bias of an Estimator}\label{sec_bias}
Let $\mathcal{V}$ be a function space containing all admissible sets of $M$ RVs. We might know $\mathcal{V}$, but not the precise identity of $V \in \mathcal{V}$. For instance, $\mathcal{V}$ may be the set of all sets of $M$ normal RVs with finite moments.
Let $p : \mathcal{V} \to \mathbb{R}$ be a PDF over $\mathcal{V}$.
The expected MSE of an estimator $\hat{\mu}_*$ is equal to
\begin{equation}\label{MSEV}
\intg{ \! p(V) \intg{ \! P(X|V) \left( \hat{\mu}_*(X) - \mu_* \right)^2 }{X\!}{X}{} }{ V\! }{ \mathcal{V} }{} ,
\end{equation}
In any given concrete setting, there is a single unknown set $V$. Therefore, $p$ does not exist `in the world'. Rather, $p$ might model our prior belief about which sets $V$ are likely in a given setting, or it might specify the $V$ for which we would like an estimator to perform well.
The MSE consists of variance and bias.
To reason in some generality about which estimators are good in practice, we discuss the \emph{non-existence of unbiased estimators} and the \emph{direction of the bias}.

\paragraph{Non-Existence of Unbiased Estimators}\label{sec_nonexistence}
By definition, $\hat{\mu}_*$ is a general unbiased estimator (GUE) for $\mathcal{V}$ if and only if
\begin{equation}\label{GUE}
\forall V \in \mathcal{V} :  \ex{\hat{\mu}_* }{ V } = \mu_* \enspace.
\end{equation}
Unfortunately, for most $\mathcal{V}$ of interest no such estimator exists.
For instance, \citet{Blumenthal:1968} show no GUE exists for two normal distributions  and \citet{Dhariyal:1985} proved this for arbitrary $M\geq 2$ and for more general distributions, including the exponential family. Essentially, the argument is that a reasonable estimator for $\mu_*$ depends smoothly on the values of the samples, whereas the real value $\mu_*$ is a piece-wise linear function with a discontinuous derivative. We can not know the location of these discontinuities without knowing the actual maximum.

Note that \eq{GUE} is already false if $\mathcal{V}$ contains only a single set of variables for which $\hat{\mu}_*$ is biased.
However, bias alone does not tell us everything, and a low bias does not necessarily imply a small expected MSE.

\paragraph{The Direction of the Bias}
In some cases, the direction of the bias is very important.
Suppose we test an algorithm for various hyper-parameters and observe that the best performance is better than some baseline. If we simply use the highest test result, it can not be concluded that the algorithm can really structurally outperform the baseline for any of the specific hyper-parameters. Although this may sound trivial, it is common in practice: when we manually tune hyper- or meta-parameters on a problem and use the best result, we are using $\max_i \hat{\mu}_i$, which has non-negative bias. It is hard to avoid optimizing on meta-parameters: these include the very (properties of the) problem we test the algorithm on.

The practical implication of this positive bias is that the algorithm will disappoint in future evaluations on similar (real-world) problems.
In contrast, if we use an estimator with \emph{non-positive bias} and our estimate is higher than the baseline, we can have much more confidence that the algorithm can reach that performance consistently with a properly tuned hyper-parameter.
This is similar to the considerations about overfitting in model selection, where CV is most often used. We prove below that CV indeed has non-positive bias, and can therefore avoid overestimations of $\mu_*$.

As another example, the performance of most machine-learning algorithms improves when more data is available. When the data collection is expensive it is useful to predict how an algorithm performs when more data is available, before actually collecting this data. An overestimation of the future performance can lead to a misallocation of resources, since the collected data may be less useful than predicted. An underestimation means we may be too pessimistic, and too often decide not to collect more data. Whether or not the false positives are more important than the false negatives depends crucially on specifics of the setting.

\section{Estimators for the Maximum Expected Value}\label{sec_estimators}
In this section, we discuss the ME and CV estimators for $\mu_*$ in detail. We bound the biases and variances of all estimators, discuss similarities and contrasts, and prove consistency. We introduce a low-variance variant of CV. We give necessary and sufficient conditions for non-zero biases to occur for all estimators, and perhaps surprisingly we show that there are settings in which the negative bias of all variants of CV is larger in size than the positive bias of ME. All proofs are given in an appendix and the end of this paper.

\subsection{The Maximum-Estimator Estimator}\label{sec_ME}
The maximum-estimator (ME) estimator for $\mu_*$ is
\begin{equation}\label{ME_est}
\est^{\mbox{\tiny ME}}_*(X) \equiv \max_i \hat{\mu}_i(X) \enspace,
\end{equation}
where $\hat{\mu}_i$ is a (possibly biased) estimator for $\mu_i$.
Because it is conceptually simple and easy to implement, the ME estimator is often used in practice.
The theorem below proves its bias is non-negative and gives necessary and sufficient conditions for a strictly positive bias. The theorem is stronger and more general than some similar earlier theorems. For instance, \citet{Smith:2006} do not consider the possibility of multiple optimal variables, and do not discuss necessity of the conditions for a strictly positive bias.

\begin{theorem}\label{theorem_overest}
For any given set $V$, $M\geq 1$ and unbiased estimators $\hat{\mu}_i$, $\ex{\hat{\mu}_i}{V} = \mu_i$,
\[
\ex{ \est^{\mbox{\tiny ME}}_* }{ V }
\geq \mu_* \enspace,
\]
with equality if and only if all optimal indices are maximal with probability one.
\end{theorem}

Theorem \ref{theorem_overest} implies a lower bound of zero for the bias of the ME.
An upper bound for arbitrary means and variances is given by \citet{Aven:1985}:
\begin{equation}\label{ME_bound}
\bias{\est^{\mbox{\tiny ME}}_*} \leq \sqrt{\frac{M-1}{M} \sum_{i}^M \var{ \hat{\mu}_i }} \enspace,
\end{equation}
which is tight when the estimators are iid \citep{Arnold:1979}, indicating that iid variables are a worst-case setting.

We do not know of previous work that bounds the variance, which we discuss next.
\begin{theorem}\label{ME_variance_bound}
The variance of the ME estimator is bounded by
$
\var{ \est^{\mbox{\tiny ME}}_* } \leq \sum_{i=1}^M \var{ \hat{\mu}_i }
$.
\end{theorem}

Theorem \ref{ME_variance_bound} and bound \eq{ME_bound} imply that $\est^{\mbox{\tiny ME}}_*$ is consistent for $\mu_*$ whenever each $\hat{\mu}_i$ is consistent for $\mu_i$ and that

\mbox{MSE}(\est^{\mbox{\tiny ME}}_*) < 2 \sum_{i=1}^M \var{ \hat{\mu}_i }
$

\subsection{The Cross-Validation Estimator}\label{sec_CV}
In general, $\mu_*$ can be considered to be a weighted average of the means of all optimal variables:

\mu_* = \frac{1}{|\mathcal{O}(V)|}\sum_{i=1}^M \mathcal{I}(i \in \mathcal{O}(V)) \mu_i
$,
where $\mathcal{I}$ is the indicator function.
We do not know $\mathcal{O}(V)$ and $\mu_i$, but with sample sets $A,B$ we can approximate these with $\mathcal{M}(A)$ and $\hat{\mu}_i(B)$ to obtain
\begin{equation}\label{maximal_weights}
\mu_* \approx \hat{\mu}_* \equiv \frac{1}{|\mathcal{M}(A)|}\sum_{i=1}^M \mathcal{I}(i \in \mathcal{M}(A)) \hat{\mu}_i(B) \enspace.
\end{equation}
If $A = B = X$, this reduces to $\est^{\mbox{\tiny ME}}_*$. However, suppose that $A$ and $B$ are independent.
This idea leads to \emph{cross-validation} (CV) estimators.
Of course, CV itself is not new.
However, it seems to be
less well-known how properties of the problem affect
the accuracy and that CV can be quite biased.

We split each $X$ into $K$ disjoint sets $X^k$ and define $\hat{\mu}^k_i \equiv \hat{\mu}_i(X^k)$. For instance, $\hat{\mu}^k_i$ might be the sample average of $X^k_i$.
We consider two different CV estimators. In both methods, for each $k \in \{ 1, \ldots, K \}$ we construct an \emph{argument set} $\hat{a}^k$ and a \emph{value set} $\hat{v}^k$.

\emph{Low-bias cross validation} (LBCV) is the `standard' CV estimator, where $K-1$ sets are used to build the argument set $\hat{a}^k$ (the model), and the remaining set is used to determine its value:
\begin{align*}
\hat{a}^k_i &\equiv \hat{\mu}_i(X \setminus X^k) \quad\mbox{and}
&\hat{v}^k_i &\equiv \hat{\mu}_i(X^k) \equiv \hat{\mu}^k_i \enspace.
\end{align*}
\emph{Low-variance cross validation} (LVCV) reverses the definitions for $\hat{a}^k$ and $\hat{v}^k$:
\begin{align*}
\hat{a}^k_i &\equiv \hat{\mu}(X_i^k) \equiv \hat{\mu}^k_i \quad\mbox{and}
&\hat{v}^k_i &\equiv \hat{\mu}(X_i \setminus X_i^k) \enspace.
\end{align*}
We do not know of any previous work that discusses this variant. However, its lower variance can sometimes result in much lower MSEs than obtained by LBCV. For both LBCV and LVCV, if $\hat{\mu}_i(X)$ is the sample average of $X_i$ and all samples are unbiased, then $E \{ \hat{a}^k_i \} = E \{ \hat{v}^k_i \} = \mu_i$.

For either approach, $\mathcal{M}^k$ is the set of indices that maximize the argument vector.
For LBCV this implies $\mathcal{M}^k = \mathcal{M}(X \setminus X^k)$ and for LVCV this implies $\mathcal{M}^k = \mathcal{M}(X^k)$.
We find the value of these indices with the value vector, resulting in
\begin{equation}\label{CV_estk}
\hat{\mu}^k_* \equiv \frac{1}{|\mathcal{M}^k|} \sum_{i \in \mathcal{M}^k} \hat{v}^k_i
\enspace.\end{equation}
We then average over all $K$ sets:
\begin{equation}\label{CV_est}
\est^{\mbox{\tiny CV}}_* \equiv \frac{1}{K} \sum_{k=1}^K \hat{\mu}^k_* =  \frac{1}{K} \sum_{k=1}^K \frac{1}{|\mathcal{M}^k|} \sum_{i\in \mathcal{M}^k}^M \hat{v}^k_i \enspace,
\end{equation}
where either $\est^{\mbox{\tiny CV}}_* = \est^{\mbox{\tiny LBCV}}_*$ or $\est^{\mbox{\tiny CV}}_* = \est^{\mbox{\tiny LVCV}}_*$, depending on the definitions of $\hat{a}^k_i$ and $\hat{v}^k_i$.

The construction of $\hat{\mu}^k_*$ performs the approximation:
$
\hat{v}^k_i \approx \hat{v}^k_{i_*} \approx \mu_{i_*} \equiv \mu_*
$,
where $i \in \mathcal{M}^k$ and $i^* \in \mathcal{O}$.
The first approximation results from using $\mathcal{M}^k$ to approximate $\mathcal{O}$ and is the main source of bias. The second approximation results from the variance of $\hat{v}^k$.

For large enough $X$, $K$ can be treated as a parameter that trades off bias and variance. For LBCV larger $K$ implies less bias and more variance, while for LVCV it implies more bias and less variance. For $K=2$, LBCV and LVCV are equivalent. If $K>2$, LVCV is more  biased but less variable than LBCV, since $\mathcal{M}^k$ is then based on fewer samples while $\hat{v}^k_i$ is based on more samples. When $\forall i : |X_i| = K$ for LBCV, $\hat{v}^k_i$ is based on a single sample, resulting in a large variance. This variant is commonly known as \emph{leave-one-out} CV. When $\forall i : |X_i| = K$ for LVCV, $\hat{a}^k_i$ is based on a single sample, potentially resulting in large bias due to large probabilities of selecting sub-optimal indices.

The bias of CV for $\mu_*$ has received comparatively little attention. Sometimes  the bias is mentioned without explanation \citep{Kohavi:1995}, and sometimes it is even claimed that CV is unbiased \citep{Mannor:2007}. Often, any observed bias is attributed to the fact that $\hat{a}^k$ can be biased when it based on $\frac{K-1}{K}|X|$ rather than $|X|$ samples \citep{Varma:2006}. This can be a factor, but the bias induced by using $\mathcal{M}^k$ for $\mathcal{O}$ is often at least as important, as will be demonstrated below.  Some confusion seems to arise from the fact that $\hat{v}^k_i$ is often unbiased for $\mu_i$. Unfortunately, this does not imply that $\est^{\mbox{\tiny CV}}_*$ is unbiased for $\mu_*$.

Next, we prove that CV estimators can have a negative bias even if $\hat{a}$ and $\hat{v}$ are unbiased, and we give necessary and sufficient conditions for a strictly negative bias.
\begin{theorem}\label{CV_underest_theorem}
If $\ex{\hat{\mu}^k_i}{V} = \mu_i$ is unbiased then

\ex{\est^{\mbox{\tiny CV}}_*}{V}
\leq \mu_*
$
is negatively biased,
with a strict inequality if and only if there is a non-zero probability that any non-optimal index is maximal.
\end{theorem}
The theorem shows that $\est^{\mbox{\tiny LVCV}}_*$ and $\est^{\mbox{\tiny LBCV}}_*$ on average underestimate $\mu_*$ if and only if there is a non-zero probability that $i \in \mathcal{M}^k(X)$ for some $i \notin \mathcal{O}(V)$. A prominent case in which this does \emph{not} hold is when all variables have the same mean, since then $i \in \mathcal{O}(V)$ for all $i$. Interestingly, this implies that CV is unbiased when the $V_i \in V$ are iid, which is a worst case for the ME.
Theorem \ref{CV_underest_theorem} implies that the bias of CV is bound from above by zero. We conjecture the bias is bound from below as follows.

\begin{conjecture}\label{CV_bound}
Let $\ex{\hat{\mu}^k_i}{V} = \mu_i$.
Then
\
\bias{\est^{\mbox{\tiny CV}}_*} > - \frac{1}{K} \sum_{k=1}^K \sqrt{ \sum_{i=1}^M \var{\hat{a}^k_i} }
\enspace.\
\end{conjecture}
We do not prove this conjecture here in full generality, but there is a proof for $M=2$ in the appendix.
It makes intuitive sense that the bias of $\est^{\mbox{\tiny CV}}_*$ depends only on the variances $\var{\hat{a}^k_i}$ if each $\hat{\mu}^k_i$ is unbiased.
The bias of the CV estimators is unaffected by the fact that $\est^{\mbox{\tiny CV}}_*$ averages over $K$ estimators $\hat{\mu}^k_*$, but $K$ does affect the bias by regulating how many samples are used for each $\hat{a}^k_i$. As mentioned earlier, for LVCV larger a $K$ implies a higher bias since then $\hat{a}^k_i$ is \emph{more} variable, while for LBCV a larger $K$ implies a lower bias since then $\hat{a}^k_i$ is \emph{less} variable.

Although CV is known for low
bias and high variance, the next theorem shows its absolute bias is
not necessarily smaller than the absolute bias of the ME.
\begin{theorem}
There exist $V$ and $N = |X|$ such that $|\biasc{\est^{\mbox{\tiny CV}}_*}{N}|>|\biasc{\est^{\mbox{\tiny ME}}_*}{N}|$ for any $K$ and for any variant of CV.
\end{theorem}
Two different experiments in Section \ref{sec_examples} prove this theorem, since there even the negative bias of leave-one-out LBCV is larger in size than the positive bias of ME.

\begin{theorem}\label{CV_variance_bound}
The variance of $\est^{\mbox{\tiny LBCV}}_*$ is bounded by
\begin{equation*}
\var{ \est^{\mbox{\tiny LBCV}}_* } \leq \frac{1}{K^2} \sum_{k=1}^K \sum_{i=1}^M \var{\hat{\mu}^k_i}
\enspace.
\end{equation*}
\end{theorem}
If each $\hat{\mu}^k_i$ is unbiased, the variance of LVCV is necessarily smaller than that of LBCV and the same bound applies trivially to $\est^{\mbox{\tiny LVCV}}_*$.
\begin{corollary}\label{CV_variance_av}
If $\hat{\mu}_i$ is the sample average of $X_i$ and $|X^k_i| = |X_i|/K$ for all $k$, then

\var{ \est^{\mbox{\tiny CV}}_* } \leq \sum_{i=1}^M \var{\hat{\mu}_i}
$ for both LBCV and LVCV
\end{corollary}
Conjecture \ref{CV_bound} and Theorem \ref{CV_variance_bound} imply that CV is consistent if each $\hat{\mu}_i$ is consistent and $K$ is fixed (or slowly increasing, see also \citet{Shao:1993}), and that

\mbox{MSE}(\est^{\mbox{\tiny CV}}_*) \leq 2 \sum_{i=1}^M \var{\hat{\mu}_i}
$

\section{Concrete Illustrations}\label{sec_examples}
To illustrate that it is non-trivial to select an accurate
estimator, we discuss some concrete examples.

\subsection{Multi-Armed Bandits for Internet Ads}
The framework of multi-armed bandits can be used to optimize
which ad is shown on a website
\citep{Langford:2008,Strehl:2010}. Consider $M$ ads with unknown fixed expected returns per visitor $\mu_i$. Bandit algorithm can be used to balance exploration and exploitation to optimize the online return per visitor, which converges to $\mu_*$. However, quick accurate estimates of $\mu_*$ can be important, for instance to base future investments on. Additionally, placing any ad may induce some cost $c$, so we may want to know quickly whether $\mu_* > c$.

\begin{figure*}[t]
\begin{center}
\includegraphics[scale=0.52]{all_100K_2K}
\caption{\label{fig_ads} The MSE for $\est^{\mbox{\tiny ME}}_*$, $\est^{\mbox{\tiny LBCV}}_*$ and $\est^{\mbox{\tiny LVCV}}_*$ for different settings, averaged over 2,000 experiments. The left-most bar is always $\est^{\mbox{\tiny ME}}_*$. The other bars are, from left to right, leave-one-out LVCV, 10-folds LVCV, 5-folds LVCV, 2-folds CV, 5-folds LBCV, 10-folds LBCV and leave-one-out LBCV. Note that 2-folds LVCV is equivalent to 2-folds LBCV, which are therefore not shown separately.}
\end{center}
\end{figure*}

For simplicity, assume each ad has the same return per click, such that only the click rate matters and each $V_i$ can be modeled with a Bernoulli variable with mean $\mu_i$ and variance $(1-\mu_i)\mu_i$. In our first experiment, there are $N=100,000$ visitors, $M=10$, $M=100$ or $M=1000$ ads, and $\forall i: \mu_i = 0.5$. All ads are shown equally often, such that $\forall i: N_i = N/M$. Because all means are equal, Theorem \ref{CV_underest_theorem} implies that CV estimators are unbiased; their MSE depends solely on the variance. In the second---more realistic---setting, the $M$ mean click rates are distributed evenly between $0.02$ and $0.05$, there are $N=300,000$ visitors, and $M=30$, $M=300$, or $M=3000$ ads.

The results are shown in the first four plots in Figure \ref{fig_ads}. We show the root MSE (RMSE), such that the units are percentage points. Within the RMSEs, the contributions of the bias and the variance are shown.
Note that $\mbox{MSE} = \mbox{bias}^2 + \mbox{variance}$, and therefore $\mbox{RMSE} = \sqrt{\mbox{bias}^2 + \mbox{variance}} \neq \mbox{bias} + \mbox{std dev}$.
This implies that the depicted contributions of bias and variance to the RMSE are not in general exactly equal to the bias and standard deviation, but this depiction does allow us to see directly how many percentage points of error are caused by bias and by variance.

In the first setting (left plot) CV is indeed unbiased. Leave-one-out LVCV has the lowest variance of all CV methods---it is barely visible---which implies it has the smallest MSE. For $M=1000$ ads, the huge bias of the ME causes it to overestimate the actual maximal click rate by more than 15\%.

In the second setting (middle three plots), there is a clear trade-off in CV: LVCV with large $K$ has large bias and small variance, whereas LBCV with large $K$ has small bias and large variance.\footnote{Sometimes the bias of LBCV seems to increase slightly for higher $K$. These are noise-related artifacts.} The bias of the CV estimators is clearly important, even though each $\hat{\mu}^k_i$ is unbiased. Even for leave-one-out LBCV the bias is non-negligible: for $M=30$ its bias is larger than the bias of the ME. Interestingly, when $M$ increases (and the number of samples per ad decreases correspondingly) the error for leave-one-out LVCV stays virtually unchanged, at approximately 1.3\%. Since the error of all other estimators increases with increasing $M$, this implies that leave-one-out LVCV goes from being by far the least accurate for $M=30$ to almost the most accurate for $M=3000$. In contrast, the ME goes from being the most accurate for $M=30$ to the least accurate for $M=3000$. The reason is that for increasing $M$, the variables are relatively more similar to iid variables, which is a best case for LVCV and a worst case for the ME. In all three cases, 10- and 5-folds LVCV are a good choice.

\subsection{Evaluation of Algorithms
We now consider a regression problem. The goal is to fit polynomials on noisy samples from a function $r(y) = 4( \sin(y) + \sin(2y) )$. Let $X = \{ (y,r(y) + \omega) \,|\, y \in Y \}$ denote a noisy data set for inputs $Y$, where $\omega$ is zero-mean Gaussian noise with variance $\sigma^2_\omega = 4$.
Let $p_i$ denote a polynomial of degree $i$, of which the coefficients are fitted with least-squares on $X$.

Let $Y = \{ 0, 0.05, \ldots, 3.95, 4 \}$ be 81 equidistant inputs.
We want to maximize the negative MSE. The lowest expected MSE of fitting each $p_i$ on 81 samples
and testing on an independent test set of 81 samples
is obtained at $4.34$ for $i=5$, which implies $\mu_* = \mu_5 = -4.34$.
We construct 1,000 independent noisy sets $X = \{ (y,r(y) + \omega) \,|\, y \in Y \}$. For each $X$, we conduct the following experiment.

For any given $Z \subseteq X$, $\hat{\mu}_i$ is defined by an inner CV loop as follows. For each $z \in Z$, we fit $p_i$ on $Z \setminus \{z\}$ and test the error on $z$ to obtain an error $e_i(z)$. We average these errors to obtain: $\hat{\mu}_i(Z) = \frac{1}{|Z|} \sum_{z \in Z} e_i(z)$.  This implies $\hat{\mu}_i$ is biased, since $p_i$ is fitted on $|Z|-1 < 81$ samples.
For the ME, $\est^{\mbox{\tiny ME}}_* = \max_i \hat{\mu}_i(X)$ which means $|Z| = 80$ samples are used to fit each $p_i$. For LBCV, $\hat{a}^k_i = \hat{\mu}_i(X \setminus X^k)$ which means $|Z|=\frac{K-1}{K}81$. For LVCV, $\hat{a}^k_i = \hat{\mu}_i(X^k)$, which means $|Z| = \frac{1}{K}81$. Since $|Z|$ can then be much smaller than 81, LVCV can be significantly biased. We consider $K\in \{2,3,9,81\}$. When $K=81$, LBCV is also known as \emph{nested leave-one-out CV}. Figure \ref{fig_ads} (right plot) shows the results.

LVCV is not shown for $K=81$ and $K=9$: LVCV with $K=81$ is meaningless, since one cannot fit a polynomial on a single point. The MSE for $K=9$ is huge. In sharp contrast with the previous settings, LVCV fares poorly---even in terms of variance---and leave-one-out LBCV is the best CV estimator. However,  interestingly the ME is more accurate than all CV estimators, and even the size of its bias ($0.018$) is much smaller than that of $n$-fold LBCV ($-0.190$).

\section{Discussion}\label{sec_disc}
Our results show that it is hard to choose an estimator that is good in general. Unfortunately, the best choice in one setting can be the worst choice in another. A poorly chosen CV estimator can be far less accurate than the ME.  This does not imply that we suggest using the ME; it is often very biased.

A potential advantage of CV estimators in some settings is a
guaranteed non-positive bias. This can be
desirable even if the estimator is less accurate. However, in our results the recommendation to always use 10-folds LBCV \citep{Kohavi:1995} seems unfounded. When each $\hat{\mu}_i$ is unbiased and especially when $M$ is large, LVCV often performs much better. On the other hand, when each estimator $\hat{\mu}_i$ has a bias that decreases with the number of samples, the bias of LVCV can become prohibitively large, as illustrated in the regression setting. This explains why 10-folds LBCV is often not a bad choice for model selection, as long as $M$ is fairly small and $\hat{\mu}_i$ is fairly biased. However, note that 5- and 10-folds LBCV were the most accurate estimator in \emph{none} of our experiments.

As a general recommendation, it may be good to try both the ME and one or more CV estimators. If the estimates are close together, this indicates they are more likely to be accurate. Although the true maximum expected value will often lie between the estimate by the ME and those by CV, one should not simply average these estimates: as we have shown that for instance the ME can be very biased in some settings, and hardly biased in others. Furthermore, the potentially excessive variance of some variants of LBCV implies that in some cases its estimate may itself be an overestimation, which is why we recommend to include LVCV in the analysis.

\paragraph{Alternative estimators} Of course, there are possible alternatives to the estimators we discussed. First, one can consider using the maximum of some lower confidence bounds on the individual value estimates. Although this does counter the overestimation of ME, it can not be guaranteed that this does not lead to an underestimation in its place. Furthermore, it is non-trivial to select a good confidence interval, and the resulting estimate will typically be much more variable than the ME.

Second, for model-selection there exist criteria such as AIC \citep{Akaike:1974} and BIC \citep{Schwarz:1978} that use a penalty term based on the number of parameters in the model. Obviously, such penalties are only useful when comparing homomorphic models with different numbers of parameters, and therefore do not apply to the more general setting we consider in this paper. Furthermore, the main purpose of these criteria is not to give an accurate estimate of the expected value of the best model, but to increase the probability of selecting it. These goals are related, but unfortunately not equivalent.

Finally, one can estimate belief distributions $\hat{F}_i$ for the location of each $\mu_i$, for instance with Bayesian inference. With these distributions, we can estimate $\mu_*$. This approach is less general, since it requires prior knowledge about $\mathcal{V}$, but then it does seem reasonable. The probability that the maximum mean is smaller than some $x$ is equal to the probability that all means are smaller than $x$. Therefore, its CDF is $\hat{F}_{\max}(x) = \prod_{i=1}^M \hat{F}_i(x)$, which we can use to estimate $\mu_*$. The resulting Bayesian estimator (BE) is
\[
\hat{\mu}_*^{\text{BE}} = \intg{ x \sum_{i=1}^M \hat{f}_i(x) \prod_{j \neq i} \hat{F}_j(x) }{ x }{ -\infty }{ \infty } \enspace,
\]
where $\hat{f}_i(x) = \frac{d}{dx} \hat{F}_i(x)$.
To show a perhaps counter-intuitive result from this approach, we discuss a small example. Consider two Bernoulli variables. We consider all means equally likely and use a uniform prior Beta distribution, with parameters $\alpha = \beta = 1$. Suppose $\mu_1 = \mu_2 = 0.5$. We draw two samples from each variable. The expected estimate for the ME is $\frac{21}{32}$, for a bias of $\frac{5}{32} \approx 0.156$. CV is unbiased, since the means are equal. For the BE $\hat{F}_i(x)$ is $1 - (1-x)^3$, $3x^2 - 2x^3$ or $x^3$, depending on how many samples for $V_i$ are equal to one. Its expected value is then $\exn{ \hat{\mu}_*^{\text{BE}} } = \frac{737}{1120} \approx 0.658$. Note that the positive bias is even higher than the bias of the ME. This is due to our uniform prior: if the prior on the individual variables is uniform, this implies the prior for the maximum expected value is negatively skewed, and its expected value is increased. The effect is already apparent with two variables, but it increases further with the number of variables due to the shape of $\hat{F}_{\max}(x) = \prod_{i=1}^M \hat{F}_i(x)$.

\section{Conclusion}\label{sec_conc}
We analyzed the bias and variance of the two most common estimators for the maximum expected value of a set of random variables. The \emph{maximum estimate} results in non-negative bias. The common alternative of \emph{cross validation} (CV) has non-positive bias, which can be preferable. Unfortunately, the accuracies of different variants of CV are very dependent on the setting; an uninformed choice can result in extremely inaccurate estimates.
No general rule---e.g., always use 10-fold CV---is always optimal.

\subsection*{Appendix}
\begin{proof}[Proof of Theorem \ref{theorem_overest}]
For conciseness, we leave $V$ and $X$ implicit. Let $j \in \mathcal{O}$ be an arbitrary optimal index, and define event $A_j \equiv (j \in \mathcal{M})$ to be true if and only if $j$ is maximal. We can write
\[
\mu_* = P(A_j) \ex{ \hat{\mu}_j }{ A_j } + P(\neg A_j) \ex{ \hat{\mu}_j }{ \neg A_j } \enspace.
\]
Note: $\ex{ \hat{\mu}_j }{ A_j } = \ex{ \est^{\mbox{\tiny ME}}_* }{A_j}$ and $\ex{ \hat{\mu}_j }{ \neg A_j } < \ex{ \est^{\mbox{\tiny ME}}_* }{\neg A_j}$. Therefore,
$
\mu_* \leq \exn{ \est^{\mbox{\tiny ME}}_* }
$,
with equality if and only if $\prob{ \neg A_j } = 0$ for all $j \in \mathcal{O}$.
\end{proof}

\begin{proof}[Proof of Theorem \ref{ME_variance_bound}]
Let $A$ and $B$ be independent sets of RVs with $\exn{ A_i } = \exn{ B_i }$ and $\exn{ A_i^2 } = \exn{ B_i^2 }$. Define
\[
C^{(i)} \equiv
(A \setminus A_i) \cup \{ B_i \} =
\{ A_1, \ldots, A_{i-1}, B_i, A_{i+1}, \ldots, A_M \}
\enspace.\]
The Efron-Stein inequality \citep{EfronStein:1981} states that for any  $g : \mathbb{R}^M \to \mathbb{R}$:
\begin{equation*}
\mbox{Var}\left( g( A  ) \right) \leq \frac{1}{2} \sum_{i=1}^M \exn{ \left( g( A ) - g( C^{(i)} ) \right)^2 } \enspace.
\end{equation*}
Let $A$ and $B$ be independent instantiations of $\hat{\mu}$ and let $g( A ) = \max_i A_i$ for any $A$. We derive
\begin{align*}
& \var{ \est^{\mbox{\tiny ME}}_* }
\leq \frac{1}{2} \sum_{i=1}^M \exn{ \left( \max_j A_j - \max_j C^{(i)}_j \right)^2 } \\
& \leq \frac{1}{2} \sum_{i=1}^M \exn{ \left( A_i - B_i \right)^2 } = \sum_{i=1}^M \var{\hat{\mu}_i} \enspace.\qedhere
\end{align*}
\end{proof}

\begin{proof}[Proof of Theorem \ref{CV_underest_theorem}]
Let $w^k_i \equiv \exn{\mathcal{I}(i \in \mathcal{M}^k)/|\mathcal{M}^k|\,}$. Then $\exn{\hat{\mu}^k_*} = \sum_i^M w^k_i \mu_i \leq \mu_*$, because $\mathcal{M}^k$ and $\hat{v}^k_i$ are independent, $\sum_i^M w^k_i = 1$ and $\exn{\hat{v}^k_i} = \mu_i$.
Note that $w^k_i > 0$ if and only if $\prob{ i \in \mathcal{M}^k}>0$. Therefore, $\exn{\hat{\mu}^k_*} < \mu_*$ if and only if there exists a $i \notin \mathcal{O}$ such that $\prob{ i \in \mathcal{M}^k } > 0$.
\end{proof}

\begin{proof}[Proof of Conjecture \ref{CV_bound} for $M=2$]
Assume without loss of generality that $\mu_1 = \mu_*$. The assumption $\exn{\hat{\mu}^k_i} = \mu_i$ implies that $\exn{ \hat{v}^k_i } = \mu_i$. Then,
\begin{align*}
\bias{\hat{\mu}^k_*}
& = \exn{ \frac{ \mathcal{I}( 2 \in \mathcal{M}^k ) }{ |\mathcal{M}^k| } } \left( \mu_2 - \mu_1 \right) \\
& \geq \prob{ 2 \in \mathcal{M}^k } \left( \mu_2 - \mu_1 \right) \\
& = \prob{ \hat{a}^k_2 \geq \hat{a}^k_1 } \left( \mu_2 - \mu_1 \right) \\
& \geq \frac{ \left( \var{ \hat{a}^k_1 } + \var{ \hat{a}^k_2 } \right) \left( \mu_2 - \mu_1 \right)}{ \var{ \hat{a}^k_1 } + \var{ \hat{a}^k_2 } + ( \mu_1 - \mu_2 )^2 } \\
& \geq -\frac{1}{2} \sqrt{ \var{ \hat{a}^k_1 } + \var{ \hat{a}^k_2 } } \enspace,
\end{align*}
where the second inequality follows from Cantelli's inequality, and the third inequality is the result of minimizing for $\mu_2 - \mu_1$. From this, it follows that for $M=2$
\[
\bias{ \est^{\mbox{\tiny CV}}_* } \leq - \frac{1}{2K} \sum_{k=1}^K \sqrt{ \sum_{i=1}^2 \var{\hat{a}^k_i} } \enspace,
\]
which is a factor $\frac{1}{2}$ tighter than the general bound in the conjecture.
\end{proof}

\begin{proof}[Proof of Theorem \ref{CV_variance_bound}]
We apply definition \eq{CV_est} and use $\sum_{k=1}^K \hat{v}^k_i = \sum_{k=1}^K \hat{\mu}_i^k$ to derive
\begin{align*}
\var{ \!\frac{1}{K} \sum_{k=1}^K \frac{1}{|\mathcal{M}^k|} \sum_{i\in \mathcal{M}^k}^M \hat{v}^k_i \!}
& \leq \var{\! \frac{1}{K} \sum_{k=1}^K \sum_{i=1}^M \hat{v}^k_i \!} \\
& \leq
\frac{1}{K^2} \sum_{k=1}^K \sum_{i=1}^M \var{\hat{\mu}^k_i} \enspace.\qedhere
\end{align*}
\end{proof}

\begin{proof}[Proof of Corollary \ref{CV_variance_av}]
Apply Theorem \ref{CV_variance_bound} with

\var{\hat{\mu}^k_i} = \sigma_i^2/|X^k_i| = K \sigma_i^2/|X_i| = K \var{\hat{\mu}_i}

\end{proof}


\section{Introduction}
Autonomous vehicles have evolved quickly in the last years, mainly thanks to recent development in the field of deep learning. Creating a self-driving vehicle can generally be divided into three subtasks, perception and localization, planning, and control and actuation. This paper is concerned with the latter, namely following a given path, ensuring correct control and actuation of the vehicle. More specifically, a Reinforcement Learning (RL) approach was used for controlling the steering angle $\delta$ and acceleration $a$ of a simulated land vehicle, trying to keep the cross-track error (CTE) and velocity error small. While traditional controlling algorithms exist for this coupled control task \cite{chebly2017coupled} and they provide promising performance, their main drawback is that precise modeling of the vehicle at hand is required. Many parameters must be measured or estimated accurately for the control algorithm to perform adequately. This motivates the use of deep learning where suitable parameters instead are learned by the algorithm itself. Although traditional supervised deep learning can be applied to solve this problem \cite{devineau2018coupled}, this will not be considered here. Instead, Deep Deterministic Policy Gradient will be applied, which is an off-policy learning algorithm suitable to use in a continuous domain where continuous control actions are desirable \cite{lillicrap2015continuous}. The algorithm and its application to the control problem will be explained closer in the coming section.

\section{Related work} \label{sec:related_work}
Reinforcement learning has been successfully applied to various scenarios, such as playing Atari video games \cite{mnih2013playing} and basic control tasks \cite{lillicrap2015continuous}. The basic idea of reinforcement learning consists of having an agent interacting with an environment by each time-step $t$ receiving an observation $x_t$, based on this taking an action $a_t$ and receiving a reward $r_t$. The agent then tries to select actions to maximize the cumulative reward. One of the more commonly known algorithms for finding action policies is called Q-learning, which is a model-free off-policy RL algorithm, introduced already in 1989 \cite{watkins1989learning}. Model-free refers to that no model of the environment is needed to learn suitable actions, while off-policy means that it can learn from previously collected observations. The Q-learning algorithm is value-based, i.e. essentially it uses observations to learn the optimal action for any giving state by keeping track of all possible state-action pairs and continuously evaluating and updating their quality $Q^*(s_t,a_t)$ (hence the name Q-learning). After learning $Q^*(s_t,a_t)$, actions are selected such that $Q^*(s_t,a_t)$ is maximized for each state. However, as the state-space grows larger, the discrete approach used in Q-learning becomes infeasible due to the curse of dimensionality. Furthermore, for a continuous state-space, each state has to be discretized, where there is a trade-off between small discretization steps and keeping low dimensionality.

The problem of continuous observation spaces can be solved using Deep Q-learning (DQN)\cite{mnih2015humanlevel}. Instead of using a table to store the Q-values, a function approximation based on a neural network is used to estimate the values. In the algorithm a continuous state-vector is fed into a neural network where the output is an estimation of the current Q-value. However, this still leaves one with having to maximize the Q-value over all possible actions, i.e. for each state, one has to search through the actions, finding the one with the highest Q-value. For a large action space or large function approximation network, this becomes computationally costly and time-consuming.

For handling continuous action spaces, one can use Deep Deterministic Policy Gradient (DDPG) \cite{lillicrap2015continuous} which is an extension of the actor-critic approach in Deterministic Policy Gradient \cite{pmlr-v32-silver14} in combination with findings done by Deep Q-learning \cite{mnih2015humanlevel}. The algorithm is based on two different neural networks, called the actor $\mu(s|\theta^\mu)$ and critic $Q(s,a|\theta^Q)$ networks, where $\theta^\mu$ and $\theta^Q$ denote their weights respectively. The actor network maps vectors $s$ in the state-space into a vector $a$ in the action space, i.e. it uses the knowledge it has about its current state and tries to determine the best possible action to take. The critic network uses both the current state $s$ and the action $a$ taken by the actor-network to decide how good that particular action is to take in this particular state, i.e. trying to approximate the Q-value. The parameters of the critic are updated by using gradient stochastic descent for minimizing the Mean-Squared Bellman Error, similar as in traditional Q-learning. For the actor however, \cite{pmlr-v32-silver14} showed that there exist a policy gradient, i.e. gradient of the policy's performance, along which one updates the parameters of the network
\begin{equation}
    \nabla_{\theta^\mu}J \approx \mathrm{E}_{s\sim\rho^\beta}[\nabla_a Q(s,a|\theta^Q)|_{a=\mu(s|\theta^\mu)}\nabla_{\theta^\mu}\mu(s|\theta^\mu)]
\end{equation}

Deep Q-learning introduced two major ideas to scale Q-learning which are also used in DDPG, namely the use of replay buffers and separate target networks. Replay buffers save a large number of experiences $(s,a,r,s',d)$, where $s'$ denotes the next state and $d$ whether that state is terminal, in a dataset $\mathcal{D}$. One later randomly samples  mini-batches from this dataset, rather than taking the observations directly from the simulation when training the actor and critic networks. By employing a replay buffer, one can suppress the issue that the networks overfit to the most recent data and achieve a more stable training process. Furthermore, it also handles the issues that the observations taken from the simulation are sequentially dependent on each other.

When training the critic network, the Mean-Squared Bellman Error (MSBE) is to be minimized with respect to the parameters $\theta$, i.e. we are trying to minimize the loss function
\begin{equation}
    \label{eq:bellman-loss}
L = \Big(Q_{\theta}(s, a)-\underbrace{\left(r+\gamma(1-d) \max _{a^{\prime}} Q_{\theta}\left(s^{\prime}, a^{\prime}\right)\right)}_{T}\Big)^2
\end{equation}
by making $Q_{\theta}$ more similar to the target $T$. Here $\gamma$ denotes the discount factor $\gamma \in [0,1]$ determining how much future rewards are valued and $a'$ is the action taken in the next state $s'$. As can be seen in \eqref{eq:bellman-loss}, the target is dependent on the parameters we are trying to learn, namely $\theta$. This can make the training very unstable, leading to the agent not learning anything. Here is where the authors of DQN proposed target networks, which are yet another set of parameters, $\theta_{targ}^Q$ and $\theta^\mu_{targ}$, which lag behind the actual parameters $\theta^Q$ and $\theta^\mu$. In the original DQN paper\cite{mnih2015humanlevel}, the target networks were updated after a fixed number of episodes, however in the DDPG algorithm\cite{lillicrap2015continuous} the authors used Polyak averaging after each training step. The update is therefore described as
\begin{equation}
    \theta_{targ} \longleftarrow (1-\rho)\theta_{targ} + \rho \theta, \ \rho \ll 1
\end{equation}
By using te target networks instead of the actual networks to calculate the target in \eqref{eq:bellman-loss} one can achieve much more stable training.

Finally, in RL one has to balance the trade-off between exploration and exploitation. By adding a certain level of randomness in the agent one allows it to explore and learn more about the state-action space before optimizing its policy. In the original DDPG paper \cite{lillicrap2015continuous}, the authors proposed an exploration policy where Gaussian noise was added to the actor-networks output, i.e. directly to the selected action before executing it in the environment. However, as described in \cite{kamran2019learning}, using such independent exploration noise can be very inefficient as a vehicle system acts as a low pass filter for the high-frequency changes in acceleration and steering angle that this type of noise provides. Instead, they proposed an exploration policy where the noise added, $\epsilon_\delta$ and $\epsilon_T$, is sinusoidal with randomly sampled mean and deviation according to
\begin{align}
\epsilon_{\delta}, \epsilon_{a} \sim N\left(\mu, \sigma^{2}\right) \\
\mu_{\delta}=A \sin \left(\omega t+\varphi_{\delta}\right) \\
\mu_{a}=A \sin \left(\omega t+\varphi_{a}\right)
\end{align}
where $\sigma^2$, $A$ and $\omega$ are sampled from a zero-mean Gaussian distribution, while $\varphi_\delta, \varphi_a$ are uniformly sampled in the range $(-\pi, \pi)$. Note here that the values  $\sigma^2$, $A$, $\varphi_\delta, \varphi_a$ and $\omega$ are kept constant over one full episode.

Furthermore, in \cite{kamran2019learning}, the authors showed that it is advantageous to have an initial explore time (e.g. the first 500 episodes) where the agent, rather than using the output of the actor-network, samples actions uniformly in the range $(-1,1)$. They showed that this drastically improved the performance of the agent and enabled it to learn much better behaviour as it allows for vast exploration in the initial training period.

\section{Method}
Before training the RL agent, an environment must be created with which it can interact. The environment in this project was produced by the authors and consisted of a kinematic bicycle model which took requested steering angles and accelerations as input, and calculated a global position $x$, $y$, velocity $v$, steering angle $\delta$, and heading $\theta$ of the agent. The choice of the kinematic bicycle model is based on the model's simplicity while capturing the overall movements of a car. However, the model was adjusted such that the requested steering angle $\delta_\text{req}$ was limited by a maximum steering rate $\Delta\delta_\text{max}=40$, i.e. the change in steering angle could not exceed a certain threshold. Further, the requested accelerations $a_\text{req}$ were also clipped to lie within adequate limits $[-a_\text{max},a_\text{max}]$ with $a_\text{max} = 5m/s^2$. These changes were introduced to simulate how actuators of a car might behave in real-world conditions. The model can be summarized as selecting a time step $\Delta t$ and vehicle length $L$, then iterating
\begin{align}
    \Delta \delta &= \text{clip}\left(\frac{\delta_\text{req}-\delta_t}{\Delta t},-\Delta\delta_\text{max},\Delta\delta_\text{max}\right)
    \\
    a_t &= \text{clip}\left(a_\text{req},-a_\text{max}, a_\text{max}\right)
    \\
    x_{t+1} & = v_t \cos(\theta_t) \Delta t\\
    y_{t+1} & = v_t \sin(\theta_t) \Delta t\\
    v_{t+1} & = a_t \Delta t \\
    \delta_{t+1} & = \Delta \delta \Delta t\\
    \theta_{t+1} & = \frac{v_t \tan(\delta_t)}{L} \Delta t
\end{align}

The model was also used to generate reference paths for the agent to follow. For this, an average velocity $v_{avg} \sim \mathcal{U}(3,20)$ was sampled once before each path generation started and then $\delta_\text{req}$ and $a_\text{req}$ where sampled randomly every time step until the path had a length of $400$m. $\delta_\text{req}$ was sampled from an uniform distribution $\mathcal{U}(-30,30)$ and $a_\text{req}$ was sampled from two different uniform distributions, $\mathcal{U}(-2,2)$ if $v_t > v_{avg}$, otherwise from $\mathcal{U}(0,2)$. The reference trajectory was then processed to consist of waypoints with reference speed every meter. Due to the randomness of the generation, the agent received a unique path to follow each episode. The agent's state observations $s$ consisted of the distance to the 25 nearest future waypoints, expressed in the agent's coordinate frame, along with the difference between current velocity $v_t$ and reference velocity $v_{ref}$ for all 25 waypoints. Additionally, the observed state also included the agent's current velocity and steering angle $v_t$, $\delta_t$. An illustration is shown in Figure \ref{fig:states}, where $(x_i,y_i)$ denote global coordinates and $(x'_i,y'_i)$ coordinates in the agents frame.

\begin{figure}[tpb]
    \centering
    \includegraphics[width=0.6\linewidth]{images/states.png}
    \caption{Illustration of state observations for agent.}
    \label{fig:states}
\end{figure}
As the agent intends to control the steering angle $\delta$ and the acceleration $a$, the action space is continuous and the actions have been chosen to be in the range of $\delta, a \in [-1, 1]$. Each command must therefore be multiplied with the maximally allowed acceleration and steering angle, as defined by the model described above before it is passed to the simulation.

The reward signal was designed with inspiration from \cite{kamran2019learning} and is described by
\begin{equation}
    R =
    \begin{cases}
      -1 & \text{if $\text{cte} > 0.2$}\\
      1.5 - P_{cte} - P_\delta - P_{v} - P_{acc} & \text{otherwise}
    \end{cases}
\end{equation}
where, cte is the cross-track error between the agent and the reference path, i.e. perpendicular distance from reference path to agent location, and $P_{cte}$, $P_\delta$, $P_{v}$ and $P_{acc}$ are penalties based on the cross-track error, steering angle, deviation from reference velocity, and the acceleration performed by the agent. The penalties were introduced to minimize cross-track errors, steering angles, velocity errors and jerky maneuvers respectively, and are defined by the following equations
\begin{align*}
    P_{cte} &= 0.8 \text{cte}, \
    P_\delta = 0.1 \delta_t/\delta_{\text{max}}, \\
    P_v &= 0.8 |v_t-v_{ref}| / v_{ref}, \
    P_{acc} = 0.2 |a_\text{req}|
\end{align*}
Additional negative reward was assigned when the normalized velocity error became to large, i.e.
\begin{equation}
    R := R - 1 \ \text{if $|v_t-v_{ref}|/v_{ref} > 0.25$}
\end{equation}
Each simulation was terminated when either the cross-track error $\geq2$ meters, the velocity $v_t \leq 0$ m/s or the agent reached the end of the path. However, only the to first two cases were marked as terminal states in the target $T$.

As mentioned previously the DDPG architecture consists of two different networks, the actor and the critic. The network architecture for each of the networks are shown in Figure \ref{fig:actor-network}. The actor-network begins with two shared fully connected layer and then branches out into two branches, each producing their own action, i.e. either the steering angle $\delta$ or the acceleration $a$. The two outputs are then concatenated before given as the final output of the actor-network. The weights and biases of the last fully connected layers of network were initialized to $\mathcal{U}(-10^{-6}, 10^{-6})$ as this was found to be crucial for the agent to learn anything at all.
\begin{figure}[tpb]
    \centering
    \includegraphics[width=0.8\linewidth]{images/images.png}
    \caption{Critic (top) and actor (bottom) network architectures.}
    \label{fig:actor-network}
\end{figure}

Furthermore, the replay-buffer size was set to $100,000$ and the sinusoidal exploration policy explained in Section \ref{sec:related_work} was used. To transition smoothly from exploration to exploitation, a noise amplitude multiplier was initialize to 1 and reduced by $0.9996$ every episode. The agent was trained in total for $10$ epochs, each lasting for $500$ episodes. The division into epochs was done mainly for learning rate scheduling.

\section{Results}
In Figure \ref{fig:training-performace} several different metrics from the training are shown for the first 2500 episodes. Starting from the top we see the average cross-track error and velocity error normalized with the travelled path length, as well as the average reward per time-step. After $\sim1000$ episodes a major bump in performance was obtained, which is caused by switching from the aggressive exploration policy used in the initial training phase to the more lenient sinusoidal noise explained in Section \ref{sec:related_work}. Simply from sampling random actions in the action space, the agent was able to learn enough about the environment to perform adequately. When trained without the initial training phase, the agent was not able to perform any satisfactory behaviour after the same number of training steps, thus validating the results in \cite{kamran2019learning}. After the initial bump in performance, the behavior of the agent only changed marginally. While it improved somewhat in all aspects shown in Figure \ref{fig:training-performace}, the biggest change was in consistency. Towards the end of the training the agent was able to maintain a lower variance in all of the aspects discussed above, including the average percent of completion w.r.t. the reference path.
\begin{figure}[htpb]
    \centering
    \includegraphics[width=0.7\linewidth]{images/training.pdf}
    \caption{Average cross-track error, reward and velocity error, all normalized with the travelled path length, shown as a function of first 2500 episodes.}
    \label{fig:training-performace}
\end{figure}

After the training, the agent was evaluated over a set of 10 test tracks randomly generated using the environment described earlier, where the performance over these is shown in Table \ref{tab:Performace average}.
\begin{table}[tbp]
    \centering
    \caption{Performance over a set of 10 test-tracks}
    \scalebox{0.8}{
    \begin{tabular}{c|c|c|c|c|c}
        Track & Avg cte & Max cte & Avg $\Delta v$ & Max $\Delta v$ & \% of path    \\ \hline
1 & 0.103 & 0.264 & 0.473 & 1.466 & 100.0 \\
2 & 0.141 & 0.32 & 0.465 & 1.813 & 100.0 \\
3 & 0.101 & 0.299 & 0.608 & 2.523 & 100.0 \\
4 & 0.119 & 0.335 & 0.658 & 2.519 & 100.0 \\
5 & 0.113 & 0.641 & 0.62 & 2.34 & 100.0 \\
\textbf{6} & \textbf{0.114} & \textbf{0.464} &\textbf{ 0.752} & \textbf{2.623} & \textbf{100.0} \\
7 & 0.141 & 0.54 & 0.65 & 2.28 & 100.0 \\
8 & 0.107 & 0.31 & 0.609 & 1.937 & 100.0 \\
9 & 0.111 & 0.385 & 0.496 & 1.84 & 100.0 \\
10 & 0.105 & 0.336 & 0.626 & 2.076 & 100.0 \\
\hline
Avg & 0.115 & 0.39 & 0.596 & 2.142 & 100.0
    \end{tabular}}
    \label{tab:Performace average}
\end{table}
From the evaluation one can conclude that the agent is performing passably. The average cross-track error is kept low ($\sim 0.12$ m) while also ensuring that the velocity is kept close to its reference (on average $\sim 0.60$ m/s deviation). For the worst case scenario, the table shows the maximal cross-track error during the 10 evaluations and it never exceeded $0.65$ m. This is important to guarantee that the path following algorithm never strays to far from its reference.

Furthermore, in Figure \ref{fig:example-performace} the evaluation on test track \# 6 (as it is deemed to be representative over the 10 evaluations) is shown in more detail. Here the reference and the actual path travelled is shown together with cross-track error and the velocity error throughout the simulation. As can be seen, the cross-track error rarely exceeds 0.3 m
\begin{figure}[tpb]
    \centering
    \includegraphics[width=0.7\linewidth]{images/test_track_6.png}
    \caption{Test Track \#6 with the cross-track error [m] and velocity [m/s] deviation shown over time.}
    \label{fig:example-performace}
\end{figure}

\section{Conclusions \& Future Work}
In this paper, we leveraged DDPG to solve the lateral and longitudinal control tasks. While conventionally solved with algorithms such as Pure Pursuit or Model Predictive Control our RL algorithm showed very promising results. The agent used relative distance and velocity to the 25 closest way-points on our reference path to determine the appropriate action. During testing, the agent was able to steer with an average cross-track error of only 0.12m while keeping the agents velocity within 0.6m/s deviation from its reference on average.

However, it should be noted that the agent was never applied in a real-world setting, but merely a rather simplistic simulator. In future work, it would be interesting not only to apply and test this on an actual vehicle but also to compare it to the performance of the more conventional controllers.

Furthermore, one could make the simulation more realistic by including e.g. noise in the state observations fed to the agent, or simply use a more sophisticated simulator.

Finally, the authors recognize that the black-box nature of our RL approach raises the issue of how we can guarantee safe behavior at all times. While conventional controllers performance can be analyzed and determined under which circumstances it will perform as intended, our solution is inherently hard to analyze begging the question; can we ever guarantee satisfactory behavior at all times, and if not, under which circumstances can we assure that the RL approach will act appropriately?

\bibliographystyle{IEEEtran}