\section{Introduction}
Tree estimators and random forests are nonparmetric estimators that enjoy widespread popularity in applied
data science, powering models in e-commerice, finance and macroneconomics, and medicine. Random forests,
introduced by Breiman \cite{breiman01}, is a bagging algorithm that aggregates estimates from a collection
of tree estimators. Since each tree has alow bias but high variance, aggregation helps performance by balancing
the bias-variance trade-off.  Other than good empirical performance,
random forests also enjoy the following advantages over other nonparametric methods:
they scale naturally with high dimensional data, as the optimal
cut at each may be computed in parallel by quantizing each covariate; categorical
variables and missing data can be easily corporated; and they are more interpretable since
variable importance can be explicitly characterized.

One fruitful application of random forests in economics is in estimating heterogeneneous
treatment effects, i.e., a function of the form $f(x) = \E(Y_1 - Y_0 \mid X = x)$.  In order
to conduct inference in econometric applications (e.g., to test the null $H_0: f(x) = 0$),
knowledge about the rate of convergence or asymptotic distribution of the estimator $\hat f(x)$
is required. Moreover, functions of $f$ are often of practical interest: for example, when we wish
to study the difference in treatment effects for two different subpopulations,
the quantity of interest is
\begin{equation}
f(x) - f(\bar x), \quad \text{where $x$ and $\bar x$ describe the two subpopulations}.
\end{equation}
More generally, we might also be interested in a weigthed treatment effect, where
a subpopulation $x$ is given an importance weight $\mu$. In this case, the functional
of $f$ is
\begin{equation}
\int_{x \in \mathcal{X}} f(x) d\mu, \quad \text{where $\mu$ is not necessarily the density of $x$},
\end{equation}
and the integral is taken over the feature space $\mathcal{X}$.
Inference on functions of $f$ requires not only the asymptotic distribution of the point
estimate $f(x)$, but also the correlation between estimates at distinct
points $f(x)$ and $f(\bar x)$. This paper studies the correlation structure of a class
of random forests models whose asymptotic distributions were worked out in \cite{crf}.
We find sufficient conditions under which the asymptotic covariance of estimates
at distinct poitns vanish relative to the variance; moreover, we provide finite sample heuristics
based on our calculations. To the best of our knowledge, this is the first set of results
on the correlation of random forest estimates.

The present paper builds on and extends the results in \cite{crf}, which in turn
builds on related work \cite{Wager2015AdaptiveCO}
on the concentration properties of tree estimators and random estimators in general.
See also \cite{athey2019}, which extends the random forest model considered here to estimate
moment conditions. Stability results established in this paper have appeared
in \cite{arsov2019stability}, who study the notions of algorithmic stability for
random forests and logistic regression and derive generalization error guarantees. Also closely related
to our paper are \cite{chernozhukov2017} and \cite{chen2018},
concerning finite sample Gaussian approximations of sums and U-statistics
in high dimensions. In this context, our paper provides a stepping stone towards applying the theory
of finite sample U-statistics to random forests, where bounds on the correlation structure plays a central
role.

The paper is structured as follows. In Section 2, we introduce the random forest model
and and state the assumptions required for our results; Section 3 contains our maion theoretical
contributions; Section 4 builds on Sections 3 and discusses heuristics useful in finite sample settings;
Section 5 evalutes our heuristics with numerical experiments, and Section 6 concludes. All proofs are found
in the appendix.

\section{Setup}
In this paper, we study Gaussian approximations
of random forest estimators. Throughout, we assume
that a random sample $(Z_i)_{i=1}^n = (X_i, Y_i)_{i=1}^n$ is given, where
$X_i \subseteq \mathcal{X} \in \mathbf{R}^p$  is a vector of regressors and $Y_i \in \mathbf{R}$
the response.

A tree estimator $T(\xi; Z_1, \dots, Z_s)$ works by recursively partitioning
the space of covariates $\mathcal{X}$ into a collection of non-overlapping
hyper-rectangles. The estimator starts with the entire space $\mathcal{X}$
(assumed to be bounded), and uses the points
$Z_1$, \dots, $Z_s$ to produce a coordinate
$j \in \{ 1, \dots, p\}$ and an index $t \in \mathbf{R}$. This divides
$\mathcal{X}$ into two hyperrectangles,
\begin{equation}
\{ x \in \mathcal{X} : x_j < t \}
\quad \text{and} \quad
\{ x \in \mathcal{X}: x_j > t \}.
\end{equation}
In the next step, each of these two sets is partitioned
into two hyperrectangles each, using the points
that land in each. The process continues until
a particular hyperrectangle satisfies a stopping criterion.
This process corresponds to a tree in the natural way,
so we will refer to a generic hyperrectangle a ``node'',
and the leaves of the tree ``terminal nodes'',
where splitting ceases.
Given the partition of $\mathcal{X}$ into terminal nodes
$N_1, \dots, N_q$, the prediction is the average
of the responses
\begin{equation}
T(x; \xi, Z_{i_1}, \dots, Z_{i_s}) =
\sum_{i=1}^q \mathbf{1}(x \in N_i) \cdot
\frac{1}{|N_i|} \sum_{i: X_i \in N_i} Y_i,
\end{equation}
where $|N_i|$ is defined to be the number of
samples $X_i$ inside $N_i$. Here, $\xi$
plays the role of randomization, as
the splitting decisions allows for randomization.

Given a base learner $T$, a random forest estimator
at $x$ is defined to be the
order $s$ $U$-statistic with kernel $T$, i.e.,
\begin{equation}
\RF(x; Z_1, \dots, Z_n)
=
\frac{1}{\binom{n}{s}}
\sum_{i_1, \dots, i_s}
\E_{\xi} T(x; \xi, Z_{i_1}, \dots, Z_{i_s}),
\end{equation}
where the randomization device $\xi$ is marginalized over.

In this paper, we study the asymptotic distribution of
the vector
\begin{equation}
\RF(x_1, \dots, x_q; Z_1, \dots, Z_n)
\coloneqq
\begin{bmatrix}
  \RF(x_1; Z_1, \dots, Z_n) \\
\vdots\\
\RF(x_q; Z_1, \dots, Z_n)
\end{bmatrix} \in \mathbf{R}^q,
\end{equation}
where the random forest is employed to produce
an estimate at $q$ points $x_1, \dots, x_q$.

Previous work by (Athey and Wager)
has shown that under regularity
conditions
\begin{equation}
\frac{\RF(x, Z_1, \dots, Z_n) - m(x)}{\sigma_n(x)} \xRightarrow{\;\rm dist\;} N(0,1)
\qquad \text{where }
m(x) = \E(Y \mid X = x)
\end{equation}
and $\sigma_n^2(x)$ the variance of $\RF(x)$. In this paper,
we extend their result to cover multivariate predictions
and establish joint normality
\begin{equation}
\Sigma^{-1/2}\{ \RF(x_1, \dots, x_q) - m(x_1, \dots, x_q) \}
\xRightarrow{\;\rm dist\;} N(0, I_{q \times q})
\end{equation}
where $m(x_1, \dots, x_q)$ is the vector with
components $\E(Y \mid X = x_k)$. This is the main technical result of the paper.
In addition, we provide numerical simulations to gauge the finite sample
behavior of the multivariate random forest estimator.

\subsection{Assumptions}
Our results builds on \cite{crf}, so that we will work with the same model of random forests
to be maintained. Specifically, we will assume the following properties of the underlying
tree algorithm.
\begin{assumption}[Honesty]
Conditional on $\{ X_i \}$, the tree structure (i.e., the splitting coordinates and splitting
indices) are independent of the tree estimates.
\end{assumption}
There are several ways to satisfy this assumption. The first is to calculate splits
based only on $X_i$: for example, a splitting coordinate and index is chosen to minimize
the average squared distance to the center of each of the two nodes as in clustering algorithms.
The second way is to perform data splitting: partition the data into two sets $\mathcal{I}_1$
and $\mathcal{I}_2$. Observations in $\mathcal{I}_1$ and $X_i \in \mathcal{I}_2$
may be freely used during the splitting process, while $Y_i \in \mathcal{I}_2$ are used to determine
terminal node values. Finally, a third method assumes the existence of side information
$\{ W_i \}$: splits are made in the domain of $\{ X_i \}$ using $\{ W_i \}$ are surrogate responses,
with $\{ Y_i \}$ being used for prediction at the terminal nodes.

For simplicity, we will assume that the first scheme is used, namely that the splitting
algorithm uses only $X_i$ and not $Y_i$ in placing the splits. Our results remain valid
for the third scheme.

\begin{assumption}[Randomized Cyclic Splits]
At each non-terminal node, the splitting algorithm decides to
do data-independent split with probability $\delta > 0$, by flipping
a ``$\delta$-coin'' with distribution is independent of everything else. The first time
the coin lands heads, the first coordinate is chosen as the splitting coordinate;
the second time, the second coordinate is chosen, and so on, such that that
on the $J$-th time the coin lands heads, the $(p \text{ mod } J) + 1$-th
coordinate is chosen.
\end{assumption}

This is modification of the assumption in \cite{crf}, in which each node has a probability
$\delta$ of being chosen at each split; the latter is most easily implemented
by flipping a $p\delta$ coin, and selecting one of the $p$ coordinates uniformly
at random to split when the coin lands heads. Our assumption does away with the
randomization in the second step.

\begin{assumption}[The Splitting Algorithm is $(\alpha, k)$-Regular]
  There exists some $\alpha \in (0, 1/2)$ such that
whenever a split occurs in a node with $m$ sample points, the two
hyper-rectangles contains at least $\alpha m$ many points
each. Moreover, splitting ceases at a node only if the node contains less than $2k-1$ points
for some $k$. In particular, this implies that $\Var T$ has bounded entries,
since the number of points in each terminal node is bounded above.
\end{assumption}
As shown in \cite{Wager2015AdaptiveCO},
this implies that with
all but exponentially small probability, the splitting axis also
shrinks by a factor between $\alpha$ and $1-\alpha$.

The following assumption is specific for our model. The first specifies that the \emph{potential}
splits do not depend on the data; in practice, this is ``essentially'' without loss of generality
as the domain of $X$ is fixed. For example, the assumption is satisfied if all splits
occur at indices that lie on a specified grid, e.g., points which can be represented by a 64-bit floating
point number. In keeping with the $(\alpha, k)$ regularity assumption above, we also assume that
no potential split may split the splitting axis by more than a proportion $\alpha$.

\begin{assumption}[Predetermined Splits]
The potential splits at each node does not depend on the realizations of the data. Furthermore,
the number of potential splits at each node is finite, and each potential
split shrinks the length of the corresponding axis by at most $\alpha$.
\end{assumption}
We will also require other assumptions regarding the ``stability''
of splits; these assumptions will be introduced later.
Finally, we follow \cite{crf} and place the following distributional assumptions
on the data generating process.
\begin{assumption}[Distributional Assumptions on $(X,Y)$]
The covariate $X$ is distributed on the $[0,1]^p$ with a density that is
bounded away from zero and infinity; without loss of generality, we may
assuem that $X$ is uniformly distributed on the $p$-dimensional hypercube.

Furthermore, the regression function
$x \mapsto \E(Y \mid X = x)$, $x \mapsto \E(Y^2 \mid X = x)$,
and $x \mapsto \E(Y^3\mid X=x)$ are uniformly Lipschitz
continuous.
\end{assumption}
\section{Gaussianity of Multivariate $U$-Statistics}
\subsection{Hajek Projections}
For $m \geq 1$ and a statistic $f(Z_1, \dots, Z_m) \in \mathbf{R}^q$, recall that the Hajek projection
of $f$ is defined to be
\begin{equation}
\mathring f(Z_1, \dots, Z_m) = \sum_{i=1}^m \E[f(Z_1, \dots, Z_m) \mid Z_i]
- (m-1) \E f(Z_1, \dots, Z_m).
\end{equation}
In particular, when $Z_1, \dots, Z_m$ is an IID sequence
and $f$ is symmetric in its arguments, we have
\begin{equation}
\mathring f(Z_1, \dots, Z_m) = \sum_{i=1}^m f_1(Z_i) - (m-1)\E f,
\end{equation}
where $f_1(z) = \E[f \mid Z_1 = z]$.

The previous calculations applied to our setting where $f$ is the
random forest estimator yields
\begin{equation}
\mathring\RF(Z_1, \dots, Z_n) - \mu = \sum_{i=1}^n \E(\RF - \mu\mid Z_i)
=
\frac{1}{\binom{n}{s}}
\sum_{i=1}^n \E
\biggl[
\sum_{i_1, \dots, i_s} \E_{\xi}T(\xi; Z_{i_1}, \dots, Z_{i_s}) - \mu \mid Z_i
 \biggr].
\end{equation}
Since our samples are independent, $\E(\E_{\xi}T(\xi;Z_{i_1}, \dots, Z_{i_s}) \mid Z_i) = \mu$
whenever $i \notin \{ i_1, \dots, i_s \}$.
As $\{ i_1, \dots, i_s \}$ runs over the size-$s$ subsets of $\{ 1, \dots, n \}$,
there are exactly $\binom{n-1}{s-1}$ subsets which contain~$i$. For each of these subsets
\begin{equation}
\E(\E_{\xi} T(\xi; Z_{i_1}, \dots, Z_{i_s}) - \mu \mid Z_i) \eqqcolon
T_1(Z_i) - \mu,
\end{equation}
where $T_1(z) = \E_{\xi} T(z, Z_2, \dots, Z_s)$.
Therefore,
\begin{equation}
\mathring\RF - \mu =
\frac{1}{\binom{n}{s}} \sum_{i=1}^n \binom{n-1}{s-1} (T_1(Z_i) - \mu)
=
\frac{s}{n} \sum_{i=1}^n (T_1(Z_i) - \mu).
\end{equation}

\subsection{Asymptotic Gaussianity via Hajek Projections}
The standard technique to derive the asymptotic distribution
of a $U$-statistic is to establish a lower bound on the
variance of its Hajek statistic; this is the approach taken by (Athey and Wager)
and we follow the approach here.
Let $V$ be the variance of $\mathring\RF$; using our previous result, we could write
\begin{equation}
\label{eq:Vvariance}
V = \Var\biggl[
\frac{s}{n} \sum_{i=1}^n (T_1(Z_i) - \mu)
 \biggr] =
\frac{s^2}{n} \Var(T_1(Z_1)) =
\frac{s}{n} \Var\biggl[
\sum_{i=1}^s T_1(Z_i)
 \biggr]
=
\frac{s}{n} \Var \mathring T,
\end{equation}
where $\mathring T$ is the Hajek projection of
the statistic $(Z_1, \dots, Z_s) \mapsto \E_{\xi}T(\xi; Z_1, \dots, Z_s)$.

Under standard regularity conditions for a multivariate
triangular array Central Limit Theorem,
\begin{equation}
V^{-1/2}(\mathring\RF - \mu) \xRightarrow{\;\rm dist\;} N(0, I).
\end{equation}
To establish the joint normality for $\RF$, write
\begin{equation}
V^{-1/2}(\RF - \mu) =
V^{-1/2}(\RF - \mathring \RF) +
V^{-1/2}(\mathring \RF - \mu).
\end{equation}
so all that is required is to prove that
$V^{-1/2}(\RF - \mathring \RF) \xrightarrow{\;\P\;} 0$. We will
show that the quantity converges in squared mean. Setting
$e = V^{-1/2}(\RF - \mathring \RF)$,
\begin{equation}
\label{eq:error squared}
  \begin{split}
\E (e^{\intercal} e) &=
\E(\RF - \mathring \RF)^{\intercal} V^{-1} (\RF - \mathring \RF)
= \E \tr V^{-1}(\RF - \mathring \RF)(\RF - \mathring \RF)^{\intercal} \\
&= \tr V^{-1} \E(\RF - \mathring \RF)(\RF - \mathring \RF)^{\intercal}
= \tr V^{-1/2} \Var(\RF - \mathring \RF)V^{-1/2}.
  \end{split}
\end{equation}
By Proposition~\ref{prop:hoeffding}, we have the following Hoeffding
decomposition with respect to the matrix $V^{-1}$
\begin{equation}
\RF - \mathring \RF = \frac{1}{\binom{n}{s}}
\biggl[
\sum_{i<j} \binom{n-2}{s-2} (T_2(Z_i, Z_j) - \mu) +
\sum_{i < j < k} \binom{n-3}{s-3} (T_3(Z_i, Z_j, Z_k) - \mu) +
\cdots
 \biggr],
\end{equation}
where $T_2$, $T_3$, etc.\ are the second and third order projections
of $T$ obeying the orthogonality conditions
\begin{equation}
\label{eq:4}
\E[ (T_k - \mu)^{\intercal} V^{-1} (T_{k'} - \mu)] = 0,
\qquad \text{for $k \neq k'$.}
\end{equation}
In addition, being projections of $T$, the higher order projections
also satisfy
\begin{equation}
\label{eq:7}
\E[(T_k - \mu)^{\intercal} V^{-1} (T_k - \mu)] \leq
\E[(T - \mu)^{\intercal} V^{-1} (T - \mu)].
\end{equation}
Therefore,
\eqref{eq:Vvariance} and \eqref{eq:error squared}
imply that
\begin{equation}
\label{eq:1}
\E(e^{\intercal}e) \leq \frac{s}{n} \tr(\Var \mathring T^{-1} \Var T).
\end{equation}

Athey and Wager obtains a bound on the diagonal elements of
$\Var \mathring T^{-1}$ and $\Var T$, namely
\begin{equation}
\label{eq:3}
\frac{(\Var T)_{kk}}{(\Var \mathring T)_{kk}} \leq c (\log s)^p,
\qquad \text{for each $k = 1, \dots, q$},
\end{equation}
for a constant $c$. In the next section, we shall bound the off-diagonal
elements of $\Var \mathring T$ and $\Var T$ in order to establish
$\E(e^{\intercal} e) \to 0$.

\begin{proposition}[Hoeffding Decomposition for Multivariate $U$-statistics]
  \label{prop:hoeffding}
Fix a positive definite matrix $M$.
Let $f(x_1, \dots, x_n) \in \mathbf{R}^p$ be a vector-valued function
that is symmetric in its arguments and let $X_1, \dots, X_n$
be a random sample such that $f(X_1, \dots, X_n)$ has finite variance. Then
there exists functions $f_1, f_2, \dots, f_n$ such that
\begin{equation}
\label{eq:5}
f(X_1, \dots, X_n) = \E(f) +  \sum_{i=1}^n f_1(X_1)
+ \sum_{i < j} f_2(X_i, X_j) + \dots + f_n(X_1, \dots, X_n)
\end{equation}
where $f_k$ is a function of $k$ arguments, such that
\begin{equation}
\label{eq:6}
\E f_k(X_1, \dots, X_k) = 0 \quad \text{and}\quad
\E [f_k(X_1, \dots, X_k)^{\intercal} M f_{\ell}(X_1, \dots, X_l)]  = 0.
\end{equation}
\end{proposition}

\subsection{Covariance Calculations}
The aim of this subsection is to prove the following
asymptotic behavior of the covariance matrix
$\Var \mathring T$ under suitable stability conditions
of the splitting algorithm
\begin{equation}
\label{eq:covariance bounds}
(\Var \mathring T)_{k, l} = o(s^{-\epsilon})
\text{ for all $k \neq l$ and some $\epsilon > 0$}.
\end{equation}
The next proposition shows that this is enough to establish
$\E(e^{\intercal} e) \to 0$.
\begin{proposition}\label{prop:trace-bound}
Suppose $\Var T$ and $\Var \mathring T$ satisfy the conditions
in \eqref{eq:covariance bounds}.  Then
\begin{equation}
\label{eq:2}
\frac{s}{n} \tr \Var \mathring T^{-1} \Var T \to 0.
\end{equation}
\end{proposition}
\begin{remark}
Proposition~\ref{prop:trace-bound} does not require the fact
$(\Var T)_{k, l} \to 0$ for the off-diagonal entries of $\Var T$,
only that they are bounded. Moreover, we only require
the much weaker bound $(\Var \mathring T)_{k, l} = o(\frac{1}{\log^p s})$.
However, tighter bounds are still useful for applications:
for one they allow us to control the ``approximation'' error $\E(e^{\intercal} e)$;
second, they are useful for bounding the off diagonal terms in
$V^{-1}(\RF - \mu)$.
\end{remark}
The rest of this section will establish the covariance bound for $\Var \mathring T$.
Recall that
\begin{equation}
\label{eq:10}
\mathring T - \mu = \sum_{i=1}^s \E(T \mid Z_i)
\quad \text{so that}\quad
\Var \mathring T = s \Var(\E(T \mid Z_1))
\text{ due to independence}.
\end{equation}
By the orthogonality condition,
\begin{equation}
\label{eq:11}
\Var \E(T \mid Z_1) = \Var[\E(T \mid Z_1) - \E(T \mid X_1)] +
\Var[\E(T \mid X_1)],
\end{equation}
and furthermore, honesty implies
\begin{equation}
\label{eq:12}
\E(T_k \mid Z_1) - \E(T_k \mid X_1) = \E(I_k \mid X_1) (Y_1 - \E(Y_1 \mid I_k = 1, X_1))
\end{equation}
where $T_k$ is the $k$-th coordinate of $T$ (i.e., the estimate of the tree
at $x_k$) and $I_k$ is the indicator variable for the event that
$X_1$ belongs to the terminal node containing $x_k$. In particular, the
off-diagonal entry at $(j, k)$ of $\Var[\E(T \mid Z_1) - \E(T \mid X_1)]$
is equal to
\begin{equation}
\label{eq:15}
\E[\E(I_k \mid X_1) \E(I_j \mid X_1)
(Y_1 - \E(Y_1 \mid X_1, I_k = 1)
(Y_1 - \E(Y_1 \mid X_1, I_j = 1)]
\end{equation}
Following the same truncation argument as in (Athey and Wager, Theorem 5),
this quantity is bounded by
\begin{equation}
\E[\E(I_k \mid X_1) \E(I_{j} \mid X_1)].
\end{equation}
Alternatively, this bound also holds in the case when $Y$ is bounded.
(Note that a direct application of the Cauchy-Schwarz inequality yields a weaker bound
$\sqrt{\E[\E(I_k \mid X_1)^2 \E(I_{j} \mid X_1)^2]}$.)
Recall that $\E(I_k \mid X_1 = x)$ is the probability that $x$
and $x_k$ belong to the same terminal node;
likewise $\E(I_j \mid X_1 = x)$ is the probability that $x$ and $x_j$
belong to the same terminal node. The following proposition shows
the intuitive result that for sufficiently large $s$, at least one of these
probabilities is small.
\begin{proposition}
\label{prop:terminal-node-probability}
Fix two distinct points $x$ and $\bar x \in [0,1]^p$,
and let $M(x, \bar x)$ be the probability that
$x$ and $\bar x$ belongs to the same terminal node. If $\delta > 1/2$, then
\begin{equation}
\label{eq:19}
M(x, \bar x) = o(s^{-(1+\epsilon)}) \quad \text{for some $\epsilon > 0$}.
\end{equation}
The same holds for the quantity $M(x, \bar x \mid X_1 = \bar x)$,
the probability conditional on $X_1 = \bar x$.
\end{proposition}
\todo{Discuss the significance of the metric $M(x, \bar x)$ that measures
how likely $x$ and $\bar x$ lie in the same terminal node.}

\todo{Simulations should show that the off-diagonal terms are strongly
correlated with $M(x, \bar x)$.}

This proposition shows that the contribution of
$\Var[\E(T \mid Z_1) - \E(T \mid X_1)]$ to the cross covariances
of $\Var \E(T \mid Z_1)$ is small (i.e., smaller than the required
$(\log s) / s$).  The requirement that $\delta > 1/2$, while needed
for our theoretical results, may not be needed in practice. The reasons
is that $\delta > 1/2$ is required for a uniform bound on the quantity
\begin{equation}
\E(I_k \mid X_1) \E(I_{j} \mid X_1),
\end{equation}
while what the proposition demands is a bound on the expectation. Indeed,
in the extreme case
$x = 0$  and $\bar x = 1$, it is easy to see that the expectation satisfies
the required bound even when $\delta \leq 1/2$. In light of this, we could
instead impose the high level condition that
\begin{equation}
M(x, \bar x)  = o\biggl( \frac{1}{\log^p s \cdot s} \biggr),
\end{equation}
where $M(x, \bar x) = \E[\E(I_k \mid X_1) \E(I_{j} \mid X_1)]$
as above.

\subsubsection{Bounding $\Var \E(T \mid X_1)$}
We turn next to bound the cross covariances
of $\Var[\E(T \mid X_1)]$.
It will be convenient to change notations, so that
$x \mapsto x_j$, $\bar x \mapsto x_k$, and let $x_1$
denote the value of $X_1$. Thus, we need to show that
\begin{equation}
\label{eq:covariance-bound}
\E[
(\E(T \mid X_1 = x_1) - \mu)
(\E(\bar T \mid X_1 = x_1) - \bar \mu))
] = o\biggl( \frac{\log s}{s} \biggr).
\end{equation}
where $T$ and $\bar T$ are the tree estimates at $x$ and $\bar x$,
and $\mu$ and $\bar \mu$ their (unconditional) expectations.

The plan is to show that if $\|x - x_1\|_{\infty}$ is small, then
\begin{equation}
\label{eq:20}
\E(T \mid X_1 = x_1) - \mu = \E(T \mid X_1 = x_1) - \E(T)
\end{equation}
is also small. Intuitively, the knowledge of sample point $X_1 = x_1$
changes the expectation of the tree estimate at $x$ only when
$x_1$ affects the position of the terminal node containing $x$. This is unlikely
to happen when $X_1 = x_1$ is farther away; as soon as $X_1$ leaves
the node containing $x$ in the splitting process, its effect on the terminal
node diminishes.

Toward this end, fix $x$, and let $\Pi$ denote the terminal node that
contains $x$.  Since the set of potential splits is unchanged when
conditioning on $X_1$, $\Pi$ is a discrete random variable and we may
write
\begin{equation}
\label{eq:tree-formula}
\E(T) = \sum_{\pi} \P(\Pi = \pi) \mu_{\pi}
\quad \text{and}\quad
\E(T \mid X_1 = x_1) = \sum_{\pi} \P(\Pi = \pi \mid X_1 = x) \mu_{\pi}'
\end{equation}
where $\mu_{\pi} = \E(T \mid \Pi = \pi)$ and
$\mu_{\pi}' = \E(T \mid \Pi = \pi, X_1 = x)$.

Recall that $\Pi$ is generated by a recursive splitting procedure,
and we can make a natural correspondence between \eqref{eq:tree-formula}
and the expectation taken over a directed acyclic graph (DAG) in the following way.
Let $[0, 1]^p$ be the root of the DAG; for every potential split at $[0,1]^p$,
there is a directed edge to a new vertex, where the vertex is the
hyperrectangle that contains $x$. If a vertex is a leaf (i.e., splitting ceases
at that node), then the vertex has no outgoing edges; otherwise, a vertex corresponding
to a non-terminal node has an outgoing edge for each potential split,
with each edge going to another vertex that is the halfspace containing $x$.

To each terminal vertex $v$ in the DAG, which naturally correspond to a terminal
$\pi$ containing $x$, associate a value $f(v) \coloneqq \mu_{\pi}$. Each edge
$e = (v \to w)$ corresponding to a split $s$ from a node $v$ to a halfspace $w$;
associate with this edge the transition probability
\begin{equation}
\label{eq:24}
p(e) \coloneqq \P(\text{$s$ is chosen at $v$} \mid \text{current node is $v$})
\eqqcolon \P(w \mid v).
\end{equation}
Given the transition probabilities, the value $f$ may be extended
to each vertex $v$ via the recursion
\begin{equation}
\label{eq:25}
f(v) \coloneqq \sum_{e: v \to w}  \P(w \mid v) f(w).
\qquad \text{Thus, $f(v)$ is the continuation value at $v$.}
\end{equation}
In this way, the expectation $\E(T) = f(\text{root}) = f([0,1]^p)$.
If instead we assign the terminal values $f'(v) = \mu'_v$ and
the transition probabilities
\begin{equation}
\label{eq:27}
p'(e) = \P(\text{$s$ is chosen at $v$} \mid \text{current node is $v$}, X_1 = x_1)
= \P'(w \mid v),
\end{equation}
then we recover $\E(T \mid X_1 = x_1) = f'([0,1]^p)$. Therefore,
to bound $\E(T \mid X_1 = x_1) - \E(T)$ requires bounding the difference
in the continuation values.

We will need to assume that $p'(e) \approx p(e)$; that is,
We will require the following assumption regarding differences in
the splitting probabilities.

\begin{splitting-stability}
For any node $v$, the total variation distance between the distributions
$\{ p(e) \}_{e: v \to w}$ and $\{ p'(e) \}_{e: v \to w}$ is bounded by
the effective volume of $v$. Specifically, there exists some $\delta>0$
such that for all $v$,
\begin{equation}
\label{eq:29}
\operatorname{TV}(p, p') \leq \biggl( \frac{1}{s|v|} \biggr)^{1+\delta}
\end{equation}
up to some constant. Here, $|v|$ denotes the volume of the hyperrectangle
at $v$, i.e.,
\begin{equation}
\label{eq:30}
|v| = \biggl| \prod_{j=1}^p (a_j, b_j) \biggr| = \prod_{j=1}^p |b_j - a_j|.
\end{equation}
\end{splitting-stability}

Recall that with probability exceeding $1/s$, the number of sample points
$X_1, \dots, X_s$ in the hyperrectangle $v$ is approximately $s|v|$, from our
uniform distribution assumption. In general, since the density of $X$
is bounded away from zero and infinity, the number of sample points will
be bounded (above and below) by a constant times $s|v|$.
Therefore, the stability assumption places a restriction on
procedure used to select optimal splits: namely, if the decision is made on the basis
of $m$ points, then by moving any one of the points changes the optimal
split with probability bounded by $m^{-(1+\delta)}$.
In practice, most splitting procedures satisfy a stronger bound. A set of sufficient
conditions is given in the following proposition.
\begin{proposition}
\label{prop:split-stability-sufficient}
Assume that the optimal split is chosen based on the quantities
\begin{equation}
f_1(\mu_1, \dots, \mu_Q), \dots, f_P(\mu_1, \dots, \mu_Q)
\end{equation}
for some $Q \geq 1$, where
$\mu_1, \dots, \mu_Q$ are the sample averages of the points being split
\begin{equation}
\mu_k  = \frac{1}{n_v} \sum_{i: X_i \in v} m_k(X_i).
\end{equation}
Specifically, the optimal split depends on the which $f_i$ achieves the largest value,
i.e., the value $\argmax_i f_i(\mu)$. Here, $f_1, \dots, f_P$ are assumed to be Lipschitz,
and the functions $m_1, \dots, m_Q$ are such that $m_k(X)$ is 1-subExponential. Then
the splitting stability assumption is satisfied.
\end{proposition}
\begin{remark}
Recall that the $X_i$ are bounded---therefore, $m_k(X)$ being subExponential
allows the use of squared loss to compute the optimal split.
\end{remark}
In general, the conditions in Proposition~\ref{prop:split-stability-sufficient}
is sufficient to guarantee the an exponential bound instead of a polynomial one.
Thus, the proposition should be viewed as simply as providing a ``plausibility argument''
that stable splitting rules are common in practice.

The next proposition shows that when the splitting probabilities are bounded
by $(s|v|)^{-(1+\delta)}$ as in the splitting stability assumption, so are the continuation values.
\begin{proposition}
\label{prop:value-bounds}
Suppose the splitting probabilities satisfy a generic bound
\begin{equation}
\TV(p, p') \leq \log(s) \Delta(s|v|) \quad \text{at each node $v$}.
\end{equation}
Then for any node $v$ containing $x$ but not $x_1$,
\begin{equation}
|f(v) - f'(v)| \leq C \Delta(s|v|) \quad \text{for some constant $C$}.
\end{equation}
(The constant $C$ does not depend on $v$.)
\end{proposition}

Finally, we can put the bounds on $\TV(p, p')$ and $|f-f'|$
together and prove the required bound on the cross diagonals.
\begin{proposition}\label{prop:2-bound}
Suppose that the splitting rule is stable as in \eqref{eq:29}
and that $\delta  > 1-\alpha$.
For $x \neq x_1$,
\begin{equation}
\label{eq:26}
|\E(T  \mid X_1 = x_1) - \E(T)| = o\biggl( \frac{1}{s^{1+\epsilon}} \biggr)
\end{equation}
for some $\epsilon > 0$. In particular, the off-diagonal
entries of $\Var \E(T \mid X_1)$ are $o(s^{-(1+\epsilon)})$.
\end{proposition}
Note that Proposition~\ref{prop:terminal-node-probability} required
that $\delta > 1/2$. Since $\alpha < 1/2$ by definition, this bound is necessarily
more restrictive. As before, this requirement is plausibly looser in
applications: we use it to give the bound
\begin{equation}
|v| \geq \alpha^L,
\end{equation}
even though the RHS is only tight only at most $1/2^L$ proportion
of the nodes. Note that the ``average'' node has area approximately
$(1/2)^L$, so that $\delta > 1/2$ may be more appropriate.

Putting everything together, we have show that required result that
\begin{equation}
\label{eq:31}
\Var(\mathring T)_{k, l} = o(s^{-\epsilon}) \qquad \text{for some $\epsilon > 0$}.
\end{equation}
Therefore, by Proposition~\ref{prop:trace-bound} and the trace calculations,
the asymptotic joint normality of the random forest estimator is established.

\section{Heuristics and Practical Recommendations}
The previous sections established the asymptotic normality result that
\begin{equation}
\label{eq:32}
V^{-1/2}(\mathring \RF - \mu) \xRightarrow{\;\rm dist\;} N(0,I),
\qquad \text{where }
V = \Var \mathring \RF = \frac{s}{n} \Var \mathring T.
\end{equation}
Recall $\mu$ is the expectation of $\RF$, while the
target function is $m(x) = \E(Y \mid X = x)$. Athey and Wager shows that
$(\RF - \mu)/\sqrt{V} \xRightarrow{\;\rm dist\;} N(0, 1)$ in the univariate case; since we have shown
that $V$ is diagonally dominant, their result carries over, so that
$V^{-1/2}(\RF - m) \xRightarrow{\;\rm dist\;} N(0, I)$.

Moreover, (Athey and Wager) proposes a jackknife estimator that can consistently
estimate $\sqrt{V}$ in the scalar case.
Our diagonally dominance result suggests that
the random forest estimates at $x$ and $\bar x \in [0, 1]^p$ are independent
in the limit $n\to\infty$,
\begin{equation}
\label{eq:33}
\begin{split}
\Var(\RF(x) + \RF(\bar x)) &= \Var(\RF(x)) + \Var(\RF(\bar x)) + 2 \Cov(\RF(x) + \RF(\bar x))
\\&\approx \Var(\RF(x)) + \Var(\RF(\bar x))
\end{split}
\end{equation}
so that the jackknife estimator for the scalar case
can be used to obtain confidence intervals. However, the approximation
above depends on decay of the off-diagonal terms. This section, we provide
a back of the envelope bound for the covariance term that may be useful
for practioners. We stress that the following calculations are mostly heuristics:
as we have shown above, the covariance term depends on quantities such as $M(x, \bar x)$
which is heavily dependent on the splitting algorithm. Since the aim is
to produce a ``usable'' result, we focus on heuristics.

To begin, recall that the covariance term is upper bounded by\footnote{This is a very crude upper bound
as we have dropped the quantity $\Delta(\alpha^{\ell} s)$
from the infinite series.}
\begin{equation}
M(x, \bar x) + \log^2s \biggl(\sum_{\ell = 0}^{\infty} p_{\ell}\biggr)
\biggl(\sum_{\ell = 0}^{\infty} \bar p_{\ell}\biggr)
\end{equation}
where $p_{\ell} = \P(L \geq \ell)$ is the probability
that $x$ and $x_1$ are not separated after $\ell$ splits
(c.f.\ proof of Proposition~\ref{prop:2-bound}) and likewise for $\bar p_{\ell}$.
If we let $I$ (resp.\ $\bar I$) denote the
indicator that $X_1$ is in the terminal node of $x$ (resp.\ $\bar x$),
then $I = 1$ is equal to the event that $L = \log_2 s$, so that
\begin{equation}
\E(I \mid X_1 = x_1) = \P(L = \log s) \leq \frac{\E L}{\log s}.
\end{equation}
Replacing the inequality with an approximation, we have
$\E L = (\log s) \E(I \mid X_1 = x_1)$. All of this shows that
the covariance term is bounded by
\begin{equation}
(\log^4s) M(x, \bar x), \quad \text{up to appropriate constants}.
\end{equation}

Towards a useful heuristic, we will consider a bound on the correlation
instead of the covariance. In our notation, the result of (Athey and Wager) lower bounds $M(x, x)$
(and $M(\bar x, \bar x)$) in their proof, while we upper
bound $M(x, \bar x)$. Ignoring the logarithmic terms, we have
\begin{equation}
\biggl|\frac{\Cov(T(x), T(\bar x))}{\sqrt{\Var T(x) \cdot \Var T(\bar x)}}\biggr|
\approx
\frac{M(x, \bar x)}{\sqrt{M(x, x) M(\bar x, \bar x)}}.
\end{equation}
Recall that $M(x, \bar x) = \E[\E(I \mid X_1) \E(\bar I \mid X_1)]$
and decays superlinearly as $\bar x$ moves away from $x$. Using the previous expression
(note that $M(x,x) \approx M(\bar x, \bar x)$ due to symmetry between $x$ and $\bar x$),
we can bound the correlation from purely geometric considerations. Since the integrand
\begin{equation}
\E(I \mid X_1) \E(\bar I \mid X_1)
\end{equation}
decays as $X_1$ moves away from $x$ (and $\bar x$), we may imagine that
its integral $M(x,x)$ comes from contributions to points $X_1$ near $x$,
say those points in a $L_{\infty}$-box of side lengths $d$ with volume $d^p$, i.e.,
points $\{ y \in [0,1]^p : \| x - y \| \leq d/2\}$.
For $M(x, \bar x)$, the contribution would come from points
that are with $d/2$ of both $x$ \emph{and} $\bar x$, and to a first degree approximation,
the volume of these points $\{ y \in [0,1]^p : \| x - y \|_{\infty} \leq d/2, \| \bar x - y \| \leq d/2 \}$
is
\begin{equation}
(d - z_1)  \dots (d - z_p) \approx d^p - (z_1 + \dots + z_p)d^{p-1}, \quad \text{where $z_i = |x_i - \bar x_i|$},
\end{equation}
where the approximation is accurate if $|z_i| \ll 1$. The proportion of the volume
of these points is therefore $1 - \frac{1}{d}\| x - \bar x \|_1$, which leads to the heuristic
\begin{equation}
\biggl|\frac{\Cov(T(x), T(\bar x))}{\sqrt{\Var T(x) \cdot \Var T(\bar x)}} \biggr|
\approx
1 - c \| x - \bar x \|_1, \qquad \text{for some constant $c$}.
\end{equation}
The RHS has the correct scaling when $x = \bar x$, i.e., the correlation equals
one when $\| x - \bar x \|_1 = 0$. To maintain correct scalinag at the other extreme
with $\| x - \bar x \|_1 = p$, we should take $c = 1/p$, so that
\begin{equation}
\biggl|\frac{\Cov(T(x), T(\bar x))}{\sqrt{\Var T(x) \cdot \Var T(\bar x)}}\biggr| \approx
1 - \frac{1}{p}\sum_{i=1}^p |x_i - \bar x_i|.
\end{equation}

Of course, this heurstic is ``wrong'' in that it does not depend on $s$;
our theoretical results show that even for non-diametrically opposed points,
the correlation drops to zero as $s \to \infty$. Therefore,
another recommendation is to use
\begin{equation}
\label{linear-heuristic}
\biggl|\frac{\Cov(T(x), T(\bar x))}{\sqrt{\Var T(x) \cdot \Var T(\bar x)}}\biggr| \approx
\min \biggl(
1 - \frac{s^{\epsilon}}{p}\sum_{i=1}^p |x_i - \bar x_i|, \; 0
\biggr),
\end{equation}
for some $\epsilon > 0$, where dependence $s^{\epsilon}$ comes from matching the heuristic
$c = 1/d$ above with the proof of Proposition~\ref{prop:terminal-node-probability}.

\section{Simulations}
In this section, we conduct numerical experiments of varying sample sizes to gauge the empirical
relevance of our theoretical results and heuristics. In our experiments, we set $p = 2$, so that
the covariates $X$ are distributed on the unit square. Instead of adopting a uniform distribution,
for ease of interpretation, the distribution of $X$ is
\begin{equation}
\label{eq:16}
X \sim
\begin{cases}
\bar N(\mu_1, I_2)  & \text{with probability $1/4$} \\
\bar N(\mu_2, I_2) & \text{with probability $1/4$}\\
\bar N(\mu_3, I_2) & \text{with probability $1/4$}\\
\bar N(\mu_4, I_2) & \text{with probability $1/4$}
\end{cases} \quad \text{where}\quad
\begin{cases}
\mu_1 = (0.3, 0.3)^{\intercal} \\
\mu_2 = (0.3, 0.7)^{\intercal} \\
\mu_3 = (0.7, 0.3)^{\intercal} \\
\mu_4 = (0.7, 0.7)^{\intercal} \\
\end{cases}
\end{equation}
and $\bar N$ denotes a truncated multivariate Gaussian distribution on the unit square.\footnote{
That is, $\bar N(\mu, \Sigma)$ denotes the conditional distribution of $x \sim N(\mu, \Sigma)$
on the event $x \in [0, 1]^2$.} Thus, $X$ has a bounded density on the unit square, and has four peaks
at $\mu_1, \dots, \mu_4$. The distribution of $Y$ conditional on $X = (x_1, x_2)$ is
\begin{equation}
\label{eq:18}
Y \sim \frac{x_1 + x_2}{2} + \frac{1}{5} N(0, 1).
\end{equation}
The random splitting probability is $\delta = 1/2$, and the regularity parameters
are $\alpha = 0.01$ and $k = 1$, so that the tree is grown to the fullest extent (i.e., terminal
nodes may contain a single observation), with each terminal node lying on the $101 \times 101$
grid of the unit square. For each sample size $n$, five thousand trees are grown, and the
estimates are aggregated to compute the correlation.

Figure~\ref{fig:1} plots the correlation of estimates at $x$ and $\bar x$
as a function of the $L_1$ norm $\|x - \bar x\|_1$. The calculation is performed by
first fixing $x$, then calculating the sample correlation (across five thousand
trees) as $\bar x$ ranges over each cell: the correlation is associated with
the $L_1$ norm $\|x - \bar x\|_1$. This process is then repeated by varying the reference point $x$,
and the correlation at $\| x - \bar x \|_1$ is the average of the correlations observed.
\begin{figure}[t]
  \centering
\includegraphics[width=\textwidth]{C.pdf}
  \caption{Correlation as a function of sample size and $L_1$ norm.}
  \label{fig:1}
\end{figure}

The figure demonsrates that the linear heuristic \eqref{linear-heuristic} given in the previous
section is conservative: it is evident that correlation decreases super-linearly as $x$ and $\bar x$
become separated. Figure~\ref{fig:2} plots the correlation on a logarithmic scale, which shows that
that correlation decay is exponential in a neighborhood of unity. In other words, simulations
suggest that the correct heuristic is of the shape
\begin{equation}
\label{eq:28}
\biggl|\frac{\Cov(T(x), T(\bar x))}{\sqrt{\Var T(x) \cdot \Var T(\bar x)}}\biggr| \approx
e^{-\lambda\| x - \bar x \|_1}
\end{equation}
for some suitable $\lambda$. Note, however, that the rate of decay of the correlation depends
heavily on the exact splitting algorithm employed. Thus, an avenue for future empirical work
is to investigate the decay behavior of popular random forest algorithms used in practice.
\begin{figure}[t]
  \centering
\includegraphics{Clog.pdf}
  \caption{Logarithm of correlation as a function of sample size and $L_1$ norm.}
  \label{fig:2}
\end{figure}

\FloatBarrier

\section{Conclusion and Points for Future Work}
  Regression trees and random forests are popular and effective
non-parametric estimators in practical applications. A recent paper by
Athey and Wager shows that the random forest estimate at any point is
asymptotically Gaussian; in this paper, we extend this result to the
multivariate case and show that the vector of estimates at multiple
points is jointly normal. Specifically, the covariance matrix of the
limiting normal distribution is diagonal, so that the estimates at any
two points are independent in sufficiently deep trees. Moreover, the
off-diagonal term is bounded by quantities capturing how likely two
points belong to the same partition of the resulting tree. Certainly stability
properties of the base learner are
central to our results, c.f.\ Propositions~\ref{prop:terminal-node-probability}
and \ref{prop:split-stability-sufficient}.
We test our proposed covariance
bound and the associated coverage rates of confidence intervals in
numerical simulations.

This paper provides the a theoretical basis for performing inferences
on functionals of target function (e.g., a heterogeneous treatment effect) when
the functional is based on values of the target function at multiple points in the feature space.
Specifically, we show that the covariance vanishes in the limit relative to the variance,
and provide heuristics on the size of the correlation in finite samples.

There are a couple interesting avenues for future work: the first is to extend our framework
to cover categorical or discrete-valued features. Here, new assumptions would be required
in order to maintain the guarantee that node sizes are ``not too small.'' Another direction is to use
our bounds---with possible improvements---on the covariance matrix of the underlying $U$-statistic
and leverage recent results \cite{chernozhukov2017,chen2018} in order to provite finite sample bounds. This would provide a more sound
theoretical underpinning of our heuristics.