\section{Introduction}
Given pairs of observations $(x_{i},y_{i}) \in \mathbb{R}^{p} \times \mathbb{R}^{q}$ for $i=1,\ldots,n$, assume they are generated by independently identically distributed (\emph{iid}) $F_{\ensuremath{X} \ensuremath{Y}}$.
A fundamental statistical question prior to the pursuit of any meaningful joint inference is the independence testing problem: the two random variables are independent if and only if $F_{\ensuremath{X} \ensuremath{Y}} = F_{\ensuremath{X}} F_{\ensuremath{Y}}$, i.e., the joint distribution equals the product of the marginals. The statistical hypothesis is formulated as:
\begin{align*}
& H_{0}: F_{\ensuremath{X} \ensuremath{Y}}=F_{\ensuremath{X}}F_{\ensuremath{Y}},\\
& H_{A}: F_{\ensuremath{X} \ensuremath{Y}} \neq F_{\ensuremath{X}}F_{\ensuremath{Y}}.
\end{align*}
For any test statistic, the testing power at a given type $1$ error level equals the probability of correctly rejecting the null hypothesis when the random variables are dependent. A test is consistent if and only if the testing power converges to $1$ as the sample size increases to infinity, and a valid test must properly control the type $1$ error level. Modern datasets are often nonlinear, high-dimensional, and noisy, where density estimation and traditional statistical methods fail to be applicable. As multi-modal data are prevalent in much data-intensive research, a powerful, intuitive, and easy-to-use method for detecting general relationships is pivotal.

The classical Pearson's correlation \cite{Pearson1895} is still extensively employed in statistics, machine learning, and real-world applications. It is an intuitive statistic that quantifies the linear association, a special but extremely important relationship. A recent surge of interests has been placed on using distance metrics and kernel transformations to achieve consistent independence testing against all dependencies. A notable example is the distance correlation (\sct{Dcorr}) \cite{SzekelyRizzoBakirov2007, SzekelyRizzo2009, SzekelyRizzo2013a,SzekelyRizzo2014}: the population \sct{Dcorr}~is defined via the characteristic functions of the underlying random variables, while the sample \sct{Dcorr}~can be conveniently computed via the pairwise Euclidean distances of given observations. \sct{Dcorr}~enjoys universal consistency against any joint distribution of finite second moments, and is applicable to any metric space of strong negative type \cite{Lyons2013}. Notably, the idea of distance-based correlation measure can be traced back to the Mantel coefficient \cite{Mantel1967, JosseHolmes2013}: the sample version differs from sample \sct{Dcorr}~only in centering, garnered popularity in ecology and biology applications, but does not have the consistency property of \sct{Dcorr}.

Developed almost in parallel from the machine learning community, the kernel-based method (\sct{Hsic}) \cite{GrettonEtAl2005,GrettonGyorfi2010} has a striking similarity with \sct{Dcorr}: it is formulated by kernels instead of distances, can be estimated on sample data via the sample kernel matrix, and is universally consistent when using any characteristic kernel. Indeed, it is shown in \cite{SejdinovicEtAl2013} that there exists a mapping from kernel to metric (and vice versa) such that \sct{Hsic}~equals \sct{Dcorr}. Another competitive method is the Heller-Heller-Gorfine method (\sct{Hhg}) \cite{HellerGorfine2013, heller2016consistent}: it is also universally consistent by utilizing the rank information and the Pearson's chi-square test, but has better finite-sample testing powers over \sct{Dcorr}~in a collection of common nonlinear dependencies. There are other consistent methods available, such as the \sct{Copula}~method that tests independence based on the empirical copula process \cite{Genest06,Genest07,Holmes09}, entropy-based methods \cite{Mendes06}, and methods tailored for univariate data \cite{Reshef2011}.

As the number of observations in many real world problems (e.g., genetics and biology) are often limited and very costly to increase,  finite-sample testing power is crucial for certain data exploration tasks: \sct{Dcorr}~has been shown to perform well in monotone relationships, but not so well in nonlinear dependencies such as circles and parabolas; the performance of \sct{Hsic}~and \sct{Hhg}~are often the opposite of \sct{Dcorr}, which perform slightly inferior to \sct{Dcorr}~in monotone relationships but excel in various nonlinear dependencies.

From another point of view, unraveling the nonlinear structure has been intensively studied in the manifold learning literature \cite{TenenbaumSilvaLangford2000, SaulRoweis2000, BelkinNiyogi2003}: by approximating a linear manifold locally via the k-nearest neighbors at each point, these nonlinear techniques can produce better embedding results than linear methods (like PCA) in nonlinear data. The main downside of manifold learning often lies in the parameter choice, i.e., the number of neighbor or the correct embedding dimension is often hard to estimate and requires cross-validation. Therefore, assuming a satisfactory neighborhood size can be efficiently determined in a given nonlinear relationship, the local correlation measure shall work better than the global correlation measure; and if the parameter selection is sufficiently adaptive, the optimal local correlation shall equal the global correlation in linear relationships.

In this manuscript we formalize the notion of population local distance correlations and \sct{MGC}, explore their theoretical properties both asymptotically and in finite-sample, and propose an improved Sample \sct{MGC}~algorithm. By combing distance correlation with the locality principle, \sct{MGC}~inherits the universal consistency in testing, is able to efficiently search over all local scales and determine the optimal correlation, and enjoys the best testing powers throughout the simulations. A number of real data applications via \sct{MGC}~are pursued in \cite{ShenEtAl2016}, e.g., testing brain images versus personality and disease, identify potential protein biomarkers for cancer, etc. And \sct{MGC}~are employed for vertex dependence testing and screening in \cite{mgc3,mgc4}.

The paper is organized as follows: In Section~\ref{sec:main1}, we define the population local distance correlation and population \sct{MGC}~via the characteristic functions of the underlying random variables and the nearest neighbor graphs, and show how the local variants are related to the distance correlation. In Section~\ref{sec:main2}, we consider the sample local correlation on finite-samples, prove its convergence to the population version, and discuss the centering and ranking scheme. In Section~\ref{sec:main3}, we present a thresholding-based algorithm for Sample \sct{MGC}, prove its convergence property, propose a theoretically sound threshold choice, manifest that \sct{MGC}~is valid and consistent under the permutation test, and finish the section with a number of fundamental properties for the local correlations and \sct{MGC}. The comprehensive simulations in Section~\ref{sec:exp} exhibits the empirical advantage of \sct{MGC}, and the paper is concluded in Section~\ref{sec:dis}.
All proofs are in Appendix~\ref{sec:proofs}, the simulation functions are presented in Appendix~\ref{appen:function}, and the code are available on Github \if11
{\footnote{\url{https://github.com/neurodata/mgc-matlab}} }\fi
\if01
\fi
and CRAN \if11
{\footnote{\url{https://CRAN.R-project.org/package=mgc}}}\fi
\if01
\fi.

\section{Multiscale Graph Correlation for Random Variables}
\label{sec:main1}

\subsection{Distance Correlation Review}
We first review the original distance correlation in \cite{SzekelyRizzoBakirov2007}. A non-negative weight function $w(t,s)$ on $(t,s) \in \mathbb{R}^{p} \times \mathbb{R}^{q}$ is defined as:
\begin{align*}
		w(t,s) &=  (c_{p}c_{q} |t|^{1+p}|s|^{1+q})^{-1},
\end{align*}
where $c_{p}=\frac{\pi^{(1+p)/2}}{\Gamma((1+p)/2)}$ is a non-negative constant tied to the dimensionality $p$, and $\Gamma(\cdot)$ is the complete Gamma function. Then the population distance covariance, variance and correlation are defined by
\begin{align*}
dCov(\ensuremath{X},\ensuremath{Y}) &=  \int_{\mathbb{R}^{p} \times \mathbb{R}^{q}} |E(g_{\ensuremath{X} \ensuremath{Y}}(t,s))-E(g_{\ensuremath{X}}(t))E(g_{\ensuremath{Y}}(s))|^{2} w(t, s)dtds, \\
dVar(\ensuremath{X}) &= dCov(\ensuremath{X},\ensuremath{X}), \\
dVar(\ensuremath{Y}) &= dCov(\ensuremath{Y},\ensuremath{Y}), \\
dCorr(\ensuremath{X},\ensuremath{Y}) &= \frac{dCov(\ensuremath{X},\ensuremath{Y})}{\sqrt{dVar(\ensuremath{X}) \cdot dVar(\ensuremath{Y})}},
\end{align*}
where $|\cdot|$ is the complex modulus, $g_{\cdot}(\cdot)$ denotes the exponential transformation within the expectation of the characteristic function, i.e., $g_{\ensuremath{X} \ensuremath{Y}}(t,s) = e^{\textbf{i} \left\langle t,\ensuremath{X} \right\rangle  +\textbf{i} \left\langle  s,\ensuremath{Y} \right\rangle }$ ($\textbf{i}$ represents the imaginary unit) and $E(g_{\ensuremath{X} \ensuremath{Y}}(t,s))$ is the characteristic function. Note that distance variance equals $0$ if and only if the random variable is a constant, in which case distance correlation shall be set to $0$. The main property of population \sct{Dcorr}~is the following.
\begin{thm*}
For any two random variables $(\ensuremath{X},\ensuremath{Y})$ with finite first moments, $dCorr(\ensuremath{X},\ensuremath{Y})=0$ if and only if $\ensuremath{X}$ and $\ensuremath{Y}$ are independent.
\end{thm*}

To estimate the population version on sample data, the sample distance covariance is computed by double centering the pairwise Euclidean distance matrix of each data, followed by summing over the entry-wise product of the two centered distance matrices. When the underlying random variables have finite second moments, the sample \sct{Dcorr}~is shown to converge to the population \sct{Dcorr}~, and is thus universally consistent for testing independence against all joint distributions of finite second moments.

\subsection{Population Local Correlations}
\label{main1a}

Next we formally define the population local distance covariance, variance, correlation by combining the k-nearest neighbor graphs with the distance covariance. For simplicity, they are named the local covariance, local variance, and local correlation from now on, and we always assume the following regularity conditions:
\begin{align*}
& 1) \mbox{ $(\ensuremath{X},\ensuremath{Y})$ have finite second moments}, \\
& 2) \mbox{ Neither random variable is a constant}, \\
& 3) \mbox{ $(\ensuremath{X},\ensuremath{Y})$ are continuous random variables}.
\end{align*}
The finite second moments assumption is required by \sct{Dcorr}, and also required by the local version to establish convergence and consistency. The non-constant condition is to avoid the trivial case and make sure population local correlations behave well. The continuous assumption is for ease of presentation, so the definition and related properties can be presented in a more elegant manner. Indeed, for any discrete random variable one can always apply jittering (i.e., add trivial white noise) to make it continuous without altering the independence testing.

\begin{defi*}
Suppose $(\ensuremath{X},\ensuremath{Y}), (\ensuremath{X}',\ensuremath{Y}'), (\ensuremath{X}'',\ensuremath{Y}''), (\ensuremath{X}''',\ensuremath{Y}''')$ are \emph{iid} as $F_{\ensuremath{X} \ensuremath{Y}}$. Let $\mb{I}(\cdot)$ be the indicator function, define two random variables
\begin{align*}
\mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}} &=\mb{I}(\int_{B(\ensuremath{X},\|\ensuremath{X}'-\ensuremath{X}\|)} dF_\ensuremath{X}(u) \leq \rho_k)  \\
\mb{I}_{\ensuremath{Y}',\ensuremath{Y}}^{\rho_{l}} &=\mb{I}(\int_{B(\ensuremath{Y}',\|\ensuremath{Y}'-\ensuremath{Y}\|)} dF_\ensuremath{Y}(v) \leq \rho_l)
\end{align*}
with respect to the closed balls $B(\ensuremath{X},\|\ensuremath{X}'-\ensuremath{X}\|)$ and $B(\ensuremath{Y}',\|\ensuremath{Y}-\ensuremath{Y}'\|)$ centered at $\ensuremath{X}$ and $\ensuremath{Y}'$ respectively. Then let $\overline{\cdot}$ denote the complex conjugate, define
\begin{align*}
h^{\rho_{k}}_{\ensuremath{X}}(t) &=(g_{\ensuremath{X}}(t)\overline{g_{\ensuremath{X}'}(t)}-g_{\ensuremath{X}}(t)\overline{g_{\ensuremath{X}''}(t)}) \mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}} \\
h^{\rho_{l}}_{\ensuremath{Y}'}(s) &=(g_{\ensuremath{Y}'}(s)\overline{g_{\ensuremath{Y}}(s)}-g_{\ensuremath{Y}'}(s)\overline{g_{\ensuremath{Y}'''}(s)}) \mb{I}_{\ensuremath{Y}',\ensuremath{Y}}^{\rho_{l}}
\end{align*}
as functions of $t \in \mathbb{R}^{p}$ and $s \in \mathbb{R}^{q}$ respectively,

The population local covariance, variance, correlation at any $(\rho_k,\rho_l) \in [0,1] \times [0,1]$ are defined as
\begin{align}
\label{eq:dcov1}
dCov^{\rho_k, \rho_l}(\ensuremath{X},\ensuremath{Y}) &= \int_{\mathbb{R}^{p} \times \mathbb{R}^{q}} \{ E(h^{\rho_{k}}_{\ensuremath{X}}(t) \overline{h^{\rho_{l}}_{\ensuremath{Y}'}(s)})-E(h^{\rho_{k}}_{\ensuremath{X}}(t))E(\overline{h^{\rho_{l}}_{\ensuremath{Y}'}(s)})\} w(t, s)dtds,\\
dVar^{\rho_k}(\ensuremath{X}) &= dCov^{\rho_k, \rho_k}(\ensuremath{X},\ensuremath{X}), \nonumber \\
dVar^{\rho_l}(\ensuremath{Y}) &= dCov^{\rho_l, \rho_l}(\ensuremath{Y},\ensuremath{Y}), \nonumber \\
dCorr^{\rho_k,\rho_l}(\ensuremath{X},\ensuremath{Y}) &= \frac{dCov^{\rho_k,\rho_l}(\ensuremath{X},\ensuremath{Y})}{\sqrt{dVar^{\rho_k}(\ensuremath{X}) \cdot dVar^{\rho_l}(\ensuremath{Y})}},
\end{align}
where we limit the domain of population local correlation to
\begin{align*}
\mathcal{S}_{\epsilon}=\big\{(\rho_{k},\rho_{l}) \in [0,1] \times [0,1] \mbox{ that satisfies } \min\{dVar^{\rho_k}(X),dVar^{\rho_l}(Y)\} \geq \epsilon\big\}
\end{align*}
for a small positive $\epsilon$ that is no larger than $\min\{dVar(X),dVar(Y)\}$.
\end{defi*}

The domain of local correlation needs to be limited so the population version is well-behaved. For example, when $\ensuremath{X}$ is a constant or $\rho_{k}=0$, $dVar^{\rho_k}(X)$ equals $0$ and the corresponding local correlation is not well-defined. All subsequent analysis for the population local correlations is based on the domain $\mathcal{S}_{\epsilon}$, which is non-empty and compact as shown in Theorem~\ref{thmMax}. In practice, it suffices to set $\epsilon$ as any small positive number, see the sample version in Section~\ref{sec:main2}. Also note that in either indicator function, the two random variables and the distribution $dF$ are independent, e.g., at any realization $(x, x')$ of $(\ensuremath{X},\ensuremath{X}')$, the first indicator equals $\mb{I}(\int_{B(x,\|x'-x\|)} dF_\ensuremath{X}(u) \leq \rho_k)$, and its expectation is taken with respect to $(\ensuremath{X},\ensuremath{X}')$.

The above definition makes use of the characteristic functions, which is akin to the original definition of \sct{Dcorr}~and easier to show consistency. Alternatively, the local covariance can be equivalently defined via the pairwise Euclidean distances. The alternative definition better motivates the sample version in Section~\ref{sec:main2}, is often handy for understanding and proving theoretical properties, and suggests that local covariance is always a real number, which is not directly obvious from Equation~\ref{eq:dcov1}.
\begin{thm}
\label{thm1}
Suppose $(\ensuremath{X},\ensuremath{Y}),(\ensuremath{X}',\ensuremath{Y}'),(\ensuremath{X}'',\ensuremath{Y}''),(\ensuremath{X}''',\ensuremath{Y}''')$ are \emph{iid} as $F_{\ensuremath{X} \ensuremath{Y}}$, and define
\begin{align*}
d^{\rho_{k}}_{\ensuremath{X}} &=(\| \ensuremath{X}-\ensuremath{X}' \| - \|\ensuremath{X}-\ensuremath{X}''\|) \mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}} \\
d^{\rho_{l}}_{\ensuremath{Y}'} &=(\| \ensuremath{Y}'-\ensuremath{Y} \| - \|\ensuremath{Y}'-\ensuremath{Y}'''\|) \mb{I}_{\ensuremath{Y}',\ensuremath{Y}}^{\rho_{l}}
\end{align*}

The local covariance in Equation~\ref{eq:dcov1} can be equally defined as
\begin{align}
\label{eq:dcov2}
dCov^{\rho_k, \rho_l}(\ensuremath{X},\ensuremath{Y}) = E(d^{\rho_{k}}_{\ensuremath{X}} d^{\rho_{l}}_{\ensuremath{Y}'}) - E(d^{\rho_{k}}_{\ensuremath{X}}) E(d^{\rho_{l}}_{\ensuremath{Y}'}),
\end{align}
which shows that local covariance, variance, correlation are always real numbers.
\end{thm}

Each local covariance is essentially a local version of distance covariance that truncates large distances at each point in the support, where the neighborhood size is determined by $(\rho_{k},\rho_{l})$. In particular, distance correlation equals the local correlation at the maximal scale, which will ensure the consistency of \sct{MGC}.

\begin{thm}
\label{thm2}
At any $(\rho_k,\rho_l)\in \mathcal{S}_{\epsilon}$, $dCov^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y}) = 0$ when $\ensuremath{X}$ and $\ensuremath{Y}$ are independent. Moreover, at $(\rho_k,\rho_l)=(1,1)$, $dCov^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y}) = dCov(\ensuremath{X},\ensuremath{Y})$. They also hold for the correlations by replacing all the $dCov$ by $dCorr$.
\end{thm}

\subsection{Population \sct{MGC}~and Optimal Scale}
The population \sct{MGC}~can be naturally defined as the maximum local correlation within the domain, i.e.,
\begin{align}
\label{eq:pmgc}
c^{*}(\ensuremath{X},\ensuremath{Y})=\max_{ (\rho_k,\rho_l) \in \mathcal{S}_{\epsilon}} \{dCorr^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y})\},
\end{align}
and the scale that attains the maximum is named the optimal scale
\begin{align}
\label{eq:pmgc1}
(\rho_k,\rho_l)^{*}=\arg\max_{ (\rho_k,\rho_l) \in \mathcal{S}_{\epsilon}} \{dCorr^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y})\}.
\end{align}
The next theorem states the continuity of the local covariance, variance, correlation, and thus the existence of population \sct{MGC}.
\begin{thm}
\label{thmMax}
Given two continuous random variables $(\ensuremath{X},\ensuremath{Y})$,
\begin{description}
\item [(a)] The local covariance is a continuous function with respect to $(\rho_{k},\rho_{l}) \in [0,1]^2$, so is local variance in $[0,1]$ and local correlation in $\mathcal{S}_{\epsilon}$.
\item [(b)] The set $\mathcal{S}_{\epsilon}$ is always non-empty unless either random variable is a constant.
\item [(c)] Excluding the trivial case in (b), the set $\{dCorr^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y}), (\rho_k,\rho_l) \in \mathcal{S}_{\epsilon}\}$ is always non-empty and compact, so an optimal scale $(\rho_k,\rho_l)^{*}$ and $c^{*}(\ensuremath{X},\ensuremath{Y})$ exist.
\end{description}
\end{thm}

Therefore, population \sct{MGC}~and the optimal scale exist, are distribution dependent, and may not be unique. Without loss of generality, the optimal scale is assumed unique for presentation purpose. The population \sct{MGC}~is always no smaller than \sct{Dcorr}~in magnitude, and equals $0$ if and only if independence, a property inherited from \sct{Dcorr}.
\begin{thm}
\label{thm3}
When $\ensuremath{X}$ and $\ensuremath{Y}$ are independent, $c^{*}(\ensuremath{X},\ensuremath{Y})=dCorr(\ensuremath{X},\ensuremath{Y})=0$; when $\ensuremath{X}$ and $\ensuremath{Y}$ are not independent, $c^{*}(\ensuremath{X},\ensuremath{Y}) \geq dCorr(\ensuremath{X},\ensuremath{Y})>0$.
\end{thm}

\section{Sample Local Correlations}
\label{sec:main2}

Sample \sct{Dcorr}~can be easily calculated via properly centering the Euclidean distance matrices, and is shown to converge to the population \sct{Dcorr}~\cite{SzekelyRizzoBakirov2007, SzekelyRizzo2013a, SzekelyRizzo2014}. Similarly, we show that the sample local correlation can be calculated via the Euclidean distance matrices upon truncating large distances for each sample observation, and the sample version converges to the respective population local correlation.

\subsection{Definition}
\label{sec:defi}
Given pairs of observations $(x_{i},y_{i}) \in \mathbb{R}^{p} \times \mathbb{R}^{q}$ for $i=1,\ldots,n$, denote $\mathcal{X}_{n}=[x_{1},\ldots,x_{n}]$ as the data matrix with each column representing one sample observation, and similarly $\mathcal{Y}_{n}$. Let $\tilde{A}$ and $\tilde{B}$ be the $n \times n$ Euclidean distance matrices of $\mathcal{X}_{n}=\{x_{i}\}$ and $\mathcal{Y}_{n}=\{y_{i}\}$ respectively, i.e., $\tilde{A}_{ij}=\|x_{i}-x_{j}\|$. Then we compute two column-centered matrices $A$ and $B$ with the diagonals excluded, i.e., $\tilde{A}$ and $\tilde{B}$ are centered within each column such that
\begin{equation}
\label{localCoef2}
    A_{ij}=
    \begin{cases}
      \tilde{A}_{ij}-\frac{1}{n-1}\sum_{s=1}^{n} \tilde{A}_{sj}, & \text{if $i \neq j$}, \\
      0, & \text{if $i=j$};
    \end{cases} \qquad \qquad
    B_{ij}=
    \begin{cases}
      \tilde{B}_{ij}-\frac{1}{n-1}\sum_{s=1}^{n} \tilde{B}_{sj}, & \text{if $i \neq j$}, \\
      0, & \text{if $i=j$};
    \end{cases}
\end{equation}

Next we define $\{R^{A}_{ij}\}$ as the ``rank'' of $x_i$ relative to $x_j$, that is, $R^{A}_{ij}=k$ if $x_i$ is the $k^{th}$ closest point (or ``neighbor'') to $x_j$, as determined by ranking the set $\{\tilde{A}_{1j},\tilde{A}_{2j},\ldots,\tilde{A}_{nj}\}$ by ascending order. Similarly define $R^{B}_{ij}$ for the $y$'s. As we assumed $(\ensuremath{X},\ensuremath{Y})$ are continuous, with probability $1$ there is no repeating observation and the ranks always take value in $\{1,\ldots,n\}$. In practice ties may occur, and we recommend either using minimal rank to keep the ties or jittering to break the ties, which is discussed at the end of this section.

For any $(k,l) \in [n]^2=\{1,\ldots,n\} \times \{1,\ldots,n\}$, we define the rank truncated matrices $A^{k}$ and $B^{l}$ as
\begin{align*}
A_{ij}^{k} &=A_{ij} \mb{I}(R^{A}_{ij} \leq k), \\
B_{ij}^{l} &=B_{ij} \mb{I}(R^{B}_{ij} \leq l).
\end{align*}
Let $\circ$ denote the entry-wise product, $\hat{E}(\cdot)=\frac{1}{n(n-1)}\sum_{i \neq j}^{n} (\cdot)$ denote the diagonal-excluded sample mean of a square matrix, then the sample local covariance, variance, and correlation are defined as:
\begin{align*}
dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) &= \hat{E}(A^{k} \circ B^{l'})- \hat{E}(A^{k})\hat{E}(B^{l}),\\
dVar^{k}(\mathcal{X}_{n}) &=\hat{E}(A^{k} \circ A^{k'})- \hat{E}^2(A^{k}), \\
dVar^{l}(\mathcal{Y}_{n}) &=\hat{E}(B^{l} \circ B^{l'})- \hat{E}^2(B^{l}), \\
dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) &=dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) / \sqrt{dVar^{k}(\mathcal{X}_{n}) \cdot dVar^{l}(\mathcal{Y}_{n})}.
\end{align*}
If either local variance is smaller than a preset $\epsilon > 0$ (e.g., the smallest positive local variance among all), then we set the corresponding $dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})=0$ instead. Note that once the rank is known, sample local correlations can be iteratively computed in $\mathcal{O}(n^2)$ rather than a naive implementation of $\mathcal{O}(n^3)$. A detailed running time comparison is presented in Section~\ref{sec:exp}.

In case of ties, minimal rank offers a consecutive indexing of sample local correlations, e.g., if $\ensuremath{Y}$ only takes two values, $R^{B}_{ij}$ takes value in $\{1,2\}$ under minimal rank, but maximal rank yields $\{\frac{n}{2},n\}$. The sample local correlations are not affected by the tie scheme, but minimal rank is more convenient to work with for implementation purposes. Alternatively, one can break ties deterministically or randomly, e.g., apply jittering to break all ties. For example, in the Bernoulli relationship of Figure~\ref{f:dependencies}, there are only three points for computing sample local correlations and the Sample \sct{MGC}~equals $0.9$. If white noise of variance $0.01$ were added to the data, we break all ties and obtain a much larger number of sample local correlations. The resulting Sample \sct{MGC}~is $0.8$, which is slightly smaller but still much larger than $0$ and implies a strong dependency.

Whether the random variable is continuous or discrete, and whether the ties in sample data are broken or not, does not affect the theoretical results except in certain theorem statements. For example, in Theorem~\ref{thm4}, the convergence still holds for discrete random variables, but the index pair $(k,l)$ does not necessarily correspond to the population version at $(\rho_{k},\rho_{l})=(\frac{k-1}{n-1}, \frac{l-1}{n-1})$, e.g., when $\ensuremath{X}$ is Bernoulli with probability $0.8$ and minimal rank is used, $k=1$ corresponds to $\rho_{k}=0.8$ instead of $\rho_{k}=\frac{k-1}{n-1}$. Nevertheless, Theorem~\ref{thm4} and all results in the paper hold regardless of continuous or discrete random variables, but the presentation is more elegant for the continuous case.

\subsection{Convergence Property}
The sample local covariance, variance, correlation are designed to converge to the respective population versions. Moreover, the expectation of sample local covariance equals the population counterpart up to a difference of $\mathcal{O}(\frac{1}{n})$, and the variance diminishes at the rate of $\mathcal{O}(\frac{1}{n})$.
\begin{thm}
\label{thm4}
Suppose each column of $\mathcal{X}_{n}$ and $\mathcal{Y}_{n}$ are generated \emph{iid} from $(\ensuremath{X},\ensuremath{Y}) \sim F_{\ensuremath{X} \ensuremath{Y}}$. The sample local covariance satisfies
\begin{align*}
E(dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})) &= dCov^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y}) +\mathcal{O}(1/n) \\
Var(dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})) &= \mathcal{O}(1/n)\\
dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) &\stackrel{n \rightarrow \infty}{\rightarrow} dCov^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y}),
\end{align*}
where $\rho_{k}=\frac{k-1}{n-1}$ and $\rho_{l}=\frac{l-1}{n-1}$. In particular, the convergence is uniform and also holds for the local correlation, i.e., for any $\epsilon$ there exists $n_{\epsilon}$ such that for all $n > n_{\epsilon}$,
\begin{align*}
|dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) -dCorr^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y})|< \epsilon
\end{align*}
for any pair of $(\rho_{k},\rho_{l}) \in \mathcal{S}_{\epsilon}$.
\end{thm}

The convergence property ensures that Theorem~\ref{thm2} holds asymptotically for the sample version.
\begin{cor}
\label{thm5}
For any $(k,l)$, $dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) \rightarrow 0$ when $\ensuremath{X}$ and $\ensuremath{Y}$ are independent. In particular, $dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n}) \rightarrow dCorr(\ensuremath{X},\ensuremath{Y})$.

\end{cor}

Moreover, one can show that $dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n}) \approx dCorr(\mathcal{X}_{n},\mathcal{Y}_{n})$ for the unbiased sample distance correlation in \cite{SzekelyRizzo2014} up-to a small difference of $\mathcal{O}(\frac{1}{n})$, which can be verified by comparing Equation~\ref{localCoef2} to Equation 3.1 in \cite{SzekelyRizzo2014}.

\subsection{Centering and Ranking}
To combine distance testing with the locality principle, other than the procedure proposed in Equation~\ref{eq:dcov2}, there are a number of alternative options to center and rank the distance matrices. For example, letting
\begin{align*}
d^{\rho_{k}}_{\ensuremath{X}} &=(\| \ensuremath{X}-\ensuremath{X}' \| - \|\ensuremath{X}-\ensuremath{X}''\|-\|\ensuremath{X}'-\ensuremath{X}''\|+\|\ensuremath{X}''-\ensuremath{X}'''\|) \mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}}, \\
d^{\rho_{l}}_{\ensuremath{Y}'} &=(\| \ensuremath{Y}'-\ensuremath{Y} \| - \|\ensuremath{Y}'-\ensuremath{Y}''\|-\|\ensuremath{Y}-\ensuremath{Y}''\|+\|\ensuremath{Y}''-\ensuremath{Y}'''\|) \mb{I}_{\ensuremath{Y}',\ensuremath{Y}}^{\rho_{l}}
\end{align*}
still guarantees the resulting local correlation at maximal scale equals the distance correlation; and letting
\begin{align*}
d^{\rho_{k}}_{\ensuremath{X}} &=\| \ensuremath{X}-\ensuremath{X}' \|  \mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}}, \\
d^{\rho_{l}}_{\ensuremath{Y}'} &=\| \ensuremath{Y}'-\ensuremath{Y} \|  \mb{I}_{\ensuremath{Y}',\ensuremath{Y}}^{\rho_{l}},
\end{align*}
makes the resulting local correlation at maximal scale equal the \sct{Mantel}~coefficient, the earliest distance-based correlation coefficient.

Nevertheless, the centering and ranking strategy proposed in Equation~\ref{eq:dcov2} is more faithful to k-nearest neighbor graph: the indicator $\mb{I}_{\ensuremath{X},\ensuremath{X}'}^{\rho_{k}}$ equals $1$ if and only if $\int_{B(\ensuremath{X},\|\ensuremath{X}'-\ensuremath{X}\|)} dF_\ensuremath{X}(u) \leq \rho_k$, which happens with probability $\rho_{k}$. Viewed another way, when conditioned on $(\ensuremath{X},\ensuremath{X}')=(x,x')$, the indicator equals $1$ if and only if $Prob(\|x'-x\|<\|\ensuremath{X}''-x\|) \leq \rho_{k}$, thus matching the column ranking scheme in Equation~\ref{localCoef2}. Indeed, the locality principle used in \cite{TenenbaumSilvaLangford2000, SaulRoweis2000, BelkinNiyogi2003} considers the k-nearest neighbors of each sample point in local computation, an essential step to yield better nonlinear embeddings.

On the centering side, the \sct{Mantel}~test appears to be an attractive option due to its simplicity in centering. All the \sct{Dcorr}, \sct{Hhg}, \sct{Hsic}~have their theoretical consistency, while the \sct{Mantel}~coefficient does not, despite it being merely a different centering of \sct{Dcorr}. An investigation of the population form of \sct{Mantel}~yields some additional insights:
\begin{defi*}
Given $\mathcal{X}_{n}$ and $\mathcal{Y}_{n}$, the \sct{Mantel}~coefficient on sample data is computed as
\begin{align*}
M(\mathcal{X}_{n},\mathcal{Y}_{n}) &= \hat{E}(\tilde{A} \circ \tilde{B})-\hat{E}(\tilde{A})\hat{E}(\tilde{B}) \\
Mantel(\mathcal{X}_{n},\mathcal{Y}_{n}) &= \frac{M(\mathcal{X}_{n},\mathcal{Y}_{n})}{\sqrt{M(\mathcal{X}_{n},\mathcal{X}_{n}) M(\mathcal{Y}_{n},\mathcal{Y}_{n})}},
\end{align*}
where $\tilde{A}_{ij}$ and $\tilde{B}_{ij}$ are the pairwise Euclidean distance, and $\hat{E}(\cdot)=\frac{1}{n(n-1)}\sum_{i \neq j}^{n} (\cdot)$ is the diagonal-excluded sample mean of a square matrix.
\end{defi*}

\begin{cor}
\label{cor1}
Suppose each column of $\mathcal{X}_{n}$ and $\mathcal{Y}_{n}$ are \emph{iid} as $F_{\ensuremath{X} \ensuremath{Y}}$, and $(\ensuremath{X},\ensuremath{Y}), (\ensuremath{X}',\ensuremath{Y}')$ are also \emph{iid} as $F_{\ensuremath{X} \ensuremath{Y}}$. Then
\begin{align*}
Mantel(\mathcal{X}_{n},\mathcal{Y}_{n}) &\rightarrow Mantel(\ensuremath{X},\ensuremath{Y}) = \frac{M(\ensuremath{X},\ensuremath{Y})}{\sqrt{M(\ensuremath{X},\ensuremath{X}) M(\ensuremath{Y},\ensuremath{Y})}},
\end{align*}
where
\begin{align*}
M(\ensuremath{X},\ensuremath{Y}) &= \int_{\mathbb{R}^{p} \times \mathbb{R}^{q}} \{|E(g_{\ensuremath{X} \ensuremath{Y}}(t,s))|^2-|E(g_{\ensuremath{X}}(t))E(g_{\ensuremath{Y}}(s))|^{2}\} w(t, s)dtds \\
&= E(\| \ensuremath{X}-\ensuremath{X}' \| \| \ensuremath{Y}-\ensuremath{Y}' \|) - E(\|\ensuremath{X}-\ensuremath{X}'\|)E(\|\ensuremath{Y}-\ensuremath{Y}'\|)) \\
&= Cov(\| \ensuremath{X}-\ensuremath{X}' \|, \| \ensuremath{Y}-\ensuremath{Y}' \|).
\end{align*}
\end{cor}
Corollary~\ref{cor1} suggests that \sct{Mantel}~is actually a two-sided test based on the absolute difference of characteristic functions: under certain dependency structure, the \sct{Mantel}~coefficient can be negative and still imply dependency (i.e., $|E(g_{\ensuremath{X} \ensuremath{Y}}(t,s))| < |E(g_{\ensuremath{X}}(t))E(g_{\ensuremath{Y}}(s))|$); whereas population \sct{Dcorr}~and \sct{MGC}~are always no smaller than $0$, and any negativity of the sample version does not imply dependency. Therefore, \sct{Mantel}~is only appropriate as a two-sided test, which is evaluated in Section~\ref{sec:exp}.

Another insight is that \sct{Mantel}, unlike \sct{Dcorr}, is not universally consistent: due to the integral $w$, one can construct a joint distribution such that the population \sct{Mantel}~equals $0$ under dependence (see Remark 3.13 in \cite{Lyons2013} for an example of dependent random variables with uncorrelated distances). However, empirically, simple centering is still effective in a number of common dependencies (like two parabolas and diamond in Figure~\ref{f:1DAll}).

\section{Sample \sct{MGC}~and Estimated Optimal Scale}
\label{sec:main3}

A naive sample version of \sct{MGC}~can be defined as the maximum of all sample local correlations
\begin{align*}
\max_{(k,l) \in [n]^2}\{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})\}.
\end{align*}
Although the convergence to population \sct{MGC}~can be guaranteed, the sample maximum is a biased estimator of the population \sct{MGC}~in Equation~\ref{eq:pmgc}. For example, under independence, population \sct{MGC}~equals $0$, while the maximum sample local correlation has expectation larger than $0$, which may negate the advantage of searching locally and hurt the testing power.

This motivates us to compute Sample \sct{MGC}~as a smoothed maximum within the largest connected region of thresholded local correlations. The purpose is to mitigate the bias of a direct maximum, while maintaining its advantage over \sct{Dcorr}~in the test statistic. The idea is that in case of dependence, local correlations on the grid near the optimal scale shall all have large correlations; while in case of independence, a few local correlations may happen to be large, but most nearby local correlations shall still be small. The idea can be similarly adapted whenever there are multiple correlated test statistics or multiple models available, for which taking a direct maximum may yield too much bias \cite{mgc3}. From another perspective, Sample \sct{MGC}~is like taking a regularized maximum.

\subsection{Sample \sct{MGC}}
The procedure is as follows:
\begin{description}[align=left]
\item [Input: ] A pair of datasets $(\mathcal{X}_{n}, \mathcal{Y}_{n})$.
\item [Compute the Local Correlation Map:] Compute all local correlations:\\
$\{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}), (k,l) \in [n]^2\}$.
\item [Thresholding: ] Pick a threshold $\tau_n \geq 0$, denote $LC(\cdot)$ as the operation of taking the largest connected component, and compute the largest region $R$ of thresholded local correlations:
\begin{align}
\label{eq:region}
&R=LC(\{(k,l) \mbox{ such that } dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})>\max\{\tau_n, dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n})\} \}).
\end{align}
Within the region $R$, set
\begin{align}
\label{eq:smgc1}
c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})&=\max_{ (k,l) \in R} \{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})\}\\
(k_{n},l_{n})^{*}&=\arg\max_{ (k,l) \in R} \{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})\}
\end{align}
as the Sample \sct{MGC}~and the estimated optimal scale. If the number of elements in $R$ is less than $2n$, or the above thresholded maximum is no more than $dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n})$, we instead set $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})=dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n})$ and $(k_{n},l_{n})^{*}=(n,n)$.

\item [Output: ] Sample \sct{MGC}~$c^{*}(\mathcal{X}_{n}, \mathcal{Y}_{n})$ and the estimated optimal scale $(k_{n},l_{n})^{*}$.
\end{description}

If there are multiple largest regions, e.g., $R_{1}$ and $R_{2}$ where their number of elements are more than $2n$ and coincide with each other, then it suffices to let $R = R_{1} \displaystyle \cup R_{2}$ and locate the \sct{MGC}~statistic within the union. The selection of at least $2n$ elements for $R$ is an empirical choice, which balances the bias-variance trade-off well in practice. The parameter can be any positive integer without affecting the validity and consistency of the test. But if the parameter is too large, \sct{MGC}~tends to be more conservative and is unable to detect signals in strongly nonlinear relationships (e.g., trigonometric functions), and performs closer and closer to \sct{Dcorr}; if the parameter is set to a very small fixed number, the bias is inflated so \sct{MGC}~tends to perform similarly as directly maximizing all local correlations.

\subsection{Convergence and Consistency}
The proposed Sample \sct{MGC}~is algorithmically enforced to be no less than the local correlation at the maximal scale, and also no more than the maximum local correlation. It also ensures in Theorem~\ref{thm3} to hold for the sample version.
\begin{thm}
\label{thm6}
Regardless of the threshold $\tau_n$, the Sample \sct{MGC}~statistic $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})$ satisfies
\begin{description}
\item [(a)] It always holds that
\begin{align*}
\max_{(k,l) \in [n]^2}\{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})\} \geq c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}) \geq dCorr^{n,n}(\mathcal{X}_{n},\mathcal{Y}_{n}).
\end{align*}
\item [(b)] When $\ensuremath{X}$ and $\ensuremath{Y}$ are independent, $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}) \rightarrow 0$; when $\ensuremath{X}$ and $\ensuremath{Y}$ are not independent, $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}) \rightarrow$ a positive constant.
\end{description}
\end{thm}

The next theorem states that if the threshold $\tau_n$ converges to $0$, then whenever population \sct{MGC}~is larger than population \sct{Dcorr}, Sample \sct{MGC}~is also larger than sample \sct{Dcorr}~asymptotically; otherwise if the threshold does not converge to $0$, Sample \sct{MGC}~may equal sample \sct{Dcorr}~despite of the first moment advantage in population. Moreover, Sample \sct{MGC}~indeed converges to population \sct{MGC}~when the optimal scale is in the largest thresholded region $R$. The empirical advantage of Sample \sct{MGC}~is illustrated in Figure~\ref{f:dependencies}.

\begin{thm}
\label{thm7}
Suppose each column of $\mathcal{X}_{n}$ and $\mathcal{Y}_{n}$ are \emph{iid} as continuous $(\ensuremath{X}, \ensuremath{Y}) \sim F_{\ensuremath{X} \ensuremath{Y}}$, and the threshold choice $\tau_n \rightarrow 0$ as $n \rightarrow \infty$.
\begin{description}
\item [(a)] Assume that $c^{*}(\ensuremath{X}, \ensuremath{Y}) > Dcorr(\ensuremath{X}, \ensuremath{Y})$ under the joint distribution. Then $c^{*}(\mathcal{X}_{n}, \mathcal{Y}_{n}) > Dcorr(\mathcal{X}_{n}, \mathcal{Y}_{n})$ for $n$ sufficiently large.
\item [(b)] Assume there exists an element within the the largest connected area of $\{(\rho_k,\rho_l) \in \mathcal{S}_{\epsilon}$ with $dCorr^{\rho_k,\rho_l}(\ensuremath{X},\ensuremath{Y})> dCorr(\ensuremath{X},\ensuremath{Y}) \}$, such that the the local correlation of that element equals $c^{*}(\ensuremath{X},\ensuremath{Y})$. Then $c^{*}(\mathcal{X}_{n}, \mathcal{Y}_{n}) \rightarrow c^{*}(\ensuremath{X},\ensuremath{Y})$.
\end{description}
\end{thm}
Alternatively, Theorem~\ref{thm7}(b) can be stated that the Sample \sct{MGC}~always converges to the maximal population local correlation within the largest connected area of thresholded local correlations. Therefore, Sample \sct{MGC}~converges either to \sct{Dcorr}~(when the area is empty) or something larger, thus improving over \sct{Dcorr}~statistic in first moment.

\subsection{Choice of Threshold}

The choice of threshold $\tau_n$ is imperative for Sample \sct{MGC}~to enjoy a good finite-sample performance, especially at small sample size. According to Theorem~\ref{thm7}, the threshold shall converge to $0$ for Sample \sct{MGC}~to prevail sample \sct{Dcorr}.

A model-free threshold $\tau_n$ was previously used in \cite{ShenEtAl2016}: for the following set
\begin{align*}
\{dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) \mbox{ s.t. }  dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})<0\},
\end{align*}
let $\sigma^2$ be the sum of all its elements squared, and set $\tau_n=5\sigma$ as the threshold; if there is no negative local correlation and the set is empty, use $\tau_n=0.05$.

Although the previous threshold is a data-adaptive choice that works pretty well empirically and does not affect the consistency of Sample \sct{MGC}~in Theorem~\ref{thm8}, it does not converge to $0$. The following finite-sample theorem from \cite{SzekelyRizzo2013a} motivates an improved threshold choice here:
\begin{thm*}
Under independence of $(\ensuremath{X},\ensuremath{Y})$, assume the dimensions of $\ensuremath{X}$ are exchangeable with finite variance, and so are the dimensions of $\ensuremath{Y}$. Then for any $n \geq 4$ and $v=\frac{n(n-3)}{2}$, as $p,q$ increase the limiting distribution of $(dCorr^{n,n}(\mathcal{X}_{n}, \mathcal{Y}_{n})+1)/2$ equals the symmetric Beta distribution with shape parameter $\frac{v-1}{2}$.
\end{thm*}

The above theorem leads to the new threshold choice:
\begin{cor}
\label{cor2}
Denote $v=\frac{n(n-3)}{2}$, $z \sim Beta(\frac{v-1}{2})$, $F^{-1}_{z}(\cdot)$ as the inverse cumulative distribution function. The threshold choice
\begin{align*}
\tau_n= 2F^{-1}_{z} \Big(1-\frac{0.02}{n}\Big)-1
\end{align*}
converges to $0$ as $n \rightarrow \infty$.
\end{cor}
The limiting null distribution of \sct{Dcorr}~is still a good approximation even when $p,q$ are not large, thus provides a reliable bound for eliminating local correlations that are larger than \sct{Dcorr}~by chance or by noise. The intuition is that Sample \sct{MGC}~is mostly useful when it is much larger than \sct{Dcorr}~in magnitude, which is often the case in non-monotone relationships as shown in Section~\ref{sec:exp} Figure~\ref{f:dependencies}. Alternatively, directly setting $\tau_{n}=0$ also guarantees the theoretical properties and works equally well when the sample size $n$ is moderately large.

\subsection{Permutation Test}
\label{sec:permutation}

To test independence on a pair of sample data $(\mathcal{X}_{n},\mathcal{Y}_{n})$, the random permutation test has been the popular choice \cite{GoodPermutationBook} for almost all methods introduced, as the null distribution of the test statistic can be easily approximated by randomly permuting one data set. We discuss the computation procedure, prove the testing consistency of \sct{MGC}, and analyze the running time.

To compute the p-value of \sct{MGC}~from the permutation test, first compute the Sample \sct{MGC}~statistic $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})$ on the observed data pair. Then the \sct{MGC}~statistic is repeatedly computed on the permuted data pair, e.g. $\mathcal{Y}_{n}=[y_{1},\ldots,y_{n}]$ is permuted into $\mathcal{Y}_{n}^{\pi}=[y_{\pi(1)},\ldots,y_{\pi(n)}]$ for a random permutation $\pi$ of size $n$, and compute $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}^{\pi})$. The permutation procedure is repeated for $r$ times to estimate the probability $Prob(c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}^{\pi}) > c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n}))$, and the estimated probability is taken as the p-value of \sct{MGC}. The independence hypothesis is rejected if the p-value is smaller than a pre-set critical level, say $0.05$ or $0.01$. The following theorem states that \sct{MGC}~via the permutation test is consistent and valid.

\begin{thm}
\label{thm8}
Suppose each column of $\mathcal{X}_{n}$ and $\mathcal{Y}_{n}$ are generated \emph{iid} from $F_{\ensuremath{X} \ensuremath{Y}}$. At any type $1$ error level $\alpha>0$, Sample \sct{MGC}~is a valid test statistic that is consistent against all possible alternatives under the permutation test.
\end{thm}

\subsection{Miscellaneous Properties}
\label{sec:misc}

In this subsection, we first show a useful lemma expressing sample local covariance in Section~\ref{sec:defi} by matrix trace and eigenvalues, then list a number of fundamental and desirable properties for the local variance, local correlation, and \sct{MGC}, akin to these of Pearson's correlation and distance correlation as shown in \cite{SzekelyRizzoBakirov2007, SzekelyRizzo2009}.

\begin{lem}
\label{lem1}
Denote $tr(\cdot)$ as the matrix trace, $\lambda_{i} [\cdot]$ as the $i$th eigenvalue of a matrix, and $J$ as the matrix of ones of size $n$. Then the sample covariance equals
\begin{align*}
dCov^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n}) &= tr (A^{k}B^{l})- tr (A^{k}J)tr(B^{l}J)\\
&=tr [ (A^{k}- tr (A^{k}J)J) (B^{l}-tr(B^{l}J)J)] \\
& = \sum_{i=1}^{n} \lambda_{i} [ (A^{k}- tr (A^{k}J)J) (B^{l}-tr(B^{l}J)J)].
\end{align*}
\end{lem}

\begin{thm}[Local Variances]
\label{thm:dvar}
For any random variable $\ensuremath{X} \sim F_{\ensuremath{X}} \in \mathbb{R}^{p}$, and any $\mathcal{X}_{n} \in \mathbb{R}^{p \times n}$ with each column \emph{iid} as $F_{\ensuremath{X}}$,
\begin{description}
\item[(a)] Population and sample local variances are always non-negative, i.e.,
\begin{align*}
dVar^{\rho_{k}}(\ensuremath{X}) \geq 0\\
dVar^{k}(\mathcal{X}_{n}) \geq 0
\end{align*}
at any $\rho_{k} \in [0,1]$ and  any $k \in [n]$.
\item[(b)] $dVar^{\rho_{k}}(\ensuremath{X}) =0$ if and only if either $\rho_k =0$ or $F_{\ensuremath{X}}$ is a degenerate distribution;

$dVar^{k}(\mathcal{X}_{n}) =0$ if and only if either $k=1$ or $F_{\ensuremath{X}}$ is a degenerate distribution.

\item[(c)] For two constants $v \in \mathbb{R}^{p},u \in \mathbb{R}$, and an orthonormal matrix $Q \in \mathbb{R}^{p \times p}$,
\begin{align*}
dVar^{\rho_{k}}(v+uQ\ensuremath{X}) &=u^2 \cdot dVar^{\rho_{k}}(\ensuremath{X})\\
dVar^{k}(v^{T} J +u\mathcal{X}_{n} Q) &=u^2 \cdot dVar^{k}(\mathcal{X}_{n} ).
\end{align*}
\end{description}
\end{thm}

Therefore, the local variances end up having properties similar to the distance variance in \cite{SzekelyRizzoBakirov2007}, except the distance variance definition there takes a square root.

\begin{thm}[Local Correlations and \sct{MGC}]
\label{thm:dcor}
For any pair of random variable $(\ensuremath{X},\ensuremath{Y}) \sim F_{\ensuremath{X}\ensuremath{Y}} \in \mathbb{R}^{p} \times \mathbb{R}^{q}$, and any $(\mathcal{X}_{n},\mathcal{Y}_{n}) \in \mathbb{R}^{p \times n} \times \mathbb{R}^{q \times n}$ with each column \emph{iid} as $F_{\ensuremath{X}\ensuremath{Y}}$,
\begin{description}
\item [(a)] Symmetric and Boundedness:
\begin{align*}
dCorr^{\rho_{k},\rho_{l}}(\ensuremath{X},\ensuremath{Y})=dCorr^{\rho_{l},\rho_{k}}(\ensuremath{Y},\ensuremath{X}) \in [-1,1]\\
dCorr^{k,l}(\mathcal{X}_{n},\mathcal{Y}_{n})=dCorr^{l,k}(\mathcal{Y}_{n},\mathcal{X}_{n}) \in [-1,1]
\end{align*}
at any $(\rho_k,\rho_l) \in (0,1]^2$ and any $(k,l) \in [2,\ldots,n]^2$.
\item [(b)] Assume $F_\ensuremath{X}$ is non-degenerate. Then at any $\rho_{k} > 0$, $dCorr^{\rho_{k},\rho_k}(\ensuremath{X},\ensuremath{Y})=1$ if and only if $(\ensuremath{X}, u \ensuremath{Y})$ are dependent via an isometry for some non-zero constant $u \in \mathbb{R}$.

Assume $F_\ensuremath{X}$ is non-degenerate. Then at any $k > 1$, $dCorr^{k,k}(\mathcal{X}_{n},\mathcal{Y}_{n})=1$ if and only if $(\ensuremath{X}, u \ensuremath{Y})$ are dependent via an isometry for some non-zero constant $u \in \mathbb{R}$.

\item [(c)] Both population and Sample \sct{MGC}~are symmetric and bounded:
\begin{align*}
c^{*}(\ensuremath{X},\ensuremath{Y})=c^{*}(\ensuremath{Y},\ensuremath{X}) \in [-1,1] \\
c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})=c^{*}(\mathcal{Y}_{n},\mathcal{X}_{n}) \in [-1,1].
\end{align*}
\item [(d)] Assume $F_\ensuremath{X}$ is non-degenerate. Then $c^{*}(\ensuremath{X},\ensuremath{Y})=1$ if and only if $(\ensuremath{X}, u \ensuremath{Y})$ are dependent via an isometry for some non-zero constant $u \in \mathbb{R}$.

Assume $F_\ensuremath{X}$ is non-degenerate. Then $c^{*}(\mathcal{X}_{n},\mathcal{Y}_{n})=1$ if and only if $(\ensuremath{X}, u \ensuremath{Y})$ are dependent via an isometry for some non-zero constant $u \in \mathbb{R}$.
\end{description}
\end{thm}

The proof of Theorem~\ref{thm:dcor}(b)(d) also shows that the local correlations and \sct{MGC}~cannot be $-1$.

\section{Experiments}
\label{sec:exp}
In the experiments, we compare Sample \sct{MGC}~with \sct{Dcorr}, \sct{Pearson}, \sct{Mantel}, \sct{Hsic}, \sct{Hhg}, and \sct{Copula}~test on $20$ different simulation settings based on a combination of simulations used in previous works \cite{SzekelyRizzoBakirov2007, SimonTibshirani2012, GorfineHellerHeller2012}. Among the $20$ settings, the first $5$ are monotonic relationships (and several of them are linear or nearly so), the last simulation is an independent relationship, and the remaining settings consist of common non-monotonic and strongly nonlinear relationships. The exact distributions are shown in Appendix.

\subsection*{The Sample Statistics}

Figure~\ref{f:dependencies} shows the sample statistics of \sct{MGC}, \sct{Dcorr}, and \sct{Pearson}~for each of the $20$ simulations in a univariate setting. For each simulation, we generate sample data $(\mathcal{X}_{n},\mathcal{Y}_{n})$ at $p=q=1$ and $n=100$ without any noise, then compute the sample statistics. From type $1-5$, the test statistics for both \sct{MGC}~and \sct{Dcorr}~are remarkably greater than $0$ and almost identical to each other. For the nonlinear relationships  (type $6-19$), \sct{MGC}~benefits from searching locally and achieves a larger test statistic  than \sct{Dcorr}'s, which can be very small in these nonlinear relationships. For type $20$, the test statistics for both \sct{MGC}~and \sct{Dcorr}~are almost $0$ as expected. On the other hand, \sct{Pearson}'s test statistic is large whenever there exists certain linear association, and almost $0$ otherwise. The comparison of sample statistics indicate that \sct{Dcorr}~may have inferior finite-sample testing power in nonlinear relationships, but a strong dependency signal is actually hidden in a local structure that \sct{MGC}~may recover.

\begin{figure}
\includegraphics[width=1.0\textwidth,trim={0cm 0cm 0cm 0cm},clip]{FigSimVisual2.pdf}
\caption{For each panel, a pair of dependent $(\mathcal{X}_{n}, \mathcal{Y}_{n})$ at $n=100$ and $p=q=1$ is generated and visualized; the accompanying color bar compares \sct{MGC}~(green), \sct{Dcorr}~(gray), and \sct{Pearson}~in the absolute value (black), all of which lie in the range of $[0,1]$ with $0$ indicating no relationship.  \sct{MGC}~yields a non-zero sample correlation for each dependency, while being almost $0$ under independence. In comparison, the distance correlation can be close to $0$ for common nonlinear dependencies, while the Pearson's correlation only measures linear association and cannot capture nonlinear dependencies. The Sample \sct{MGC}~statistic is shown above each panel.}
\label{f:dependencies}
\end{figure}

\subsection*{Finite-Sample Testing Power}
Figure~\ref{f:noise}  shows the finite-sample testing power of \sct{MGC}, \sct{Dcorr}, and \sct{Pearson}~for a linear  and a quadratic relationship at $n=20$ and $p=q=1$ with white noise (controlled by a constant). The testing power of \sct{MGC}~is estimated as follows: we first generate dependent sample data $(\mathcal{X}_{n},\mathcal{Y}_{n})$ for $r=10,000$ replicates, compute Sample \sct{MGC}~for each replicate to estimate the alternative distribution of \sct{MGC}. Then we generate independent sample data $(\mathcal{X}_{n},\mathcal{Y}_{n})$ using the same marginal distributions for $r=10,000$ replicates, compute Sample \sct{MGC}~to estimate the null distribution, and estimate the testing power at type $1$ error level $\alpha=0.05$. The testing power of \sct{Dcorr}~is estimated in the same manner, while the testing power of \sct{Pearson}~is directly computed via the t-test. \sct{MGC}~has the best power in the quadratic relationship, while being almost identical to \sct{Dcorr}~and \sct{Pearson}~in the linear relationship.

\begin{figure}[!ht]
\centering
\includegraphics[width=0.48\textwidth,trim={1.5cm 0 0cm 0cm},clip]{FigNoiseT1}
\includegraphics[width=0.48\textwidth,trim={1.5cm 0 0cm 0cm},clip]{FigNoiseT6}
  \caption{Comparing the power of \sct{MGC}, \sct{Dcorr}, and \sct{Pearson}~in noisy linear relationship (left), and noisy quadratic relationship (right). For the linear relationship at $n=20$ and $p=q=1$, all three methods are almost the same with \sct{Pearson}~being slightly higher power; for the quadratic relationship, \sct{MGC}~has a much higher power than \sct{Dcorr}~and \sct{Pearson}. The phenomenon is consistent throughout the remaining dependent simulations: for testing in monotonic relationships, \sct{Pearson}, \sct{Dcorr}, and \sct{MGC}~almost coincide with each other; for strongly nonlinear relationships, \sct{MGC}~almost always supersedes \sct{Dcorr}, and \sct{Dcorr}~is better than \sct{Pearson}.
}
\label{f:noise}
\end{figure}

The same phenomenon holds throughout all the simulations we considered, i.e., \sct{MGC}~achieves  almost the same power as \sct{Dcorr}~in monotonic relationships, while being able to improve the power in monotonic and strongly nonlinear relationships. The testing power of \sct{MGC}~versus all other methods are shown in Figure~\ref{f:1DAll} for the univariate settings, and we plot the power versus the sample size from $5$ to $100$ for each simulation.
Note that the noise level is tuned for each dependency for illustration purposes.

\begin{figure}[htbp]
\includegraphics[width=1.0\textwidth,trim={0 0.5cm 3.2cm 0},clip]{Fig1DPowerAll}
\caption{
Comparing the testing power of \sct{MGC}, \sct{Dcorr}, \sct{Mantel}, \sct{Hsic}, \sct{Hhg}, and \sct{Copula}.
for $20$ different univariate simulations.
Estimated via $10,000$ replicates of repeatedly generated dependent and independent sample data, each panel shows the estimated testing power at the type $1$ error level $\alpha=0.05$ versus sample sizes ranging from $n=5$ to  $100$. Excluding the independent simulation (\#20) where all methods yield power $0.05$, \sct{MGC}~exhibits the highest or nearly highest power in most dependencies.
Note that we only show the ticks for the first panel, because they are the same for every panel, i.e., the x-axis always ranges from $5$ to $100$ while the y-axis always ranges from $0$ to $1$.}
\label{f:1DAll}
\end{figure}

Figure~\ref{f:nDAll} compares the testing performance for the same $20$ simulations
with a fixed sample size $n=100$
 and increasing dimensionality. The relative powers in the univariate and multivariate settings are then summarized in Figure~\ref{f:Summary2}. \sct{MGC}~is overall the most powerful method, followed by \sct{Hhg}~and \sct{Hsic}. Since non-monotone relationships are prevalent among the $20$ settings, it is not a surprise that \sct{Dcorr}~is overall worse than \sct{Hhg}~and \sct{Hsic}, both of which also excel at nonlinear relationships.

\begin{figure}[htbp]
\includegraphics[width=1.0\textwidth,trim={0 0.5cm 3.2cm 0},clip]{FigHDPowerAll}
\caption{The testing power computed in the same procedure as in Figure~\ref{f:1DAll}, except the $20$ simulations are now run at fixed sample size $n=100$ and increasing dimensionality $p$. Again, \sct{MGC}~empirically achieves similar or higher power than the previous popular approaches for all dimensions on most settings.
The ticks for y axis is only shown in the first panel, as the power has the same range in $[0,1]$ for every panel.}
\label{f:nDAll}
\end{figure}

\begin{figure}[!ht]
\centering
\includegraphics[width=0.48\textwidth,trim={1.5cm 0 0cm 0cm},clip]{Fig1DPowerSummary}
\includegraphics[width=0.48\textwidth,trim={1.5cm 0 0cm 0cm},clip]{FigHDPowerSummary}
  \caption{The relative Power of \sct{MGC}~to other methods for testing  the $20$ simulations under one-dimensional and high-dimensional scenarios. (Left) For each simulation type, we average the testing power of each method in Figure~\ref{f:1DAll} over the sample size, then divide each average power by the average power of \sct{MGC}. The last column (which also serves as the legend) shows the median power among all relative powers of type $1-19$. The same for the right panel, except it averages over the dimensionality in Figure~\ref{f:nDAll}. The relative power percentage indicates that \sct{MGC}~is a very powerful method for finite-sample testing.
}
\label{f:Summary2}
\end{figure}

Note that the same $20$ simulations were also used in \cite{ShenEtAl2016} for evaluation purposes. The main difference is that the Sample \sct{MGC}~algorithm is now based on the improved threshold with theoretical guarantee. Comparing to the previous algorithm, the new threshold slightly improves the testing power in monotonic relationships (the first $5$ simulations).

\subsection*{Running Time}
\label{sec:time}

Sample \sct{MGC}~can be computed and tested in the same running time complexity as distance correlation: Assume $p$ is the maximum feature dimension of the two datasets, distance computation and centering takes $\mc{O}(n^2 p)$, the ranking process takes $\mc{O}(n^2 \log n)$, all local covariances and correlations can be incrementally computed in $O(n^2)$ (the pseudo-code is shown in \cite{ShenEtAl2016}), the thresholding step of Sample \sct{MGC}~takes $O(n^2)$ as well. Overall, Sample \sct{MGC}~can be computed in $\mc{O}(n^2 \max\{\log n,p\})$. In comparison, the \sct{Hhg}~statistic requires the same complexity as \sct{MGC}, while distance correlation saves on the $\log n$ term.

As the only part of \sct{MGC}~that has the additional $\log n$ term is the column-wise ranking process, a multi-core architecture can reduce the running time to $\mc{O}(n^2 \max\{\log n,p\}/T)$. By making $T=\log(n)$ ($T$ is no more than $30$ at $1$ billion samples), \sct{MGC}~effectively runs in $\mc{O}(n^2 p)$ and is of the same complexity as \sct{Dcorr}. The permutation test multiplies another $r$ to all terms except the distance computation, so overall the \sct{MGC}~testing procedure requires $\mc{O}(n^2 \max\{r,p\})$, which is the same as \sct{Dcorr}, \sct{Hhg}, and \sct{Hsic}. Figure~\ref{f:time} shows that \sct{MGC}~has approximately the same complexity as \sct{Dcorr}, and is slower by a constant in the actual running time.

\begin{figure}
\centering
\includegraphics[width=0.48\textwidth,trim={0cm 0 0cm 0cm},clip]{FigTime1D}
\includegraphics[width=0.48\textwidth,trim={0cm 0 0cm 0cm},clip]{FigTimeHD}
\caption{Compute the test statistics of \sct{MGC}, \sct{Dcorr}, and \sct{Hsic}~for $100$ replicates, then plot the average running time in log scale (clocked using Matlab 2017a on a Windows 10 machine with I7 six-core CPU). The sample data is repeatedly generated using the quadratic relationship in Appendix~\ref{appen:function}, the sample size increases from $25$ to $500$, and the dimensionality is fixed at $p=1$ on the left and $p=1000$ on the right. In either panel, the three lines differ by some constants in the log scale, suggesting the same running time complexity but different constants. \sct{MGC}~has a higher intercept than the other two, which translates to about a constant of $6$ times of \sct{Dcorr}~and $3$ times of \sct{Hsic}~at $n=500$ and $p=1$, and about $3$ at $p=1000$. Note that the increase in $p$ has a relatively small effect in the running time, because the dimensionality $p$ takes part only in the distance matrix computation and is thus relatively cheap.}
\label{f:time}
\end{figure}

\section{Conclusion}
\label{sec:dis}
In this paper, we formalize the population version of local correlation and \sct{MGC}, connect them to the sample counterparts, prove the convergence and almost unbiasedness from the sample version to the population version, as well as a number of desirable properties for a well-defined correlation measure. In particular, population \sct{MGC}~equals $0$ and the sample version converges to $0$ if and only if independence, making Sample \sct{MGC}~valid and consistent under the permutation test. Moreover, Sample \sct{MGC}~is designed in a computationally efficient manner, and the new threshold choice achieves both theoretical and empirical improvements. The numerical experiments confirm the empirical advantages of \sct{MGC}~in a wide range of linear, nonlinear, high-dimensional dependencies.

There are many potential future avenues to pursue. Theoretically, proving when and how one method dominates another in testing power is highly desirable. As the methods in comparison have distinct formulations and different properties, it is often difficult to compare them directly. However, a relative efficiency analysis may be viable when limited to methods of similar properties, such as \sct{Dcorr}~and \sct{Hsic}, or local statistic and global statistic. In terms of the locality principle, the geometric meaning of the local scale in \sct{MGC}~is intriguing --- for example, does the family of local correlations fully characterize the joint distribution, and what is the relationship between the optimal local scale and the dependency geometry --- answering these questions may lead to further improvement of \sct{MGC}, and potentially make the family of local correlations a valuable tool beyond testing.

Method-wise, there are a number of alternative implementations that may be pursued. For example, the sample local correlations can be defined via $\epsilon$ ball instead of nearest neighbor graphs, i.e., truncate large distances based on absolute magnitude instead of the nearest neighbor graph. The maximization and thresholding mechanism may be further improved, e.g., thresholding based on the covariance instead of correlation, or design a better regularization scheme. There are many alternative approaches that can maintain consistency in this framework, and it will be interesting to investigate a better algorithm. In particular, we name our method as ``multiscale graph correlation" because the local correlations are computed via the k-nearest neighbor graphs, which is one way to generalize the distance correlation.

Application-wise, the \sct{MGC}~method can directly facilitate new discoveries in many kinds of scientific fields, especially data of limited sample size and high-dimensionality such as in neuroscience and omics \cite{ShenEtAl2016}. Within the domain of statistics and machine learning, \sct{MGC}~can be a very competitive candidate in any methodology that requires a well-defined dependency measure, e.g., variable selection \cite{LiZhongZhu2012}, time series \cite{Zhou2012}, etc. Moreover, the very idea of locality may improve other types of distance-based tests, such as the energy distance for K-sample testing \cite{SzekelyRizzo2013b}.

\clearpage
\bibliographystyle{ieeetr}