\section{Introduction}
\label{sec1}

Modern scientific research is characterized by a collection of massive and diverse data sets.  One of the most important goals is to integrate these different data sets for making better predictions and statistical inferences. Given a target problem to solve, transfer learning \citep{Torrey10} aims at transferring the knowledge from different but related samples to improve the learning performance of the target problem.  A typical example  of transfer learning is that one can improve the accuracy of recognizing cars by using not only the labeled data for cars  but some labeled data for trucks \citep{weiss2016survey}.
Besides classification, another relevant class of transfer learning problems is linear regression using auxiliary samples. In health-related studies, some biological or clinical  outcomes are hard to obtain due to ethical or cost issues, in which case transfer learning  can be leveraged to boost the prediction and estimation performance of these outcomes by gathering information from different but related biological outcomes.

Transfer learning has been applied to problems in medical and biological applications, including   predictions of protein localization \citep{Mei11}, biological imaging diagnosis \citep{Shin16}, drug sensitivity prediction \citep{Turki17} and integrative analysis of``multi-omics'' data, see, for instance, \citet{Sun16}, \citet{Hu19}, and \citet{Wang19}.  It has also been applied to  natural language processing \citep{Daume07} and recommendation systems \citep{Pan13} in machine learning literature.
The application that motivated the research in this paper is to integrate gene expression data sets measured in different issues to understand the gene regulations using the Genotype-Tissue Expression (GTEx) data (\url{https://gtexportal.org/}).
 These datasets are always high-dimensional with relatively small sample sizes. When studying the gene regulation relationships  of a specific tissue or cell-type, it is possible to borrow information from other issues  in order to enhance the learning accuracy.  This motivates us to consider transfer learning in high-dimensional linear regression.

\subsection{Transfer Learning in High-dimensional Linear Regression}
\label{sec1-1}
Regression analysis is one of  the most widely used statistical methods to understand the association of an outcome with a set of covariates. In many modern applications, the dimension of the covariates is usually very high as compared to the sample size. Typical examples include the genome-wide association and gene expression studies.
In this paper, we consider transfer learning in high-dimensional linear regression models. Formally, our target model can be written as
\begin{equation}
\label{m0}
    y^{(0)}_i=(x_i^{(0)})^{\intercal}\beta+\eps^{(0)}_i,~i=1,\dots,n_0,
\end{equation}
where $((x_i^{(0)})^{\intercal}, y_i^{(0)}), ~i=1,\dots, n_0$, are independent samples, $\beta\in{\real}^p$ is the regression coefficient of interest, and $\eps_i^{(0)}$ are independently distributed random noises such that ${\mathbb{E}}[\eps_i^{(0)}|x_i^{(0)}]=0$. In the high-dimensional regime, where $p$ can be larger and much larger than $n_0$, $\beta$ is often assumed to be sparse such that the number of nonzero elements of $\beta$, denoted by $s$, is much smaller than $p$.

In the context of transfer learning, we observe  additional samples from $K$ auxiliary studies,  That is, we observe $((x_i^{(k)})^{\intercal}, y_i^{(k)})$ generated from the auxiliary model
\begin{align}
y_i^{(k)}=(x_i^{(k)})^{\intercal}w^{(k)}+\eps^{(k)}_i,~i=1,\dots,n_k,~k=1,\dots, K,
\end{align}
where $w^{(k)}\in{\real}^p$ is the true coefficient vector for the $k$-th study, and $\eps^{(k)}_i$ are the random noises such that ${\mathbb{E}}[\eps^{(k)}_i|x_i^{(k)}]=0$.
 The regression coefficients $w^{(k)}$ are unknown and different from our target $\beta$ in general. The number of auxiliary studies, $K$, is allowed to grow but practically $K$ may not be too large. We will study the estimation and prediction of target model (\ref{m0}) utilizing the primary data $((x_i^{(0)})^{\intercal}, y_i^{(0)}), ~i=1,\dots, n_0,$ as well as the data from $K$ auxiliary studies $((x_i^{(k)})^{\intercal}, y_i^{(k)}),~ i=1,\dots, n_k,~k=1,\dots, K$.

If useful information can be borrowed from the auxiliary samples, the target model and some of the auxiliary models need to possess a certain level of similarity. If an auxiliary model is ``similar'' to the target model, we say that this auxiliary sample/study is informative. In this work, we characterize the informative level of the $k$-th auxiliary study  using the sparsity of the difference between $w^{(k)}$ and $\beta$. Let $\delta^{(k)}=\beta-w^{(k)}$ denote the contrast between $w^{(k)}$ and $\beta$. The set of informative auxiliary samples are those whose contrasts are sufficiently sparse:
\begin{equation}
\label{def-A}
\mathcal{A}_q=\{1\leq k\leq K: \|\delta^{(k)}\|_q\leq h\},
\end{equation}
for some $q\in[0,1]$. That is, the set $\mathcal{A}_q$, which contains the auxiliary studies  whose contrast vectors have $\ell_q$-sparsity at most $h$, is called the {\it informative set}. It will be seen later that as long as $h$ is relatively small to the sparsity of $\beta$, the studies in $\mathcal{A}_q$ can be useful in improving the prediction and estimation of $\beta$. In the case of $q=0$, the set $\mathcal{A}_q$ corresponds to the auxiliary samples whose contrast vectors have at most $h$ nonzero elements. We also consider approximate sparsity constraints $(q\in (0,1])$, which allows all of the coefficients to be nonzero but their absolute magnitude decays at a relatively rapid rate.
For any $q\in[0,1]$, smaller $h$ implies that the auxiliary samples in $\mathcal{A}_q$ are more informative;  larger cardinality of $\mathcal{A}_q$ ($|\mathcal{A}_q|$) implies that a larger number of informative auxiliary samples. Therefore, smaller $h$ and larger $|\mathcal{A}_q|$ should be favorable. We allow $\mathcal{A}_q$ to be empty in which case none of the auxiliary samples are informative. For the auxiliary samples outside of $\mathcal{A}_q$, we do not assume sparse $\delta^{(k)}$ and hence $w^{(k)}$ can be very different from $\beta$ for $k\notin \mathcal{A}_q$.

There is a paucity of methods and fundamental theoretical results for high-dimensional linear regression in the transfer learning setting. In the case where the set of informative auxiliary samples $\mathcal{A}_q$ is known, there is a lack of rate optimal estimation and prediction methods. A closely related topic is multi-task learning \citep{Ando05, LMT09}, where the goal is to simultaneously estimate multiple models using multiple response data. The multi-task learning considered in \citet{LMT09} estimates multiple high-dimensional sparse linear models under the assumption that the support of all the regression coefficients are the same. The goal of transfer learning is however different, as one is only interested in estimating the target model and this remains to be a largely unsolved problem.
 \citet{CW19} studied the  minimax and adaptive methods for nonparametric classification in the transfer learning setting under similarity assumptions on all the auxiliary samples to the target distribution \citep[Definition 5]{CW19}.
In the more challenging setting where the  set $\mathcal{A}_q$ is unknown as is typical in real applications, it is unclear how to avoid the effects of adversarial auxiliary samples.  Additional challenges include the heterogeneity among the design matrices, which does not arise in the conventional high-dimensional regression problems and hence requires novel proposals.

\subsection{Our Contributions}

In the setting where the informative set $\mathcal{A}_q$ is known, we propose a transfer learning algorithm, called Oracle Trans-Lasso, for estimation and prediction of the target regression vector and prove its minimax optimality under mild conditions. The result demonstrates a faster rate of convergence when $\mathcal{A}_q$ is non-empty and $h$ is sufficiently smaller than $s$, in which case the knowledge from the informative auxiliary samples can be optimally transferred to substantially help solve the regression problem under the target model.

In the more challenging setting where $\mathcal{A}_q$ is unknown a priori, we introduce a data-driven algorithm, called Trans-Lasso, to adapt to the unknown $\mathcal{A}_q$. The adaption is achieved by aggregating a number of candidate estimators. The desirable properties of the aggregation method guarantee that the Trans-Lasso is not much worse than the best one among the candidate estimators. We carefully construct the candidate estimators and, leveraging the properties of aggregation, demonstrate the robustness and the efficiency of Trans-Lasso under mild conditions. In terms of robustness, the Trans-Lasso is guaranteed to be not much worse than the Lasso estimator using only the primary samples no matter how adversarial the auxiliary samples are. In terms of efficiency, the knowledge from a subset of the informative auxiliary samples can be transferred to the target problem under proper conditions.  Furthermore, If  the contrast vectors in the informative samples are sufficiently sparse,  the Trans-Lasso estimator performs as if the informative set $\mathcal{A}_q$ is known.

When the distributions of the designs are distinct in different samples, the effects of heterogeneous designs are studied. The performance of the proposed algorithms is justified theoretically and numerically in various settings.

\subsection{Related Literature}
Methods for incorporating auxiliary information into statistical inference have received much recent interest. In this context, \cite{Tony19} and \cite{Xia2020GAP} studied  the two-sample larges-scale multiple testing problems. \cite{Ban19} considered the high-dimensional sparse estimation and \cite{Mao19} focused on matrix completion.
The auxiliary information in the aforementioned papers is given as some extra covariates while we have some additional raw data, which are high-dimensional, and it is not trivial to find the best way to summarize the information. \citet{Bastani18} studied estimation and prediction in high-dimensional linear models with one informative auxiliary study, where the sample size of the auxiliary study is larger than the number of covariates. This work considers more general scenarios under weaker assumptions. Specifically, the sample size of auxiliary samples can be smaller than the number of covariates and some auxiliary studies can be non-informative, which is more practical in applications.

The problem we study here is certainly related to the high-dimensional prediction and estimation in the conventional settings where only samples from the target model are available.
Several $\ell_1$ penalized or constrained minimization methods have been proposed for prediction and estimation for high-dimensional linear regression; see, for example, \citet{Lasso, FL01, Zou06, CT07, Zhang10}. The minimax optimal rates for estimation and prediction are studied in \citet{Raskutti11} and \citet{Ver12}.

\subsection{Organization and Notation}
The rest of this paper is organized as follows. Section \ref{sec2} focuses on the setting where the informative set $\mathcal{A}_q$ is known and with the sparsity in \eqref{def-A} measured in $\ell_1$-norm.  A transfer learning algorithm is proposed for estimation and prediction of the target regression vector and its minimax optimality is established. In Section \ref{sec3}, we study the estimation and prediction of the target model when $\mathcal{A}_q$ is unknown for $q=1$. In Section \ref{sec4}, we justify the theoretical performance of our proposals under heterogeneous designs and extend our main algorithms to deal with $\ell_q$-sparse contrasts for $q\in[0,1)$. In Section \ref{sec-simu}, the numerical performance of the proposed methods is studied in various settings. In Section \ref{sec-data}, the proposed algorithms are applied to an analysis of a Genotype-Tissue Expression (GTEx) dataset to investigate  the association of gene expression of one gene with other genes in a target tissue by leveraging data measured on other related tissues or cell types.

We finish this section with notation. Let $X^{(0)}\in{\real}^{n_0\times p}$ and $y^{(0)}\in{\real}^{n_0}$ denote the design matrix and the response vector for the primary data, respectively. Let $X^{(k)}\in{\real}^{n_k\times p}$ and $y^{(k)}\in{\real}^{n_k}$ denote the design matrix and the response vector for the $k$-th sample, respectively. For a class of matrices $R_l\in{\real}^{n_l\times p_0}$, $l\in \mathcal{L}$, we use $\{R_l\}_{l\in \mathcal{L}}$ to denote $R_l$, $l\in \mathcal{L}$. Let $n_{\mathcal{A}_q}=\sum_{k\in \mathcal{A}_q}n_k$.
For a generic semi-positive definite matrix $\Sigma\in{\real}^{m\times m}$, let $\Lambda_{\max}(\Sigma)$ and $\Lambda_{\min}(\Sigma)$ denote the largest and smallest eigenvalues of $\Sigma$, respectively. Let $\textrm{Tr}(\Sigma)$ denote the trace of $\Sigma$. Let $e_j$ be such that its $j$-th element is 1 and all other elements are zero. Let $a\vee b$ denote $\max\{a,b\}$ and $a\wedge b$ denote $\min\{a,b\}$. We use $c,c_0,c_1,\dots$ to denote generic constants which can be different in different statements. Let $a_n=O(b_n)$ and $a_n\lesssim b_n$ denote $|a_n/b_n|\leq c<\infty$ for some constant $c$ when $n$ is large enough. Let $a_n\asymp b_n$ denote $a_n/b_n\rightarrow c$ for some positive constant $c$ as $n\rightarrow \infty$. Let $a_n=O_P(b_n)$ and $a_n\lesssim_{{\mathbb{P}}} b_n$ denote ${\mathbb{P}}(|a_n/b_n|\leq c)\rightarrow 1$ for some constant $c<\infty$. Let $a_n=o_P(b_n)$ denote ${\mathbb{P}}(|a_n/b_n|>c)\rightarrow 0$ for any constant $c>0$.
\section{Estimation with Known Informative Auxiliary Samples}
\label{sec2}

In this section, we consider transfer learning for high-dimensional linear regression when the  informative set $\mathcal{A}_q$ is known. We focus on the $\ell_1$-sparse characterization of the contrast vectors and leave the $\ell_q$-sparsity, $q\in[0,1)$, to Section \ref{sec4}. The notation $\mathcal{A}_1$ will be abbreviated as $\mathcal{A}$ in the sequel without special emphasis.

\subsection{Oracle Trans-Lasso Algorithm}
\label{sec2-1}
We propose a transfer learning algorithm, called {\it Oracle Trans-Lasso},  for estimation and prediction when $\mathcal{A}$ is known.  As an overview, we first compute an initial estimator using the primary sample and all the informative auxiliary samples.
However, its probabilistic limit is biased from $\beta$ as $w^{(k)}\neq \beta$ in general. We then correct its bias using the primary data in the second step.  Algorithm \ref{algo1} formally presents our proposed Oracle Trans-Lasso algorithm.

\vspace{0.1in}\begin{algorithm}
 \SetKwInOut{Input}{Input}
    \SetKwInOut{Output}{Output}
\SetAlgoLined
 \Input{Primary data $(X^{(0)},y^{(0)})$ and informative auxiliary samples $\{X^{(k)},y^{(k)}\}_{k\in\mathcal{A}}$}
 \Output{$\hat{\beta}$}

\underline{Step 1}. Compute
\begin{align}
\label{eq-hbeta-init}
\hat{w}^{\mathcal{A}}&=\mathop{\rm arg\, min}_{w\in{\real}^p} \Big\{\frac{1}{2(n_{\mathcal{A}}+n_0)}\sum_{k \in \mathcal{A}\cup\{0\}}\|y^{(k)}-X^{(k)}w\|_2^2+\lambda_w\|w\|_1\Big\}\;
\end{align}
for $\lambda_w=c_1\sqrt{\log p/(n_0+n_{\mathcal{A}})}$ with some constant $c_1$.

\underline{Step 2}. Let
\begin{equation}
\label{eq-hbeta}
\hat{\beta}=\hat{w}^{\mathcal{A}}+\hat{\delta}^{\mathcal{A}},
\end{equation}
 where
\begin{equation}
\label{eq-hgam}
  \hat{\delta}^{\mathcal{A}}=\mathop{\rm arg\, min}_{\delta\in{\real}^p} \left\{\frac{1}{2n_0}\|y^{(0)}-X^{(0)}(\hat{w}^{\mathcal{A}}+\delta)\|_2^2+\lambda_{\delta}\|\delta\|_1\right\}
\end{equation}
for $\lambda_{\delta}=c_2\sqrt{\log p/n_0}$ with some constant $c_2$.
 \caption{\textbf{Oracle Trans-Lasso algorithm}} \label{algo1}
\end{algorithm}

In Step 1, $\hat{w}^{\mathcal{A}}$ is realized based on the Lasso \citep{Lasso} using the primary sample and all the informative auxiliary samples. Its probabilistic limit is $w^{\mathcal{A}}$, which can be defined via the following moment condition
\[
   {\mathbb{E}}\left[\sum_{k \in \mathcal{A}\cup\{0\}} (X^{(k)})^{\intercal}(y^{(k)}-X^{(k)}w^{\mathcal{A}})\right]=0.
\]
Denoting ${\mathbb{E}}[x_i^{(k)}(x_i^{(k)})^{\intercal}]=\Sigma^{(k)}$, $w^{\mathcal{A}}$ has the following explicit form:
\begin{align}
\label{eq-wA}
w^{\mathcal{A}}=\beta+\delta^{\mathcal{A}}
\end{align}
for $\delta^{\mathcal{A}}=\sum_{k \in \mathcal{A}}\alpha_k\delta^{(k)}$ and $\alpha_k=n_k/(n_{\mathcal{A}}+n_0)$, if $\Sigma^{(k)}=\Sigma^{(0)}$ for all $k\in \mathcal{A}$.
That is, the probabilistic limit of $\hat{w}^{\mathcal{A}}$, $w^{\mathcal{A}}$, has bias $\delta^{\mathcal{A}}$, which is a weighted average of $\delta^{(k)}$. Step 1 is related to the approach for high-dimensional misspecified models \citep{Bu15} and moment estimators. The estimator $\hat{w}^{\mathcal{A}}$ converges relatively fast as the sample size used in Step 1 is relatively large. Step 2 corrects the bias, $\delta^{\mathcal{A}}$, using the primary samples. In fact, $\delta^{\mathcal{A}}$ is a sparse high-dimensional vector whose $\ell_1$-norm is no larger than $h$.
Hence, the error of step 2 is under control for a relatively small $h$.
The choice of the tuning parameters $\lambda_w$ and $\lambda_{\delta}$ will be further specified in Theorem \ref{thm0-l1}.

\subsection{Theoretical Properties of Oracle Trans-Lasso}
Formally, the parameter space we consider can be written as
\begin{equation}
\label{eq-Thetaq}
  \Theta_{q}(s,h)=\left\{(\beta,\delta^{(1)},\dots,\delta^{(K)} ): \|\beta\|_0\leq s, ~\max_{k \in \mathcal{A}_q} \|\delta^{(k)}\|_q\leq h\right\}
\end{equation}
for $\mathcal{A}_q\subseteq \{1,\dots, K\}$ and $q\in[0,1]$.
We study the rate of convergence for the Oracle Trans-Lasso algorithm under the following two conditions.
\begin{condition}
\label{cond1}{\rm
For each $k\in \mathcal{A}\cup\{0\}$, each row of $X^{(k)}$ is \textit{i.i.d.} Gaussian distributed with mean zero and covariance matrix $\Sigma$. The smallest and largest eigenvalues of $\Sigma$ are bounded away from zero and infinity, respectively.
}\end{condition}

\begin{condition}
\label{cond2}{\rm
For each $k\in\mathcal{A}\cup\{0\}$, the random noises $\eps^{(k)}_i$ are \textit{i.i.d.} sub-Gaussian distributed mean zero and variance $\sigma^2_k$.   For some constant $C_0$, it holds that $\max_{\mathcal{A}\cup\{0\} }{\mathbb{E}}[\exp\{t\eps^{(k)}_i\}]\leq \exp\{t^2C_0\}$ for all $t\in {\real}$ and $\max_{0\leq k\leq K} {\mathbb{E}}[(y_i^{(k)})^2]$ is bounded away from infinity.
}\end{condition}

Condition \ref{cond1} assumes random designs with Gaussian distribution. The Gaussian assumption provides convenience for bounding the restricted eigenvalues of sample Gram matrices. Moreover, the designs are identically distributed for $k\in \mathcal{A}\cup\{0\}$. This assumption is for simplifying some technical conditions and will be relaxed in Section \ref{sec4}. Without loss of generality, we also assume the design matrices are normalized such that $\|X^{(k)}_{.,j}\|_2^2=n_k$ and $\Sigma_{j,j}=1$ for all $1\leq j\leq p$, $k\in \mathcal{A}\cup\{0\}$. Condition \ref{cond2} assumes sub-Gaussian random noises for primary and informative auxiliary samples and the second moment of the response vector is finite. Conditions \ref{cond1} and \ref{cond2} put no assumptions on the non-informative auxiliary samples as they are not used  in the Oracle Trans-Lasso algorithm.
In the next theorem, we prove the convergence rate of the Oracle Trans-Lasso.

\begin{theorem}[Convergence Rate of Oracle Trans-Lasso]
\label{thm0-l1}
\text{  }\text{ }
Assume that Condition \ref{cond1} and Condition \ref{cond2} hold true. We take
$\lambda_w=$\text{ } $\max_{k\in\mathcal{A}\cup\{0\}}c_1\sqrt{{\mathbb{E}}[(y^{(k)}_i)^2]\log p/(n_{\mathcal{A}}+n_0})$ and $\lambda_\delta=c_2\sqrt{\log p/n_0}$ for some sufficiently large constants $c_1$ and $c_2$ only depending on $C_0$.
If $s\log p/(n_{\mathcal{A}}+n_0)+h(\log p/n_0)^{1/2}=o((\log p/n_0)^{1/4})$, then it holds that
\begin{align}
&\sup_{\beta\in\Theta_1(s,h)}\frac{1}{n_0}\|X^{(0)}(\hat{\beta}-\beta)\|_2^2\vee\|\hat{\beta}-\beta\|_2^2\nonumber\\
&=O_P\left(\frac{s\log p}{n_{\mathcal{A}}+n_0}+\frac{s\log p}{n_0}\wedge h\sqrt{\frac{\log p}{n_0}}\wedge h^2\right) \label{l1-re1}.
\end{align}
\end{theorem}
Theorem \ref{thm0-l1} provides the convergence rate of $\hat{\beta}$ for any $\beta\in\Theta_1(s,h)$.
In the trivial case where $\mathcal{A}$ is empty, the right-hand side in (\ref{l1-re1}) is $O_P(s\log p/n_0)$, which is the convergence rate for the Lasso only using primary samples. When $\mathcal{A}$ is non-empty, the right-hand side of (\ref{l1-re1}) is sharper than $s\log p/n_0$ if  $h\sqrt{\log p/n_0}\ll s$ and $n_{\mathcal{A}}\gg n_0$. That is, if the informative auxiliary samples have contrast vectors sufficiently sparser than $\beta$ and the total sample size is significantly larger than the primary sample size, then the knowledge from the auxiliary samples can significantly improve the learning performance of the target model. In practice, even if $n_{\mathcal{A}}$ is comparable to $n_0$, the Oracle Trans-Lasso can still improve the empirical performance as shown by some numerical experiments provided in Section \ref{sec-simu}.

The sample size requirement in Theorem \ref{thm0-l1} guarantees the lower restricted eigenvalues of the sample Gram matrices in Step 1 and Step 2 are bounded away from zero with high probability. The proof of Theorem \ref{thm0-l1} involves an error analysis of $\hat{w}^{\mathcal{A}}$ and that of $\hat{\delta}^{\mathcal{A}}$. While $w^{\mathcal{A}}$ may be  neither $\ell_0$- nor $\ell_1$-sparse, it can be decomposed into an $\ell_0$-sparse component plus an $\ell_1$-sparse component as illustrated in (\ref{eq-wA}). Exploiting this sparse structure is a key step in proving Theorem \ref{thm0-l1}.
Regarding the choice of tuning parameters, $\lambda_w$ depends on the second moment of $y_i^{(k)}$, which can be consistently estimated by $\|y^{(k)}\|_2^2/n_k$. The other tuning parameter $\lambda_{\delta}$ depends on the noise levels, which can be estimated by the scaled Lasso \citep{scaled-lasso}.  In practice, cross validation can be performed for selecting tuning parameters.

 We now establish the minimax lower bound for estimating $\beta$ in the transfer learning setup, which shows the minimax optimality of the  Oracle Trans-Lasso algorithm in $\Theta_1(s,h)$.
\begin{theorem}[Minimax lower bound for $q=1$]
\label{thm2-low}
Assume Condition \ref{cond1} and Condition \ref{cond2}. If $\max\{s\log p/(n_{\mathcal{A}}+n_0),~h(\log p/n_0)^{1/2}\}=o(1)$, then
{\small
\begin{align*}
&{\mathbb{P}}\left(\inf_{\hat{\beta}}\sup_{\beta\in \Theta_1(s,h)} \|\hat{\beta}-\beta\|_2^2\geq c_1\frac{s\log p}{n_{\mathcal{A}}+n_0}+ c_2 \frac{s\log p}{n_0}\wedge h\left(\frac{\log p}{n_0}\right)^{1/2}\wedge h^2\right)\geq \frac{1}{2}
\end{align*}
}
for some positive constants $c_1$ and $c_2$.
\end{theorem}
Theorem \ref{thm2-low} implies that $\hat{\beta}$ obtained by the Oracle Trans-Lasso algorithm is minimax rate optimal in $\Theta_1(s,h)$ under the conditions of Theorem \ref{thm0-l1}. To understand the lower bound, the term $s\log p/(n_{\mathcal{A}}+n_0)$ is the optimal convergence rate when $w^{(k)}=\beta$ for all $k \in \mathcal{A}$. This is an extremely ideal case where we have $n_{\mathcal{A}}+n_0$ \textit{i.i.d.} samples from the target model. The second term in the lower bound is the optimal convergence rate when $w^{(k)}=0$ for all $k \in \mathcal{A}$, i.e., the auxiliary samples are not helpful at all. Let $\mathcal{B}_q(r)=\{u\in {\real}^p:\|u\|_q\leq r\}$ denote the $\ell_q$-ball with radius $r$ centered at zero. In this case, the definition of $\Theta_1(s,h)$ implies that $\beta\in \mathcal{B}_0(s)\cap \mathcal{B}_1(h)$ and the second term in the lower bound is indeed the minimax optimal rate for estimation when $\beta\in \mathcal{B}_0(s)\cap \mathcal{B}_1(h)$ with $n_0$ \textit{i.i.d.} samples \citep{TS14}.

\section{Unknown Set of Informative Auxiliary Samples}
\label{sec3}
The  Oracle Trans-Lasso algorithm is based on the knowledge of the informative set $\mathcal{A}$. In some applications, the informative set $\mathcal{A}$ is not given, which makes the transfer learning problem more challenging. In this section, we propose a data-driven method for estimation and prediction when $\mathcal{A}$ is unknown. The proposed algorithm is described in detail in Section \ref{sec3-1} and \ref{sec3-2}. Its theoretical properties are studied in Section \ref{sec3-3}.

\subsection{The Trans-Lasso Algorithm}
\label{sec3-1}

Our proposed algorithm, called Trans-Lasso, consists of two main steps. First, we construct a collection of candidate estimators, where each of them is based on an estimate of $\mathcal{A}$. Second, we perform an aggregation step \citep{RT11, DRZ12, Dai18} on these candidate estimators. Under proper conditions, the aggregated estimator is guaranteed to be not much worse than the best candidate estimator under consideration in terms of prediction. For technical reasons, we need the candidate estimators and the sample for aggregation to be independent. Hence, we start with sample splitting.
We need some more notation. For a generic estimate of $\beta$, $b$, denote its sum of squared prediction error as
\[
   \widehat{Q}(\mathcal{I},b)=\sum_{i\in \mathcal{I}}\|y^{(0)}_i-(x^{(0)}_i)^{\intercal}b\|_2^2,
\]
where $\mathcal{I}$ is a subset of $\{1,\dots,n_0\}$.
Let $\Lambda^{L+1}=\{\nu\in{\real}^{L+1}: \nu_l\geq 0,\sum_{l=0}^L\nu_l=1\}$ denote an $L$-dimensional simplex.
The Trans-Lasso algorithm is presented in Algorithm \ref{algo2}.

\begin{algorithm}
 \SetKwInOut{Input}{Input}
    \SetKwInOut{Output}{Output}
\SetAlgoLined
 \Input{ Primary data $(X^{(0)},y^{(0)})$ and samples from $K$ auxiliary  studies  $\{X^{(k)},y^{(k)}\}_{k=1}^K$.}
 \Output{$\hat{\beta}^{\hat{\theta}}$.}

\underline{Step 1}.
Let $\mathcal{I}$ be a random subset of $\{1,\dots,n_0\}$ such that $|\mathcal{I}|\approx n_0/2$. Let $\mathcal{I}^c=\{1,\dots,n_0\}\setminus \mathcal{I}$.

\underline{Step 2}.
 Construct $L+1$ candidate sets of $\mathcal{A}$, $\big\{\widehat{G}_0,\widehat{G}_1,\dots,\widehat{G}_{L}\big\}$ such that $\widehat{G}_0=\emptyset$ and $\widehat{G}_1,\dots,\widehat{G}_{L}$ are based on (\ref{eq-hGl}) using $\left(X_{\mathcal{I},.}^{(0)},y_{\mathcal{I}}^{(0)}\right)$ and $\{X^{(k)},y^{(k)}\}_{k=1}^M$.

\underline{Step 3}. For each $0\leq l\leq L$, run the Oracle Trans-Lasso algorithm with primary sample $(X_{\mathcal{I},.}^{(0)},y_{\mathcal{I}}^{(0)})$ and auxiliary samples $\{X^{(k)},y^{(k)}\}_{k\in\widehat{G}_l}$. Denote the output as $\hat{\beta}(\widehat{G}_l)$ for $0\leq l\leq L$.

\underline{Step 4}.
Compute
{\small
\begin{align}
  &\hat{\theta}=\label{theta-bma}\\
  &\mathop{\rm arg\, min}_{\theta\in\Lambda^{L+1}}\left\{\widehat{Q}\big(\mathcal{I}^c,\sum_{l=0}^L\hat{\beta}(\widehat{G}_l)\theta_l\big)+\sum_{l=0}^L\theta_l \widehat{Q}(\mathcal{I}^c,\hat{\beta}(\widehat{G}_l))+\frac{2\lambda_{\theta}\log (L+1)}{n_0}\|\theta\|_1\right\}\nonumber
 \end{align}
 }
for some $\lambda_{\theta}>0$. Output
\begin{equation}
\label{hbeta-htheta}
   \hat{\beta}^{\hat{\theta}}=\sum_{l=0}^L\hat{\theta}_{l}\hat{\beta}(\widehat{G}_l).
\end{equation}
 \caption{\textbf{Trans-Lasso Algorithm}} \label{algo2}
\end{algorithm}
As an illustration, steps 2 and 3 of the Trans-Lasso algorithm are devoted to constructing some initial estimates of $\beta$, $\hat{\beta}(\widehat{G}_l)$. They are computed using the Oracle Trans-Lasso algorithm by treating each $\widehat{G}_l$ as the set of informative auxiliary samples. We construct $\widehat{G}_l$ to be a class of estimates of $\mathcal{A}$ and the detailed procedure is provided in Section \ref{sec3-2}.
Step 4 is based on the Q-aggregation proposed in \citet{DRZ12} with a uniform prior and a simplified tuning parameter. The Q-aggregation can be viewed as a weighted version of least square aggregation and exponential aggregation \citep{RT11} and it has been shown to be rate optimal both in expectation and with high probability for model selection aggregation problems.

The framework of model selection aggregation is a good fit for the transfer learning task under consideration.
On one hand, it guarantees the robustness of Trans-Lasso in the following sense. Notice that $\hat{\beta}(\widehat{G}_0)$ corresponds to the Lasso estimator only using the primary samples and it is always included in our dictionary. The purpose is that, invoking the property of model selection aggregation, the performance of $\hat{\beta}^{\hat{\theta}}$ is guaranteed to be not much worse than the performance of the original Lasso estimator under mild conditions. This shows the performance of Trans-Lasso will not be ruined by adversarial auxiliary samples. Formal statements are provided in Section \ref{sec3-3}.
On the other hand, the gain of Trans-Lasso relates to the qualities of $\widehat{G}_1,\dots,\widehat{G}_L$.
If
\begin{equation}
\label{cond-agg}
{\mathbb{P}}\left(\widehat{G}_l\subseteq\mathcal{A}, ~\text{for some}~ 1\leq l\leq L\right)\rightarrow 1,
\end{equation}
i.e., $\widehat{G}_l$ is a nonempty subset of the informative set $\mathcal{A}$, then the model selection aggregation property implies that the performance of $\hat{\beta}^{\hat{\theta}}$ is not much worse than the performance of the Oracle Trans-Lasso with $\sum_{k\in \widehat{G}_l}n_k$ informative auxiliary samples.
Ideally, one would like to achieve $\widehat{G}_l=\mathcal{A}$ for some $1\leq l\leq L$ with high probability. However, it can rely on strong assumptions that  may not be guaranteed in practical situations.

To motivate our constructions of $\widehat{G}_l$, let us first point out a naive construction of candidate sets, which consists of $2^K$ candidates. These candidates are all different combinations of $\{1,\dots, K\}$, denoted by $\widehat{G}_1,\dots, \widehat{G}_{2^K}$.
It is obvious that $\mathcal{A}$ is an element of this candidate sets. However, the number of candidates is too large and it can be computationally burdensome. Furthermore, the cost of aggregation can be significantly high, which is of order $K/n_0$ as will be seen in Lemma \ref{thm-ag1}.
In contrast, we would like to pursue a much smaller number of candidate sets such that the cost of aggregation is almost negligible and (\ref{cond-agg}) can be achieved under mild conditions.
We introduce our proposed construction of candidate sets in the next subsection.

\subsection{Constructing the Candidate Sets for Aggregation}
\label{sec3-2}
As illustrated in Section \ref{sec3-1}, the goal of Step 2 is to have a class of candidate sets, $\{\widehat{G}_0,\dots,\widehat{G}_{L}\}$,  that satisfy (\ref{cond-agg}) under certain conditions.
Our idea is to exploit the sparsity patterns of the contrast vectors.
Specifically, recall that the definition of $\mathcal{A}$ implies that $\{\delta^{(k)}\}_{k \in \mathcal{A}}$ are sparser than $\{\delta^{(k)}\}_{k \in \mathcal{A}^c}$, where $\mathcal{A}^c=\{1,\dots,K\}\setminus \mathcal{A}$. This property motivates us to find a sparsity index $R^{(k)}$ and its estimator $\widehat{R}^{(k)}$ for each $1\leq k\leq K$ such that
\begin{align}
\label{cond-rk}
 \max_{k \in \mathcal{A}^o}R^{(k)}< \min_{k \in \mathcal{A}^c}R^{(k)}\quad\text{and}\quad   {\mathbb{P}}\left(\max_{k \in \mathcal{A}^o}\widehat{R}^{(k)}< \min_{k \in \mathcal{A}^c}\widehat{R}^{(k)}\right)\rightarrow 1,
\end{align}
where $\mathcal{A}^o$ is some subset of $\mathcal{A}$.
In words, the sparsity indices in $\mathcal{A}^o$ are no larger than the sparsity indices in $\mathcal{A}^c$ and so are their estimators with high probability. To utilize (\ref{cond-rk}), we can define the candidate sets as
\begin{align}
\label{eq-hGl}
   \widehat{G}_l=\left\{1\leq k\leq K: \widehat{R}^{(k)}~ \text{is among the first $l$ smallest of all}\right\}
\end{align}
for $1\leq l\leq K$.
That is, $\widehat{G}_l$ is the set of auxiliary samples whose estimated sparsity indices are among the first $l$ smallest.
A direct consequence of (\ref{cond-rk}) and (\ref{eq-hGl}) is that ${\mathbb{P}}(\widehat{G}_{|\mathcal{A}^o|}=\mathcal{A}^o)\rightarrow 1$ and hence the desirable property (\ref{cond-agg}) is satisfied.
To achieve the largest gain in transfer learning, we would like to find proper sparsity indices such that (\ref{cond-rk}) holds for $|\mathcal{A}^o|$ as large as possible.
Notice that $\widehat{G}_{K+1}=\{1,\dots, K\}$ is always included as candidates according to (\ref{eq-hGl}). Hence, in the special cases where all the auxiliary samples are informative or none of the auxiliary samples are informative, it holds that $\widehat{G}_{|\mathcal{A}|}=\mathcal{A}$ and the Trans-Lasso is not much worse than the Oracle Trans-Lasso.
The more challenging cases are $0<|\mathcal{A}|<K$.

As $\{\delta^{(k)}\}_{k\in\mathcal{A}^c}$ are not necessarily sparse, the estimation of $\delta^{(k)}$ or functions of $\delta^{(k)}$, $1\leq k\leq K$, is not trivial.
We consider using $R^{(k)}=\|\Sigma\delta^{(k)}\|_2^2$, which is a function of the population-level marginal statistics, as the oracle sparsity index for $k$-th auxiliary sample. The advantage of $R^{(k)}$ is that it has a natural unbiased estimate without further assumptions.
Let us relate $R^{(k)}$ to the sparsity of $\delta^{(k)}$ using a Bayesian characterization of sparse vectors assuming $\Sigma^{(k)}=\Sigma$ for all $0\leq k\leq K$. If $\delta^{(k)}_j$ are \textit{i.i.d.} Laplacian distributed with mean zero and variance $\nu_k^2$ for each $k$, then it follows from the properties of Laplacian distribution \citep{LK15} that
\[
   {\mathbb{E}}[\|\delta^{(k)}\|_1]=p\nu_k={\mathbb{E}}^{1/2}[\|\Sigma\delta^{(k)}\|^2_2]\frac{p}{\textrm{Tr}^{1/2}(\Sigma\Sig)},
\]
where $\textrm{Tr}^{1/2}(\Sigma\Sig)/p$ does not depend on $k$. Hence, the rank of ${\mathbb{E}}[\|\Sigma\delta^{(k)}\|^2_2]$ is the same as the rank of ${\mathbb{E}}[\|\delta^{(k)}\|_1]$. As $\max_{k\in \mathcal{A}}\|\delta^{(k)}\|_1< \min_{k\in \mathcal{A}^c}\|\delta^{(k)}\|_1$, it is reasonable to expect $\max_{k\in \mathcal{A}}\|\Sigma\delta^{(k)}\|^2_2< \min_{k\in \mathcal{A}^c}\|\Sigma\delta^{(k)}\|^2_2$.
 Obviously, the above derivation holds for many other zero mean prior distributions besides Laplacian. This illustrates our motivation for considering $R^{(k)}$ as the oracle sparsity index.

We next  introduce the estimated version, $\widehat{R}^{(k)}$, based on the primary data $\{(x_i^{(0)})^{\intercal},y_{i}^{(0)}\}_{i\in \mathcal{I}}$ (after sample splitting) and auxiliary samples $\{X^{(k)},y^{(k)}\}_{k=1}^K$.
We first perform a SURE screening \citep{FL08} on the marginal statistics to reduce the effects of  random noises.
We summarize our proposal for Step 2 of the Trans-Lasso as follows (see Algorithm \ref{algo2.2}). Let $n_*=\min_{0\leq k\leq K}n_k$.

\begin{algorithm}[H]
\underline{Step 2.1}.
For $1\leq k\leq K$, compute the marginal statistics
\begin{align}
\widehat{\Delta}^{(k)}&=\frac{1}{n_k}\sum_{i=1}^{n_k}x_i^{(k)}y_i^{(k)}-\frac{1}{|\mathcal{I}|}\sum_{i\in \mathcal{I}}x_i^{(0)}y_i^{(0)},\label{eq-hDeltak}\
\end{align}
For each $k\in\{1,\dots, K\}$, let $\widehat{T}_k$ be obtained by SURE screening such that
\begin{align*}
  & \widehat{T}_k=\left\{1\leq j\leq p:~|\widehat{\Delta}_j^{(k)}|~\text{is among the first}~t_*~\text{largest of all}\right\}
   \end{align*}
for a fixed $t_*=n_*^{\alpha},~0\leq \alpha<1$.

\underline{Step 2.2}. Define the estimated sparse index for the $k$-th auxiliary sample as
\begin{align}
\label{eq-hRk}
   \widehat{R}^{(k)}=\left\|\widehat{\Delta}_{\widehat{T}_k}^{(k)}\right\|_2^2.
\end{align}
\underline{Step 2.3}.
Compute $\widehat{G}_l$ as in (\ref{eq-hGl}) for $l=1,\dots, L$.
\caption{\textbf{Step 2 of the Trans-Lasso Algorithm}} \label{algo2.2}
\end{algorithm}

One can see that $\widehat{\Delta}^{(k)}$ are empirical marginal statistics such that ${\mathbb{E}}[\widehat{\Delta}^{(k)}]=\Sigma\delta^{(k)}$ for $k\in\mathcal{A}$.
The set $\widehat{T}_k$ is the set of first $t_*$ largest marginal statistics for the $k$-th sample.
The purpose of screening the marginal statistics is to reduce the magnitude of noise. Notice that the un-screened version $\|\widehat{\Delta}^{(k)}\|_2^2$ is a sum of $p$ random variables and it contains noise of order $p/(n_k\wedge n_0)$, which diverges fast as $p$ is much larger than the sample sizes. By screening with $t_*$ of order $n_*^{\alpha}$, $\alpha<1$, the errors induced by the random noises is under control. In practice, the auxiliary samples with very small sample sizes can be removed from the analysis as their contributions to the target problem is mild.
Desirable choices of $\widehat{T}_k$ should keep the variation of $\Sigma\delta^{(k)}$ as much as possible.
Under proper conditions, SURE screening can consistently select a set of strong marginal statistics and hence is appropriate for the current purpose.
In Step 2.2, we compute $\widehat{R}^{(k)}$ based on the marginal statistics which are selected by SURE screening.
In practice, different choices of $t_*$ may lead to different realizations of $\widehat{G}_l$. One can compute multiple sets of $\{\widehat{R}^{(k)}\}_{k=1}^K$ with different $t_*$ which give multiple sets of $\{\widehat{G}_l\}_{l=1}^K$. It will be seen from Lemma \ref{thm-ag1} that a finite number of choices on $t_*$ does not affect the rate of convergence.

\subsection{Theoretical Properties of Trans-Lasso}
\label{sec3-3}
In this subsection, we derive the theoretical guarantees for the Trans-Lasso algorithm. We first establish the model selection aggregation type of results for the Trans-Lasso estimator $\hat{\beta}^{\hat{\theta}}$.
\begin{lemma}[Q-aggregation for Trans-Lasso]
\label{thm-ag1}
Assume that Condition \ref{cond1} and Condition \ref{cond2} hold true. Let $\hat{\theta}$ be computed with
$\lambda_{\theta}\geq 4\sigma_0^2$. With probability at least $1-t$, it holds that
\begin{align}
& \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}^{\hat{\theta}}-\beta)\right\|_2^2\leq \min_{0\leq l\leq L} \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}(\widehat{G}_l)-\beta)\right\|_2^2+\frac{\lambda_{\theta}\log (L/t)}{n_0}.\label{re0-ms}
\end{align}
If $\|\Sigma\|_2L\leq c_1n_0$ for some small enough constant $c_1$, then
\begin{align}
\label{re1-ms}
\left\|\hat{\beta}^{\hat{\theta}}-\beta\right\|_2^2\lesssim_{{\mathbb{P}}} \min_{0\leq l\leq L} \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}(\widehat{G}_l)-\beta)\right\|_2^2\vee \|\hat{\beta}(\widehat{G}_l)-\beta\|_2^2+\frac{\log L}{n_0}.
\end{align}
\end{lemma}
\begin{remark}
\label{re1}{\rm
Assume that Conditions \ref{cond1} and \ref{cond2} hold. Let $\hat{\theta}$ be obtained with
$\lambda_{\theta}\geq 4\sigma_0^2$. For any $L\geq 1$, it holds that
\[
 \big\|\hat{\beta}^{\hat{\theta}}-\beta\big\|_2^2\lesssim_{{\mathbb{P}}} \min_{0\leq l\leq L} \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}(\widehat{G}_l)-\beta)\right\|_2^2\vee \|\hat{\beta}(\widehat{G}_l)-\beta\|_2^2+\sqrt{\frac{\log L}{n_0}}.
 \]
}\end{remark}

Lemma \ref{thm-ag1} implies that the performance of $\hat{\beta}^{\hat{\theta}}$ only depends on the best candidate regardless of the performance of other candidates under mild conditions.
As commented before, this result guarantees the robustness and efficiency of Trans-Lasso, which can be formally stated as follows.
As the original Lasso is always in our dictionary, (\ref{re0-ms}) and (\ref{re1-ms}) imply that $\hat{\beta}^{\hat{\theta}}$ is not much worse than the Lasso in prediction and estimation. Formally, ``not much worse'' refers to the last term in (\ref{re0-ms}), which can be viewed as the cost of ``searching'' for the best candidate model within the dictionary which is of order $\log L/n_0$. This term is almost negligible, say, when $L=O(K)$, which corresponds to our constructed candidate estimators. This demonstrates the robustness of $\hat{\beta}^{\hat{\theta}}$ to adversarial auxiliary samples. Furthermore, if (\ref{cond-agg}) holds, then the prediction and estimation errors of Trans-Lasso are comparable to the Oracle Trans-Lasso based on auxiliary samples in $\mathcal{A}^o$.

The prediction error bound in (\ref{re0-ms}) follows from Corollary 3.1 in \citet{DRZ12}. However, the aggregation methods do not have theoretical guarantees in estimation error in general. Indeed, an estimator with $\ell_2$-error guarantee is crucial for more challenging tasks, such as out-of-sample prediction and inference. For our transfer learning task, we show in (\ref{re1-ms}) that the estimation error is of the same order if the cardinality of the dictionary is $L\leq cn_0$ for some small enough $c$. For our constructed dictionary, it suffices to require $K\leq cn_0$. In many practical applications, $K$ is relatively small compared to the sample sizes and hence this assumption is not very strict.
In Remark \ref{re1}, we provide an upper bound on the estimation error which holds for arbitrarily large $L$ but is slower than (\ref{re1-ms}) in general.

In the following, we provide sufficient conditions such that the desirable property (\ref{cond-rk}) holds with $\widehat{R}^{(k)}$ defined in (\ref{eq-hRk}) and hence (\ref{cond-agg}) is satisfied.
 For each $k \in\mathcal{A}^c$, define a set
\begin{align}
\label{eq-Hk}
   H_k=\left\{1\leq j\leq p: |\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta|>n_*^{-\kappa}, ~\kappa<\alpha/2\right\}.
\end{align}
Recall that $\alpha$ is defined such that $t_*=n^\alpha$. In fact, $H_k$ is the set of ``strong'' marginal statistics that  can be consistently selected into $\widehat{T}_k$ for each $k\in \mathcal{A}^c$.
We see that $\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta=\Sigma_{j,.}\delta^{(k)}$ if $\Sigma^{(k)}=\Sigma^{(0)}$ for $k\in\mathcal{A}^c$. The definition of $\mathcal{H}_k$ in (\ref{eq-Hk}) allows for heterogeneous designs among non-informative auxiliary samples.
\begin{condition}
\label{cond4}{\rm
(a) For each $k\in \mathcal{A}^c$, each row of $X^{(k)}$ is \textit{i.i.d.} Gaussian with mean zero and covariance matrix $\Sigma^{(k)}$. The largest eigenvalue of $\Sigma^{(k)}$ is bounded away from infinity for any $k\in \mathcal{A}^c$.
For each $k\in \mathcal{A}^c$, the random noises $\eps^{(k)}_i$ are \textit{i.i.d.} Gaussian with mean zero and variance $\sigma^2_k$.

(b)It holds that $\log p\vee \log K\leq c_1 \sqrt{n_*}$ for a small enough constant $c_1$. Moreover,
\begin{align}
\label{eq1-cond4b}
 \min_{k \in\mathcal{A}^c}\sum_{j\in H_k}|\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta|^2\geq \frac{c_1\log p}{n_*^{1-\alpha}}
\end{align}
for some large enough constant $c_1>0$.
} \end{condition}

The Gaussian assumptions in Condition \ref{cond4}(a) guarantee the desirable properties of SURE screening for the non-informative auxiliary studies. In fact, the Gaussian assumption can be relaxed to be sub-Gaussian random variables according to some recent studies \citep{SURE2}. For the conciseness of the proof, we consider Gaussian distributed random variables.
Condition \ref{cond4}(b) first puts constraint on the relative dimensions. It is trivial in the regime that $p\vee K\leq n_*^{\xi}$ for any finite $\xi>0$.  The expression (\ref{eq1-cond4b}) requires that for each $k\in\mathcal{A}^c$, there exists a subset of strong marginal statistics such that their squared sum is beyond some noise barrier. This condition is mild by choosing $\alpha$ such that $\log p\ll n_*^{1-\alpha}$ and $\alpha=1/2$ is an obvious choice revoking the first part of Condition \ref{cond4}(b). For instance, if $\min_{k\in \mathcal{A}^c}\|{\mathbb{E}}[\widehat{\Delta}^{(k)}]\|_{\infty}\geq c_0>0$, then (\ref{eq1-cond4b}) holds with any $\alpha\leq 1/2$. In words, a sufficient condition for (\ref{eq1-cond4b}) is that at least one marginal statistic in the $k$-th study is of constant order for $k\in\mathcal{A}^c$. We see that larger $n_*$ makes Condition \ref{cond4} weaker. As mentioned before, it is helpful to remove the auxiliary samples with very small sample sizes from the analysis.

In the next theorem, we demonstrate the theoretical properties of $\widehat{R}^{(k)}$ and provide a complete analysis of the Trans-Lasso algorithm. Let $\mathcal{A}^o$ be a subset of $\mathcal{A}$ such that
\[
   \mathcal{A}^o=\left\{k\in \mathcal{A}:\|\Sigma^{(0)}\delta^{(k)}\|_2^2\leq c_1\min_{k \in\mathcal{A}^c}\sum_{j\in H_k}|\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta|^2\right\}
\]
for some $c_1<1$ and $H_k$ defined in (\ref{eq-Hk}). In general, one can see that the informative auxiliary samples with sparser $\delta^{(k)}$ are more likely to be included into $\mathcal{A}^o$. Specially, the fact that $\max_{k\in \mathcal{A}}\|\Sigma^{(0)}\delta^{(k)}\|_2^2\leq \|\Sigma^{(0)}\|_2^2h^2$ implies $\mathcal{A}^o=\mathcal{A}$ when $h$ is sufficiently small. We will show (\ref{cond-rk}) for such $\mathcal{A}^o$ with $\widehat{R}^{(k)}$ defined in (\ref{eq-hRk}).
 Let $n_{\mathcal{A}^o}=\sum_{k\in \mathcal{A}^o}n_k$.

\begin{theorem}[Convergence Rate of the Trans-Lasso]
\label{sec4-lem1}
Assume that the conditions of Theorem \ref{thm0-l1} and Condition \ref{cond4} hold. Then
\begin{align}
\label{cond-agg2}
   {\mathbb{P}}\left(\max_{k \in \mathcal{A}^o}\widehat{R}^{(k)}< \min_{k \in \mathcal{A}^c}\widehat{R}^{(k)} \right)\rightarrow 1.
\end{align}
Let $\hat{\beta}^{\hat{\theta}}$ be computed using the Trans-Lasso algorithm with $\lambda_{\theta}\geq 4\sigma^2_0$. If $K\leq cn_0$ for a sufficiently small constant $c>0$, then
\begin{align}
& \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}^{\hat{\theta}}-\beta)\right\|_2^2\vee\left\|\hat{\beta}^{\hat{\theta}}-\beta\right\|_2^2\nonumber\\
&=O_P\left(\frac{s\log p}{n_{\mathcal{A}^o}+n_0}+\frac{s\log p}{n_0}\wedge h\sqrt{\frac{\log p}{n_0}}\wedge h^2+\frac{\log K}{n_0}\right).
\label{re3-agg}
\end{align}
\end{theorem}
\begin{remark}
\label{re3}{\rm
Under the conditions of Theorem \ref{sec4-lem1}, if
\[ \|\Sigma^{(0)}\|_2^2h^2\leq \alpha\min_{k \in\mathcal{A}^c}\sum_{j\in H_k}|\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta|^2~\text{for some}~\alpha<1,
\]
then ${\mathbb{P}}\left(\max_{k \in \mathcal{A}}\widehat{R}^{(k)}< \min_{k \in \mathcal{A}^c}\widehat{R}^{(k)} \right)\rightarrow 1$ and
\begin{align*}
& \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}^{\hat{\theta}}-\beta)\right\|_2^2\vee\left\|\hat{\beta}^{\hat{\theta}}-\beta\right\|_2^2\\
&=O_P\left(\frac{s\log p}{n_{\mathcal{A}}+n_0}+\frac{s\log p}{n_0}\wedge h\sqrt{\frac{\log p}{n_0}}\wedge h^2+\frac{\log K}{n_0}\right).
\end{align*}
}\end{remark}
The result in (\ref{cond-agg2}) implies the estimated sparse indices in $\mathcal{A}^o$ and in $\mathcal{A}^c$ are separated with high probability. As illustrated before, a consequence of (\ref{cond-agg2}) is (\ref{cond-agg}) for the candidate sets $\widehat{G}_l$ defined in (\ref{eq-hGl}). Together with Theorem \ref{thm0-l1} and Lemma \ref{thm-ag1}, we arrive at (\ref{re3-agg}).
In Remark \ref{re3}, we develop a sufficient condition for  $\mathcal{A}^o=\mathcal{A}$, which requires sufficiently small $h$. Under this condition, the estimation and prediction errors of the Trans-Lasso are comparable the  case where  $\mathcal{A}$ is known, i.e. the adaptation to $\mathcal{A}$ is achieved. Remark \ref{re3} is a direct consequence of Theorem \ref{sec4-lem1} and the fact that $\max_{k\in\mathcal{A}}\|\Sigma^{(0)}\delta^{(k)}\|_2^2\leq \|\Sigma^{(0)}\|_2^2h^2$.

It is worth mentioning that Condition \ref{cond4} is only employed to show the gain of Trans-Lasso  and the robustness property of Trans-Lasso holds without any conditions on the non-informative samples (Lemma \ref{thm-ag1}). In practice, missing a few informative auxiliary samples may not be a very serious concern. One can see that when $n_{\mathcal{A}^o}$ is large enough such that the first term on the right-hand side of (\ref{re3-agg}) no longer dominates, increasing the number of auxiliary samples will not improve the convergence rate. In contrast, it is more important to guarantee that the estimator is not affected by the adversarial auxiliary samples.
The empirical performance of Trans-Lasso is carefully studied in Section \ref{sec-simu}.

\section{Extensions to Heterogeneous Designs and $\ell_q$-sparse Contrasts}
\label{sec4}
In this section, we extend the algorithms and theoretical results developed in Sections \ref{sec2} and \ref{sec3}. Section \ref{sec4-1} considers the case where the design matrices are heterogeneous with difference covariance structures and Section \ref{sec4-2} generalizes the sparse contrasts from $\ell_1$-constraint to  $\ell_q$-constraint for $q\in [0,1)$ and presents a rate-optimal estimator in this setting.

\subsection{Heterogeneous Designs}
\label{sec4-1}

The Oracle Trans-Lasso algorithm  proposed in Section \ref{sec2} can be directly applied to the  setting where the design matrices are heterogeneous.
To establish the theoretical guarantees in the heterogeneous case, we first introduce a relaxed version of Condition \ref{cond1} as follows.
\begin{condition}
\label{cond1b}{\rm
For each $k\in \mathcal{A}\cup\{0\}$, each row of $X^{(k)}$ is \textit{i.i.d.} Gaussian with mean zero and covariance matrix $\Sigma^{(k)}$. The smallest and largest eigenvalues of $\Sigma^{(k)}$ are bounded away from zero and infinity, respectively, for all $k\in \mathcal{A}\cup\{0\}$.
}\end{condition}
Define
\[
    C_{\Sigma}=1+\max_{j\leq p}\max_{k \in \mathcal{A}}\Big\|e_j^{\intercal}\big(\Sigma^{(k)}-\Sigma^{(0)}\big)\Big(\sum_{k \in \mathcal{A}\cup \{0\}}\alpha_k\Sigma^{(k)}\Big)^{-1}\Big\|_1.
\]
The parameter $C_{\Sigma}$ characterizes the differences between $\Sigma^{(k)}$ and $\Sigma^{(0)}$ for $k\in\mathcal{A}$. Notice that $C_{\Sigma}$ is a constant if $\max_{1\leq j\leq p}\|e_j^{\intercal}(\Sigma^{(k)}-\Sigma^{(0)})\|_0\leq C<\infty$ for all $k \in \mathcal{A}$, where examples include block diagonal $\Sigma^{(k)}$ with constant block sizes or banded $\Sigma^{(k)}$ with constant bandwidths for $k \in \mathcal{A}$.
The following theorem characterizes the rate of convergence of the Oracle Trans-Lasso estimator in terms of $C_{\Sigma}$.

\begin{theorem}[Oracle Trans-Lasso with heterogeneous designs]
\label{thm1-l1}
Assume that Condition \ref{cond2} and Condition \ref{cond1b} hold true. We take $\lambda_w= \max_{k\in\mathcal{A}\cup\{0\}}c_1\sqrt{{\mathbb{E}}[(y^{(k)}_i)^2]\log p/(n_{\mathcal{A}}+n_0})$ and $\lambda_\delta=c_2\sigma_0 \sqrt{\log p/n_0}$  for some sufficiently large constants $c_1$ and $c_2$ only depending on $C_0$.
If $s\log p/(n_{\mathcal{A}}+n_0)+C_{\Sigma}h(\log p/n_0)^{1/2}=o((\log p/n_0)^{1/4})$, then
\begin{align}
&\frac{1}{n_0}\|X^{(0)}(\hat{\beta}-\beta)\|_2^2\vee\|\hat{\beta}-\beta\|_2^2\nonumber\\
&=O_P\left(\frac{s\log p}{n_{\mathcal{A}}+n_0}+\frac{s\log p}{n_0}\wedge C_{\Sigma}h\sqrt{\frac{\log p}{n_0}}\wedge C_{\Sigma}^2h^2\right) \label{l1-re2}.
\end{align}
\end{theorem}
When $\mathcal{A}$ is non-empty, the right-hand side of (\ref{l1-re1}) is sharper than $s\log p/n_0$ if $n_{\mathcal{A}_0}\gg n_0$ and $C_{\Sigma}h\sqrt{\log p/n_0}\ll s$. We see that small $C_{\Sigma}$ is favorable. This implies that  the informative auxiliary samples should not only have sparse contrasts but have similar Gram matrices to the primary one.
When $\mathcal{A}$ is unknown, we consider $\tilde{\mathcal{A}}^o$, a subset of $\mathcal{A}$ such that
\[
  \tilde{\mathcal{A}}^o=\left\{k\in \mathcal{A}:\|\Sigma^{(k)}w^{(k)}-\Sigma^{(0)}\beta\|_2^2<c_1\min_{k \in\mathcal{A}^c}\sum_{j\in H_k}|\Sigma^{(k)}_{j,.}w^{(k)}-\Sigma^{(0)}_{j,.}\beta|^2\right\}
\]
for some $c_1<1$ and $H_k$ defined in (\ref{eq-Hk}). This is a generalization of $\mathcal{A}^o$ to the case of heterogeneous designs.

\begin{corollary}[Trans-Lasso with heterogeneous designs]
\label{cor1-l1}
Assume the conditions of Theorem \ref{thm1-l1} and Condition \ref{cond4}.
Let $\hat{\beta}^{\hat{\theta}}$ be computed via the Trans-Lasso algorithm with $\lambda_{\theta}\geq 4\sigma^2_0$. If $K\leq cn_0$ for a small enough constant $c$, then
\begin{align}
& \frac{1}{|\mathcal{I}^c|}\left\|X^{(0)}_{\mathcal{I}^c,.}(\hat{\beta}^{\hat{\theta}}-\beta)\right\|_2^2\vee \|\hat{\beta}^{\hat{\theta}}-\beta\|_2^2\nonumber\\
&=O_P\left(\frac{s\log p}{n_{\tilde{\mathcal{A}}^o}+n_0}+\frac{s\log p}{n_0}\wedge C_{\Sigma}h\sqrt{\frac{\log p}{n_0}}\wedge C_{\Sigma}^2h^2+\frac{\log K}{n_0}\right).
\label{re4-agg}
\end{align}
\end{corollary}
Corollary \ref{cor1-l1} provides an upper bound for the Tran-Lasso with heterogeneous designs. The numerical experiments for this setting are studied in Section \ref{sec-simu}.

 \subsection{$\ell_q$-sparse Contrasts}
 \label{sec4-2}

 We have so far focused on the  $\ell_1$-sparse characterization of the contrast vectors. The established framework can be extended to the settings where the contrast vectors are characterized in terms of the $\ell_q$-norm for $q\in[0,1)$. We discuss the exact sparse contrasts ($q=0$) here and leave the results for $q\in(0,1)$  to the Appendix. We first consider the case when  $\mathcal{A}_0$ is known and present the propopsed algorithm in Algorithm \ref{algo3}.

\vspace{0.1in}
\begin{algorithm}[H]
 \SetKwInOut{Input}{Input}
    \SetKwInOut{Output}{Output}
\SetAlgoLined
 \Input{Primary data $(X^{(0)},y^{(0)})$ and informative auxiliary samples $\{X^{(k)},y^{(k)}\}_{k\in\mathcal{A}_0}$}
 \Output{$\hat{\beta}(\mathcal{A}_0)$}

\underline{Step 1}.
Estimate each $\delta^{(k)}$, $k\in\mathcal{A}_0$, via
{\small
\begin{align*}
   &\hat{\delta}^{(k)}=\mathop{\rm arg\, min}_{\delta\in{\real}^p}\left\{\frac{1}{2}\delta^{\intercal}\widehat{\Sigma}^{\mathcal{A}_0}\delta-\delta^{\intercal}[(X^{(k)})^{\intercal}y^{(k)}/n_k-(X^{(0)})^{\intercal}y^{(0)}/n_0]+\lambda_k\|\delta\|_1\right\},\nonumber
\end{align*}
}
where $\widehat{\Sigma}^{\mathcal{A}_0}=\sum_{k\in \mathcal{A}_0\cup\{0\}}(X^{(k)})^{\intercal}X^{(k)}/(n_{\mathcal{A}_0}+n_0)$ and $\lambda_k>0$.

\underline{Step 2}. Compute
{\small
\begin{align*}
&\hat{\beta}(\mathcal{A}_0)=\mathop{\rm arg\, min}_{b\in{\real}^p} \Big\{\frac{1}{2(n_{\mathcal{A}_0}+n_0)}\sum_{k \in \mathcal{A}_0\cup\{0\}}\|y^{(k)}-X^{(k)}\hat{\delta}^{(k)}-X^{(k)}b\|_2^2+\lambda_{\beta}\|b\|_1\Big\}\;\nonumber
\end{align*}
}
for some $\lambda_{\beta}>0$.
 \caption{\textbf{Oracle Trans-Lasso algorithm for $q=0$}}\label{algo3}
\end{algorithm}

In the above algorithm, we estimate each $\delta^{(k)}$ based on the following moment equation:
\begin{align}
\label{eq-m1}
   {\mathbb{E}}[x^{(k)}_iy_i^{(k)}]- {\mathbb{E}}[x^{(0)}_iy_i^{(0)}]-\Sigma\delta^{(k)}=0,~k\in \mathcal{A}_0,
\end{align}
assuming that $\Sigma^{(k)}=\Sigma^{(0)}=\Sigma$ for all $k \in \mathcal{A}_0$. In the realization of Step 1, we replace $ {\mathbb{E}}[x^{(k)}_iy_i^{(k)}]$ and ${\mathbb{E}}[x^{(0)}_iy_i^{(0)}]$ by their unbiased sample versions, and the population Gram matrix $\Sigma$ by its unbiased estimator $\widehat{\Sigma}^{\mathcal{A}_0}$.
 In Step 2, we use the ``bias-corrected'' samples to estimate $\beta$. In contrast to the Oracle Trans-Lasso proposed in Section \ref{sec2-1}, the contrast vectors are estimated individually in the above algorithm. This is because the $\ell_1$-norm has the sub-additive property while $\ell_0$-norm does not. The computational cost in Step 1 can be relatively high if $\mathcal{A}_0$ is large. Moreover, Step 1 heavily relies on the homogeneous designs among informative auxiliary samples.
In the next theorem, we prove the convergence rate of $\hat{\beta}(\mathcal{A}_0)$.
\begin{theorem}[Convergence rate of $\hat{\beta}(\mathcal{A}_0)$]
\label{thm1-l0}
Assume that Condition \ref{cond1} and Condition \ref{cond2} hold true. Suppose that
{\small
\begin{equation}
\label{ms-ss}
\frac{h\log p}{n_0}=o\Big(\big(\frac{\log p}{n_0+n_{\mathcal{A}_0}}\big)^{1/4}\Big),~ \frac{s\log p}{n_{\mathcal{A}_0}+n_0}=o(1),~\text{and}~~n_{\mathcal{A}_0}\gtrsim |\mathcal{A}_0|n_0.
\end{equation}
}
For $\hat{\beta}(\mathcal{A}_0)$ computed with $\lambda_k\geq c_1(\|y^{(k)}\|_2/n_k+ \|y^{(0)}\|_2/n_0)\sqrt{\log p}$ and \\ $\lambda_{\beta}=c_2\sqrt{\log p/(n_0+n_{\mathcal{A}_0})}$ for large enough constants $c_1$ and $c_2$ only depending on $C_0$,
{\small
\begin{align}
 &\sup_{\beta\in\Theta_0(s,h)}\frac{1}{n_0+n_{\mathcal{A}_0}}\sum\limits_{k\in\mathcal{A}_0\cup\{0\}}\|X^{(k)}(\hat{\beta}(\mathcal{A}_0)-\beta)\|_2^2 \vee \|\hat{\beta}(\mathcal{A}_0)-\beta\|_2^2\nonumber\\
 &=O_P\left( \frac{s\log p}{n_{\mathcal{A}_0}+n_0}+\frac{(h\wedge s)\log p}{n_0}\right)\label{l0-re1}.
\end{align}
}
\end{theorem}
We see from (\ref{l0-re1}) that $\hat{\beta}(\mathcal{A}_0)$ has sharper convergence rate than the Lasso when $n_{\mathcal{A}_0}\gg n_0$ and $h\ll s$. The first two requirements in (\ref{ms-ss}) guarantee that the lower restricted eigenvalues of the sample Gram matrices are bounded away from zero. The last expression in (\ref{ms-ss}) requires the average auxiliary sample size is asymptotically no smaller than the primary sample size $n_0$. This is a checkable condition in practice. When $|\mathcal{A}_0|$ is fixed, this condition can be trivially satisfied. Indeed, if the auxiliary sample sizes are too small, there would not be much improvement with transfer learning.
We next establish the minimax lower bound for $\beta\in\Theta_0(s,h)$.
\begin{theorem}[Minimax lower bound for $q=0$]
\label{thm-mini2}
Assume Condition \ref{cond1} and Condition \ref{cond2}. Suppose that $\max\{h\log p/n_0, s\log p/(n_{\mathcal{A}_0}+n_0)\}=o(1)$.
There exists some constants $c_1$ and $c_2$ such that
\begin{align*}
{\mathbb{P}}\left(\inf_{\hat{\beta}}\sup_{\Theta_0(s,h)} \|\hat{\beta}-\beta\|_2^2\geq c_1\frac{s\log p}{n_{\mathcal{A}_0}+n_0}+ c_2\frac{(h\wedge s)\log p}{n_0}\right)\geq \frac{1}{2}.
\end{align*}
\end{theorem}
Theorem \ref{thm-mini2} demonstrates the minimax optimality of $\hat{\beta}(\mathcal{A}_0)$ in $\Theta_0(s,h)$ under the conditions of Theorem \ref{thm1-l0}.
When $\mathcal{A}_0$ is unknown, one can consider the Trans-Lasso algorithm where the Oracle Trans-Lasso is replaced with the Oracle Trans-Lasso for $q=0$. We see that Lemma \ref{thm-ag1} still holds and a similar result as Theorem \ref{sec4-lem1} can be established with $\mathcal{A}$ replaced by $\mathcal{A}_0$. For the sake of conciseness, it is omitted here.

While $\ell_0$-sparsity is widely assumed and studied in the high-dimensional literature, $\ell_0$-sparse contrast vectors may not be a realistic situation. First, an $\ell_0$-sparse $\delta^{(k)}$ implies that $\beta$ and $w^{(k)}$ have the same coefficients in most coordinates which may be impractical. Second, a typical data preprocessing step is to standardize the data such that $\|y^{(k)}\|_2^2=n_k$. While standardization does not change the $\ell_0$-norm of $\beta$ or $w^{(k)}$, it can change the $\ell_0$-norm of $\delta^{(k)}$. Hence, this work focuses on $\ell_1$-sparse contrasts, which is more practical in applications.

\section{Simulation Studies}
\label{sec-simu}
In this section, we evaluate the empirical performance of our proposals and some other comparable methods in various numerical experiments. Specifically, we evaluate the performance of four methods: the original Lasso, the Oracle Trans-Lasso proposed in Section \ref{sec2-1}, the Trans-Lasso proposed in Section \ref{sec3-1}, and a naive Trans-Lasso method, which naively assumes $\mathcal{A}=\{1,\dots, K\}$ in the Oracle Trans-Lasso. The purpose of including the naive Trans-Lasso is to understand the overall informative level of the auxiliary samples. In the Appendix, we report the performance of the estimated sparse indices $\widehat{R}^{(k)}$ in achieving (\ref{cond-agg2}).

\subsection{Identity Covariance Matrix for the Designs}
\label{sec5-1}
We consider $p=500$, $n_0=150$, and  $n_1,\dots,n_K=100$ for $K=20$. The covariates $x^{(k)}_i$ are \textit{i.i.d.} Gaussian with mean zero and identity covariance matrix for all $0\leq k\leq K$ and $\eps^{(k)}_i$ are \textit{i.i.d.} Gaussian with mean zero and variance one for all $0\leq k\leq K$. For the target parameter $\beta$, we set $s=16$, $\beta_{j}=0.3$ for $j\in \{1,\dots,s\}$, and $\beta_j=0$ otherwise. For the regression coefficients in auxiliary samples, we consider two configurations.

(i) Let
\begin{align*}
   w^{(k)}_{j}=\beta_{j}-0.3\mathbbm{1}(j\in H_k).
   \end{align*}
For a given $\mathcal{A}$, if $k \in \mathcal{A}$, we set $H_k$ to be a random subset of $[p]$ with $|H_k|=h\in\{2,6,12\}$ and if $k\notin \mathcal{A}$, we set $H_k$ to be a random subset of $[p]$ with $|H_k|=50$.

(ii)
For a given $\mathcal{A}$, if $k \in \mathcal{A}$, let $H_k$ be a random subset of $[p]$ with $|H_k|=p/2$ and let
\[
   w^{(k)}_{j}=\beta_{j}+\xi_j\mathbbm{1}(k\in H_k), ~\text{where}~ \xi_j\sim_{i.i.d.} \text{Laplace}(0, 2h/p),
\]
   where $h\in\{2,6,12\}$ and $\text{Laplace}(a, b)$ is Laplacian distribution with mean $a$ and dispersion $b$.
If $k\notin \mathcal{A}$, we set $H_k$ to be a random subset of $[p]$ with $|H_k|=p/2$ and let
\[
    w^{(k)}_j=\beta_j+\xi_j\mathbbm{1}(j\in H_k),~\text{where}~\xi_j\sim_{i.i.d.} \text{Laplace}(0, 100/p).
\]
The setting (i) can be treated as either $\ell_0$- or $\ell_1$-sparse contrasts.
In practice, the true parameters are unknown and we use $\mathcal{A}$ to denote the set of auxiliary samples without distinguishing $\ell_0$- or $\ell_1$-sparsity. We consider $|\mathcal{A}|\in\{0,4,8,\dots,20\}$.

To avoid searching for tuning parameters, we use the raw data other than the standardized data.
 For the Lasso method, the tuning parameter is chosen to be $\sqrt{2\log p/n_0}$. For the Oracle Trans-Lasso, we set $\lambda_w=\sqrt{2\log p/(n_0+n_{\mathcal{A}})}$ and $\lambda_{\delta}=\sqrt{2\log p/n_0}$. The naive Trans-Lasso is computed based on the Oracle Trans-Lasso with $\mathcal{A}=\{1,\dots, K\}$. For the Trans-Lasso,  we set $\mathcal{I}=\{1,\dots,n_0/2\}$ in Step 1. The sets $\widehat{G}_0,\dots,\widehat{G}_L$ are computed based on SURE screening with $t_*=n_0^{3/4}$. For the Q-aggregation, we implement Algorithm 2 (GD-BMAX) in \citet{Dai18}, which solves a dual representation of the Q-aggregation. Using their notations, we set $\omega=\sqrt{2}$, $\nu=0.5$, and $t_k=1$ in GD-BMAX. We mention that as for demonstration, the tuning parameters are not deliberately chosen. We then perform a cross-fitting by using the first half of the samples for aggregation and the other half for constructing the dictionary. Our final estimator is the average of these two Trans-Lasso estimators.

The sum of squared estimation errors (SSE) are reported in Figure \ref{fig1-simu}. As we expected, the performance of the Lasso does not change as $|\mathcal{A}|$ changes. On the other hand, all three other Trans-Lasso based algorithms have estimation errors decreasing as $|\mathcal{A}|$ increases.  As $h$ increases, the problem gets harder and the estimation errors of all three methods increase. In settings (i) and (ii), the performance of Oracle Trans-Lasso and Trans-Lasso are comparable in most occasions. When $h=12$ in setting (i), we see a relatively large gap between the SSEs of Trans-Lasso and Oracle Trans-Lasso. One main reason is that the proposed $\widehat{R}^{(k)}$ does not consistently separate informative auxiliary samples from others in this case (Table \ref{table-rank} in the Appendix). This is because, the sparsity levels of $\delta^{(k)}$ are similar for $k\in\mathcal{A}$ and $k\notin \mathcal{A}$. In other cases, the proposed $\widehat{R}^{(k)}$ can separate informative auxiliary samples from others reasonably well.
On the other hand, the naive Trans-Lasso has worse performance than the Lasso when $|\mathcal{A}|$ is relatively small. This shows that the scenarios under consideration are hard in the sense that some naive methods cannot adapt the unknown $\mathcal{A}$ uniformly.

\begin{figure}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Ident-i.eps}}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Ident-ii.eps}}
\caption{\label{fig1-simu}Estimation errors of the Lasso, Naive Trans-Lasso, Oracle Trans-Lasso, and Trans-Lasso for the settings with identity covariance matrices. The two rows correspond to configurations (i) and (ii), respectively. Each point is summarized from 200 independent simulations.}
\end{figure}

\subsection{Homogeneous Designs among $\mathcal{A}\cup \{0\}$}
\label{sec5-2}
In this subsection, we consider $x^{(k)}_i$ as \textit{i.i.d.} Gaussian with mean zero and a Toeplitz covariance matrix whose first row is
\[\Sigma^{(k)}_{1,.}=(1,0.8,\dots,0.8^K, 0_{p-K-1})
\]
for $k\in \mathcal{A}\cup\{0\}$. For $k\notin \mathcal{A}\cup\{0\}$, $x^{(k)}_i$ are \textit{i.i.d.} Gaussian with mean zero and a Toeplitz covariance matrix whose first row is
\begin{equation}
\label{eq-Sigk1}
\Sigma^{(k)}_{1,.}=(1,\underbrace{1/(k+1),\dots,1/(k+1)}_{2k-1}, 0_{p-2k}).
\end{equation}
 Other true parameters and the dimensions of the samples are set to be the same as in Section \ref{sec5-1}. From the results presented in Figure \ref{fig2-simu}, we see that the Trans-Lasso and Oracle Trans-Lasso have reliable performance when $\Sigma^{(k)}\neq \Sigma^{(k')}$ for $k\in\mathcal{A}\cup\{0\}$ and $k'\notin \mathcal{A}\cup\{0\}$. In Table \ref{table-rank} in the Appendix, we see that our proposed $\widehat{R}^{(k)}$ can separate informative auxiliary samples from others consistently in settings (i) and (ii).

 On the other hand, we can observe from Figure \ref{fig2-simu} that the SSE of the Trans-Lasso can be slightly below those of the Oracle Trans-Lasso when $0<|\mathcal{A}|< K$. There are two potential reasons. As a cross-fitting step is performed in the Trans-Lasso, the samples for computing the Trans-Lasso and for other methods are different empirically. Second, our definition of $\mathcal{A}$ may not always be the best subset of auxiliary samples that  give the smallest estimation errors.
\begin{figure}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Homo-i.eps}}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Homo-ii.eps}}
\caption{\label{fig2-simu}Estimation errors of the Lasso, Naive Trans-Lasso, Oracle Trans-Lasso, and Trans-Lasso for the settings with homogeneous covariance matrices among $k\in\mathcal{A}\cup\{0\}$. The two rows correspond to configurations (i) and (ii), respectively. Each point is summarized from 200 independent simulations.}
\end{figure}

\subsection{Heterogeneous Designs}
We now consider $x^{(k)}_i$ as \textit{i.i.d.} Gaussian with mean zero and a Toeplitz covariance matrix whose first row is (\ref{eq-Sigk1}) for $k=1,\dots, K$. Moreover, $\Sigma^{(0)}=I_p$. Other parameters and the dimensions of the samples are set to be the same as in Section \ref{sec5-1}.
Figure \ref{simu-fig4}  shows that the general patterns observed in previous subsections still hold. We observe a relatively large gap between the SSEs of the Oracle Trans-Lasso and Trans-Lasso in the scenario when $h=12$ in setting (i). Again, this is because $\widehat{R}^{(k)}$ can only separate a subset of $\mathcal{A}$ from $\mathcal{A}^c$. In other cases, the performance of Trans-Lasso is comparable to the Oracle Trans-Lasso.
\begin{figure}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Hetero-i.eps}}
\makebox{\includegraphics[width=0.98\textwidth, height=4.5cm]{Hetero-ii.eps}}
\caption{\label{simu-fig4}Estimation errors of the Lasso, Naive Trans-Lasso, Oracle Trans-Lasso, and Trans-Lasso for the settings with heterogeneous covariance matrices. The two rows correspond to configurations (i) and (ii), respectively. Each point is summarized from 200 independent simulations.}
\end{figure}

\section{Application to Genotype-Tissue Expression Data}
\label{sec-data}
In this section, we demonstrate  the performance of our proposed transfer learning algorithm in analyzing the Genotype-Tissue Expression (GTEx) data (\url{https://gtexportal.org/}). Overall, the data sets  measure  gene expression levels from 49 tissues of 838 human donors, in total comprising 1,207,976 observations of 38,187 genes.  In our analysis, we focus on genes that are related to central nervous systems, which were assembled as MODULE\underline{ }137 ( \url{https://www.gsea-msigdb.org/gsea/msigdb/cards/MODULE_137.html}).
This module includes a total of  545 genes  and additional 1,632 genes that  are significantly enriched in the same experiments as the genes of the module.  A   complete list of genes can be found at \url{http://robotics.stanford.edu/~erans/cancer/modules/module_137}.

\subsection{Data Analysis Method}
To demonstrate the replicability of our proposal, we consider multiple target genes and multiple target tissues and estimate their corresponding models one by one.
For an illustration of the computation process,  we  consider gene \texttt{JAM2} (Junctional adhesion molecule B), as the response variable, and treat other genes in this module as covariates.  \texttt{JAM2} is a protein coding gene on chromosome 21 interacting with a variety of immune cell types and may play a role in lymphocyte homing to secondary lymphoid organs \citep{JAM2}.  It is of biological interest to understand how other CNS genes  can predict its expression levels in different tissues/cell types.

As an example, we consider the association between \texttt{JAM2} and other genes in a brain tissue as the target models and the association between \texttt{JAM2} and other genes in other tissues as the auxiliary models. As there are multiple brain tissues in the dataset, we treat each of them as the target at each time. The list of target tissues can be found in Figure \ref{fig-data-1}. The min, average, and max of primary sample sizes in these target tissues are 126, 177, and 237, respectively. The gene \texttt{JAM2} is expressed in 49 tissues in our dataset and we use 47 tissues  with more than 120 measurements on \texttt{JAM2}. The average number of auxiliary samples for each target model is 14,837 over all the non-target tissues. The covariates used are the  genes that are in the enriched MODULE\underline{ }137 and do not have missing values in all of the 47 tissues. The final covariates include a total of  1,079 genes. The data is standardized before analysis.

 We compare the prediction performance of Trans-Lasso with the Lasso. To understand the overall informative level of the auxiliary samples, we also compute the Naive Trans-Lasso which treats all the auxiliary samples are informative. For evaluation, we split the target sample into 5 folds and use 4 folds to train the three algorithms and use the remaining fold to test their prediction performance. We repeat this process 5 times each with a different fold of test samples.

We mention that one individual can provide expression measurements on multiple tissues and these measurements are hard to be independent. As the dependence of the samples can reduce the efficiency of the estimation algorithms, using auxiliary samples may still be beneficial. However, one need to choose proper tuning parameters. The tuning parameter for the Lasso and $\lambda_w$ in the Naive-Trans-Lasso are chosen by 8-fold cross validation. The $\lambda_{\delta}$ in the Naive Trans-Lasso is set to be $\lambda_w\sqrt{\sum_{k=0}^K n_k/n_0}$. For our proposal Trans-Lasso, we use two-thirds of the training sample to construct the dictionary and use one-third of the training sample for aggregation. The sparsity indices are computed in the same way as in our simulations. For computing each $\hat{\beta}(\widehat{G}_l)$, the $\lambda_w$ is chosen by 8-fold cross validation and $\lambda_{\delta}$ is set to be the corresponding $\lambda_w\sqrt{\sum_{k\in \widehat{G}_l} n_k/n_0}$. The tuning parameters in aggregation are chosen as in our simulation.

\subsection{Prediction Performance of the Trans-Lasso for \texttt{JAM2} Expression}
Figure \ref{fig-data-1} demonstrates the errors of Naive Trans-Lasso, and Trans-Lasso relative to the Lasso for predicting gene expression \texttt{JAM2} using other genes. The prediction errors in the raw scale are provided in the Appendix. We see that the Trans-Lasso algorithm always achieves the smallest prediction errors across different tissues. Its average gain is 17\% comparing to the Lasso. This shows that our characterization of the similarity of the target model and a given auxiliary model is suitable to the current problem and our proposed sparsity index for aggregation is effective in detecting good auxiliary samples. In tissues such as Amygdala and Nucleus accumbens basal ganglia, the Trans-Lasso achieves relatively significant improvement and it has more accurate prediction than the Naive Trans-Lasso. This implies that the knowledge from the auxiliary tissues have been successfully transferred into these target tissues for modeling \texttt{JAM2} even if not all the tissues are informative. In tissues such as Pituitary, the improvement of the Naive Trans-Lasso and Trans-Lasso are mild. This implies that the regression model for \texttt{JAM2} in Pituitary is relatively distinct from the models in other tissues so that little knowledge can be transferred. Moreover, in tissues such as Frontal Cortex, the prediction performance of naive Trans-Lasso can be worse than the Lasso whereas the Trans-Lasso still has the smallest prediction error. This again demonstrates the robustness of our proposal.

 \begin{figure}[H]
 \centering
 \includegraphics[height=6cm, width=0.95\textwidth]{JAM2-pred2.eps}
 \caption{ \label{fig-data-1}Prediction errors of the naive Trans-Lasso and Trans-Lasso relative to the Lasso evaluated via 5-fold cross validation for gene \texttt{JAM2} in multiple tissues.}
 \end{figure}

 \subsection{Prediction Performance of Other 25 Genes on Chromsome 21}

To demonstrate the replicability of our proposal, we also consider other genes on Chromosome 21 which are in Module\underline{ }137 as our target genes. We report the overall prediction performance of these 25 genes in Figure \ref{fig-data-2}. A complete list of these genes and some summary information can be found in the Appendix. Generally speaking, we see that the Trans-Lasso has the best overall performance among all target tissues. Specifically, the improvement of Trans-Lasso is significant in tissues including Cerebellar Hemisphere, Cortex, and Frontal Cortex. The naive Trans-Lasso is has comparable accuracy to the Lasso in most cases, which implies that the overall similarity between the auxiliary tissues and target tissues is not very strong.

 \begin{figure}
 \centering
 \makebox{\includegraphics[height=6cm, width=0.99\textwidth]{Chr21-pred2.eps}}
 \caption{ \label{fig-data-2}Prediction errors of the naive Trans-Lasso and Trans-Lasso relative to the Lasso for the 25 genes on Chromosome 21 and in Module\underline{ }137, in multiple target tissues.}

 \end{figure}

\section{Discussion}
\label{sec:discussion}

This paper studies high-dimensional linear regression in the presence of additional auxiliary samples, where the similarity of the target model and a given auxiliary model is characterized by  the sparsity of their contrast vector.  Transfer learning algorithms for estimation and prediction are developed. The results show that if the informative set is known, the proposed Oracle Trans-Lasso is minimax optimal over a range of parameter spaces and the accuracy for estimation and prediction can be improved.
It has been considered in \citet{Bastani18} the setting of known informative set with one large-scale auxiliary study, which is a special case of our problem set-up. However, their upper bound analysis is not minimax rate optimal in the parameter space considered in this work.
Adaptation to the unknown informative set is also considered. It is shown that adaptation can be achieved by aggregating a collection of candidate estimators. Numerical experiments and real data applications support the theoretical findings.

Transfer learning for high-dimensional linear regression  is an important problem with a wide range of potential applications. However, statistical theory and methods have not been well developed in the literature. Using our similarity characterization of the auxiliary studies, it is also interesting to study statistical inference such as constructing confidence intervals and hypothesis testing for high-dimensional linear regression with auxiliary samples.
In view of the results derived in this paper, one may expect weaker sample size conditions in the transfer learning setting than the conventional case. It is interesting to provide a precise characterization and develop a minimax optimal confidence interval in the transfer learning setting. On the other hand, different measurements of the similarity can be used when they are appropriate, which can lead to new methods and insights into the underlying structure of the transfer learning algorithm.

\section*{Acknowledgements}
This research was supported by NIH grants GM129781 and GM123056 and NSF Grant DMS-1712735.