\subsubsection*{Abstract}
In this paper we obtain an energy estimate for a complete strictly hyperbolic operator with second order coefficients satisfying a
log-Zygmund-continuity condition with respect to $t$, uniformly with respect to $x$, and a log-Lipschitz-continuity condition with respect to $x$,
uniformly with respect to $t$.

\subsubsection*{Keywords}
Hyperbolic equations, log-Zygmund regularity, loss of derivatives.

\subsubsection*{MS Classification 2000}
35L15

\section{Introduction}
Let us consider the second order operator
\begin{equation} \label{eq:P}
 P\,=\,\partial^2_t\,-\,\sum_{i,j=1}^n\partial_{x_i}(a_{ij}(t)\partial_{x_j})
\end{equation}
and suppose that $P$ is strictly hyperbolic, i.e. there exist two positive constants $\lambda_0\leq\Lambda_0$ such that
\begin{equation} \label{eq:P_elliptic}
 \lambda_0\,|\xi|^2\,\leq\,\sum_{i,j=1}^n a_{ij}(t)\,\xi_i\xi_j\,\leq\,\Lambda_0\,|\xi|^2
\end{equation}
for all $t\in\mathbb{R}$ and all $\xi\in\mathbb{R}^n$.

It is well-known (see e.g.~\cite{Horm} and~\cite{Treves}) that, if the coefficients $a_{ij}$ are Lipschitz-continuous, then the following energy
estimate holds for the operator $P$: for all $s\in\mathbb{R}$, there exists $C_s>0$ such that
\begin{eqnarray} \label{eq:en-estim_t_Lip}
& & \sup_{t\in[0,T]}\left(\|u(t,\cdot)\|_{H^{s+1}}\,+\,\|\partial_tu(t,\cdot)\|_{H^s}\right)\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C_s\,\left(\|u(0,\cdot)\|_{H^{s+1}}\,+\,\|\partial_tu(0,\cdot)\|_{H^s}\,+\,
\int^T_0\|Pu(t,\cdot)\|_{H^s}\,dt\right) \nonumber
\end{eqnarray}
for every function $u\in\mathcal{C}^2([0,T];H^\infty(\mathbb{R}^n))$.\\
In particular, the previous energy estimate implies that the Cauchy problem for (\ref{eq:P}) is well-posed in the space $H^\infty$, with no loss
of derivatives.

On the contrary, if the coefficients $a_{ij}$ are not Lipschitz-continuous, then (\ref{eq:en-estim_t_Lip}) is no more true in general, as it is shown
by an example given by Colombini, De Giorgi and Spagnolo in the paper~\cite{Colomb-DeG-S}. Nevertheless,
under suitable weaker regularity assumptions on the coefficients, one can recover the $H^\infty$-well-posedness again, but this time
from an energy estimate with loss of derivatives.

A first result of this type was obtained in the quoted paper~\cite{Colomb-DeG-S}. The authors supposed that there
was a constant $C>0$ such that, for all $\varepsilon\in]0,T]$,
\begin{equation} \label{eq:cont_t_Log-Lip_int}
 \int^{T-\varepsilon}_0|a_{ij}(t+\varepsilon)-a_{ij}(t)|\,dt\,\leq\,C\,\varepsilon\,\log\left(1+\frac{1}{\varepsilon}\right)\,.
\end{equation}
The Fourier trasform with respect to $x$ of the equation, togheter with the new ``approximate energy technique''
(i.e. the approximation of the coefficients is different in different zones of the phase space), enabled them to obtain the following energy estimate:
there exist strictly positive constants $K$ (indipendent of $s$) and $C_s$ such that
\begin{eqnarray} \label{eq:en-estim_t_Log-Lip}
& & \sup_{t\in[0,T]}\left(\|u(t,\cdot)\|_{H^{s+1-K}}\,+\,\|\partial_tu(t,\cdot)\|_{H^{s-K}}\right)\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C_s\,\left(\|u(0,\cdot)\|_{H^{s+1}}\,+\,\|\partial_tu(0,\cdot)\|_{H^s}\,+\,
\int^T_0\|Pu(t,\cdot)\|_{H^s}\,dt\right) \nonumber
\end{eqnarray}
for all $u\in\mathcal{C}^2([0,T];H^\infty(\mathbb{R}^n))$.

Considering again the case that the coefficients of $P$ depend only on the time variable, in the recent paper~\cite{Tarama}
(see also~\cite{Yama}) Tarama has weakened the regularity hypotesis further, supposing a log-Zygmund type integral condition, i.e.
that there exists a constant $C>0$ such that, for all $\varepsilon\in]0,T/2]$,
\begin{equation} \label{eq:cont_t_Log-Zyg_int}
 \int^{T-\varepsilon}_{\varepsilon}|a_{ij}(t+\varepsilon)+a_{ij}(t-\varepsilon)-2a_{ij}(t)|\,dt\,\leq\,C\,\varepsilon\,\log\left(1+\frac{1}{\varepsilon}\right)\,.
\end{equation}
Nevertheless, he has been able to prove the well-posedness to the Cauchy problem for (\ref{eq:P}) in the space $H^\infty$: the improvement with respect
to~\cite{Colomb-DeG-S} was obtained introducing a new type of approximate energy, which involves the second derivatives
of the approximating coefficients.

Much more difficulties arise if the operator $P$ has coefficients depending both on the time variable $t$ and on the space variables $x$. This
case was considered by Colombini and Lerner in the paper~\cite{Colomb-Lern}. They supposed a pointwise log-Lipschitz regularity condition, i.e.
that there exists $C>0$ such that, for all $\varepsilon\in]0,T]$,
\begin{equation} \label{eq:cont_t-x_Log-Lip_punt}
 \sup_{\begin{array}{c}
        \scriptstyle y,z\in[0,T]\times\mathbb{R}^n \\
	\scriptstyle |z|=\varepsilon
       \end{array}} |a_{ij}(y+z)-a_{ij}(y)|\,\leq\,C\,\varepsilon\,\log\left(1+\frac{1}{\varepsilon}\right)\,.
\end{equation}
Because the coefficients of the operator $P$ depend also on the space variables, here the Littlewood-Paley dyadic decomposition with respect
to $x$ takes the place of the Fourier trasform, and it is, togheter with the approximate energy technique, the key tool to obtain the energy estimate:
for all fixed $\theta\in]0,1/4]$, there exist $\beta$, $C>0$ and $T^*\in]0,T]$ such that
\begin{eqnarray} \label{eq:en-estim_t-x_Log-Lip}
& & \sup_{t\in[0,T^*]}\left(\|u(t,\cdot)\|_{H^{-\theta+1-\beta t}}\,+\,\|\partial_tu(t,\cdot)\|_{H^{-\theta-\beta t}}\right)\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C\,\left(\|u(0,\cdot)\|_{H^{-\theta+1}}\,+\,\|\partial_tu(0,\cdot)\|_{H^{-\theta}}\,+\,
\int^T_0\|Pu(t,\cdot)\|_{H^{-\theta-\beta t}}\,dt\right) \nonumber
\end{eqnarray}
for all $u\in\mathcal{C}^2([0,T^*];H^\infty(\mathbb{R}^n))$.\\
In this case, the loss of derivatives gets worse with the increasing of time.

In a recent paper (\hspace{-0.1cm}~\cite{Colomb-DelSant_2}), Colombini and Del Santo considered the case of one space variable (i.e. $n=1$) and
studied again the case of the coefficient $a$ depending both on $t$ and $x$, but under a special regularity condition: they mixed condition
(\ref{eq:cont_t_Log-Zyg_int}) togheter with (\ref{eq:cont_t-x_Log-Lip_punt}). In particular, they supposed $a$ to be log-Zygmund-continuous
with respect to $t$, uniformly with respect to $x$, and log-Lipschitz-continuous with respect to $x$, uniformly with respect to $t$. The dyadic
decomposition technique and the Tarama's approximate energy enabled them to obtain an estimate similar to (\ref{eq:en-estim_t-x_Log-Lip}).

The reason why they focused on the case $n=1$ is that the case of several space variables needs some different and new ideas in the definition of the
microlocal energy: this point still remains as an open problem.

\bigskip
In the present note, we will consider the case of the non-homogeneous operator
\begin{equation} \label{eq:L}
Lu\,=\,\partial_{t}^{2}u\,-\,\partial_x(a(t,x)\partial_x u)\,+\,b_0(t,x)\partial_tu\,+\,b_1(t,x)\partial_x u\,+\,c(t,x)u\;,
\end{equation}
where the coefficient $a$ satisfy the same regularity assumptions as in~\cite{Colomb-DelSant_2}. We will also suppose that $b_0$,
$b_1\in L^\infty(\mathbb{R}_t;\mathcal{C}^{\omega}(\mathbb{R}_x))$, $\omega>0$, and $c$ bounded on $\mathbb{R}_t\times\mathbb{R}_x$.
We will apply the Littlewood-Paley decomposition and the Tarama's approximate energy again to obtain an energy estimate with a loss of
derivatives that depends on $t$, as in (\ref{eq:en-estim_t-x_Log-Lip}).

One can find the estimate of the second order coefficient $a$ in the paper~\cite{Colomb-DelSant_2}, however, for reader's convenience,
we will give here all the details.

\section{Main result}
Let $a:\mathbb{R}^2\rightarrow\mathbb{R}$ be a function such that, for positive constants $\lambda_0\leq\Lambda_0$ and $C_0$, one has, for all
$(t,x)\in\mathbb{R}^2$ and all $\tau>0$, $y>0$,
\begin{eqnarray}
 & \lambda_0\leq a(t,x)\leq\Lambda_0   \label{(N)eq_condiz_1.1} \\
 & \sup_{(t,x)}|a(t+\tau,x)+a(t-\tau,x)-2a(t,x)|\leq C_0\,\tau\log\left(\frac{1}{\tau}+1\right) \label{(N)eq_condiz_1.2} \\
 & \sup_{(t,x)}|a(t,x+y)-a(t,x)|\leq C_0\,y\log\left(\frac{1}{y}+1\right) \label{(N)eq_condiz_1.3}
\end{eqnarray}
Moreover, let
\begin{equation} \label{(N)eq_condiz_B}
b_0\,,\,b_1\;\in\,L^\infty(\mathbb{R}_t;\mathcal{C}^{\omega}(\mathbb{R}_x))\,,
\end{equation}
where $\omega>0$, and
\begin{equation} \label{(N)eq_condiz_c}
c\;\in\,L^{\infty}(\mathbb{R}_t\times\mathbb{R}_x)\;.
\end{equation}

\begin{teorema} \label{(N)teor_teorema1}
Let us consider, on the whole space $\mathbb{R}^2$, the operator
\begin{equation} \label{(N)operatore}
Lu\,=\,\partial_{t}^{2}u\,-\,\partial_x(a(t,x)\partial_x u)\,+\,b_0(t,x)\partial_tu\,+\,b_1(t,x)\partial_x u\,+\,c(t,x)u\;,
\end{equation}
where the coefficients $a$, $b_0$, $b_1$ and $c$ satisfy the hypothesis (\ref{(N)eq_condiz_1.1})-(\ref{(N)eq_condiz_c}).\\
Then, for all fixed
$$
\theta\,\in\,\left]0,\min\left\{\frac{1}{2},\frac{\omega}{1+\log2}\right\}\right[\;,
$$
there exist $\beta^*>0$, a time $T\in\mathbb{R}$ and a constant $C>0$ such that
\begin{eqnarray} \label{(N)eq_tesi}
 & & \sup_{[0,T]}\left(\|u(t,\cdot)\|_{H^{1-\theta-\beta^*t}}+
\|\partial_t u(t,\cdot)\|_{H^{-\theta-\beta^*t}}\right)\,\leq \\
 & & \qquad\qquad\qquad\qquad\leq\,C\,\left(\|u(0,\cdot)\|_{H^{1-\theta}}+
\|\partial_t u(0,\cdot)\|_{H^{-\theta}}+\int_{0}^{T}\|Lu(t,\cdot)\|_{H^{-\theta-\beta^*t}}dt\right) \nonumber
\end{eqnarray}
for all $u\in\mathcal{C}^2([0,T];H^\infty(\mathbb{R}_x))$.
\end{teorema}

\section{Proof of Theorem \ref{(N)teor_teorema1}}
\subsection{Approximation of the coefficient $a(t,x)$}
Let $\rho\in\mathcal{C}_{0}^{\infty}(\mathbb{R})$ be an even function such that:
\begin{enumerate}
 \item $0\leq\rho\leq1$
 \item $supp\,\rho\,\subset\,[-1,1]$
 \item $\int\rho(s)ds=1$
 \item $|\rho'(s)|\leq2$;
\end{enumerate}
for all $0<\varepsilon\leq1$, we set $\rho_{\varepsilon}(s)=\frac{1}{\varepsilon}\,\rho\left(\frac{s}{\varepsilon}\right)$.

Then, for all $0<\varepsilon\leq1$, we define
\begin{equation} \label{(N)eq_a_eps}
a_\varepsilon(t,x)\,:=\,\int_{\mathbb{R}_t\times\mathbb{R}_x}\rho_\varepsilon(t-s)\,\rho_\varepsilon(x-y)\,a(s,y)\,dsdy\;.
\end{equation}

\begin{lemma} \label{(N)lemma_approx}
The following inequalities hold true:
\begin{enumerate}
 \item for all $\varepsilon\in]0,1]$, for all $(t,x)\in\mathbb{R}^2$, one has
\begin{equation} \label{(N)eq_condiz_1.1_eps}
\lambda_0\,\leq\,a_\varepsilon(t,x)\,\leq\,\Lambda_0\;;
\end{equation}
 \item for all $\varepsilon\in]0,1]$, one has
\begin{equation} \label{(N)eq_a_eps-a}
\sup_{(t,x)}|a_\varepsilon(t,x)-a(t,x)|\,\leq\,
\frac{3}{2}C_0\,\varepsilon\,\log\left(\frac{1}{\varepsilon}+1\right)\;;
\end{equation}
 \item for all $\sigma\in]0,1[$, a constant $c_\sigma>0$ exists such that, for all $\varepsilon\in]0,1]$,
\begin{equation} \label{(N)eq_d_t_a_eps}
\sup_{(t,x)}|\partial_t a_\varepsilon(t,x)|\,\leq\,c_\sigma (\Lambda_0+C_0)\,\varepsilon^{\sigma-1}\;;
\end{equation}
 \item for all $\varepsilon\in]0,1]$, one has
\begin{eqnarray}
\sup_{(t,x)}|\partial_x a_\varepsilon(t,x)| & \leq &
C_0\|\rho'\|_{L^1}\log\left(\frac{1}{\varepsilon}+1\right) \label{(N)eq_d_x_a_eps} \\
\sup_{(t,x)}\left|\partial_{t}^{2} a_\varepsilon(t,x)\right| & \leq &
\frac{C_0}{2}\|\rho''\|_{L^1}\frac{1}{\varepsilon}\log\left(\frac{1}{\varepsilon}+1\right) \label{(N)eq_d_t^2_a_eps} \\
\sup_{(t,x)}|\partial_t\partial_x a_\varepsilon(t,x)| & \leq &
C_0\|\rho'\|_{L^1}^{2}\frac{1}{\varepsilon}\log\left(\frac{1}{\varepsilon}+1\right) \label{(N)eq_d_t_d_x_a_eps}\;.
\end{eqnarray}
\end{enumerate}
\end{lemma}

\begin{proof}
Inequalities in (\ref{(N)eq_condiz_1.1_eps}) immediately follow from the fact that $|\rho|\leq1$.

Relation (\ref{(N)eq_a_eps-a}), instead, follows from (\ref{(N)eq_condiz_1.2}), after one has observed that
$$
a_\varepsilon(t,x)-a(t,x)=\frac{1}{2}\int_{\mathbb{R}_t}\rho_\varepsilon(s)\int_{\mathbb{R}_y}\rho_\varepsilon(x-y)(a(t+s,y)+a(t-s,y)-2a(t,y))\,dy\,ds\;,
$$
where we have used the fact that $\rho$ is an even function.

Moreover, one has
$$
\partial_t^2a_\varepsilon(t,x)=\frac{1}{2}\int\rho_\varepsilon''(s)\int\rho_\varepsilon(x-y)(a(t+s,y)+a(t-s,y)-2a(t,y))\,dy\,ds\;,
$$
from which one can deduce (\ref{(N)eq_d_t^2_a_eps}).

Inequalities (\ref{(N)eq_d_x_a_eps}) and (\ref{(N)eq_d_t_d_x_a_eps}) derive from (\ref{(N)eq_condiz_1.3}) in a very similar way.

Finally, relation (\ref{(N)eq_d_t_a_eps}) is a consequence of the fact that (\ref{(N)eq_condiz_1.1}) and (\ref{(N)eq_condiz_1.2}) imply that
for all $\sigma\in]0,1[$, a constant $c_\sigma'>0$ exists such that, for all $\tau>0$, one has
\begin{equation} \label{(N)eq_2.6}
\sup_{(t,x)}|a(t+\tau,x)-a(t,x)|\,\leq\,c_\sigma'\,(\Lambda_0+C_0)\,\tau^\sigma\;.
\end{equation}
\end{proof}

\subsection{Littlewood-Paley decomposition} \label{sub-sec:L-P_dec}
We collect here some well-known facts about dyadic decomposition, referring to~\cite{Bony},~\cite{Colomb-Lern} and~\cite{Metivier} for the details.

Let $\varphi_0\in\mathcal{C}_{0}^{\infty}(\mathbb{R}_\xi)$ be an even function, decreasing on $[0,+\infty[$, such that $0\leq\varphi_0\leq1$ and
$$
\varphi_0(\xi)=1\,\mbox{ if }\,|\xi|\leq1\quad\mbox{,}\quad\varphi_0(\xi)=0\,\mbox{ if }\,|\xi|\geq2.
$$
We set $\varphi(\xi)\,=\,\varphi_0(\xi)-\varphi_0(2\xi)$ and, for $\nu\in\mathbb{N}\backslash\{0\}$, $\varphi_{\nu}(\xi):=\varphi(2^{-\nu}\xi)$.

For a tempered distribution $u\in H^{-\infty}(\mathbb{R})$, we define
$$
u_\nu(x)\,:=\,\varphi_\nu(D_x)u(x)\,=\,\frac{1}{2\pi}\int e^{ix\xi}\varphi_\nu(\xi)\,\widehat{u}(\xi)\,d\xi\,=\,
\frac{1}{2\pi}\int\widehat{\varphi_\nu}(y)u(x-y)\,dy\,;
$$
for all $\nu$, $u_\nu$ is an entire analytic function belonging to $L^2$.\\
Moreover, for all $s\in\mathbb{R}$ there exists a constant $C_s>0$ such that
\begin{equation} \label{eq:dyadic-dec_Sobolev-norm}
 \frac{1}{C_s}\,\sum^{+\infty}_{\nu=0}2^{2\nu s}\|u_\nu\|^2_{L^2}\,\leq\,\|u\|^2_{H^s}\,\leq\,
C_s\,\sum^{+\infty}_{\nu=0}2^{2\nu s}\|u_\nu\|^2_{L^2}
\end{equation}
and the following inequalities (called ``Bernstein's inequalities'') hold:
\begin{eqnarray}
 \|\partial_x u_\nu\|_{L^2} & \leq & 2^{\nu+1}\,\|u_\nu\|_{L^2}\qquad\qquad\;\mbox{  for all } \nu\geq0 \label{eq:Bern-ineq_deriv<} \\
 \|u_\nu\|_{L^2} & \leq & 2^{-(\nu-1)}\,\|\partial_x u_\nu\|_{L^2}\qquad\mbox{ for all } \nu\geq1 \label{eq:Bern-ineq_<deriv}\,.
\end{eqnarray}

We end this subsection quoting a result which will be useful in the following; for its proof, see~\cite{Colomb-Lern}.\\
We denote with $[A,B]$ the commutator between two linear operators $A$ and $B$ and with $\mathcal{L}(L^2)$ the space of bounded linear operators
from $L^2$ to $L^2$.
\begin{lemma} \label{lemma:comm}
 \begin{enumerate}
  \item There exist $C>0$, $\nu_0\in\mathbb{N}$ such that, for all $a\in L^\infty(\mathbb{R})$ satisfying
$$
\sup_{x\in\mathbb{R}}|a(x+y)-a(x)|\,\leq\,C_0\,y\,\log\left(1+\frac{1}{y}\right)
$$
for all $y>0$, one has, for all $\nu\geq\nu_0$,
\begin{equation} \label{eq:comm_Log-Lip}
 \|\,[\varphi_\nu(D_x),a(x)]\,\|_{\mathcal{L}(L^2)}\,\leq\,C\,(\|a\|_{L^\infty}+C_0)\,2^{-\nu}\,\nu\,.
\end{equation}
 \item There exist $C>0$, $\nu_0\in\mathbb{N}$ such that, for all $b\in\mathcal{C}^\omega(\mathbb{R})$ and all $\nu\geq\nu_0$, one has
\begin{equation} \label{eq:comm_Holder}
 \|\,[\varphi_\nu(D_x),b(x)]\,\|_{\mathcal{L}(L^2)}\,\leq\,C\,\|b\|_{\mathcal{C}^\omega}\,2^{-\nu}\,.
\end{equation}
 \end{enumerate}
\end{lemma}

\subsection{Approximate and total energy}
Let $T_0>0$ and $u\in\mathcal{C}^2([0,T_0];H^\infty(\mathbb{R}_x))$. If we set $u_\nu(t,x)=\varphi_\nu(D_x)u(t,x)$, we obtain
\begin{eqnarray} \label{(N)eq_ni}
(Lu)_{\nu} & = & \partial_{t}^{2}u_{\nu}\,-\,\partial_x(a(t,x)\partial_x u_{\nu}) - \partial_x([\varphi_{\nu}(D_x),a]\partial_x u) \\
           & + &  b_0(t,x)\partial_t u_\nu + [\varphi_\nu(D_x),b_0]\partial_t u\,+\,
b_1(t,x)\partial_x u_\nu + [\varphi_\nu(D_x),b_1]\partial_x u \nonumber\\
           & + &  c(t,x)u_\nu + [\varphi_\nu(D_x),c]u \nonumber\;.
\end{eqnarray}

Now, we introduce the approximate energy of $u_\nu$ (see~\cite{Colomb-DelSant_2} and~\cite{Tarama}), setting
\begin{equation} \label{(N)eq_e_nu-eps}
e_{\nu,\varepsilon}(t):=\int_{\mathbb{R}}\left(\frac{1}{\sqrt{a_\varepsilon}}\left|\partial_t u_\nu+
\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right|^2+\sqrt{a_\varepsilon}|\partial_x u_\nu|^2+|u_\nu|^2\right)\,dx\;,
\end{equation}
and, taken $\theta$ as in the hypothesis of Theorem \ref{(N)teor_teorema1}, we define the total energy of $u$:
\begin{equation} \label{(N)eq_E_tot}
E(t)\,:=\,\sum_{\nu=0}^{+\infty}e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t)\;,
\end{equation}
where $\beta>0$ will be fixed later on.

\begin{oss} \label{remark:E(0)-E(t)}
From (\ref{eq:dyadic-dec_Sobolev-norm}) and Bernstein's inequalities (\ref{eq:Bern-ineq_deriv<})-(\ref{eq:Bern-ineq_<deriv}), it's easy to see that
there exist two positive constants $C_\theta$ and $C_\theta'$ such that
\begin{eqnarray*}
E(0) & \leq & C_\theta\,(\|\partial_tu(0,\cdot)\|_{H^{-\theta}}\,+\,\|u(0,\cdot)\|_{H^{1-\theta}}) \\
E(t) & \geq & C_\theta'\,(\|\partial_tu(t,\cdot)\|_{H^{-\theta-\beta^*t}}\,+\,
\|u(t,\cdot)\|_{H^{1-\theta-\beta^*t}})
\end{eqnarray*}
where we have set $\beta^*=\beta(\log2)^{-1}$.
\end{oss}

\medskip
First, we derive $e_{\nu,\varepsilon}$, defined by (\ref{(N)eq_e_nu-eps}), with respect to the time variable, and we obtain
\begin{eqnarray*}
\frac{d}{dt}e_{\nu,\varepsilon}(t) & = & \int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(\partial_t^2u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \\
 & + & \int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(R_\varepsilon u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \\
 & + & \int\partial_t\sqrt{a_\varepsilon}\,\,|\partial_xu_\nu|^2\,dx \\
 & + & \int2\sqrt{a_\varepsilon}\,\,\Re(\partial_xu_\nu\,\cdot\,\overline{\partial_x\partial_tu_\nu})\,dx \\
 & + & \int2\,\,\Re(u_\nu\,\cdot\,\overline{\partial_tu_\nu})\,dx\;,
\end{eqnarray*}
where $R_\varepsilon v\,=\,\partial_t\left(\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}\right)v-
\left(\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}\right)^2v$.
Now, we can put in the previous relation the value of $\partial_t^2u_\nu$, given by (\ref{(N)eq_ni}).

Integrating by parts and taking advantage of the spectral localisation of $u_\nu$, we have
\begin{eqnarray*}
 & & \int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(\partial_x(a\partial_xu_\nu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx\,= \\
 & & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad=\,\int2\frac{\partial_x\sqrt{a_\varepsilon}}{a_\varepsilon}\,a\,\,\Re\left(\partial_xu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \\
 & & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad-\,\int\frac{\partial_t\sqrt{a_\varepsilon}}{a_\varepsilon}\,a\,|\partial_xu_\nu|^2\,dx \\
 & & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad-\,\int2\frac{a}{\sqrt{a_\varepsilon}}\,\,\Re\left(\partial_xu_\nu\,\cdot\,
\overline{\partial_x\partial_tu_\nu}\right)\,dx \\
 & & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad-\,\int\frac{a}{\sqrt{a_\varepsilon}}\,\,
\partial_x\left(\frac{\partial_t\sqrt{a_\varepsilon}}{\sqrt{a_\varepsilon}}\right)\,
\Re(\partial_xu_\nu\,\cdot\,\overline{u_\nu})\,dx\,;
\end{eqnarray*}
taken care of the fact that $b_0$, $b_1$ and $c$ are real-valued, finally we obtain the complete expression for the time derivative of the approximate
energy:
\begin{eqnarray} \label{(N)eq_dt_e_nu,eps}
 & & \frac{d}{dt}e_{\nu,\varepsilon}(t)\,=\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re
\left((Lu)_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \\
 & & \qquad\qquad\quad\,+\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(R_\varepsilon u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,+\,\int\partial_t\sqrt{a_\varepsilon}\left(1-\frac{a}{a_\varepsilon}\right)|\partial_xu_\nu|^2\,dx \nonumber \\
 & & \quad\qquad\qquad\,+\,\int2\left(\sqrt{a_\varepsilon}-\frac{a}{\sqrt{a_\varepsilon}}\right)\Re(\partial_xu_\nu\,\cdot\,
\overline{\partial_x\partial_tu_\nu})\,dx \nonumber \\
 & & \qquad\qquad\quad\,+\,\int2\frac{\partial_x\sqrt{a_\varepsilon}}{a_\varepsilon}\,a\,\,\Re\left(\partial_xu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,
\int\frac{a}{\sqrt{a_\varepsilon}}\,\,\partial_x\left(\frac{\partial_t\sqrt{a_\varepsilon}}{\sqrt{a_\varepsilon}}\right)\,
\Re(\partial_xu_\nu\,\cdot\,\overline{u_\nu})\,dx \nonumber \\
 & & \quad\qquad\qquad\,+\,\int2\,\Re(u_\nu\,\cdot\,\overline{\partial_tu_\nu})\,dx \nonumber \\
 & & \quad\qquad\qquad\,+\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left((\partial_x([\varphi_\nu(D_x),a]\partial_xu))\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,b_0(t,x)\,\,\Re\left(\partial_tu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(([\varphi_\nu(D_x),b_0]\partial_tu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,b_1(t,x)\,\,\Re\left(\partial_xu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(([\varphi_\nu(D_x),b_1]\partial_xu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,c(t,x)\,\,\Re\left(u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber \\
 & & \qquad\qquad\quad\,-\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(([\varphi_\nu(D_x),c]u)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx \nonumber\;.
\end{eqnarray}

\subsection{Estimate for the approximate energy}
We want to obtain an energy estimate; so, let us start to control each term of (\ref{(N)eq_dt_e_nu,eps}).\\
Through the rest of the proof, we will denote with $C$, $C'$, $C''$ and $\widehat{C}$ constants depending only on $\lambda_0$, $\Lambda_0$, $C_0$ and on
the norms of the coefficients of the lower order terms of the operator $L$ in their respective functional spaces, and which are allowed to vary
from line to line.

\subsubsection{Terms with $a$ and $a_\varepsilon$}

Thanks to relations (\ref{(N)eq_condiz_1.1}), (\ref{(N)eq_d_t_a_eps}) with $\sigma=1/2$, (\ref{(N)eq_d_t^2_a_eps}) and Bernstein's inequalities,
we deduce that there exists $C>0$, depending only on $\lambda_0$, $\Lambda_0$ and $C_0$, such that, for all $\nu\in\mathbb{N}$,
$$
\left|\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re\left(R_\varepsilon u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}
\right)\,dx\right|\,\leq\,C\,\frac{1}{\varepsilon}\,\log\left(\frac{1}{\varepsilon}+1\right)\,
2^{-\nu}\,e_{\nu,\varepsilon}(t)\;.
$$

In the same way, from (\ref{(N)eq_condiz_1.1}), (\ref{(N)eq_a_eps-a}) and (\ref{(N)eq_d_t_a_eps}), we have
$$
\left|\int\partial_t\sqrt{a_\varepsilon}\left(1-\frac{a}{a_\varepsilon}\right)|\partial_xu_\nu|^2\,dx\right|\,\leq\,
C\,\log\left(\frac{1}{\varepsilon}+1\right)\,e_{\nu,\varepsilon}(t)\;,
$$
for a constant $C$ depending again only on $\lambda_0$, $\Lambda_0$ and $C_0$.

Moreover, again from (\ref{(N)eq_condiz_1.1}) and (\ref{(N)eq_a_eps-a}) and Bernstein's inequalities, we  obtain
\begin{eqnarray*}
\left|\int2\left(\sqrt{a_\varepsilon}-\frac{a}{\sqrt{a_\varepsilon}}\right)\Re(\partial_xu_\nu\,\cdot\,
\overline{\partial_x\partial_tu_\nu})\,dx\right| & \leq & C\,\varepsilon\,\log\left(\frac{1}{\varepsilon}+1\right)\,
\|\partial_xu_\nu\|_{L^2}\,\|\partial_x\partial_tu_\nu\|_{L^2} \\
 & \leq & C\,\varepsilon\,\log\left(\frac{1}{\varepsilon}+1\right)\,2^{\nu+1}\,
\|\partial_xu_\nu\|_{L^2}\,\|\partial_tu_\nu\|_{L^2}\;;
\end{eqnarray*}
but we have
$$
\|\partial_tu_\nu\|_{L^2}\,\leq\,\left\|\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu
\right\|_{L^2}\,+\,\left\|\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right\|_{L^2}
$$
and
\begin{eqnarray*}
\left\|\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_0\right\|_{L^2} & \leq & C\,\varepsilon^{-1/2}\|u_0\|_{L^2} \\
\left\|\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right\|_{L^2} & \leq & C\,\varepsilon^{-1/2}\,2^{-\nu}\|u_\nu\|_{L^2}\qquad(\nu\geq1)\;,
\end{eqnarray*}
that give us the following:
$$
\left|\int2\left(\sqrt{a_\varepsilon}-\frac{a}{\sqrt{a_\varepsilon}}\right)\Re(\partial_xu_\nu\cdot
\overline{\partial_x\partial_tu_\nu})dx\right|\,\leq\,
C(\varepsilon\,2^\nu+1)\log\left(\frac{1}{\varepsilon}+1\right)e_{\nu,\varepsilon}(t)\;.
$$

In a very similar way, from (\ref{(N)eq_d_x_a_eps}) one has
$$
\left|\int2\frac{\partial_x\sqrt{a_\varepsilon}}{a_\varepsilon}\,a\,\,\Re\left(\partial_xu_\nu\cdot
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)dx\right|
\,\leq\,C\log\left(\frac{1}{\varepsilon}+1\right)e_{\nu,\varepsilon}(t)\;;
$$
moreover, from (\ref{(N)eq_d_t_a_eps}) with $\sigma=1/2$, (\ref{(N)eq_d_x_a_eps}) and (\ref{(N)eq_d_t_d_x_a_eps}) we deduce
$$
\left|\int\frac{a}{\sqrt{a_\varepsilon}}\,\,\partial_x\left(\frac{\partial_t\sqrt{a_\varepsilon}}{\sqrt{a_\varepsilon}}\right)\,
\Re(\partial_xu_\nu\,\cdot\,\overline{u_\nu})\,dx\right|\,\leq\,C\,\frac{1}{\varepsilon}\,
\log\left(\frac{1}{\varepsilon}+1\right)\,2^{-\nu}\,e_{\nu,\varepsilon}(t)\;.
$$

Finally, we have
$$
\left|\int2\,\Re(u_\nu\,\cdot\,\overline{\partial_tu_\nu})\,dx\right|\,\leq\,
C\,\varepsilon^{-1/2}\,2^{-\nu}\,e_{\nu,\varepsilon}(t)\;.
$$

\subsubsection{Terms with $b_0$, $b_1$ and $c$}
Thanks to the hypothesis (\ref{(N)eq_condiz_B})-(\ref{(N)eq_condiz_c}), one has that there exist suitable constants, depending only on $\lambda_0$, $\Lambda_0$, $C_0$
and on the norms of $b_0$ and $b_1$ in the space $L^\infty(\mathbb{R}_t;\mathcal{C}^{\omega}(\mathbb{R}_x))$ and of $c$ in
$L^\infty(\mathbb{R}_t\times\mathbb{R}_x)$, such that
\begin{eqnarray*}
& & \left|\int\frac{2}{\sqrt{a_\varepsilon}}\,b_0(t,x)\,\,\Re\left(\partial_tu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\|\partial_t u_\nu\|_{L^2}\,
\left\|\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right\|_{L^2} \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,(C+C'\,2^{-\nu}\varepsilon^{-1/2})\,e_{\nu,\varepsilon}(t)\;; \\
& & \left|\int\frac{2}{\sqrt{a_\varepsilon}}\,b_1(t,x)\,\,\Re\left(\partial_xu_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx\right|
\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\int\sqrt[4]{a_{\varepsilon}}|\partial_x u_\nu|\,
\frac{1}{\sqrt[4]{a_\varepsilon}}\left|\partial_tu_\nu+
\frac{\partial_t\sqrt{a_{_\varepsilon}}}{2\sqrt{a_\varepsilon}}u_\nu\right|\,dx \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,2C\,\int\sqrt{a_\varepsilon}|\partial_x u_\nu|^2\,+\,
\frac{1}{\sqrt{a_\varepsilon}}\left|\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu
\right|^2\,dx \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,2\,e_{\nu,\varepsilon}(t)\;; \\
& & \left|\int\frac{2}{\sqrt{a_\varepsilon}}\,c(t,x)\,\,\Re\left(u_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\int|u_\nu|\,\frac{1}{\sqrt[4]{a_\varepsilon}}
\left|\partial_tu_\nu+\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right|\,dx \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,2C\,e_{\nu,\varepsilon}(t)\;,
\end{eqnarray*}
where we have delt with $\|\partial_tu_\nu\|_{L^2}$ as before.

\bigskip
Now, we join the approximation parameter $\varepsilon$ with the dual variable $\xi$, setting
$$
\varepsilon\,=\,2^{-\nu}\;;
$$
so, from (\ref{(N)eq_dt_e_nu,eps}) and the previous inequalities, we obtain
\begin{eqnarray}
& & \frac{d}{dt}e_{\nu,2^{-\nu}}(t)\,\leq\,\widetilde{C}\,(\nu+1)\,e_{\nu,2^{-\nu}} \label{(N)eq_dt_e_nu,eps_II}\\
& & \qquad\qquad\quad\quad\,+\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left((Lu)_\nu\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx \nonumber \\
& & \qquad\qquad\quad\quad\,+\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left((\partial_x([\varphi_\nu(D_x),a]\partial_xu))\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx \nonumber \\
& & \qquad\qquad\quad\quad\,-\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(([\varphi_\nu(D_x),b_0]\partial_tu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx \nonumber \\
& & \qquad\qquad\quad\quad\,-\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(([\varphi_\nu(D_x),b_1]\partial_xu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx \nonumber \\
& & \qquad\qquad\quad\quad\,-\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(([\varphi_\nu(D_x),c]u)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx \nonumber\;,
\end{eqnarray}
for a suitable constant $\widetilde{C}$, which depends only on
$\lambda_0$, $\Lambda_0$, $C_0$ and on the norms of the coefficients of the operator $L$ in their respective functional spaces.

\subsection{Estimates for the commutator terms}
Now, we have to deal with the commutator terms. As we will see, it's useful to consider immediately the sum over $\nu\in\mathbb{N}$.

First, we report an elementary lemma (see also~\cite{Colomb-DelSant_2}), which we will use very often in the next calculations.
\begin{lemma} \label{(N)lemma_1}
There exist two continuous, decreasing functions $\alpha_{1}$, $\alpha_2:\,]0,1[\,\rightarrow\,]0,+\infty[$ such that
$\lim_{c\rightarrow0^+}\alpha_j(c)=+\infty$ for $j=1\,,\,2$ and such that, for all $\delta\in]0,1]$ and all $n\geq1$, the following inequalities hold:
$$
\sum_{j=1}^{n}\,e^{\delta j}\,j^{-1/2}\,\leq\,\alpha_1(\delta)\,e^{\delta n}\,n^{-1/2}\,,\qquad\qquad
\sum_{j=n}^{+\infty}\,e^{-\delta j}\,j^{1/2}\,\leq\,\alpha_2(\delta)\,e^{-\delta n}\,n^{1/2}\,.
$$
\end{lemma}

Before going on, we take $\beta>0$ and $T\in]0,T_0]$ such that
$$
\beta T\,=\,\frac{\theta}{2}\,\log2\;.
$$
\begin{oss} \label{oss:beta_T}
Notice that, thanks to the hypothesis of Theorem \ref{(N)teor_teorema1}, this condition implies that
$$
\beta T\,\leq\,\frac{\omega-\theta}{2}\;.
$$
Moreover, for all $t\in[0,T]$, we have:
\begin{eqnarray*}
\beta t+\frac{\theta}{2}\log2 & \geq & \frac{\theta}{2}\log2\;>0\\
\beta t+\frac{\theta}{2}\log2 & \leq & \theta\log2\;\leq\;\frac{1}{2}\log2\;<1\\
(1-\theta)\log2-\beta t & \geq & \left(1-\frac{3}{2}\theta\right)\log2\;\geq\;\left(1-\frac{3}{4}\right)\log2\;>0\\
(1-\theta)\log2-\beta t & \leq & (1-\theta)\log2\;\leq\;\log2\;<1\,.
\end{eqnarray*}
\end{oss}

Finally, we set (with the same notations used in the subsection \ref{sub-sec:L-P_dec})
$$
\psi_{\mu}\,=\,\varphi_{\mu-1}+\varphi_{\mu}+\varphi_{\mu+1}\qquad(\varphi_{-1}\equiv0)\;.
$$
As $\psi_\mu\equiv1$ on the support of $\varphi_\mu$, we can write
$$
\partial_xu_\mu\,=\,\varphi_\mu(D_x)\partial_xu\,=\,\Psi_\mu(\varphi_\mu(D_x)\partial_xu)\,=\,\Psi_\mu\partial_xu_\mu\;,
$$
where $\Psi_\mu$ is the operator related to $\psi_\mu$. So, given a generic function $f(t,x)$, one has
\begin{equation} \label{(N)eq_nu-mu}
[\varphi_\nu(D_x),f]\partial_xu\,=\,[\varphi_\nu(D_x),f]\left(\sum_{\mu\geq0}\partial_xu_\mu\right)\,=\,
\sum_{\mu\geq0}([\varphi_\nu(D_x),f]\Psi_\mu)\partial_xu_\mu\;.
\end{equation}

After these preliminary remarks, we now can go on with commutators' estimates.

\subsubsection{Term with $[\varphi_\nu(D_x),a]$}

Due to Bernstein's inequalities, we have
$$
\left\|\partial_x\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)\right\|_{L^2}\,\leq\,C\,2^\nu\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\;.
$$
So, using (\ref{(N)eq_nu-mu}) and the fact that $a_\varepsilon$ is real-valued, one has
\begin{eqnarray*}
& & \left|\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(\partial_x([\varphi_\nu(D_x),a]\partial_xu)\,\cdot\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx\right|\,\leq\, \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\sum_\mu\|([\varphi_\nu(D_x),a]\Psi_\mu)\partial_xu_\mu\|_{L^2}\,
2^\nu\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2} \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\sum_\mu\|[\varphi_\nu(D_x),a]\Psi_\mu\|_{\mathcal{L}(L^2)}\,\left(e_{\mu,2^{-\mu}}(t)\right)^{1/2}\,
2^\nu\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\;,
\end{eqnarray*}
with the constant $C$ which depends only on $\lambda_0$, $\Lambda_0$ and $C_0$.\\
Then,
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta(\nu+1)t}2^{-2\nu\theta}
\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(\partial_x([\varphi_\nu(D_x),a]\partial_xu)
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)dx\right|\,\leq \\
& & \;\quad\qquad\leq\,C\,\sum_{\nu,\mu}k_{\nu\mu}\,(\nu+1)^{1/2}e^{-\beta(\nu+1)t}2^{-\nu\theta}
\left(e_{\nu,2^{-\nu}}\right)^{1/2}\,(\mu+1)^{1/2}e^{-\beta(\mu+1)t}2^{-\mu\theta}\left(e_{\mu,2^{-\mu}}\right)^{1/2}\;,
\end{eqnarray*}
where we have set
\begin{equation} \label{(N)eq_kernel_comm[a]}
k_{\nu\mu}\,=\,e^{-(\nu-\mu)\beta t}\,2^{-(\nu-\mu)\theta}\,2^{\nu}\,(\nu+1)^{-1/2}\,(\mu+1)^{-1/2}\,
\|[\varphi_\nu(D_x),a]\Psi_\mu\|_{\mathcal{L}(L^2)}\;.
\end{equation}
Observe that, if $|\nu-\mu|\geq3$, then $\varphi_\nu\psi_\mu\equiv0$, so $[\varphi_\nu(D_x),a]\Psi_\mu=\varphi_\nu(D_x)[a,\Psi_\mu]$.
Therefore, from lemma \ref{lemma:comm}, in particular from (\ref{eq:comm_Log-Lip}), we deduce that
$$
\|[\varphi_{\nu}(D_x),a(t,x)]\Psi_\mu\|_{\mathcal{L}(L^2)}\leq
\left\{
\begin{array}{ll}
C\,2^{-\nu}(\nu+1) & \mbox{ if } |\nu-\mu|\leq2\;, \\
 & \\
C\,2^{-max\{\nu,\mu\}}(\max\{\nu,\mu\}+1) & \mbox{ if } |\nu-\mu|\geq3\;,
\end{array}
\right.
$$
where the constant $C$ depends only on $\Lambda_0$ and $C_0$.

Now our aim is to apply Schur's lemma, so to estimate the quantity
\begin{equation} \label{(N)eq_lemma_schur}
\sup_{\mu}\sum_{\nu}|k_{\nu\mu}|\,+\,\sup_{\nu}\sum_{\mu}|k_{\nu\mu}|\;.
\end{equation}
To do this, we will use lemma \ref{(N)lemma_1} and the inequalities stated in remark (\ref{oss:beta_T}).

\begin{enumerate}
 \item Fix $\mu\leq2$.
\begin{eqnarray*}
\sum_{\nu\geq0}|k_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu+1)^{-\frac{1}{2}}
\sum_\nu e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,(\nu+1)^{\frac{1}{2}} \\
& = & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu+1)^{-\frac{1}{2}}
\sum_\nu e^{-(\nu+1)(\beta t+\theta\log2)}\,(\nu+1)^{\frac{1}{2}} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\alpha_2(\beta t+\theta\log2) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2(\theta\log2)\;.
\end{eqnarray*}
 \item Now, take $\mu\geq3$ and first consider
\begin{eqnarray*}
\sum_{\nu=0}^{\mu-3}|k_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu+1)^{\frac{1}{2}}
\sum_{\nu=0}^{\mu-3} e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,(\nu+1)^{-\frac{1}{2}} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu+1)^{\frac{1}{2}}
\sum_{\nu=0}^{\mu-3}e^{(\nu+1)(-\beta t+(1-\theta)\log2)}\,(\nu+1)^{-\frac{1}{2}} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu+1)^{\frac{1}{2}}\,
\alpha_1(-\beta t+(1-\theta)\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\quad\;\cdot\,
e^{(-\beta t+(1-\theta)\log2)(\mu-2)}\,(\mu-2)^{-\frac{1}{2}} \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_1\left(\left(1-\frac{3}{2}\theta\right)\log2\right)\;.
\end{eqnarray*}

For the second part of the sum, one has
\begin{eqnarray*}
\sum_{\nu=\mu-2}^{+\infty}|k_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu+1)^{-1/2}
\sum_{\nu=\mu-2}^{+\infty}e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,(\nu+1)^{1/2} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu+1)^{-1/2}\alpha_2(\beta t+\theta\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{-(\beta t+\theta\log2)(\mu-1)}\,(\mu-1)^{1/2} \\
& \leq & C\,2^{\frac{7}{2}\theta}\,\alpha_2(\theta\log2)\;.
\end{eqnarray*}
 \item Fix now $\nu\geq0$; we have
\begin{eqnarray*}
\sum_{\mu=0}^{\nu+2}|k_{\nu\mu}| & \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,(\nu+1)^{1/2}
\sum_{\mu=0}^{\nu+2}e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu+1)^{-1/2} \\
& \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,(\nu+1)^{1/2}\,\alpha_1(\beta t+\theta\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{(\beta t+\theta\log2)(\nu+3)}\,(\nu+3)^{-1/2} \\
& \leq & C\,2^{\frac{7}{2}\theta}\,\alpha_1(\theta\log2)\;.
\end{eqnarray*}
For the second part of the series, the following inequality holds:
\begin{eqnarray*}
\sum_{\mu=\nu+3}^{+\infty}|k_{\nu\mu}| & \leq & C\,e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,(\nu+1)^{-\frac{1}{2}}
\sum_{\mu=\nu+3}^{+\infty}e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu+1)^{\frac{1}{2}} \\
& \leq & C\,e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,(\nu+1)^{-\frac{1}{2}}\,
\alpha_2(-\beta t+(1-\theta)\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\;\;\cdot\,
e^{(-\beta t+(1-\theta)\log2)(\nu+4)}\,(\nu+4)^{\frac{1}{2}} \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2\left(\left(1-\frac{3}{2}\theta\right)\log2\right)\;.
\end{eqnarray*}
\end{enumerate}

In conclusion, there exists a positive function $\varPi$, with $\lim_{\theta\rightarrow0^+}\varPi(\theta)=+\infty$, such that
$$
\sup_{\mu}\sum_{\nu}|k_{\nu\mu}|\,+\,\sup_{\nu}\sum_{\mu}|k_{\nu\mu}|\,\leq\,C\,\varPi(\theta)\;,
$$
and so
\begin{eqnarray*}
& & \hspace{-1cm}\left|\sum_{\nu\geq0}e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}
\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left(\partial_x([\varphi_\nu(D_x),a]\partial_xu)
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\quad\qquad\qquad \leq\,C\,\varPi(\theta)\,
\sum_{\nu=0}^{+\infty}(\nu+1)\,e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t)\;.
\end{eqnarray*}

\subsubsection{Terms with $[\varphi_\nu(D_x),b_0]$ and $[\varphi_\nu(D_x),b_1]$}

Now, let us consider
\begin{eqnarray*}
& & \left|\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),b_0(t,x)]\partial_t u\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right)}\right)\,dx\right|\,\leq \\
& & \quad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,2\,\|[\varphi_{\nu}(D_x),b_0(t,x)]\partial_t u\|_{L^2}\,
\left\|\frac{1}{\sqrt{a_{2^{-\nu}}}}\left|\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}
u_\nu\right|\;\right\|_{L^2} \\
& & \quad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,2\,\|[\varphi_{\nu}(D_x),b_0(t,x)]\partial_t u\|_{L^2}\,(e_{\nu,2^{-\nu}}(t))^{1/2}\;.
\end{eqnarray*}

Thanks to relation (\ref{(N)eq_nu-mu}), we have
\begin{eqnarray*}
\|[\varphi_{\nu}(D_x),b_0(t,x)]\partial_t u\|_{L^2} & = &
\left\|[\varphi_{\nu}(D_x),b_0(t,x)]\sum_{\mu\geq0}\Psi_\mu\partial_tu_\mu\right\|_{L^2} \\
 & \leq & \sum_{\mu\geq0}\|[\varphi_{\nu}(D_x),b_0(t,x)]\Psi_\mu\|_{\mathcal{L}(L^2)}\|\partial_tu_\mu\|_{L^2}\;.
\end{eqnarray*}
As we have done before, we have, for constants depending only on $\lambda_0$, $\Lambda_0$ and $C_0$,
\begin{eqnarray*}
\|\partial_tu_\mu\|_{L^2}\,=\,
\left(\int\frac{\sqrt{a_{2^{-\mu}}}}{\sqrt{a_{2^{-\mu}}}}|\partial_tu_\mu|^{2}dx\right)^{1/2} & \leq & \left(C\,e_{\mu,2^{-\mu}}(t)+
C'\,\left\|\frac{\partial_t\sqrt{a_{2^{-\mu}}}}{2\sqrt{a_{2^{-\mu}}}}u_\mu\right\|_{L^2}\right)^{1/2} \\
& \leq & \left(C e_{\mu,2^{-\mu}}(t)\,+\,C'\frac{1}{2^{-\mu}}2^{-2\mu}\|\partial_x u_\mu\|_{L^2}\right)^{1/2} \\
& \leq & (C+ C'\,2^{-\mu})^{1/2}\left(e_{\mu,2^{-\mu}}\right)^{1/2} \\
& \leq & C\,\left(e_{\mu,2^{-\mu}}\right)^{1/2}\;.
\end{eqnarray*}

Due to lemma \ref{lemma:comm}, we obtain
$$
\|[\varphi_{\nu}(D_x),b_0(t,x)]\Psi_\mu\|_{\mathcal{L}(L^2)}\leq
\left\{
\begin{array}{ll}
C\,2^{-\nu\omega} & \quad\mbox{ if } |\nu-\mu|\leq2 \\
 & \\
C\,2^{-max\{\mu,\nu\}\omega} & \quad\mbox{ if } |\nu-\mu|\geq3
\end{array}
\right.
$$
where $C$ is a constant depending only on $\|b_0\|_{L^\infty(\mathbb{R}_t;\,\mathcal{C}^{\omega}(\mathbb{R}_x))}$.\\
As a matter of fact, the kernel of operator $[\varphi_\nu(D_x),b_0]$ is
$$
n(x,y)\,=\,\widehat{\varphi}(2^\nu(x-y))\,2^{\nu}\,(b_0(t,x)-b_0(t,y))\;;
$$
so, to evaluate its norm we apply Schur's lemma and, thanks to the fact that $b_0$ is $\omega$-h\"older, we get the desired estimate.

Therefore,
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}2^{-2\nu\theta} \int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\Delta_{\nu},b_0]\partial_tu\,
\overline{\left(\partial_t u_\nu+
\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\leq\,\sum_{\nu,\mu\geq0}e^{-\beta t(\nu+1)}\,2^{-\nu\theta}\,\left(e_{\nu,2^{-\nu}}\right)^{1/2}\,
e^{-\beta t(\mu+1)}\,2^{-\mu\theta}\,\left(e_{\mu,2^{-\mu}}\right)^{1/2}\,l_{\nu\mu}\;,
\end{eqnarray*}
where we have defined
\begin{equation} \label{(N)eq_kernel_comm[b_0]}
l_{\nu\mu}\,:=\,e^{-(\nu-\mu)\beta t}\,2^{-(\nu-\mu)\theta}\,\|[\varphi_{\nu}(D_x),b_0(t,x)]\Psi_\mu\|_{\mathcal{L}(L^2)}\;.
\end{equation}

As made before, we are going to estimate $l_{\nu\mu}$ applying Schur's lemma.

\begin{enumerate}
 \item Let us fix $\mu\leq2$. Then
\begin{eqnarray*}
\sum_{\nu\geq0}|l_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,
\sum_{\nu\geq0}e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,2^{-\nu\omega} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\sum_{\nu\geq0}e^{-(\nu+1)(\beta t+\theta\log2)}\,(\nu+1)^{1/2} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\alpha_2(\beta t+\theta\log2) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2(\theta\log2)\,.
\end{eqnarray*}
 \item Now, take $\mu\geq3$ and consider first
\begin{eqnarray*}
\sum_{\nu=0}^{\mu-3}|l_{\nu\mu}| & \leq & C\,e^{\mu\beta t}\,2^{\mu\theta}\,2^{-\mu\omega}
\sum_{\nu=0}^{\mu-3} e^{-\nu\beta t}\,2^{-\nu\theta} \\
& \leq & C\,e^{\mu(\beta t-(\omega-\theta)\log2)}\,(\mu-2) \\
& \leq & C\,e^{-\mu(\omega-\frac{\theta}{2})\log2}\,(\mu-2) \\
& \leq & C\,M(\omega,\theta)\;,
\end{eqnarray*}
where $M(\omega,\theta)$ is the maximum of the function $z\mapsto e^{-\gamma z}\,(z-2)$, with $\gamma=\left(\omega-\frac{\theta}{2}\right)\log2$.\\
For the second part of the sum, we have
\begin{eqnarray*}
\sum_{\nu=\mu-2}^{+\infty}|l_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}
\sum_{\nu=\mu-2}^{+\infty}e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,2^{-\nu\omega}
\frac{(\nu+1)^{1/2}}{(\nu+1)^{1/2}} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,(\mu-1)^{-1/2}\,\alpha_2(\beta t+\theta\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{-(\mu-1)\beta t}\,2^{-(\mu-1)\theta}\,(\mu-1)^{1/2} \\
& \leq &  C\,e^{2(\beta t+\theta\log2)}\,\alpha_2(\theta\log2) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2(\theta\log2)\,.
\end{eqnarray*}
\item Fix now $\nu$. Initially, we have
\begin{eqnarray*}
\sum_{\mu=0}^{\nu+2}|l_{\nu\mu}| & \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,2^{-\nu\omega}
\sum_{\mu=0}^{\nu+2}e^{(\mu+1)(\beta t+\theta\log2)}\,\frac{(\mu+1)^{1/2}}{(\mu+1)^{1/2}} \\
& \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,2^{-\nu\omega}\,(\nu+3)^{1/2}\,\alpha_1(\beta t+\theta\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{(\nu+3)\beta t}\,2^{(\nu+3)\theta}\,(\nu+3)^{-1/2} \\
& \leq & C\,e^{2(\beta t+\theta\log2)}\,\alpha_1(\theta\log2) \\
& \leq & C\,2^{3\theta}\,\alpha_1(\theta\log2)\;.
\end{eqnarray*}
Moreover,
\begin{eqnarray*}
\sum_{\mu=\nu+3}^{+\infty}|l_{\nu\mu}| & \leq & C\,e^{-\nu\beta t}\,2^{-\nu\theta}
\sum_{\mu=\nu+3}^{+\infty}e^{-\mu(-\beta t+(\omega-\theta)\log2)}\,\frac{\mu^{1/2}}{\mu^{1/2}} \\
& \leq & C\,e^{-\nu\beta t}\,2^{-\nu\theta}\,(\nu+3)^{-1/2}\,\alpha_2((\omega-\theta)\log2-\beta t)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{(\nu+3)\beta t}\,2^{-(\nu+3)(\omega-\theta)}\,(\nu+3)^{1/2}\\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,2^{-(\nu+3)\omega}\,\alpha_2\left(\left(\omega-\frac{3}{2}\theta\right)\log2\right) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2\left(\left(\omega-\frac{3}{2}\theta\right)\log2\right)\,.
\end{eqnarray*}
\end{enumerate}

From all these inequalities, thanks to Schur's lemma, one has that there exists a constant $\widetilde{M}(\omega,\theta)$,
depending only on $\|b_0\|_{L^\infty(\mathbb{R}_t;\,\mathcal{C}^{\omega}(\mathbb{R}_x))}$ and on the parameter $\theta$ (which we have fixed at the beginning of
the calculations), such that
$$
\sup_{\mu}\sum_{\nu}|l_{\nu\mu}|\,+\,\sup_{\nu}\sum_{\mu}|l_{\nu\mu}|\,\leq\,C\,\widetilde{M}(\omega,\theta)\;;
$$
from this relation, finally we get
\begin{eqnarray*}
& & \hspace{-0.5cm}\left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}2^{-2\nu\theta} \int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,
\Re\left([\varphi_{\nu}(D_x),b_0]\partial_t u\,
\overline{\left(\partial_t u_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\widetilde{M}(\omega,\theta)
\sum_{\nu=0}^{+\infty}(\nu+1)\,e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t)\;.
\end{eqnarray*}

With regard to the term with the commutator $[\varphi_\nu(D_x),b_1(t,x)]$, notice that, as we made before, one has, for constants depending
only on $\lambda_0$, $\Lambda_0$ and $C_0$,
\begin{eqnarray*}
& & \left|\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),b_1]\partial_xu\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)\,dx\right|\,\leq \\
& & \qquad\qquad\leq\,C\,\sum_{\mu\geq0}\|[\varphi_\nu(D_x),b_1]\Psi_\mu\|_{\mathcal{L}(L^2)}\,
\|\sqrt[4]{a_{2^{-\nu}}}\partial_xu_\mu\|_{L^2}\,\left\|\frac{1}{\sqrt[4]{a_{2^{-\nu}}}}
\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)\right\|_{L^2} \\
& & \qquad\qquad\leq\,C\,\sum_{\mu\geq0}\|[\varphi_\nu(D_x),b_1]\Psi_\mu\|_{\mathcal{L}(L^2)}\,\left(e_{\nu,2^{-\nu}}\right)^{1/2}\left(e_{\mu,2^{-\mu}}\right)^{1/2}\;.
\end{eqnarray*}

Therefore,
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}\,2^{-2\nu\theta}\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),b_1]\partial_xu\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)dx\right|\,\leq\\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C\,\sum_{\nu,\mu\geq0}e^{-\beta(\nu+1)t}\,2^{-\nu\theta}\,\left(e_{\nu,2^{-\nu}}\right)^{1/2}\,
e^{-\beta(\mu+1)t}\,2^{-\mu\theta}\,\left(e_{\mu,2^{-\mu}}\right)^{1/2}\,h_{\nu\mu}\;,
\end{eqnarray*}
where we have set again
$$
h_{\nu\mu}\,=\,e^{-(\nu-\mu)\beta t}\,2^{-(\nu-\mu)\theta}\,\|[\varphi_\nu(D_x),b_1]\Psi_\mu\|_{\mathcal{L}(L^2)}\;.
$$
As $b_0$ and $b_1$ satisfy to the same hypothesis, the commutator $[\varphi_\nu(D_x),b_1]$ verifies the same inequalities as $[\varphi_\nu(D_x),b_0]$;
so, if we repeat the same calculations, we obtain
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}\,2^{-2\nu\theta}\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),b_1]\partial_xu\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\widetilde{M}(\omega,\theta)
\sum_{\nu=0}^{+\infty}(\nu+1)\,e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t)\;.
\end{eqnarray*}

\subsubsection{Term with $[\varphi_\nu(D_x),c]$}

Finally, we have to deal with the commutator $[\varphi_\nu(D_x),c(t,x)]$.

First, observe that there exist constants, depending only on $\lambda_0$, $\Lambda_0$ and $C_0$ as usual, such that one has
\begin{eqnarray*}
& & \left|\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),c]\partial_xu\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)\,dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C\,\|[\varphi_{\nu}(D_x),c]u\|_{L^2}\,\left\|\frac{1}{\sqrt[4]{a_{2^{-\nu}}}}\left(\partial_tu_\nu+
\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)\right\|_{L^2} \\
& & \qquad\qquad\qquad\qquad\qquad\leq\,C\,\sum_{\mu\geq0}\|[\varphi_\nu(D_x),c]\Psi_\mu\|_{\mathcal{L}(L^2)}\,\|u_\mu\|_{L^2}\,
\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\\
& & \qquad\qquad\qquad\qquad\qquad\leq\,2\,C\,\sum_{\mu\geq0}\|[\varphi_\nu(D_x),c]\Psi_\mu\|_{\mathcal{L}(L^2)}\,
2^{-\mu}\,\|D_xu_\mu\|_{L^2}\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\\
& & \qquad\qquad\qquad\qquad\qquad\leq\,2\,C\,\sum_{\mu\geq0}\|[\varphi_\nu(D_x),c]\Psi_\mu\|_{\mathcal{L}(L^2)}\,2^{-\mu}\,
\left(e_{\mu,2^{-\mu}}(t)\right)^{1/2}\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\;.
\end{eqnarray*}

Thereby, we get the estimate
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}\,2^{-2\nu\theta}\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),c]\partial_xu
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\leq\,2\,C\sum_{\nu,\mu\geq0}e^{-\beta t(\nu+1)}\,2^{-\nu\theta}\,\left(e_{\nu,2^{-\nu}}(t)\right)^{1/2}\,
e^{-\beta t(\mu+1)}\,2^{-\mu\theta}\,\left(e_{\mu,2^{-\mu}}(t)\right)^{1/2}\,m_{\nu\mu}\;,
\end{eqnarray*}
where we have defined
$$
m_{\nu\mu}\,=\,e^{-(\nu-\mu)\beta t}\,2^{-(\nu-\mu)\theta}\,2^{-\mu}\,\|[\varphi_\nu(D_x),c]\Psi_\mu\|_{\mathcal{L}(L^2)}\;.
$$

The kernel of the operator $[\varphi_\nu(D_x),c]$ is
$$
n'(x,y)\,=\,\widehat{\psi}(2^\nu(x-y))\,2^{\nu}\,(c(t,y)-c(t,x))\,;
$$
so, remembering that $c$ is bounded over $\mathbb{R}\times\mathbb{R}$, from Schur's lemma one gets
$$
\|[\varphi_\nu(D_x),c]\|_{\mathcal{L}(L^2)}\,\leq\,C\qquad\qquad\forall\,\nu\geq0\,,
$$
where the constant $C$ depends only on $\|c\|_{L^\infty(\mathbb{R}_t\times\mathbb{R}_x)}$.

Again, we are going to estimate the kernel $m_{\nu\mu}$ to apply Schur's lemma.
\begin{enumerate}
 \item First, we take $\mu\leq2$ and we have
\begin{eqnarray*}
\sum_{\nu\geq0}|m_{\nu\mu}| & \leq &  C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,2^{-\mu}
\sum_{\nu\geq0}e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\sum_{\nu\geq0}e^{-(\nu+1)(\beta t+\theta\log2)}\,(\nu+1)^{1/2} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\alpha_2(\beta t+\theta\log2) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2(\theta\log2)\;.
\end{eqnarray*}
 \item Now, we fix $\mu\geq3$ and we consider the first part of the series:
\begin{eqnarray*}
\sum_{\nu=0}^{\mu-3}|m_{\nu\mu}| & \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,2^{-\mu}\,
\sum_{\nu=0}^{\mu-3} e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,2^{\mu-\nu}\frac{(\nu+1)^{1/2}}{(\nu+1)^{1/2}} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu-2)^{1/2}\,
\sum_{\nu=0}^{\mu-3} e^{(\nu+1)((1-\theta)\log2-\beta t)}\,(\nu+1)^{-1/2} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{-(\mu+1)(1-\theta)}\,(\mu-2)^{1/2}\,
\alpha_1((1-\theta)\log2-\beta t)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{-(\mu-2)\beta t}\,2^{(\mu-2)(1-\theta)}\,(\mu-2)^{-1/2} \\
& \leq & C\,e^{3\beta t}\,2^{3\theta}\,\alpha_1\left(\left(1-\frac{3}{2}\theta\right)\log2\right) \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_1\left(\left(1-\frac{3}{2}\theta\right)\log2\right)\;.
\end{eqnarray*}
For the second part, one has:
\begin{eqnarray*}
\sum_{\nu=\mu-2}^{+\infty}|m_{\nu\mu}| & \leq &  C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,2^{-\mu}
\sum_{\nu=\mu-2}^{+\infty}e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\frac{(\nu+1)^{1/2}}{(\nu+1)^{1/2}} \\
& \leq & C\,e^{(\mu+1)\beta t}\,2^{(\mu+1)\theta}\,2^{-\mu}\,\alpha_2(\beta t+\theta\log2)\,
e^{-(\mu-1)\beta t}\,2^{-(\mu-1)\theta} \\
& \leq & C\,2^{3\theta}\,\alpha_2(\theta\log2)\;.
\end{eqnarray*}
 \item Now, we fix $\nu\geq0$. Initially, we consider
\begin{eqnarray*}
\sum_{\mu=0}^{\nu+2}|m_{\nu\mu}| & \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,
\sum_{\mu=0}^{\nu+2}e^{(\mu+1)(\beta t+\theta\log2)}\,2^{-\mu}\,\frac{(\mu+1)^{1/2}}{(\mu+1)^{1/2}} \\
& \leq & C\,e^{-(\nu+1)\beta t}\,2^{-(\nu+1)\theta}\,(\nu+3)^{1/2}\,\alpha_1(\beta t+\theta\log2)\,\cdot \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\cdot\,e^{(\nu+3)\beta t}\,2^{(\nu+3)\theta}\,(\nu+3)^{-1/2} \\
& \leq & C\,e^{2(\beta t+\theta\log2)}\,\alpha_1(\theta\log2) \\
& \leq & C\,2^{3\theta}\,\alpha_1(\theta\log2)\;.
\end{eqnarray*}
The second part of the series, instead, can be treated as follow:
\begin{eqnarray*}
\sum_{\mu=\nu+3}^{+\infty}|m_{\nu\mu}| & \leq & C\,e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,
\sum_{\mu=\nu+3}^{+\infty}e^{(\mu+1)\beta t}\,2^{-\mu}\,2^{-(\mu+1)(1-\theta)}\,2^{\mu-\nu}\,
\frac{(\mu+1)^{\frac{1}{2}}}{(\mu+1)^{\frac{1}{2}}} \\
 & \leq & C\,e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,(\nu+4)^{-\frac{1}{2}}\,
\sum_{\mu=\nu+3}^{+\infty}e^{-(\mu+1)((1-\theta)\log2-\beta t)}\,(\mu+1)^{\frac{1}{2}} \\
& \leq & C\,e^{-(\nu+1)\beta t}\,2^{(\nu+1)(1-\theta)}\,\alpha_2((1-\theta)\log2-\beta t)\,e^{(\nu+4)\beta t}\,2^{-(\nu+4)(1-\theta)} \\
& \leq & C\,2^{\frac{9}{2}\theta}\,\alpha_2\left(\left(1-\frac{3}{2}\theta\right)\log2\right)\;.
\end{eqnarray*}
\end{enumerate}
Finally, we obtain:
\begin{eqnarray*}
& & \left|\sum_{\nu\geq0}e^{-2\beta t(\nu+1)}\,2^{-2\nu\theta}\,\int\frac{2}{\sqrt{a_{2^{-\nu}}}}\,\,\Re\left([\varphi_{\nu}(D_x),c]\partial_xu\,
\overline{\left(\partial_tu_\nu+\frac{\partial_t\sqrt{a_{2^{-\nu}}}}{2\sqrt{a_{2^{-\nu}}}}u_\nu\right)}
\right)dx\right|\,\leq \\
& & \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\leq\,C\,\varPi(\theta)\,
\sum_{\nu=0}^{+\infty}(\nu+1)\,e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t)\;,
\end{eqnarray*}
where the function $\varPi$ is the same used in the estimate of the term $[\varphi_\nu(D_x),a]$.

\subsection{End of the proof of theorem \ref{(N)teor_teorema1}}

Now we can complete the proof of theorem \ref{(N)teor_teorema1}.

First, remembering the definition of total energy given by (\ref{(N)eq_E_tot}), we have that there exists a constant $C>0$, depending only on
$\theta$, such that
$$
\left|\sum_{\nu=0}^{+\infty}e^{-\beta(\nu+1)t}\,2^{-2\nu\theta}\,\int\frac{2}{\sqrt{a_\varepsilon}}\,\,\Re
\left((Lu)_\nu\,\cdot\,\overline{\left(\partial_tu_\nu+
\frac{\partial_t\sqrt{a_\varepsilon}}{2\sqrt{a_\varepsilon}}u_\nu\right)}\right)\,dx\right|\,\leq\,C\,(E(t))^{1/2}\,\|Lu\|_{H^{-\theta-\beta^*t}}\,.
$$
Therefore, if we set $\widetilde{\varPi}(\omega,\theta)=\max\{\widetilde{M}(\omega,\theta),\varPi(\theta)\}$, from relation (\ref{(N)eq_dt_e_nu,eps_II})
and from the estimates proved in the previous subsection, we have that, for suitable constants, depending only on $\lambda_0$, $\Lambda_0$, $C_0$
and on the norms of the coefficients of operator $L$ in their respective functional spaces, the following inequality holds:
\begin{eqnarray*}
\frac{d}{dt}E(t) & \leq & \left(C+C'\,\widetilde{\varPi}(\omega,\theta)-2\beta\right)\,
\sum_{\nu=0}^{+\infty}\,(\nu+1)\,e^{-2\beta(\nu+1)t}\,2^{-2\nu\theta}\,e_{\nu,2^{-\nu}}(t) \\
 & + & C''\,(E(t))^{1/2}\,\|Lu\|_{H^{-\theta-\beta^*t}}\,.
\end{eqnarray*}
Now, let us fix $\beta$ large enough, such that $C+C'\,\widetilde{\varPi}(\omega,\theta)-2\beta\leq0$: we can always do this, on condition that we take
$T$ small enough. With this choice, we have
$$
\frac{d}{dt}E(t)\,\leq\,C''\,(E(t))^{1/2}\,\|Lu\|_{H^{-\theta-\beta^*t}}\;;
$$
now, the thesis of the theorem follows from Gronwall's lemma and remark \ref{remark:E(0)-E(t)}.


\section{Introduction}

Neural Architecture Search (NAS) has advanced state-of-the-art on various tasks, such as image classification \citep{dblp:conf/cvpr/zophvsl18,Pham2018EfficientNA,2018regularized, dblp:conf/cvpr/tancpvshl19}, machine translation \citep{DBLP:journals/taslp/FanTXQLL20, DBLP:conf/icml/SoLL19}, and language modeling \citep{ Pham2018EfficientNA, DBLP:conf/iclr/LiuSY19, jiang-etal-2019-improved, li-etal-2020-learning}. Despite the remarkable results, conventional NAS methods are computationally expensive, requiring training millions of architectures during search. For instance, obtaining a state-of-the-art machine translation model with an evolutionary algorithm requires more than 250 GPU years \citep{DBLP:conf/icml/SoLL19}.

Several techniques have been proposed to improve the search efficiency, such as sharing parameters among all architectures \citep{Pham2018EfficientNA, DBLP:conf/aaai/CaiCZYW18, DBLP:conf/cvpr/ZhongYWSL18}, predicting the performance instead of full training \citep{ DBLP:conf/eccv/LiuZNSHLFYHM18, DBLP:conf/iclr/BakerGRN18, DBLP:conf/eccv/WenLCLBK20, DBLP:journals/corr/abs-2003-12857}, and searching over a continuous space \citep{ DBLP:conf/iclr/LiuSY19, jiang-etal-2019-improved, li-etal-2020-learning}. Unfortunately, these approaches still suffer from the high cost of predicting the performance of each candidate architecture. An inherent reason for this is that obtaining accurate performance requires training numerous neural networks to convergence, as described in Sec. \ref{sec:PE}. However, it is unnecessary to predict the model performance as in previous NAS methods. Rather, all we need is to distinguish architectures of different quality in NAS, say, ranking these architectures.

\input{analysis/search-cost.tex}

In this paper, we approach the problem by formulating NAS as a ranking task. Here we propose RankNAS, a ranking model for comparing different architectures. One of the key challenges is that directly ranking all architectures in a large search space is still computationally infeasible. Therefore, we adopt the pairwise method \citep{DBLP:conf/icml/BurgesSRLDHH05, DBLP:conf/icml/WauthierJJ13}, where the ranking problem is reduced to a binary classification problem over architecture pairs. To speed up RankNAS further, we develop an architecture selection method that chooses the most promising architectures for evaluation according to the importance of features, e.g., the topology of architectures.

We test RankNAS on well-established machine translation and language modeling benchmarks. Experiments show that RankNAS is orders of magnitude faster than standard NAS systems and can find better architectures. Notably, RankNAS is generic to different tasks and evaluation metrics. It achieves competitive results on hardware-aware NAS tasks and is 10$\times$ faster than the HAT baseline \citep{wang2020hatht}. It also discovers new architectures that outperform vanilla Transformer by +1.8 BLEU points on the IWSLT'14 De-En data and +1.5 BLEU points on the WMT'14 En-De data, surpassing the Evolved Transformer \citep{DBLP:conf/icml/SoLL19} with 150,000$\times$ less search cost.

\section{Preliminaries}
NAS generally consists of two steps: 1) sample architectures from the pre-defined search space, and 2) estimate the performance of these samples. This work focuses on the performance estimation step, which is the efficiency bottleneck of NAS.

\subsection{Search Space}
\label{sec:search-space}
The search space contains all possible architectures for the search. In this work, we take the Transformer architecture for description, but the discussed problem and solutions are general and can be applied to other models. Following HAT \citep{wang2020hatht}, we represent a Transformer architecture as a set of features and search for the optimal model configuration.

An overview of the search space is shown in Figure \ref{fig:search-space}. It is extended from the HAT's space and inspired by manually designed Transformer variants, including Relative Position Representations \cite{DBLP:conf/naacl/ShawUV18} and Deep Transformer \cite{wang-etal-2019-learning-deep}.  The search space can also be represented as a \textit{supernet} where each sub-network is a unique architecture. The search space contains around $10^{23}$ possible architectures, as detailed in Appendix \ref{appendix-space}. It is computationally prohibited to explore such a large space with an exhaustive method.

\input{method/search-space.tex}

\subsection{Performance Estimation}
\label{sec:PE}
Let $\mathcal{A}$ denotes the search space, and each architecture in it is represented by a feature vector $\alpha$. Formally, the goal of NAS is to find the optimal architecture $\alpha^{*}$ with the best performance. The performance can be measured by some metrics, such as accuracy or latency. The performance estimation process consists of two steps: 1) estimate the performance of all architectures, and 2) choose the architecture with the optimal performance.

Without loss of generality, we define $\mathcal{S}(\cdot)$ as the performance evaluated by some metrics. The task here is to find the most promising architecture with maximum $\mathcal{S}(\cdot)$. Standard NAS methods solve this problem by learning to estimate the performance of each architecture. The objective is given by:
\begin{equation}
\label{eq:nas}
\begin{aligned}
\alpha^{*}= \ & \operatorname*{argmax}_{\alpha} \mathcal{S}_{val}(w^{*}, \alpha)\\
\textrm{s.t.} \ w^{*}= \ & \operatorname*{argmax}_{w} \mathcal{S}_{train}(w, \alpha)
\end{aligned}
\end{equation}
where $w$ is the weights associated with the architecture. $\mathcal{S}_{val}$ and $\mathcal{S}_{train}$ are the evaluation results on the validation set and training set, respectively.

Optimizing Eq. \ref{eq:nas} is time-consuming as obtaining the optimal weights for each architecture requires training them to converge. Although we can share the weights among all architectures to amortize the cost, performance evaluation is still nontrivial and requires numerous training steps.

\section{NAS as Ranking}
As mentioned in Sec. \ref{sec:PE}, the goal of NAS is to find promising architectures that achieve high performance on unseen data. NAS requires distinguishing whether the architectures are “good” or “bad” rather than predicting accurate performance. Therefore, it is natural to treat NAS as a ranking problem, in which the explicit goal is to rank different architectures correctly.

\subsection{Pairwise Ranking}
\paragraph{Problem Formulation.}
Given an architecture $\alpha$, we define a score $s$ on it by a function $r(\cdot)$:

\begin{equation}
s = r(\alpha, p)
\end{equation}

\noindent where $p$ is the parameter of the scoring function. We implement the scoring function with a gradient boosting decision tree, as detailed in Sec. \ref{sec:ranking-setup}.

We want to optimize $p$ such that $s$ assigns high scores to good architectures and low scores to bad architectures. This induces a ranking of the candidate architectures in the search space. It is infeasible to sort all candidate architectures in a large search space directly. A solution is to reduce the listwise ranking problem to the pairwise ranking problem. Fortunately, the properties of the NAS task allow us to achieve the goal. As described in \citet{dudziak2021brpnas}, the relation between any pair of performance is \textit{antisymmetric}, \textit{transitive} and \textit{connex}. This makes it possible to rank all architectures via pairwise comparisons, substantially reducing the training complexity.

\paragraph{Training Set Construction.}
\label{sec:traing-set}
In pairwise ranking, the learning task is framed as a \textit{binary classification} of architecture pairs into two categories: correctly ordered and incorrectly ordered. Given an architecture pair $(\alpha_{i}, \alpha_{j})$ and the order of performance $\bar{P}_{ij}$, we can construct training examples $(\alpha_{i}, \alpha_{j}, \bar{P}_{ij})$ for the classification by comparing the two values. Note that $\bar{P}_{ij}$ is a 0-1 variable. For example, if $\alpha_{i}$ is better than $\alpha_{j}$, we would add $(\alpha_{i}, \alpha_{j}, 1)$ and $(\alpha_{j}, \alpha_{i}, 0)$ to the training set.

\paragraph{Optimization.}
\label{sec:ranknas}
Consider a pair of architectures $(\alpha_{i}, \alpha_{j})$, scored by $s_i$ and $s_j$, respectively. The probability of $\alpha_{i}$ being better than $\alpha_{j}$ is given by the difference through an activation function $g$:
\begin{equation}
P_{i j}= {g\left(s_{i}-s_{j}\right)}
\end{equation}
We assume that $P_{i j}\geq0.5$ means $\alpha_{i}$ is better than $\alpha_{j}$ while $P_{i j}<0.5$ means $\alpha_{j}$ is better than $\alpha_{i}$. Here we use a logistic function to achieve this goal:
\begin{equation}
\label{eq:pij}
P_{i j}=\frac{1}{1+e^{-(s_{i}-s_{j})}}
\end{equation}
Similarly, $P_{j i}$ can be induced by:
\begin{equation}
\label{eq:pji}
P_{j i}= \frac{1}{1+e^{-(s_{j}-s_{i})}} = 1 - P_{i j}
\end{equation}
Denote the gold score of $\alpha_{i}$ being better than $\alpha_{j}$ as $\bar{P}_{i j}$. We use the cross-entropy loss function for the classification. The loss for a pair of inputs is:
\begin{equation}
\label{eq:ranknas}
\begin{aligned}
L_{i j} &=-(\bar{P}_{ij} \log P_{ij} + \bar{P}_{ji} \log P_{ji}) \\
&={(1-\bar{P}_{i j}) \cdot (s_{i}-s_{j})} + \\
& \ \ \ \ \ \log {(1+e^{-(s_{i}-s_{j})})}
\end{aligned}
\end{equation}

Compared with Eq. \ref{eq:nas}, Eq. \ref{eq:ranknas} just requires $\bar{P}_{i j}$. In particular, we use the intermediate performance measured on the validation set during training. It is much easier than assessing the accurate performance of candidate architectures. In this sense, the ranking model is ``easier'' to learn and may not need many training samples as in performance prediction. RankNAS also enables efficient optimization through gradient methods. Algorithm \ref{algo:training} describes the complete training process of the ranking model.

\begin{algorithm}[t]
\caption{Training of RankNAS}
\label{algo:training}
\KwIn {search space $\mathcal{A}$ and ranking model $r$}
\While{$r$ not converged}{
\textbf{training example construction:} sample $(\alpha_{i}, \alpha_{j})$ from $\mathcal{A}$, compute $\bar{P}_{ij}$ by comparing their performance\;
\textbf{classification:} compute scores $(s_i, s_j)$\;
\textbf{optimization:} optimize $r$ w.r.t. Eq. \ref{eq:ranknas}.
}
\end{algorithm}

\subsection{Applying Pairwise Ranking}
Although the training time of the ranking model is heavily reduced, it is still challenging to apply it to the ranking of all architectures in the search space $\mathcal{A}$. The challenge is that exploring all architectures is computationally expensive, even when the task is a binary classification.

\input{method/overview.tex}
\input{analysis/loss-features.tex}

\paragraph{Correlations between Features and Performance.}
We start by analyzing the effect of architectural features on estimated performance. Figure \ref{fig:performance-features} illustrates the impact of the FFN dimension on latency and the validation loss on the machine translation task. We observe that: (a) different architectural features have very different correlations with the same evaluation metric, and (b) the same features also have different influences on different metrics. For example, the latency monotonically increases when scaling the FFN dimension on CPUs, while it is almost unchanged on GPUs. Hence, it is natural to improve search efficiency by eliminating unimportant features.

\paragraph{Feature Importance.}
\label{sec:feature-selection}
Inspired by previous feature selection methods \citep{DBLP:journals/ml/Breiman01, DBLP:journals/jmlr/FisherRD19}, we measure the importance of an architectural feature (e.g., the number of layers) by calculating the increase in the model error after permuting the feature.

We assume that each architecture $\alpha$ is represented by a feature vector $\bm{f} \in \mathcal{R}^{M \times N}$, where $M$ is the number of different features, and $N$ is the dimension of feature vectors. Also, we assume a set $C$ that contains $n$ architectures sampled from the search space. We first estimate the original model error $L_{total}$ on $C$ using the accumulation of the prediction errors. For any feature $\bm{f}_i \in \bm{f}$, we randomize it for each architecture in $C$. Then the randomized architectural features are passed to the ranking model and yield an error $L_i$. The importance of the $i$-th feature $\bm{f}_i$ is defined by:
\begin{equation}
    I(\bm{f}_i) = \frac{L_{i}}{L_{total}}
\end{equation}
where a higher value implies $\bm{f}_i$ is more important.

\paragraph{Search Space Pruning.}
It is easy to select valuable architectural features with the above measure. Given all features $\bm{f} \in \mathcal{R}^{M \times N}$, we discard those with a score less than a threshold $\theta$ and obtain the selected features $\bm{f'} \in \mathcal{R}^{M' \times N}$, where $M' < M$. Then we can prune the search space according to the selected features. For instance, if the feature \textit{Embedding Dimension} is not selected, we will keep it \textit{fixed} during the search. Finally, we only search over the architectures in the reduced search space.

An overview of the search process is presented in Figure \ref{fig:overview}. As described in Sec. \ref{sec:ranknas}, the training of the proposed ranking model is much cheaper than previous methods, which need to optimize the parameters for all architectures. Pruning search space further reduces the number of architectures to be evaluated. Also, the sampling procedure can be implemented with any existing NAS search strategy, e.g., Random Search (RS) or Evolution Algorithm (EA).

\section{Experiments}
\label{sec:experiment}
\subsection{Experimental Setups}
We evaluate our methods on language modeling and machine translation tasks. In the experiments, we search for hardware-aware architectures and high-accuracy architectures.

\input{res/hat-search.tex}

\paragraph{Training Setups.}
For machine translation, we experiment on the IWSLT’14 De-En and WMT’14 En-De tasks using the identical settings as \citet{wang2020hatht}. For language modeling, we experiment on the WikiText-103 dataset \citep{Merity2017PointerSM} with the same settings as \citet{baevski2019adaptiveir}. We set the maximum number of tokens per sample to 1,843 to fit the memory constraints and apply gradient accumulation to keep the same batch size as \citet{baevski2019adaptiveir} ’s work. All models are trained with mixed precision on 8 NVIDIA RTX 2080 Ti GPUs except for IWSLT ones, which only take one GPU for training.

\paragraph{Ranking Model Setups.}
\label{sec:ranking-setup}
We implement the ranking model (binary classifier) described in Sec.\ref{sec:ranknas} with LightGBM \citep{DBLP:conf/nips/KeMFWCMYL17} and set the learning rate to 0.1. To prevent overfitting, we set the maximum number of leaves to 30 and the tree's maximum depth to 6. We also use the default regularization terms and apply the early stopping strategy to the training. Specifically, the training stops if the validation score does not increase for 5 rounds. After training the ranking model, we apply the search space pruning method to find the most valuable architectural features for different tasks and hardware. There are two hyper-parameters for pruning: the sample size and the threshold. We set them to 200/1.15 and 300/1.25 for the hardware-aware architecture search and high-accuracy architecture search, respectively.

\input{res/lm-search.tex}

\paragraph{Architecture Search Setups.}
Table \ref{tab:iwslt-search-space} and Table \ref{tab:wmt-search-space} presents the search space of high-accuracy search for the translation tasks. We refer the readers to \citet{wang2020hatht}'s work for more details about the search space of hardware-aware architecture search. RankNAS is not restricted to a specific search strategy. We compare different search strategies in the experiments, including Random Search (RS) and Evolutionary Algorithm (EA). We apply uniform sampling for RS and use the same settings as \citet{wang2020hatht}'s work for EA. More specifically, the random search process will stop if the best-so-far architecture does not change for 3 epochs.

\input{res/accuracy-search.tex}
\paragraph{Evaluation Metrics.}
We report the results obtained by averaging 5 runs with different seeds. We calculate BLEU scores with case-sensitive tokenization using Moses, and apply the compound splitting BLEU for WMT, the same as HAT. We test the latency of models on an Intel Xeon Silver 4114 CPU and an NVIDIA GTX 1080Ti GPU. A machine translation model's latency is the time of translating a single sentence with a fixed length - 30 for WMT and 23 for IWSLT. For language modeling, the latency is the cost of decoding a single sentence without mini-batching, averaged over the whole test set. Following \citet{wang2020hatht}'s work, we measure each model's latency 300 times and remove the fastest and slowest 10\% and then take the average of the rest 80\%. Note that we report the total number of trainable parameters in a model, while \citet{wang2020hatht} emit the parameters of the embedding layers. The search cost is the GPU hours measured on or normalized to a single RTX 2080Ti.

\subsection{Results}

\paragraph{Hardware-Aware Architecture Search.}
\label{exp:hat}
The hardware-aware NAS aims to maximize the accuracy under specified latency constraints on different hardware platforms. We first rank architectures by their latencies and pick those that meet the constraint to achieve this goal. Then we rank the selected architectures by their losses on the validation set and choose the best one. For machine translation tasks, we use the same search space as HAT \citep{wang2020hatht}, which contains around $10^{15}$ possible architectures. For the language modeling task, we use the following search space: [10, 12, 14] for decoder layer number, [768, 1024] for embedding dimension, [3072, 4096, 5120] for hidden dimension, and [8, 12, 16] for the head number in attention modules. We add a simple linear projection without bias if two adjacent layers have different hidden sizes.

Table \ref{tab:hat-search} shows the results of RankNAS comparing to HAT \citep{wang2020hatht} and Transformer \citep{DBLP:conf/nips/VaswaniSPUJGKP17} on the machine translation tasks. Our method is effective in reducing the search cost for different tasks and hardware platforms. For instance, it requires 10.53$\times$ less cost to find a comparable architecture on the WMT task. The discovered architectures also have the lowest latencies with the same or better BLEU scores on most tasks. For example, the architecture designed for the CPU is 2.68$\times$ faster than the standard Transformer.

We present the architecture search results for language modeling on the WikiText-103 test data in Table \ref{tab:lm-search}. All models are evaluated with a context window of 2,560 tokens, following \citet{baevski2019adaptiveir}. Our method significantly accelerates the baseline on different devices. Specifically, our method speeds up the baseline by 2.59$\times$ on the CPU and 1.83$\times$ on the GPU. Our model also obtains a perplexity of 18.13, which outperforms Transformer-XL \citep{dai2019transformerxlal} and is comparable to the state-of-the-art language model, e.g., Sandwich-Transformer \citep{DBLP:conf/acl/PressSL20}.

\paragraph{High-Accuracy Architecture Search.}
\label{exp:accuracy-search}
Unlike hardware-aware architecture search, the high-accuracy architecture search only optimizes accuracy and does not consider latency. In the experiments, we enlarge the HAT's search space by introducing two additional features \textit{Relative Attention Position} \citep{DBLP:conf/naacl/ShawUV18} and \textit{Layer Norm Position}, as shown in Table \ref{tab:iwslt-search-space} and Table \ref{tab:wmt-search-space}. This expands the size of search space to $10^{23}$, 8 orders of magnitude larger than HAT.

\input{res/mt-ranker.tex}

We compare RankNAS with state-of-the-art machine translation models designed by human experts and models discovered by other NAS methods. The results are presented in Table \ref{tab:accuracy-search}. RankNAS consistently outperforms other methods in both the IWSLT and WMT tasks. It demonstrates that RankNAS can also design high-accuracy architectures. Notably, the discovered architectures achieve a +1.8 BLEU improvement on the IWSLT task and a +1.5 BLEU improvement on the WMT task than the standard Transformers baseline \citep{DBLP:conf/nips/VaswaniSPUJGKP17}. We show that RankNAS surpasses the Evolved Transformer \citep{DBLP:conf/icml/SoLL19}, with orders of magnitude fewer search costs. RankNAS also matches the performance of gradient-based methods, including NAO \citep{DBLP:journals/taslp/FanTXQLL20} and DARTSformer \citep{DBLP:conf/acl/ZhaoDSZWC21}.

\section{Analysis}
We analyze both the accuracy and efficiency of our search method and study the effect of different features on model performance.

\subsection{Architecture Ranking Accuracy}
\label{analysis:accuracy}
To study the accuracy of the proposed method, we evaluate it on the IWSLT translation task. We randomly sample 200 different architectures from the HAT search space (small) and the enlarged search space (large) introduced in Sec. \ref{exp:accuracy-search}. We train these architectures from scratch and measure their BLEU scores on the test set. Table \ref{tab:mt-predictor} presents the Kendall and Spearman rank correlation coefficient between the predicted results and the real scores. RankNAS outperforms HAT in terms of different ranking correlations. For example, RankNAS achieves a high Kendall's Tau of 0.883 and 0.826 on small and large spaces. This indicates that the predicted ranking is very close to the real results.

\input{analysis/selected-features.tex}

\paragraph{Importance of Ranking Accuracy.}
\label{sec:analysis-accuracy}
Although our ranking model is more accurate than prior methods, a question remains: how does ranking accuracy affect the search quality? We analyze the impact of different ranking models on the high-accuracy NAS task. Figure \ref{fig:bleu-sample} compares two ranking models with different ranking correlation coefficients. The results are obtained by best-so-far models trained from scratch on the IWSLT'14 De-En data. Results show that inaccurate ranking leads to poor search results. It indicates that an accurate ranking model is essential for architecture search.

\subsection{Analysis of Discovered Architectures}
We present the discovered architectures in Appendix \ref{appendix-arch} and analyze important features for different hardware on the IWSLT'14 De-En task.

Figure \ref{fig:selected-features} (top) plots the selected features for the CPU. It shows that the decoder FFN dimension is the most important feature for predicting latency, followed by the decoder's arbitrary attention and the encoder FFN dimension. We also find that the decoder embedding dimension has a similar impact on latency as the number of decoder layers.

Figure \ref{fig:selected-features} (bottom) illustrates the results for the GPU. Similar to the CPU, the latency on the GPU has a high correlation to the decoder attention module. The main difference is that the latency on GPU is insensitive to FFN or embedding dimensions but more sensitive to the number of decoder layers.

The results indicate that we can design \textit{``shallow and wide''} models for GPUs and \textit{``deep and thin''} models for CPUs to achieve the Pareto-optimal state. Similar design insights have been verified in recent works, such as \citet{wang-etal-2019-learning-deep}, \citet{hu-etal-2020-niutrans}, \citet{DBLP:conf/aaai/LiLXZ21}, and \citet{lin-etal-2021-weight}.

\subsection{Search Efficiency}
Experiments in Sec. \ref{sec:experiment} show that our method has much lower search costs than previous works. We now analyze how does our method accelerates the architecture search.

\paragraph{Ranking Model Training Efficiency.}
The overall search cost includes the training time of the ranking model and the cost of the search process. Figure \ref{fig:search-cost} compares our method and HAT on the IWSLT’14 De-En task. The two methods share the same search space and sampling strategy for search. We observe that the ranking model training takes most of the time. RankNAS speeds up the ranking model training by 10.34 times compared with HAT. Pruning the search space further reduces the 75\% time of the search process. Thus the overall search cost is significantly reduced. It indicates that efficient training of the ranking model is essential to accelerate the search process.

\paragraph{Architecture Search Efficiency.}
We also analyze the efficiency of our proposed methods on the IWSLT hardware-aware task. Figure \ref{fig:loss} shows the loss curves on the validation set of the models found by our method with different sampling strategies. We observe that RankNAS is compatible with different strategies. Also, the evolutionary algorithm outperforms random search in terms of the rate of convergence and the search result.

\input{analysis/bleu-sample}

\section{Related Work}
Many efforts have been made to improve the NAS efficiency for different tasks \citep{dblp:conf/cvpr/tancpvshl19, dblp:journals/corr/abs-1812-03443, dblp:journals/corr/abs-1812-00332, DBLP:journals/corr/abs-1911-00105, Chen2020AdaBERTTB}. A common approach to accelerating the search process is to use a proxy, such as reduced model size, training data, or training steps. However, it is inaccurate for estimating the model's performance and diminishes the NAS quality \citep{DBLP:conf/iclr/BakerGRN18,dudziak2021brpnas}. Another popular way is to share parameters among all architectures to reduce the training time \citep{dblp:conf/cvpr/tancpvshl19, dblp:journals/corr/abs-1812-00332}. However, it is infeasible to train all architecture candidates fairly to obtain their accurate performance.

\input{analysis/loss.tex}

Recent works explored performance prediction based on architectural properties, i.e., the network topology and the model size \citep{DBLP:conf/eccv/LiuZNSHLFYHM18, 8901943, DBLP:conf/eccv/WenLCLBK20, DBLP:conf/eccv/NingZZWY20}. For instance, Hardware-Aware Transformer (HAT) \citep{wang2020hatht} encoded architectures into feature vectors and predicted the latency with a Multilayer Perceptron (MLP) for the target hardware. BRP-NAS \citep{dudziak2021brpnas} proposed an end-to-end performance predictor based on a Graph Convolutional Network (GCN). Although these methods greatly improve the performance estimation efficiency, they still require many samples and train numerous neural networks to converge, thereby increasing the search cost. Instead, we are motivated by the fact that NAS is expected to distinguish different candidate architectures. Thus, NAS can be solved by learning pairwise ranking rather than obtaining the accurate performance of architectures.

\section{Conclusion}
We have presented RankNAS, a simple yet efficient NAS algorithm for both hardware-aware and high-accuracy architecture search. We have shown that pairwise ranking can significantly improve search efficiency. We also have proposed a search space pruning method to help the ranking model be more efficient during the search. Our approach outperforms prior methods in both efficiency and accuracy. RankNAS requires 80\% less time in ranking model training on the hardware-aware search task and accelerates the overall search process by 11.53 times. Also, the architectures discovered by our method outperform state-of-the-art Transformer models in terms of efficiency and accuracy.

\section*{Acknowledgements}
This work was supported in part by the National Science Foundation of China (Nos. 61876035 and 61732005), the National Key R\&D Program of China (No.2019QY1801), and the Ministry of Science and Technology of the PRC (Nos. 2019YFF0303002 and 2020AAA0107900). The authors would like to thank the anonymous reviewers for their comments and suggestions.