\section{{FedAGM}}
\label{sec:method}

This section first describes FedAGM algorithm and discuss how it solves the problem of client heterogeneity. Then we show that FedAGM is desirable for the real-world federated learning settings.

\subsection{{Acceleration of global momentum}}

To reduce the gap between the local and global objective function, our main idea is to incorporate global gradient information into local updates by accelerating the current model with global momentum.

The Pseudo-code for our proposed method, FedAGM, is shown in~\cref{alg:proposed_method}.
{
In each round $t \in \{0,1,\dots,T\}$, the central server quantifies the server update direction $\Delta^t=-(\theta^{t}-\theta^{t-1})$, and broadcasts the accelerated global model $\theta^t - \lambda \Delta^t$ to the currently active clients $S_{t}$.}
Starting with the integrated model as an initial point, each participating client optimizes a local objective function.
{The local objective function is defined as the sum of its local empirical loss function and a penalized loss function which is based on the local online model and the received server model:}
\begin{equation}
{\underset{\theta_{i,k}^t}{\operatorname{argmin}}}~ L_i(\theta_{i,k}^t) = \alpha f_i(\theta_{i,k}^t
) + \frac{\beta}{2}\|\theta_{i,k}^t-(\theta^t-\lambda\Delta^t)\|^2,
\label{eq:local_objective}
\end{equation}
{where $\alpha$ and $\beta$ control the relative importance of individual terms.}
After $K$ local updates, each client uploads their trained model $\theta_{i,K}^t$ to the server, and then the server constructs the next server model $\theta^{t+1}$ as shown in~\cref{alg:proposed_method}.

\setlength{\textfloatsep}{10pt}
\begin{algorithm}[t]
\caption{FedAvg}
\label{alg:fedavg}
\SetAlgoLined

\KwIn{$\theta^0$, number of clients $N$, number of local iterations $K$, number of communication rounds $T$}
\For{$\text{each round}~t = 0, \dots ,T$}{
Sample subset of clients $S_t \subseteq \{1, \dots, N\}$

Server sends $\theta^t$ to all clients $i \in S_t$

\For{$\text{each client}~ i \in S_t,~\textbf{in parallel}$}{
\For{$k = 0, \dots ,K$}{
{${\underset{\theta_{i,k}^t}{\operatorname{argmin}}}\ f_i(\theta_{i,k}^t
)$}
}
Client sends  $\theta_{i,K}^t$ back to the server}
\textbf{In server:}

\quad $\theta^{t+1}$ = $\frac{1}{|S_t|} \Sigma_{i \in S_t}\theta_{i,K}^{t}$
}
\textbf{Return} ~ $\theta^{t+1}$
\end{algorithm}

Since we define the server model update direction as $\Delta^{t+1}=-(\theta^{t+1}-\theta^t)$,
the following Lemma~\ref{thm:delta_momentum} holds.
\begin{lemma}
\label{thm:delta_momentum}
Let the averaged client's amount of local updates be $\nabla (\theta^t) =-\frac{1}{|S_t|}\underset{i \in S_t}{\Sigma}(\theta_{i,K}^{t}-(\theta^t-\lambda\Delta^t))$. Then $\Delta^t$ is the exponential moving average of       ~$\nabla (\theta^t)$, i.e.
\begin{equation*}
    \Delta^{t+1}=\tau\nabla(\theta^t)+\lambda\Delta^t.
\end{equation*}
\end{lemma}

\begin{myproof}
\begin{align*}
\Delta^{t+1}&=-(\theta^{t+1}-\theta^{t})
           \\ &= -\frac{\tau}{|S_t|} \Sigma_{i \in S_t}\theta_{i,K}^{t} - (1-\tau) (\theta^t-\lambda\Delta^t)+\theta^t
           \\ &= -\frac{\tau}{|S_t|} \Sigma_{i \in S_t}(\theta_{i,K}^{t} - (\theta^t-\lambda\Delta^t)) + \lambda\Delta^t
           \\&=\tau\nabla(\theta^t) + \lambda \Delta^t
\end{align*}
\end{myproof}

Lemma~\ref{thm:delta_momentum} implies that $\Delta^t$ is an exponential moving average of the total local updates computed at the projected point with the decay coefficient $\lambda$ and server learning rate $\tau$.
Although only a subset of clients participate\revision{s} in each communication round, $\Delta^t$ maintains past global gradient information, which serves as an approximation to the gradient of the global loss function $f(\theta)$.

Therefore, we can view integrating $\lambda \Delta^t$ to the global model $\theta^t$ at server-to-client transmission as taking a lookahead through the interim parameters where the accumulated global velocity will lead the global model.
This anticipatory update in client side leads each client to find a local minima adjacent to the trajectory of the global gradients, which helps FedAGM avoid inconsistent local updates.
\begin{algorithm}[t]
\caption{FedAGM}
\label{alg:proposed_method}
\SetAlgoLined
\KwIn{$\alpha$, $\beta$, $\lambda$, $\tau$, $\theta^0$, number of clients $N$, number of local iterations $K$, number of communication rounds $T$}
Initialize $\Delta^{0}=0$

 \For{$\text{each round}~t = 0, \dots ,T$}{
  Sample subset of clients $S_t \subseteq \{1, \dots, N\}$

Server sends $\theta^t-\lambda\Delta^t$ to all clients $i \in S_t$

\For{$\text{each client}~ i \in S_t,~\textbf{in parallel}$}{
Set $\theta_{i,0}^t=\theta^t-\lambda\Delta^t$

\For{$k = 0, \dots ,K$}{
{${\underset{\theta_{i,k}^t}{\operatorname{argmin}}}~ L_i(\theta_{i,k}^t) = \newline
\hspace*{3.33em} \alpha f_i(\theta_{i,k}^t
) + \frac{\beta}{2}\|\theta_{i,k}^t-(\theta^t-\lambda\Delta^t)\|^2$  }
}
Client sends  $\theta_{i,K}^t$
back to the server}
\textbf{In server:}

\quad $\theta^{t+1}$ = $\frac{\tau}{|S_t|} \Sigma_{i \in S_t}\theta_{i,K}^{t} + (1-\tau) (\theta^t-\lambda\Delta^t)$

\quad $\Delta^{t+1}=-(\theta^{t+1}-\theta^{t}$)
}
\textbf{Return} ~$\theta^{t+1}$
\end{algorithm}
{\subsection{Local regularization with global momentum}}

{In addition to the initial point acceleration for local training, our proposed local objective function in~\cref{eq:local_objective} also takes advantage of the global gradient information to further align the gradients of individual clients.}
In detail, due to the regularization term in the local objective function, the local update direction is as follows:
\begin{equation}
\begin{aligned}
\nabla L_{i}(\theta_{i, k}^{t}) &=\alpha \nabla f_{i}(\theta_{i, k}^{t})+\beta(\theta_{i, k}^{t}-(\theta^{t}-\lambda \Delta^{t})) \\
&=\alpha \nabla f_{i}(\theta_{i, k}^{t}) +\beta \lambda \Delta^{t} +\beta (\theta_{i, k}^{t}-\theta^{t}) \\
&\approx(\alpha + \beta\lambda) \nabla f_{i}(\theta_{i, k}^{t}) +\beta \lambda (\nabla f(\theta) - \nabla f_{i}(\theta_{i, k}^{t})) \\
&\quad +\beta (\theta_{i, k}^{t}-\theta^{t}).
\end{aligned}
\end{equation}
This implies that FedAGM corrects the local gradient toward global gradient direction at every local gradient step, which also prevents each client from falling into its own biased minima.

\subsection{Discussion}

While our formulation is related to existing works which also handle client heterogeneity by employing global gradient information for the local update, FedAGM has the following major advantages.
First, the server and clients only communicate model weights without imposing additional network overhead for transmitting gradients and other  information~\cite{karimireddy2019scaffold, xu2021fedcm}.
Note that the increase in communication cost challenges many realistic federated learning applications involving clients with limited network bandwidths.
Also, FedAGM does not require the server to compute or maintain any historical information of the model, which leads to extra saving of computational cost in the server.
Second, FedAGM is robust to the low-rate client participation situations and allows new-arriving clients to join the training process immediately without warmup because, unlike~\cite{karimireddy2019scaffold, acar2020federated, li2021model}, the clients are not supposed to store their local states.

\iffalse
\paragraph{\revision{[Advantages of not modifying local objective function]}}
Moreover, while several works~\cite{karimireddy2019scaffold, xu2021fedcm} correct the direction of local update by modifying local SGD procedure, repetitive correction with the constant term for every iteration can impose overshooting toward the global gradient which can be noisy in partial participation scenario.
In contrast, in FedAGM, we modifies the initial point of local update in direction of the global information only once, which impose more flexibility for local update.
\fi

\iffalse

as linear combination of averaged parameters received from clients $\frac{1}{|S_t|} \Sigma_{i \in S_t}\theta_{i,K}^{t}$ and the parameters distributed to clients.

For the regularization term, is to penalize
in model communication process where
The server aggregates the gradient information of previous active clients and sends the projected model $\theta^t - \lambda \Delta^t$ to the client instead of $\theta^t$.
\iffalse
\jkkim{The main modification in FedAGM compared to FedAvg lies in the model that the server broadcasts to the client.
Unlike FedAvg passing the current global model($\theta^t)$, FedAGM passes the accelerated global model($\theta^t-\lambda\Delta^t)$ where $\Delta^t$ is global update in the previous step and $\lambda$ is hyperparameter.
}
\fi
\jkkim{Starting with the projected parameter as an initial point, clients active in current round take local steps to minimize their local objective functions.}
After $K$ local updates, each client returns their parameters $\theta_{i,K}^t$. The server calculates the next model as linear combination of averaged parameters received from clients $\frac{1}{|S_t|} \Sigma_{i \in S_t}\theta_{i,K}^{t}$ and the parameters distributed to clients.
We define server model update direction as $\Delta^{t+1}=-(\theta^{t+1}-\theta^t)$.
Then the following~\cref{thm:delta_momentum} holds.

Lemma 4.1 means that $\Delta^t$ is an exponential moving average of the total local updates computed at the projected point with the decay coefficient $\lambda$.
Although only a subset of clients participate in each communication round, past local update information included in $\Delta^t$ is maintained.
Updating  the  current  model  using  it  can have the similar effect as increasing the participation rate which makes FedAGM be robust to partial client participation in federated learning.
Therefore, since the momentum term $\Delta^t$ serves as an approximation to the gradient of the global loss function $f(\theta)$, we have the local update direction as follows:
\begin{equation}
\begin{aligned}
\nabla L_{i}(\theta_{i, k}^{t}) &=\alpha \nabla f_{i}(\theta_{i, k}^{t})+\beta(\theta_{i, k}^{t}-(\theta^{t}-\lambda \Delta^{t})) \\
&=\alpha \nabla f_{i}(\theta_{i, k}^{t}) +\beta \lambda \Delta^{t} +\beta (\theta_{i, k}^{t}-\theta^{t}) \\
&\approx(\alpha + \beta\lambda) \nabla f_{i}(\theta_{i, k}^{t}) +\beta \lambda (\nabla f(\theta) - \nabla f_{i}(\theta_{i, k}^{t})) \\
&\quad +\beta (\theta_{i, k}^{t}-\theta^{t})
\end{aligned}
\end{equation}

\iffalse
\paragraph{\revision{[Correct local update with global information $\Delta^t$ without communication cost]}}
\jkkim{
If each client knows the global update information and utilizes it for local update, it can be expected that client-drift due to data heterogeneity will be reduced. At first glance, this method seems to be possible only by communicating an additional message. However, the following lemma suggests that there is a detour.}
\begin{lemma}
\label{local correction}
Let the model that the $i$th client undergoes the $k$th local iteration be $\theta_{i,k}^t$. Then the local model is updated in the following way.
\begin{equation*}
    \theta_{i,k+1}^t=\theta_{i,k}^t-\eta_l\alpha \nabla f_i(\theta_{i,k}^t)  -\eta_l \beta(\theta_{i,k}^t-\theta^t)+ \eta_l\beta \lambda \Delta^t
\end{equation*}
\end{lemma}
\begin{myproof}
Since $i$th client's local objective is \\
$L_i(\theta_{i,k}^t)=\alpha f_i(\theta_{i,k}^t
)+\frac{\beta}{2}\|\theta_{i,k}^t-(\theta^t+\lambda\Delta^t)\|^2$,\\
Its gradient is obtained as follows.
\begin{align*}
\nabla L_i(\theta_{i,k}^t) &=\alpha\nabla f_i(\theta_{i,k}^t
)+\beta(\theta_{i,k}^t-(\theta^t+\lambda\Delta^t))\\
&=\alpha\underbrace{\nabla f_i(\theta_{i,k}^t
)}_\text{local loss func}+\beta\underbrace{(\theta_{i,k}^t-\theta^t)}_\text{prox}-\underbrace{\beta\lambda\Delta^t}_\text{global momentum}
\end{align*}
\end{myproof}
\fi
This implies that FedAGM has a correction toward global gradient direction with proximal term, which prevent each client from falling into its biased minima.

\jkkim{
\iffalse
This suggests a surprising fact. We can use global information for local update correction without communicating messages bigger than the model size.
\fi
\iffalse
This is effective in improving performance in a federated setting.
\fi
}
\iffalse
Moreover, $\Delta^t$ contains the update information of clients that participated in the past in the form of an exponential moving average, updating the current model using it \iffalse in this direction \fi can have the similar effect as increasing the participation rate.\fi
Furthermore, you can view adding $\lambda \Delta^t$ to current model $\theta^t$ when sending the model for local update as taking a lookahead through the interim parameters where the accumulated global velocity will lead the current model.
If the velocity update leads to bad client loss, then the client will make a correction to minimize their loss properly.
This anticipatory update in client side allow the search to slow down when approaching the local optima and reduce the likelihood of going too far by the added global velocity, which helps FedAGM avoid oscillations by client heterogeneity and achieves better performance.

\fi

\section{Discussion}
\label{sec:discussion}
\iffalse
\subsection{Comparision with NAG}
Let $x^t:=\theta^t+\lambda \Delta^t$ and $\nabla (\theta^t) := \theta^{t+1} - x^t$. Then $\Delta^{t+1} = \tau\nabla(\theta^t) + \lambda \Delta^t$. After $t$ communication round we have:

\begin{equation}
\begin{aligned}
x^{t+1}&=\theta^{t+1}+\lambda \Delta^{t+1}
\\&=x^t +\tau\nabla(\theta^t)+\lambda(\tau\nabla(\theta^t) + \lambda \Delta^t)
\\&
=x^t+\tau\nabla(\theta^t) +\tau\underset{i=0}{\overset{t}{\Sigma}}\lambda^{i+1}\nabla(\theta^{t-i})
\end{aligned}
\end{equation}

FedAGM is a federated algorithm. But, if we have N=1 as total client who always participate and do single local update, then we recover the Nesterov Accelerated momentum, which is widely used in centralized learning.
\fi
\section{Conclusion}
\label{sec:conclusion}

This paper tackles a realistic federated learning scenario, where a large number of clients with heterogeneous data and limited participation constraints hurt the convergence and performance of the model.
To address this problem, we proposed a novel federated learning framework, which naturally aggregates previous global gradient information and incorporates it to guide client updates.
The proposed algorithm transmits the global gradient information to clients without additional communication cost by simply adding the global information to the current model when broadcasting it to clients.
We showed that the proposed method is desirable with the realistic federated learning scenarios since it does not require any constraints such as communication or memory overhead.
We demonstrate the effectiveness of the proposed method in terms of robustness and communication-efficiency in the presence of client heterogeneity through extensive evaluation on multiple benchmarks.

\iffalse
\vspace{-0.2cm}
\paragraph{Limitations}
A drawback of this work is simplicity, which may undermine the novelty of this paper.
However, simplicity has rather many positive aspects in terms of efficiency and reproducibility, and we identified many practical benefits of our approach.

\vspace{-0.3cm}
\paragraph{Negative social impact}

While a large number of edge devices participate to train a single model in federated learning, there is no security consideration in federated learning, which may be susceptible to adversarial clients.
To mitigate this issue, a central server needs mechanisms to monitor adversarial clients.
\fi

\section{Preliminaries}
\label{sec:preliminary}

\subsection{Problem setting and notations}
The goal of federated learning is to construct a single model that minimizes the following objective function.
\begin{equation}
\label{global_objective}
    \underset{\theta\in \mathbb{R}^d}{\operatorname{argmin}}\left[f(\theta)=\frac{1}{N}\underset{i=1}{\overset{N}{\Sigma}}f_i(\theta)\right],
\end{equation}
where $f_i(\theta)=\mathbb{E}_{z \sim D_i}[f_i(\theta,z)]$ is the loss function of the $i$-th client.
Note that clients may have heterogeneous data distributions, but communication of training data between clients and the central server is strictly prohibited due to privacy concerns.
All the basic notations throughout this paper are listed in~\cref{tab:notation}.

\begin{table}[t]
\centering
\caption{Summary of notation used in the paper}
\scalebox{0.9}{
\begin{tabular}{@{}cl@{}}
\hline
$N$ & total number of clients \\
$T$ & total number of communication rounds \\
$S_t$ & sampled number of clients in round $t$\\
$K$ & total number of local iterations \\
$i$ & client index \\
$D_i$ & data distribution of $i$th client \\
$\theta^t$ & shared model parameters in round $t$\\
 $\theta_{i,k}^t$ & local model of $i$th client in round $t$ and step $k$  \\
\hline
\end{tabular}}
\label{tab:notation}
\end{table}

\subsection{FedAvg algorithm}

A standard algorithm for solving federated learning tasks is FedAvg~\cite{mcmahan2017communication}, as described by the pseudo-code in~\cref{alg:fedavg}.
Specifically, a central server sends a common model $\theta^t$ to the clients.
Each client performs several steps of gradient descent to minimize its local loss function then returns the resultant model parameters $\theta_{i,K}^t$.
A new common model for the next round training is constructed by averaging the models sent by all participants in the current round.

While the property of taking multiple local updates in FedAvg before aggregation cuts down the communication cost required for training, in practice, this property leads to the so-called client drift~\cite{karimireddy2019scaffold} issue, where the individual client updates do not align well due to over-fitting on the local client data.
This phenomenon inhibits FedAvg from converging to the optimum of the average loss over all clients.

\section{Related work}
\label{sec:related_work}

Federated learning was first proposed in McMahan~\etal~\cite{mcmahan2017communication}, which introduces the key properties of federated learning as non-iid client data, massively distribution, and partial participation, and then proposes FedAvg algorithm as a solution.
Several works explore the negative influence of heterogeneity in federated learning empirically~\cite{zhao2018federated} and derive convergence
rates depending on the amount of heterogeneity~\cite{li2018federated, wang2019adaptive, khaled2019first, li2019convergence, hsieh2020non, wang2020tackling}.
In this work, we focus on the problem of non-iidness, also known as client statistical heterogeneity where participating clients have different data distributions.

To improve FedAvg in the presence of heterogeneous clients, there is a long line of work which penalize local models not to drift towards their local minima by regularizing the local objectives.
These approaches regularize the client loss function with a proximal term~\cite{li2018federated} or use primal-dual approaches~\cite{zhang2020fedpd, acar2020federated}.
In activation regularization perspective, client update is regularized to have similar activation to the downloaded global model by contrastive learning~\cite{li2021model}, using mixup with global statistics~\cite{yoon2021fedmix}, and using generative models~\cite{zhu2021data}.
There is another line of work which reduce inter-client variance to eliminate inconsistent update across clients.
several approaches use control variates~\cite{karimireddy2019scaffold, liang2019variance, karimireddy2020mime, li2019feddane} or global gradient momentum~\cite{xu2021fedcm} to reduce biases in client update.
\cite{khanduri2021stem, das2020faster} apply STORM algorithm~\cite{cutkosky2019momentum} to reduce variance caused by both server-level and client-level SGD procedures.
Another way to de-bias client updates is to estimate the global posterior using local posterior sampling by running Markov chain Monte Carlo (MCMC) on client side~\cite{al2020federated}.
However, most of these methods require full participation, additional communication cost or client storage, which can be problematic in realistic federated learning tasks.

On the other hand, several works incorporate momentum~\cite{wang2019slowmo, hsu2019measuring} and adaptive gradient methods~\cite{reddi2021adaptive} in server optimization to accelerate the convergence of the federated learning.
While these methods only involve server-level optimization, our method incorporates the momentum of global gradient information into both server- and client-level optimization.
\section{Introduction}
\label{sec:introduction}

Federated learning, first introduced in~\cite{mcmahan2017communication}, is an emerging large-scale machine learning framework that a central server learns a shared model without direct observation of training examples using a large number of remote clients with separate datasets.
This decentralized learning concept allows the federated learning to achieve the basic level of data privacy.
Since the remote clients such as mobile or IoT devices have limited communication bandwidth, federated learning algorithms are particularly sensitive to the communication cost.

FedAvg~\cite{mcmahan2017communication} is known as the baseline algorithm of federated learning.
In FedAvg, a subset of the clients updates their models based on a gradient descent method using their local data and then upload their resultant models to the server for updating the global model parameters via model averaging.
As discussed in the extensive analysis on the convergence of FedAvg~\cite{stich2018local, yu2019parallel, wang2021cooperative, stich2019error, basu2020qsparse}, multiple local updates conducted before aggregation greatly reduce the communication cost that is required for training a model in the sever.

However, federated learning faces two key challenges: high heterogeneity in the data distributed over clients and limited participation rate, so-called cross-device setting~\cite{kairouz2019advances}, where only a subset of clients is active at a given time due to the limited communication bandwidth.

Several studies~\cite{zhao2018federated, karimireddy2019scaffold} have shown that multiple local updates in the clients with non-iid data lead to client-drift, diverging updates in the individual clients.
Such a phenomenon introduces high variance associated with the averaging step of FedAvg for the global update, which hampers the convergence to the optimal average loss over all clients~\cite{li2018federated, wang2019adaptive, khaled2019first, li2019convergence, hsieh2020non, wang2020tackling}.
The client-drift challenge is exacerbated when the participation rate per communication round is low due to unreliable connections between the server and the clients.

\iffalse
This phenomenon are exacerbated by low participation rate per communication round due to unreliable connection link between the server and the clients.
\fi

\iffalse
To properly handle client heterogeneity, we propose a novel federated learning algorithm, Federated Averaging with Acceleration of Global Momentum (FedAGM), which close the gap between local and global loss function by conveying momentum of global gradient information to clients through simply adding it to the current global and incorporating the momentum into client local updates.
Instead of aggregating previous global gradient information in server and transmitting it with the current model to clients, FedAGM automatically carry out the update and transmission of momentum term to client updates through gradient acceleration in server-to-client communication procedure.
\fi

To properly address the client heterogeneity issue, we propose a novel optimization algorithm for federated learning, Federated Averaging with Acceleration of Global Momentum (FedAGM)), which conveys the momentum of global gradient information to clients and enables the momentum to be incorporated into the local updates in the individual clients.
This approach turns out to be effective for reducing the gap between the global and local losses.
Contrary to the existing methods that require additional steps to compute the momentum, FedAGM transmits the global model integrated with the momentum and saves the extra communication and computational cost consequently.
In addition, FedAGM incorporates a regularization term in the objective function of clients to make the global and local gradients more consistent.
FedAGM incurs no extra computation and communication costs on both server and client sides.

\iffalse
, where server stores statistical information about previous global update information and
\jkkim{
Server stores statistical information about previous global update information.
Using this, clients can accelerate their initial point in advance to point supposed to be on the global model trajectory.
Each client is regularized on every local iteration to reduce the weight $L_2$ norm between the local online model and the initial point.
This allows them to receive local corrects consistent with the global update direction and reduce variance between clients local update.
}
\fi
\iffalse
The server stores momentum which accumulates previous global update information.
Then it sends the accelerated model on the predicted trajectory using momentum to each client.
Each client gets local correct at every iteration to find a local minima near the predicted trajectory.
Each client is regularized to find a local minima near the predicted global trajectory at every iteration. \fi

\iffalse
To properly address client heterogeneity issue, we propose a novel optimization algorithm for federated learning, Federated Averaging with Acceleration of Global Momentum (FedAGM), which conveys the momentum of global gradient information to clients
and enables the momentum to be incorporated into the local updates in the individual clients.
This approach turns out to be effective for reducing the gap between the local and global losses.
Contrary to the existing methods that require additional step to computes the momentum, FedAGM transmits the global model integrated with the momentum and saves the extra communication and computational cost consequently.
\bhhan{[client side regularization]}
\jkkim{
We propose to regularize the weight $L_2$ norm with the accelerated point in each client local objective.
Global momentum accelerated at the server stage jumps out in the gradient and acts as a local correction term in each local updates.
Local correction term regularizes each client to find a local minima adjacent to the direction of the global optimum instead of falling to the nearest local minima which are distinct to the global optimum, which helps to handle client-drift phenomena.
}
\fi

Although there have been a growing number of works that handle the client heterogeneity in federated learning, FedAGM has the following major advantages.
Unlike previous methods focusing on the strategies for either client-level optimization~\cite{xu2021fedcm, acar2020federated, karimireddy2019scaffold, li2021model, li2018federated, zhang2020fedpd, karimireddy2020mime, li2019feddane, yang2020federated, liang2019variance} or server-level updates~\cite{reddi2021adaptive, wang2019slowmo, hsu2019measuring}, FedAGM incorporates the momentum based on the global gradient information into both server and client updates.
This feature allows the proposed method to achieve the same level of task-specific performance with fewer communication rounds.
Moreover, while most of existing methods require additional requirements compared to FedAvg such as full participation~\cite{liang2019variance, zhang2020fedpd, khanduri2021stem}, additional communication bandwidth~\cite{xu2021fedcm, karimireddy2019scaffold, zhu2021data, karimireddy2020mime, li2019feddane, das2020faster}, or storage costs on clients to store local states~\cite{acar2020federated, karimireddy2019scaffold, li2021model}, FedAGM is completely free from any additional communication and memory overhead, which ensures the compatibility with large-scale and low-participation federated learning scenarios.
The main contributions of this paper are summarized as follows.
 \begin{itemize}

	\item[$\bullet$] We propose a communication-efficient federated optimization algorithm, which deals with heterogeneous clients effectively.
	 The proposed approach computes and transmits the global model with a momentum efficiently, which facilitates the optimization in clients.

	\iffalse \jkkim{by approximating the global model trajectory with a past global update history and searching the local minima around it.}\fi

	\iffalse computing the global model with a momentum efficiently and incorporating it into both server- and client-level optimizations. \fi

	\item[$\bullet$] We also revise the objective function of clients, which augments a regularization term to the local gradient direction, which further aligns the gradients of the server and the individual clients.

	 \item[$\bullet$] We show that the proposed approach does not require any additional communication cost and memory overhead, which is desirable for the real-world settings of federated learning.

	\item[$\bullet$] Through extensive evaluation on multiple benchmarks, we demonstrate that our optimization technique is communication-efficient and robust to client heterogeneity, especially when the participation ratio is low.

\end{itemize}

The rest of this paper is organized as follows.
We first review the core algorithmic idea of federated learning, FedAvg, and its variants in~\cref{sec:related_work,sec:preliminary}.
Then, we formally describe the proposed federated learning framework and demonstrate the effectiveness of our method in~\cref{sec:method,sec:experiment}. Finally, we conclude the paper in~\cref{sec:conclusion}.

\section{Experiments}
\label{sec:experiment}

This section presents empirical evaluations of FedAGM and competing federated learning methods, to highlight the robustness to data heterogeneity of the proposed method in terms of performance and communication-efficiency.

\subsection{Experimental setup}

\paragraph{Datasets and baselines}
We conduct a set of experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet\footnote{\url{https://www.kaggle.com/c/tiny-imagenet}}~\cite{le2015tiny} with various data heterogeneity levels and participation rates.
Note that Tiny-ImageNet ($200$ classes with $10,000$ samples) is more natural and realistic compared to the simple datasets, such as MNIST and CIFAR, used for evaluation of many previous methods~\cite{mcmahan2017communication, karimireddy2019scaffold}.
We generate IID data split by randomly assigning training data to individual clients without replacement.
For the non-IID data, we simulate the data heterogeneity by sampling the label ratios from a Dirichlet distribution with parameter \{0.3, 0.6\}, following~\cite{hsu2019measuring}.
We keep the training data balanced, so each client holds the same amount of data.

We compare our method, FedAGM, with several state-of-the-art federated learning techniques, which include FedAvg~\cite{mcmahan2017communication}, FedProx~\cite{li2018federated}, FedAvgm~\cite{hsu2019measuring}, FedAdam~\cite{reddi2021adaptive}, FedDyn~\cite{acar2020federated}, FedCM~\cite{xu2021fedcm}.
We adopt a standard ResNet-18~\cite{he2016deep} as backbone network for all benchmarks, but we replace batch normalization by group normalization as suggested in~\cite{hsieh2020non}.

\begin{table*}[t!]
\centering
\caption{Comparison of FedAGM with baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet with a moderate number of clients. In these experiments, the number of clients, participation rate, and Dirichlet parameter are set to 100, 5\%, and $0.3$, respectively. Accuracy at target round and the communication round to reach target test accuracy is based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.}
\vspace{-0.2cm}
\label{tab:clients100_prate5}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{3}{*}{Method}                                   & \multicolumn{4}{c}{CIFAR10}                                                                     & \multicolumn{4}{c}{CIFAR100   }                                                                & \multicolumn{4}{c}{Tiny-ImageNet}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & \multicolumn{1}{c}{500R}           & \multicolumn{1}{c}{1000R}     & 84\%         & 87\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 47\%         & 53\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{800R}            & \multicolumn{1}{c}{35\%} & \multicolumn{1}{c}{38\%}                     \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 74.36 & 82.53 & 840 & 1000+ & 41.88  & 47.83  & 924 & 1000+ & 33.94 & 35.37 & 645 & 1000+ \\
FedProx~\cite{li2018federated}         & 73.70 & 82.68 & 826 & 1000+ & 42.43  & 48.32  & 881 & 1000+ & 34.14 & 35.53 & 613 & 1000+ \\
FedAvgm~\cite{hsu2019measuring}        & 80.56 & 85.48 & 519 & 828 & 46.98  & 53.29  & 515 & 936 & 36.32 & 38.51 & 416 & 829 \\
FedAdam~\cite{reddi2021adaptive}       & 73.18 & 81.14 & 878 & 1000+  & 44.80  & 52.48  & 691 & 1000+ & 33.22 & 38.91 & 658 & 945 \\
FedDyn~\cite{acar2020federated}        & {\bf85.67} & 88.84 & 340 & 794  & 52.66  & 58.46  & 293 & 504 & 38.86 & 42.82 & 327 & 440  \\
FedCM~\cite{xu2021fedcm}               & 78.92 & 83.71 & 624 & 1000+ & 52.44  & 58.06  & 293 & 572  & 31.61 & 37.87 & 694 & 1000+\\
FedAGM (ours)                          & 85.13 & {\bf89.10} & {\bf326} & {\bf450} & {\bf55.79}  & {\bf62.51}  & {\bf260} & {\bf389} & {\bf42.26}& \bf{46.31} & {\bf226} & {\bf331}\\
\cline{1-13}
\end{tabular}}
\vspace{0.3cm}
\end{table*}
\begin{table*}[t!]
\centering
\caption{Comparison of FedAGM with baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet with a large number of clients. In these experiments, number of clients, participation rate, and Dirichlet parameter are set to 500, 2\%, and $0.3$, respectively.
}
\vspace{-0.2cm}
\label{tab:clients500_prate2}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{3}{*}{Method}                   & \multicolumn{4}{c}{CIFAR10}                                                                     & \multicolumn{4}{c}{CIFAR100   }                                                                & \multicolumn{4}{c}{Tiny-ImageNet}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 73\%         & 77\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 36\%         & 40\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{800R}            & 24\%         & 30\%                     \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 58.74 & 71.45 & 1000+ & 1000+ & 30.16& 38.11& 842& 1000+ & 23.63 &29.48 & 523  & 1000+ \\
FedProx~\cite{li2018federated}& 57.88 & 70.75 & 1000+ & 1000+ & 29.28 & 36.16 & 966 & 1000+ &25.45  &31.71 &445 &799 \\
FedAvgm~\cite{hsu2019measuring} & 65.85 & 77.49 & 753  & 959 & 31.80 & 40.54 & 724 & 955 & 26.75 & 33.26 & 386 & 687 \\
FedAdam~\cite{reddi2021adaptive}  & 61.53 & 69.94 & 1000+ & 1000+  & 24.40 & 30.83 & 1000+ & 1000+ &21.88 &28.08  & 648& 1000+ \\
FedDyn~\cite{acar2020federated}  & 69.14 & 82.56 & 600 & 719  & 33.52 & 45.01 & 576  & 714 & 27.74 & 37.25 & 387 & 580 \\
FedCM~\cite{xu2021fedcm} & 69.27  & 76.57 & 742  & 1000+ & 27.23 & 38.79 & 872 & 1000+ &19.41 &24.09 & 975 & 1000+ \\
FedAGM (ours)       & \textbf{76.50} & \textbf{83.93} & \textbf{410} & \textbf{519} & \textbf{35.68} & \textbf{48.40}  & \textbf{505} & \textbf{616} &\textbf{31.47} &\textbf{38.48} & \textbf{246} & \textbf{447}\\
\hline
\end{tabular}}
\vspace{-0.1cm}
\end{table*}

\paragraph{Validation metrics}
To evaluate the generalization performance of the methods, we use the entire test set in the CIFIAR-10~\cite{krizhevsky2009learning}, CIFAR-100~\cite{krizhevsky2009learning}, and Tiny-ImageNet datasets.
Since both the speed of learning as well as the final performance are important quantities for federated learning, we measure: (i) the performance attained at a specified number of rounds, and (ii) the number of rounds needed for an algorithm to attain the desired level of target accuracy, following~\cite{al2020federated}.
For methods that could not achieve aimed accuracy within the maximum communication round, we append the communication round with a $+$ sign.
We also report the communication savings of FedAGM compared to each method in parenthesis.
Note that all the comparison methods except FedCM have the same communication cost per round whereas FedCM costs twice than other methods for server-to-client communication due to additional transmission of global gradient information.

\paragraph{Implementation details}

We use PyTorch~\cite{paszke2019pytorch} to implement FedAGM and the other baselines.
We follow~\cite{acar2020federated, xu2021fedcm} for evaluation protocol.
For local update, we use the SGD optimizer with a learning rate $0.1$ for all approaches on the three benchmarks.
We apply exponential decay on the local learning rate, and the decay parameter is selected from $\{1.0, 0.998, 0.995\}$.
We apply no momentum for local SGD, but apply weight decay of $0.001$ to prevent overfitting.
We also use gradient clipping to increase the stability of the algorithms.
The number of local training epochs over each client update is set to 5, and the batch size is set so that the total iteration for local updates is set to $50$ for all experiments.
We set the global learning rate as $1$ for all methods except for FedADAM which is set to $0.01$.
We list the details of the hyperparameters specific to FedAGM and the baselines in the supplementary materials.

\iffalse
The SGD weight decay is set to 0.00001 and the SGD momentum is set to 0.9. The batch size is set to 64. The number of local epochs is set to 300 for SOLO. The number
of local epochs is set to 10 for all federated learning approaches unless explicitly specified. The number of communication rounds is set to 100 for CIFAR-10/100 and 20
for Tiny-ImageNet, where all federated learning approaches
have little or no accuracy gain with more communications.
For MOON, we set the temperature parameter to 0.5 by default like [3].
\fi

\begin{figure*}[t]
\centering
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar10_mid_100.pdf}
\caption{5\% participation, 100 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar10_low_500.pdf}
\caption{2\% participation, 500 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar10_low_100.pdf}
\caption{1\% participation, 100 clients}
\end{subfigure}
\caption{
The convergence plot of FedAGM and comparison methods on CIFAR-10 with different client heterogeneity.
}
\label{fig:moderate_curve}
\end{figure*}

\begin{figure*}[t]
\centering
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar100_mid_100.pdf}
\caption{5\% participation, 100 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar100_low_500.pdf}
\caption{2\% participation, 500 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.32\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/cifar100_low_100.pdf}
\caption{1\% participation, 100 clients}
\end{subfigure}
\caption{
The convergence plot of FedAGM and comparison methods on CIFAR-100 with different client heterogeneity.
}
\label{fig:massive_curve}
\end{figure*}

\subsection{Evaluation on a moderate number of clients}

We first present the performance of the proposed approach, FedAGM, on CIFAR-10, CIFAR-100, and Tiny-ImageNet in comparison to baseline methods for a moderate-scale federated learning setting.
This setting has 100 devices while keeping the participation rate constant per round.

\cref{tab:clients100_prate5}~provides a detailed comparison of FedAGM to the baselines in terms of the attained performance and the speed of learning on the three benchmark tasks.
\cref{tab:clients100_prate5}~shows that FedAGM outperforms competitive methods in most cases.
In particular, FedAGM outperforms the momentum-based methods which incorporate momentum in either the server-side update (FedAvgM, FedAdam) or the client-side update (FedCM).
This is partly because, in FedAGM, the accelerated global model enables each client to lookahead the global update trajectory and find local minima near the global gradient trajectory, which aligns client updates.
Note that since FedCM requires twice the communication cost as the server transmits the current model the associated momentum at each server-to-client communication.
\subsection{Evaluation on a large number of clients}

To further validate the effectiveness of the proposed method in handling client heterogeneity, we perform experiments with a large number of clients, which are more realistic federated learning scenarios.
In this setting, since the total number of clients is increased by 5 times compared to that of the previous experiments,
the number of training data that one client has is reduced by 80\%.
Specifically, each client has 100 data points in CIFAR-10 and CIFAR-100, respectively.

\begin{table}[t]
\centering
\caption{Comparison of FedAGM with baselines on CIFAR-10 for more limited participation rate (1\%). The number of clients, and Dirichlet parameter is set to 100, and $0.3$, respectively.}
\label{tab:cifar10_p_0.01}
\scalebox{0.9}{
\begin{tabular}{lcc}
\multirow{2}{*}{Method}  & accuracy (\%, $\uparrow$ ) & rounds (\#, $\downarrow$) \\
& 1000R           &  70\%  \\
\cline{1-3}
FedAvg~\cite{mcmahan2017communication}  & 64.54       & 1000+ ($>$ 1.43$\times$)   \\
FedProx~\cite{li2018federated}          & 65.47       & 1000+ ($>$ 1.43$\times$)  \\
FedAvgm~\cite{hsu2019measuring}         & 63.73       & 1000+ ($>$ 1.43$\times$)   \\
FedAdam~\cite{reddi2021adaptive}        & 69.29       & 1000+ ($>$ 1.43$\times$)    \\
FedDyn~\cite{acar2020federated}         & 72.18       & 854 (1.22$\times$) \\
FedCM~\cite{xu2021fedcm}                & 55.03       & 1000+ ($>$ 1.43$\times$)   \\
FedAGM (ours)                           & {\bf76.72}  & {\bf696}  \\
\cline{1-3}
\end{tabular}}
\end{table}
\begin{table}[t]
\centering
\caption{Comparison of FedAGM with baselines on CIFAR-100 for more limited participation rate (1\%). The number of clients, and Dirichlet parameter is set to 100, and $0.3$, respectively.
}
\label{tab:cifar100_p_0.01}
\scalebox{0.9}{
\begin{tabular}{lcc}
\multirow{2}{*}{Method}  & accuracy (\%, $\uparrow$ ) & rounds (\#, $\downarrow$) \\
& 1000R           &  40\%  \\
\cline{1-3}
FedAvg~\cite{mcmahan2017communication}  & 40.44       & 994 (1.30$\times$)   \\
FedProx~\cite{li2018federated}          & 40.16       & 995(1.30$\times$)   \\
FedAvgm~\cite{hsu2019measuring}         & 36.30       & 1000+ ($>$ 1.30$\times$)   \\
FedAdam~\cite{reddi2021adaptive}        & 18.02       & 1000+ ($>$ 1.30$\times$)   \\
FedDyn~\cite{acar2020federated}         & 42.64       & 849 ($>$ 1.11) \\
FedCM~\cite{xu2021fedcm}                & 27.16       & 1000+ ($>$ 1.30$\times$)   \\
FedAGM (ours)                           & {\bf45.15}  & {\bf768} \\
\cline{1-3}
\end{tabular}}
\vspace{+0.2cm}
\end{table}

\begin{table*}[t]
\centering
\caption{Comparison of FedAGM with baselines on CIFAR-10 with 100 clients of Dirichlet 0.6 split on 10\%, 5\%, and 1\% client participation.
The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.
}
\vspace{-0.2cm}
\label{tab:cifar10_d_0.6}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{4}{*}{Method }&\multicolumn{12}{c}{Participation rate}\\
\cline{2-13}
               & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 83\%         & 87\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 81\%         &  85\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}   & 71\%         &  75\%
                                                              \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} &83.09  & 87.15 & 497 &907 &80.56 & 85.97& 520& 832 & 62.84 &76.92 & 767 & 943 \\
FedProx~\cite{li2018federated}         &83.22  &87.33  & 494 & 855 & 80.39 & 85.53 & 524 & 889 &61.27  &75.16 &780 &994 \\
FedAvgm~\cite{hsu2019measuring}        & 86.58 & 89.70 &346 & 568& 84.65 & 87.96 & 355 & 541 & 62.67 & 75.05 & 804&998  \\
FedAdam~\cite{reddi2021adaptive}       & 84.97 & 87.59 & 340&  715 & 80.25 & 83.52 &526& 1000+ & 60.32& 75.16 &828 &952  \\
FedDyn~\cite{acar2020federated}        & \bf{89.98} & 90.78 & 244 & 303  & 80.10 & 86.47 & 551 & 826& 67.50&79.57 & 580 & 661 \\
FedCM~\cite{xu2021fedcm}               & 87.50 & 89.29 &  226 &452 & 82.84 &86.64 & 385  & 714 & 42.57 &53.75 &1000+ &1000+ \\
FedAGM (ours)                          & 89.24 & \bf{91.10} & \bf{184} & \bf{286} & \bf{87.57} &  \bf{90.56} & \bf{218} & \bf{332}& \bf{73.59}& \bf{81.50} &\bf{463} &\bf{584} \\
\cline{1-13}
\end{tabular}}
\vspace{0.3cm}
\end{table*}
\begin{table*}[t]\centering
\caption{Comparison of FedAGM with baselines on CIFAR-10 with 100 clients of IID split on 10\%, 5\%, and 1\% client participation.
The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.
}
\vspace{-0.2cm}
\label{tab:cifar10_iid}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{4}{*}{Method }&\multicolumn{12}{c}{Participation rate}\\
\cline{2-13}
               & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 84\%         & 88\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 82\%         &  86\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}            & 82\%         & 86\%                    \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 86.83 & 89.84 & 374 &600 &85.28 &88.69 & 372& 545  &77.03 & 85.06& 724 & 1000+ \\
FedProx~\cite{li2018federated}         & 86.93 & 89.71 & 359 & 606 & 84.79 &87.99  & 384 & 594 & 79.2 &86.13 &645 & 977\\
FedAvgm~\cite{hsu2019measuring}        & 88.51 & 90.23 & 256& 429&87.67&89.96  & 258 & 375 & 79.09 & 85.79 &685 & 1000+ \\
FedAdam~\cite{reddi2021adaptive}       & 89.05 & 90.87 & 204&  397 & 85.29 & 87.97 & 286 & 633 &70.78 & 83.01 &912 &1000+  \\
FedDyn~\cite{acar2020federated}        & \bf{91.57} &91.98  & 233 &  295 & 88.41 & 89.91 & 265 &355 &76.49 & 84.79& 747 & 1000+ \\
FedCM~\cite{xu2021fedcm}               &89.93  & 91.18 &  166 &296 & 87.38 & 89.65&  181 &  352&69.99 & 76.19& 1000+&1000+ \\
FedAGM (ours)                       &  91.08 &  \bf{92.26} & \bf{154} &  \bf{239} & \bf{90.57} & \bf{92.29} &  \bf{157} & \bf{295} & \bf{81.05} & \bf{87.01} & \bf{536} & \bf{892} \\
\cline{1-13}
\end{tabular}}
\end{table*}

\cref{tab:clients500_prate2}~shows the result of FedAGM and other methods on CIFAR-10 and CIFAR-100.
We first observe that the overall performance of all algorithms is lower than those with a moderate number of clients.
This is because, as the number of training data for each client decreases, each client is more likely to fall into its distinct local optimum, which intensifies client drift.
For instance, it takes FedAGM 450 rounds to achieve 84\% with 100 devices, while it takes 519 rounds to achieve 77\% with 500 devices.
A similar trend is observed for CIFAR-100 and other methods.
Nevertheless, we can observe that FedAGM outperforms the compared methods consistently on all benchmarks in terms of accuracy and communication efficiency.
This is partly because that FedAGM effectively aligns the gradients of the server and the individual clients.

\subsection{Evaluation on the low participation rate}

To validate the robustness to partial participation nature of federated learning, we simulate an extremely constrained federated learning scenario, with a very low participation level ($1\%$) on 100 clients.
\cref{tab:cifar10_p_0.01,tab:cifar100_p_0.01}~show the results of FedAGM and competitive methods on CIFAR-10 and CIFAR-100, respectively.

Although the overall accuracy at 1000 rounds is degraded for all methods due to the limited participants per round, the proposed algorithm outperforms all comparison methods on all datasets in terms of both generalization performance and communication efficiency.
Performance gaps between FedAGM and baselines are much larger than the experiments in high participation setting~(\cref{tab:clients100_prate5}).
Note that FedDyn shows a large performance drop as participation rate drops; this is mainly because it needs to store local states, which can easily get stale, and hurt the convergence of the algorithm.
In contrast, our method does not rely on the past information stored in local devices and is relatively unaffected by this issue, which validates our claim that the proposed method is robust to the low participation property of federated learning.

\subsection{Convergence plots}
Convergence plots of FedAGM and competing baselines on CIFAR-10 and CIFAR-100 under various settings are provided in~\cref{fig:moderate_curve,fig:massive_curve}.
We observe that FedAGM outperforms other strong baselines across different participation rates and heterogeneity levels.
Note that, the performance gap between FedCM and baselines is larger in 1\% participation and 100 devices.
This also validates our claim the FedAGM is robust to the low participation cases of federated learning.

\begin{table*}[t]\centering
\caption{Comparison of FedAGM with baselines on CIFAR-100 with 100 clients of Dirichlet 0.6 split on 10\%, 5\%, and 1\% client participation. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.
}
\vspace{-0.2cm}
\label{tab:cifar100_d_0.6}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{4}{*}{Method }&\multicolumn{12}{c}{Participation rate}\\
\cline{2-13}
               & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 56\%         & 60\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 54\%         &  58\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}   &  38\%         & 42\%                     \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 45.22&50.02 &1000+ &1000+  & 43.91 & 49.18& 1000+ &1000+ &29.98  & 44.11 & 744 &911   \\
FedProx~\cite{li2018federated}  & 44.43 & 49.24 & 1000+ &1000+  & 43.15 &48.45 &1000+ &  1000+      & 29.77 & 42.61 & 790 & 962 \\
FedAvgm~\cite{hsu2019measuring} & 47.27&51.80  &1000+  &1000+  &46.66  & 52.49 & 1000+ & 1000+      &28.41& 44.56 & 771 &915    \\
FedAdam~\cite{reddi2021adaptive}  & 53.74 & 58.82 & 659 & 1000+ &45.95 & 51.63 & 1000+& 1000+    &18.89  & 24.53 & 1000+&  1000+   \\
FedDyn~\cite{acar2020federated} & \bf{59.86} & \bf{62.68}  &  \bf{355} & \bf{506} & 53.74 &59.70 & 518&   776   &33.96 &47.23  &592&  768       \\
FedCM~\cite{xu2021fedcm} &58.33  &61.75 &  377 & 647 &53.75 &60.48 & 511&    740          & 19.77 & 27.20 &  1000+ & 1000+  \\
FedAGM (ours)  & 58.21 & 62.60 & 397  & 595 &\bf{58.82} &\bf{63.88} & \bf{325}&  \bf{471}                   & \bf{36.75} & \bf{49.34} & \bf{544} & \bf{672}   \\
\cline{1-13}
\end{tabular}}
\vspace{0.3cm}
\end{table*}

\begin{table*}[t]\centering
\caption{Comparison of FedAGM with baselines on CIFAR-100 with 100 clients of IID split on 10\%, 5\%, and 1\% client participation.
The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.
}
\vspace{-0.2cm}
\label{tab:cifar100_iid}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\multirow{4}{*}{Method }&\multicolumn{12}{c}{Participation rate}\\
\cline{2-13}
               & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{2-13}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 56\%         & 60\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 52\%         &  56\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}   &  42\%         & 48\%                      \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 45.38 &50.37  & 1000+ &1000+ & 43.60& 48.01& 1000+&1000+  & 35.92 & 49.15& 678 &940  \\
FedProx~\cite{li2018federated}         &44.96  & 49.16 & 1000+ & 1000+ & 43.27 & 47.23 &1000+  & 1000+ & 36.50 &48.38 &691 & 984\\
FedAvgm~\cite{hsu2019measuring}        & 47.19 & 51.48 &1000+ &1000+ & 47.43 & 52.83 & 880 & 1000+ &  39.07& 51.75 &578 &822  \\
FedAdam~\cite{reddi2021adaptive}       & 55.40 & 60.08 & 545&  997 & 52.23 & 57.73 & 496 & 835 &25.84 & 35.08 &1000+ & 1000+ \\
FedDyn~\cite{acar2020federated}  &   \bf{59.91}& 63.37      &335  &\bf{513}  &  54.88 & 60.69 & 412 &558 &34.50 & 43.91& 779 & 1000+ \\
FedCM~\cite{xu2021fedcm}               &59.37  & 63.32 &  335 &531 & 57.10 &62.48 & 288  & 466 & 22.68&34.37 & 1000+&1000+ \\
FedAGM (ours)     &  59.64&  \bf{63.50}      &\bf{321}&  523& \bf{58.23} & \bf{62.48} &  \bf{280} & \bf{391} & \bf{40.07}& \bf{52.75} & \bf{536} & \bf{769}\\
\cline{1-13}
\end{tabular}}
\vspace{-0.1cm}
\end{table*}

\subsection{Evaluation on less heterogeneous setting}
\cref{tab:cifar10_d_0.6,tab:cifar10_iid,tab:cifar100_d_0.6,tab:cifar100_iid}~show that FedAGM matches or outperforms the performance of competitive methods when data heterogeneity is not severe (Dirichlet 0.6) or absent (IID) in most cases.
Note that, while the compared methods show performance degradation as the participation rate decreases, FedAGM shows little degradation as the participation rate decreases for both benchmarks.
This implies that FedAGM is more robust for low participation rates than other baselines.
This is partly because low client heterogeneity reduces noise in the momentum of global gradient, which attributes to the smooth trajectory of global update.
Since FedAGM effectively incorporates the momentum for local updates, FedAGM is relatively unaffected by the partial participation of federated learning.

\subsection{Ablation study}

\paragraph{Sensitivity to decay coefficient $\lambda$}
As shown in our~\cref{alg:proposed_method}, $\lambda$ controls how much the server projects the current model toward the global gradient to initialize the local update.
To analyze the effect of $\lambda$ on the performance of FedAGM algorithms, we evaluate the generalization performance of FedAGM after $1000$ rounds varying $\lambda$ as \{0.75, 0.8, 0.85, 0.9, 0.95\} on CIFAR-10 with 500 clients of different data splits on $2 \%$ participation.
\cref{tab:lamb_ablation} shows that FedAGM converges well for all the selection of $\lambda$ while there is little performance drop when $\lambda$ is set to $0.95$.
We note that selecting too large $\lambda$ will harm the performance of FedAGM since large acceleration toward momentum can impose oscillation for the global optimization procedure.
Despite this, the proposed method still outperforms most existing algorithms.

\begin{table}[t]
\centering
\caption{$\lambda$ sensitivity of FedAGM for test accuracy at 1000 rounds on CIFAR-10 with large number of clients.}
\label{tab:lamb_ablation}
\scalebox{0.9}{
\begin{tabular}{cccccc}
$\lambda$       & 0.75 & 0.8 & 0.85 & 0.9 & 0.95 \\
\hline
Dirichlet (0.3) & 81.32 & 82.52 & 82.80  & 81.82 & 78.25  \\
Dirichlet (0.6) & 83.97 & 84.89 & 85.28  & 84.56 & 82.07  \\
IID             & 85.52 & 86.92 & 86.83 & 87.08 & 84.37 \\
\hline

\end{tabular}}
\end{table}
\begin{table}[t]
\centering
\caption{Effect of regularization at local objective function on CIFAR-100.
'P' and 'C' denotes participation rate and number of clients, respectively.}
\label{abl:beta}
\scalebox{0.9}{
\begin{tabular}{cccc}
\multirow{2}{*}{P / C}  & \multirow{2}{*}{Method}  & accuracy (\%, $\uparrow$ ) & rounds (\#, $\downarrow$) \\
&& 1000R           &  42\%  \\
\cline{1-4}
\multirow{2}{*}{5\% / 100}&$\beta= 0$     &    59.50      &  200 \\
&$\beta \neq 0$                         &   \bf{62.51}    &   \bf{197} \\
\cline{1-4}
\multirow{2}{*}{1\% / 100}&$\beta= 0$     &    42.95   & 914  \\
&$\beta \neq 0$                         & \bf{45.15}      & \bf{895}  \\
\cline{1-4}
\multirow{2}{*}{2\% / 500}&$\beta= 0$  & 46.80       & 736   \\
&$\beta \neq 0$                      & \bf{48.40}       & \bf{678}   \\
\cline{1-4}
\end{tabular}}
\end{table}

\paragraph{Effect of adding regularization on local objective function}

To analyze the effect of incorporating the regularization term in the local objective function, we compare the performance by setting $\beta=0$ and $\beta \neq 0$ on CIFAR-100.
\cref{abl:beta}~shows that FedAGM with the regularization term achieves better performance in terms of convergence rate and accuracy for all levels of participation.
This implies that correcting local gradients for every local update helps to mitigate inconsistent local updates between clients.

\iffalse

\begin{table*}[t]
\centering
\caption{CIFAR 10, 500 clients,0.02}
\begin{tabular}{lcccc}
\hline
\multicolumn{1}{c}{\begin{tabular}[c]{@{}c@{}}Participation \\ rate\end{tabular}} & \multicolumn{4}{c}{2\%}                                                                          \\
\hline
\multirow{2}{*}{Method \textbackslash{}@}                                          & \multicolumn{2}{c}{accuracy (\%, )}              & \multicolumn{2}{c}{rounds (\#, )}            \\
                                                                                & 500R                 & \multicolumn{1}{c}{1000R} & 73\%                 & 77\%                   \\
\hline
Ours(mu\textgreater{}0)                                                            & 73.61                & 82.80                      & 484                  & 606                    \\
Ours(mu=0)                                                                         & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}       & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}  \\
FedAvg                                                                             & 58.74                & 71.45                      & -                    & -                      \\
FedAvgm                                                                            & 65.86                & 77.49                      & 753                  & 959                    \\
FedProx                                                                            & 57.88                & 70.75                      & -                    & -                      \\
FedDyn                                                                             & 69.14                & 82.56                      & 600                  & 719                    \\
FedAdam                                                                            & 61.53                & 69.94                      & -                    & -                      \\
FedCM                                                                              & 69.27                & 76.57                      & 742                  & -                      \\
\hline
\end{tabular}
\end{table*}

\begin{table*}[t]\centering
\caption{CIFAR 100, 500 clients,0.02}
\begin{tabular}{lcccc}
\hline
\multicolumn{1}{c}{\begin{tabular}[c]{@{}c@{}}Participation \\ rate\end{tabular}} & \multicolumn{4}{c}{2\%}                                                                          \\
\hline
\multirow{2}{*}{Method \textbackslash{}@}                                          & \multicolumn{2}{c}{accuracy (\%, )}              & \multicolumn{2}{c}{rounds (\#, )}            \\
                                                                                & 500R                 & \multicolumn{1}{c}{1000R} & 36\%                 & 40\%                   \\
\hline
Ours(mu\textgreater{}0)                                                            & 34.94                & 47.35                      & 526                  & 642                    \\
Ours(mu=0)                                                                         & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}       & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}  \\
FedAvg                                                                             & 30.16                & 38.11                      & 842                  & -                      \\
FedAvgm                                                                            & 31.80                & 40.54                      & 724                  & 955                    \\
FedProx                                                                            & 29.28                & 36.16                      & 966                  & -                      \\
FedDyn                                                                             & 33.52                & 45.01                      & 576                  & 714                    \\
FedAdam                                                                            & 24.40                & 30.83                      & -                    & -                      \\
FedCM                                                                              & 27.23                & 38.79                      & 872                  & -                      \\
\hline
\end{tabular}
\end{table*}

\begin{table*}[t]\centering
\caption{Tiny-ImageNet, 500 clients, 0.02}
\begin{tabular}{lcccc}
\hline
\multicolumn{1}{c}{\begin{tabular}[c]{@{}c@{}}Participation \\ rate\end{tabular}} & \multicolumn{4}{c}{2\%}                                                                          \\
\hline
\multirow{2}{*}{Method \textbackslash{}@}                                          & \multicolumn{2}{c}{accuracy (\%, )}              & \multicolumn{2}{c}{rounds (\#, )}            \\
                                                                                & 500R                 & \multicolumn{1}{c}{1000R} & 73\%                 & 77\%                   \\
\hline
Ours(mu\textgreater{}0)                                                            & 73.61                & 82.80                      & 484                  & 606                    \\
Ours(mu=0)                                                                         & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}       & \multicolumn{1}{l}{} & \multicolumn{1}{l}{}  \\
FedAvg                                                                             & 58.74                & 71.45                      & -                    & -                      \\
FedAvgm                                                                            & 65.86                & 77.49                      & 753                  & 959                    \\
FedProx                                                                            & 57.88                & 70.75                      & -                    & -                      \\
FedDyn                                                                             & 69.14                & 82.56                      & 600                  & 719                    \\
FedAdam                                                                            & 61.53                & 69.94                      & -                    & -                      \\
FedCM                                                                              & 69.27                & 76.57                      & 742                  & -                      \\
\hline
\end{tabular}
\end{table*}

\fi

\iffalse
\begin{table*}[h]\centering
\caption{CIFAR10, Dirichlet 0.3}
\label{tab:cifar10_d_0.3}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\cline{1-13}
Participation rate                                   & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{1-13}
\multirow{2}{*}{Method } & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 84\%         & 87\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 81\%         &  85\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}   & 71\%         &  75\%
                                                              \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 77.01& 84.60 & 885 & 1000+ & 74.36 & 82.53 & 840 & 1000+ & 46.72 & 64.54 & 1000+ & 1000+  \\
FedProx~\cite{li2018federated}         & 77.20 & 83.72 & 1000+ & 1000+ & 73.70 & 82.68 & 826 & 1000+ & 47.60 & 65.47 & 1000+ & 1000+  \\
FedAvgm~\cite{hsu2019measuring}        & 83.58 & 87.72 & 527  & 889 & 80.56 & 85.48 & 519 & 828  & 46.05 & 63.73 & 1000+ & 1000+  \\
FedAdam~\cite{reddi2020adaptive}       & 80.86 & 84.61 & 777 & 1000+ & 73.18 & 81.14 & 878 & 1000+ & 53.15 & 69.29 & 976   & 1000+  \\
FedDyn~\cite{acar2020federated}        & \textbf{88.29} & 89.89 & 323  & \textbf{403} & {\bf85.67} & 89.04 & 340 & 794   & 53.59 & 72.18 & 794   & 854    \\
FedCM~\cite{xu2021fedcm}               & 84.87  & 87.75 & 376  & 789 & 78.92 & 83.71 & 624 & 1000+ & 10.00 & 10.00 & 1000+ & 1000+  \\
FedAGM (ours)                          & 87.44 & \textbf{90.10} & \textbf{289} & 458 & 85.13 & {\bf89.07} & {\bf 326} & {\bf 450}   & {\bf 64.30} & {\bf 76.72} & {\bf 616}   & {\bf 696}    \\
\cline{1-13}
\end{tabular}}
\end{table*}

\begin{table*}[h]\centering
\caption{CIFAR100, Dirichlet 0.3}
\label{tab:cifar100_d_0.3}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
\cline{1-13}
Participation rate                                   & \multicolumn{4}{c}{10 \%}                                                                     & \multicolumn{4}{c}{5 \%}                                                                & \multicolumn{4}{c}{1 \%}                                                                                \\
\cline{1-13}
\multirow{2}{*}{Method } & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
                                          & 500R           & \multicolumn{1}{c}{1000R}     & 55\%         & 59\%                            & 500R           & \multicolumn{1}{c}{1000R}    & 47\%         &  53\%                           & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R}   &  36\%         & 40\%                     \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication}   & 45.51& 49.63& 1000+& 1000+ & 41.88  & 47.83  & 924 & 1000+ & 26.28  & 40.44 & 829 & 994       \\
FedProx~\cite{li2018federated}           & 44.66 & 48.18 & 1000+ & 1000+ & 42.43  & 48.32  & 881 & 1000+ & 25.30  & 40.16 & 832 & 995       \\
FedAvgm~\cite{hsu2019measuring}          & 48.19 & 52.77 & 1000+ & 1000+ & 46.98  & 53.29  & 515 & 936  & 23.07  & 36.30 & 995  & 1000+      \\
FedAdam~\cite{reddi2020adaptive}         & 52.67 & 58.01 & 688 & 1000+ & 44.80  & 52.48  & 691 & 1000+   & 14.68  & 18.02 & 976  & 1000+      \\
FedDyn~\cite{acar2020federated}          & 57.36 & 60.79 & 398  & 625 & 52.96  & 58.22  & 293 & 504  & 30.24  & 42.64 & 645   & 849   \\
FedCM~\cite{xu2021fedcm}                 & 56.68 & 60.41 & 403 & 702 & 52.44  & 58.06  & 293 & 572  & 2.57 & 3.09  & 1000+   & 1000+        \\
FedAGM (ours)                            & \textbf{58.47} & \textbf{62.41}  & \textbf{350} & \textbf{546} & {\bf55.79}  & {\bf62.51}  & {\bf260} & {\bf389}  & {\bf34.79}  & {\bf46.03} & {\bf546}  & {\bf678}   \\
\cline{1-13}
\end{tabular}}
\end{table*}
\fi

\section{Evaluation on Low Participation Rate with a Large Number of Clients}
The low participation rate and the large number of clients are factors that make the convergence of federated learning difficult.
To show the robustness to client heterogeneity and participation rates, we perform experiments when both factors are present, that is when the total number of clients is 500 and the participation rate is 1\%.
Local epochs and local iterations are set to 5 and 50, respectively.
Note that this setting makes each client have a very small number of training examples and increases client heterogeneity significantly.
\cref{tab:abl_500cl_1partrate} again shows that FedAGM has the best performance on all metrics.
Note that the performance gap between FedAGM and its strongest competitor, FedDyn, becomes larger than when the participation rate is 2\%: from 1.37\%p to 1.68\%p in CIFAR 10 and from 3.39\%p to 4.76\%p in CIFAR 100 at round 1000.
\begin{table*}[h]
\centering
\caption{Comparison of federated learning methods on CIFAR-10 and CIFAR-100 with the large number of clients (500): Dirichlet (0.3) split on low client participation (1\%). Accuracy at the target round and the communication round to reach target test accuracy are based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted by  {\bf bold}.}

\label{tab:abl_500cl_1partrate}
\scalebox{0.9}{
\begin{tabular}{lcccccccc}
\multirow{3}{*}{Method}                                   & \multicolumn{4}{c}{CIFAR-10}                                                                     & \multicolumn{4}{c}{CIFAR-100}                                                                                \\
\cline{2-9}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 64\% & 68\% & 500R           & \multicolumn{1}{c}{1000R}    & 30\%         & 35\%                 \\
\cline{1-9}
FedAvg~\cite{mcmahan2017communication} &54.71 & 68.96 & 792 & 949 & 26.94 & 35.69  & 636 & 950  \\
FedProx~\cite{li2018federated}                     & 55.18  & 69.80 & 773 & 919 & 26.92 & 35.41  & 648 &  963 \\
FedAvgm~\cite{hsu2019measuring}             &  57.82 & 71.12 & 669   &812  & 29.29 & 39.36  & 530 &  755 \\
FedAdam~\cite{reddi2021adaptive}           &    47.97 & 55.11 &  1000+  &  1000+ & 17.72 & 23.92  & 1000+  & 1000+  \\
FedDyn~\cite{acar2020federated}                 & 58.28 & 74.77 & 621   & 752  & 29.68 & 40.42  & 512 &   703 \\
FedCM~\cite{xu2021fedcm}                        &    49.21& 60.38 &  1000+  & 1000+ & 16.32 & 22.59  & 1000+ & 1000+  \\
FedAGM (ours)                                                & {\bf63.70} & {\bf76.45} & {\bf509} & {\bf618} & {\bf 31.74} & {\bf45.18}  & {\bf458} & {\bf581} \\
\cline{1-9}
\end{tabular}}
\end{table*}

\section{Effect of More Local Iterations}

The increase of local iterations under non-iid environments is prone to result in more divergence across client models and degraded performance of an algorithm.
We evaluate the accuracy and communication-efficiency of the proposed method with aggravated client heterogeneity by varying the number of local iterations, \textit{i.e.}, $K \in \{50, 70, 100\}$, on CIFAR-100.
In this experiments, the number of clients, participation rate, and Dirichlet parameter is set to 100, 5\% and $0.3$, respectively. \cref{tab:abl_local_iterations}~presents that FedAGM outperforms the compared methods consistently for all cases in terms of accuracy and communication efficiency.

\begin{table*}[h]
\centering
\caption{Comparison of federated learning methods with three different local iterations on CIFAR-100: Dirichlet (0.3) split on 5\% client participation out of 100 clients. Accuracy at the target round and the communication round to reach target test accuracy are based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.}
\label{tab:abl_local_iterations}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
 Local iterations & \multicolumn{4}{c}{$K$ = 50} & \multicolumn{4}{c}{$K$ = 70} & \multicolumn{4}{c}{$K$ = 100} \\
\cline{2-13}
 \multirow{2}{*}{Methods} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} &\multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 47\% & 53\% & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 47\% & 53\% & 500R           & \multicolumn{1}{c}{1000R}    & 47\%         & 53\%                 \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 41.88  & 47.83  & 924 & 1000+ & 42.45 & 48.29 & 852 & 1000+ & 41.92 & 48.16  & 896 & 1000+  \\
FedProx~\cite{li2018federated}                 & 42.43  & 48.32  & 881 & 1000+        & 43.31 & 49.54 & 752 & 1000+ & 42.01 & 48.17  & 888 &  1000+ \\
FedAvgm~\cite{hsu2019measuring}               & 46.98  & 53.29  & 515 & 936 & 46.17 &52.34  & 544 & 1000+ & 45.72 &  52.74 & 578 & 1000+  \\
FedAdam~\cite{reddi2021adaptive}              & 44.80  & 52.48  & 691 & 1000+  & 43.76 &  49.19& 756 & 1000+  & 43.00 &  47.51 & 994  &  1000+ \\
FedDyn~\cite{acar2020federated}                  & 52.66  & 58.46  & 293 & 504  & 52.85 & 59.98 &326  &509   & 51.98 & 59.13  &347  & 532   \\
FedCM~\cite{xu2021fedcm}                            & 52.44  & 58.06  & 293 & 572 & 48.30 & 54.89 & 467 & 812 & 46.90 & 54.20  & 502 & 893  \\
FedAGM (ours)                                              & {\bf55.79}  & {\bf62.51}  & {\bf260} & {\bf389}  & \bf{54.23} & \bf{61.23} & \bf{295} & \bf{459} & \bf{54.71} & \bf{63.12}  & \bf{284} & \bf{461} \\
\cline{1-13}
\end{tabular}}
\end{table*}

\subsection{Convergence plot on Tiny-ImageNet}

To validate the performance of the proposed method on a more realistic dataset, we compare FedAGM with the baselines on Tiny-ImageNet in the two federated settings: one with 100 total clients and 5\% client participation, and the other with 500 clients and 2\% client participation.
In both settings, local epochs and local iterations are set to 5 and 50 respectively.
\cref{fig:tiny_curve} shows that FedAGM consistently outperforms the compared methods every communication rounds.
Note that FedAGM shows a faster convergence rate, especially at the early stage of training, than other algorithms.

\begin{figure*}[h]
\centering
\begin{subfigure}[b]{0.4\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/tiny_mid_100.pdf}
\caption{5\% participation, 100 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.4\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/tiny_low_500.pdf}
\caption{2\% participation, 500 clients}
\end{subfigure}
\caption{
The convergence plot of FedAGM and the compared methods on Tiny-ImageNet with the different levels of client heterogeneity.
}
\label{fig:tiny_curve}
\end{figure*}

\section{Evaluation on Tiny-ImageNet}

\bhhan{We found a small mistake in our implementation for the experiments on the Tiny-ImageNet dataset, and report the correct numbers in \cref{tab:abl_tiny_imagenet}.
Also, for consistency with the results on other datasets, we present the results at round 500 and 1000.}

To validate the performance of the proposed method on a more realistic dataset, we compare FedAGM with the baselines on Tiny-ImageNet in the two federated settings: one with 100 total clients and 5\% client participation, and the other with 500 clients and 2\% client participation.
In both settings, local epochs and local iterations are set to 5 and 50 respectively.
\cref{tab:abl_tiny_imagenet}~shows that FedAGM achieves the state-of-the-art performance for all cases.
This validates that FedAGM not only works well on small datasets, but shows good performance on more realistic large-scale datasets.
\cref{fig:tiny_curve} also shows that FedAGM consistently outperforms the compared methods every communication rounds.
Note that FedAGM shows a faster convergence rate, especially at the early stage of training, than other algorithms.

\begin{table*}[h]
\centering
\caption{Comparison of federated learning methods on Tiny-ImageNet with two different settings. Accuracy at the target round and the communication round to reach target test accuracy are based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted by {\bf bold}.
Note that we put wrong results of FedAGM on Tiny-ImageNet of Tab. 1 and 2 in the main paper due to our mistake, and here are the correct results of FedAGM, which is denoted by an asterisk (*). We apologize for the confusion.}
\label{tab:abl_tiny_imagenet}
\scalebox{0.9}{
\begin{tabular}{lcccccccc}
\multirow{3}{*}{Method}                                   & \multicolumn{4}{c}{5$\%$,100 clients}                                                                     & \multicolumn{4}{c}{2$\%$,500 clients }                                                                                \\
\cline{2-9}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 35\% & 38\% & 500R           & \multicolumn{1}{c}{1000R}    & 24\%         & 30\%                 \\
\cline{1-9}
FedAvg~\cite{mcmahan2017communication}  & 33.94 & 35.42 & 645 &  1000+&23.63  & 29.48  & 523 &  1000+ \\
FedProx~\cite{li2018federated}                        & 34.14 & 35.53 & 613 & 1000+ &25.45 & 31.71 & 445  & 799  \\
FedAvgm~\cite{hsu2019measuring}                & 36.32 & 38.51 & 416 & 829 & 26.75 & 33.26  & 386 & 687  \\
FedAdam~\cite{reddi2021adaptive}                & 33.22 & 38.91 &  658& 945  & 21.88 & 28.08  & 648  & 1000+  \\
FedDyn~\cite{acar2020federated}                   & 38.86 & 42.82 & 327 & 440  & 27.74 & 37.25  & 387 & 580   \\
FedCM~\cite{xu2021fedcm}                            & 31.61 & 37.87 & 694 & 1000+ & 19.41 & 24.09  & 975 &  1000+ \\
FedAGM* (ours)                                                & \bf{42.26} & \bf{46.31} & \bf{226} & \bf{331} &\bf{31.47} & \bf{38.48}  & \bf{246} & \bf{447} \\
\cline{1-9}
\end{tabular}}
\end{table*}

\begin{figure*}[h]
\centering
\begin{subfigure}[b]{0.4\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/tiny_mid_100.pdf}
\caption{5\% participation, 100 clients}
\end{subfigure}
\hspace{0.05cm}
\begin{subfigure}[b]{0.4\linewidth}
\centering
\includegraphics[width=1\linewidth]{figures/tiny_low_500.pdf}
\caption{2\% participation, 500 clients}
\end{subfigure}
\caption{
The convergence plot of FedAGM and the compared methods on Tiny-ImageNet with the different levels of client heterogeneity.
}
\label{fig:tiny_curve}
\end{figure*}

\section{Evaluation on Low Participation Rate with a Large Number of Clients}
The low participation rate and the large number of clients are factors that make the convergence of federated learning difficult.
To show the robustness to client heterogeneity and participation rates, we perform experiments when both factors are present, that is when the total number of clients is 500 and the participation rate is 1\%.
Local epochs and local iterations are set to 5 and 50, respectively.
Note that this setting makes each client have a very small number of training examples and increases client heterogeneity significantly.
\cref{tab:abl_500cl_1partrate} again shows that FedAGM has the best performance on all metrics.
Note that the performance gap between FedAGM and its strongest competitor, FedDyn, becomes larger than when the participation rate is 2\%: from 1.37\%p to 1.68\%p in CIFAR 10 and from 3.39\%p to 4.76\%p in CIFAR 100 at round 1000.
\begin{table*}[h]
\centering
\caption{Comparison of federated learning methods on CIFAR-10 and CIFAR-100 with the large number of clients (500): Dirichlet (0.3) split on low client participation (1\%). Accuracy at the target round and the communication round to reach target test accuracy are based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted by  {\bf bold}.}

\label{tab:abl_500cl_1partrate}
\scalebox{0.9}{
\begin{tabular}{lcccccccc}
\multirow{3}{*}{Method}                                   & \multicolumn{4}{c}{CIFAR-10}                                                                     & \multicolumn{4}{c}{CIFAR-100}                                                                                \\
\cline{2-9}
 & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 64\% & 68\% & 500R           & \multicolumn{1}{c}{1000R}    & 30\%         & 35\%                 \\
\cline{1-9}
FedAvg~\cite{mcmahan2017communication} &54.71 & 68.96 & 792 & 949 & 26.94 & 35.69  & 636 & 950  \\
FedProx~\cite{li2018federated}                     & 55.18  & 69.80 & 773 & 919 & 26.92 & 35.41  & 648 &  963 \\
FedAvgm~\cite{hsu2019measuring}             &  57.82 & 71.12 & 669   &812  & 29.29 & 39.36  & 530 &  755 \\
FedAdam~\cite{reddi2021adaptive}           &    47.97 & 55.11 &  1000+  &  1000+ & 17.72 & 23.92  & 1000+  & 1000+  \\
FedDyn~\cite{acar2020federated}                 & 58.28 & 74.77 & 621   & 752  & 29.68 & 40.42  & 512 &   703 \\
FedCM~\cite{xu2021fedcm}                        &    49.21& 60.38 &  1000+  & 1000+ & 16.32 & 22.59  & 1000+ & 1000+  \\
FedAGM (ours)                                                & {\bf63.70} & {\bf76.45} & {\bf509} & {\bf618} & {\bf 31.74} & {\bf45.18}  & {\bf458} & {\bf581} \\
\cline{1-9}
\end{tabular}}
\end{table*}

\section{Effect of More Local Iterations}

The increase of local iterations under non-iid environments is prone to result in more divergence across client models and degraded performance of an algorithm.
We evaluate the accuracy and communication-efficiency of the proposed method with aggravated client heterogeneity by varying the number of local iterations, \textit{i.e.}, $K \in \{50, 70, 100\}$, on CIFAR-100.
In this experiments, the number of clients, participation rate, and Dirichlet parameter is set to 100, 5\% and $0.3$, respectively. \cref{tab:abl_local_iterations}~presents that FedAGM outperforms the compared methods consistently for all cases in terms of accuracy and communication efficiency.

\begin{table*}[h]
\centering
\caption{Comparison of federated learning methods with three different local iterations on CIFAR-100: Dirichlet (0.3) split on 5\% client participation out of 100 clients. Accuracy at the target round and the communication round to reach target test accuracy are based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.}
\label{tab:abl_local_iterations}
\scalebox{0.9}{
\begin{tabular}{lcccccccccccc}
 Local iterations & \multicolumn{4}{c}{$K$ = 50} & \multicolumn{4}{c}{$K$ = 70} & \multicolumn{4}{c}{$K$ = 100} \\
\cline{2-13}
 \multirow{2}{*}{Methods} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} &\multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 47\% & 53\% & \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 47\% & 53\% & 500R           & \multicolumn{1}{c}{1000R}    & 47\%         & 53\%                 \\
\cline{1-13}
FedAvg~\cite{mcmahan2017communication} & 41.88  & 47.83  & 924 & 1000+ & 42.45 & 48.29 & 852 & 1000+ & 41.92 & 48.16  & 896 & 1000+  \\
FedProx~\cite{li2018federated}                 & 42.43  & 48.32  & 881 & 1000+        & 43.31 & 49.54 & 752 & 1000+ & 42.01 & 48.17  & 888 &  1000+ \\
FedAvgm~\cite{hsu2019measuring}               & 46.98  & 53.29  & 515 & 936 & 46.17 &52.34  & 544 & 1000+ & 45.72 &  52.74 & 578 & 1000+  \\
FedAdam~\cite{reddi2021adaptive}              & 44.80  & 52.48  & 691 & 1000+  & 43.76 &  49.19& 756 & 1000+  & 43.00 &  47.51 & 994  &  1000+ \\
FedDyn~\cite{acar2020federated}                  & 52.66  & 58.46  & 293 & 504  & 52.85 & 59.98 &326  &509   & 51.98 & 59.13  &347  & 532   \\
FedCM~\cite{xu2021fedcm}                            & 52.44  & 58.06  & 293 & 572 & 48.30 & 54.89 & 467 & 812 & 46.90 & 54.20  & 502 & 893  \\
FedAGM (ours)                                              & {\bf55.79}  & {\bf62.51}  & {\bf260} & {\bf389}  & \bf{54.23} & \bf{61.23} & \bf{295} & \bf{459} & \bf{54.71} & \bf{63.12}  & \bf{284} & \bf{461} \\
\cline{1-13}
\end{tabular}}
\end{table*}

\iffalse
\begin{table*}[h]
\centering
\caption{Comparison with federated learning methods for different local iterations on CIFAR-100 with 100 clients of Dirichlet (0.3) split on 5\% client participation. Accuracy at target round and the communication round to reach target test accuracy is based on running exponential moving average with parameter $0.9$. The arrows indicate whether higher ($\uparrow$) or lower ($\downarrow$) is better. The best performance in each column is denoted in {\bf bold}.}
\label{tab:abl_local_iterations}
\scalebox{0.9}{
\begin{tabular}{lcccccccc}
 Local iterations & \multicolumn{4}{c}{$K$ = 70} & \multicolumn{4}{c}{$K$ = 100} \\
\cline{2-9}
 \multirow{2}{*}{Methods} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$ )} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)} & \multicolumn{2}{c}{accuracy (\%, $\uparrow$)} & \multicolumn{2}{c}{rounds (\#, $\downarrow$)}  \\
& \multicolumn{1}{c}{500R} & \multicolumn{1}{c}{1000R} & 47\% & 53\% & 500R           & \multicolumn{1}{c}{1000R}    & 47\%         & 53\%                 \\
\cline{1-9}
FedAvg~\cite{mcmahan2017communication}  & 42.45 & 48.29 & 852 & 1000+ & 41.92 & 48.16  & 896 & 1000+  \\
FedProx~\cite{li2018federated}                        & 43.31 & 49.54 & 752 & 1000+ & 42.01 & 48.17  & 888 &  1000+ \\
FedAvgm~\cite{hsu2019measuring}                & 46.17 &52.34  & 544 & 1000+ & 45.72 &  52.74 & 578 & 1000+  \\
FedAdam~\cite{reddi2021adaptive}                & 43.76 &  49.19& 756 & 1000+  & 43.00 &  47.51 & 994  &  1000+ \\
FedDyn~\cite{acar2020federated}                   & 52.85 & 59.98 &326  &509   & 51.98 & 59.13  &347  & 532   \\
FedCM~\cite{xu2021fedcm}                            & 48.30 & 54.89 & 467 & 812 & 46.90 & 54.20  & 502 & 893  \\
FedAGM (ours)                                                & \bf{54.23} & \bf{61.23} & \bf{295} & \bf{459} & \bf{54.71} & \bf{63.12}  & \bf{284} & \bf{461} \\
\cline{1-9}
\end{tabular}}
\end{table*}
\fi

\section{Sensitivity to $\lambda$}
$\lambda$ controls how much the server projects the current model toward the global gradient to initialize the local update.
To further analyze the effect of $\lambda$ on the performance of FedAGM algorithms,
we evaluate the accuracy of the proposed method with different client heterogeneity by varying $\lambda$ as \{0.75, 0.8, 0.85, 0.9, 0.95\}, on CIFAR-10.
\cref{tab:lamb_ablation} shows that the proposed algorithm outperforms most baselines regardless of $\lambda$, and the proposed method shows steady results in terms of the choices of $\lambda$ for different client heterogeneity.

\begin{table}[h]
\centering
\caption{Sensitivity to $\lambda$ of FedAGM in terms of accuracy [\%] at rounds 1000 on CIFAR-10.
Three different levels of client heterogeneity are tested, and the number of clients and participation rate are set to 500 and 2\%, respectively.}
\label{tab:lamb_ablation}
\scalebox{0.9}{
\begin{tabular}{cccccc}
$\lambda$       & 0.75 & 0.8 & 0.85 & 0.9 & 0.95 \\
\hline
Dirichlet (0.3) & 81.32 & 82.52 & 82.80  & 81.82 & 78.25  \\
Dirichlet (0.6) & 83.97 & 84.89 & 85.28  & 84.56 & 82.07  \\
IID             & 85.52 & 86.92 & 86.83 & 87.08 & 84.37 \\
\hline
\end{tabular}}
\end{table}

\section{Hyperparameter Setting}

Our hyperparameter setting is as follows.
For the experiments on CIFAR-10 and CIFAR-100, we choose 5 as the number of local training epochs (50 iterations) and 0.1 as the local learning rate.
We set the batch size of the local update to 50 and 10 for the 100 and 500 client participation, respectively.
The learning rate decay parameter of each algorithm is selected from \{0.995, 0.998, 1\} to achieve the best performance.
The global learning rate is set to 1, except for FedAdam, which is tested with 0.01. \iffalse set to 1.0 except for FedAdam, which is set to 0.01.\fi
For the experiments on Tiny-ImageNet, we match the total local iterations of local updates with other benchmarks by setting the batch size of local update as 100 and 20 for the 100 and 500 client participation, respectively.

As for algorithm-dependent hyperparameters, $\alpha$ in FedCM is selected from \{0.1, 0.3, 0.5\}, $\alpha$ in FedDyn is selected from \{0.001, 0.01,  0.1\}, and $\alpha$ in FedAGM is selected from \{0.9, 1.0\}.
 $\tau$ in FedAdam is set to 0.001.
$\beta$ in FedAvgM is selected from \{0.4, 0.6, 0.8\}, and $\beta$ in FedProx and FedAGM is selected from \{0.1, 0.01, 0.001\}.
For FedAGM, $\lambda$ is selected from \{0.8, 0.85, 0.9\} and $\tau$ is selected from \{0.2, 1.0\}.

\paragraph{Reproducibility statement}
More details for the implementation and hyperparameters are organized at the demo code in the supplementary material, and we will release the source code to facilitate the reproduction of our results.

{\small
\bibliographystyle{ieee_fullname}