\subsection{ST Algorithm}
The ST algorithm works in two phases namely, learning phase and trekking phase. In the learning phase, each player initially plays an arm drawn uniformly at random in each round. Once a player observes collision-free play on an arm, she starts playing the arms sequentially drawing an arm with higher index (up to modulo $K$) in each round. After all the players observe a collision-free play, the arms played by them are orthogonal in each round and no collisions occur. We refer to the part of learning phase in which all players orthogonalize as Random Hopping (RH) sub-phase and the part in which each  player select arms sequentially as Sequential Hopping (SH) sub-phase. The learning phase runs for $T_0$ rounds which is set such that all players find orthogonal arms and learn correct ranking of the arms with high probability. After $T_0$ rounds, each player re-index the arms according to decreasing value of their estimated means.  If a player is on the top channel in the $T_0$ round she continues to play it henceforth, otherwise, she enters into the trekking phase.

In the trekking phase, each player sets the arm played at the end of learning phase as their reserved arm and checks for availability of their next best arm. Specifically, a player updates its current reserved arm, say $i$,  to $i-1$ if no collision is observed on arm $i-1$ in the next $i-1$ rounds. Otherwise she goes back to arm $i$ and plays it in all the subsequent rounds. We refer to this latter scenario as `player is locked'. When a player's reserved arm changes, her previously reserved arm is `released' and can become a reserved arm for another player. Updating of a reserved arm is continued either till the player is locked, or the top arm becomes her reserved arm in which case she locks on it. Thus, in the trekking phase, the players `trek' towards the better arms and all $N$ players settle on one of the distinct top $N$ arms.
\begin{algorithm}[!t]

	\caption{Static Trekking (ST)} \label{Algo1}
	\begin{algorithmic}
		\STATE	Input: {$K, \delta$ }
		\STATE Compute $T_0$ as in (\ref{eqn:ST_T0})
		\STATE	$(\hat{\mu}, I)$= Learning $(T_0, K)$
		\STATE	 Trekking$( \hat{\mu}, I)$
	\end{algorithmic}
\end{algorithm}

\begin{algorithm}[!t]
	\caption*{\textbf{Subroutine:} Learning}

	\begin{algorithmic}[1]
		\STATE Input: $T_{0}, K$
		\STATE Set $L=0$ and $V_k=0, S_k=0 \;\; \forall k \in [K]$
		\FOR{$t=1 \dots T_{0}$}
		\IF{($L == 1$)}
		\STATE Choose arm, $I_t = I_{t-1}+1$ modulo $K$
		\ELSE
		\STATE Randomly choose arm, $I_t \sim U(1, ... ,K)$
		\ENDIF
		\STATE Increment $S_{I_t}$ by 1

		\IF{ no collision ($\eta_{I_t,t}==0$)}
		\STATE Set $L=1$ and $V_{I_t}\leftarrow V_{I_t} + r_{I_t}$
		\ENDIF
		\ENDFOR
		\STATE Estimate arm means, $\hat{\mu}_k = \frac{V_{k}}{S_{k}} \; \forall k$. $\hat{\mu}:=\{\hat{\mu}_k\}_{k \in [K]}$
		\STATE Re-index arms according to their rank in $\hat{\mu}$. Call it $\pi$
		\IF {Index of $I_{T_0}$ in $\pi$ is $1$}
		\STATE Play first arm in $\pi$ henceforth
		\ELSE
		\STATE Trekking($\pi$, Index of $I_{T_0}$ in $\pi$)
		\ENDIF
	\end{algorithmic}
\end{algorithm}
\begin{algorithm}[!t]
	\caption*{\textbf{Subroutine:} Trekking (TrekU)}
	\begin{algorithmic}[1]
		\STATE Input : {$\pi$ (ordered list of arms), $J$ (arm index) }

		\STATE Set $Y_k= 1\;\; \forall k \in [K]$ and  $L=0$ (lock indicator)
		\FOR {$t=T_{0}+1 \dots $}
		\IF{$L==1$}
		\STATE Select the same arm, $I_t=I_{t-1}$
		\ELSIF{$Y_{J}\leq J-1$}
		\STATE Select the (same) next best arm, $I_t=J-1$
		\STATE  $Y_{J}\leftarrow Y_{J} + 1$
		\ELSIF {$I_{t-1}$ not equals to $1$}
		\STATE Select the next best arm, $I_t=I_{t-1}-1$
		\STATE Update reserved arm $J=I_{t-1}$
		\ELSE
		\STATE Lock on the top arm, $L=1$
		\ENDIF
		\IF{ collision ($\eta_{I_t,t}==1$)}
		\STATE Lock on reserved arm, $I_{t}=J$ and $L=1$
		\ENDIF
		\ENDFOR
	\end{algorithmic}
\end{algorithm}

Observing arm $i$ for $i$ rounds by a player before making it as her reserved arm prevents two players from locking on the same arm. To see this, consider that arm $2$ is the reserved arm for a player. This player needs one slot to check if arm $1$ is taken by any other player (by observing collision on it), and in case it is taken, she needs another slot to return and get locked on arm $2$. During these $2$ slots the player with reserved arm $3$ should not lock on arm $2$. Extending this argument for any arm $i$, if each player observes arm $i$ for $i$ rounds, all players are ensured not to lock on the same arm and orthogonality is achieved on the top $N$ arms. The pseudo-code of ST is given in  Algorithm \ref{Algo1} which is run by each player faithfully. We suppress the player index to simplify notations.

The success of trekking phase depends on the players having the correct ranking of the arms. We next show that setting $T_0$ long enough, each player learns correct ranking of arms and the trekking phase then guarantees that each player will lock on one of the top $N$ distinct arm within a bounded number of rounds and no regret is incurred after that.
\subsection{Analysis of ST Algorithm}
\label{sec:AST}
In this subsection, we bound the expected regret and number of collisions of the ST algorithm. For each player let $\hat{\mu}_k$ denotes the empirical mean of arm $k$. We begin with the following definition given in \cite{ICML16_MultiplayerBandits_RosenkiShamir}.

\begin{defn}
	An $\epsilon$-correct ranking of $K$ arms is a sorted list of their empirical mean such that $\forall i, j: \hat{\mu}_i$ is listed before $\hat{\mu}_j$ if $\mu_i - \mu_j \geq \epsilon$.
\end{defn}

\begin{thm}
	\label{thm:StaticST}
Let $\Delta>0$ be the gap between the mean rewards of $N^{th}$ and $(N+1)^{th}$ best arm. Then for all $\epsilon < \Delta$ and $\delta \in (0,1)$, with probability at least $1-\delta$, the expected regret of the ST algorithm from $T$ rounds is bounded as
\begin{equation}
\label{eqn:STRegret}
R \leq N(T_{rh} + T_{sh}(1-N/K) + T_{tr}),
\end{equation}
where $T_{sh}:=T_{sh}(\delta/2), T_{rh}:=T_{rh}(\delta/2)$ and $T_{tr}$ are given as follows:
\begin{equation*}
\label{eqn:ST_TL}
\hspace{-.2cm}T_{rh}=\left \lceil \hspace{-.1cm}\frac{\log\left (\frac{\delta}{2K} \right)} {\log\left(1-\frac{1}{4K}\right)} \hspace{-.1cm}\right\rceil,
\end{equation*}
\begin{equation*}
 T_{sh}=\frac{2K}{\epsilon^2}\log \left (\frac{4KN}{\delta}\right),
\end{equation*}
\begin{equation*}
T_{tr}=(K^2-(N-1)^2)/2 + 1.
\end{equation*}

Further, the number of collisions is bounded  with probability at least $1-\delta$ as
\begin{equation}
C\leq N T_{rh} + 4N.
\end{equation}

\end{thm}
A total of $T_0=T_{rh}+T_{sh}$ rounds guarantee that all players orthogonalize and learn $\epsilon$-correct ranking of the arms with probability at least $1-\delta$. The length of the learning phase is set to $T_0$ in ST algorithm.  Note that the value of $T_{sh}$ depends on $N$, which may not be known. We redefine $T_{sh} $ by replacing $N$ with $K$ and use the following value of $T_0$ which upper bounds earlier value and depends only on $K$ and $\delta$.

 \begin{equation}
 \label{eqn:ST_T0}
T_0=\left \lceil \frac{\log(\delta/2K)}{\log\left(1-1/4K\right)}\right \rceil + \frac{2K}{\epsilon^2}\log \left (\frac{4K^2}{\delta}\right).
 \end{equation}

The bounds hold under the assumption that a lower bound on the reward gap is known. This assumption is also made in \cite{KDD14_ConcurrentBandits_AvnerMannor} and \cite{ICML16_MultiplayerBandits_RosenkiShamir}.  The bounds are in expectation and conditioned on the fact that that players learn the $\epsilon$-correct ranking of the arms which happens with probability $1-\delta$ if learning phase is run for $T_0$ number of rounds.

The proof of Thm \ref{thm:StaticST} is given in the Appendix. The proof consists of bounding  the expected number of rounds required for each player to 1) observe a collision-free play through random selection of arms 2) learn $\epsilon$-correct ranking of arms and 3) settle on one of the top $N$ arms through trekking.  All the bounds are independent of $T$. After the three events, each player will lock on one of the top $N$ distinct arm hence no regret in the subsequent plays. The collision bound is obtained by bounding the expected number of rounds for all the players to orthogonalize in the learning phase and noting that at most two collisions are observed by each player during the trekking phase.

In contrast to the MC algorithm, the ST algorithm aims to orthogonalize the players as early as possible in the learning phase. This significantly brings down the collisions and improves the regret. We next compare the collision and regret performance of the state-of-the-art MC algorithm.
\subsection{Performance comparison with the MC algorithm}\label{staticthm}
 The MC algorithm also works in phases. In the learning phase, each player selects an arm uniformly at random from $[K]$ in each round and obtains $\epsilon$-correct ranking of the arms and an estimate of number of players. In the next phase, named musical chairs, each player plays an arm selected uniformly at random from the top $N$ arms till it observes a collision-free play on an arm and locks on it.
 \begin{thm}
 	\label{thm:MCBound}
 For the same setup in Thm \ref{thm:StaticST}, the expected regret of MC with probability at least $1-\delta$ is bounded as
 \begin{equation}
 \label{eqn:RegretMC}
 R^{MC}\leq NT_0^{MC} + 2N^2\exp(2),
 \end{equation}
 where
 \begin{equation*}
 T_0^{MC}= \max\left( \frac{16K}{\epsilon^2}\ln \left(\frac{4K^2}{\delta}\right), \frac{K^2\log (4/\delta)}{0.02} \right).
 \end{equation*}

The expected number of collisions in the MC algorithm with $N\geq 2$ is lower bounded as
 \begin{equation}
 C^{MC}>(N/K)T_0^{MC}.
  \end{equation}
\end{thm}
The proof for regret is given in \cite{ICML16_MultiplayerBandits_RosenkiShamir} and the proof for collision is given in the Appendix.

{\bf Regret comparison:} The dominant terms in the regret bounds of the ST and MC algorithms are  $\frac{2K}{\epsilon^2}\log \left (\frac{4K^2}{\delta}\right)$
(Eq. (\ref{eqn:STRegret}), (\ref{eqn:ST_T0})) and $\max\left( \frac{16K}{\epsilon^2}\ln \left(\frac{4K^2}{\delta}\right), \frac{K^2\log (4/\delta)}{0.02} \right)$ (Eq. \ref{eqn:RegretMC}), respectively. Comparing the two, the regret bound of ST is smaller by at least a factor of $8$. MC obtains ranking of arms by uniform sampling of arms which requires about $4$ plays of an arm to get one reward sample. Whereas in ST reward is observed in each round after orthogonalization. Thus giving a gain of factor $4$. Another factor $2$ in MC is due to an application of Chernoff bound (this can be tightened).

{\bf Collision comparison:} The average number of collisions incurred by ST during the learning phase is no more than that  incurred by MC during the MC phase -- in the former case an arm is selected uniformly at random from set $[K]$ whereas it is  from the small set $[N]$ in the latter case till a collision-free arm is found. We next compare the number of collisions in the trekking phase of ST and learning phase of MC. In the former case collisions are at most $4K$, whereas it is at least $\frac{K^2\log (4/\delta)}{0.02}$ in the latter case. Hence collisions in ST are smaller by at least a factor of $12.5K$ compared to the MC algorithm.

When $N=K$, the regret in ST is minimal -- regret is non-zero only in the $T_{rh}$ rounds of the learning and $K$ rounds of the trekking phase. Whereas the performance of  MC degrades as $N$ increases. In Section \ref{sec:SR}, Fig \ref{Algo1} we see that the difference in regret of ST and that of MC increases with $N$ validating the proposed hypothesis.

\begin{algorithm}[!b]
	\caption*{\textbf{Subroutine:} Modified Trekking (TrekD)}
		\label{alg:TrekS1}
	\begin{algorithmic}[1]
		\STATE Input : \hspace{-.1cm}{$\pi_1$ (ordered list of estimated arms), $J$ (arm index) }
		\STATE Set $Y_k= 1\;\; \forall k \in [K], L=0, R=1, J_n=1$
		\FOR {$t=T_{0}+1 \dots $}
		\IF{locked ($L==1) $}
		\STATE Select the same arm, $I_t=J$
		\ELSIF { returned to reserved arm ($R==1$)}
		\STATE Select the next `worst' arm $I_t \leftarrow \pi_1(J_n)$ and sense
		\STATE Update next `worst' arm $J_n \leftarrow J_n +1$, set $R \leftarrow 0$ and $Y_{I_t} \leftarrow Y_{I_t} +1$
		\ELSIF{collision ($\eta_{I_{t-1},{t-1}}==1$)}
		\STATE Set $Y_{I_t} \leftarrow Y_{I_t} +1$
		\IF {back-off time lapsed ($Y_{I_t}> K-J$)}
		\STATE Return to reserved arm $R\leftarrow 1$
		\ELSE
		\STATE Stay on the same arm $I_t \leftarrow I_{t-1}$
		\ENDIF

		\ELSE
		\STATE	Set $L\leftarrow 1, I_t\leftarrow I_{t-1},  J\leftarrow I_{t-1}$
		\ENDIF

		\ENDFOR
	\end{algorithmic}
\end{algorithm}

\subsection{ST Algorithm Using Modified Trekking Approach}
\label{sec:mtrek}
In this subsection, we discuss another version of the ST algorithm that is more suitable to the dynamic version where players can enter and leave anytime. In the current version if a player on the top arm leaves, regret is incurred till all the players shift to their next best arm. But, if players start from the top arm and go down as per the ordered list (trekking downwards) instead of checking for their next best arm (trekking upwards), any arm freed up by a leaving player will be taken up earlier. However, in the new 'trekking downward' approach many players may select an arm simultaneously. To overcome this, we introduce 'back off' mechanism to resolve which player locks on an arm.  The ST algorithm based on these modifications is described below and its pseudo-code is given below. We refer to the earlier trekking approach as TrekU and the modified version as TrekD.

Suppose that a player is on arm $i$ at the end of learning phase. The player sets it back-off time as  $K-i+1$ rounds and it remains same for entire trekking phase. A player begins trekking by playing an arm $1$ for the next $K-i+1$ rounds and set the arm $2$ as its reserved arm. If a collision free play is observed within the $K-i+1$ rounds, she enters into lock state and updates her reserved arm to $1$. We again refer to this scenario as `player is locked'. If collisions are observed in each of the $K-i+1$ rounds, the player moves to the reserved arm, i.e. $2$, and updates the next reserved arm as $3$.  The player then check for availability for arm $2$ applying the same procedure. The process is repeated until player gets locked on one of the top arms. Thus, in the modified trekking phase, the players `trek' from the top arm towards bottom arm and all $N$ players settle on one of the distinct top $N$ arms.

The number of rounds required by TrekD to settle the players on the top $N$ channels is at most $(N-1)(K-1)+1$ (see Lemma~\ref{lma:Trekking_Static1} in the Appendix). This is higher than the corresponding $(K^2 -(N-1)^2)/2+1$ bound for the TrekU. It is not hard to realize scenarios where the bounds are tight for both cases. Though TrekU is better on an average, we will see later that TrekD extend to the dynamic scenarios more naturally.

\subsection*{Proof of Thm \ref{thm:StaticST}}
\label{APA}
We prove Theorem $1$ using the following lemmas which gives expected number of rounds for players to 1) orthogonalize through random hopping 2) learn $\epsilon$-correct ranking of arms and 3) settle on the top $N$ arms. The first two events correspond to  random selection and sequential selection of arms in the learning phase, respectively.  We refer to them as random hopping (RH) and sequential hopping (SH) sub-phase.

\begin{lemma}
	\label{lma:RanHop_Static}
	Let $\delta \in (0,1)$. If RH sub-phase is run for  $ T_{rh}(\delta):=\left \lceil \frac{\log(\delta/K)}{ \log\left(1-1/4K\right)} \right \rceil $ number of rounds then all the players will orthogonalize with probability at least $1-\delta$.
\end{lemma}

\noindent \textbf{Proof:}  Let $p_c$ denote the collision
probability of a players when all the players are randomly selecting an arm to play from $[K]$ in each round $t$. Probability that each a player  will observe a collision-free play on an arm after  $T_{01}$ rounds is given by:
\[\sum_{t=1}^{T_{01}}p_c^{t-1}(1-p_c).\]
Setting this value to be at least larger than $1- \frac{\delta}{N}$ for each player we get
\begin{eqnarray}
	\lefteqn{\sum_{t=1}^{T_{01}} p_c^{t-1}(1-p_c) \geq 1- \frac{\delta}{N}} \nonumber\\
	&\iff&1-{p_c}^{T_{01}} \geq 1- \frac{\delta}{N} \nonumber \\
	&\iff& T_{01} \log{p_c} \leq \log\bigg({\frac{\delta}{N}}\bigg) \nonumber \\
	&\iff& T_{01} \geq \frac{\log\big({\frac{\delta}{N}}\big)}{\log{p_c}} \label{eqn:TRHBound}.
\end{eqnarray}
We next give an uniform upper bound on $p_c$. Note that in any round some players may be selecting arms sequentially (call them SH players) while others uniformly at random (call them RH players). Fix a round $t$ and let $N_r \geq 1 $ denote the number of players selecting arms uniformly at random. Let $p_{cr}$ denote the probability that collision is observed from a RH player. We have

\begin{eqnarray*}
\lefteqn{1-p_c= \Pr\{\mbox{no colision from RH players}\}} \\
&+& \Pr\{\mbox{no colision from SH players}\} \\
&\geq & \sum_{j=1}^{N_r}\frac{(1-p_{cr})}{K}\geq \frac{(1-p_{cr})}{K} \\
&=& \frac{(1-1/K)^{N_r-1}}{K}\\
&\geq& \frac{(1-1/K)^{N-1}}{K}\geq 1/4K (\mbox{ for all } K>1).
\end{eqnarray*}

Substituting the bound on $p_c$ in (\ref{eqn:TRHBound}) and using union bound we see that within
$ T_{rh}(\delta)$ rounds all the players will orthogonalize with probability at least $1-\delta$. \hfill\IEEEQED

\begin{lemma}
	\label{lma:Learning_Static}
	For any $\epsilon>0$ and $\delta \in (0,1)$ let $T_{sh}(\delta):=\frac{2K}{\epsilon^2}\log(2KN/\delta)$. If the learning phase is run for $T_0(\delta) := T_{sh}(\delta/2) + T_{rh}(\delta/2)$ rounds, then all the players will have $\epsilon$-correct ranking of arms with probability atleast $1-\delta$.
\end{lemma}
{\bf Proof:} The proof of this lemma is similar to Lemma $1$ in \cite{Arxiv15_MultiplayerBandits_RosenkiShamir}. We repeat it here for completeness. From Lemma \ref{lma:RanHop_Static}, after $T_{rh}(\delta/2)$ rounds of the learning phase (RH sub-phase) all the players are orthogonalized with probability at least $\delta/2$. Conditioned on this event, we will show that players learn $\epsilon$-ranking of arms with probability atleast $\delta/2$ after $\frac{2K}{\epsilon^2}\log(4KN/\delta)$ number of rounds.

Recall a player has an $\epsilon$-correct rank of arms if $\forall k \in [K]$ she has an estimate $\hat{\mu}_j$ such that $\left | \hat{\mu}_k-\mu_k \right| \leq\frac{\epsilon}{2} $. We will upper bound the probability that no SU has $\epsilon-$ correct ranking given that each player has $C_{m}$ observations of each arm.
Consider the following events:

\noindent$O_n$ - event that player $n$ has observed each arm atleast $C_{m}$ number of times.\\
$A$ - event that all players have $\epsilon$-correct ranking. \\
$A_n$ - event that player $n$ has $\epsilon$- correct ranking. \\
$B$ - event that all players have atleast $C_{m}$ observations of each arm. \\
$B_n$ - event that player $n$ has atleast $C_{m}$ observations of each arm.

In the following we use $\overline{X}$ which denotes complement of event $X$.\\

We have
\begin{align*}
&Pr(\overline{A}_n|B_n) \\
&\leq Pr\left ( \exists k \in [K] \mbox{ such that }| \hat{\mu}_k-\mu_k | >\frac{\epsilon}{2} | \ B_n\right) \\
 &\leq \sum_{k=1}^{K} Pr\left ( | \hat{\mu}_k-\mu_k | >\frac{\epsilon}{2} | \ B_n\right) (\mbox{Union bound})\\
&= \sum_{k=1}^{K} \sum_{j=C_{m}}^{\infty} Pr\left (| \hat{\mu}_k-\mu_k | >\frac{\epsilon}{2} | \ O_n = j \right)Pr \left( O_n = j| \ B_n \right) \\
\tag {By Hoeffding's Inequality} &\leq \sum_{k=1}^{K} \sum_{j=C_m}^{\infty} 2  \exp \left(\frac{-j \epsilon^2}{2} \right) Pr \left( O_n = j| \ B_n \right) \\
&\leq \sum_{k=1}^{K} 2 \exp \left(\frac{-C_{m}  \epsilon^2}{2} \right) \sum_{j=C_m}^{\infty} Pr \left( O_n = j| \ B_n \right) \\
&\leq \sum_{k=1}^{K} 2 \exp \left(\frac{-C_{m} \cdot \epsilon^2}{2} \right)  \\
&\leq  2K  \exp \left(\frac{-C_m \epsilon^2}{2} \right)
\end{align*}

We can apply Hoeffding's Inequality since each observation of the arm is independent of the number of times we observe that arm. Setting the bound to be  less than $\frac{\delta}{2N}$, we get
\[   C_{m} \geq  \frac{2}{\epsilon^2} \ln \left ( \frac{4 KN}{\delta} \right) \]

After the RH sub-phase,  the players select orthogonal arms hence there will be no collision and get reward sample in each round. Further, since they select the arms sequentially, the number of plays of  each arm is in the same proposition. Hence if $T_{sh}(\delta/2)=(2K/ \epsilon^2) \ln \left ( 4 KN/ \delta) \right) $ rounds are played after the RH sub-phase, all players will have $\epsilon$-correct ranking of the arms with probability at least $1-\delta/2$ (by applying union bound). \hfill\IEEEQED

\begin{lemma}
\label{lma:Trekking_Static}
In the trekking phase of ST all the players settle on the top $N$ arms in at most $T_{tr}:=(K^2-(N-1)^2)/2 + 1$ number of rounds.
\end{lemma}
{\bf Proof:} Recall that in the trekking phase each player plays its next best arm, say $i>1$,  for at least $(i-1)$ rounds before taking it as their reserved arm. If there are $n$ players with reserved arms better than than arm $i$,  then a player with reserved arm $i$ can lock only on the $(n+1)$th arm and also all other $n$ players lock on the top arms before her. Hence the maximum number of rounds before the player locks on the $(n+1)$th arm is given by
 \[(i-1)+(i-2)\cdots +( n) + 1= \left (\sum_{k=1}^{i-n} i-k\right ) + 1\]
$1$ is added in the summation to count the round in which the player falls-back on its reserved arm and locks.
The worst case happens when one of the player starts the trekking phase with the worst arm as her reserved arm. Thus setting $i=K$ and $n=N-1$ in the above summation we get the maximum number of rounds in the trekking phase as
\begin{eqnarray*}
T_{tr}&=& \left (\sum_{k=1}^{K-(N-1)} K-k\right ) + 1\\
 &\leq& (K^2 - (N-1)^2)/2 + 1.
\end{eqnarray*}

\hfill\IEEEQED

\noindent {\bf Proof of Thm 1:} From Lemma \ref{lma:Learning_Static} and Lemma \ref{lma:Trekking_Static} all the players settle on the top $N$ arms without any overlap after $T_{rh}(\delta/2)+T_{sh}(\delta/2) + T_{tr}$ rounds with probability at least $1-\delta$ and regret from the subsequent rounds is zero.
Hence expected regret of ST with probability at least $(1-\delta)$  is
 \[R \leq N(T_{rh}(\delta/2)+T_{sh}(\delta/2) + T_{tr}).\]

The upper bound can be tightened as follows. Notice that in the SH sub-phase, each player selects each arm $1/K$ fraction of the time and in particular the top $N$-arms $N/K$ fraction of the time. When a player selects any of the top $N$ arms, her contribution to regret in that round is zero. Hence each player in the SH sub-phase contribute to regret only $(1-N/K)$ fraction of the time. Adding this factor in the above regret bound we get
 \[R \leq N(T_{rh}(\delta/2)+T_{sh}(\delta/2)(1-N/K) + T_{tr}).\]

We next bound the number of collisions. During the learning phase, collision occurs only in the RH sub-phase. During the trekking phase, each player can experience at most two collisions -- one before and after locking on an arm. For a player collision can happen before locking when she selects an arm on which another player is locked. Collision can happen after locking when another trekking player selects the arm on which she is locked. Thus total number of collision with probability atleast  $1-\delta$ is bounded as

\[C \leq N T_{rh}(\delta/2) + 2(2N).\]  \hfill\IEEEQED

\subsection*{Length of Modified Trekking Phase:}

\begin{lemma}
	\label{lma:Trekking_Static1}
	In the modified trekking phase (TrekD) presented in Section~\ref{sec:mtrek}, all the players settle on the top $N$ arms in at most $T_{tr}:=(N-1)(K-1)+1$ number of rounds.
\end{lemma}
{\bf Proof:} Recall that in the modified trekking phase each player plays an arm for at most ($K-i+1$) rounds before locking on it. The time taken by any player to get locked is then at most the number of arms tried before locking multiplied by her back-off time. Note that player on arm $2$ (if any) is the last to back-off from every arm that is taken over by another player and is the last to lock.
Since the player on arm $2$ has to try at most $N-1$ different arms before she locks, she will lock (and so are others) after at most $T_{tr}:=(N-1)(K-1)+1$ rounds.\hfill\IEEEQED

\subsection*{Proof of Thm  	\ref{thm:MCBound}}
The bound on the regret is given in \cite{ICML16_MultiplayerBandits_RosenkiShamir}[Thm 1]. In the MC algorithm the learning phase is run for $T_0^{MC}$ number of rounds in which each player select arm randomly from $[K]$ in each round. The probability of observing a collision for a player in each round is $(1-(1-1/K)^{N-1})$. Hence expected number of collisions are at least

\[C^{MC} \geq NT_0^{MC} (1-(1-1/K)^{N-1}).\]
For $N\geq 2$, we have $(1-1/K)^{N-1} \leq 1-1/K$. Hence we get $C^{MC} \geq NT_0^{MC}/K$ as claimed. \hfill\IEEEQED

\subsection*{Proof of Proposition \ref{prop:DynamicDT}}
The proof of this Theorem follows along the ideas similar to that in \cite{ICML16_MultiplayerBandits_RosenkiShamir}[Thm 2.]

We first bound the regret. The regret in each epoch is composed of the following three terms:

\begin{itemize}
	\item Regret due to learning and trekking phase
	\item Regret due to entering players
	\item Regret due to leaving players
\end{itemize}
Let $T_0:=T_0(\delta/T)$ denote the length of learning phase in each epoch.
Then with probability $1-\delta/T$ all the players in that epoch will have $\epsilon$-correct ranking of the arms leading to zero regret after the trekking phase.
Note that $T_0$ is a function of $T$ and grows logarithmically in $T$.

 \noindent
{\bf Regret due to learning and trekking phase:} The length of the this period is $T_0 + T_{tr}$ and adds at most $K(T_0+ T_{tr})$ regret.

\noindent
{\bf Regret due to entering players:}
Recall that we allow a new player to enter after the learning and trekking phase in each epoch. Each new player collides with at most one player in each round. If $e_i$ is the number of player that enter in an epoch they add
at most $2e_i(T_{ep}-T_0-T_{tr})$ regret. A factor $2$ is because reward from two optimal arms is lost each time a collision happens.

\noindent
{\bf Regret due to leaving :}
Recall that player can leave at any time. A player leaving during the learning phase do not cause any regret, whereas if a player leaves after the learning phase, the arm on which she was locked may not be taken over by any other player and regret is incurred for the remaining rounds. Hence, if $l_i$ players leave in an epoch, it add at most $l_i (T_{ep}- T_0-T_{tr})$ regret.

\noindent
Let
$e=\sum_ {i=1}^{T/T_{ep}} e_i$ and $l=\sum_ {i=1}^{T/T_{ep}}l_i$ denote the total number of entering and leaving players across all epochs. Combining regret from all the three parts from each epoch and adding over all the epochs, we get
\begin{align*}
R &\leq  \frac{T}{T_{ep}}(K(T_0+ T_{tr})) + 2e (T_{ep} -T_0 -T_{tr}) \\
& \quad + l (T_{ep} -T_0 -T_{tr})
\end{align*}
We set the value of $T_{ep}$ as
\[T_{ep} =\sqrt{\frac{KT(T_0+T_{tr})}{2x}}.\]
which minimizes the upper bound. Finally, the result follows by taking union bound over $T$.

We next bound the number of collisions. Note that a leaving players will not cause any collision. In each epoch number of collision in the learning phase is at most $T_{rh}:=T_{rh}(\delta/2T)$ and the trekking phase is at most $4K$. Each entering player will cause at most $2(T_{ep}-T_0-T_{tr})$. Hence total number of collisions is at most

\[C\leq \frac{T}{T_{ep}}(KT_{rh}+4K)+ 2x(T_{ep}-T_0-T_{tr}).\] \hfill\IEEEQED

\subsection*{Proof of Thm \label{thm:DynamicDT}}
\noindent
From the proof of Lemma $1$, recall that $T_0= \mathcal{O}(\log T)$. Hence
$T_{ep}=\mathcal{O}(\sqrt{T\log T /x})$. We get
\begin{eqnarray*}
R &\leq& \mathcal{O}\left (\frac{T}{T_{ep}}T_0 + xT_{ep} \right ) \\
&=& \mathcal{O} \left(
\frac{\sqrt{x}T}{\sqrt{T\log T}} \log T +  x \sqrt{\frac{T \log T}{x}}\right) \\
&=& \mathcal{O} \left(
\sqrt{xT\log T}  +   \sqrt{xT \log T}\right)\\
&=&\tilde{\mathcal{O}}(\sqrt{xT}),
\end{eqnarray*}
 where $\tilde{\mathcal{O}}$ hides logarithmic factor in $T$.

 The bound on the number of collisions also follows similarly by noting that
 \begin{equation}
  C \leq \mathcal{O}\left (\frac{T}{T_{ep}}T_0 + xT_{ep} \right ).
 	\end{equation}

\hfill\IEEEQED

\subsection*{Proof of Theorem \ref{thm:DynamicDTS}}
 \noindent
 We prove the Theorem using the following lemma

 \begin{lemma}

 	Consider the same setup as in Prop. \ref{prop:DynamicDT}. For any $\delta \in (0,1)$ let
      length of the learning phase in DTS is set as
 	\[
 	\tilde{T}_0=\left \lceil \frac{\log(\delta/2(K+x))}{\log\left(1-1/4K\right)}\right \rceil + \frac{2K}{\epsilon^2}\log \left (\frac{4K(K+x)}{\delta}\right),\]
 	Then, expected regret of DTS after $T$ rounds is bounded with probability at least $1-\delta$ as follows:

 	\begin{equation*}
 	R \leq K\tilde{T}_0+ e\tilde{T}_0 + xKT_l + (T/T_l)\tilde{T}_{tr} K
 	\end{equation*}
 \end{lemma}
where
\[\tilde{T}_{tr}:= (K/2)^2 + K/2.\]

  Further, expected number of collisions is bounded with probability at least $1-\delta$ as
 \begin{equation*}
C\leq (K+x)\left \lceil \frac{\log(\delta/2(K+x))}{\log\left(1-1/4K\right)}\right \rceil + (T/T_l)\tilde{T}_{tr}K.
 \end{equation*}

{\bf Proof:} We first bound the regret. The regret is composed of the following terms.

 \begin{itemize}
 	\item Regret due to learning phase of players who joined from the start
 	\item Regret due to learning phase of players entering the game late
 	\item Regret due to players leaving the game
 	\item Regret due to continuous trekking of all players
 \end{itemize}

{\bf Regret due to players who joined from the start:}\\
Let $N_m  \leq K$ denote the number of players that join the game at $t=0$ and $\tilde{T}_0$ denote the length of the learning phase. The regret due to learning phase is upper bounded $K\tilde{T}_0$.

{\bf Regret due to learning phase of players entering late:}\\
Let $e$ denote the number of players that enter the game late. We note that the entering players do not disturb already settled players as they sense selected arms before playing. Thus when a new player enters regret is incurred only due to them and not due to already settled players which is upper bounded by $e\tilde{T}_0$. Further, if the at the end of learning phase, the entering players may not have estimates for mean rewards for some arms and their estimates are obtained during the trekking phase by playing them for $T_l$ rounds each. This will cause additional regret bounded by $KT_l$. The worst case happens when a player do not get estimates of any arm. Hence the regret upper bound due to entering players is
$e\tilde{T}_0 + eKT_l$

{\bf Regret due to leaving players:}\\
When a player leaves, the freed-up arm will be taken by one of the existing player after at most $T_l$ rounds -- within $T_l$ rounds after the player leaves one of the existing players enters into the trekking state and takes over the free-up arm. Hence $l$ leaving players cause at most $lT_l$ rounds of regret.

{\bf Regret due to continuous trekking of all players:}\\
We first argue that the regret due to the players having estimates of all or only few of the arms can be treating in the same fashion.  Consider a player that does not have estimates of all the arms (entering/late player). This player checks for an arm for which  estimate is not available yet before checking for the best available arm for the set of estimated arms and locks on it whenever it is available to get its estimate.  At most $KT_l$ (this could be spread over multiple trekking cycles) rounds incurred due to this. This factor is already accounted in the regret computed in the second point. Hence we compute regret during the continuous trekking assuming that all players have estimates of all the arms. We refer to number of rounds a players spends in trekking state before he enters into locked state as one trekking cycle.

We next bound length of a trekking cycle. A player with reserved arm $i$ requires at most $(i-1)(K- i +1) + i$ to complete checking of availability of better arms -- the players spends at most $K-i +1$ rounds (back-off) on each of the arms $1,2, i-1$. If none of them is available she locks back on $i$. Addition of $i$ accounts for each return to arm $i$.  Optimizing over the value of $i$, the maximum length of a trekking cycle is given by $\tilde{T}_{tr}:= (K/2)^2 + K/2$.

Over period $T$, number of trekking cycles for each player is at most $T/T_l$ and in each trekking regret is at most $\tilde{T}_{tr}$. Since at most $K$ players can be in the game at any time, regret due to continuous trekking is upper bound by
           \[(T/T_l)\tilde{T}_{tr}K.\]

Combining all the terms, the regret is upper bound as

\begin{eqnarray}
R &\leq& K\tilde{T}_0+ e\tilde{T}_0 + eKT_l + lT_l + (T/T_l)\tilde{T}_{tr} K  \nonumber \\
&\leq& K\tilde{T}_0+ e\tilde{T}_0 + eKT_l + lKT_l + (T/T_l)\tilde{T}_{tr} K \nonumber \\
\label{eqn:DTS_RegretBound}
&\leq& K\tilde{T}_0+ e\tilde{T}_0 + xKT_l + (T/T_l)\tilde{T}_{tr} K.
\end{eqnarray}

Setting
\begin{equation*}
\tilde{T}_0=\left \lceil \frac{\log(\delta/2(K+x))}{\log\left(1-1/4K\right)}\right \rceil + \frac{2K}{\epsilon^2}\log \left (\frac{4K(K+x)}{\delta}\right).
\end{equation*}
and using arguments similar to that in Thm 1, and applying union bound over all players (at most $(K+x)$), the regret bound holds with probability $1-\delta$.

To bound the collision, note that the  players incur collisions during RH sub-phase of the learning phase and  whenever it enters into the trekking state during the trekking phase.  Since at most $(N_m + e)$ players enter into RH sub-phase and at most $K$ player in the game at any time we get that collisions are bounded with probability $1-\delta$ by
\begin{equation}
\label{eqn:DTS_Collision}
C\leq (K+x)\left \lceil \frac{\log(\delta/2(K+x))}{\log\left(1-1/4K\right)}\right \rceil + (T/T_l)\tilde{T}_{tr}K.
\end{equation}
\hfill\IEEEQED

We now return to the proof of Theorem \ref{thm:DynamicDTS}. Ignoring the constants, the regret bound in \ref{eqn:DTS_RegretBound} is given by
\[R \leq \mathcal{O}(xKT_l + (T/T_l)\tilde{T}_{tr} K )\]
Differentiating the bound w.r.t to $T_l$, we find the optimal value of the bound is given as
\[R \leq \mathcal{O}\left(2K\sqrt{xT\tilde{T}_{tr}}  \right).\]
and it is achieved by setting
\[T_l=\sqrt{\frac{T\tilde{T}_{tr}}{x}}.\]
Similarly, by plugging the above value of $T_l$ in (\ref{eqn:DTS_Collision}) we get
\[C \leq \mathcal{O}\left( K\sqrt{xT\tilde{T}_{tr}} \right).\]
\hfill\IEEEQED

\section{Introduction}
\label{sec:submission}
\input{Intro}

\section{ Problem Setup}
\label{sec:setup}
\input{Setup}
\section{Static Trekking Algorithm}
\label{sec:AlgoNoSensing}
\input{Algorithms}
\section{Epoch Based Dynamic Trekking Algorithm}
\label{sec:AlgoNoSensingd}
\input{Algorithmsd}
\section{Epoch-Free Dynamic Trekking Algorithm}
\label{sec:AlgoSensing}
\input{AlgoSensing}
\section{Simulation Results}
\label{sec:SR}
\input{simulation_results}
\section{Conclusions and Future Directions}
\label{sec:Conclu}
\input{Conclusion}

\appendices
\label{Appendix}
\input{Appendix}
\ifCLASSOPTIONcaptionsoff
\newpage
\fi

\bibliographystyle{IEEEtran}

\subsection{Related Work}
Most works on stochastic bandits with multiple-payers require some negotiation or pre-agreement phase to avoid collisions between the players. The $\mbox{dUCB}_4$ algorithm in  \cite{TIT14_DecentralizedLearning_KalthilNayyarJain} achieves this using Bertsekas' auction mechanism for players to negotiate unique arm. The Time Divisions Fair Sharing (TDFS) algorithm in \cite{TSP10_DistributedLearning_LiuZhao} requires players to agree on a time division of slots before the game. Such negotiations are hard to realize in a completely distributed setup like ad hoc CRN \cite{ad_hoc_1,ad_hoc_2}.  The $\rho^{\mbox{{\tiny RAND}}}$ algorithm in \cite{JSAC11_DistributedLearning_Anadakumar} is communication free and completely decentralized. Performance improvements of  $\rho^{\mbox{{\tiny RAND}}}$ are studied recently in \cite{ALT2018_MultiplayerBandits_BessonKaufma}. However, these algorithms consider only static case and assumes prior knowledge of number of players. The modified $\rho^{\mbox{{\tiny EST}}}$ algorithm overcomes latter issue, but its guarantees holds only asymptotically. Other set of works in \cite{TWC16_DistributedStochastic_ZandiDongGrami}, \cite{TMC11_CongnitiveMediumAccess_LaiGamalJiangPoor} considers selfish behavior of players and analyze their equilibrium behavior. However, all these algorithms work only for the static case and cannot extend to the dynamic scenarios which is the focus of this work.

The works most similar to ours are \cite{INFOCOM16_MultiUserLax_AvnerMannor}, \cite{KDD14_ConcurrentBandits_AvnerMannor} and \cite{ICML16_MultiplayerBandits_RosenkiShamir} which consider communication free setting with unknown number of players that can vary during the game. The algorithm in \cite{INFOCOM16_MultiUserLax_AvnerMannor} also considers the case where the arm characteristics are different across players but does not guarantee network optimal reward. The major drawback of this algorithm is that it assumes that the players gets to know information of the channels selected by all other players in each time slot. In ad-hoc CRN, complex hardware is needed to gain such information and hence, it is not feasible for battery operated users \cite{sns}. The MEGA algorithm in \cite{KDD14_ConcurrentBandits_AvnerMannor} uses the classical $\epsilon$-greedy MAB algorithm and ALOHA based collision avoidance mechanism. Though collision frequency reduces in MEGA as the game proceeds it may not go to zero as shown in \cite{ICML16_MultiplayerBandits_RosenkiShamir}. To overcome this \cite{ICML16_MultiplayerBandits_RosenkiShamir} develop Musical Chairs (MC) algorithm that incurs collisions only in the initial phase and guarantees collision free play subsequently. Though MC performs better than MEGA, its performance in the initial rounds is poor -- MC uses collision information to estimate the number of players and forces a large number of collisions to get a good estimate.

Our approach reduces total collisions  by circumventing the need to estimate the number of players and guarantees collision free play after few rounds.
The first part of our algorithms find orthogonal arm allocations through random hopping and then follow a common (deterministic) hopping pattern which is similar to the two phase channel/arm access scheme in \cite{JSAC2014_DesignAnalysis_ZhangYao}. However, it considers only the static case with identical arms.

The trekking approach has been discussed in \cite{wiopt} where we considered the static case. In this paper, we provide algorithms for both the static and dynamic case and their analysis. The proposed algorithms in \cite{wiopt} are specifically designed for CRN in a licensed spectrum whereas the current work focuses on a more general MPMAB setting and hence the analysis is substantially different than in \cite{wiopt}.

\noindent
{\bf Organization of the paper:}In Section \ref{sec:setup} we introduce the notations and setup the problem. In Section \ref{sec:AlgoNoSensing} we give algorithms and analyze their performance for the static and dynamic scenarios in sections \ref{sec:AlgoNoSensingd} and \ref{sec:AlgoNoSensingd}, respectively.  In section  \ref{sec:AlgoSensing}, we modify the algorithm for the dynamic scenario to work without requiring any shared global clock.  We validate our claims through both synthetic and real test-bed setup in Section \ref{sec:SR}. Conclusions and future directions are given in Section \ref{sec:Conclu}. All the proofs are in the appendix given at the end of the paper.

\subsection{Static Case}

\subsection{Static Case}
For static case, we consider a game of $T=10000$ rounds. We consider $K=10$ and $\mu^1 =  \{0.22,0.29,0.36,0.43,0.50,0.57,0.64,0.71,0.78,0.85\}$ with $\Delta=0.07$ and  $\mu^2 =  \{0.05,0.15,0.25,0.35,0.45,0.55,0.65,0.75,0.85,0.95\}$ with $\Delta=0.1$. Analytically, the $T_0$ of the ST algorithm is at least $4$ times smaller than $T_0^{MC}$ of the MC algorithm. However, $T_0^{MC}$ used in the simulation results presented in \cite{ICML16_MultiplayerBandits_RosenkiShamir} is much smaller than that obtained from their mathematical expressions. For fair comparison, we assume $T_0^{MC}=T_0=3000$ rounds. Later, we present the results using actual values of $T_0^{MC}$ and $T_0$.

In Fig.~\ref{st11}, we compare the cumulative regret of the MC and ST algorithms for $N=\{3,5,9\}$ where $N$ indicates the number of active players. The MC algorithm has constant regret plot after learning and MC phases while the ST algorithm has constant regret plot after learning and trekking phases. The constant regret plot is an indication of players settling in top channels and no further increase in the regret thereafter. Before that, the regret of MC algorithm is significantly higher than that of the ST algorithm due to random hopping approach in the MC algorithm compared to collision-free sequential hopping in the ST algorithm. Also, the regret of the ST algorithm is highest when $N=K/2=5$ while the regret of the MC algorithm increases as the value of $N$ increases.

Next, we compare the number of collisions faced by all the players until the end of the horizon in Fig.~\ref{st12}. Note that $y-axis$ is shown on a logarithmic scale for clarity of the plots. It can be observed that the number of collisions is significantly higher in the MC algorithm compared to the ST algorithm and the difference increases as the value of $N$ increases. This, in turn, means that the difference between the regret of the MC algorithm and ST algorithm increases as the value of $N$ increases. The corresponding plots are shown in Fig.~\ref{st13}. Similarly, the plots corresponding to $\mu^2$ are shown in Fig.~\ref{st2}. All the plots presented in Fig.~\ref{st1} and Fig.~\ref{st2} validate our claims in Section~\ref{staticthm}.

The simulation results presented in Fig.~\ref{st1} and Fig.~\ref{st2} assume $T_0^{MC}=T_0=3000$. As discussed in Section~\ref{sec:AST}, the actual value of $T_0$ for the ST algorithm is much smaller than $T_0^{MC}$ of the MC algorithm. Here, we choose the value of $T_0$ and $T_0^{MC}$ such that it guarantees desired minimum number of observations of each arm, $C_m$ at each player. The value of the $C_m$ is given in Lemma 2 and is equal to 200 for the parameters considered here. The corresponding value of $T_0^{MC}$ and $T_0$ are 6200 and 2000, respectively. The plots of the cumulative regret, and the number of collisions for two different arm statistics, $\mu^1$ and $\mu^2$ are shown in Fig.~\ref{st3} and Fig.~\ref{st4}, respectively. It can be observed that the difference between the regret of ST and MC algorithms is higher in this case compared to the results in Fig.~\ref{st1} and Fig.~\ref{st2} where $T_0^{MC}=T_0$. Similar observations can be made for the number of collisions as well.

\begin{figure*}[!t]
	\centering
	\subfigure[]{\label{st31}\includegraphics[width=55mm]{figures/static/mu1_N4_N8_diff_tcc_.pdf}}
	\subfigure[]{\label{st32}\includegraphics[width=58mm]{figures/static/mu1_coll_diff_tcc_.pdf}}\hspace{0.1cm}
	\subfigure[]{\label{st33}\includegraphics[width=55mm]{figures/static/mu1_diff_U1_diff_tcc_.pdf}}
		\vspace{-0.2cm}
	\caption{(a) The cumulative regret comparison between MC and ST algorithm for $N=4$ and $N=8$, (b) Number of collisions for different values of $N$ with $y-axis$ shown on the logarithmic scale, and (c) The plots showing the difference between the regret of MC and ST algorithms for various values of $N$. Here, we consider the arms with statistics $\mu^1$, $T_0=2000$ and $T_0^{MC}=6200$.}
	\label{st3}
\end{figure*}
\begin{figure*}[!t]
	\centering
	\subfigure[]{\label{st41}\includegraphics[width=55mm]{figures/static/mu2_N4_N8_diff_tcc_.pdf}}
	\subfigure[]{\label{st42}\includegraphics[width=58mm]{figures/static/mu2_coll_diff_tcc_.pdf}}\hspace{0.1cm}
	\subfigure[]{\label{st43}\includegraphics[width=55mm]{figures/static/mu2_diff_U1_diff_tcc_.pdf}}
	\vspace{-0.2cm}
	\caption{(a) The cumulative regret comparison between MC and ST algorithm for $N=4$ and $N=8$, (c) Number of collisions for different values of $N$ with $y-axis$ shown on the logarithmic scale, and (c) The plots showing the difference between the regret of MC and ST algorithms for various values of $N$. Here, we consider the arms with statistics $\mu^2$, $T_0=2000$ and $T_0^{MC}=6200$.}
	\label{st4}
\end{figure*}

\subsection{Dynamic Case}
In this sub-section, we consider various scenarios to compare the performance of the DT, DTS and DMC algorithms. The game starts with one player and the subsequent entry and exit of the players are shown using dotted red and dashed gray lines, respectively. The leaving players are chosen randomly.

\subsubsection{Scenario 1-3: Restricted Entry/Exit}
In this sub-section, we study the effect of the number of players and the rate at which they enter or leave the game. To do this, we consider three scenarios where players can enter or leave the game anytime except during learning period of each epoch of the DMC and DT algorithms.

We begin with Scenario 1 which is similar to the one discussed in \cite{ICML16_MultiplayerBandits_RosenkiShamir}. In Scenario 1, the game starts with one player and $T=500000$ rounds. At $166667^{th}$ round, second player enters while the first player leaves later at $333333^{th}$ round. Similar to \cite{ICML16_MultiplayerBandits_RosenkiShamir}, we consider four arms with $\mu^3 =\{0.05, 0.35,0.65,0.95\}$ and epoch length, $T_{ep}$ of 34757 rounds. Due to single player till 166667 rounds, the regret of the DMC and DT algorithm is identical while the regret of the DTS algorithm is lower due to epoch-free approach. In fact, the DTS algorithm incurs regret only when the player enters or leaves the game. The regret of the DT algorithm is lower than that of the MC algorithm from 166667 till 333333 rounds during which there are two players in the game. Thereafter, the regret incurred by both the algorithms is identical. Thus, the DT algorithm is superior to the DMS algorithm whenever there are more than one player in the game.

Next, we increase the number of players and the rate at which they enter or leave the game. The corresponding plots are shown in Fig.~\ref{dc1} and Fig.~\ref{dc2} where the number of players and the entering/leaving rate is higher in latter compared to former. As expected, the regret of the DTS algorithm is lowest followed by that of the DT algorithm and MC algorithm incurs highest regret among three algorithms.

\subsubsection{Scenario 4-5: Un-restricted Entry/Exit}
In this sub-section, we allow players to enter or exit the game at any round. The DMC algorithm do not allow the player to enter or leave the game during learning and MC phases while DT algorithm restricts entry during learning and trekking phases. In case of DTS algorithm, there is no such restriction on the player entry or exit during the game. It can be observed from the Fig.~\ref{dc3} and Fig.~\ref{dc4} that DTS algorithm performs significantly better than the DT and DMC algorithms. Though the DT algorithm is superior to the DMC algorithm, the difference between the regret of the DT and DMC algorithm is smaller than the scenarios where the players can not enter or exit during the learning and trekking/MC phases.

\begin{figure}[!h]
	\centering
	\subfigure[]{\label{dc01}\includegraphics[width=40mm]{figures/A1_mu3_CR_.pdf}}  \hspace{0.25cm}
	\subfigure[]{\label{dc02}\includegraphics[width=41mm]{figures/A1_mu3_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for Scenario 1 where players can enter or leave any time except during learning phase of the DMC and DT algorithm.}

	\label{dc0}
\end{figure}

\begin{figure}[!h]
	\centering
	\subfigure[]{\label{dc11}\includegraphics[width=40mm]{figures/A2_m1_.pdf}}  \hspace{0.25cm}
	\subfigure[]{\label{dc12}\includegraphics[width=41mm]{figures/A2_m1_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for Scenario 2 where players can enter or leave any time except during learning phase of the DMC and DT algorithm. }
	\label{dc1}
\end{figure}

\begin{figure}[!h]
	\centering
	\subfigure[]{\label{dc21}\includegraphics[width=40mm]{figures/A2_m1_ip2_.pdf}}  \hspace{0.25cm}
	\subfigure[]{\label{dc22}\includegraphics[width=41mm]{figures/A2_m1_ip2_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for Scenario 3 where players can enter or leave any time except during learning phase of DMC and DT algorithm.}
	\label{dc2}
\end{figure}

\begin{figure}[!h]
	\centering
	\subfigure[]{\label{dc31}\includegraphics[width=40mm]{figures/A3_m1_.pdf}}  \hspace{0.25cm}
	\subfigure[]{\label{dc32}\includegraphics[width=41mm]{figures/A3_m1_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for Scenario 4 where players can enter or leave any time including the learning phase of the DMC and DT algorithm.}
	\label{dc3}
\end{figure}

\begin{figure}[!h]
	\centering
	\subfigure[]{\label{dc41}\includegraphics[width=40mm]{figures/A3_m2_.pdf}}  \hspace{0.25cm}
	\subfigure[]{\label{dc42}\includegraphics[width=41mm]{figures/A3_m2_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for Scenario 5 where players can enter or leave any time including the learning phase of the DMC and DT algorithm.}
	\label{dc4}
\end{figure}

For the five different scenarios considered for dynamic case, it can be observed that the difference between the regret as well as number of collisions of the DMC algorithm and proposed algorithms increases as the number of players and the entering/leaving rate is increased. Thus, higher the dynamism of the game/network, better is the performance of the proposed algorithm compared to the state-of-the-art DMC algorithm. Similar observation is also valid for the number collisions. The Fig.~\ref{dcl} shows that the proposed algorithms offer a significantly fewer number of collisions than the DMC algorithm for all the scenarios considered in dynamic case. The difference between the number of collisions in the DMC algorithm and the proposed algorithms increases significantly as the number of players and the entering/leaving rate increases.

\begin{figure}[!h]

	\includegraphics[scale=0.4]{figures/d_coll_ap.pdf}%
	\centering
	\caption{Collision comparison for different scenarios considered for dynamic case with logarithmic scale on $y-axis$.}
	\label{dcl}
\end{figure}

\subsubsection{Scenario 6-7: Special cases}
Next two scenarios are same as the one considered in \cite{ICML16_MultiplayerBandits_RosenkiShamir}. We begin with the Scenario $6$ where the game starts with a set of six players and $10$ arms with statistics, $\boldmath{\mu}^1$. At every $T^{0.84}$ rounds, we alternate between a player leaving and a player entering the game. The leaving player is chosen at random from the set of current players. Fig.~\ref{dt11} and Fig.~\ref{dt12} show the cumulative regret and average regret, respectively, for the DMC, DT and DTS algorithms with $T= 5*10^5$. We also plot the results for larger horizon of  $T= 10*10^6$ rounds (Scenario $7$). Figure~\ref{dt21} and Figure~\ref{dt22} show the cumulative regret and average regret, respectively, for the DMC, DT and DTS algorithms for long horizon of $T= 10*10^6$ rounds. It can be observed that proposed DT and DTS algorithms offer better performance than the DMC algorithm for small as well as large horizons.

\begin{figure}[!h]

	\centering
	\subfigure[]{\label{dt11}\includegraphics[width=40mm]{figures/A1_mu1_CR_.pdf}}
	\subfigure[]{\label{dt12}\includegraphics[width=40mm]{figures/A1_mu1_AR_.pdf}}

	\caption{(a) Cumulative regret and (b) Average regret comparison for dynamic case for Scenario 6.}
	\label{dt1}
\end{figure}
\begin{figure}[!h]

	\centering
	\subfigure[]{\label{dt21}\includegraphics[width=40mm]{figures/A1_mu1_CR_LH_.pdf}}
	\subfigure[]{\label{dt22}\includegraphics[width=41mm]{figures/A1_mu1_AR_LH_.pdf}}
	\vspace{-0.45cm}

	\caption{(a) Cumulative regret and (b) Average regret comparison for dynamic case for Scenario 7.}
	\label{dt2}
\end{figure}

\subsubsection{Scenario 2: Testbed}
In Fig.~\ref{dt3}, we include results of the experiments conducted in real radio environment (shown in dotted lines) for the DMC and DT algorithms using the CRN set-up with network parameters set exactly as in Scenario $2$ . The simulation and experimental results show identical behavior validating the feasibility of the proposed algorithm in real environment. Similar results were observed for other scenarios as well but we omitted them due to limited space constraints. All the simulation and experimental results validate the analytical guarantees and gains of proposed algorithms over existing algorithms.

\begin{figure}[!h]

	\includegraphics[scale=0.4]{figures/A2_m11_.pdf}%
	\centering
\caption{Cumulative regret comparison for dynamic case for Scenario 2. The plots with markers refer to experimental results using an actual CR-Network testbed.}
	\label{dt3}
\end{figure}

\section{Introduction}

\input{Appendix}

\section{Appendix B: Simulation Results}
\label{SR_extra}
\input{simulation_results_ap}