\section{Introduction}
\label{sec:Introduction}
Manipulation of heavy and bulky objects is a challenging task for  manipulators and humanoid robots. An object is considered heavy if the manipulator's joint torques are not large enough to balance the object weight while lifting it off the ground. Thus, heavy objects cannot be manipulated with usual pick-and-place strategy due to actuator saturation.
Consider the manipulation scenario shown in Fig.~\ref{Fig:Motivation}, where a heavy object has to be moved from an initial pose $\mathcal{C}_O$ to a final pose $\mathcal{C}_F$ by a dual-armed robot. The object has to negotiate a step during the manipulation which implies that the final pose cannot be achieved by either pick-and-place strategies or by pushing. One possible way to move the object and negotiate the step is to use a sequence of pivoting motions, which we call object gaiting, and this is a common strategy used by humans to manipulate heavy objects. Therefore, the goal of this paper is to develop an algorithmic approach to compute a plan for manipulating heavy objects by a sequence of pivoting motions.

\begin{figure}[t]
    \centering
    \includegraphics[scale=0.4]{Figures/Motivation.pdf}
    \caption{Dual-handed manipulation of a heavy object between two given poses $\mathcal{C}_O$ and $\mathcal{C}_F$ by a sequence of pivoting motions.}
\label{Fig:Motivation}
\end{figure}

In a pivoting motion, we move the object while maintaining point or line contact with the environment. A point contact acts like a spherical joint, whereas a line contact acts like a revolute joint. The location and axes of these joints change during a gaiting motion. These joints are force-closed joints and can only be implemented through adequate frictional force at the object-ground contact that prevents slippage. Thus, a plan for pivoting operations consists of (a) {\em Motion plan}: a sequence of joint angles of the manipulators (that are within joint limits) and the corresponding object poses that maintains contact with the ground, (b) {\em Force plan}: a sequence of joint torques that are within the actuator limits and ensure that there is enough force at the object ground contact to prevent slippage. Furthermore, we also want to ensure that the manipulator does not lose the grasp of the object and there is no slippage at the hand-object contact. In this paper, we will focus on the motion planning problem. We have studied the problem of computing the force plan or force synthesis problem in~\cite{Patankar2020} and we will combine it with our motion plan to generate torques to achieve the motion.

The key challenge in solving the motion planning problem is that the kinematic constraints of the object maintaining a spherical or a revolute joint with the ground during the motion corresponds to nonlinear manifold constraints in the joint space of the manipulator. In sampling-based motion planning in joint space ($\mathbb{J}$-space), these constraints are hard to deal with, although there have been some efforts in this direction~\cite{BerensonSK11,JailletP12,Stilman10,KimU16,YaoK07, bonilla2015sample, KingstonMK2019}. Furthermore, in manipulation by gaiting, where we are performing a sequence of pivoting operations, these manifold constraints are not known beforehand since they depend on the choice of the pivot points (or lines) which has to be computed as a part of the plan. In this paper, we present a novel task-space ($\mathbb{T}$-space) based approach for generating the motion plan that exploits the fact that the kinematic constraints of a revolute or spherical joint constrains the motion of the object to a subgroup of $SE(3)$.

We present a two-step approach for computing the motion plan. In the first step, we develop an algorithm to compute a sequence of intermediate poses for the object to go from the initial to the goal pose. Two consecutive intermediate poses implicitly determine a point or line on the object and the ground that stay fixed during motion, thus encoding motion about a revolute or a spherical joint. In the second step, we use Screw Linear Interpolation (ScLERP) to determine a task space path between two intermediate poses, along with resolved motion rate control (RMRC)~\cite{whitney1969resolved,Pieper68} to convert the task space path to a joint space path. The advantage of using ScLERP is that it automatically satisfies the kinematic motion constraints during the pivoting motion without explicitly encoding it~\cite{Sarker2020}.
Thus, the joint space path that we compute along with the object path automatically ensures that the kinematic contact constraints are satisfied. This computationally efficient approach for motion planning for manipulation by pivoting is the key contribution of this paper. We also show that our motion plan can be combined with the second order cone programming (SOCP) based approach to compute joint torques and grasping forces~\cite{Patankar2020}, while ensuring that all no-slip constraints at the contacts and actuator limits are satisfied. We demonstrate our approach in simulation using a dual-armed Baxter robot.

\section{Related Work}
The use of external environment contacts to enhance the in-hand manipulation capability was first studied by Chavan-Dafle in \cite{Dafle2014}. More recently Hou \textit{et. al} have referred to the use of environment contact as \textit{shared grasping} wherein they treat the environment as an additional finger \cite{hou2020manipulation}. They have provided stability analysis of shared grasping by using \textit{Hybrid Force-Velocity Control} (HFVC).
Murooka \textit{et. al.} \cite{Murooka2015} proposed a method for pushing a heavy object by an arbitrary region of a humanoid robot. Polverini \textit{et. al.} \cite{Polverini2020} also developed a control architecture for a humanoid robot which is able to exploit the complexity of the environment to perform the pushing task of a heavy object.
Pivoting was first was first introduced by Aiyama \textit{et. al.} \cite{aiyama1993pivoting} as a new method of graspless/non-prehensile manipulation. Based on this method, Yoshida \textit{et. al.} \cite{yoshida2007pivoting,yoshida2008whole,yoshida2010pivoting} developed a whole-body motion planner for a humanoid robot to autonomously plan a pivoting
strategy for manipulating bulky objects. They first planned a sequence of collision-free Reeds and Shepp paths (especially straight and circular paths in $\mathbb{R}^2$), then convert these paths into a sequence of pivoting motions. However, this method is limited to the motion on Reeds and Shepp curves to satisfy a nonholonomic constraint, which is not always required. Thus, it is not a general, efficient, and optimum way to manipulate objects between two given poses, especially when there are no obstacles in the workspace.
Hence, we propose a general gait planning method as an optimization problem by defining the \textit{intermediate poses} and using the ScLERP to manipulate the object by gaiting between any two arbitrary poses.

\section{Preliminaries}

\noindent
\textbf{Quaternions and Rotations}: The quaternions are the set of hypercomplex numbers, $\mathbb{H}$. A quaternion $Q \in \mathbb{H}$ can be represented as a 4-tuple $Q = (q_0, \boldsymbol{q}_r) = (q_0, q_1, q_2, q_3)$, $q_0 \in \mathbb{R}$ is the real scalar part,
$\boldsymbol{q}_r=(q_1, q_2, q_3) \in \mathbb{R}^3$ corresponds to the imaginary part.
The conjugate, norm, and inverse of a quaternion $Q$ is given by
$Q^* = (q_0, -\boldsymbol{q}_r)$, $\lVert Q \rVert = \sqrt{Q Q^*} = \sqrt{Q^* Q}$,
and $Q^{-1} = Q^*/{\lVert Q \rVert}^2$, respectively. Addition and multiplication of two quaternions
$P = (p_0, \boldsymbol{p}_r)$ and
$Q = (q_0, \boldsymbol{q}_r)$ are performed as $P+Q = (p_0 + q_0, \boldsymbol{p}_r + \boldsymbol{q}_r)$ and $PQ = (p_0 q_0 - \boldsymbol{p}_r \cdot \boldsymbol{q}_r, p_0 \boldsymbol{q}_r + q_0 \boldsymbol{p}_r + \boldsymbol{p}_r \times \boldsymbol{q}_r)$.
The quaternion $Q$ is a \textit{unit quaternion}
if ${\lVert Q \rVert} = 1$, and consequently, $Q^{-1} = Q^*$. Unit quaternions are used to represent the set of all rigid body rotations,  $SO(3)$, the Special Orthogonal group of dimension $3$. Mathematically,  $SO(3)=\left\{\boldsymbol{R} \in \mathbb{R}^{3 \times 3}\left|\boldsymbol{R}^{\mathrm{T}} \boldsymbol{R}=\boldsymbol{R} \boldsymbol{R}^{\boldsymbol{T}}=\boldsymbol{I}_3,\right| \boldsymbol{R} \mid=1\right\}$, where $\boldsymbol{I}_3$ is a $3\times3$ identity matrix and $\left| \cdot \right|$ is the determinant operator. The unit quaternion corresponding to a rotation is $Q_R = (\cos\frac{\theta}{2}, \boldsymbol{l} \sin\frac{\theta}{2})$, where $\theta \in [0,\pi]$ is the angle of rotation about a unit axis $\boldsymbol{l} \in \mathbb{R}^3$.

\noindent
\textbf{Dual Quaternions and Rigid Displacements}:
In general, dual numbers are defined as $d = a + \epsilon b$ where $a$ and $b$ are elements of an algebraic field, and $\epsilon$ is a \textit{dual unit} with $\epsilon ^ 2 = 0, \epsilon \ne 0$.
Similarly, a dual quaternion $D$ is defined as $D= P + \epsilon Q$
where $P, Q \in \mathbb{H}$. The conjugate, norm, and inverse of the dual quaternion $D$ is represented as $D^* = P^* + \epsilon Q^*$, $\lVert D \rVert = \sqrt{D D^*} = \sqrt{P P^* + \epsilon (PQ^* + QP^*)}$, and $D^{-1} = D^*/{\lVert D \rVert}^2$,
respectively. Another definition for the conjugate of $D$ is represented as $D^\dag = P^* - \epsilon Q^*$. Addition and multiplication of two dual quaternions $D_1= P_1 + \epsilon Q_1$ and $D_2= P_2 + \epsilon Q_2$ are performed as $D_1 + D_2 = (P_1 + P_2) + \epsilon (Q_1 + Q_2)$ and $D_1 D_2 = (P_1 P_2) + \epsilon (P_1 Q_2 + Q_1 P_2) $.
The dual quaternion $D$ is a \textit{unit dual quaternion} if ${\lVert D \rVert} = 1$, i.e., ${\lVert P \rVert} = 1$ and $PQ^* + QP^* = 0$, and consequently, $D^{-1} = D^*$. Unit dual quaternions can be used to represent the group of rigid body displacements, $SE(3) = \mathbb{R}^3 \times SO(3)$, $S E(3)=\left\{(\boldsymbol{R}, \boldsymbol{p}) \mid \boldsymbol{R} \in S O(3), \boldsymbol{p} \in \mathbb{R}^{3}\right\}$. An element $\boldsymbol{T} \in SE(3)$, which is a pose of the rigid body, can also be expressed by a $4 \times 4$ homogeneous transformation matrix as
$\boldsymbol{T} = \left[\begin{smallmatrix}\boldsymbol{R}&\boldsymbol{p}\\\boldsymbol{0}&1\end{smallmatrix}\right]$ where $\boldsymbol{0}$ is a $1 \times 3$ zero vector. A rigid body displacement (or transformation) is represented by a unit dual quaternion $D_T = Q_R + \frac{\epsilon}{2} Q_p Q_R$ where $Q_R$ is the unit quaternion corresponding to rotation and  $Q_p = (0, \boldsymbol{p}) \in \mathbb{H}$ corresponds to the translation.

\noindent
\textbf{Screw Displacement}: Chasles-Mozzi theorem states that the general Euclidean displacement/motion of a rigid body from the origin $\boldsymbol{I}$ to $\boldsymbol{T} = (\boldsymbol{R},\boldsymbol{p}) \in SE(3)$
can be expressed as a rotation $\theta$ about a fixed axis $\mathcal{S}$, called the \textit{screw axis}, and a translation $d$ along that axis (see Fig.~\ref{Fig:ScrewDisplacement}). Plücker coordinates can be used to represent the screw axis by $\boldsymbol{l}$ and $\boldsymbol{m}$, where $\boldsymbol{l} \in \mathbb{R}^3$ is a unit vector that represents the direction of the screw axis $\mathcal{S}$, $\boldsymbol{m} = \boldsymbol{r} \times \boldsymbol{l}$, and $\boldsymbol{r} \in \mathbb{R}^3$ is an arbitrary point on the axis. Thus, the screw parameters are defined as $\boldsymbol{l}, \boldsymbol{m}, \theta, d$.
The screw displacements can be expressed by the dual quaternions as $D_T = Q_R + \frac{\epsilon}{2} Q_p Q_R = (\cos \frac{\Phi}{2}, L \sin \frac{\Phi}{2})$ where $\Phi = \theta + \epsilon d$ is a dual number and $L = \boldsymbol{l} + \epsilon \boldsymbol{m}$ is a
dual vector.
A power of the dual quaternion $D_T$ is then defined as $D_T^{\tau} = (\cos \frac{\tau \Phi}{2}, L \sin \frac{\tau \Phi}{2})$, $\tau >0$.

\begin{figure}[!htbp]
    \centering
    \includegraphics[scale=0.57]{Figures/ScrewDisplacement.pdf}
    \caption{Screw displacement from pose $\mathcal{C}_1$ to pose $\mathcal{C}_2$.}
\label{Fig:ScrewDisplacement}
\end{figure}

\noindent
\textbf{Screw Linear Interpolation (ScLERP)}: To perform a one degree-of-freedom smooth screw motion (with a constant rotation and translation rate) between two object poses in $SE(3)$, the screw linear interpolation (ScLERP) can be used. The ScLERP provides a \textit{straight line} in $SE(3)$ which is the closest path between two given poses in $SE(3)$.
If the poses are represented by unit dual quaternions $D_{1}$ and $D_{2}$, the path provided by the ScLERP is derived by $D(\tau) = D_1 (D_1^{-1}D_2)^{\tau}$ where $ \tau \in[0,1]$ is a scalar path parameter.
As $\tau$ increases from 0 to 1, the object moves between two poses along the path
$D(\tau)$ by the rotation $\tau \theta$ and translation $\tau d$. Let $D_{12} = D_1^{-1}D_2$. To compute $D_{12}^\tau$, the screw coordinates $\boldsymbol{l}, \boldsymbol{m}, \theta, d$ are first extracted from $D_{12} = P + \epsilon Q = (p_0,\boldsymbol{p}_r) + \epsilon (q_0,\boldsymbol{q}_r) = (\cos\frac{\theta}{2}, \boldsymbol{l} \sin\frac{\theta}{2}) + \epsilon Q$ by $\boldsymbol{l} = \boldsymbol{p}_r/ \lVert \boldsymbol{p}_r \lVert $, $\theta = 2 \, \mathrm{atan2}(\lVert \boldsymbol{p}_r \lVert, p_0)$, $d = \boldsymbol{p} \cdot \boldsymbol{l}$, and $\boldsymbol{m} = \frac{1}{2} (\boldsymbol{p} \times \boldsymbol{l} + (\boldsymbol{p}-d \boldsymbol{l})\cot \frac{\theta}{2})$ where $\boldsymbol{p}$ is derived from $2QP^* = (0, \boldsymbol{p})$ and $\mathrm{atan2}(\cdot)$ is the two-argument arctangent. Then, $D_{12}^\tau = (\cos \frac{\tau \Phi}{2}, L \sin \frac{\tau \Phi}{2})$ is directly derived from $\left(\cos \frac{\tau \theta}{2}, \sin \frac{\tau \theta}{2}\boldsymbol{l}\right)+\epsilon \left( -\frac{\tau d}{2}\sin \frac{\tau \theta}{2}, \frac{\tau d}{2}\cos \frac{\tau \theta}{2}\boldsymbol{l}+\sin \frac{\tau \theta}{2}\boldsymbol{m} \right) $. Note that $\theta =0, \pi$ corresponds to pure translation between two poses and the screw axis is at infinity.

\section{Problem Statement}
\label{sec:ProblemStatement}
Let us assume that we want to manipulate a heavy cuboid object quasi-statically by using $n$ manipulators, while maintaining contact with environment, from an initial pose $\mathcal{C}_O \in SE(3)$ to a final pose $\mathcal{C}_F \in SE(3)$.
We also assume that the object always remains in the manipulators' workspace.
Figure~\ref{Fig:Cube_Manipulator} shows a cuboid object in contact with the environment at the vertex $v$ and also with the $i$-th manipulator's end-effector at the contact point $c_i$ (where $i=1,..,n$). Contact coordinate frames \{$c_i$\} and \{$v$\}  are attached to the object at each manipulator and environment contact, respectively, such that $\bm{n}$-axis of the frames is normal (inward) to the object surface and two other axes, $\bm{t}$ and $\bm{o}$, are tangent to the surface. The coordinate frame \{$b$\} is attached to the object center of mass, coordinate frame \{$e_i$\} is attached to the $i$-th end-effector, and \{$s$\} is the inertial coordinate frame.
Let $\boldsymbol{\Theta}^i = [\theta_1^i, \theta_2^i, \cdots, \theta_{l_i}^i] \in \mathbb{R}^{l_i}$ be the vector of joint angles of the $i$-th $l_i$-DoF manipulator, which represents the \textit{joint space} ($\mathbb{J}$-space) or the \textit{configuration space} ($\mathbb{C}$-space) of the manipulator.
Moreover, $\mathcal{E}^i \in SE(3)$ is defined as the pose of the end-effector of the $i$-th manipulator where $\mathcal{E}^i = \mathcal{FK}(\boldsymbol{\Theta}^i)$ and $\mathcal{FK}(\cdot)$ is the manipulator forward kinematics map. Therefore, $\boldsymbol{\Theta}_O^i \in \mathbb{R}^{l_i}$ and $\mathcal{E}_O^i \in SE(3)$ represent the initial configuration of the $i$-th manipulator (in $\mathbb{J}$-space) and pose of $i$-th end-effector, respectively, corresponding to the object initial pose $\mathcal{C}_O$ and $\boldsymbol{\Theta}_F^i \in \mathbb{R}^{l_i}$ and $\mathcal{E}_F^i \in SE(3)$ represent the final configuration of the $i$-th manipulator (in $\mathbb{J}$-space) and pose of $i$-th end-effector, respectively, corresponding to the object final pose $\mathcal{C}_F$. We assume that the position of the manipulator-object contact $c_i$ is given and the transformation between the frames $\{e_i\}$ and $\{c_i\}$ remains constant during the manipulation, i.e., there is no relative motion at the contact interface.

Our motion planning problem is now defined as computing a sequence of joint angles $\boldsymbol{\Theta}^i(j)$, where $j=1,\cdots,m$, $\boldsymbol{\Theta}^i(1) = \boldsymbol{\Theta}^i_O$, $\boldsymbol{\Theta}^i(m) = \boldsymbol{\Theta}^i_F$, to manipulate the object while maintaining contact with the environment from its initial pose $\mathcal{C}_O$ to a final pose $\mathcal{C}_F$ when $(\mathcal{C}_O, \mathcal{E}_O^i, \boldsymbol{\Theta}_O^i)$ and $(\mathcal{C}_F, \mathcal{E}_F^i)$ ($i=1,...,n$) are given. Moreover, our force planning problem is computing the minimum contact wrenches required to be applied at $c_i$ during the object manipulation to balance the external wrenches (e.g., gravity) and also the environment contact wrenches using the method we have presented in \cite{Patankar2020}.

\begin{figure}[!htbp]
    \centering
    \includegraphics[scale=0.5]{Figures/Cuboid_Manipulator.pdf}
    \caption{An cuboid object being tilted at one of its vertices.}
\label{Fig:Cube_Manipulator}
\end{figure}

\textbf{Solution Approach Overview}: Generally speaking, to move an object while maintaining contact we can use two primitive motions, namely, (1) \textit{sliding} on a vertex, edge, or face of the object in contact with the environment (Fig.~\ref{Fig:SRP}-\subref{Fig:SRP_S}) and
(2) \textit{pivoting} about an axis passing through a vertex, edge, or face of the object in contact with the environment (Fig.~\ref{Fig:SRP}-\subref{Fig:SRP_T},\subref{Fig:SRP_P}, Fig.~\ref{Fig:Motivation}). All other motions can be made by combining these primitive motions. Note that we consider \textit{tumbling} as a special case of pivoting when the axis of rotation passes through an object edge or face. Manipulation by sliding (or pushing) can be useful in many scenarios like picking a penny off a table. However, in heavy and bulky object manipulation scenarios, sliding may not give feasible solutions. Thus, in this paper, we will focus on manipulation using the pivoting primitive.

Our \textit{manipulation strategy} can be described briefly as follows.
(i) Given the initial and final pose of the object, we first determine if multiple pivoting moves have to be made and, if necessary, compute intermediate poses of the object. (ii) Using the dual quaternion representation of these poses, we compute  paths in $SE(3)$ using the ScLERP for the object and end-effectors. These paths automatically satisfies all the basic task related constraints (without any additional explicit representation of the constraints).
(iii) We use the (weighted) pseudoinverse of the Jacobian to derive the joint angles in the $\mathbb{J}$-space from the computed $\mathbb{T}$-space path. (iv) Finally, we compute the minimum required contact wrenches and manipulators' joint torques required for object manipulation. Note that the steps (ii) to (iv) can be done either sequentially or they can be interleaved in a single discrete time-step.

\section{Pivoting}
Pivoting is a motion where an object is moved while maintaining a point or line contact with a support surface. When an object maintains a point contact, the constraints on motions are same as those imposed by a spherical joint. Thus, the motion of the object is restricted to $SO(3)$, which is a subgroup of $SE(3)$, and the axis of rotation passes through the contact point. During pivoting with line contact (or tumbling), the constraint on the motion is same as that imposed by a revolute joint with the axis of the joint being the line of contact. Thus, in this case, the motion of the object is restricted to $SO(2)$, which is also a subgroup of $SE(3)$. This mathematical structure of pivoting motions is key to our approach as we discuss below.

Suppose an object can reach a goal pose from a start pose using a single pivoting motion. This can happen when the start and the goal poses are such that there is a common vertex, say $v$, between the start and goal poses that lie on the support surface (see Fig.~\ref{Fig:SRP}-\subref{Fig:SRP_P}). In such situations, when planning in $\mathbb{T}$-space, one should be careful about the interpolation scheme for generating the motion of the object. If we use linear interpolation between the end poses in the space of parameters (a popular choice being linear interpolation for position and spherical linear interpolation for orientation using unit quaternion parameterization of orientation), the resulting intermediate poses will not ensure that the contact between the object and the support surface is maintained. The motion obtained will also change with the choice of the coordinate frames for the initial and final pose. The advantage of using ScLERP is that it is coordinate invariant. Furthermore, since the pivoting motions also belongs to a subgroup of $SE(3)$, ScLERP ensures that all the intermediate poses will lie in the same subgroup that contains the initial and goal pose (i.e., all intermediate poses will have the vertex $v$ fixed to the support surface). Thus, it is not necessary to explicitly enforce the pivoting constraints for motion planning. Lemma $1$ formalizes this discussion.

\begin{figure}[!htbp]
    \centering
    \subfloat[]{\includegraphics[scale=0.36]{Figures/S.pdf}\label{Fig:SRP_S}}

    \subfloat[]{\includegraphics[scale=0.36]{Figures/T.pdf}\label{Fig:SRP_T}}
    \subfloat[]{\includegraphics[scale=0.36]{Figures/P.pdf}\label{Fig:SRP_P}}
    \caption{Examples of the primitive motions for manipulating polyhedral objects by exploiting the environment contact, (a) sliding or pushing on a face, (b) pivoting about an edge (tumbling), (c) pivoting about a vertex.}
\label{Fig:SRP}
\end{figure}

\begin{lemma}
Let $D_1 = Q_{R1} + \frac{\epsilon}{2} Q_{p1} Q_{R1}$ and $D_2 = Q_{R2} + \frac{\epsilon}{2} Q_{p2} Q_{R2}$ be two unit dual quaternions representing two poses of a rigid body. If a point $\boldsymbol{v} \in \mathbb{R}^3$ in the rigid body has the same position in both poses, the position of this point remains the same in all the poses provided by the ScLERP $D(\tau) = D_1 (D_1^{-1}D_2)^{\tau}$ where $ \tau \in[0,1]$.
\label{lemma:FixedPoint}
\end{lemma}
\begin{proof}
Let $Q_v = (0,\boldsymbol{v}) \in \mathbb{H}$ be a pure quaternion representing the point $\boldsymbol{v}$. Since the point $\boldsymbol{v}$ has the same position in both poses $D_1$ and $D_2$, therefore
\begin{align}
D_1(1+\epsilon Q_v)D_1^\dag & =  D_2(1+\epsilon Q_v)D_2^\dag, \\
\therefore  \,\, Q_{p2} - Q_{p1} & = Q_{R1} Q_v Q_{R1}^*  - Q_{R2} Q_v Q_{R2}^*.
\label{eq:Qp2_Qp1}
\end{align}
Therefore, the transformation from $D_1$ to $D_2$ is derived as
\begin{equation}
\begin{split}
        D_{12} & = D_1^{*}D_2 = Q_{R1}^* Q_{R2} + \frac{\epsilon}{2}Q_{R1}^* (Q_{p2} - Q_{p1}) Q_{R2}\\
        &= Q_{R1}^* Q_{R2} + \frac{\epsilon}{2}(Q_v Q_{R1}^* Q_{R2} - Q_{R1}^* Q_{R2} Q_v).
\end{split}
\label{eq:D_1D_2_}
\end{equation}
By representing the rotation $Q_{R1}^* Q_{R2}$ as $(\cos\frac{\theta}{2}, \boldsymbol{l} \sin\frac{\theta}{2}) \in \mathbb{H}$ (where $\boldsymbol{l}$ is a unit vector along the  screw axis and $\theta$ is rotation about the screw axis), ($\ref{eq:D_1D_2_}$) can be simplified as
\begin{equation}
    D_{12} = (\cos\frac{\theta}{2}, \boldsymbol{l}\sin\frac{\theta}{2}) + \epsilon (0, \boldsymbol{v} \times \boldsymbol{l} \sin\frac{\theta}{2}) = P + \epsilon Q.
\label{eq:D_12}
\end{equation}
The translation $d$ along the screw axis is determined by $d = \boldsymbol{p} \cdot \boldsymbol{l}$ where $\boldsymbol{p}$ is derived from $2QP^* = (0, \boldsymbol{p})$. By using (\ref{eq:D_12}),
\begin{equation}
\boldsymbol{p} = \boldsymbol{v} \times \boldsymbol{l} \sin\frac{\theta}{2} \cos\frac{\theta}{2} - (\boldsymbol{v} \times \boldsymbol{l}) \times \boldsymbol{l} \sin^2\frac{\theta}{2},
\label{eq:p}
\end{equation}
and $d = \boldsymbol{p} \cdot \boldsymbol{l} = 0$. Therefore, the transformation $D(\tau)$ is a pure rotation about the fixed point $\boldsymbol{v}$ on the screw axis.
\end{proof}

Furthermore, when using multiple manipulators to pivot an object and we assume that there is no relative motion at the hand-object contact, the motion of each end-effector can be obtained independently by ScLERP using a shared interpolation parameter. This will ensure that the  constraint that the relative end-effector poses of the manipulators are unchanged during motion is maintained  without explicitly encoding it (this follows from Lemma $3$ of~\cite{Sarker2020} and so we do not repeat the formal statements and proofs here). In the next section, we use pivoting as a primitive motion for motion planning between any two given poses in $\mathbb{T}$-space.

\section{Motion Planning in Task Space}
\label{sec:MotionPlanningTS}
To manipulate a polyhedral object between any two given poses $\mathcal{C}_O$ and $\mathcal{C}_F$ while maintaining contact with the environment, multiple pivoting moves can be combined by defining a set of appropriate \textit{intermediate poses}. The set of the intermediate poses $\mathcal{C}_I = \{\mathcal{C}_I^1,\mathcal{C}_I^2, \cdots, \mathcal{C}_I^h \}$ are defined in a way that the motion between any two successive poses $\{\mathcal{C}_O, \mathcal{C}_I, \mathcal{C}_F \}$ can be represented by a single constant screw pivoting move.
Thus, we can conveniently represent the motion between any two given object poses $\mathcal{C}_O$ and $\mathcal{C}_F$ in $SE(3)$ by using ScLERP to ensure that the object maintains its contact with the environment continuously. The object manipulation strategies on a flat surface can be categorized into 3 cases; (\textbf{Case I}) If $\mathcal{C}_O$ and $\mathcal{C}_F$ have a contact edge or vertex in common, the final pose can be achieved by pivoting the object about the common point or edge (Fig.~\ref{Fig:SRP}-\subref{Fig:SRP_T},\subref{Fig:SRP_P}).
(\textbf{Case II}) If $\mathcal{C}_O$ and $\mathcal{C}_F$ do not have any edge or vertex in common but the same face of the object is in contact with the environment in both poses, different strategies can be considered. One of the strategies is using a sequence of pivoting motions about the object edges (tumbling). In this motion, the travel distance is discrete and depends on the object size and it may not be suitable for manipulating some objects like furniture.
In this situation, we can manipulate the object is \textit{object gaiting} (Fig.~\ref{Fig:IntermediateConfigs_Edges}-a) which is defined as a sequence of pivoting motions on two adjacent object vertices in contact
(see \ref{subsec:IntermediatePosesObjectGaiting} and \ref{subsec:GaitPlanning}).
(\textbf{Case III}) If
the adjacent or opposite faces of the object are in contact with the environment in both poses, a combination of pivoting and gaiting is required to achieve the final pose as shown in Fig.~\ref{Fig:Examples}. Depending on the manipulators' physical limitations, object gaiting is more efficient only when a specific face of the object is in contact with the environment. For instance, manipulation on the longer edge of the cuboid shown in Fig.~\ref{Fig:IntermediateConfigs_Edges}-a may be more difficult than two other edges.

\begin{figure}[!htbp]
    \centering
    \subfloat[]{\includegraphics[scale=0.26]{Figures/CaseIV_A.pdf}}
    \subfloat[]{\includegraphics[scale=0.26]{Figures/CaseIV_B.pdf}}
    \subfloat[]{\includegraphics[scale=0.26]{Figures/CaseV_A.pdf}}

    \caption{Examples of the object manipulation with primitive motions when two adjacent (a,b) or opposite (c) object faces are in contact with the environment in initial and final poses (P: Pivoting, G: Gaiting).}
\label{Fig:Examples}
\end{figure}

\subsection{Intermediate Poses in Object Gaiting}
\label{subsec:IntermediatePosesObjectGaiting}
Let us assume that the axes of the body frame $\{b\}$ are parallel to the cuboid edges and the inertia frame $\{s\}$ is attached to the supporting plane such that $Z$-axis is perpendicular to the plane (Fig.~\ref{Fig:ObjectGaiting}). Three successive intermediate poses while pivoting about the vertex $a$ are shown in Fig.~\ref{Fig:ObjectGaiting}-a,b. The object is initially in the pose $\mathcal{C}_I^1 = (R_{1}, p_{1})$ (Fig.~\ref{Fig:ObjectGaiting}-a) holding on the contact edge $ab$. The angle $\gamma$ can be determined such that the object weight pass through the contact edge $ab$ to reduce the required contact forces during the manipulation. The pose $\mathcal{C}_I^2 = (R_{2}, p_{2})$ (Fig.~\ref{Fig:ObjectGaiting}-a) is achieved by rotating the object by a small angle $\beta$ along the edge passing through the vertex $a$; therefore, $R_{2} = R_{1} R_{x}({-\beta})$ and only the vertex $a$ is in the contact. Note that the angle $\beta$ can be adjusted during the motion to allow the object to pass over small obstacles in the environment. Finally, the pose $\mathcal{C}_I^3 = (R_{3}, p_{3})$ (Fig.~\ref{Fig:ObjectGaiting}-b) is determined by rotating $\mathcal{C}_I^1$ by an angle $\alpha$ along $Z$-axis about the vertex $a$; therefore, $R_{3} = R_Z({\alpha}) R_{1}$ and the edge $ab$ is again in contact with the environment. This procedure can be also repeated for the vertex $b$. Using the intermediate poses ScLERP can be used to derive a smooth motion for object gaiting while maintaining contact with the environment.

\begin{figure}[!htbp]
    \centering
    \subfloat[]{\includegraphics[scale=0.55]{Figures/I2.pdf}} \quad \quad
    \subfloat[]{\includegraphics[scale=0.55]{Figures/I3.pdf}}
    \caption{Intermediate poses in object gaiting while pivoting.}
\label{Fig:ObjectGaiting}
\end{figure}

\subsection{Gait Planning}
\label{subsec:GaitPlanning}
In order to manipulate the object from an initial pose $\mathcal{C}_O$ to a final pose $\mathcal{C}_F$ by object gaiting, a sequence of the rotation angle $\alpha$ between these two poses should be properly determined (Fig.~\ref{Fig:IntermediateConfigs_Edges}-a). Let $k$ be the number of required edge contacts and $\bm{\alpha} = [\alpha_1, \cdots, \alpha_k]^T \in \mathbb{R}^k$ be the angles between the contact edges as shown in Fig.~\ref{Fig:IntermediateConfigs_Edges}-b. We can find $\bm{\alpha}$ using an optimization problem as
\begin{equation}
\begin{aligned}
&{\underset {\bm{\alpha}}{\operatorname {minimize}}}&&  \lVert \bm{\alpha} \rVert \\[-8pt]
&\operatorname {subject\;to} && \boldsymbol{x} = \pm w \sum_{i=1}^k{\left( -1 \right) ^{i}\left[ \begin{array}{@{\mkern0mu} c @{\mkern0mu}}
	\cos \left( \alpha _O \pm \bar{\alpha} \right)\\
	\sin \left( \alpha _O \pm \bar{\alpha} \right)\\
\end{array} \right]},\\[-5pt]
&&& \alpha _{F} - \alpha _O  = \pm \sum_{i=1}^k{\left( -1 \right) ^{i}\alpha_i },\\
&&& \left| \alpha _i \right| \leq \alpha_{\text{max}},\ \ i=1,...,k,
\end{aligned}
\label{eq:GaitPlanning}
\end{equation}
where $\bar{\alpha} = \sum_{j=1}^i{\left( -1 \right) ^{j}\alpha _j }$, $\alpha_{\text{max}}$ is the maximum allowed rotation angle, $w$ is the length of the edge contact, and $\alpha_{O}$ and $\alpha_{F}$ represent the orientation of the contact edges $a_O b_O$ and $a_F b_F$ relative to $X$-axis, respectively. The negative sign correspond to the case that the first gait begins from the edge $a_O$, where $\boldsymbol{x} = \boldsymbol{b}_{F} - \boldsymbol{a}_{O}$ if $k$ is an odd number and $\boldsymbol{x} = \boldsymbol{a}_{F} - \boldsymbol{a}_{O}$ if $k$ is an even number, moreover, the positive sign correspond to the case that the first gait begins from the edge $b_O$, where $\boldsymbol{x} = \boldsymbol{a}_{F} - \boldsymbol{b}_{O}$ if $k$ is an odd number and $\boldsymbol{x} = \boldsymbol{b}_{F} - \boldsymbol{b}_{O}$ if $k$ is an even number. $\boldsymbol{a}_{O}$, $\boldsymbol{b}_{O}$, $\boldsymbol{a}_{F}$, $\boldsymbol{b}_{F} \in \mathbb{R}^2$ are the coordinates of the contact vertices in $\mathcal{C}_O$ and $\mathcal{C}_F$ poses along $X$- and $Y$-axis of the frame $\{s\}$. In the optimization problems (\ref{eq:GaitPlanning}), the first constraint represents the distance of the last contact vertex ($a_F$ or $b_F$) relative to the first contact vertex ($a_O$ or $b_O$) in $X$ and $Y$ directions. The second constraint represents the relative angle between the contact edges $a_O b_O$ and $a_F b_F$, and the last constraint considers the manipulators' limitations to rotate the object.
In order to find the feasible minimum number of edge contacts, $k$, required to manipulate the object between two poses $\mathcal{C}_O$ and $\mathcal{C}_F$, we need to repeat (\ref{eq:GaitPlanning}) for different values of $k$.

\begin{figure}[!htbp]
    \centering
    \subfloat[]{\includegraphics[scale=0.4]{Figures/ObjectGaiting.pdf}} \quad \,
    \subfloat[]{\includegraphics[scale=0.34]{Figures/IntermediateConfigs_Edges_Obstacle.pdf}}
    \caption{A sequence of contact edges for object gaiting between two poses $\mathcal{C}_O$ and $\mathcal{C}_F$ when the first gait begins from the edge $a_O$.}
\label{Fig:IntermediateConfigs_Edges}
\end{figure}

\section{Mapping from $\mathbb{T}$-space to $\mathbb{J}$-space}
Since it is assumed that the transformation between the end-effector frame $\{e_i\}$ and contact frame $\{c_i\}$ remains constant,
after planning a path in the $\mathbb{T}$-space, we can compute the end-effector poses $\mathcal{E}_i$ for each object intermediate pose.
Then, we use the ScLERP for each of these end-effector poses individually with a shared screw parameter \cite{daniilidis1999hand,kavan2006dual}. To find the joint angles of the manipulators in $\mathbb{J}$-space, we use the (weighted) pseudoinverse of the manipulators' Jacobian \cite{Klein1983}.
Let $\boldsymbol{\Theta}_{t}$ and $\boldsymbol{\chi}_{t}$ be the vector of joint angles and end-effector’s pose at the step $t$, respectively.
For each manipulator, given the current end effector pose $\boldsymbol{\chi}_{t}$ and the target end effector pose $\boldsymbol{\chi}_{t+1}$ (obtained from ScLERP) we have the corresponding joint angles $\boldsymbol{\Theta}_{t+1}$ as
\begin{equation}
    \boldsymbol{\Theta}_{t+1} = \boldsymbol{\Theta}_{t} + \lambda \boldsymbol{J}(\boldsymbol{\Theta}_{t}) (\boldsymbol{\chi}_{t+1} - \boldsymbol{\chi}_{t}),
    \label{eq:IK}
\end{equation}
where  $0 < \lambda \le 1$ is a step length parameter (refer to \cite{Sarker2020} for a complete algorithm). Here $\boldsymbol{J}$ is the (weighted) pseudo-inverse of the manipulator Jacobian. By using (\ref{eq:IK}) between any two successive poses in $\{\mathcal{C}_O, \mathcal{C}_I, \mathcal{C}_F \}$, $\boldsymbol{\Theta}^i(j)$ ($j=1,\cdots,m$) for the $i$-th manipulator is computed.

\section{Implementation and Results}
In this section, we briefly present the simulation results for manipulating a heavy cuboid object on a flat surface and over a step.
Videos of our simulations are presented in the video attachment to the paper.

\noindent
\textbf{Manipulation on a Flat Surface}: In this example, we plan motion to reorient a heavy object from an initial pose $\mathcal{C}_O$ to a final pose $\mathcal{C}_F$, in its vicinity, by object gaiting as shown in Fig.~\ref{Fig:Example_Flat}-\subref{Fig:Example_Flat_Object}.
Existing planning algorithms~\cite{yoshida2010pivoting} cannot efficiently solve this problem, because their motion plan is essentially restricted to move on Reeds and Shepp curves.
By using the proposed optimization problem (\ref{eq:GaitPlanning}), we can find the minimum number of contact edges required to manipulate the object between these two poses. The simulation results are shown in Fig.~\ref{Fig:Example_Flat}-\subref{Fig:Example_Flat_Edges}. As shown, at least 3 contact edges (in total 7 intermediate poses) are required to reach the final pose by starting pivoting from the edge $a$.

\begin{figure}[!htbp]
    \centering
    \subfloat[]{\includegraphics[scale=0.44]{Figures/Example_Flat.pdf}\label{Fig:Example_Flat_Object}} \qquad
    \subfloat[]{\includegraphics[scale=0.34]{Figures/Example_Flat_Edges.pdf}\label{Fig:Example_Flat_Edges}}
    \caption{Object gaiting on a flat surface where $a_O = [0, \, 0]$, $\alpha_{O} = 0^{\circ}$, $a_F = [0.13, \, 0.13]$m, $\alpha_F = -80^{\circ}$,  $w = 0.2$m, $\alpha_{\text{max}} = 35^{\circ}$, $\alpha_1 = -10.55^{\circ}$, $\alpha_2 = 29.56^{\circ}$, $\alpha_3 = -12.63^{\circ}$, $\alpha_4 = 27.25^{\circ}$.}
\label{Fig:Example_Flat}
\end{figure}

\noindent
\textbf{Manipulation over a Step}: In this example, we plan motion and force to manipulate a heavy object over a step (Fig.~\ref{Fig:Example_Step}) by both 7-DoF arms of Baxter robot.
The computed motion plan includes 3 stages: (1) pivoting about the object edge ($\mathcal{C}_I^1$), (2) pivoting about the vertex $v$ ($\mathcal{C}_I^2$), where the object face and only the vertex $v$ are in contact with the environment, (3) changing the location of the end-effectors' contacts and pivoting about the step edge ($\mathcal{C}_F$). Thus, we have two intermediate poses $\{\mathcal{C}_I^1,\mathcal{C}_I^2\}$.
We implemented $\mathbb{T}$-space planning, conversion to $\mathbb{J}$-space, and our force planning method described in \cite{Patankar2020} to find the minimum required normal forces $f_{c_{n,1}}$ and $f_{c_{n,2}}$ at both object--end-effector contacts $\{c_1\}$ and $\{c_2\}$ in each motion stage.
In Fig.~\ref{Fig:contact_force_results}, the variations of the normal contact forces with respect to the number of iterations to reach the goal pose at 3 stages of object manipulation over a step are shown.
In stage 1, $f_{c_{n,1}}$ and $f_{c_{n,2}}$ first decrease and become negligible at a particular object tilting angle where the weight of the object passes through its support edge, and then increases. In stage 2, since the motion is not symmetric, there is a difference between the right and left end-effector normal contact forces in order to balance the the object weight. In stage 3, the object-environment contact points are initially located closer to the object center of mass; thus, less contact forces are initially required and by pivoting the object, these forces increases.

\begin{figure}[!htbp]
    \centering
    \includegraphics[scale=0.45]{Figures/Example_Step.pdf}
    \caption{Object manipulation over a step.}
\label{Fig:Example_Step}
\end{figure}

\begin{figure}[!htbp]
    \centering
    \includegraphics[scale=0.29]{Figures/fc_complete.pdf}
    \caption{The normal contact forces at $\{c_1\}$ and $\{c_2\}$ where the object weight is $m = 2$kg, maximum joint torque for shoulder and elbow joints is $\tau_{\text{max}} = 50$Nm, and maximum joint torque for  wrist joints is $\tau_{\text{max}} = 15$Nm.}
\label{Fig:contact_force_results}
\end{figure}

\section{Conclusion and Future Work}
In this paper, we have proposed a novel approach for manipulating heavy objects while using a sequence of pivoting motions. We have implemented our proposed motion and force planning on two different scenarios; reorienting an object using gaiting and also manipulating a heavy object over a step. Given the initial and final poses of the object, we first compute the required intermediate poses. These poses can be derived by an optimization problem which computes the optimal values of the rotation angles between contact edges while \textit{object gaiting}. Then, by using ScLERP, we can interpolate between these intermediate poses while satisfying all the task-related constraints.
Using RMRC we can map the task-space based plan to the joint-space allowing us to compute the contact forces and the joint torques required to manipulate the object. Future work includes the relaxation of the quasi-static assumption for the force planning and experimental evaluation of the proposed approach.

\addtolength{\textheight}{-10.5cm}

\bibliographystyle{IEEEtran}


\section{Introduction} \label{sec:intro}

With the development of natural language processing and deep learning, multilingual machine translation has gradually attracted the interest of researchers \citep{dabre-etal-2020-multilingual}.
Moreover, the multilingual machine translation model demands less space than multiple bilingual unidirectional machine translation models, making it more popular among developers \citep{liu2020multilingual, zhang-etal-2020-improving, fan2020beyond}.

However, existing multilingual machine translation models face imbalance problems.
On the one hand, various sizes of training corpora for different language pairs cause imbalance.
Typically, the training corpora size of some high resource languages (HRLs) is hundreds or thousands of times that of some low resource languages (LRLs) \citep{schwenk2019ccmatrix}, resulting in lower competence of LRL learning.
On the other hand, translation between different languages has different difficulty, which also leads to imbalance.
In general, translation between closely related language pairs is more effortless than that between distant language pairs, even if the training corpora is of the same size \citep{barrault-etal-2020-findings}.
This would lead to low learning competencies for distant languages compared to closely related languages.
Therefore, multilingual machine translation is inherently imbalanced, and dealing with this imbalance is critical to advancing multilingual machine translation \citep{dabre-etal-2020-multilingual}.

To address the above problem, existing balancing methods can be divided into two categories, i.e., static and dynamic.
1) Among static balancing methods, temperature-based sampling \citep{arivazhagan2019massively} is the most common one, compensating for the gap between different training corpora sizes by oversampling the LRLs and undersampling the HRLs.
2) Researchers have also proposed some dynamic balancing methods \citep{jean2019adaptive, wang-etal-2020-balancing}.
\citet{jean2019adaptive} introduce an adaptive scheduling, oversampling the languages with poorer results than their respective baselines.
In addition, MultiDDS-S \citep{wang-etal-2020-balancing} focus on learning an optimal strategy to automatically balance the usage of training corpora for different languages at multilingual training.

Nevertheless, the above methods focus too much on balancing LRLs, resulting in lower competencies for HRLs compared to that trained only on bitext corpora.
Consequently, the performance on the HRLs by the multilingual translation model is inevitably worse than that of bitext models by a large margin \citep{lin-etal-2020-pre}.
Besides, knowledge learned by related HRLs is also beneficial for LRLs \citep{neubig-hu-2018-rapid}, while is neglected by previous approaches, limiting the performance on LRLs.

\iffalse
Therefore, in this paper, we we manage to balance the competencies of languages and propose a \emph{\textbf{C}ompetence-based \textbf{C}urriculum \textbf{L}earning Approach for \textbf{M}ultilingual Machine Translation}, named CCL-M.
Specifically, we treat the HRLs as easy samples, the LRLs as hard samples, and learn all the samples from easy to hard through curriculum learning \citep{bengio2009curriculum}.
Hence, we define two competencies to help scheduling: 1) \emph{Self-evaluated Competence}, evaluating how well a language is learned; 2) \emph{HRLs-evaluated Competence}, evaluating how well an LRL is initialized using HRLs' \emph{Self-evaluated Competence}.
We further define two sets, the training set and the candidate set, to determine whether to train a language.
At the beginning of the training, we assign the HRLs to the training set and the LRLs to the candidate set.
For the languages in the training set, we gradually calculate their \emph{Self-evaluated Competence} and apply sampling weight to the languages with the reciprocal of their \emph{Self-evaluated Competence}.
For the LRLs out of the training set, we also gradually calculate their \emph{HRLs-evaluated Competence} and add the LRLs to the training set when its \emph{HRLs-evaluated Competence} is larger than a certain threshold.
Eventually, all the languages are added to the training set.
\fi

Therefore, in this paper, we try to balance the learning competencies of languages and propose a \emph{\textbf{C}ompetence-based \textbf{C}urriculum \textbf{L}earning Approach for \textbf{M}ultilingual Machine Translation}, named CCL-M.
Specifically, we firstly define two competence-based evaluation metrics to help schedule languages, which are 1) \emph{Self-evaluated Competence}, for evaluating how well the language itself has been learned; and 2) \emph{HRLs-evaluated Competence}, for evaluating whether an LRL is ready to be learned by the LRL-specific HRLs' \emph{Self-evaluated Competence}.
Based on the above two competence-based evaluation metrics, we design the CCL-M algorithm to gradually add new languages into the training set.
Furthermore, we propose a novel competence-aware dynamic balancing sampling method for better selecting training samples at multilingual training.

We evaluate our approach on the multilingual Transformer \citep{vaswani2017attention} and conduct experiments on the TED talks\footnote{\url{https://www.ted.com/participate/translate}} to validate the performance in two multilingual machine translation scenarios, i.e., \emph{many-to-one} and \emph{one-to-many} ("\emph{one}" refers to English).
Experimental results show that our approach brings in consistent and significant improvements compared to the previous state-of-the-art approach \citep{wang-etal-2020-balancing} on multiple translation directions in the two scenarios.

Our contributions\footnote{We release our code on \url{https://github.com/zml24/ccl-m}.} are summarized as follows:
\begin{itemize}
\item
We propose a novel competence-based curriculum learning method for multilingual machine translation.
To the best of our knowledge, we are the first that integrate curriculum learning into multilingual machine translation.
\item
We propose two effective competence-based evaluation metrics to dynamically schedule which languages to learn, and a competence-aware dynamic balancing sampling method for better selecting training samples at multilingual training.
\item
Comprehensive experiments on the TED talks dataset in two multilingual machine translation scenarios, i.e., \emph{many-to-one} and \emph{one-to-many}, demonstrating the effectiveness and superiority of our approach,
which significantly outperforms the previous state-of-the-art approach.
\end{itemize}

\section{Background}

\subsection{Multilingual Machine Translation}
Bilingual machine translation model translates a sentence of source language $S$ into a sentence of target language $T$ (\citealp{sutskever2014sequence}; \citealp{cho-etal-2014-learning}; \citealp{bahdanau2014neural}; \citealp{luong-etal-2015-effective}; \citealp{vaswani2017attention}), which is trained as
\begin{equation}
\theta^* = \argmin_\theta \mathcal{L} (\theta; S, T) ,
\end{equation}
where $\mathcal{L}$ is the loss function, $\theta^*$ is the model parameters.

Multilingual machine translation system aims to train multiple language pairs in a single model, including \emph{many-to-one} (translation from multiple languages into one language), \emph{one-to-many} (translation from one language to multiple languages), and \emph{many-to-many} (translation from several languages into multiple languages) \citep{dabre-etal-2020-multilingual}.
Specifically, we denote the training corpora of $n$ language pairs in multilingual machine translation as $\{S_1, T_1\}$, $\{S_2, T_2\}$, $\dots$, $\{S_n, T_n\}$ and multilingual machine translation aims to train a model $\theta^*$ as
\begin{equation}
\theta^* = \argmin_\theta \frac{1}{n} \sum_{i = 1}^n \mathcal{L} (\theta; S_i, T_i) .
\end{equation}

\subsection{Sampling Methods}

Generally, the size of the training corpora for different language pairs in multilingual machine translation varies greatly.
Researchers hence developed two kinds of sampling methods, i.e., static and dynamic, to sample the language pairs at training \citep{dabre-etal-2020-multilingual}.

There are three mainstream static sampling methods, i.e., uniform sampling, proportional sampling, and temperature-based sampling \citep{arivazhagan2019massively}.
These methods sample the language pairs by the predefined fixed sampling weights $\psi$.

\paragraph{Uniform Sampling.} Uniform sampling is the most straightforward solution \citep{johnson-etal-2017-googles}. The sampling weight $\psi_i$ for each language pair $i$ of this method is calculated as follows
\begin{equation}
\psi_i = \frac{1}{\vert \mathcal{S}_\text{lang} \vert ,}
\end{equation}
where $\mathcal{S}_\text{lang}$ is the language sets for training.

\paragraph{Proportional Sampling.} Another method is sampling by proportion \citep{neubig-hu-2018-rapid}. This method improves the model's performance on high resource languages and reduces the performance of the model on low resource languages. Specifically, we calculate its sampling weight $\psi_i$ for each language pair $i$ as
\begin{equation}
\psi_i = \frac{\vert \mathcal{D}^i_\text{Train} \vert}{\sum_{k \in \mathcal{S}_\text{lang} } \vert \mathcal{D}^k_\text{Train} \vert} ,
\end{equation}
where $\mathcal{D}_\text{Train}$ is the training corpora of language $i$.

\paragraph{Temperature-based Sampling.} It samples the language pairs according to the corpora size exponentiated by a temperature term $\tau$ (\citealp{arivazhagan2019massively}; \citealp{conneau-etal-2020-unsupervised}) as
\begin{equation}
\psi_i = \frac{p_k^{1 / \tau}}{\sum_{k \in \mathcal{S}_\text{lang}} p_k^{1 / \tau}} \ \text{where} \ p_i = \frac{\vert \mathcal{D}^i_\text{Train} \vert}{\sum_{k \in \mathcal{S}_\text{lang} } \vert \mathcal{D}^k_\text{Train} \vert} .
\end{equation}
Obviously, $\tau = \infty$ is the uniform sampling and $\tau = 1$ is the proportional sampling. Both of them are a bit extreme from the perspective of $\tau$.
In practice, we usually select a proper $\tau$ to achieve a balanced result.

On the contrary, dynamic sampling methods (e.g., MultiDDS-S\citep{wang-etal-2020-balancing}) aim to automatically adjust the sampling weights by some predefined rules.

\paragraph{MultiDDS-S.} MultiDDS-S \citep{wang-etal-2020-balancing} is a dynamic sampling method performing differentiable data sampling.
It takes turns to optimize the sampling weights of different languages and the multilingual machine translation model, showing more significant potential than static sampling methods. This method optimizes the sample weight $\psi$ to minimize the development loss as follows
\begin{equation}
\psi^* = \argmin_\psi \mathcal{L} (\theta^*; \mathcal{D}_\text{Dev}) , \\
\end{equation}
\begin{equation}
\theta^* = \argmin_\theta \sum_{i = 1}^n \psi_i \mathcal{L} (\theta; \mathcal{D}_\text{Train}) ,
\end{equation}
where $\mathcal{D}_\text{Dev}$ and $\mathcal{D}_\text{Train}$ denote the development corpora and the training corpora, respectively.

\section{Methodology}

In this section, we first define a directed bipartite language graph, on which we deploy the languages to train.
Then, we define two competence-based evaluation metrics, i.e., the \emph{Self-evaluated Competence} $c$ and the \emph{HRLs-evaluated Competence} $\hat{c}$, to help decide which languages to learn.
Finally, we elaborate the entire CCL-M algorithm.

\subsection{Directed Bipartite Language Graph}

Formally, we define a directed bipartite language graph $G(V, E)$, in which one side is full of HRLs and the other side of LRLs.
Each vertex $v_i$ on the graph represents a language, and the weight of each directed edge (from HRLs to LRLs) $e_{ij}$ indicates the similarity between a HRL $i$ and an LRL $j$:
\begin{equation}
e_{ij} = \text{sim}(i, j) .
\end{equation}
Inspired by TCS \citep{wang-neubig-2019-target}, we measure it using vocabulary overlap and define the language similarity between language $i$ and language $j$ as
\begin{equation}
\text{sim}(i, j) = \frac{\vert \text{vocab}_k(i) \cap \text{vocab}_k(j) \vert}{k} ,
\label{eq:sim}
\end{equation}
where $\text{vocab}_k(\cdot)$ represents the top $k$ most frequent subwords in the training corpus of a specific language.

\begin{figure*}[!t]
\centering
\includegraphics[width=0.9\textwidth]{cl.png}
\caption{Diagram of the CCL-M Algorithm. This graph shows how to gradually add the LRLs to the training set $\mathcal{S}_\text{selected}$ using graph coloring. "aze" stands for Azerbaijani, "bel" stands for Belarusian, etc. The number after the colon indicates current \emph{HRLs-evaluated Competence}, and suppose the corresponding threshold $t$ is set to 0.8. Subfigure (a) represents the state before training. Subfigure (b) indicates that "slk" (Slovak) is added to the training set because the \emph{HRLs-evaluated Competence} is higher than the threshold. Subfigure (c) indicates that "aze" (Azerbaijani) and "glg" (Glacian) are added to the training set, and Subfigure (d) indicates that all the LRLs are added to the training set. Notice we use the abbreviation of language (xxx) to indicate language pairs (xxx-eng or eng-xxx), which is more general.}
\label{ccl-m}
\end{figure*}

\subsection{Competence-based Evaluation Metrics}

\paragraph{Self-evaluated Competence.} We define how well a language itself has been learned as the \emph{Self-evaluated Competence} $c$.
In the following paragraphs, we first introduce the concept of \emph{Likelihood Score} and then give a formula for calculating the \emph{Self-evaluated Competence} in multilingual training based on the relationship between current \emph{Likelihood Score} and the \emph{Likelihood Score} of model trained on bitext corpus.

For machine translation, we usually use the label smoothed \citep{szegedy2016rethinking} cross-entropy loss $\mathcal{L}$ to measure how well the model is trained, and calculate it as
\begin{equation}
\mathcal{L} = - \sum_i p_i \log_2 q_i ,
\end{equation}
where $p$ is the label smoothed actual probability distribution, and $q$ is the model output probability distribution\footnote{We select 2 as the base number for all relevant formulas and experiments in this paper.}.

We find that the exponential of negative label smoothed cross-entropy loss is a likelihood to some extent, which is negatively correlated to the loss.
Since neural network usually optimizes the model by minimizing the loss, we use the likelihood as a positive correlation indicator to measure competence.
Therefore, we define a \emph{Likelihood Score} $s$ to estimate how well the model is trained as follows
\begin{equation}
s = 2^{-\mathcal{L}} = \prod_i q_i^{p_i} .
\end{equation}

Inspired by \citet{jean2019adaptive}, we estimate the \emph{Self-evaluated Competence} $c$ of a specific language by calculating the quotient of its current \emph{Likelihood Score} and baseline's \emph{Likelihood Score}.
Finally, we obtain the formula as follows
\begin{equation} \label{self-competence}
c = \frac{s}{s^*} = 2^{\mathcal{L}^* - \mathcal{L}} ,
\end{equation}
where $\mathcal{L}$ is the current loss on the development set, $\mathcal{L}^*$ is the \emph{benchmark}  loss of the converged bitext model on the development set, and $s$ and $s^*$ are their corresponding \emph{Likelihood Scores}, respectively.

\paragraph{HRLs-evaluated Competence.} Furthermore, we define how well an LRL is ready to be learned as its \emph{HRLs-evaluated Competence} $\hat{c}$.
We believe that each LRL can learn adequate knowledge from its similar HRLs before training.
Therefore, we estimate each LRL's \emph{HRLs-evaluated Competence} by the LRL-specific HRLs' \emph{Self-evaluated Competence}.

Specifically, we propose two methods for calculating the \emph{HRLs-evaluated Competence}, i.e., \emph{maximal} ($\text{CCL-M}_\text{max}$) and \emph{weighted average} ($\text{CCL-M}_\text{avg}$).
The $\text{CCL-M}_\text{max}$ only migrates the knowledge from the HRL that is most similar to the LRL, so we calculate \emph{maximal} \emph{HRLs-evaluated Competence} $\hat{c}_{\text{max}}$ for each LRL $j$ as
\begin{equation}
\hat{c}_{\text{max}}(j)= c_{\argmax_{i \in \mathcal{S}_\text{HRLs}} e_{ij}} ,
\end{equation}
where $\mathcal{S}_{\text{HRLs}}$ is the set of the HRLs.

On the other side, the $\text{CCL-M}_\text{avg}$ method pays attention to all HRLs.
In general, the higher the language similarity, the more knowledge an LRL can migrate from HRLs.
Therefore, we calculate \emph{weighted average} \emph{HRLs-evaluated Competence} $\hat{c}_{\text{avg}}$ for each LRL $j$ as
\begin{equation}
\hat{c}_{\text{avg}}(j)= \sum_{i \in \mathcal{S}_\text{HRLs}} \left ( \frac{e_{ij}}{\sum_{k \in \mathcal{S}_\text{HRLs}} e_{kj}} \cdot c_i \right ) .
\end{equation}

\subsection{The CCL-M Algorithm}

Now we detailly describe the \emph{\textbf{C}ompetence-based \textbf{C}urriculum \textbf{L}earning for \textbf{M}ultilingual Machine Translation}, namely the CCL-M algorithm. The algorithm is divided into two parts: 1) curriculum learning scheduling framework, guiding when to add a language to the training set; 2) competence-aware dynamic balancing sampling, guiding how to sample languages in the training set.

First, we present how to schedule which languages on the directed bipartite language graph should be added to the training set  according to the two competence-based evaluation metrics as shown in Figure \ref{ccl-m} and Algorithm \ref{alg:the_alg}, where $\mathcal{S}_{\text{LRLs}}$ is the set of LRLs, and $f(\cdot)$ is the function calculating the \emph{HRLs-evaluated Competence} $\hat{c}$ for LRLs.
Initialized as Line \ref{lst:1}, we add all languages on the HRLs side to the training set $\mathcal{S}_\text{selected}$ at the beginning of training, leaving all languages on the LRLs side in the candidate set $\mathcal{S}_\text{candidate}$.
Then, we regularly sample the development corpora of different languages and calculate current \emph{HRLs-evaluated Competence} of the languages in the candidate set $\mathcal{S}_\text{candidate}$ as shown in Line \ref{lst:8} and \ref{lst:9}.
Further, the "if" condition in Line \ref{lst:13} illustrates that we would add the LRL to the training set $\mathcal{S}_\text{selected}$ when its \emph{HRLs-evaluated Competence} is greater than a pre-defined threshold $t$.
However, as the calculation of Equation \ref{self-competence}, the upper bound of the \emph{Self-evaluated Competence} for a specific language may not always be 1 at multilingual training.
This may cause that some LRLs remain out of the training set $\mathcal{S}_\text{selected}$ for some thresholds.
To ensure the completeness of our algorithm, we will directly add the languages still in the candidate set $\mathcal{S}_\text{candidate}$ to the training set $\mathcal{S}_\text{selected}$ after a long enough number of steps, which is described between Line \ref{lst:22} and Line \ref{lst:32}.

\begin{algorithm}[!t]
\SetAlgoLined
\KwIn{Randomly initialized model $\theta$; language graph $G$; \emph{benchmark} losses $\mathcal{L}_i^*$; training corpora $\mathcal{D}_\text{Train}$; development corpora $\mathcal{D}_\text{Dev}$;}
\KwOut{The converged model $\theta^*$;}
$\mathcal{S}_\text{selected} \gets \mathcal{S}_\text{HRLs}$, $\mathcal{S}_\text{candidate} \gets \mathcal{S}_\text{LRLs}$,
$\psi \gets 0$\; \label{lst:1}
\For{$i \in \mathcal{S}_\text{\normalfont{selected}}$}{
    $\psi_i \gets \frac{1}{\vert \mathcal{S}_\text{\normalfont{selected}} \vert}$\; \label{lst:3}
}
\While{$\theta$ \normalfont{not converge}}{
    train the model on $\mathcal{D}_\text{Train}$ for some steps with sampling weight $\psi$\;
    \For{$i \in \mathcal{S}_\text{\normalfont{selected}} \cup \mathcal{S}_\text{\normalfont{candidate}}$}{
        sample $\mathcal{D}_\text{Dev}$ and calculate $\mathcal{L}_i$\; \label{lst:8}
        $c_i \gets 2^{\mathcal{L}_i^* - \mathcal{L}_i}$\; \label{lst:9}
    }
    \For{$i \in \mathcal{S}_\text{\normalfont{candidate}}$}{
        $\hat{c}_i \gets f(G, i, c_{\mathcal{S}_\text{HRLs}})$\;
        \If{$\hat{c}_i \geq t$}{ \label{lst:13}
            $\mathcal{S}_\text{selected} \gets \mathcal{S}_\text{selected} \cup \{ i \}$\;
            $\mathcal{S}_\text{candidate} \gets \mathcal{S}_\text{candidate} \setminus \{ i \} $\;
        }
    }
    \For{$i \in \mathcal{S}_\text{\normalfont{selected}}$}{
        $\psi_i \gets \frac{1}{c_i}$\;
    }
 }
\If{$\mathcal{S}_\text{\normalfont{candidate}} \neq \varnothing$}{ \label{lst:22}
    $\mathcal{S}_\text{selected} \gets \mathcal{S}_\text{selected} \cup \mathcal{S}_\text{candidate}$\;
    \While{$\theta$ \normalfont{not converge}}{
        train the model on $\mathcal{D}_\text{Train}$ for some steps with sampling weight $\psi$\;
        \For{$i \in \mathcal{S}_\text{\normalfont{selected}}$}{
            sample $\mathcal{D}_\text{Dev}$ and calculate $\mathcal{L}_i$\;
            $c_i \gets 2^{\mathcal{L}_i^* - \mathcal{L}_i}$\;
            $\psi_i \gets \frac{1}{c_i}$\;
        }
    }
} \label{lst:32}
\caption{The CCL-M Algorithm}
\label{alg:the_alg}
\end{algorithm}

Then, we introduce our competence-aware dynamic balancing sampling method, which is based on the \emph{Self-evaluated Competence}.
For languages in the training set $\mathcal{S}_\text{selected}$, we randomly select samples from the development corpora and calculate their \emph{Self-evaluated Competence}.
Those languages with low \emph{Self-evaluated Competence} should get more attention, therefore we simply assign the sampling weight $\psi_i$ to each language $i$ in the training set to the reciprocal of its \emph{Self-evaluated Competence}, as follows
\begin{equation}
\psi_i \propto \frac{1}{c_i} = 2^{\mathcal{L} - \mathcal{L^*}} .
\end{equation}
Notice that the uniform sampling is used for the training set $\mathcal{S}_\text{selected}$ at the beginning of training as a balancing cold-start strategy.
The corresponding pseudo code can be found in Line \ref{lst:3}.

\section{Experiments}

\begin{table*}[!t]
\centering
{
\centering
\begin{tabular}{l|cc|cc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c|}{\textbf{M2O}} & \multicolumn{2}{c}{\textbf{O2M}} \\
& \textbf{Related} & \textbf{Diverse} & \textbf{Related} & \textbf{Diverse} \\
\midrule
Bitext Models & 20.37 & 22.38 & 15.73 & 17.83 \\
Uniform Sampling $(\tau = \infty)$ & 22.63 & 24.81 & 15.54 & 16.86 \\
Temperature-Based Sampling $(\tau = 5)$ & 24.00 & 26.01 & 16.61 & 17.94 \\
Proportional Sampling $(\tau = 1)$ & 24.88 & 26.68 & 15.49 & 16.79 \\
\midrule
MultiDDS \cite{wang-etal-2020-balancing} & 25.26 & 26.65 & 17.17 & 18.40 \\
MultiDDS-S \cite{wang-etal-2020-balancing} & 25.52 & 27.00 & 17.32 & 18.24 \\
\midrule
$\text{CCL-M}_\text{max}$ (Ours) & 26.59** & 28.29** & \textbf{18.89}** & \textbf{19.53}** \\
$\text{CCL-M}_\text{avg}$ (Ours) & \textbf{26.73}** & \textbf{28.34}** & 18.74** & \textbf{19.53}** \\
\bottomrule
\end{tabular}
}
\caption{Average BLEU scores (\%) on test sets of the baselines and our methods.
$\text{CCL-M}_\text{max}$ is the CCL-M algorithm using \emph{maximal HRLs-evaluated Competence}, $\text{CCL-M}_\text{max}$ is the CCL-M algorithm using \emph{weighted average HRLs-evaluated Competence}.
Bold indicates the highest value.
"$**$" indicates significantly \citep{koehn-2004-statistical} better than MultiDDS-S with t-test $p < 0.01$.
}
\label{tab:results}
\end{table*}

\subsection{Dataset Setup}

Following \citet{wang-etal-2020-balancing}, we use the 58-languages-to-English TED talks parallel data \cite{qi-etal-2018-pre} to conduct experiments.
Two sets of language pairs with different levels of language diversity are selected: \emph{related} (language pairs with high similarity) and \emph{diverse} (language pairs with low similarity).
Both of them consist of 4 high resource languages (HRLs) and 4 low resource languages (LRLs).

For the \emph{related} language set, we select 4 HRLs (Turkish: "tur", Russian: "rus", Portuguese: "por", Czech, "ces") and its related LRLs (Azerbaijani: "aze", Belarusian: "bel", Glacian: "glg", Slovak: "slk"). For the \emph{diverse} language set, we select 4 HRLs (Greek: "ell", Bulgarian: "bul", French: "fra", Korean: "kor") and 4 LRLs (Bosnian: "bos", Marathi: "mar", Hindi: "hin", Macedonian: "mkd") as \citep{wang-etal-2020-balancing}.
Please refer to Appendix for a more detailed description.

We test two kinds of multilingual machine translation scenarios for each set: 1) \emph{many-to-one} (M2O): translating 8 languages to English; 2) \emph{one-to-many} (O2M): translating English to 8 languages.
The data is preprocessed by SentencePiece\footnote{\url{https://github.com/google/sentencepiece}} \citep{kudo-richardson-2018-sentencepiece} with a vocabulary size of 8k for each language.
Moreover, we add a target language tag before the source and target sentences in O2M as \citep{johnson-etal-2017-googles}.

\subsection{Implementation Details} \label{model}

\paragraph{Baseline.} We select three static heuristic strategies: uniform sampling, proportional sampling, and temperature-based sampling ($\tau = 5$), and the bitext models for the baseline.
In addition, we compare our approach with the previous state-of-the-art sampling method, MultiDDS-S \citep{wang-etal-2020-balancing}. All baseline methods use the same model and the same set of hyper-parameters as our approach.

\paragraph{Model.} We validate our approach upon the multilingual Transformer \citep{vaswani2017attention} implemented by fairseq\footnote{\url{https://github.com/pytorch/fairseq}} \citep{ott-etal-2019-fairseq}.
The number of layers is 6 and the number of attention heads is 4, with the embedding dimension $d_{\text{model}}$ of 512 and the feed-forward dimension $d_{\text{ff}}$ of 1024 as \citep{wang-etal-2020-balancing}.
For training stability, we adopt Pre-LN \citep{xiong2020layer} for the layer-norm \citep{ba2016layer} module.
For M2O tasks, we use a shared encoder with a vocabulary of 64k.
Similarly, for O2M tasks, we use a shared decoder with a vocabulary of 64k.

\paragraph{Training Setup.} We use the Adam optimizer \citep{kingma2014adam} with $\beta_1 = \text{0.9}$, $\beta_2 = \text{0.98}$ to optimize the model.
Further, the same learning rate schedule as \citet{vaswani2017attention} is used, i.e., linearly increase the learning rate for 4000 steps to 2e-4 and decay proportionally to the inverse square root of the step number.
We accumulate the batch size to 9,600 and adopt half-precision training implemented by apex\footnote{\url{https://github.com/NVIDIA/apex}} for faster convergence \citep{ott-etal-2018-scaling}.
For regularization, we also use a dropout \citep{srivastava2014dropout} $p = \text{0.3}$ and a label smoothing \citep{szegedy2016rethinking} $\epsilon_{ls} = \text{0.1}$.
As for our approach, we sample 256 candidates from each languages' development corpora every 100 steps to calculate the \emph{Self-evaluated Competence} $c$ for each language and \emph{HRLs-evaluated Competence} $\hat{c}$ for each LRL.

\paragraph{Evaluation.} \label{sec:eval}
In practice, we perform a grid search for the best threshold $t$ in \{0.5, 0.6, 0.7, 0.8, 0.9, 1.0\}, and select the checkpoints with the lowest weighted loss\footnote{This loss is calculated by averaging the loss of each samples in development corpora of all languages, which is equivalent to taking the proportional weighted average of the loss for each language.} on the development sets to conduct the evaluation.
The corresponding early stopping patience is set to 10.
For target sentence generation, we set the beam size to 5 and a length penalty of 1.0.
Following \citet{wang-etal-2020-balancing}, we use the SacreBLEU \citep{post-2018-call} to evaluate the model performance.
In the end, we compare our result with MultiDDS-S using paired bootstrap resampling \citep{koehn-2004-statistical} for significant test.

\subsection{Results}

\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{
\centering
\begin{tabular}{l|cc|cc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c|}{\textbf{Related M2O}} & \multicolumn{2}{c}{\textbf{Diverse M2O}} \\
& \textbf{LRLs} & \textbf{HRLs} & \textbf{LRLs} & \textbf{HRLs} \\
\midrule
Bi. & 10.45 & \textbf{30.29} & 11.18 & \textbf{33.58} \\
MultiDDS-S & 22.51 & 28.54 & 22.72 & 31.29 \\
\midrule
$\text{CCL-M}_\text{max}$ & 23.14* & 30.04** & 23.31* & 33.26** \\
$\text{CCL-M}_\text{avg}$ & \textbf{23.30}* & 30.15** & \textbf{23.55}* & 33.13** \\
\bottomrule
\end{tabular}
}
\caption{Average BLEU scores (\%) on test sets of the HRLs and the LRLs for the best baselines and our methods in M2O tasks. Bitext models (``Bi." for short) and MultiDDS-S are selected from the baselines since ``Bi." performs better on the HRLs and MultiDDS-S performs better on the LRLs. Bold indicates the highest value.
"$*$" and "$**$" indicates significantly better than MultiDDS-S with t-test $p < 0.05$ and $p < 0.01$, respectively.
}
\label{tab:m2o hrl and lrl}
\end{table}

\paragraph{Main Results.} \label{main}
The main results are listed in Table \ref{tab:results}.
As we can see, both methods significantly outperform the baselines and MultiDDS with averaged BLEU scores of over +1.07 and +1.13, respectively, indicating the superiority of our approach.
Additionally, the $\text{CCL-M}_\text{avg}$ is slightly better than the $\text{CCL-M}_\text{max}$ in more cases.
This is because the $\text{CCL-M}_\text{avg}$ can get more information provided by the HRLs, and can more accurately estimate when to add an LRL into the training.
Moreover, we find that O2M tasks are much more complicated than M2O tasks since decoders shared by multiple languages might generate tokens in wrong languages.
Consequently, the BLEU scores of O2M tasks are more inferior than M2O tasks by a large margin.

\paragraph{Results on HRLs and LRLs in M2O.}

We further study the performance of our approach on LRLs and the HRLs in M2O tasks and list the results in Table \ref{tab:m2o hrl and lrl}.
As widely known, the bitext model performs poorly on LRLs while performs well on HRLs.
Also, we find our method performs much better than MultiDDS-S, both on LRLs and HRLs.
Although our method does not strictly match the performance of the bitext model on HRLs, the gap between them is much smaller than that of MultiDDS-S and bitext models.
All of the above proves the importance of balancing learning competencies of different languages.

\paragraph{Results on HRLs and LRLs in O2M.}

\begin{table}[t]
\centering
\resizebox{\columnwidth}{!}{
\centering
\begin{tabular}{l|cc|cc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c|}{\textbf{Related O2M}} & \multicolumn{2}{c}{\textbf{Diverse O2M}} \\
& \textbf{LRLs} & \textbf{HRLs} & \textbf{LRLs} & \textbf{HRLs} \\
\midrule
Bi. & 8.25 & \textbf{23.22} & 7.82 & \textbf{27.83} \\
MultiDDS-S & 15.31 & 19.34 & 13.98 & 22.52 \\
\midrule
$\text{CCL-M}_\text{max}$ & \textbf{16.54}** & 21.24** & \textbf{14.36}* & 24.71** \\
$\text{CCL-M}_\text{avg}$ & 16.33** & 21.14** & 13.82 & 25.42** \\
\bottomrule
\end{tabular}
}
\caption{Average BLEU scores (\%) on test sets of the HRLs and the LRLs for the best baselines and our methods in O2M tasks.
Bitext models (``Bi." for short) and MultiDDS-S are selected from the baselines since ``Bi." performs better on the HRLs and MultiDDS-S performs better on the LRLs. Bold indicates the highest value.
"$*$" and "$**$" indicates significantly better than MultiDDS-S with t-test $p < 0.05$ and $p < 0.01$, respectively.
}
\label{tab:o2m hrl and lrl}
\end{table}

\begin{figure*}[!t]
\centering
\includegraphics[width=0.9\textwidth]{threshold.png}
\caption{Weighted losses on development sets and average BLEU scores (\%) on test sets for different thresholds (the abscissa) in four scenarios. The blue line represents $\text{CCL-M}_\text{max}$, the green line represents $\text{CCL-M}_\text{avg}$.
The yellow dotted line represents MultiDDS-S \citep{wang-etal-2020-balancing}.}
\label{grid}
\end{figure*}

As shown in Table \ref{tab:o2m hrl and lrl}, our approach also performs well on the more difficult scenario, i.e., the O2M.
Apparently, our approach almost doubles the performance of the LRLs from bitext models.
Consistently, there is a roughly -2 and -3 BLEU decay for the HRLs in the \emph{related} and \emph{diverse} language sets.
Compared to MultiDDS-S, both our approach in the LRLs and the HRLs are significantly better.
This again proves the importance of balancing the competencies of different languages.
Additionally, the performance on HRLs in O2M task has a larger drop from the bitext model than that in M2O task.
This is because the decoder shares a 64k vocabulary for all languages in O2M tasks, but each language has only 8k vocabulary.
Thus, it is easier for the model to output misleading tokens that do not belong to the target language during inference.

\section{Analysis}

\subsection{Effects of Different Threshold $t$}
\label{threshold}

We firstly conduct a grid search for the best \emph{HRLs-evaluated Competence} threshold $t$.
As we can see from Figure \ref{grid}, the more HRLs are trained (the larger the threshold $t$ is), the better the model's performance is in M2O tasks.
This phenomenon again suggests that M2O tasks are easier than O2M tasks.
The curriculum learning framework performs better in the \emph{related} set than that in the \emph{diverse} set in M2O tasks, because languages in the \emph{related} set are more similar.
Still, our method is better than MultiDDS-S, as shown in Figure \ref{grid}.
This again demonstrates the positive effect of our curriculum learning framework.

Experimental results also reveal that the optimal threshold $t$ for O2M tasks may not be 1 because more training on HRLs would not produce optimal overall performance.
Furthermore, the optimal threshold for the \emph{diverse} language set is lower than that for the \emph{related} language set as the task in the \emph{diverse} language set is more complicated.

\subsection{Effects of Different Sampling Methods}

\begin{table}[t]
\resizebox{\columnwidth}{!} {
\centering
\begin{tabular}{l|cc|cc}
\toprule
\multirow{2}{*}{\textbf{Method}} & \multicolumn{2}{c|}{\textbf{M2O}} & \multicolumn{2}{c}{\textbf{O2M}} \\
& \textbf{Related} & \textbf{Diverse} & \textbf{Related} & \textbf{Diverse} \\
\midrule
$\text{CCL-M}_\text{avg}$ & 26.73 & 28.34 & \textbf{18.74} & \textbf{19.53} \\
\ \ \ $+$ Uni. & 24.59 & 27.13 & 18.29 & 18.21 \\
\ \ \ $+$ Temp. & 25.28 & 27.50 & 18.65 & 19.28 \\
\ \ \ $+$ Prop. & \textbf{27.21} & \textbf{28.72} & 18.20 & 18.80 \\
\bottomrule
\end{tabular}
}
\caption{Average BLEU scores (\%) on test sets by the $\text{CCL-M}_\text{avg}$ algorithm using our dynamic sampling method and three static sampling methods. "Uni." refers to the uniform sampling, "Temp." refers to the temperature-based sampling ($\tau = 5$), and "Prop." refers to the proportional sampling. Bold indicates the highest value.
}
\label{tab:sample}
\end{table}

We also analyze the effects of different sampling methods.
Substituting our competence-aware dynamic sampling method in the $\text{CCL-M}_\text{avg}$ with three static sampling methods, we get the results in Table \ref{tab:sample}.
Consistently, our method performs best among the sampling methods in O2M tasks, which shows the superiority of sampling by language-specific competence.

Surprisingly, we find that proportional sampling surpasses our proposed dynamic method in M2O tasks.
This also indicates that more training on HRLs has a positive effect in M2O tasks, since proportional sampling would train more on the HRLs than the dynamic sampling we proposed.
In addition, all three static sampling methods outperform their respective baselines in Table \ref{tab:results}. Some of them are even better than the previous state-of-the-art sampling method, i.e., MultiDDS-S.
This shows that our curriculum learning approach has a strong generability.

\section{Related Work}

Curriculum learning was first proposed by \citet{bengio2009curriculum} with the idea of learning samples from easy to hard to get a better optimized model.
As a general method for model improvement, curriculum learning has been widely used in a variety of machine learning fields \citep{gong2016multi, kocmi-bojar-2017-curriculum, hacohen2019power, platanios-etal-2019-competence, narvekar2020curriculum}

There are also some previous curriculum learning researches for machine translation.
For example, \citep{kocmi-bojar-2017-curriculum} divide the training corpus into smaller buckets using some features such as sentence length or word frequency and then train the buckets from easy to hard according to the predefined difficulty.
\citet{platanios-etal-2019-competence} propose competence-based curriculum learning for machine translation, which treats the model competence as a variable in training and samples the training corpus in line with the competence.
In detail, they believes that competence is positively related to the training steps, and uses linear or square root functions for experiments.
We bring the concept of competence and redefine it in this paper with a multilingual context.
Further, we define \emph{Self-evaluated Competence} and \emph{HRLs-evaluated Competence} as the competence of each language pair to capture the model's multilingual competence more accurately.

\section{Conclusion}

In this paper, we focus on balancing the learning competencies of different languages in multilingual machine translation and propose a competence-based curriculum learning framework for this task.
The experimental results show that our approach brings significant improvements over baselines and the previous state-of-the-art balancing sampling method, MultiDDS-S.
Furthermore, the ablation study on sampling methods verifies the great generalibility of our curriculum learning framework.

\section*{Acknowledgements}
We would like to thank anonymous reviewers for their suggestions and comments. This work was supported by the National Key Research and Development Program of China (No. 2020YFB2103402).