

Implementation differences with SwitchTransformers
The biggest difference is the way the tokens are routed. NLLB-MoE uses a top-2-gate which means that for each input, only the top two experts are selected based on the 
highest predicted probabilities from the gating network, and the remaining experts are ignored. In SwitchTransformers, only the top-1 probabilities are computed, 
which means that tokens have less probability of being forwarded. Moreover, if a token is not routed to any expert, SwitchTransformers still adds its unmodified hidden 
states (kind of like a residual connection) while they are masked in NLLB's top-2 routing mechanism. 
Generating with NLLB-MoE
The available checkpoints require around 350GB of storage. Make sure to use accelerate if you do not have enough RAM on your machine.
While generating the target text set the forced_bos_token_id to the target language id. The following
example shows how to translate English to French using the facebook/nllb-200-distilled-600M model.
Note that we're using the BCP-47 code for French fra_Latn. See here
for the list of all BCP-47 in the Flores 200 dataset.
thon