Mask-filling: Given a sequence of tables T and a question Q with [MASK], what is the word at [MASK].
Q： Since all three attention types were pruned in transformerbig experiments, the speed-up is considerablethe model is 1.5 times faster with [MASK] BLEU loss.In their paper among many reported models, NAME_1 al. (2018b) a
T:<C> .  wmt14,Model Transformer - big  16,26.7,Model Transformer - big - 8  8,<max>27.2,Model Transformer - big - 4  4,26.1,Model Transformer - big - 2  2,26.0,<C> .  Avg .,Model Transformer - big  16,30.1,Model Transformer - big - 8  8,<max>30.4,Model Transformer - big - 4  4,29.8,Model Transformer - big - 2  2,29.5,<C> .  wmt16,Model Transformer - big  16,33.9,Model Transformer - big - 8  8,<max>34.2,Model Transformer - big - 4  4,34.2,Model Transformer - big - 2  2,33.6,<C> .  wmt15,Model Transformer - big  16,<max>29.8,Model Transformer - big - 8  8,29.7,Model Transformer - big - 4  4,29.0,Model Transformer - big - 2  2,29.0,. Model Heads wmt14 wmt15 wmt16 Avg. Transformer-big 16 26.7 29.8 33.9 30.1 Transformer-big-8 8 27.2 29.7 34.2 30.4 Transformer-big-4 4 26.1 29.0 34.2 29.8 Transformer-big-2 2 26.0 29.0 33.6 29.5 Table 1: A transformer-big with different number of heads for English→German. 