Precise meaning of ```d_head``` and ```d_inner```
I am trying to train a transformerXL model from scratch, but I am struggling to understand the meaning of thed_headandd_innerin theconfig.I understandd_headas being the dimension of the value vector after attetion has been applied, but I have no clue whatd_innershould be.The doc only states that:d_inner (int, optional, defaults to 4096) — Inner dimension in FFWhat does FF mean here?
2021-12-14T17:18:12Z
It stands for FeedForward.d_inneris the dimensionality of the hidden layer of the feedforward neural network (FF, FFN, or also called MLP as it’s a multilayer perceptron) inside the layers of the Transformer-XL model.
2021-12-15T09:59:41Z
I see. So just to be clear then: Does this mean that the very first attention layer of the transformer needs an input of dimensiond_embed, and the second attention layer expects an input of dimensionsd_inner?Edit: re-reading the original “attention is all you need” paper, I realize that there are actuallytwofeed-forward layers, and thatd_inneris the dimension of the first one. The second one bring the input to the correct dimension ofd_embedagain.