

F
feature extraction
The process of selecting and transforming raw data into a set of features that are more informative and useful for machine learning algorithms. Some examples of feature extraction include transforming raw text into word embeddings and extracting important features such as edges or shapes from image/video data.
feed forward chunking
In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
google-bert/bert-base-uncased).
For an input of size [batch_size, sequence_length], the memory required to store the intermediate feed forward
embeddings [batch_size, sequence_length, config.intermediate_size] can account for a large fraction of the memory
use. The authors of Reformer: The Efficient Transformer noticed that since the
computation is independent of the sequence_length dimension, it is mathematically equivalent to compute the output
embeddings of both feed forward layers [batch_size, config.hidden_size]_0, , [batch_size, config.hidden_size]_n
individually and concat them afterward to [batch_size, sequence_length, config.hidden_size] with n = sequence_length, which trades increased computation time against reduced memory use, but yields a mathematically
equivalent result.
For models employing the function [apply_chunking_to_forward], the chunk_size defines the number of output
embeddings that are computed in parallel and thus defines the trade-off between memory and time complexity. If
chunk_size is set to 0, no feed forward chunking is done.
finetuned models
Finetuning is a form of transfer learning which involves taking a pretrained model, freezing its weights, and replacing the output layer with a newly added model head. The model head is trained on your target dataset.
See the Fine-tune a pretrained model tutorial for more details, and learn how to fine-tune models with 🤗 Transformers.
H
head
The model head refers to the last layer of a neural network that accepts the raw hidden states and projects them onto a different dimension. There is a different model head for each task. For example: