
Processors
Processors can mean two different things in the Transformers library:
- the objects that pre-process inputs for multi-modal models such as Wav2Vec2 (speech and text)
  or CLIP (text and vision)
- deprecated objects that were used in older versions of the library to preprocess data for GLUE or SQUAD.
Multi-modal processors
Any multi-modal model will require an object to encode or decode the data that groups several modalities (among text,
vision and audio). This is handled by objects called processors, which group together two or more processing objects
such as tokenizers (for the text modality), image processors (for vision) and feature extractors (for audio).
Those processors inherit from the following base class that implements the saving and loading functionality:
[[autodoc]] ProcessorMixin
Deprecated processors
All processors follow the same architecture which is that of the
[~data.processors.utils.DataProcessor]. The processor returns a list of
[~data.processors.utils.InputExample]. These
[~data.processors.utils.InputExample] can be converted to
[~data.processors.utils.InputFeatures] in order to be fed to the model.
[[autodoc]] data.processors.utils.DataProcessor
[[autodoc]] data.processors.utils.InputExample
[[autodoc]] data.processors.utils.InputFeatures
GLUE
General Language Understanding Evaluation (GLUE) is a benchmark that evaluates the
performance of models across a diverse set of existing NLU tasks. It was released together with the paper GLUE: A
multi-task benchmark and analysis platform for natural language understanding
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched), CoLA, SST2, STSB,
QQP, QNLI, RTE and WNLI.
Those processors are: