What is said about  deep bidirectional representations in the following text?

Language model pre-training has been shown to
be effective for improving many natural language
processing tasks (NAME_1 NAME_2 NAME_3, 2015; NAME_4 NAME_5 NAME_6.,
2018a; NAME_7 NAME_5 NAME_6., 2018; NAME_8 NAME_9,
2018). These include sentence-level tasks such as
natural language inference (NAME_10 NAME_5 NAME_6., 2015;
NAME_11 NAME_5 NAME_6., 2018) NAME_2 paraphrasing (NAME_12
NAME_2 NAME_13, 2005), which aim to predict the re-
lationships between sentences by analyzing them
holistically, as well as token-level tasks such as
named entity recognition NAME_2 question answering,
where models are required to produce fine-grained
output at the token level (NAME_14 NAME_2
De NAME_15, 2003; NAME_16 NAME_5 NAME_6., 2016).
There are two existing strategies for apply-
ing pre-trained language representations to down-
stream tasks: feature-based NAME_2 fine-tuning. The
feature-based approach, such as ELMo (NAME_4
NAME_5 NAME_6., 2018a), uses task-specific architectures that
include the pre-trained representations as addi-
tional features. The fine-tuning approach, such as
the Generative Pre-trained Transformer (OpenAI
GPT) (NAME_7 NAME_5 NAME_6., 2018), introduces minimal
task-specific parameters, NAME_2 is trained on the
downstream tasks by simply fine-tuning all pre-
trained parameters. The two approaches share the
same objective function during pre-training, where
they use unidirectional language models to learn
general language representations.
We argue that current techniques restrict the
power of the pre-trained representations, espe-
cially for the fine-tuning approaches. The ma-
jor limitation is that standard language models are
unidirectional, NAME_2 this limits the choice of archi-
tectures that can be used during pre-training. For
example, in OpenAI GPT, the authors use a left-to-
right architecture, where every token can only at-
tend to previous tokens in the self-attention layers
of the Transformer (NAME_17 NAME_5 NAME_6., 2017). Such re-
strictions are sub-optimal for sentence-level tasks,
NAME_2 could be very harmful when applying fine-
tuning based approaches to token-level tasks such
as question answering, where it is crucial to incor-
porate context from both directions.
In this paper, we improve the fine-tuning based
approaches by proposing BERT: Bidirectional
Encoder Representations from Transformers.
BERT alleviates the previously mentioned unidi-
rectionality constraint by using a “masked lan-
guage model” (MLM) pre-training objective, in-
spired by the Cloze task (NAME_18, 1953). The
masked language model 